Overview of the NIST 2024 AI Risk Management Framework
The NIST 2024 report presents useful guidelines for digital forensics examiners who use generative artificial intelligence (GAI). The recommendations in this work are drawn exclusively from the National Institute of Standards and Technology’s (NIST) Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST states that their publication “provides a set of suggested actions to help organizations govern, map, measure, and manage these risks” (National Institute of Standards and Technology [NIST], 2024, p. 1).
Scope of the NIST Report
The NIST publication describes itself as “a cross-sectoral profile of and companion resource for the AI Risk Management Framework (AI RMF 1.0) for Generative AI” (NIST, 2024, p. 1). Its development was “informed by public feedback and consultations with diverse stakeholder groups” (NIST, 2024, p. 2). The working group’s focus was limited to “Governance, Content Provenance, Pre-deployment Testing, and Incident Disclosure” (NIST, 2024, p. 2).
The NIST publication limits its scope to “risks for which there is an existing empirical evidence base at the time this profile was written” (NIST, 2024, p. 3). It does not present a digital-forensics-specific study or establish procedures for all forms of AI. Accordingly, the following guidelines select the publication’s suggested actions for consideration in digital forensics practice without presenting them as forensic-specific requirements established by NIST.
NIST cautions that “not every suggested action applies to every AI Actor or is relevant to every AI Actor Task” (NIST, 2024, p. 13). The recommendations below retain the source’s wording and action identifiers. Page numbers refer to the publication’s printed pagination, rather than the PDF viewer’s page count.
Recommended Actions
Governance, Authorized Use, and Accountability
Align AI use with applicable legal requirements. “Align GAI development and use with applicable laws and regulations, including those related to data privacy, copyright and intellectual property law” (NIST, 2024, p. 13, Action GV-1.1-001).
Establish acceptable-use policies. “Establish transparent acceptable use policies for GAI that address illegal use or applications of GAI” (NIST, 2024, p. 15, Action GV-1.4-002).
Require performance thresholds before deployment. “Establish minimum thresholds for performance or assurance criteria”. The same action specifies “reviewed processes and approval thresholds reflecting measurement of GAI capabilities and risks” (NIST, 2024, p. 14, Action GV-1.3-002).
Maintain an inventory of AI systems. “Enumerate organizational GAI systems for incorporation into AI system inventory and adjust AI system inventory requirements to account for GAI risks” (NIST, 2024, p. 16, Action GV-1.6-001). Inventory considerations include “human oversight roles and responsibilities” and “underlying foundation models, versions of underlying models, and access modes” (NIST, 2024, p. 16, Action GV-1.6-003).
Retain testing and transparency records. “Maintain a document retention policy to keep history for test, evaluation, validation, and verification (TEVV), and digital content transparency methods for GAI” (NIST, 2024, p. 16, Action GV-1.5-003).
Verification, Accuracy, and Appropriate Reliance
Assess output against known ground truth. NIST directs organizations to “assess the accuracy, quality, reliability, and authenticity of GAI output by comparing it to a set of known ground truth data”. The action also calls for “a variety of evaluation methods” (NIST, 2024, p. 24, Action MP-2.3-001).
Fact-check generated information. “Deploy and document fact-checking techniques to verify the accuracy and veracity of information generated by GAI systems, especially when the information comes from multiple (or unknown) sources” (NIST, 2024, p. 25, Action MP-2.3-003).
Verify sources and citations. “Review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities” (NIST, 2024, p. 31, Action MS-2.5-003).
Do not generalize from anecdotal assessments. “Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments” (NIST, 2024, p. 31, Action MS-2.5-001).
Evaluate capability claims empirically and test in practical settings. “Evaluate claims of model capabilities using empirically validated methods” (NIST, 2024, p. 30, Action MS-2.3-002). NIST also recommends evaluating performance in real-world scenarios to “reveal issues that might not surface in controlled and optimized testing environments” (NIST, 2024, p. 39, Action MS-4.2-002).
Review AI-generated code. “Review GAI system outputs for validity and safety: Review generated code to assess risks that may arise from unreliable downstream decision-making” (NIST, 2024, p. 32, Action MS- 2.6-004).
Content Provenance, Authenticity, and Traceability
Trace the origin and modification of digital content.“Employ methods to trace the origin and modifications of digital content” (NIST, 2024, p.28, Action MS-1.1-001). NIST also recommends maintaining records of third-party changes,“including sources, timestamps, metadata” (NIST, 2024, p. 21, Action GV-6.1-008).
Evaluate tools used to assess provenance and authenticity. “Integrate tools designed to analyze content provenance and detect data anomalies, verify the authenticity of digital signatures, and identify patterns associated with misinformation or manipulation” (NIST, 2024, p. 28, Action MS-1.1-002).
Measure authentication error rates. “Evaluate the rate of false positives and false negatives in content provenance, as well as true positives and true negatives for verification” (NIST, 2024, p. 33, Action MS-2.7-005).
Document content generation, modification, and sharing. NIST recommends digital content transparency solutions that document “each instance where content is generated, modified, or shared” to “provide a tamper-proof history of the content, promote transparency, and enable traceability” (NIST, 2024, p. 34, Action MS-2.8003).
Test methods for identifying synthetic content. “Develop and implement testing techniques to identify GAI produced content (e.g., synthetic media) that might be indistinguishable from human-generated content” (NIST, 2024, p. 25, Action MP-2.3-004).
Privacy, Security, and Third-Party Resources
Monitor for sensitive-data exposure. “Conduct periodic monitoring of AI-generated content for privacy risks; address any possible instances of PII or sensitive data exposure” (NIST, 2024, p. 26, Action MP-4.1-001).
Establish approved providers and assess contractual safeguards. “Inventory all third-party entities with access to organizational content and establish approved GAI technology and service provider lists” (NIST, 2024, p. 21, Action GV-6.1-007). NIST additionally calls for contracts specifying “content ownership, usage rights, quality standards, security requirements, and content provenance expectations” (NIST, 2024, p. 20, Action GV-6.1-004).
Conduct regular adversarial testing. “Implement plans for GAI systems to undergo regular adversarial testing to identify vulnerabilities and potential manipulation or misuse” (NIST, 2024, p. 25, Action MP-2.3005). Security testing should include “GAI attacks (e.g., prompt injection)” and “ML attacks (e.g., adversarial examples/prompts, data poisoning, membership inference, model extraction, sponge examples)” (NIST, 2024, p. 33, Action MS-2.7-007).
Reassess modified models and new applications. “Re-assess model risks a er fine-tuning or retrieval-augmented generation implementation and for any third-party GAI models deployed for applications and/or use cases that were not evaluated in initial testing” (NIST, 2024, p. 42, Action MG-3.1-003).
Practitioner Competence, Bias Assessment, and Incident Management
Assess practitioner understanding of provenance. “Evaluate whether GAI operators and end-users can accurately understand content lineage and origin” (NIST, 2024, p. 25, Action MP-3.4-001). “Adapt existing training programs to include modules on digital content transparency” (NIST, 2024, p. 25, Action MP-3.4-002).
Assess bias and performance disparities. “Conduct fairness assessments to measure systemic bias” (NIST, 2024, p. 36, Action MS-2.11002). The action also directs practitioners to “measure GAI system performance across demographic groups and subgroups” (NIST, 2024, p. 36, Action MS-2.11-002).
Document human overrides.“Monitor and document instances where human operators or other systems override the GAI's decisions” (NIST, 2024, p. 39, Action MS-4.2-004).
Establish fallback and deactivation procedures. “Establish policies and procedures to test and manage risks related to rollover and fallback technologies for GAI systems, acknowledging that rollover and fallback may include manual processing” (NIST, 2024, p. 22, Action GV-6.2-006). NIST also recommends a plan “to halt development or deployment of a GAI system that poses unacceptable negative risk” (NIST, 2024, p. 15, Action GV-1.3-007).
Issues, Challenges and Pitfalls
The following entries identify risks and limitations described in the NIST report. They are not presented as findings from a digital-forensics-specific experiment.
Confidently presented false information and fabricated citations. GAI systems can “generate and confidently present erroneous or false content in response to prompts” (NIST, 2024, p. 6). Outputs may also include “confabulated logic or citations that purport to justify or explain the system’s answer” (NIST, 2024, p. 6).
Overreliance and automation bias. Users may “over-rely on GAI systems or may unjustifiably perceive GAI content to be of higher quality than that produced by other sources” NIST identifies automation bias as “excessive deference to automated systems” (NIST, 2024, p. 9).
Privacy leakage and sensitive-information inference. “Models may leak, generate, or correctly infer sensitive information about individuals”. NIST further warns that inferences can negatively affect individuals “even if the inferences are not accurate” (NIST, 2024, p. 7).
Harmful bias and unequal performance. “Harmful bias in GAI systems can also lead to harms via disparities between how a model performs for different subgroups or languages”. NIST cautions that systems may be “inappropriately trusted to perform similarly across all subgroups” (NIST, 2024, p. 8).
Homogenization and model collapse. “Overly homogenized outputs can themselves be incorrect, or they may lead to unreliable decision-making or amplify harmful biases”. “Model collapse can occur when model training over-relies on synthetic data” (NIST, 2024, p. 9).
Misinformation, disinformation, and diminished trust in evidence. “GAI systems can ease the unintentional production or dissemination of false, inaccurate, or misleading content (misinformation) at scale”. Such content “may erode public trust in true or valid evidence and information” (NIST, 2024, p. 10).
Prompt injection and data poisoning. “Indirect prompt injection attacks occur when adversaries remotely (i.e., without a direct interface) exploit LLM-integrated applications by injecting prompts into data likely to be retrieved”. Data poisoning occurs when an adversary “compromises a training dataset used by a model to manipulate its outputs or operation” (NIST, 2024, p. 11).
Circumvention of safety controls and harmful recommendations. Output restrictions “may still produce harmful recommendations in response to other less explicit, novel prompts”. NIST describes deliberate circumvention as “ʻjailbreaking,ʼ or, manipulating prompts to circumvent output controls” (NIST, 2024, p. 7).
Third-party opacity and benchmark errors. Third-party components “might be improperly obtained or not properly vetted, leading to diminished transparency or accountability for downstream users”. Additionally, “test datasets commonly used to benchmark or validate models can contain label errors” (NIST, 2024, p. 12).
Intellectual-property infringement. “If a GAI system’s training data included copyrighted material, GAI outputs displaying instances of training data memorization . . . could infringe on copyright”. The source also identifies ongoing debate concerning “the use or emulation of personal identity, likeness, or voice without permission” (NIST, 2024, p. 11).
Synthetic abusive imagery and diversion of investigative resources. NIST warns that GAI can facilitate child sexual abuse material and nonconsensual intimate imagery. In discussing synthetic child sexual abuse material, it states that “the prevalence of such images can divert time and resources from efforts to find real-world victims” (NIST, 2024, p. 11).
Dangerous information and offensive capabilities. The source identifies “eased access to or synthesis of materially nefarious information or design capabilities related to chemical, biological, radiological, or nuclear (CBRN) weapons”. It also identifies “lowered barriers for offensive cyber capabilities” (NIST, 2024, p. 4).
Environmental costs and measurement limitations.“Training, maintaining, and operating (running inference on) GAI systems are resource intensive activities, with potentially large energy and environmental footprints”. “Currently there is no agreed upon method to estimate environmental impacts from GAI” (NIST, 2024, p. 8).
Unknown risks and immature assessment methods.“Some GAI risks are unknown and are therefore difficult to properly scope or evaluate”. NIST states that estimation challenges are aggravated by “a lack of visibility into GAI training data” and “the generally immature state of the science of AI measurement and safety” (NIST, 2024, p. 3).
AI Use Statement
Perplexity AI was used in the development of this information.
References
National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.600-1
Perplexity AI. (n.d.). Perplexity [Generative AI tool]. https://www.perplexity.ai/





