NIST AI RMF Playbook: Measure and Manage suggested actions for safety-critical AI

Why this page: the site has an AI RMF explainer and several crosswalk pages, but no page on the Playbook, which is where the AI RMF’s outcomes turn into things a team can do. This page quotes only the public Playbook text. It does not reproduce the whole Playbook, and the editorial paragraphs are labeled.

What the Playbook is

NIST says the Playbook is a companion AI RMF playbook for voluntary use that suggests ways to navigate and use the AI Risk Management Framework (AI RMF) to incorporate trustworthiness considerations in the design, development, deployment, and use of AI systems. It includes suggested actions, references, and related guidance to achieve the outcomes for the four functions in the AI RMF: Govern, Map, Measure, and Manage. It is not a checklist: Organizations may utilize this information by borrowing as many – or as few – suggestions as apply to their industry use case or interests.

The Playbook is based on AI RMF 1.0 (released on January 26, 2023), and NIST says The Playbook will be updated after the AI RMF 1.0 is revised. Check the page for the current state before citing a subcategory number.

Measure 2.5: validity and limits

The outcome reads The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the conditions under which the technology was developed are documented. Suggested actions include Define the operating conditions and socio-technical context under which the AI system will be validated. and Define and document processes to establish the system’s operational conditions and limits. Another asks teams to Assess and document system variance.

For a certification-minded team, “operational conditions and limits” is the Playbook’s version of an operational design domain statement. That is an editorial reading, not NIST’s wording.

Measure 2.6: safe failure

The outcome says The AI system to be deployed is demonstrated to be safe, its residual negative risk does not exceed the risk tolerance, and can fail safely, particularly if made to operate beyond its knowledge limits. Suggested actions include Stress-test system performance under likely scenarios (e.g., concept drift, high load) and beyond known limitations, in consultation with domain experts. and Document the range of conditions under which the system has been tested and demonstrated to fail safely. Another says Test the system under conditions similar to those related to past known incidents or near-misses and measure system performance and safety characteristics.

Manage 2.4: bypass and deactivation

The outcome says Mechanisms are in place and applied, responsibilities are assigned and understood to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use. Suggested actions include Regularly review established procedures for AI system bypass actions, including plans for redundant or backup systems to ensure continuity of operational and/or business functionality. and Preserve materials for forensic, regulatory, and legal review.

Using it as program evidence (editorial judgment)

This section is the site’s judgment, not NIST’s. A supplier can answer a buyer’s question about validation, safe failure and bypass by pointing to a document set that follows these three subcategories: an operating-conditions statement, a stress-test and near-miss test report, and a bypass procedure with thresholds. That is evidence of process. The Playbook does not say following it satisfies any aviation, DoD or EU requirement. For how it sits next to ISO/IEC 42001 see the crosswalk, and for a generic evidence list see the AI audit evidence checklist.

Questions

Is the Playbook mandatory?

NIST describes it as for voluntary use.

Does it cover generative AI?

Generative AI has its own NIST profile; see NIST AI 600-1 for defense contractors. The Playbook text quoted above is from the AI RMF 1.0 companion.