CDAO test and evaluation of Artificial Intelligence models

CDAO’s published T&E documents still say Artificial Intelligence and AI-enabled capabilities. This article uses those names. Where this site speaks in its own voice about current U.S. executive-branch policy it uses Super Intelligence (SI); that voice does not rewrite CDAO titles. For red-teaming language inside the same office, see AI red-teaming requirements for defense programs.

What the Model T&E document is

CDAO’s April 2024 Test and Evaluation of Artificial Intelligence Models Framework is titled What to Consider in a Test & Evaluation Strategy and is marked Cleared for Public Release. It discusses the test and evaluation of AI models, both standalone and integrated into system-of-systems, in a Defense context. It is intended to help AIEC developers and working-level testers incorporate operational realism into testing throughout an AIEC’s lifecycle.

The same pages say what it is not. The document does NOT provide: Binding policy and requirements; T&E at the system-of-systems level; a comprehensive AI Model T&E guide; detailed T&E implementation.

Four focus areas — and what “SI T&E” means here

CDAO organizes the T&E of AI-enabled capabilities into four focus areas that are neither mutually exclusive nor cleanly delineated in real testing:

1. AI Model T&E — evaluating and documenting AI models and data across performance dimensions informed by system and mission constraints. The April 2024 document covers this focus area.
2. Systems Integration (SI) T&E — evaluating an AI component within its larger system to ensure that the AIEC functions as a holistic unit and identify its limitations and risks. In this CDAO document, SI means Systems Integration, not Super Intelligence.
3. Human Systems Integration (HSI) T&E — evaluating an AIEC’s ability to help stakeholders observe and orient to their environment, make informed decisions, and carry out their missions.
4. Operational T&E (OT&E) — evaluating an AIEC performing representative missions within an operationally realistic environment against a realistic adversary.

A Test & Evaluation Strategy is described as a high-level document in DoD acquisitions that guides test planning and execution.

Performance is more than correctness

The framework’s “performance iceberg” says correctness is just the tip of the iceberg. Aspects listed below the water line include bias, explainability, robustness, drift, uncertainty, resilience, representativeness and latency. On bias, it says fairness bias can result in harm to individuals and groups — a violation of the DoD Ethical AI Principles. On data, it says NIST’s AI Risk Management Framework and CDAO’s National AI Infrastructure T&E Capability (NAITIC) Gap study discuss additional differences between software and AI models.

The Department adopted five AI ethical principles in February 2020. Reliable requires safety, security and effectiveness subject to testing and assurance across the life cycle. That is the principle-level hook. It does not turn the April 2024 PDF into a DFARS clause.

Where this sits next to the RAI Pathway and the AIA Toolkit

The RAI Strategy and Implementation Pathway assigns CDAO, with OUSD(R&E) and DOT&E, an AI T&E toolkit that shall include tools and technologies required to detect both natural degradation of and adversarial attacks on AI. That is a departmental line of effort, not a contractor clause.

The live Tradewinds page titles the public self-evaluation resource the AI Assurance (AIA) Toolkit. It helps teams assess their AI projects at any stage of development, ensuring alignment with DoW AI Ethical Principles, and is accessible in both IL2 and IL5 environments. See the CDAO AIA Toolkit listing and DoD RAI Toolkit vs NIST AI RMF.

JATIC software is not an accreditation

The Joint AI Test Infrastructure Capability (JATIC) develops software products for AI Test & Evaluation (T&E) and AI Assurance. It is managed by the DoD Chief Digital and Artificial Intelligence Office (CDAO). The FAQ says the use of our products within your testing and evaluation will not currently provide any certification or accreditation. It adds that they may help provide the necessary documentation required for certain DoD processes such as DoD Directive 3000.09. See the JATIC listing and HEART. For 3000.09’s actual scope, see Autonomy and DoD Directive 3000.09.

NIST’s ARIA Evaluation Planning Manual (AI 200-3) is a separate NIST evaluation-planning method. We found no DoD issuance that incorporates AI 200-3 as a contract requirement.

What we searched for and did not find

We searched the CDAO T&E PDF, the RAI Pathway, the AIA Toolkit page and the JATIC FAQ for a sentence that makes completing the Model T&E framework a CMMC practice or a DFARS clause. We did not find one. The T&E PDF itself says it does not provide binding policy and requirements.

Analysis, not sourced fact: a program can still treat the four-focus-area TES questions as the DoD-facing T&E evidence list, use the AIA Toolkit for ethical-principle self-evaluation, and use JATIC libraries for model-level measurements. That stacking is a reading of the documents, not a requirement we found.

Questions

Does CDAO T&E replace DO-178C for airborne ML?

No source we cite says that. CDAO’s document is a Defense-context T&E strategy primer. Airborne software assurance is still DO-178C / DO-330.

Is “SI T&E” Super Intelligence test and evaluation?

Not in this CDAO document. Systems Integration (SI) T&E is one of four AI-enabled-capability T&E focus areas. Live CDAO pages still say Artificial Intelligence.