AI assurance & verification tools
33 listings are filed under AI assurance & verification. Framework tags on these listings, by count: NIST AI RMF (7), EU AI Act (5), DO-178C / DO-330 (2), EASA AI Concept Paper (2). Most-tagged industries: Cross-industry, Aerospace & aviation, Defense & national security. 15 are open source under an OSI-approved license. A tag appears only when the cited public page names it. Combine this type with other filters.
-
Advai
UK AI testing and monitoring platform for performance, safety, security and robustness, with Defence & Government among named sectors. Platform covers agents, LLMs, computer vision, time series and multimodal systems; Model Arena for vendor selection.
-
Adversarial Robustness Toolbox (ART)
Python library hosted by LF AI & Data for evaluating and defending ML models against evasion, poisoning, extraction and inference. PyPI 1.20.1 (7 July 2025) lists Python 3.10–3.12. Supports TensorFlow, Keras, PyTorch and similar frameworks. MIT-licensed. DARPA HR001120C0013 acknowledgment.
-
Counterfit
MIT-licensed CLI (banner Version 1.1.0) that wraps Adversarial Robustness Toolbox, TextAttack and AugLy to assess ML-system security. Linux or Windows WSL; optional Azure Container Instance. Python 3.8+. Windows and macOS native support is still listed as work in progress.
-
Croissant (MLCommons)
MLCommons metadata format and Apache-2.0 Python library (pip install mlcroissant; Python 3.10+) that describes ML datasets in four layers, plus a v1.0 RAI vocabulary (dataBiases, dataLimitations, dataUseCases). Discoverable via Google Dataset Search; loaders for TensorFlow, JAX and PyTorch.
-
CycloneDX (AI/ML-BOM)
OWASP/Ecma TC54 BOM standard (CycloneDX 1.7, released 21 October 2025) published as ECMA-424 2nd edition (10 December 2025). Homepage names 260+ tools, the defense industrial base, satellite and space systems, and medical-device HBOM/SBOM. The ML-BOM guide names the EU AI Act and NIST AI RMF.
-
DeepTeam
Open-source Python framework for red teaming LLM applications: 120+ vulnerabilities and 20+ single- and multi-turn attack methods, with framework modules for OWASP Top 10 for LLMs 2025, MITRE ATLAS, the EU AI Act and the NIST AI RMF Measure function.
-
ERAN
ETH SRI Lab Apache-2.0 analyzer (DeepZ, DeepPoly, GPUPoly) for complete and incomplete verification of feedforward, convolutional and residual networks, including ACAS Xu. Python 3.6+ and TensorFlow 1.11+. Uses ELINA; Gurobi for RefineZono/RefinePoly.
-
Fiddler AI
AI control plane for agentic and predictive models: continuous evaluations, inline guardrails and governance. Government page names DoD ML/LLM deployment, AWS GovCloud, and air-gapped options. Homepage FAQ names SaaS, VPC or on-premises.
-
garak
NVIDIA LLM vulnerability scanner (Generative AI Red-teaming & Assessment Kit). Probes hallucination, leakage, prompt injection, jailbreaks and more. Generators include Hugging Face, OpenAI, Bedrock, NIM and REST. pip or conda, Python 3.11–3.13. Apache-2.0.
-
Giskard
LLM-agent testing (Python SDK and Hub UI), red-teaming and runtime Guards with an EU AI Act Policy Pack (OPA/Rego). On-premise Hub for public-sector and defense workloads. Named sectors: banking and insurance.
-
HEART (Hardened Extension of ART)
MIT-licensed Python 3.9–3.11 library (pip install heart-library; 0.7.0) that extends ART for DoD T&E. Modules: attacks, estimators, metrics. Aligns to MAITE; depends on ART ≥1.20.1. CDAO contract W519TC-23-9-2037. Optional Gradio front end.
-
HiddenLayer AI Security Platform
AI security platform with government and defense pages: NIST AI RMF alignment, contractor and open-source model scanning, real-time monitoring, and air-gapped options for classified disconnected environments. The homepage also names AI Discovery, supply-chain, attack simulation and runtime security.
-
Hugging Face Dataset Cards
Hub dataset documentation: each dataset README.md is a dataset card with YAML metadata. Force-modality tags (3d, audio, geospatial, image, tabular, text, timeseries, video) and RL-environment tags describe compatibility; they do not create a new repository type.
-
Hugging Face Model Cards
Hub model cards are README.md files with YAML metadata (license, datasets, library_name, pipeline_tag, model-index evals). CDAO's Generative AI RAI Toolkit names the Hugging Face model-card template as a documentation tool for traceability.
-
IARPA TrojAI
IARPA program (BAA W911NF-19-S-0012, closed) and NIST T&E resources for detecting Trojan/backdoor attacks in trained AIs before deployment. T&E partners: JHU APL, NIST, SEI and Sandia. NIST still hosts a leaderboard and baseline detector; that page is no longer updated.
-
Inspect (UK AI Security Institute)
UK AISI MIT-licensed LLM evaluation framework (pip install inspect-ai; Python ≥3.10). 200+ ready-to-run evals, 20+ model providers, and sandboxes for Docker, Kubernetes, Modal, Proxmox and Vagrant.
-
Marabou
SMT-based neural-network verifier (Marabou 2.0). ONNX main format; also .nnet, TensorFlow and VNNLIB. pip install maraboupy (Python 3.8–3.11). Default: DeepPoly bound tightening and DeepSoI. Optional Gurobi, parallelism and Split-and-Conquer. ACAS Xu example. Research product.
-
MathWorks AI Verification Library for Deep Learning Toolbox
Formal robustness verification, output bounds, adversarial examples and out-of-distribution detection for MATLAB, ONNX and PyTorch networks, with airborne runway-sign and ACAS Xu worked examples.
-
MathWorks DO Qualification Kit
Documentation, test cases and procedures to qualify supported Simulink and Polyspace tools for DO-178C, DO-278A, DO-254 and DO-330. The FAQ says the kit does not prequalify tools; qualification is project-specific.
-
Model Card Toolkit
Archived Python toolkit (pip model-card-toolkit, Python 3.7+) that generates Model Cards as HTML from proto/JSON. The TFX ModelCardGenerator moved to tfx-addons from v2.0.0. Apache-2.0.
-
NASA ProgPy
NASA ProgPy 1.8.0 (2024 NASA Software of the Year): pip install progpy; Python ≥3.9,<3.14. PHM and remaining-useful-life research with an optional LSTM datadriven extra and a prog_server SOA. Partners named: DLR, Northrop Grumman and Vanderbilt. NASA-1.3 — no OSI badge.
-
NIST Dioptra
NIST test platform (docs and release 1.2.1) for assessing trustworthy characteristics of AI models. The official README says it supports the Measure function of the NIST AI RMF. Docker images for nginx, MLflow, REST API and CPU workers. Public source under CC BY 4.0.
-
nnenum
Python 3.9+ neural-network verifier (pip install .; nnenum 1.0.1, GPL-3.0-or-later) for ReLU networks. Combines zonotopes with star sets, uses the ImageStar trick through ONNX Runtime, and ships ACAS Xu ONNX/vnnlib examples. VNN-COMP 2020: only tool to verify all ACAS-Xu benchmarks.
-
NNV (Neural Network Verification Toolbox)
MATLAB toolbox for set-based neural-network verification using Star sets and zonotopes. 10+ architectures and 48 layer types; ONNX and VNNLIB. Official aerospace domain and all 45 ACAS Xu networks. AFOSR, DARPA and NSF.
-
Promptfoo
Open-source gen-AI red-teaming tool that scans 50+ vulnerability types and maps plugins to the NIST AI RMF, ISO/IEC 42001 and EU AI Act. Runs locally against HTTP APIs, browsers or models. MIT-licensed.
-
PyRIT
Microsoft AI Red Team framework for automated and human-led generative-AI red teaming. Python 3.13 recommended; pip install pyrit. Ships Crescendo, TAP and Skeleton Key, CoPyRIT GUI, pyrit_scan/pyrit_shell, and SQLite or Azure SQL memory. MIT-licensed.
-
QuantPi
Enterprise AI validation layer that tests models, agents and physical AI before and after deployment. Public cloud, on-premises via HPE Private Cloud AI, and air-gapped. MCP integration; SSO/RBAC and zero data retention. TÜV Süd’s first CARAT partner laboratory worldwide.
-
Re:In (Moasoftware)
Data-quality platform the vendor maps to ARP6983/ED-324 §§6.2–6.5 (bias, diversity, sufficiency, balance, augmentation, reliability reports). W-Shape four stages. Dated 25 August 2026; updated 6 October 2026.
-
Repello AI
AI security platform: Inventory and Workstation Lens discover AI applications, coding agents and MCP servers; ARTEMIS red-teams prompt injection, data disclosure and unsafe tool use; ARGUS applies runtime policies with reviewable events.
-
Resaro Approved Intelligence Platform
On-premise, air-gapped TEVV platform that produces ASQI-structured deployment evidence for civil and defence AI. CI/CD-native workflows over text and image; used in the IMDA/AI Verify Global AI Assurance Pilot.
-
Saimple
Static analyzer that uses abstract interpretation to assess neural-network robustness, citing ISO/IEC 24029. Sold as SaaS and on-premises. Numalis names Article 15 explainability reports and EASA MLEAP work.
-
Sigstore model-transparency
Python CLI and API (model_signing) that signs ML models with Sigstore, keys or certificates so users can verify integrity and provenance. DSSE in-toto bundle; optional PKCS #11. Apache-2.0.
-
SPDX AI Profile
SPDX 3.0.1 AI Profile for documenting AI packages, models and datasets. Conformant AIPackage elements must carry hasConcludedLicense and hasDeclaredLicense. Records training, limitations, safety-risk level (serious/high/medium/low) and energy use by stage.