CNCF AI Ecosystem OSCAL Benchmarks

CNCF AI Ecosystem OSCAL Benchmarks

tests/data/cncf-ai-benchmarks/ holds 31 files named <Name>-component-definition.json, such as PyTorch-component-definition.json, TensorFlow-component-definition.json and Kubernetes-component-definition.json.¹ Each file’s metadata gives oscal-version 1.2.1, while the schema renders OSCAL 1.2.3. Each file’s remarks say that the component definition is derived from the project’s governance benchmark, as in “Component definition derived from PyTorch Governance Benchmark”.

tests/data/README.md calls them real-world OSCAL component definitions covering CNCF AI ecosystem projects, and treats every file in the folder as a ComponentDefinitionDocument, whatever its prefix.² NOTICE says that they come from oscal-compass-lab/ai-benchmarks, https://github.com/oscal-compass-lab/ai-benchmarks, licensed under the Apache License, Version 2.0.³

Two roles in the tests

• tests/data/cncf-ai-benchmarks/ holds the 31 files as they are, and each must be a valid ComponentDefinitionDocument.
• tests/data/invalid/cncf-ai-benchmarks/ holds copies with the same names and one fault injected in each: metadata.last-modified uses the malformed +0000 time zone offset instead of RFC 3339’s +00:00, which the generated JSON Schema’s date-time pattern rejects. Each copy must fail.

The files use OSCAL’s native hyphenated keys, such as component-definition. tests/test_data.py rewrites their keys with underscores and checks each against ComponentDefinitionDocument in the generated JSON Schema.⁴ The _test-linkml-validate recipe in project.justfile, which just test runs, rewrites them the same way in a temporary copy and runs linkml-validate with src/oscal/schema/oscal_component.yaml.⁵ See Test data and schema tests.

Footnotes

1. tests/data/cncf-ai-benchmarks, the 31 files and their metadata ↩

2. tests/data/README.md, the cncf-ai-benchmarks items and Naming conventions ↩

3. NOTICE, the Test fixtures section ↩

4. tests/test_data.py, test_cncf_ai_benchmarks_match_generated_json_schema and test_cncf_ai_benchmarks_invalid_fail_generated_json_schema ↩

5. project.justfile, the cncf-ai-benchmarks part of _test-linkml-validate ↩