Test data and schema tests

Test data and schema tests

tests/data/README.md sorts the example data into folders:¹

• valid/ holds YAML fixtures that conform to the datamodel;
• invalid/ holds YAML fixtures that must fail validation, and invalid/cncf-ai-benchmarks/;
• problem/ holds fixtures that document known model quirks not yet handled correctly, split into problem/valid/ and problem/invalid/, which individual tests reference directly;
• cncf-ai-benchmarks/ and luigi_carpio/ hold third-party OSCAL component definitions: see CNCF AI Ecosystem OSCAL Benchmarks and NIST 800-53 Rev 5 to AWS Service Mapping (Luigi Carpio).

A file in valid/ or invalid/ is named ClassName-suffix.yaml. The part before the first hyphen is an exact class name from the schema, and the suffix is a number or any other string a file name allows. CONTRIBUTING.md asks for an invalid example for each constraint.

The examples

tests/data/valid/ holds 36 files and tests/data/invalid/ holds 26 counter-examples. Each of the eight models, from Catalog to MappingCollection, has at least one of each, which CHANGELOG.md describes as an example of each model and a counter-example for each one’s required content.² The counter-examples include Profile-missing-imports.yaml, SystemSecurityPlan-missing-system-characteristics.yaml, MappingCollection-missing-provenance.yaml and AtFrequency-zero-period.yaml. problem/valid/ holds 5 files and problem/invalid/ holds 3.

tests/test_data.py

The module’s docstring says that every class is loaded from oscal.datamodel.oscal, the dataclasses generated from src/oscal/schema/oscal.yaml.³ Its tests do four things:

• load each valid file into the dataclass its name gives, and check it against that class’s definition in the generated JSON Schema, project/jsonschema/oscal.schema.json;
• check that each counter-example fails to load into its dataclass and fails the generated JSON Schema. Four counter-examples are skipped for the dataclass, because the Python model cannot enforce their constraint: AssessmentPlan-bad-last-modified.yaml, CatalogDocument-bad-uuid-only.yaml, AtFrequency-zero-period.yaml and PortRange-negative-start.yaml. A comment says that the dataclasses check no minimum, which the generated JSON Schema does;
• rewrite the hyphenated keys of each CNCF fixture with underscores and check it against ComponentDefinitionDocument in the generated JSON Schema: the valid copies must pass, and the fault-injected copies must fail;
• check that three enums marked allow_any: true carry the annotation, and record as expected failures that the generated enum classes do not yet accept a value outside the declared set.

The docstring also says that some generated classes read hyphenated alias keys, so the loader injects those keys by hand for a few classes, such as Observation, Risk and AssessmentPlan. Validate OSCAL data against the schema says what each check leaves out.

tests/test_schema_rules.py

Its seven tests check that the schema agrees with the OSCAL release it is pinned to, and that the adapt copy’s names stay apart from its neighbours’:⁴

• the header’s oscal_tag, oscal_commit and source match the pins in scripts/check_upstream_release.py, and the tag is v followed by the header’s version;
• every module carries the umbrella’s version, and each module’s source is that release’s JSON schema;
• every module’s id lies under the umbrella’s namespace, and every module’s default prefix is oscal:;
• each enum names its metaschema lists in oscal_allowed_values, and holds exactly their values: see OSCAL metaschema copy;
• no class, enum, type or subset name of .lokf/oscal.yaml is LOKF’s, and a slot name the two share carries LOKF’s IRI;
• no such name is shared with tests/neighbours/slsa.yaml or tests/neighbours/github-actions.yaml, and the only keys shared with them under another IRI are the seven listed: see Element names and the shared namespace.

A comment says that the two neighbours are adapt copies that other lmodel repositories release, each the schema asset of a release checked against its .sha256 and attestation: slsa v0.7.0 and github-actions v0.1.0.

How they run

just test runs five recipes.⁵ _test-schema generates the project into tmp/. _test-python regenerates the Python models and runs pytest. _test-examples runs linkml-run-examples over tests/data/valid/ and tests/data/invalid/, and writes what it converts into examples/output/, which git ignores. The other two come from project.justfile.⁶ _test-linkml-validate runs linkml-validate on every file in valid/ and invalid/ against the module that defines its class, or the umbrella, and on the CNCF fixtures with their keys rewritten. test-luigi-carpio checks the Luigi Carpio fixture. Build and test runs just test on every pull request.

Footnotes

1. tests/data/README.md, the folder list and Naming conventions ↩

2. CHANGELOG.md, the Added items of the Unreleased section ↩

3. tests/test_data.py, the docstring, NON_FAILING_INVALID_FIXTURES and the test functions ↩

4. tests/test_schema_rules.py, the seven test functions and the comment above COPY ↩

5. justfile, the test, _test-schema, _test-python and _test-examples recipes ↩

6. project.justfile, _test-linkml-validate and test-luigi-carpio ↩