Data model for verification methods (follow-up to #101 / #186) #191

Context

#101 identified a gap: control tables say what a control requires, not how to check it is implemented.

#186 piloted this as a free-text verification_method column on Agentic_AIUC1.md. Review on #186 found three problems:

1. Method text is stored per row. The same method repeats across entries and will drift.
2. Review state and citation sit inside prose (DRAFT — … (NPW C02)), so tools can't read them, and no field records who reviewed a method.
3. The pilot used a legacy table: the pilot framework AIUC-1 already publishes their own evidence layer.

Goal

This issue proposes a structured data model for verification methods:

• Hold each verification method once and reuse it across controls and frameworks.
• Store method–control links, with their own review state.
• Track provenance, licensing, and review state.
• Allow both methods from third-party catalogues and methods created in the project.
• Keep Markdown tables readable.
• Non-breaking: files and consumers that don't use verification data are unaffected.

Overview

Two new source files, hand-edited through PRs similar to incidents.json, each with a companion schema:

1. Method file: data/verification-methods.json

1.1 File structure

The file holds the method objects (1.2) in a methods array:

{
  "version": "1.0",
  "description": "Verification methods: how to check a control is implemented. Methods with a `source` are adapted from the cited source; the original wording is in `source.text`.",
  "methods": [ { "...": "method objects, see 1.2" } ]
}

1.2 Method object

1.3 method_type values

Based on the assessment methods in NIST SP 800-53A.

1.4 frequency (optional)

A recommended minimum, not a requirement.

"frequency": { "mode": "continuous" }

frequency is an object, not a plain string, so that details such as interval or triggers can be added later without breaking existing data (section 5).

1.5 Provenance

Provenance is expressed by source:

• source present: the method is derived from the cited source. The source's original wording is kept in source.text.
• source: null: the method was originated in the project.

1.5 source object

1.6 status

Deprecated methods stay in the methods file for consistency, and ID never used for something else.

1.7 Change rules

• Editorial change (clarity, typo; what is tested and the pass criteria are unchanged): same ID. unmodified becomes adapted.
• Substantive change (what is tested, or the pass criteria): a new method with a new ID. The old method becomes deprecated, with status_note naming the replacement (e.g. "Replaced by VM-0031"). Its links should be re-pointed in the same PR (2.1).

1.8 Examples

Adapted from an external catalogue:

{
  "id": "VM-0001",
  "name": "Injection test across ingestion paths",
  "method_type": "test",
  "procedure": "Submit a maintained injection payload set through each distinct ingestion path (user input, retrieved documents, tool output).",
  "expected_result": "No payload alters the agent goal or triggers an unauthorised action; all attempts are logged.",
  "source": {
    "name": "NPW Agentic AI Control Catalogue",
    "version": "2.4.0",
    "id": "C02",
    "url": "https://www.newpacificway.com/ai-controls",
    "license": "CC BY-SA 4.0",
    "text": "Injection test executed through each distinct ingestion path."
  },
  "status": "draft",
  "reviewed_by": [],
  "frequency": { "mode": "event_driven" },
  "evidence": "Test report with payload set version and per-path results."
}

Originated in the project:

{
  "id": "VM-0002",
  "name": "Supply-chain scope of third-party adversarial testing",
  "method_type": "examine",
  "procedure": "Review the scope section of the current third-party adversarial test report.",
  "expected_result": "Tools, connectors and MCP servers used by the agent are explicitly in scope.",
  "source": null,
  "status": "draft",
  "reviewed_by": []
}

2. Links file: data/verification-links.json

2.1 Why a separate links file

Links are stored once, in their own file, rather than inside method records or framework registries for the following reasons:

1. A link needs its own review. "Is this method well written?" and "Does this method verify control X?" are separate judgments.
2. Separate change lifecycles. Editing a method returns it to draft. Linking a reviewed method to a new control does not touch the method record, so its review stands.
3. One place for both levels. Control-level and row-level links live in the same table; a row-level link is a control-level link with the risk entry set.
4. Fewer merge conflicts. Contributors linking the same method to different controls edit independent records, not one shared method object.
5. Registries stay owned by ingest tooling. scripts/ingest-framework.mjs writes a framework registry whole on re-ingest, so links stored on registry controls would be lost on the next sync.

2.2 Control level and row level

A link is applied either to a control (control level) or to risk–control pair (row level). Both levels are needed:

• One control often serves several risks. 94% of 3,706 mapping rows use a control that is also mapped to other risks of the same framework.
• Some methods depend on the risk. Controls that state an outcome ("limit agent actions", "apply a cybersecurity baseline") are verified differently depending on the risk they address. For example, CoSAI WS4-AGT-2.2 "Agent cybersecurity baseline" is mapped to ASI03 (identity: verify authentication), ASI05 (code execution: verify input validation) and ASI07 (inter-agent communication: verify secure channels). These three threats require different verification methods for the same control.
• Some methods don't depend on the risk. Controls that names a specific mechanism ("sign deployed artefacts") can be verified in the same way for every risk.

Rule for contributors: does the method verify the control the same way regardless of the risk?

• Yes: a control-level link, entry_id: null.
• No: a row-level link, entry_id set to the risk entry.

2.3 File structure

The file holds the link objects (2.4) in a links array:

{
  "version": "1.0",
  "links": [
    {
      ...
    },
    {
      ...
    }
  ]
}

2.4 Link object

Combination (method_id, framework, control_id, entry_id) is unique. No separate link ID or status field is needed.

2.5 Link lifecycle

• A new link starts with empty reviewed_by (draft). It counts as reviewed once an SME adds their handle.
• A wrong or obsolete link is deleted; Git keeps the history.
• When a method is deprecated with a replacement, its links are re-pointed to the replacement in the same PR, and their reviewed_by is cleared.
• When a method is deprecated without a replacement, its links are deleted.
• Editing a method does not change its links: the method's review covers its text, and the link's review covers its applicability.

3. Validation rules (scripts/validate.js)

Methods file

1. The methods file validates against verification-methods-schema.json, including enum values, additionalProperties: false and the 80-character name limit.
2. IDs are unique and match ^VM-\d{4}$.
3. reviewed requires a non-empty reviewed_by.
4. deprecated requires status_note.
5. When source is present, all its fields are required.

Links file

6. The links file validates against verification-links-schema.json.
7. The key (method_id, framework, control_id, entry_id) is unique.
8. method_id exists in the methods file. A link to a deprecated method is a warning, not an error.
9. (framework, control_id) resolves to a control in data/frameworks/.
10. When entry_id is set, a mapping row exists for that entry, framework and control IDs.

4. Considerations for future development

elements that can be added later without impacting existing data:

Disclosure: NPW (New Pacific Way Ltd.) is my company, and the NPW Agentic AI Control Catalogue is its catalogue. Methods derived from it will say so in their source field and in the PR.