An open, versioned, publicly-contestable format for scoring AI verification coverage and frontier-system risk.
FRV:0.1.0/CAP:6.3/CTL:4.0/GOV:5.5/DEP:6.0/SCORE:5.6/CONTESTED
Structured AI risk assessment isn't a new idea — the FLI AI Safety Index, Shanghai AI Lab / Concordia AI's Frontier AI Risk Management Framework, NVIDIA's internal frontier-risk assessment process, ForesightSafety Bench, and AIVSS (CVSS extended for agentic vulnerabilities) all already exist, and they're all good. What none of them share is a format. Each publishes its own one-off rubric, in its own report, on its own schedule. You can't take a Shanghai AI Lab score and a FLI grade and put them in the same spreadsheet row, because nothing about either is machine-readable or structured the same way.
That's the CVE/CVSS story before CVSS existed: everyone had a severity opinion, nobody had a shared vector. FRV isn't trying to out-rank those frameworks — it's trying to be the thing that lets a factor from any of them (or a lab's own self-assessment, or an independent auditor's finding) sit in one comparable, evidence-linked, versioned record.
The other half of the gap is narrower and more urgent: AI 2040: Plan A (AI Futures Project) proposes verifying an international AI-training slowdown through inference-only retrofits of datacenters — hardware and recomputation-based checks that a cluster is doing inference, not training. Their own supplement is candid that this is proof-of-concept, not deployed, and their get-involved page is actively recruiting people who can work on it. Amodo Design's verification SITREP is the first attempt at tracking where each of the five inference-only measures actually stands — but it's a blog post. It goes stale the day it's published, and there's no way to file a structured objection to one of its claims. FRV's record format is built to be that tracker's structured, versioned, forkable successor, generalized so it also works for model-level risk.
- Structured, not vibes — a score is a weighted combination of named, individually-justified factors, never a single asserted number.
- Evidence-linked — every factor cites its supporting eval result, red-team report, audit, self-report, or (this is the interchange piece) another framework's own published score.
- Publicly contestable — disputes are data. File a structured challenge against one factor with counter-evidence, and it travels with the record instead of living in someone's inbox.
- Versioned like CVSS — every record is tagged with the methodology version that produced it, so the standard can move without breaking historical comparisons.
Full methodology: spec/METHODOLOGY.md · Record schema: spec/frv-record.schema.json
pip install -e ".[dev]"
frv score examples/records/verification-coverage-datacenter-x.yaml
frv explain examples/records/frontier-model-agentic-coding.yaml
pytestfrv score prints the vector string, and the contested range if any open dispute has a proposed alternative score. frv explain prints the full human-readable breakdown, evidence and all.
demo/index.html is a single, dependency-free file that mirrors the Python scoring logic in JS. Open it in a browser, or point a Hugging Face static Space at the demo/ folder — demo/README.md already has the Space frontmatter.
spec/ the actual standard: methodology + JSON Schema
src/frv/ reference implementation — models, scorer, CLI, markdown renderer
examples/ illustrative (fictional) records — one verification-coverage, one model-risk
tests/ unit tests, plus schema validation of every example
demo/ the browser calculator
examples/records/third-party-encoding-amodo-sitrep.yaml is the odd one out — not fictional. It's an independent FRV encoding of Amodo Design's real, published Plan A Verification SITREP, using every one of Amodo's own stated maturity-stage labels via spec/framework-mappings.md's maturity-stage table. Not authored or reviewed by Amodo — the point of building it was to show what their own status report looks like structured, versioned, and open to a dispute filed against any one factor, not to assert new opinions on their behalf.
structured— shipped in v0.2, seesafety_caseevidence typespec/safety-case.schema.jsonstructured— shipped in v0.2, seecompute_attestationevidence typespec/compute-attestation.schema.jsoncross-record comparison— shipped in v0.2 asfrv diff- Signature verification for
compute_attestation.signature(currently unvalidated) - Reference weight profiles, so scores are comparable across assessors who currently declare their own category weights
recompute-sim models detection probability for the recomputation spot-check mechanism in AI 2040: Plan A's verification supplement — a parameter explorer, not a finished verification system. Its output is exactly the kind of thing a compute_attestation receipt with method: recomputation_spot_check can cite. See examples/records/verification-coverage-with-attestation.yaml for a record that does this end to end.
Two more mature projects on the same account are worth knowing about before treating FRV as the whole picture: Fides (MIT, 108 tests) is a real, working inference-only verification tap graded against Amodo's own SITREP — the kind of thing FRV's examples/records/third-party-encoding-amodo-sitrep.yaml describes structurally, Fides actually implements. ai-2040-verification goes further still (real trained model, real DiFR server, real red-team attacks) but currently has no license — all rights reserved by default until that's fixed. FRV's job is scoring and citing evidence like theirs in a comparable format, not replacing it.
Citing another framework is one thing; converting its scale onto FRV's is the part that actually makes records comparable. src/frv/converters.py does that conversion — frv convert-score from the CLI:
frv convert-score --letter-grade B --framework "FLI AI Safety Index" --domain "existential safety"prints ready-to-paste YAML for a factor's evidence list, plus a suggested Factor.score. The mapping itself — why B is 7.0 and not, say, 6.0 — is documented and disputable in spec/framework-mappings.md, not buried in code.
The format, methodology, and reference scorer are Apache-2.0 — fork it, disagree with the weightings, file an issue. If you want an independent FRV assessment done, a hosted scoring/verification API wired into a release process, or help standing up an internal review pipeline on this schema: [add your contact / issues link before publishing].
v0.1.0. Draft. This is a starting point built to be argued with — see spec/METHODOLOGY.md § 5 for the list of things it doesn't solve yet.
Not affiliated with the AI Futures Project, Shanghai AI Lab, Concordia AI, the Future of Life Institute, or NVIDIA. Their work is cited as evidence sources and prior art, not endorsed as reviewed or approved of this project.
Apache 2.0 — see LICENSE.