Skip to content

C10: draft dataset quality gate and corruption evidence #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
SatyaA27 wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

C10: draft dataset quality gate and corruption evidence #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
70b082d
Add draft C10 dataset conformance quality gate
SatyaA27 Oct 16, 2026
8440f4b
Reject orphan episode rows in shared native Parquet files
SatyaA27 Oct 16, 2026
34fd6de
Retain reproducible C10 CLI and native test evidence
SatyaA27 Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions .github/workflows/c10-dataset-quality.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: C10 reference quality gate

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

jobs:
c10-reference:
runs-on: ubuntu-24.04
defaults:
run:
shell: bash
env:
C10_CHECKER_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
C10_EVIDENCE_DIR: .verification/c10-evidence
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
persist-credentials: false
- name: Prepare evidence directory
run: |
mkdir -p "$C10_EVIDENCE_DIR"
printf '%s\n' "$C10_CHECKER_SHA" > "$C10_EVIDENCE_DIR/expected-checker-sha.txt"
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install scoped C10 verification dependencies
run: |
python -m pip install -r tools/requirements-c10-native.txt
sudo apt-get update
sudo apt-get install -y --no-install-recommends ffmpeg
- name: Run nonempty C10 core suite
if: ${{ !cancelled() }}
run: |
set -o pipefail
python tools/run_verification_tests.py tests test_c10_dataset.py 2>&1 | tee "$C10_EVIDENCE_DIR/core-suite.log"
- name: Run nonempty C10 native suite
if: ${{ !cancelled() }}
run: |
set -o pipefail
python tools/run_verification_tests.py tests test_c10_native.py 2>&1 | tee "$C10_EVIDENCE_DIR/native-suite.log"
- name: Run nonempty C10 evidence suite
if: ${{ !cancelled() }}
run: |
set -o pipefail
python tools/run_verification_tests.py tests test_c10_evidence.py 2>&1 | tee "$C10_EVIDENCE_DIR/evidence-suite.log"
- name: Prove reverted arithmetic check is detected
if: ${{ !cancelled() }}
run: |
set -o pipefail
python tools/prove_c10_negative.py 2>&1 | tee "$C10_EVIDENCE_DIR/revert-negative.log"
- name: Generate reproducible synthetic CLI evidence
if: ${{ !cancelled() }}
run: |
set -o pipefail
python tools/generate_c10_evidence.py \
--output-dir "$C10_EVIDENCE_DIR/synthetic" \
--expected-checker-sha "$C10_CHECKER_SHA" \
2>&1 | tee "$C10_EVIDENCE_DIR/generation.log"
- name: Upload C10 report and test evidence
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: c10-evidence-${{ github.event.pull_request.head.sha || github.sha }}-${{ github.run_attempt }}
path: .verification/c10-evidence/
include-hidden-files: true
if-no-files-found: error
retention-days: 30
101 changes: 101 additions & 0 deletions docs/c10-check-fixture-matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
# C10 check-to-corrupted-fixture acceptance matrix

Generated from the named cases in `tests/test_c10_dataset.py` and
`tests/test_c10_native.py`. Core corruption cases remain valid against the
unchanged C9 Draft 2020-12 schema. Wrapper cases deliberately corrupt native
artifacts or exercise missing/contradictory evidence. No real recording is altered.

| Stable diagnostic | Deliberate corruption / missing-evidence case | Expected | Named unittest |
| --- | --- | --- | --- |
| `C10_ACTION_BOUNDS` | `action_range` | REJECT | `C10SemanticTests.test_reject_action_range` |
| `C10_ACTION_LATENCY` | `action_timestamp_semantics` | REJECT | `C10SemanticTests.test_reject_action_timestamp_semantics` |
| `C10_ACTION_LATENCY` | `action_value_semantics` | REJECT | `C10SemanticTests.test_reject_action_value_semantics` |
| `C10_ACTION_LATENCY` | `command_latency` | REJECT | `C10SemanticTests.test_reject_command_latency` |
| `C10_BINDING` | `binding_revision` | REJECT | `C10SemanticTests.test_reject_binding_revision` |
| `C10_BINDING` | `episode_length` | REJECT | `C10SemanticTests.test_reject_episode_length` |
| `C10_BINDING` | `external_native_identity_contradiction` | REJECT | `C10SemanticTests.test_reject_external_native_identity_contradiction` |
| `C10_BINDING` | `format_version` | REJECT | `C10SemanticTests.test_reject_format_version` |
| `C10_BINDING` | `orphan_native_episode` | REJECT | `C10SemanticTests.test_reject_orphan_native_episode` |
| `C10_COMPLETENESS` | `interrupted_recording` | REJECT | `C10SemanticTests.test_reject_interrupted_recording` |
| `C10_COMPLETENESS` | `success_with_abort_event` | REJECT | `C10SemanticTests.test_reject_success_with_abort_event` |
| `C10_COVERAGE` | `duplicate_frame` | REJECT | `C10SemanticTests.test_reject_duplicate_frame` |
| `C10_COVERAGE` | `frame_coverage` | REJECT | `C10SemanticTests.test_reject_frame_coverage` |
| `C10_COVERAGE` | `native_frame_index` | REJECT | `C10SemanticTests.test_reject_native_frame_index` |
| `C10_EVENTS` | `duplicate_event_id` | REJECT | `C10SemanticTests.test_reject_duplicate_event_id` |
| `C10_EVENTS` | `event_missing_frame` | REJECT | `C10SemanticTests.test_reject_event_missing_frame` |
| `C10_EVENTS` | `event_outside_recording` | REJECT | `C10SemanticTests.test_reject_event_outside_recording` |
| `C10_EVENTS` | `inverted_event` | REJECT | `C10SemanticTests.test_reject_inverted_event` |
| `C10_FEATURES` | `camera_fps_descriptor` | REJECT | `C10SemanticTests.test_reject_camera_fps_descriptor` |
| `C10_FEATURES` | `feature_shape` | REJECT | `C10SemanticTests.test_reject_feature_shape` |
| `C10_FEATURES` | `legacy_camera_aliases` | REJECT | `C10SemanticTests.test_reject_legacy_camera_aliases` |
| `C10_FEATURES` | `native_vector_name_order` | REJECT | `C10SemanticTests.test_reject_native_vector_name_order` |
| `C10_FEATURES` | `native_vector_shape` | REJECT | `C10SemanticTests.test_reject_native_vector_shape` |
| `C10_HEALTH` | `fresh_reused_sample` | REJECT | `C10SemanticTests.test_reject_fresh_reused_sample` |
| `C10_HEALTH` | `fresh_without_receipt` | REJECT | `C10SemanticTests.test_reject_fresh_without_receipt` |
| `C10_MAPPING` | `affine_arithmetic` | REJECT | `C10SemanticTests.test_reject_affine_arithmetic` |
| `C10_MAPPING` | `mapping_validity_interval` | REJECT | `C10SemanticTests.test_reject_mapping_validity_interval` |
| `C10_PROFILE` | `profile_dimension_order` | REJECT | `C10SemanticTests.test_reject_profile_dimension_order` |
| `C10_PROFILE` | `profile_units` | REJECT | `C10SemanticTests.test_reject_profile_units` |
| `C10_SELECTION` | `selected_time_disagrees_with_mapping` | REJECT | `C10SemanticTests.test_reject_selected_time_disagrees_with_mapping` |
| `C10_SELECTION` | `selection_source` | REJECT | `C10SemanticTests.test_reject_selection_source` |
| `C10_SEQUENCE` | `nonconsecutive_source_id_reuse` | REJECT | `C10SemanticTests.test_reject_nonconsecutive_source_id_reuse` |
| `C10_SEQUENCE` | `sequence_reset` | REJECT | `C10SemanticTests.test_reject_sequence_reset` |
| `C10_SKEW` | `reported_skew` | REJECT | `C10SemanticTests.test_reject_reported_skew` |
| `C10_STREAM_QUALITY` | `measured_source_fps` | REJECT | `C10SemanticTests.test_reject_measured_source_fps` |
| `C10_STREAM_QUALITY` | `sequence_gap` | REJECT | `C10SemanticTests.test_reject_sequence_gap` |
| `C10_STREAM_QUALITY` | `stale_observation_age` | REJECT | `C10SemanticTests.test_reject_stale_observation_age` |
| `C10_SYNC` | `declared_policy_tolerance_mismatch` | REJECT | `C10SemanticTests.test_reject_declared_policy_tolerance_mismatch` |
| `C10_SYNC` | `inter_observation_skew` | REJECT | `C10SemanticTests.test_reject_inter_observation_skew` |
| `C10_TASK` | `orphan_native_task` | REJECT | `C10SemanticTests.test_reject_orphan_native_task` |
| `C10_TASK` | `task_text` | REJECT | `C10SemanticTests.test_reject_task_text` |
| `C10_TIME_ORDER` | `descending_authoritative_time` | REJECT | `C10SemanticTests.test_reject_descending_authoritative_time` |
| `C10_TIME_ORDER` | `timestamp_order` | REJECT | `C10SemanticTests.test_reject_timestamp_order` |
| `C10_VECTOR_VALUES` | `vector_infinity` | REJECT | `C10SemanticTests.test_reject_vector_infinity` |
| `C10_VECTOR_VALUES` | `vector_nonfinite` | REJECT | `C10SemanticTests.test_reject_vector_nonfinite` |
| `C10_VECTOR_VALUES` | `vector_width` | REJECT | `C10SemanticTests.test_reject_vector_width` |
| `C10_ARTIFACT` | `absent_configuration_artifact` | NOT_VERIFIED | `C10WrapperTests.test_wrapper_absent_configuration_artifact` |
| `C10_ARTIFACT` | `artifact_bytes_changed` | REJECT | `C10WrapperTests.test_wrapper_artifact_bytes_changed` |
| `C10_ARTIFACT` | `configuration_declared_hash` | REJECT | `C10WrapperTests.test_wrapper_configuration_declared_hash` |
| `C10_ARTIFACT` | `configuration_symlink_escape` | REJECT | `C10WrapperTests.test_wrapper_configuration_symlink_escape` |
| `C10_CALIBRATION` | `calibration_identity` | REJECT | `C10WrapperTests.test_wrapper_calibration_identity` |
| `C10_CALIBRATION` | `calibration_residual_exceeds` | REJECT | `C10WrapperTests.test_wrapper_calibration_residual_exceeds` |
| `C10_CALIBRATION` | `unapproved_calibration` | NOT_VERIFIED | `C10WrapperTests.test_wrapper_unapproved_calibration` |
| `C10_INPUT` | `duplicate_json_key` | REJECT | `C10WrapperTests.test_wrapper_duplicate_json_key` |
| `C10_INPUT` | `malformed_nested_policy` | REJECT | `C10WrapperTests.test_wrapper_malformed_nested_policy` |
| `C10_NATIVE_DATA` | `native_row_range` | REJECT | `C10WrapperTests.test_wrapper_native_row_range` |
| `C10_NATIVE_DATA` | `orphan_native_episode_rows` | REJECT | `C10WrapperTests.test_wrapper_orphan_native_episode_rows` |
| `C10_NATIVE_DATA` | `truncated_parquet` | REJECT | `C10WrapperTests.test_wrapper_truncated_parquet` |
| `C10_NATIVE_METADATA` | `duplicate_native_episode` | REJECT | `C10WrapperTests.test_wrapper_duplicate_native_episode` |
| `C10_PARQUET_DTYPE` | `native_double_vectors` | REJECT | `C10WrapperTests.test_wrapper_native_double_vectors` |
| `C10_PROFILE` | `source_profile_identity` | REJECT | `C10WrapperTests.test_wrapper_source_profile_identity` |
| `C10_REAL_EPISODE` | `synthetic_is_not_real_evidence` | NOT_VERIFIED | `C10WrapperTests.test_wrapper_synthetic_is_not_real_evidence` |
| `C10_RELEASE` | `recording_release_commit` | REJECT | `C10WrapperTests.test_wrapper_recording_release_commit` |
| `C10_RELEASE` | `unapproved_release` | NOT_VERIFIED | `C10WrapperTests.test_wrapper_unapproved_release` |
| `C10_SCHEMA` | `invalid_c9_schema` | REJECT | `C10WrapperTests.test_wrapper_invalid_c9_schema` |
| `C10_VIDEO` | `decoded_video_frame_count` | REJECT | `C10WrapperTests.test_wrapper_decoded_video_frame_count` |
| `C10_VIDEO` | `frozen_decoded_pixels` | REJECT | `C10WrapperTests.test_wrapper_frozen_decoded_pixels` |

Additional explicit tests cover clean semantic PASS, independent fixture objects,
inclusive action/synchronization/clock-map boundaries, ties-to-even mapping,
exact JSON/YAML fractional clock lexemes, pending tolerance with an approved
policy, pending/unmapped clocks, honest missing/dropped slots, stale producer
labels requiring review, missing episode identity, source-profile identity,
explicit action reference offsets, unavailable native data, native metadata
totals, strict JSON and nonfinite vectors, CLI exits and unchanged dataset bytes.

`test_every_core_diagnostic_has_named_corruption` fails if a semantic diagnostic
is added without a named corruption. All CLI diagnostic families also appear
above. The acceptance matrix proves diagnostic-specific fault detection, not
independent correctness of every algorithm or physical acceptance.

`test_real_video_decode_or_explicit_unavailable` decodes a temporary synthetic
MP4 and corrupts its bytes when ffmpeg/ffprobe are available. Without them it
explicitly asserts NOT_VERIFIED and prints which branch was exercised. It does
not use unittest skip. The decoded-frame-count and frozen-pixel tests use a
deterministic subprocess double and are distinguished from actual decoding.

The revert demonstration uses `python tools/prove_c10_negative.py`: one unchanged
negative test passes on current code, then fails with exactly one assertion
failure after the arithmetic comparison is reverted in a temporary source copy.
The original checkout and fixtures remain intact.
42 changes: 42 additions & 0 deletions docs/c10-compatibility-matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# C10 compatibility review

Status: **Draft conformance reference, pending architecture review.** This review
does not approve a runtime home, source profile, release pin or numeric threshold.
The reference tooling under `tools/` and its synthetic `tests/` exercise the
accepted C9 contract; recorder, ingestion and training implementation ownership
remains unresolved.

The C9 v1 shape was accepted in [issue #18](https://github.com/openAMRobot/openamrobot-interfaces/issues/18#issuecomment-5919324379)
and [PR #16](https://github.com/openAMRobot/openamrobot-interfaces/pull/16)
merged on 2 October 2026 as `fa7c438e33bd807c174fa0747119d7d627faa3bf`.
The older Proposed label in the C9 document does not override that decision.
Comparison baseline: [dataset tooling](https://github.com/4u-79/openamrobot-c-baseline/tree/a29ce7c89b7c42c9147ad81e6a751de9dad045c9),
commit `a29ce7c89b7c42c9147ad81e6a751de9dad045c9`.

| Area | Inspected baseline behavior | Accepted C9 v1 / reconciliation needed |
| --- | --- | --- |
| Native format | Calls its layout v3.0; expects JSONL task/episode registries and one Parquet/video file per episode. | Native LeRobot v3.0 remains primary; sidecar at `meta/openamrobot/episodes/episode-{episode_index:06d}.json`. Resolve actual native metadata/file bindings using the chosen writer; the label alone proves no loader compatibility. |
| Cameras | Wrist keys agree; requires `observation.images.camera_scene` and `observation.images.camera_base_nav` with fixed shapes. | Requires `observation.images.head` and `observation.images.base` plus wrists; additional explicitly bound cameras allowed. An approved source mapping must resolve the differing keys and physical/stereo identity; no silent aliases. |
| State/action | Exactly 19 channels; historical fixed-mast rev18.2 profile, mixed base velocity, joint radians and normalized grippers; zero-limit torso channels. Both validator constants are 19; README and validator docstring retain stale 21-channel wording. | Named ordered dimensions, units, frames and action semantics control interpretation. The supplied OpenArm development profile has 16 positions in degrees. Neither vector length nor latest URDF authorizes conversion. The historical profile must remain distinct from the current lift/arm configuration. |
| Source profiles | Enum `ROBOT`, `GAZEBO`, `ISAAC`, `CALIBRATION`, `DIAGNOSTICS`; defaults to requiring `ROBOT`. | Source-profile identity and version must describe the actual configuration, serialization, clocks and units. The enum alone establishes none of these facts. Synthetic or simulation evidence cannot establish physical acceptance. |
| Clock representation | Two frame-level float64 seconds columns: host monotonic `timestamp` and `timestamp_device`. Integrity gate assumes epoch alignment; contract validator instead removes the median host/device offset. | Recording-relative integer nanoseconds; each stream retains raw device/host receipt evidence and explicit affine mapping, unit, source domain and validity interval. A host receipt is an arrival proxy. Reconstructing absent acquisition evidence from native FPS is invalid. |
| Skew | Integrity gate: maximum absolute host/device difference. Contract validator: maximum deviation of that difference from its median. These measure different quantities. | Recompute selected time using the declared mapping and ties-to-even nanosecond rounding. Recompute per-stream absolute skew against observation reference; inter-observation skew is max minus min across cameras/state. Assess action latency separately. Missing/unmapped timing is unassessable, never zero. |
| Thresholds | Defaults: 30 ms skew, 30 frames minimum, camera container FPS at least 95% of configured FPS; frozen run at least 2 s and phase lag beyond 3 frames produce nonblocking flags. | These are baseline parameters, not accepted C9 release limits. C9 pending tolerance is null; a declared value plus evidence does not prove approval. An explicit reviewed policy must supply numeric acceptance values. Container FPS cannot establish acquisition freshness. |
| Range checks | Hard-coded 19-channel limits, 5% of span margin; zero-span channels use absolute `1e-4` tolerance. `FIX-OOB-STATS` injects NaN/Inf rather than a finite range violation. | Check finite values and profile-specific bounds separately, by dimension name/unit and immutable configuration evidence. Do not apply radians/normalized-gripper limits to the degrees profile. |
| Calibration/configuration | README describes hashes; inspected gate attaches source profile and declared length from the episode registry, without checking C9 calibration/configuration artifacts. | Each camera needs calibration ID/version and structured measured residual or explicit not-measured reason. Configuration hashes identify exact artifact bytes and serialization. Checking identifier shape/equality does not verify artifact existence, contents or physical accuracy. |
| Provenance/pins | `requirements.txt` pins `lerobot==0.4.1`; README describes a future release manifest. | Inspected upstream snapshot is v0.6.1, `7e241bd630a3719a56157a497ce5d08f244784f1`; this is not a validated release pin. Record actual recorder version/commit separately; retain unknowns and null validated pin until release-owner approval. |
| Completeness/events | Eight fault classes cover jitter, missing video frames, phase lag, repeated images, broken Parquet, vector width, nonfinite values and orphan task. Most operate on in-memory synthetic records. | Bind every native frame/task/episode and stream source sample; verify recording boundaries/completeness, independent outcome and event linkage. Synthetic fixtures do not demonstrate native dataset loading, media decoding or real capture. |

The [13 October integration decision](https://github.com/openAMRobot/openamrobot-interfaces/issues/18#issuecomment-6152412257)
requires camera mappings, configuration/action profiles, pins and timestamps to
be reconciled before a real-episode gate, training or another schema freeze.
The accepted additive schema shape remains intact. The
[organization decisions register](https://github.com/openAMRobot/.github/blob/539da1982bccf8e2315483f41ac16734e0bed328/decisions.yaml)
records the current lift/arm and configuration distinctions; it supplies no
C10 runtime-home or dataset synchronization acceptance decision.

Acceptance remains blocked on an approved runtime owner/location and policy,
an approved source profile with immutable configuration/calibration evidence,
a validated LeRobot release pin, and one actual native episode plus matching C9
sidecar and end-to-end report. A passing conformance fixture alone closes none
of those items.
Loading
Loading