Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ extensions/labwired-vscode/share/tools.json
fixtures/**/.pio/
fixtures/**/evidence/
fixtures/coverage/smoke/
# Recorded twin-vs-desk differential fixtures (stand-in artifacts, tracked on purpose)
!fixtures/twin-desk-diff/*.bin
workspaces/**/.pio/
workspaces/**/evidence/
share/smoke/.dead-token
Expand Down
4 changes: 4 additions & 0 deletions bin/labwired-agent
Original file line number Diff line number Diff line change
Expand Up @@ -916,6 +916,10 @@ Usage:
labwired agent compose … Assemble plot elements (uart | capture)
labwired agent hardware plan --profile FILE --out DIR
labwired agent hardware run --profile FILE --out DIR --confirm DIGEST
labwired agent hardware diff --artifact FILE --twin-evidence DIR --twin-receipt SHA256
[--desk-evidence DIR --desk-receipt SHA256] [--out FILE]
Same firmware, twin vs desk board; exit 0 agree, 3 disagree,
4 desk-unavailable, 5 twin-unavailable, 2 invalid
labwired agent install-deps Install sim + probe-rs into prefix
labwired agent package … Portable prefix: info | path | env | uninstall
labwired agent version Version
Expand Down
35 changes: 35 additions & 0 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,3 +133,38 @@ results and execution or evidence `FAIL` results exit `3`.
Planning never creates the evidence directory or builds, flashes, or opens an
instrument. Runs lock the explicit target, probe, port, and analyzer identities
so two sessions cannot control the same lab resource.

## Differential: the same firmware on the twin and on a desk board

One firmware artifact, two independent targets. Run the twin lane and the
physical lane as ordinary `hardware run` invocations against their own profiles,
then compare the two authenticated evidence bundles:

```bash
labwired agent hardware diff --artifact build/firmware.bin \
--twin-evidence .labwired/evidence-twin --twin-receipt <64-character-twin-receipt> \
--desk-evidence .labwired/evidence-desk --desk-receipt <64-character-desk-receipt> \
--out .labwired/twin-desk-diff.json
```

Each `--*-receipt` is the out-of-bundle `manifestSha256` that `hardware run`
returned for that bundle. A bundle supplied without its receipt is never
accepted as evidence for its side.

`hardware diff` emits one structured JSON verdict and binds it to exit codes:

| Verdict | Exit | Meaning |
|---|---|---|
| `agree` | `0` | Both targets decided the same behaviors the same way. |
| `disagree` | `3` | The twin and the board decided a shared behavior differently. This is a first-class published result, not an error. |
| `desk-unavailable` | `4` | No desk bundle, or the desk bundle recorded no physical evidence (for example no probe was detected). Never a silent pass. |
| `twin-unavailable` | `5` | No twin bundle, or the twin bundle decided nothing. |
| `invalid` | `2` | The comparison was refused: unequal artifact digests, a twin bundle claiming a physical-evidence grade, no shared behavior, or a CLI usage error. |

Each side publishes only its own grade, and the grades are disjoint: a model
grade is reachable only from a twin run, a physical-evidence grade only from a
board. `hardware diff` has no code path that copies a level or a pass from one
side to the other, so a hardware green is never upgraded to a twin green or the
reverse. Both sides must be bound to the same artifact digest; a mismatch is
`invalid`, never agreement. The grade names themselves, and what each one
requires, are defined in [VERIFY.md](VERIFY.md).
26 changes: 26 additions & 0 deletions docs/VERIFY.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,3 +115,29 @@ They cannot represent arbitrary twin steps, fix serial capture at 115200 baud,
and preserve prebuilt artifacts as imported rather than compiled. Migrate any
workflow needing other providers, observations, baud rates, or behavior-level
claims to the checked-in generic profile.

## Comparing the twin against a desk board

`hardware diff` takes one firmware artifact and two authenticated evidence
bundles — one from the twin, one from a physical board — and publishes whether
they agree. A disagreement is a first-class result, not an error.

The two sides use disjoint grades and neither is ever converted into the other:

- The twin side reaches `model_verified`, from `model_observed` or
`surrogate_model_observed`. It can never record `hardware_observed`; a twin
bundle that claims it is rejected as `invalid`.
- The desk side reaches `hardware_observed`, which requires exact flash plus
independent evidence for the configured behavior. A desk record carrying a
model grade is inconclusive desk evidence, never a desk pass.
- `compiled` on the twin side is inconclusive, not behavior evidence.

Each side is summarized from its own bundle alone, so there is no expression
that can upgrade a hardware green to a twin green or the reverse. Both bundles
must bind to the same artifact digest; a mismatch is refused rather than
smoothed into agreement, and `agree` requires at least one behavior that both
sides actually decided.

Verdicts bind to exit codes: `agree` 0, `invalid` 2, `disagree` 3,
`desk-unavailable` 4, `twin-unavailable` 5. A failed invocation exits 2, so it
can never be mistaken for a disagreement.
7 changes: 7 additions & 0 deletions fixtures/twin-desk-diff/agree/desk.receipt.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"bundle": "agree/desk",
"side": "desk",
"artifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"manifestSha256": "5b627cf5eb053314f3261cb90fcba4dbcb2be301583a58970d7262d4e6d04acb",
"result": "PASS"
}
31 changes: 31 additions & 0 deletions fixtures/twin-desk-diff/agree/desk/.owner.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"schema": 1,
"owner": "29ce09f1-3ffb-4c9b-b9c4-626b5823fd24",
"createdAt": "2026-08-16T16:11:24.398Z",
"targetIdentity": {
"id": "desk-c3",
"chip": "esp32c3"
},
"behaviors": [
{
"id": "heartbeat",
"provider": "serial",
"requiredLevel": "hardware_observed"
},
{
"id": "led-blink",
"provider": "logic-csv",
"requiredLevel": "hardware_observed"
}
],
"stages": [
{
"id": "build",
"provider": "platformio"
},
{
"id": "flash",
"provider": "probe-rs"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
desk capture for heartbeat
level=hardware_observed
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"behaviorId": "heartbeat",
"provider": "serial",
"level": "hardware_observed",
"artifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"targetIdentity": {
"id": "desk-c3",
"chip": "esp32c3"
},
"startedAt": "2026-08-16T16:11:24.288Z",
"endedAt": "2026-08-16T16:11:28.288Z",
"toolVersion": "probe-rs recorded fixture",
"rawEvidenceRefs": [
"observations/heartbeat.desk.capture.log"
],
"diagnostics": {
"detail": "desk observed heartbeat"
},
"flashedArtifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"rawEvidence": [
{
"path": "observations/heartbeat.desk.capture.log",
"sha256": "5aba5981cb190ef4cd9d90f657261630d61d8845bcbc6885b24a4703c45f4bc6",
"size": 51
}
],
"requiredLevel": "hardware_observed"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
desk capture for led-blink
level=hardware_observed
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"behaviorId": "led-blink",
"provider": "logic-csv",
"level": "hardware_observed",
"artifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"targetIdentity": {
"id": "desk-c3",
"chip": "esp32c3"
},
"startedAt": "2026-08-16T16:11:24.288Z",
"endedAt": "2026-08-16T16:11:28.288Z",
"toolVersion": "probe-rs recorded fixture",
"rawEvidenceRefs": [
"observations/led-blink.desk.capture.log"
],
"diagnostics": {
"detail": "desk observed led-blink"
},
"flashedArtifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"rawEvidence": [
{
"path": "observations/led-blink.desk.capture.log",
"sha256": "5b3cacc1d608a94df1f8c95d3393e8e8c7c4f2a341da0048f2e1bdc1fbbe8fcf",
"size": 51
}
],
"requiredLevel": "hardware_observed"
}
4 changes: 4 additions & 0 deletions fixtures/twin-desk-diff/agree/desk/result.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"result": "PASS",
"reasons": []
}
5 changes: 5 additions & 0 deletions fixtures/twin-desk-diff/agree/desk/stages/build/result.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"stageId": "build",
"provider": "platformio",
"level": "not-run"
}
5 changes: 5 additions & 0 deletions fixtures/twin-desk-diff/agree/desk/stages/flash/result.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"stageId": "flash",
"provider": "probe-rs",
"level": "not-run"
}
7 changes: 7 additions & 0 deletions fixtures/twin-desk-diff/agree/twin.receipt.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"bundle": "agree/twin",
"side": "twin",
"artifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"manifestSha256": "5a66eead2d11fd4fa2fca5352dbdbe8560753b7e5e4b291cd54fd17486da8191",
"result": "PASS"
}
31 changes: 31 additions & 0 deletions fixtures/twin-desk-diff/agree/twin/.owner.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"schema": 1,
"owner": "62bb0ca3-5b92-440d-bf02-323553bac607",
"createdAt": "2026-08-16T16:11:24.299Z",
"targetIdentity": {
"id": "desk-c3",
"chip": "esp32c3"
},
"behaviors": [
{
"id": "heartbeat",
"provider": "serial",
"requiredLevel": "model_observed"
},
{
"id": "led-blink",
"provider": "logic-csv",
"requiredLevel": "model_observed"
}
],
"stages": [
{
"id": "build",
"provider": "platformio"
},
{
"id": "twin",
"provider": "labwired-sim"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
twin capture for heartbeat
level=model_observed
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"behaviorId": "heartbeat",
"provider": "serial",
"level": "model_observed",
"artifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"targetIdentity": {
"id": "desk-c3",
"chip": "esp32c3"
},
"startedAt": "2026-08-16T16:11:24.288Z",
"endedAt": "2026-08-16T16:11:28.288Z",
"toolVersion": "labwired-sim recorded fixture",
"rawEvidenceRefs": [
"observations/heartbeat.twin.capture.log"
],
"diagnostics": {
"detail": "twin observed heartbeat"
},
"nativeArtifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"rawEvidence": [
{
"path": "observations/heartbeat.twin.capture.log",
"sha256": "075b5e80a20b12fb8149f6d12d5fbbaa93ce2860f143dda0cb806d2f958de887",
"size": 48
}
],
"requiredLevel": "model_observed"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
twin capture for led-blink
level=model_observed
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"behaviorId": "led-blink",
"provider": "logic-csv",
"level": "model_observed",
"artifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"targetIdentity": {
"id": "desk-c3",
"chip": "esp32c3"
},
"startedAt": "2026-08-16T16:11:24.288Z",
"endedAt": "2026-08-16T16:11:28.288Z",
"toolVersion": "labwired-sim recorded fixture",
"rawEvidenceRefs": [
"observations/led-blink.twin.capture.log"
],
"diagnostics": {
"detail": "twin observed led-blink"
},
"nativeArtifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"rawEvidence": [
{
"path": "observations/led-blink.twin.capture.log",
"sha256": "706f06775ea2096934a20e3349cf37f8617b6108171ae90f20265282a7021703",
"size": 48
}
],
"requiredLevel": "model_observed"
}
4 changes: 4 additions & 0 deletions fixtures/twin-desk-diff/agree/twin/result.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"result": "PASS",
"reasons": []
}
5 changes: 5 additions & 0 deletions fixtures/twin-desk-diff/agree/twin/stages/build/result.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"stageId": "build",
"provider": "platformio",
"level": "not-run"
}
5 changes: 5 additions & 0 deletions fixtures/twin-desk-diff/agree/twin/stages/twin/result.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"stageId": "twin",
"provider": "labwired-sim",
"level": "not-run"
}
7 changes: 7 additions & 0 deletions fixtures/twin-desk-diff/contaminated/twin.receipt.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"bundle": "contaminated/twin",
"side": "twin",
"artifactSha256": "1bd0221812559331be433e5b1e002945ebba25b6503b1fafc85adc4cd1555fcf",
"manifestSha256": "fd5ebe8fec4af5cb8f487fbe8a3f49a074ea9d27758896e516e7ad9a9f4ec992",
"result": "PASS"
}
31 changes: 31 additions & 0 deletions fixtures/twin-desk-diff/contaminated/twin/.owner.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"schema": 1,
"owner": "da33f05a-b86a-4932-abd3-b1d0535f4784",
"createdAt": "2026-08-16T16:11:24.591Z",
"targetIdentity": {
"id": "desk-c3",
"chip": "esp32c3"
},
"behaviors": [
{
"id": "heartbeat",
"provider": "serial",
"requiredLevel": "model_observed"
},
{
"id": "led-blink",
"provider": "logic-csv",
"requiredLevel": "model_observed"
}
],
"stages": [
{
"id": "build",
"provider": "platformio"
},
{
"id": "twin",
"provider": "labwired-sim"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
twin capture for heartbeat
level=hardware_observed
Loading