diff --git a/CHANGELOG.md b/CHANGELOG.md index 708edbd..b6fed7f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,6 +52,8 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht * **Document large-capture and dataset-acquisition footguns (#447).** Added a "Working with large captures" section to `troubleshooting.md` (inspect with `capture-info` first, use its `suggested_max_frames`/`suggested_seconds`, bound runs with `--max-frames`/`--seconds`/`--offset`, and the automatic 50 MB / 500k-frame cap) and an "Acquiring dataset data" section clarifying `datasets fetch` (provenance only) vs `datasets replay` (retrieves data) and the real-time `--rate 1.0` default. Added a cookbook recipe, *Acquire a bounded dataset slice for offline analysis* (search → inspect → list-files → bounded replay), a getting-started tip box cross-linking both sections, and a note that `j1939 pgn`/`spn` no longer require a capture (reference mode, #442). Closes #447. +* **Refresh stale command summaries in the high-level docs.** The overview, command-spec status list, and feature-matrix summaries had drifted behind the per-command reference sections in the same files. Brought them back in line: added the command families the summaries had fallen behind on (`simulate`, `compare`, `capture-info`, `sequence replay`, `datasets download`, `j1939 map`, the full UDS active-workflow set, the `doip` command group, `xcp info`/`dump`, `plot`, `re corpus`, `skills`, and the `fuzz` commands), described `tui` as the full-screen Textual dashboard instead of the retired text-mode shell, removed a "Current Gaps" bullet that still listed already-implemented payload/replay/arbitration-id fuzzing as unexposed, and listed `fuzz guided` in the README command catalog. Documentation-only; no command-surface or behaviour change. + ### Fixed * **The full-screen TUI no longer crashes when DM1 active faults are present.** The J1939 summary ribbon and the bus-status bar rendered their text as console markup, so a DM1 fault summary containing a bracketed SPN/FMI sample (e.g. `[spn=175/fmi=5]`) — or the bracketed `[paused]` status indicator — raised a `MarkupError` and took down the app the moment a `j1939 dm1` command with active faults folded into the ribbon. Both `Static` widgets now render their content as literal text, so brackets are shown verbatim. A regression test drives `j1939 dm1` against a fault fixture inside the TUI. diff --git a/README.md b/README.md index f37383f..a6c13a7 100644 --- a/README.md +++ b/README.md @@ -182,6 +182,7 @@ Fully implemented and tested: **Active-transmit fuzzing** (gated by the [active-transmit safety design](docs/design/active-transmit-safety.md); `--dry-run` is the safe planning path) - `fuzz payload`, `fuzz replay`, `fuzz arbitration-id`, `fuzz identify` — payload/replay/ID-walk fuzzing and bisecting a fuzz log to the culprit frame - `fuzz signal`, `fuzz spn` — DBC-signal and J1939-SPN-aware mutation with sentinel coverage +- `fuzz guided` — response-feedback coverage-guided fuzzing with a persisted seed corpus **Session, export, and utilities** - `session save`, `load`, `show` — session management diff --git a/docs/command_spec.md b/docs/command_spec.md index 084182a..b90d04d 100644 --- a/docs/command_spec.md +++ b/docs/command_spec.md @@ -6,30 +6,34 @@ This document describes the current implemented CLI contract. Implemented and verified in the current codebase: -* live and deterministic transport workflows for `capture`, `send`, and `generate` -* file-backed `filter` and `stats` over candump capture logs +* live and deterministic transport workflows for `capture`, `send`, `generate`, and `simulate` +* file-backed `filter`, `stats`, `compare`, and `capture-info` over candump capture logs * default `python-can` support for `capture` and `send` -* deterministic `replay` +* deterministic `replay` and `sequence replay` * live `gateway` bridging between CAN interfaces via `python-can` * structured `export` for capture files and saved sessions * DBC-backed `decode`, `encode`, and `dbc inspect` * DBC provider workflows for `dbc provider list`, `dbc search`, `dbc fetch`, `dbc cache list`, `dbc cache prune`, and `dbc cache refresh` -* dataset provider workflows for `datasets provider list`, `datasets search`, `datasets inspect`, `datasets fetch`, `datasets cache list`, `datasets cache refresh`, `datasets convert`, `datasets stream`, and `datasets replay` -* J1939 `monitor`, `decode`, `pgn`, `spn`, `tp`, `dm1`, `faults`, `summary`, `inventory`, and `compare` +* dataset provider workflows for `datasets provider list`, `datasets search`, `datasets inspect`, `datasets fetch`, `datasets download`, `datasets cache list`, `datasets cache refresh`, `datasets convert`, `datasets stream`, and `datasets replay` +* J1939 `monitor`, `decode`, `pgn`, `spn`, `tp`, `dm1`, `faults`, `summary`, `inventory`, `compare`, and `map` * session `save`, `load`, and `show` * shell one-shot command execution -* initial text-mode `tui` shell over the shared command layer -* UDS `scan`, `trace`, and `services` -* XCP (measurement/calibration) `scan`, `trace`, `read`, and `commands` for XCP-on-CAN +* full-screen Textual `tui` dashboard over the shared command layer +* UDS `scan`, `trace`, `services`, `subservices`, `ecu-reset`, `tester-present`, `security-seed`, `dump-dids`, `read-memory`, and `auto` +* DoIP `discovery`, `services`, `ecu-reset`, `tester-present`, `security-seed`, and `dump-dids` for Diagnostic-over-IP workflows +* XCP (measurement/calibration) `scan`, `info`, `dump`, `trace`, `read`, and `commands` for XCP-on-CAN * J1587/J1708 `decode` and `pids` for legacy heavy-vehicle diagnostic captures * J2497 (PLC4TRUCKS) `decode` and `mids` for passive trailer power-line frame captures +* `plot` for decoded signal time-series rendering (optional `[plot]` extra) * `config show` for effective transport configuration inspection -* `re signals`, `re counters`, `re entropy`, `re correlate`, and `re anomalies` for passive file-backed analysis +* `re signals`, `re counters`, `re entropy`, `re correlate`, `re anomalies`, and `re corpus` for passive file-backed analysis * `re match-dbc` and `re shortlist-dbc` for provider-backed DBC candidate ranking against captures * `re suggest` for heuristic signal-name suggestions (with an optional, off-by-default external-LLM enrichment) * `plugins list`, `plugins info`, `plugins enable`, and `plugins disable` for Python entry-point plugin inspection and toggles * `web serve` for the read-only browser dashboard over the JSONL event envelope * `cannelloni decode` and `cannelloni send` for cannelloni CAN-over-UDP wire-format interop +* skills provider workflows for `skills provider list`, `skills search`, `skills fetch`, `skills cache list`, and `skills cache refresh` +* active-transmit `fuzz payload`, `fuzz replay`, `fuzz arbitration-id`, `fuzz signal`, `fuzz spn`, `fuzz guided`, and `fuzz identify`, all gated by the active-transmit safety model * structured JSON and JSONL output * explicit error schema and exit codes @@ -1937,10 +1941,7 @@ canarchy shell --command "capture can0 --text" ## Current Gaps -Planned capabilities that are intentionally not exposed in the CLI yet: - -* active fuzzing workflows for replay mutation, payload mutation, and arbitration-ID probing - -These deeper capabilities are also not implemented yet even where the command surface exists: +These deeper capabilities are not fully implemented yet even where the command surface exists: * deeper live transport integration beyond the current `python-can` transport path +* the follow-up active-transmit safety controls tracked in the `fuzz` section (configurable rate-cap ceiling, TOML target allowlist, explicit `KILL_SWITCH_TRIGGERED` alert) diff --git a/docs/feature-matrix.md b/docs/feature-matrix.md index 602af66..5a90444 100644 --- a/docs/feature-matrix.md +++ b/docs/feature-matrix.md @@ -90,8 +90,8 @@ The workflow matrix does not fully capture several important reasons someone mig * `cantools` excels at database-heavy engineering: multiple schema formats, inspection, plotting, monitor workflows, and C code generation. * CANarchy now reads and writes the full cantools database set — `decode`, `encode`, `dbc inspect`, `dbc convert`, and `dbc generate-c` accept DBC, ARXML, KCD, and SYM by filename suffix (#320), and `dbc convert` serializes back out to DBC / KCD / SYM (#385) while `dbc generate-c` emits C source/header/fuzzer files via the cantools C source generator (#386) — alongside provider-backed DBC discovery, `dbc inspect --search`, cantools-backed `dbc inspect --layout` bit diagrams / signal trees / choice tables, and initial reverse-engineering DBC matching. It is still earlier in depth than mature database-centric or visual RE tools. * CANarchy's passive reverse-engineering helpers cover signal-candidate inference (`re signals`), counter detection (`re counters`), byte-entropy ranking (`re entropy`), reference-series correlation (`re correlate`), provider-backed DBC matching (`re match-dbc` / `re shortlist-dbc`), timing/ID anomaly detection (`re anomalies` — robust median/MAD timing outliers plus unknown/dropped-id flags, with cyclic-vs-event classification from the DBC `cycle_time` / send type or a coefficient-of-variation guard so event traffic is not falsely flagged), and cross-capture corpus analysis (`re corpus` — per-ID coverage matrix, cycle-time drift across captures, signal-stability scoring, and new/dropped-ID detection across a fleet of captures). Naming the ranked candidates is assisted by `re suggest`, which proposes signal names offline from reference-DBC overlap, the bundled J1939 SPN/PGN catalog, and range/behaviour templates, with an optional, off-by-default and explicitly-confirmed external-LLM enrichment (`--llm`) that sends only candidate metadata, never payload bytes (#332). -* CANarchy's protocol breadth is currently strongest for J1939 and UDS, with ISO-TP reassembly already used internally by UDS workflows and standalone ISO-TP utility commands planned in the design/test specs for #328. Passive J1939 network understanding extends to `j1939 map`, which derives a CMAP-style nodes/edges topology from a capture — source-address nodes carrying decoded Address Claimed NAME fields and identification strings, plus observed PGN flow edges to unicast or broadcast destinations — as a structured artifact suitable for graphing and diffing, with no active probing (#417). UDS discovery and tracing also run over DoIP (Diagnostic over IP, ISO 13400-2): `uds scan` and `uds trace` accept a `doip://:?logical_address=0x0E80` target and reach an ECU by logical address over TCP, emitting the same canonical transaction envelope (#326). XCP (measurement/calibration, ASAM MCD-1) is covered for XCP-on-CAN via `xcp scan` (CONNECT-based responder discovery), `xcp trace` (named command/response transactions), and `xcp read` (raw DAQ DTO measurement payloads) (#327). Legacy heavy-vehicle diagnostics are covered via `canarchy j1587`: `j1587 decode` parses J1708 capture files (MID + PID-framed parameters + checksum) and resolves common J1587 PIDs to names/units/values against a bundled catalog, and `j1587 pids` lists that catalog (#415). The trailer power-line network (SAE J2497, "PLC4TRUCKS") — which CAN-centric tooling generally ignores — is covered passively via `canarchy j2497`: `j2497 decode` parses captured J2497 frames (which reuse the J1708/J1587 frame format) into per-frame events (MID, message data, checksum validity) with common MIDs resolved to ECU names, and `j2497 mids` lists that MID catalog (#416). Live PLC access requires a power-line carrier modem and external hardware and is out of scope; CANarchy is the analysis layer over captured frames. -* CANarchy's active-transmit fuzzing now spans raw payloads (`fuzz payload`, with bitflip / random / boundary plus AFL-style havoc / splice / interesting-value strategies), capture replay mutation (`fuzz replay`), arbitration-id walks (`fuzz arbitration-id`), DBC-aware signal mutation (`fuzz signal` — in-bounds, out-of-bounds, boundary, enum-gap, and full-field modes), and J1939 SPN-aware mutation (`fuzz spn` — operational bounds plus the not-available / error sentinels), all behind the active-transmit safety model. Coverage-guided fuzzing without instrumentation is provided by `fuzz guided` (#350), which scores mutations by the novelty of the target's observed responses — UDS NRCs, DM1 fault emergence, response-timing buckets, and silence transitions — keeping productive lineages in a persisted seed corpus. +* CANarchy's protocol breadth is currently strongest for J1939 and UDS, with ISO-TP reassembly already used internally by UDS workflows and standalone ISO-TP utility commands planned in the design/test specs for #328. Passive J1939 network understanding extends to `j1939 map`, which derives a CMAP-style nodes/edges topology from a capture — source-address nodes carrying decoded Address Claimed NAME fields and identification strings, plus observed PGN flow edges to unicast or broadcast destinations — as a structured artifact suitable for graphing and diffing, with no active probing (#417). UDS discovery and tracing also run over DoIP (Diagnostic over IP, ISO 13400-2): `uds scan` and `uds trace` accept a `doip://:?logical_address=0x0E80` target and reach an ECU by logical address over TCP, emitting the same canonical transaction envelope (#326), and a dedicated `doip` command group adds UDP entity `discovery` plus `services`, `ecu-reset`, `tester-present`, `security-seed`, and `dump-dids` active diagnostics over a DoIP session (#465). XCP (measurement/calibration, ASAM MCD-1) is covered for XCP-on-CAN via `xcp scan` (CONNECT-based responder discovery), `xcp info` (slave capability interrogation), `xcp dump` (bounded memory upload), `xcp trace` (named command/response transactions), and `xcp read` (raw DAQ DTO measurement payloads) (#327, #467). Legacy heavy-vehicle diagnostics are covered via `canarchy j1587`: `j1587 decode` parses J1708 capture files (MID + PID-framed parameters + checksum) and resolves common J1587 PIDs to names/units/values against a bundled catalog, and `j1587 pids` lists that catalog (#415). The trailer power-line network (SAE J2497, "PLC4TRUCKS") — which CAN-centric tooling generally ignores — is covered passively via `canarchy j2497`: `j2497 decode` parses captured J2497 frames (which reuse the J1708/J1587 frame format) into per-frame events (MID, message data, checksum validity) with common MIDs resolved to ECU names, and `j2497 mids` lists that MID catalog (#416). Live PLC access requires a power-line carrier modem and external hardware and is out of scope; CANarchy is the analysis layer over captured frames. +* CANarchy's active-transmit fuzzing now spans raw payloads (`fuzz payload`, with bitflip / random / boundary plus AFL-style havoc / splice / interesting-value strategies), capture replay mutation (`fuzz replay`), arbitration-id walks (`fuzz arbitration-id`), DBC-aware signal mutation (`fuzz signal` — in-bounds, out-of-bounds, boundary, enum-gap, and full-field modes), and J1939 SPN-aware mutation (`fuzz spn` — operational bounds plus the not-available / error sentinels), all behind the active-transmit safety model. Coverage-guided fuzzing without instrumentation is provided by `fuzz guided` (#350), which scores mutations by the novelty of the target's observed responses — UDS NRCs, DM1 fault emergence, response-timing buckets, and silence transitions — keeping productive lineages in a persisted seed corpus. `fuzz identify` (#464) then narrows a fuzz log to the culprit frame by replaying bisected windows and recording effect / no-effect observations. * SavvyCAN excels at visual exploration and reverse-engineering-oriented desktop analysis. * Caring Caribou excels at automotive security workflows, including UDS fuzzing, DoIP, and XCP-oriented work. * TruckDevil excels at truck and J1939-focused ECU assessment workflows. diff --git a/docs/overview.md b/docs/overview.md index c31561b..b06ca1f 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -13,22 +13,26 @@ Core principles: Implemented and exercised in the current codebase: -* `python-can`-backed live transport workflows plus deterministic scaffold transport for `capture` and `send` -* file-backed `filter`, `stats`, `decode`, `j1939 decode`, and `replay` using candump logs -* DBC-backed `decode`, `encode`, and `dbc inspect`, including provider-ref resolution and `dbc_source` provenance in structured output +* `python-can`-backed live transport workflows plus deterministic scaffold transport for `capture`, `send`, `generate`, and `simulate` +* file-backed `filter`, `stats`, `compare`, `capture-info`, `decode`, `j1939 decode`, and `replay` using candump logs +* `sequence replay` for coordinated transmit of a YAML/JSON sequence of DBC-encoded frames +* live `gateway` bridging between CAN interfaces via `python-can` +* database-backed `decode`, `encode`, `dbc inspect`, `dbc convert`, and `dbc generate-c` across DBC / ARXML / KCD / SYM, including provider-ref resolution and `dbc_source` provenance in structured output * DBC provider and cache workflows for catalog search, fetch, and refresh through the optional opendbc integration * J1939 `monitor`, `decode`, `pgn`, `spn`, `tp`, `dm1`, `faults`, `summary`, `inventory`, `compare`, and `map` -* UDS `scan`, `trace`, and `services` -* `re signals` for file-backed signal candidate inference -* `re counters` for file-backed likely-counter detection -* `re entropy` for file-backed per-ID and per-byte entropy ranking -* `re correlate` for correlating candidate fields against timestamped reference series -* `re match-dbc` and `re shortlist-dbc` for provider-backed DBC candidate ranking against captures +* UDS `scan`, `trace`, `services`, `subservices`, `ecu-reset`, `tester-present`, `security-seed`, `dump-dids`, `read-memory`, and `auto` +* DoIP `discovery`, `services`, `ecu-reset`, `tester-present`, `security-seed`, and `dump-dids` for Diagnostic-over-IP workflows +* XCP-on-CAN `scan`, `info`, `dump`, `trace`, `read`, and `commands` +* J1587/J1708 and J2497 (PLC4TRUCKS) passive decode workflows for legacy heavy-vehicle buses +* reverse-engineering helpers: `re signals`, `re counters`, `re entropy`, `re correlate`, `re anomalies`, `re corpus`, `re match-dbc`, `re shortlist-dbc`, and `re suggest` +* active-transmit fuzzing (`fuzz payload`, `replay`, `arbitration-id`, `signal`, `spn`, `guided`, `identify`) behind the active-transmit safety model +* public CAN dataset provider workflows for search, inspect, fetch, download, convert, stream, and replay * repository-backed skills provider workflows for discovering, fetching, caching, and referencing agent-oriented CANarchy skills +* Python entry-point `plugins` inspection and toggles * session `save`, `load`, and `show` * structured `export` for capture files and saved sessions * shell command reuse through `canarchy shell --command ...` -* initial text-mode `tui` shell over the shared command layer +* a full-screen Textual `tui` dashboard and a read-only `web serve` browser dashboard over the shared command layer * `config show` for effective transport configuration inspection * structured `--json`, `--jsonl`, and `--text` output modes @@ -36,9 +40,10 @@ Some protocol-oriented commands still use explicit sample/reference providers ra ## Current Gaps -Planned but not yet exposed in the CLI: +Not fully implemented yet even where the command surface exists: -* active fuzzing workflows for replay mutation, payload mutation, and arbitration-ID probing +* deeper live transport integration beyond the current `python-can` transport path +* follow-up active-transmit safety controls (configurable rate-cap ceiling, TOML target allowlist, explicit `KILL_SWITCH_TRIGGERED` alert) ## Recommended Reading