Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht

* **Document large-capture and dataset-acquisition footguns (#447).** Added a "Working with large captures" section to `troubleshooting.md` (inspect with `capture-info` first, use its `suggested_max_frames`/`suggested_seconds`, bound runs with `--max-frames`/`--seconds`/`--offset`, and the automatic 50 MB / 500k-frame cap) and an "Acquiring dataset data" section clarifying `datasets fetch` (provenance only) vs `datasets replay` (retrieves data) and the real-time `--rate 1.0` default. Added a cookbook recipe, *Acquire a bounded dataset slice for offline analysis* (search → inspect → list-files → bounded replay), a getting-started tip box cross-linking both sections, and a note that `j1939 pgn`/`spn` no longer require a capture (reference mode, #442). Closes #447.

* **Refresh stale command summaries in the high-level docs.** The overview, command-spec status list, and feature-matrix summaries had drifted behind the per-command reference sections in the same files. Brought them back in line: added the command families the summaries had fallen behind on (`simulate`, `compare`, `capture-info`, `sequence replay`, `datasets download`, `j1939 map`, the full UDS active-workflow set, the `doip` command group, `xcp info`/`dump`, `plot`, `re corpus`, `skills`, and the `fuzz` commands), described `tui` as the full-screen Textual dashboard instead of the retired text-mode shell, removed a "Current Gaps" bullet that still listed already-implemented payload/replay/arbitration-id fuzzing as unexposed, and listed `fuzz guided` in the README command catalog. Documentation-only; no command-surface or behaviour change.

### Fixed

* **The full-screen TUI no longer crashes when DM1 active faults are present.** The J1939 summary ribbon and the bus-status bar rendered their text as console markup, so a DM1 fault summary containing a bracketed SPN/FMI sample (e.g. `[spn=175/fmi=5]`) — or the bracketed `[paused]` status indicator — raised a `MarkupError` and took down the app the moment a `j1939 dm1` command with active faults folded into the ribbon. Both `Static` widgets now render their content as literal text, so brackets are shown verbatim. A regression test drives `j1939 dm1` against a fault fixture inside the TUI.
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,7 @@ Fully implemented and tested:
**Active-transmit fuzzing** (gated by the [active-transmit safety design](docs/design/active-transmit-safety.md); `--dry-run` is the safe planning path)
- `fuzz payload`, `fuzz replay`, `fuzz arbitration-id`, `fuzz identify` — payload/replay/ID-walk fuzzing and bisecting a fuzz log to the culprit frame
- `fuzz signal`, `fuzz spn` — DBC-signal and J1939-SPN-aware mutation with sentinel coverage
- `fuzz guided` — response-feedback coverage-guided fuzzing with a persisted seed corpus

**Session, export, and utilities**
- `session save`, `load`, `show` — session management
Expand Down
29 changes: 15 additions & 14 deletions docs/command_spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,30 +6,34 @@ This document describes the current implemented CLI contract.

Implemented and verified in the current codebase:

* live and deterministic transport workflows for `capture`, `send`, and `generate`
* file-backed `filter` and `stats` over candump capture logs
* live and deterministic transport workflows for `capture`, `send`, `generate`, and `simulate`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required changelog entry

This documentation refresh changes high-level command summaries in four docs, so it is more than the typo/nav exceptions, but the diff does not include CHANGELOG.md. AGENTS.md requires every "significant documentation update" to have a [Unreleased] changelog entry, so the release notes will miss this docs-currency fix unless it is added in this commit.

Useful? React with 👍 / 👎.

* file-backed `filter`, `stats`, `compare`, and `capture-info` over candump capture logs
* default `python-can` support for `capture` and `send`
* deterministic `replay`
* deterministic `replay` and `sequence replay`
* live `gateway` bridging between CAN interfaces via `python-can`
* structured `export` for capture files and saved sessions
* DBC-backed `decode`, `encode`, and `dbc inspect`
* DBC provider workflows for `dbc provider list`, `dbc search`, `dbc fetch`, `dbc cache list`, `dbc cache prune`, and `dbc cache refresh`
* dataset provider workflows for `datasets provider list`, `datasets search`, `datasets inspect`, `datasets fetch`, `datasets cache list`, `datasets cache refresh`, `datasets convert`, `datasets stream`, and `datasets replay`
* J1939 `monitor`, `decode`, `pgn`, `spn`, `tp`, `dm1`, `faults`, `summary`, `inventory`, and `compare`
* dataset provider workflows for `datasets provider list`, `datasets search`, `datasets inspect`, `datasets fetch`, `datasets download`, `datasets cache list`, `datasets cache refresh`, `datasets convert`, `datasets stream`, and `datasets replay`
* J1939 `monitor`, `decode`, `pgn`, `spn`, `tp`, `dm1`, `faults`, `summary`, `inventory`, `compare`, and `map`
* session `save`, `load`, and `show`
* shell one-shot command execution
* initial text-mode `tui` shell over the shared command layer
* UDS `scan`, `trace`, and `services`
* XCP (measurement/calibration) `scan`, `trace`, `read`, and `commands` for XCP-on-CAN
* full-screen Textual `tui` dashboard over the shared command layer
* UDS `scan`, `trace`, `services`, `subservices`, `ecu-reset`, `tester-present`, `security-seed`, `dump-dids`, `read-memory`, and `auto`
* DoIP `discovery`, `services`, `ecu-reset`, `tester-present`, `security-seed`, and `dump-dids` for Diagnostic-over-IP workflows
* XCP (measurement/calibration) `scan`, `info`, `dump`, `trace`, `read`, and `commands` for XCP-on-CAN
* J1587/J1708 `decode` and `pids` for legacy heavy-vehicle diagnostic captures
* J2497 (PLC4TRUCKS) `decode` and `mids` for passive trailer power-line frame captures
* `plot` for decoded signal time-series rendering (optional `[plot]` extra)
* `config show` for effective transport configuration inspection
* `re signals`, `re counters`, `re entropy`, `re correlate`, and `re anomalies` for passive file-backed analysis
* `re signals`, `re counters`, `re entropy`, `re correlate`, `re anomalies`, and `re corpus` for passive file-backed analysis
* `re match-dbc` and `re shortlist-dbc` for provider-backed DBC candidate ranking against captures
* `re suggest` for heuristic signal-name suggestions (with an optional, off-by-default external-LLM enrichment)
* `plugins list`, `plugins info`, `plugins enable`, and `plugins disable` for Python entry-point plugin inspection and toggles
* `web serve` for the read-only browser dashboard over the JSONL event envelope
* `cannelloni decode` and `cannelloni send` for cannelloni CAN-over-UDP wire-format interop
* skills provider workflows for `skills provider list`, `skills search`, `skills fetch`, `skills cache list`, and `skills cache refresh`
* active-transmit `fuzz payload`, `fuzz replay`, `fuzz arbitration-id`, `fuzz signal`, `fuzz spn`, `fuzz guided`, and `fuzz identify`, all gated by the active-transmit safety model
* structured JSON and JSONL output
* explicit error schema and exit codes

Expand Down Expand Up @@ -1937,10 +1941,7 @@ canarchy shell --command "capture can0 --text"

## Current Gaps

Planned capabilities that are intentionally not exposed in the CLI yet:

* active fuzzing workflows for replay mutation, payload mutation, and arbitration-ID probing

These deeper capabilities are also not implemented yet even where the command surface exists:
These deeper capabilities are not fully implemented yet even where the command surface exists:

* deeper live transport integration beyond the current `python-can` transport path
* the follow-up active-transmit safety controls tracked in the `fuzz` section (configurable rate-cap ceiling, TOML target allowlist, explicit `KILL_SWITCH_TRIGGERED` alert)
4 changes: 2 additions & 2 deletions docs/feature-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,8 @@ The workflow matrix does not fully capture several important reasons someone mig
* `cantools` excels at database-heavy engineering: multiple schema formats, inspection, plotting, monitor workflows, and C code generation.
* CANarchy now reads and writes the full cantools database set — `decode`, `encode`, `dbc inspect`, `dbc convert`, and `dbc generate-c` accept DBC, ARXML, KCD, and SYM by filename suffix (#320), and `dbc convert` serializes back out to DBC / KCD / SYM (#385) while `dbc generate-c` emits C source/header/fuzzer files via the cantools C source generator (#386) — alongside provider-backed DBC discovery, `dbc inspect --search`, cantools-backed `dbc inspect --layout` bit diagrams / signal trees / choice tables, and initial reverse-engineering DBC matching. It is still earlier in depth than mature database-centric or visual RE tools.
* CANarchy's passive reverse-engineering helpers cover signal-candidate inference (`re signals`), counter detection (`re counters`), byte-entropy ranking (`re entropy`), reference-series correlation (`re correlate`), provider-backed DBC matching (`re match-dbc` / `re shortlist-dbc`), timing/ID anomaly detection (`re anomalies` — robust median/MAD timing outliers plus unknown/dropped-id flags, with cyclic-vs-event classification from the DBC `cycle_time` / send type or a coefficient-of-variation guard so event traffic is not falsely flagged), and cross-capture corpus analysis (`re corpus` — per-ID coverage matrix, cycle-time drift across captures, signal-stability scoring, and new/dropped-ID detection across a fleet of captures). Naming the ranked candidates is assisted by `re suggest`, which proposes signal names offline from reference-DBC overlap, the bundled J1939 SPN/PGN catalog, and range/behaviour templates, with an optional, off-by-default and explicitly-confirmed external-LLM enrichment (`--llm`) that sends only candidate metadata, never payload bytes (#332).
* CANarchy's protocol breadth is currently strongest for J1939 and UDS, with ISO-TP reassembly already used internally by UDS workflows and standalone ISO-TP utility commands planned in the design/test specs for #328. Passive J1939 network understanding extends to `j1939 map`, which derives a CMAP-style nodes/edges topology from a capture — source-address nodes carrying decoded Address Claimed NAME fields and identification strings, plus observed PGN flow edges to unicast or broadcast destinations — as a structured artifact suitable for graphing and diffing, with no active probing (#417). UDS discovery and tracing also run over DoIP (Diagnostic over IP, ISO 13400-2): `uds scan` and `uds trace` accept a `doip://<host>:<port>?logical_address=0x0E80` target and reach an ECU by logical address over TCP, emitting the same canonical transaction envelope (#326). XCP (measurement/calibration, ASAM MCD-1) is covered for XCP-on-CAN via `xcp scan` (CONNECT-based responder discovery), `xcp trace` (named command/response transactions), and `xcp read` (raw DAQ DTO measurement payloads) (#327). Legacy heavy-vehicle diagnostics are covered via `canarchy j1587`: `j1587 decode` parses J1708 capture files (MID + PID-framed parameters + checksum) and resolves common J1587 PIDs to names/units/values against a bundled catalog, and `j1587 pids` lists that catalog (#415). The trailer power-line network (SAE J2497, "PLC4TRUCKS") — which CAN-centric tooling generally ignores — is covered passively via `canarchy j2497`: `j2497 decode` parses captured J2497 frames (which reuse the J1708/J1587 frame format) into per-frame events (MID, message data, checksum validity) with common MIDs resolved to ECU names, and `j2497 mids` lists that MID catalog (#416). Live PLC access requires a power-line carrier modem and external hardware and is out of scope; CANarchy is the analysis layer over captured frames.
* CANarchy's active-transmit fuzzing now spans raw payloads (`fuzz payload`, with bitflip / random / boundary plus AFL-style havoc / splice / interesting-value strategies), capture replay mutation (`fuzz replay`), arbitration-id walks (`fuzz arbitration-id`), DBC-aware signal mutation (`fuzz signal` — in-bounds, out-of-bounds, boundary, enum-gap, and full-field modes), and J1939 SPN-aware mutation (`fuzz spn` — operational bounds plus the not-available / error sentinels), all behind the active-transmit safety model. Coverage-guided fuzzing without instrumentation is provided by `fuzz guided` (#350), which scores mutations by the novelty of the target's observed responses — UDS NRCs, DM1 fault emergence, response-timing buckets, and silence transitions — keeping productive lineages in a persisted seed corpus.
* CANarchy's protocol breadth is currently strongest for J1939 and UDS, with ISO-TP reassembly already used internally by UDS workflows and standalone ISO-TP utility commands planned in the design/test specs for #328. Passive J1939 network understanding extends to `j1939 map`, which derives a CMAP-style nodes/edges topology from a capture — source-address nodes carrying decoded Address Claimed NAME fields and identification strings, plus observed PGN flow edges to unicast or broadcast destinations — as a structured artifact suitable for graphing and diffing, with no active probing (#417). UDS discovery and tracing also run over DoIP (Diagnostic over IP, ISO 13400-2): `uds scan` and `uds trace` accept a `doip://<host>:<port>?logical_address=0x0E80` target and reach an ECU by logical address over TCP, emitting the same canonical transaction envelope (#326), and a dedicated `doip` command group adds UDP entity `discovery` plus `services`, `ecu-reset`, `tester-present`, `security-seed`, and `dump-dids` active diagnostics over a DoIP session (#465). XCP (measurement/calibration, ASAM MCD-1) is covered for XCP-on-CAN via `xcp scan` (CONNECT-based responder discovery), `xcp info` (slave capability interrogation), `xcp dump` (bounded memory upload), `xcp trace` (named command/response transactions), and `xcp read` (raw DAQ DTO measurement payloads) (#327, #467). Legacy heavy-vehicle diagnostics are covered via `canarchy j1587`: `j1587 decode` parses J1708 capture files (MID + PID-framed parameters + checksum) and resolves common J1587 PIDs to names/units/values against a bundled catalog, and `j1587 pids` lists that catalog (#415). The trailer power-line network (SAE J2497, "PLC4TRUCKS") — which CAN-centric tooling generally ignores — is covered passively via `canarchy j2497`: `j2497 decode` parses captured J2497 frames (which reuse the J1708/J1587 frame format) into per-frame events (MID, message data, checksum validity) with common MIDs resolved to ECU names, and `j2497 mids` lists that MID catalog (#416). Live PLC access requires a power-line carrier modem and external hardware and is out of scope; CANarchy is the analysis layer over captured frames.
* CANarchy's active-transmit fuzzing now spans raw payloads (`fuzz payload`, with bitflip / random / boundary plus AFL-style havoc / splice / interesting-value strategies), capture replay mutation (`fuzz replay`), arbitration-id walks (`fuzz arbitration-id`), DBC-aware signal mutation (`fuzz signal` — in-bounds, out-of-bounds, boundary, enum-gap, and full-field modes), and J1939 SPN-aware mutation (`fuzz spn` — operational bounds plus the not-available / error sentinels), all behind the active-transmit safety model. Coverage-guided fuzzing without instrumentation is provided by `fuzz guided` (#350), which scores mutations by the novelty of the target's observed responses — UDS NRCs, DM1 fault emergence, response-timing buckets, and silence transitions — keeping productive lineages in a persisted seed corpus. `fuzz identify` (#464) then narrows a fuzz log to the culprit frame by replaying bisected windows and recording effect / no-effect observations.
* SavvyCAN excels at visual exploration and reverse-engineering-oriented desktop analysis.
* Caring Caribou excels at automotive security workflows, including UDS fuzzing, DoIP, and XCP-oriented work.
* TruckDevil excels at truck and J1939-focused ECU assessment workflows.
Expand Down
29 changes: 17 additions & 12 deletions docs/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,32 +13,37 @@ Core principles:

Implemented and exercised in the current codebase:

* `python-can`-backed live transport workflows plus deterministic scaffold transport for `capture` and `send`
* file-backed `filter`, `stats`, `decode`, `j1939 decode`, and `replay` using candump logs
* DBC-backed `decode`, `encode`, and `dbc inspect`, including provider-ref resolution and `dbc_source` provenance in structured output
* `python-can`-backed live transport workflows plus deterministic scaffold transport for `capture`, `send`, `generate`, and `simulate`
* file-backed `filter`, `stats`, `compare`, `capture-info`, `decode`, `j1939 decode`, and `replay` using candump logs
* `sequence replay` for coordinated transmit of a YAML/JSON sequence of DBC-encoded frames
* live `gateway` bridging between CAN interfaces via `python-can`
* database-backed `decode`, `encode`, `dbc inspect`, `dbc convert`, and `dbc generate-c` across DBC / ARXML / KCD / SYM, including provider-ref resolution and `dbc_source` provenance in structured output
* DBC provider and cache workflows for catalog search, fetch, and refresh through the optional opendbc integration
* J1939 `monitor`, `decode`, `pgn`, `spn`, `tp`, `dm1`, `faults`, `summary`, `inventory`, `compare`, and `map`
* UDS `scan`, `trace`, and `services`
* `re signals` for file-backed signal candidate inference
* `re counters` for file-backed likely-counter detection
* `re entropy` for file-backed per-ID and per-byte entropy ranking
* `re correlate` for correlating candidate fields against timestamped reference series
* `re match-dbc` and `re shortlist-dbc` for provider-backed DBC candidate ranking against captures
* UDS `scan`, `trace`, `services`, `subservices`, `ecu-reset`, `tester-present`, `security-seed`, `dump-dids`, `read-memory`, and `auto`
* DoIP `discovery`, `services`, `ecu-reset`, `tester-present`, `security-seed`, and `dump-dids` for Diagnostic-over-IP workflows
* XCP-on-CAN `scan`, `info`, `dump`, `trace`, `read`, and `commands`
* J1587/J1708 and J2497 (PLC4TRUCKS) passive decode workflows for legacy heavy-vehicle buses
* reverse-engineering helpers: `re signals`, `re counters`, `re entropy`, `re correlate`, `re anomalies`, `re corpus`, `re match-dbc`, `re shortlist-dbc`, and `re suggest`
* active-transmit fuzzing (`fuzz payload`, `replay`, `arbitration-id`, `signal`, `spn`, `guided`, `identify`) behind the active-transmit safety model
* public CAN dataset provider workflows for search, inspect, fetch, download, convert, stream, and replay
* repository-backed skills provider workflows for discovering, fetching, caching, and referencing agent-oriented CANarchy skills
* Python entry-point `plugins` inspection and toggles
* session `save`, `load`, and `show`
* structured `export` for capture files and saved sessions
* shell command reuse through `canarchy shell --command ...`
* initial text-mode `tui` shell over the shared command layer
* a full-screen Textual `tui` dashboard and a read-only `web serve` browser dashboard over the shared command layer
* `config show` for effective transport configuration inspection
* structured `--json`, `--jsonl`, and `--text` output modes

Some protocol-oriented commands still use explicit sample/reference providers rather than true transport-backed execution. See [Architecture](architecture.md) and [Command Spec](command_spec.md) for the current boundary.

## Current Gaps

Planned but not yet exposed in the CLI:
Not fully implemented yet even where the command surface exists:

* active fuzzing workflows for replay mutation, payload mutation, and arbitration-ID probing
* deeper live transport integration beyond the current `python-can` transport path
* follow-up active-transmit safety controls (configurable rate-cap ceiling, TOML target allowlist, explicit `KILL_SWITCH_TRIGGERED` alert)

## Recommended Reading

Expand Down