RoboTraceOpt

English | 简体中文

RoboTraceOpt analyzes ROS 2 runtime behavior across application, middleware, and Linux layers. It combines cross-layer tracing, evidence-graph diagnosis, and constrained configuration optimization for robotic systems.

Explicit adapters unify application-level RuntimeEvent records, ROS 2 traces, and Linux runtime evidence. The diagnosis layer builds typed evidence graphs, reports uncertainty instead of forcing a root-cause label, and only allows optimization trials whose actions match the diagnosed cause.

What is included

• RuntimeEvent v2 instrumentation for three ROS 2 workloads.
• Adapters for RuntimeEvent, ros2_tracing, eBPF scheduling records, and SocketCAN/vcan ACK lifecycles.
• Topology-constrained trace-stage association and typed evidence graphs.
• Auditable root-cause inference with conflict handling and abstention.
• A bounded action registry and reproducible guided, random, and unguided search protocols.
• Candidate validation and offline rollback decisions.
• Balanced repeated campaigns with paired bootstrap confidence bounds.
• Development experiment runners for F1-F6 fault characterization.

Repository layout

ros2_core/     ROS 2 Humble packages and launch files
cpp_core/      portable C++17 planner, diagnosis, and optimizer libraries
diagnosis/     evidence adapters, association, graph construction, inference
experiments/   fault catalog, controlled runners, matched comparisons
optimizer/     action constraints, search plans, objectives, validation
scripts/       build, capture, smoke, and experiment entry points
tests/         unit and contract tests
docs/          public schemas, environment notes, and migration references

Portable C++ core

The first incremental C++ migration is available under cpp_core. It provides dependency-free C++17 libraries for planner safety contracts, trace-stage diagnosis, evidence graphs, and deterministic optimization. The libraries have one CMake entry point, CTest coverage, installable package exports, and a cross-module executable example.

cmake -S cpp_core -B build/cpp-core -DCMAKE_BUILD_TYPE=Release
cmake --build build/cpp-core --parallel
ctest --test-dir build/cpp-core --output-on-failure

The current Python and ROS 2 implementations remain the default while adapters and shadow-mode validation are completed. See docs/CPP_MIGRATION.md for the staged replacement plan and compatibility gates.

RDK X5 preparation

X5 setup, two-adapter physical CAN wiring, pilot execution, the short defense demonstration, and recovery steps are documented in docs/hardware/X5_RUNBOOK.md. The required capture artifacts, CANable/SLCAN branch, and claim boundaries are documented in docs/hardware/PHYSICAL_CAN_EVIDENCE.md.

Preview package installation and run the read-only software preflight:

bash scripts/bootstrap_x5.sh --dry-run
python3 scripts/preflight_x5.py --mode software

With two UP physical CAN interfaces, rehearse the complete demonstration without starting a workload:

python3 scripts/run_x5_demo.py \
  --dry-run \
  --runtime-interface can0 \
  --peer-interface can1 \
  --bitrate 500000 \
  --output-dir data/raw/demos/x5_plan_01

The current physical result is a two-interface SocketCAN smoke with a responder peer. It is development evidence, not an ECU HIL result or a substitute for the frozen native X5 tracing/eBPF matrix. A retained normal/drop pair with the required capture artifacts is necessary before reporting an F6 diagnosis or performance conclusion.

Environment

The primary development environment is Ubuntu 22.04 with ROS 2 Humble. The core workspace can be built from WSL or native Ubuntu:

bash scripts/build_core.sh
source ~/.cache/robotraceopt_build/install/setup.bash

The repository keeps these environment partitions separate:

Partition	Environment	What it supports
Development and WSL evidence	Ubuntu 22.04 / ROS 2 Humble	WSL smoke, association, RuntimeEvent proxy, and readiness checks
Archived native F3/F4 candidate	Ubuntu 24.04 / ROS 2 Jazzy	Retained tooling and candidate files only; no completed native claim is accepted
X5 hardware path	arm64 Ubuntu 22.04.5 / ROS 2 Humble	X5 software/physical-CAN preparation and separately qualified hardware smoke

The Humble workspace is the development and paper baseline. Jazzy has provisioning, capability, and runner tooling for a future qualification pass. The negative Jazzy preflight case is an environment guard, not a Jazzy workload result. Files retained under the historical native F3/F4 package path are an archived candidate projection and must not be cited as a completed native session. Results from different environments must not be merged.

Run the migrated workloads:

bash scripts/run_smoke_workload.sh all 8

Run the Python test suite:

python3 -m unittest discover -s tests -q
python3 -m unittest \
  tests.optimizer.test_action_registry \
  tests.optimizer.test_diagnosis_guided_sampler \
  tests.optimizer.test_runtime_objective \
  tests.optimizer.test_candidate_validator \
  tests.optimizer.test_rollback \
  tests.optimizer.test_trial_planner \
  tests.optimizer.test_runtime_trial \
  tests.optimizer.test_search_summary \
  tests.optimizer.test_diagnosis_gate \
  tests.optimizer.test_runtime_profiles \
  tests.optimizer.test_closed_loop \
  tests.optimizer.test_closed_loop_cli \
  tests.optimizer.test_campaign_schedule \
  tests.optimizer.test_paired_bootstrap \
  tests.optimizer.test_repeated_campaign_cli -q

AI planner reliability

The AI planner supports explicit mock, OpenAI-compatible, and deterministic replay backends through one versioned request/result contract. It records only normalized decision evidence when configured, rejects stale/duplicate output, and fails closed before the final CAN guard. Configuration, replay, fault campaign semantics, and the distinction between command delivery and task success are documented in docs/ai/OPENAI_COMPATIBLE_PROXY_SETUP.md. The default vision mode is metadata-only; image bytes are sent only when payload_base64 is configured with a real JPEG, PNG, or WebP payload. The mock camera does not constitute a real VLM experiment.

replay is an implemented backend, not only a launch parameter. The node wires ReplayPlannerClient through ros2_core/src/vlm_planner_pkg/src/vlm_planner_node.py, and tests/planner/test_model_recording_replay.py covers unique-match delivery, ambiguous-match rejection, and validation before ROS publication. It replays a normalized decision JSONL recording; it does not replay raw model or image inputs. A recording must be supplied by a prior run because this checkout has no data/ recording.

Evidence boundaries

Generated raw and processed experiment data is intentionally excluded from Git. Development evidence is kept separate from calibration and held-out test partitions. RuntimeEvent-only and vcan results are labeled as proxy evidence and are not presented as formal syscall, scheduler, or physical CAN attribution. A physical_can_evidence=true capture establishes only the recorded physical SocketCAN transport path; by itself it is not an ECU HIL, functional-safety, actuator, or formal experiment result.

The repository contains implementation and public technical documentation only. Private research documents and local experiment data are excluded.

Current checkout artifact status

This checkout has no data/ directory and Git tracks zero paths below data/. Commands in this README and the runbooks create local ignored files under data/raw/, data/processed/, or data/reports/; those files are not present here and their absence is not a zero-result experiment. The committed result artifacts are the sanitized projections under docs/evidence/, each with its own manifest and claim boundary.

Archived native F3/F4 candidate package

The historical native F3/F4 package path is retained for repository compatibility and audit history. Its public projection omits the raw CTF, ROS 2, RuntimeEvent, bpftrace/eBPF, and per-event identity records needed to independently requalify the session. It is therefore not accepted as evidence that a native formal experiment was completed.

Do not cite the archived counts, figures, or metadata as native execution, paired F3/F4 effects, scheduler or syscall attribution, diagnosis accuracy, optimization benefit, ECU HIL behavior, or actuator safety. A future native claim requires a new clean session whose raw artifacts, environment report, identity mapping, manifests, and checksums pass the current qualification contract. Until then, the native runbooks describe planned procedures only.

WSL2 run-held-out association evaluation

The earlier WSL2 / Ubuntu 22.04 / ROS 2 Humble overlap logs also support a separate run-held-out evaluation of path association. Runs 01-05 calibrate the timestamp baseline and runs 06-10 are held out for testing in each scenario. The predictor receives only event identity, trace ID, sequence ID, stage, and timestamp; oracle identity is joined only after public groups are generated.

Held-out scenario	Oracle traces	trace_id_contract precision	Recall	F1	Run-bootstrap 95% F1 CI
Dual 10 Hz	6,024	1.0000	0.9772	0.9884	[0.9793, 0.9939]
Dual mixed-rate	5,178	1.0000	1.0000	1.0000	[1.0000, 1.0000]

This is association and path-validity evidence, not F1-F6 root-cause classification. The runs share one host and boot, use mock-mode delays, and record a dirty source tree at commit 65273ea; their input hashes and source tree hash are retained. They are therefore reported as limited WSL2 association evidence and do not close the native-execution, held-out diagnosis, or formal optimization gaps.

The complete sanitized run-held-out association package includes the frozen evaluator, aggregate scoring outputs, split manifest, and input-artifact hashes. Row-level predictions remain in the private audit copy.

Limited supporting evidence

Two overlooked campaigns are now preserved with narrower claim boundaries:

• The WSL2 RuntimeEvent proxy-overhead package contains 60 whole-run summaries: disabled, buffered, and per-event flush at nominal 5 Hz and stress 20 Hz. Median process CPU was 2.037%, 3.476%, and 3.250% at 5 Hz, and 4.075%, 7.607%, and 7.253% at 20 Hz. This block-ordered, dirty-tree WSL2 campaign is not native or four-mode tracing/fused evidence.
• The X5 physical-CAN smoke package preserves one 40-second arm64 PREEMPT_RT capture with 34 sends, 34 matched ACKs, 100% payload matching, and 6.039 ms / 6.238 ms send-to-ACK P50/P95. The planner was mock and no drop/timeout comparator exists, so this is neither ECU HIL nor evidence for the current fail-closed model runtime.

See the public evidence index for package manifests and the claim boundary of every published result.

Project lineage

RoboTraceOpt consolidates engineering work from ROS2Probe and RoboTraceRT into one maintained codebase.

Formal experiment readiness

The formal-session protocol freezes selected Chapter 6 cases before any ROS 2 process starts. Generate a read-only platform report after sourcing ROS 2 and the built workspace:

python3 scripts/check_platform_capabilities.py \
  --label x86-wsl \
  --output-json data/raw/environment/x86-wsl.json

Current WSL development can rehearse only the cases whose reported requirements are ready. This command writes a 42-run plan for F1, mock F6, and the two optimization campaigns without starting a workload:

python3 scripts/run_formal_experiment_session.py \
  --matrix experiments/protocol/formal_experiment_matrix.json \
  --capability-report data/raw/environment/x86-wsl.json \
  --case diagnosis_f1_control \
  --case diagnosis_f1_injected \
  --case diagnosis_f6_control \
  --case diagnosis_f6_injected \
  --case optimization_executor \
  --case optimization_qos \
  --dataset-role pilot \
  --session-name readiness_dry_run_20260718_01 \
  --seed 20260718 \
  --output-dir data/raw/experiments/pilot/readiness_dry_run_20260718_01 \
  --dry-run

This dry-run does not contain measurement evidence. WSL is denied for calibration and held-out test roles even when individual tools appear available.

Fault evidence commit point

A successful formal fault case writes artifact_manifest.json last. The manifest names the required RuntimeEvent, run/oracle/command, identity, tracing, eBPF, scheduler, and summary artifacts for that fault and records a SHA-256 for every file or CTF directory. The outer session verifies this manifest before accepting the case and revalidates its nested artifacts during every integrity reconstruction. A missing or changed artifact makes the case failed or the session invalid; it is preserved and is never silently replaced.

F3/F4 now invoke the eBPF collector during the workload window instead of only checking that the tool is installed. Capture starts only when the live process-manifest/v2 reports ebpf_identity_status=comparable; the runner does not match tasks by process name as a fallback. F2/F3/F5 perform a full ROS 2 trace export after CTF capture, retaining every selected event rather than the bounded sampling used by public fixtures.

This integration closes the runner's artifact-admission contract for a qualified session, but it does not turn a tool-ready or WSL capture into native evidence and does not establish X5 measurement results. WSL dry-runs and synthetic tests remain readiness checks; formal conclusions still require a qualified native Linux or X5 test session with real artifacts.

On the actual X5, first generate a new report with --label rdk-x5. After the report allows every selected requirement and Git is clean, the held-out entry is:

python3 scripts/run_formal_experiment_session.py \
  --matrix experiments/protocol/formal_experiment_matrix.json \
  --capability-report data/raw/environment/rdk-x5.json \
  --case diagnosis_f1_injected \
  --case diagnosis_f2_injected \
  --case diagnosis_f3_injected \
  --case diagnosis_f4_injected \
  --case diagnosis_f6_injected \
  --dataset-role test \
  --session-name x5_test_01 \
  --seed 20260718 \
  --output-dir data/raw/experiments/test/x5_test_01

An interrupted session is continued with the same frozen arguments plus --resume. Resume verifies the manifest sidecar, matrix, capability report, Git commit, role, seed, and session name. Successful, failed, and interrupted cases are terminal and are never rerun in place; a new measurement attempt uses a new session name. Physical CAN is not part of this first formal matrix. Control variants and F5 are intentionally excluded here because they remain development-only until their formal evidence profiles are frozen.