Repository navigation
Conversation
alexandraabbas
left a comment
There was a problem hiding this comment.
Review posted on behalf of @alexandraabbas (written with Claude Code).
Good catch and a clean fix. Reading the message ids off SampleInitEvent.state["messages"] is the right provenance (I checked the demo log: the ids there match sample.messages), the fallback to the old first-human-message rule when no sample_init is recorded is correct for OpenClaw inputs, and the scanner and frame docstrings are updated. A few things before merge.
Must fix
- Rebase and re-bump. The branch bumps 0.4.4 to 0.4.5, but
mainis at 0.7.0. Rebase ontomainand bump to 0.7.1 inpyproject.tomlanduv.lock. - README wording. The added sentence under Human interventions reads as a contract statement in the middle of a bullet about what the section shows. Shorten to something like "Messages Inspect recorded as the sample's initial input are task setup, not interventions." and keep the mechanics in the frame docstring where they already are.
Smaller points
_initial_input_ids()is defensive at every level (getattr(event, "state"),isinstance(state, dict),isinstance(messages, list),isinstance(message, dict)).SampleInitEvent.stateis Inspect's own serialisedTaskState, so a missingmessageslist there is a broken log, not a shape to tolerate. Oneisinstance(state, dict)guard is plenty; let the rest raise. This repo prefers loud failure over silent fallback (see "Honest absence" in AGENTS.md).- The scanner now requests
sample_initevents for every transcript. That event carries the full initial state, which can be large on long-context tasks. Fine for now, but worth a one-line comment in theevents=list saying why it is requested, since the other entries each have a reason stated.
Tests
The mechanical parametrized test is good. The API test that rewrites the demo log and runs two full scans is the one that actually proves the id contract against a real Inspect log, so keep it, but compare changed.interventions against a frozen expectation rather than running a second scan of the untouched log; the second scan doubles the test's cost for an equality we already know from the existing demo-log tests.
12dbd2e to
ccebafa
Compare
|
Thank you for the review and sorry for the late reply. I rebased on |
While testing tasks with a supplied conversation history, I noticed that user examples could be counted as human interventions. Assistant examples could also shift the turn assigned to later steering. A conversation provided as task input should contribute neither interventions nor generated turns.
The cause is in
human_intervention()insrc/transect/scanners/base.py. Its nestedexecute()loop only skipped the first human-channel user message as the prompt, then treated later examples as interventions and advancednext_turnfor assistant examples.interventions_df()exported those incorrect entries.The scanner now requests
sample_initevents, and the new_initial_input_ids()helper reads the message IDs fromSampleInitEvent.state["messages"]. Those messages are skipped before intervention counting and turn attribution. Genuine steering still counts, including messages before the first model call, while logs without initial-state provenance retain the existing fallback.Regression tests cover supplied history, genuine steering, and a real Inspect scan/reload. The affected API and mechanical tests pass (62 tests), as do the full unit suite, Ruff, mypy and Pyright. Seven browser tests were skipped because Chromium could not run in this sandbox. Local tests disabled only Scout’s progress memory monitor because /proc is unavailable.
Related to #101.