Skip to content

fix: exclude initial conversation history from human interventions - #102

Open
YMuskrat wants to merge 5 commits into
AI-Safety-Institute:mainfrom
YMuskrat:fix/initial-history-interventions
Open

YMuskrat wants to merge 5 commits into
AI-Safety-Institute:mainfrom
YMuskrat:fix/initial-history-interventions

Conversation

@YMuskrat

@YMuskrat YMuskrat commented Oct 27, 2026 •

Copy link
Copy Markdown

While testing tasks with a supplied conversation history, I noticed that user examples could be counted as human interventions. Assistant examples could also shift the turn assigned to later steering. A conversation provided as task input should contribute neither interventions nor generated turns.

The cause is in human_intervention() in src/transect/scanners/base.py. Its nested execute() loop only skipped the first human-channel user message as the prompt, then treated later examples as interventions and advanced next_turn for assistant examples. interventions_df() exported those incorrect entries.

The scanner now requests sample_init events, and the new _initial_input_ids() helper reads the message IDs from SampleInitEvent.state["messages"]. Those messages are skipped before intervention counting and turn attribution. Genuine steering still counts, including messages before the first model call, while logs without initial-state provenance retain the existing fallback.

Regression tests cover supplied history, genuine steering, and a real Inspect scan/reload. The affected API and mechanical tests pass (62 tests), as do the full unit suite, Ruff, mypy and Pyright. Seven browser tests were skipped because Chromium could not run in this sandbox. Local tests disabled only Scout’s progress memory monitor because /proc is unavailable.

Related to #101.

@alexandraabbas alexandraabbas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review posted on behalf of @alexandraabbas (written with Claude Code).

Good catch and a clean fix. Reading the message ids off SampleInitEvent.state["messages"] is the right provenance (I checked the demo log: the ids there match sample.messages), the fallback to the old first-human-message rule when no sample_init is recorded is correct for OpenClaw inputs, and the scanner and frame docstrings are updated. A few things before merge.

Must fix

  1. Rebase and re-bump. The branch bumps 0.4.4 to 0.4.5, but main is at 0.7.0. Rebase onto main and bump to 0.7.1 in pyproject.toml and uv.lock.
  2. README wording. The added sentence under Human interventions reads as a contract statement in the middle of a bullet about what the section shows. Shorten to something like "Messages Inspect recorded as the sample's initial input are task setup, not interventions." and keep the mechanics in the frame docstring where they already are.

Smaller points

  • _initial_input_ids() is defensive at every level (getattr(event, "state"), isinstance(state, dict), isinstance(messages, list), isinstance(message, dict)). SampleInitEvent.state is Inspect's own serialised TaskState, so a missing messages list there is a broken log, not a shape to tolerate. One isinstance(state, dict) guard is plenty; let the rest raise. This repo prefers loud failure over silent fallback (see "Honest absence" in AGENTS.md).
  • The scanner now requests sample_init events for every transcript. That event carries the full initial state, which can be large on long-context tasks. Fine for now, but worth a one-line comment in the events= list saying why it is requested, since the other entries each have a reason stated.

Tests

The mechanical parametrized test is good. The API test that rewrites the demo log and runs two full scans is the one that actually proves the id contract against a real Inspect log, so keep it, but compare changed.interventions against a frozen expectation rather than running a second scan of the untouched log; the second scan doubles the test's cost for an equality we already know from the existing demo-log tests.

@YMuskrat
YMuskrat force-pushed the fix/initial-history-interventions branch from 12dbd2e to ccebafa Compare November 18, 2026 00:06
@YMuskrat

Copy link
Copy Markdown
Author

Thank you for the review and sorry for the late reply. I rebased on main, bumped to 0.7.1, and shortened the README wording. I also simplified _initial_input_ids() and added a comment explaining why the scanner requests sample_init. The regression test now runs one scan and checks the interventions against a fixed expectation. make check passed locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants