Skip to content

Add Claude observability parity and prepare v0.2.0 - #35

Merged
tmusser merged 17 commits into
mainfrom
feature/claude-observability-parity-v020
Jul 10, 2026
Merged

Add Claude observability parity and prepare v0.2.0#35
tmusser merged 17 commits into
mainfrom
feature/claude-observability-parity-v020

Conversation

@tmusser

@tmusser tmusser commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Summary

Brings the Claude harness up to the same observability contract as the merged Codex checkpoint path and prepares the repository for a v0.2.0 release.

Claude parity

  • Capture distinct Claude workspace states after completed file-changing tool results.
  • Briefly pause the Claude process group while copying stable snapshots.
  • Defer public and hidden evaluator execution until after Claude exits.
  • Keep hidden evaluator output outside the agent session.
  • Select the initial evaluator for initial runs and the resume evaluator for full/stripped resume runs.
  • Record complete/partial coverage, stable snapshot status, skipped states, snapshot pause overhead, evaluator overhead, and native observation unit.
  • Keep polling fallback conservative: stable/deferred snapshots are retained, but first-green remains non-conclusive because sampling can miss states.
  • Refuse exact stream coverage when only a final snapshot exists or any file-changing tool result is unresolved.

Provider capability rubric

Adds docs/capabilities.md with a Claude/Codex matrix covering:

  • native observation units;
  • stable snapshots;
  • deferred evaluation;
  • first functional and bench-ready green;
  • post-green tails;
  • coverage guarantees and fallback boundaries;
  • cross-provider comparison guidance.

The README includes a short provider-observability callout and link.

v0.2.0 preparation

  • Bump package version from 0.1.0 to 0.2.0.
  • Update the README release badge.
  • Add v0.2.0 changelog notes and explicit claim boundaries.

The tag/release should be created from main after this PR merges; this PR deliberately does not create a tag from an unmerged branch.

Adversarial boundary

Parity means equivalent evidence standards, not interchangeable units. Claude turns/tool-result boundaries and Codex provider items remain distinct. Claims about the first evaluator-green observed boundary require checkpoint_coverage_complete=true; neither path claims an instruction-level instant. Post-functional work is not automatically waste when it completes required bench-ready evidence.

Validation

  • Focused Claude/Codex parity tests: passed.
  • Fake-Claude red-to-green end-to-end test: passed.
  • Final-only and capped-coverage negative tests: passed.
  • Full benchmark_harness/tests suite: passed.
  • Shell syntax and git diff --check: passed.
  • Release metadata checks: passed.
  • Final repository CI: passed on Python 3.11 and Python 3.12.

The final diff contains only the 11 intended implementation, test, documentation, and release-metadata files.

@github-actions

Copy link
Copy Markdown
Contributor

Claude parity focused tests failed:

F.........................                                               [100%]
=================================== FAILURES ===================================
_ test_stream_json_observer_reports_first_green_turn_with_complete_stable_coverage _
benchmark_harness/tests/test_claude_solution_latency_parity.py:85: in test_stream_json_observer_reports_first_green_turn_with_complete_stable_coverage
    assert summary["first_functional_green_turn"] == 2
E   assert 1 == 2
=========================== short test summary info ============================
FAILED benchmark_harness/tests/test_claude_solution_latency_parity.py::test_stream_json_observer_reports_first_green_turn_with_complete_stable_coverage - assert 1 == 2
1 failed, 25 passed in 3.47s

@github-actions

Copy link
Copy Markdown
Contributor

Claude parity focused tests failed:

F.........................                                               [100%]
=================================== FAILURES ===================================
_ test_stream_json_observer_reports_first_green_turn_with_complete_stable_coverage _
benchmark_harness/tests/test_claude_solution_latency_parity.py:85: in test_stream_json_observer_reports_first_green_turn_with_complete_stable_coverage
    assert summary["first_functional_green_turn"] == 2
E   assert 1 == 2
=========================== short test summary info ============================
FAILED benchmark_harness/tests/test_claude_solution_latency_parity.py::test_stream_json_observer_reports_first_green_turn_with_complete_stable_coverage - assert 1 == 2
1 failed, 25 passed in 3.58s

@github-actions

Copy link
Copy Markdown
Contributor

Claude parity hygiene validation failed:

== bash syntax ==
bash_exit=0
== git diff --check ==
docs/scorecard.md:118: new blank line at EOF.
diff_exit=2

tmusser commented Jul 10, 2026

Copy link
Copy Markdown
Owner Author

Final validation update: the earlier red→green attribution failures and EOF hygiene failure were intermediate staging findings and are superseded. The cleaned head ccb35a045aa0fc2d8bf3b86f76e476dd6074dafb passes standard CI on Python 3.11 and 3.12. Focused Claude/Codex parity tests, the full harness suite, shell syntax, diff hygiene, release metadata, and negative coverage tests are green. Final diff: 11 intended files only.

@tmusser
tmusser marked this pull request as ready for review July 10, 2026 11:23
@tmusser
tmusser merged commit 4c4c316 into main Jul 10, 2026
2 checks passed
@tmusser
tmusser deleted the feature/claude-observability-parity-v020 branch July 10, 2026 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant