Skip to content

feat: compare pinned harness replay variants - #106

Merged
divo12 merged 17 commits into
mainfrom
feat/replay-variant-comparison
Aug 16, 2026
Merged

divo12 merged 17 commits into
mainfrom
feat/replay-variant-comparison

Conversation

@divo12

@divo12 divo12 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

User description

Summary

  • add immutable, provider-neutral harness variant and replay case contracts
  • compare baseline and candidate executions using canonical SessionSnapshot and RunTrace values
  • block promotion on failing critical criteria while preserving ordered artifacts and usage deltas
  • reject malformed executor identity, trace, artifact, completeness, and cumulative-usage results

Stack

Verification

  • focused replay and public API tests: 12 passed
  • Ruff: passed
  • mypy strict: passed
  • full Dream suite: all relevant tests passed; 3 pre-existing research-claw tests fail because this host has no python executable on PATH (only python3)

Review fixes

  • reject whitespace-only identifiers and empty replay suites
  • reject duplicate assertion/execution artifacts
  • validate every trace event session id
  • reject cumulative usage below the source snapshot
  • CodeAnt major: reject NaN and infinite source/result costs before promotion deltas

CodeAnt-AI Description

Add replay comparison, default OTLP tracing, and safe session recovery

What Changed

  • Compare pinned baseline and candidate harness variants across ordered replay cases, including outcomes, required artifacts, trace identity, completeness, and usage changes
  • Block promotion when a candidate fails a critical assertion while preserving noncritical degradations and per-case cost deltas
  • Export session traces to OTLP by default alongside durable JSONL traces, with endpoint and service settings configurable through environment variables; OTEL_SDK_DISABLED=true keeps JSONL-only tracing
  • Prevent concurrent session writers from overwriting newer snapshots, and report save conflicts with the affected revisions
  • Report role-session recovery as typed events, preserving concurrent replacements and isolating observer failures from session execution

Impact

✅ Safer harness variant promotion
✅ OTLP traces available without changing existing JSONL durability
✅ Fewer session transcript overwrites during concurrent recovery

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

@codeant-ai

codeant-ai Bot commented Aug 9, 2026

Copy link
Copy Markdown

🤖 CodeAnt AI — Review Status

Status Commit Started (UTC) Finished (UTC)
✅ Incremental review completed 2bf545f Aug 16, 2026 · 17:58 17:59
✅ Reviewed your PR 25ff6ab Aug 09, 2026 · 12:48 12:50

@codeant-ai

codeant-ai Bot commented Aug 9, 2026

Copy link
Copy Markdown

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

divo12 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@codeant-ai codeant-ai Bot added the size:XL This PR changes 500-999 lines, ignoring generated files label Aug 9, 2026
Comment thread src/dream/replay.py

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

divo12 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

Base automatically changed from feat/session-snapshot-trace to main August 16, 2026 16:11
divo12 and others added 9 commits August 16, 2026 22:20
Add bounded default-on OTLP export as an additive typed sink while preserving JSONL durability and explicit collector-free opt-out.

Co-authored-by: Cursor <cursoragent@cursor.com>
feat(observability): default-on OTLP export beside JSONL traces
Preserve snapshot truth across resume races and expose recovery actions without allowing observers to control session lifecycle.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the reviewed current-main implementation while joining the original PR branch without rewriting remote history.
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the reviewed current-main implementation while joining the original PR branch without rewriting remote history.
@codeant-ai

codeant-ai Bot commented Aug 16, 2026

Copy link
Copy Markdown

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

divo12 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@codeant-ai codeant-ai Bot added size:XXL This PR changes 1000+ lines, ignoring generated files and removed size:XL This PR changes 500-999 lines, ignoring generated files labels Aug 16, 2026
@divo12
divo12 merged commit b624fc8 into main Aug 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant