Repository navigation
chore: v0.1.29 cut: version, compose pins, changelog - #5
Merged
Merged
Conversation
…er the test cut Both the CLI (the RAGTruth adapter's calibration corpus) and the shell's split calibrate held out on the graded test cut, and the after round then graded the same cut, so a gate decision read test labels (in the 450-per-task pilot it refused the contrastive round, 0.68 against 0.76, on test rows). The corpus is now the calibration split alone: carve_dev (lithrim_bench/harness/calib_corpus.py) relabels a deterministic, source-disjoint 30% of each task's sources, ordered by the hash of the source id, as dev; the demos train on the rest and the gate scores on dev. run_optimize and scripts/optimize_judge.py take the held-out split (default test, unchanged for existing corpora; never the trainset); the CLI passes dev when the corpus carries it and an enriched round keeps the held-out rows; the BFF's split calibrate builds its corpus the same way and reports calibration/dev/test_untouched. The judge editor says how held-out is chosen, and both loop pages and the hold-out policy say it too, with the pilot's history. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mpt as an object
A RAGTruth QA source is a {question, passages} object stored as JSON text on a prose case;
the chat format passed the text to the paper prompt, which indexes the object, so every QA
row crashed the export (CLI and service alike). A prose source that parses to a JSON
object now reaches the prompt as that object; a row the training prompt still cannot fill
is refused by name with its task, never dropped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ning export draws on The training export needs graded calibration rows, and the shell could only grade the test split. The import card (after a load with a calibration split) and the palette offer 'Grade the calibration split (paid)', which grades split=calibration as round=calibration behind the cost confirm; its copy says the test split is not touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…a calibration round
The export control gains a format picker (generic, paper, azure-chat) and says which split
the round exports from; the training formats are offered only on a calibration-split round,
matching the service's 422. The importer manifest gains training_prompt_module (RAGTruth
names examples/ragtruth/arms/paper_prompt.py), the chat export defaults to it so the shell
sends no path, and GET /v1/importers lists each importer's training formats. The journey
fixture's QA sources now have RAGTruth's real {question, passages} shape; an end-to-end test
grades the calibration split as its own round and exports the chat training file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… shell Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate compared a new round's held-out score with the pinned set's sidecar score. A set pinned before the dev slice was scored on the test cut, a new round on the dev slice, so the gate could pin or refuse on the difference between two held-out sets. The manifest now names the held-out split and the sidecar records the held-out identity (split, size, SHA-256 of the sorted case ids, heldout_identity). pin_demos treats a mismatch, or an identity on one side only, as not comparable and pins with that reason stated (the first-pin rule; re-scoring the old set would need a paid call in a file step); with no identity on either side the old comparison holds. The response carries comparable, and the editor's pin line says when there was nothing comparable to beat. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
A short description.
Why
The problem or motivation.
Checklist
ruff check .is cleanpytest -qis green without a model key (credential-free)make demostill runs