Skip to content

data-contract.md

Latest commit

 

History

History
189 lines (143 loc) · 7.66 KB

data-contract.md

File metadata and controls

189 lines (143 loc) · 7.66 KB
Raw

Data contract

This page is the canonical description of what steerbench reads and what it refuses to produce. Two separate contracts live here:

  1. the CSV schemas a sweep producer has to emit for steer-report to render a card, and
  2. the repeng boundary — steerbench consumes steering vectors, it does not extract them.

The module docstrings in report.py and vectors.py point here; if the two ever disagree, the code wins and this page is the bug.

1. CSV schemas

Three CSVs feed the four-part card. They are produced by the GPU sweep in experiments/modal_app.py (the steerbench[gpu] harness) and consumed on CPU by report.load_dose_curve, report.load_layer_curve and report.load_side_effects.

Rules that hold for all three:

  • Header row required. All three are read with csv.DictReader.
  • Extra columns are tolerated and ignored. A producer may carry whatever diagnostics it likes.
  • Missing required columns fail loudly, before any row is parsed: ValueError: <path> is missing columns: [...].
  • Every value in a required numeric column must parse — there is no missing-value sentinel and no row skipping.

1a. Dose response — load_dose_curve

Default path artifacts/dose_response.csv; steer-report --dose-csv.

One row per (coeff, seed) pair.

column required meaning
coeff ✅ injection coefficient (the dose); the x axis
seed ✅ generation seed for this sample
formality ✅ the effect axis — behaviour score for the concept
repetition ✅ coherence axis, higher is worse
ppl ✅ coherence axis (perplexity), lower is better
alpha_norm — normalised dose; carried by the sweep, ignored here

The first rows of the committed artifacts/dose_response.csv, showing two seeds at the same coeff:

coeff,seed,alpha_norm,formality,repetition,ppl
-60.0,0,-0.13114754098360656,2.0294648914518314,0.29126370083816894,10.909140628067348
-60.0,1,-0.13114754098360656,2.848055555555556,0.10565149136577706,12.38439366631976

The required set is exactly {coeff, seed, formality, repetition, ppl} — _SWEEP_COLUMNS plus the x column.

Known drift. The effect column name is currently hardcoded to formality, but the cross-model runs in artifacts/ (sentiment, verbosity, and the redosed layer sweeps) emit it as effect, so steer-report cannot read them as committed. That is issue #30; this page will need a one-line update when it lands.

1b. Layer sweep — load_layer_curve

Default path artifacts/layer_sweep.csv; steer-report --layer-csv.

The same effect/coherence tail as the dose sweep, keyed on layer instead of coeff — one row per (layer, seed) pair, one vector injected at every depth.

column required meaning
layer ✅ injection layer index; the x axis
seed ✅ generation seed for this sample
formality ✅ the effect axis
repetition ✅ coherence axis, higher is worse
ppl ✅ coherence axis, lower is better
layer_pos — ignored
dir_norm — per-layer direction L2, ≈1.0 since repeng unit-normalises; ignored
resid_norm, coeff, alpha_norm — ignored

The x column is a parameter (load_layer_curve(path, x_column="layer")), so a sweep keyed on something else can be read without changing the file.

1c. Side effects — load_side_effects

steer-report --side-csv; optional. If it is not passed, the CLI writes a header-only stub and the card renders an empty "Side effects" table rather than failing. If it is passed and does not exist, that is an error — a typo should not silently degrade to the stub.

One row per benchmark slice. All three columns are required and there are no optional ones.

column required meaning
benchmark ✅ slice name, e.g. mmlu, gsm8k
unsteered_acc ✅ accuracy with no steering applied
steered_acc ✅ accuracy with the vector injected 
benchmark,unsteered_acc,steered_acc
mmlu,0.62,0.55
gsm8k,0.40,0.41

SideEffect.delta is steered_acc - unsteered_acc, so negative means the vector degraded the capability. The renderer colours it on that sign.

1d. Raw-per-seed, not pre-aggregated

This is the part that most often surprises a new producer:

Emit one raw row per seed. Do not pre-aggregate.

report.py computes the statistics itself, in _combine:

  • the reported centre is statistics.fmean of the per-seed values;
  • the reported spread is the population standard deviation (statistics.pstdev) across seeds, and is 0.0 for a single seed;
  • n_seeds counts distinct seed values in the group.

Rows are grouped by the x column and the resulting points are sorted by x, so row order in the file does not matter and duplicate x values are expected.

Handing the reader a pre-averaged file is not detected — it will silently be treated as a one-sample-per-x sweep and every error bar will collapse to zero. CONTRIBUTING.md asks for 3+ seeds for exactly this reason.

1e. Coherence direction

Coherence is carried as two raw axes, matching what metrics.CoherenceScore returns, with no sign convention imposed on the producer. Dump exactly what metrics.py gives you. The direction is owned by the consumer, in report.COHERENCE_DIRECTION:

axis column direction
perplexity ppl lower_is_better
repetition repetition higher_is_worse

A point is coherent when it satisfies both axes — the cliff can appear in either one:

  • analyze_dose — perplexity within perplexity_tol (default 0.5) of the baseline point, i.e. ppl <= baseline_ppl * 1.5, and repetition <= repetition_cap (default 0.5). The baseline is the point with coeff nearest zero, not the smallest coeff, so a two-sided sweep is handled symmetrically.
  • analyze_layers — perplexity within perplexity_tol (default 1.0, and deliberately more generous) of the best perplexity anywhere in the sweep, plus the same repetition cap. A degenerate trap means a catastrophic blowup, not a hair above the tightest floor.

2. The repeng boundary

steerbench consumes steering vectors. It does not reimplement extraction.

Producing a direction from contrastive prompts is repeng's job. steerbench.vectors exists to read repeng's output durably and to describe it, and there is deliberately no training or extraction code anywhere in src/.

What load_vector accepts:

  • .gguf — repeng's native ControlVector.export_gguf format: architecture controlvector, KV fields controlvector.model_hint and controlvector.layer_count, and one tensor per layer named direction.{layer}.
  • .pt — a plain dict[int, tensor] mapping layer to direction, as a fallback for anything that is not repeng.

What save_vector writes is a superset of the native format: the same tensors and KV fields, plus steerbench.concept and steerbench.repeng_version. repeng's import_gguf ignores KV fields it does not recognise, so a file steerbench wrote stays natively loadable by repeng. That is the whole point of the superset — the boundary is one-way in code but round-trippable in data.

Practically, for a contributor:

  • Adding a new concept means adding example prompts and a vector, not extraction code.
  • Adding a new model means a new sweep producing the CSVs above.
  • If you find yourself writing PCA over hidden states, you are in the wrong repo — that belongs upstream in repeng.