Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
- Row-level set evidence checks for required columns, binary membership, unique members, and reconciled A-only/overlap/B-only/neither counts.
- Schematic layouts for partial overlap, disjoint, subset, and equal-set relationships, with authoritative region labels preserved in Vega-Lite metadata.
- A runnable `examples/set_membership.py` artifact and dedicated set-membership contract guidance.
- A checked-in Vega-Lite proof artifact for the set-membership example.
- A package-build CI job that builds both distributions, checks metadata, installs the wheel in isolation, and smoke-tests the installed CLI and JSON report shape.
- CI coverage across the complete supported Python 3.10-3.13 range.

Expand All @@ -16,6 +17,8 @@
- Hardened CI with read-only permissions, disabled checkout credentials, pip caching, concurrency cancellation, job timeouts, dependency checks, and source compilation.
- Preserved the chart-specific verdict and statistical-diagnostic trap checks across every supported Python version.
- Made pytest fail closed on unknown configuration, undeclared markers, and unexpected `xfail` passes.
- Restored the README's diagnostic trap outcomes, companion-artifact explanation, and suite navigation after the set-membership insertion.
- Refreshed the roadmap and workflow artifacts to distinguish released v0.2.0 behavior from the unreleased set-membership slice.

## [0.2.0] - 2026-07-12

Expand Down
21 changes: 19 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,14 @@ Good diagnostic claims:

### Try the diagnostic traps

The runnable fixtures in [examples/traps/README.md](examples/traps/README.md) make the diagnostic boundaries concrete.
The runnable fixtures in [examples/traps/README.md](examples/traps/README.md) make the diagnostic boundaries concrete:

- [Severe QQ tail departure](examples/traps/qq_heavy_tails.vl.json) → `REVIEW` because the normality claim overstates the evidence.
- [QQ plot without a reference line](examples/traps/qq_missing_reference_line.vl.json) → `BLOCK` because the visual contract is incomplete.
- [Residuals with an obvious fitted-value pattern](examples/traps/residual_obvious_pattern.vl.json) → `REVIEW` because “no pattern” contradicts the data.
- [Four-point residual diagnostic](examples/traps/diagnostic_tiny_sample.vl.json) → `BLOCK` because the sample cannot support interpretation.

Each fixture includes a Vega-Lite spec, CSV data, and separate claim text so the verdict can be reproduced directly through the CLI.

## Set Membership Charts

Expand Down Expand Up @@ -290,4 +297,14 @@ chart = Chart.set_membership(

The renderer supports partial overlap, disjoint, subset, and equal-set relationships. Circle geometry is schematic; labeled A-only, overlap, B-only, and neither counts are authoritative and are preserved in `usermeta`.

Run `python examples/set_membership.py` to write `examples/output/set_membership_chart.vl.json`. See [the set membership contract](docs/SET_MEMBERSHIP.md) for the evidence shape, audit rules, and intentional two-set boundary.
Run `python examples/set_membership.py` to write [the generated Vega-Lite proof artifact](examples/output/set_membership_chart.vl.json). See [the set membership contract](docs/SET_MEMBERSHIP.md) for the evidence shape, audit rules, and intentional two-set boundary.

## Companion Artifact

This repo was built using `ai-engineering-skills` and is intended as the analytical-integrity proof artifact companion to `context-to-action-skills`.

See the [agent workflow case study](docs/AGENT_WORKFLOW_CASE_STUDY.md) and [build manifest](artifacts/BUILD_MANIFEST.md) for the proof trail.

## Part of the Suite

See the [suite map](docs/SUITE_MAP.md) for how `chart-contract`, `ai-engineering-skills`, and `context-to-action-skills` fit together as one story.
64 changes: 30 additions & 34 deletions ROADMAP.md
Original file line number Diff line number Diff line change
@@ -1,47 +1,43 @@
# Roadmap

## v0.1 polish
## Released: v0.2.0

- README hero clarity
- rendered example artifact
- audit output example
- packaging hygiene
The v0.2.0 release established `chart-contract` as a deterministic agent gate:

## v0.2.0 — Agent Gate
- CLI audits for Vega-Lite specs with optional CSV or JSON evidence
- text, JSON, and Markdown reports with stable `READY`, `REVIEW`, and `BLOCK` behavior
- distribution intents: `Chart.histogram()`, `Chart.boxplot()`, and `Chart.violin()`
- statistical diagnostic intents: `Chart.qq()`, `Chart.ecdf()`, and `Chart.residual()`
- runnable trap fixtures and agent-facing claim guidance

Goal: make `chart-contract` usable as a deterministic CLI gate for chart audits from disk before any code changes spill into v0.3.
## Current main: Unreleased

Release checklist:
The current unreleased line adds and hardens:

- [x] CLI entrypoint and spec loader: audit a Vega-Lite spec from disk and accept an explicit claim argument.
- [x] Optional data input: support CSV and JSON data files with deterministic parse and file errors.
- [x] Report emitters: produce text, JSON, and Markdown audit reports from the same audit result.
- [x] CI-friendly exit codes: map pass, warn, and fail outcomes to stable process exit codes for agents and CI.
- [x] Runnable trap fixtures: add examples that exercise weak claims, missing provenance, and other audit failures.
- [x] README and docs usage: add copy-paste examples that show the agent gate flow end to end.
- [x] Release prep: bump the package version and add a v0.2.0 changelog entry.
- `Chart.set_membership()` for audited two-set membership and Venn-style relationships
- row-level universe, binary-membership, unique-member, and reconciled-region checks
- schematic partial-overlap, disjoint, subset, and equal-set layouts with authoritative labels
- CI across Python 3.10-3.13 plus isolated wheel build and install checks
- generated proof artifacts and current-state documentation kept in sync with the implementation

Explicit non-goals:
## Next

- Do not add new chart intents.
- Do not add `ChartContract` yet.
- Do not add semantic denominator/grain rules yet.
- Do not add auto-correction.
- Do not build a dashboard or chart generator.
- decide the release version and release notes for the set-membership slice
- keep generated example artifacts deterministic and reviewable
- extend CLI/spec auditing only where evidence can be reconstructed without pretending arbitrary visuals are semantically complete
- add new intents only when their data, claim, and visual contracts can be tested explicitly

## v0.3 distribution and diagnostics preview
## Later

- Distribution intents: `Chart.histogram()`, `Chart.boxplot()`, `Chart.violin()`
- Statistical diagnostic intents: `Chart.qq()`, `Chart.ecdf()`, `Chart.residual()`
- Distribution audit rules: numeric metric checks, sample-size thresholds, grouped category thresholds, histogram bins, violin density warnings
- Diagnostic audit rules: QQ reference support and tie density, fitted-value types, residual sample size, residual variation, zero-reference guarantee
- Deterministic normal quantiles and ECDF preparation using only the standard library plus pandas
- Example artifacts: `examples/distribution_charts.py`, `examples/statistical_diagnostics.py`
- matrix or UpSet-style membership intent for more than two sets
- additional reference distributions after their semantics and tests are explicit
- optional renderer and export improvements
- more annotation primitives

## Later
## Continuing boundaries

- Additional reference distributions only after their semantics and tests are explicit
- More visual intents only after the audit contract is strong
- Optional renderer and export improvements
- More annotation primitives
- v0.3 starts only after the gate is stable enough to trust
- no automatic chart correction
- no dashboard or chart-generator product surface
- no formal statistical certification
- no area-proportional Venn fitting in the two-set intent
- no broad plotting-library coverage without an auditable contract
52 changes: 28 additions & 24 deletions artifacts/HANDOFF.md
Original file line number Diff line number Diff line change
@@ -1,39 +1,43 @@
# HANDOFF

RESUME PACKET
## Resume Packet

- Goal: harden QQ and residual diagnostics with claim-support traps, reference-line contracts, and agent-facing claim guidance.
- Workflow State: runtime rules, first-party spec semantics, four trap triplets, tests, docs, and CI are published in draft PR #5; the full permanent GitHub Actions suite passed.
- Branch: `agent/diagnostic-claim-traps`
- Pull request: #5 (`feat: harden diagnostic claim audits`)
- Next task: review the deterministic thresholds and trap verdicts, then squash-merge if the boundaries are acceptable.
- Verification: full pytest, all legacy CLI smoke checks, and all four diagnostic trap checks passed in GitHub Actions run #155.
- Read first: `src/chart_contract/statistical_audit.py`, `src/chart_contract/audit.py`, `src/chart_contract/renderers/altair.py`, `tests/test_statistical_intents.py`, `docs/DIAGNOSTIC_CLAIMS.md`, `artifacts/VERIFY.md`
- Goal: restore README content displaced by the set-membership insertion and align current-state docs and proof artifacts with `main`.
- Workflow state: `Chart.set_membership()` is on `main`, its full CI matrix passed, and this cleanup branch contains documentation and generated-artifact follow-through only.
- Branch: `agent/restore-docs-and-proof-artifacts`
- Base: `main` at `b5ae6c6edf852e1d510951b69193a63c3cf845c0`
- Next task: review the cleanup diff, confirm CI, and merge if the restored navigation and current-state wording are accurate.
- Read first: `README.md`, `ROADMAP.md`, `artifacts/SPEC.md`, `examples/output/set_membership_chart.vl.json`, and `CHANGELOG.md`.

## Current Repo State

- First-party QQ specs declare `chart_contract_intent=qq` and `qq_reference_distribution=normal`; residual and ECDF specs preserve their statistical intent without changing metadata for older chart types.
- QQ audits require a fitted normal reference line and warn when outer-tail departure reaches 0.8 sample standard deviations.
- Residual audits warn when absolute fitted/residual correlation reaches 0.5 or ordered-thirds mean shift reaches one residual standard deviation.
- Nonnumeric fitted values fail deterministically without entering the pattern calculation.
- `audit_spec()` applies these semantics only when first-party statistical intent metadata is declared.
- Trap fixtures distinguish complete-but-overstated diagnostics (`REVIEW`) from structurally invalid or undersampled diagnostics (`BLOCK`).
- v0.2.0 provides the CLI gate, distribution intents, and statistical diagnostic intents.
- Unreleased `main` adds audited two-set membership charts with explicit row-level evidence and schematic geometry.
- CI covers Python 3.10-3.13 and separately builds, inspects, installs, and smoke-tests the wheel.
- The set-membership example now has a checked-in Vega-Lite proof artifact matching the documented output path.
- Diagnostic trap links, the companion-artifact explanation, and the suite map are restored in the README.
- The roadmap and spec now distinguish released v0.2.0 behavior from the unreleased set-membership slice.

## Important Decisions

- Treat reference lines as visual evidence contracts, not optional decoration.
- Keep obvious-pattern detection deterministic and explainable rather than adding opaque tests or smoothing.
- Use `REVIEW` for claims contradicted by visible tail/pattern evidence; reserve `BLOCK` for missing reference layers, invalid fields, or fewer than five complete observations.
- Preserve the existing `usermeta` contract for trend, rank, compare, histogram, boxplot, and violin charts.
- Keep normality and model-adequacy language explicitly non-certifying.
- Restore displaced README material rather than choosing between feature documentation and suite context.
- Keep this PR free of runtime changes; the set-membership implementation already passed CI on `main`.
- Treat labeled region counts as evidence and circle area as schematic.
- Describe the package as an auditable harness, not a statistical certifier or general visualization library.
- Keep more-than-two-set membership as a separate future matrix or UpSet-style intent.

## Verification

- Parse the generated set-membership Vega-Lite artifact as JSON.
- Check Markdown structure, internal links, trailing whitespace, and final newlines.
- Rely on GitHub Actions for the full supported Python matrix and isolated wheel checks.

## Remaining Risks

- The thresholds catch obvious tail, monotonic, and broad curved structure; they do not exhaust every residual failure mode such as heteroskedasticity or autocorrelation.
- QQ tail checks use a fitted normal location-scale line and are intentionally not a formal normality test.
- First-party spec semantics depend on `usermeta.chart_contract_intent`; arbitrary external specs remain under the generic audit contract.
- The branch history is noisy from connector-assisted publication, so squash merge is recommended.
- The generated artifact is review evidence, not a browser screenshot; visual appearance still depends on Vega-Lite rendering.
- `audit_spec()` cannot reconstruct row-level membership evidence from arbitrary external layered specs.
- The package remains versioned at 0.2.0 while set membership is recorded under Unreleased; the next release number is intentionally undecided.

## Next Recommended Task

Review draft PR #5 with special attention to the 0.8-SD QQ threshold, the residual correlation/thirds thresholds, and the REVIEW-versus-BLOCK trap split.
Choose the release version for the set-membership slice after this cleanup merges, then prepare release notes without adding another visual intent at the same time.
82 changes: 47 additions & 35 deletions artifacts/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,73 +2,85 @@

## Objective

Build `chart-contract`, a lightweight Python harness for claim-first, audited analytical charts that turns analytical claims into inspected `trend`, `rank`, and `compare` charts with Altair/Vega-Lite output and deterministic PASS/WARN/FAIL audit findings.
Build `chart-contract`, a lightweight Python harness for claim-first analytical charts that makes the claim, evidence shape, visual intent, provenance, and known limitations inspectable before a chart is shared.

The package should emit Altair/Vega-Lite output and deterministic `PASS`/`WARN`/`FAIL` findings summarized as `READY`, `REVIEW`, or `BLOCK`.

## Audience

- Analysts and analytics engineers who want auditable chart defaults.
- AI-assisted builders who need a thin contract layer before sharing charts.

## v0.1 Scope

- `Chart.trend()`, `Chart.rank()`, `Chart.compare()`
- `chart.audit()`
- `chart.to_altair()`
- `chart.to_vega_lite()`
- experimental `audit_spec()`
- Agent workflows that need a deterministic CLI gate and durable audit report.

## Current Scope

- descriptive intents: `Chart.trend()`, `Chart.rank()`, and `Chart.compare()`
- distribution intents: `Chart.histogram()`, `Chart.boxplot()`, and `Chart.violin()`
- statistical diagnostic intents: `Chart.qq()`, `Chart.ecdf()`, and `Chart.residual()`
- two-set membership intent: `Chart.set_membership()`
- `chart.audit()` for first-party chart contracts
- experimental `audit_spec()` for supported Vega-Lite evidence shapes
- `chart.to_altair()` and `chart.to_vega_lite()`
- `chart-contract audit spec` with text, JSON, and Markdown reports
- Altair/Vega-Lite as the only renderer
- deterministic PASS/WARN/FAIL findings
- docs, tests, examples, and build-proof artifacts
- deterministic, explainable audit findings and stable CLI exit behavior
- docs, tests, traps, examples, and generated proof artifacts

## v0.3 Preview
## Evidence Boundaries

- Distribution intents: `Chart.histogram()`, `Chart.boxplot()`, `Chart.violin()`
- Statistical diagnostic intents: `Chart.qq()`, `Chart.ecdf()`, `Chart.residual()`
- distribution-specific audit rules for numeric value fields, sample size, grouped categories, histogram bins, and violin density warnings
- statistical diagnostic rules for QQ reference support, tie density, fitted-value types, residual sample size, and residual variation
- deterministic normal-reference QQ and ECDF preparation without SciPy or external data fetching
- `examples/distribution_charts.py` and `examples/statistical_diagnostics.py`
- Trend, rank, compare, distribution, diagnostic, and membership claims require explicit fields and usable observations.
- QQ and residual charts provide visual diagnostic guardrails, not formal normality or model-adequacy certification.
- Set membership requires one row per unique universe member and exactly two explicit boolean or integer `0`/`1` membership columns.
- Venn-style circle geometry is schematic; labeled region counts are authoritative.
- Arbitrary external Vega-Lite specs are audited only where the required evidence can be reconstructed deterministically.

## Non-Goals

- UI, dashboards, or Streamlit
- automatic chart correction
- renderers beyond Altair/Vega-Lite
- broad plotting-library coverage beyond the explicitly supported intents
- broad plotting-library coverage beyond explicitly supported intents
- external data fetching, LLM calls, telemetry, or theme systems
- three-or-more-set Venn diagrams or area-proportional Venn fitting
- unverifiable claims of statistical, accessibility, or design certification

## Acceptance Criteria

- Public API supports the usage shown in the brief.
- Audit layer catches required completeness, data-contract, visual-form, claim-support, provenance, and Tufte-inspired integrity checks.
- `audit_spec()` flags non-zero bar baselines and overcrowded pie/arc charts.
- Examples run on synthetic data and write Vega-Lite JSON into `examples/output/`.
- README leads with the bad-chart to corrected-chart story.
- Artifacts document the `ai-engineering-skills` workflow and verification evidence.
- Public API supports every intent listed in Current Scope.
- Audit findings cover contract completeness, usable data, visual form, claim support, provenance, and explainable visual-integrity checks.
- The CLI returns stable reports and exit codes for `READY`, `REVIEW`, and `BLOCK`.
- First-party generated specs preserve intent and evidence metadata required for downstream auditing.
- Examples run on synthetic data and write inspectable Vega-Lite JSON into `examples/output/`.
- CI tests the supported Python range and validates an isolated built wheel.
- README, roadmap, changelog, and workflow artifacts describe current behavior without overstating guarantees.

## Constraints

- Python 3.10+
- Minimal dependencies: `pandas`, `altair`, `pytest`
- Keep models simple and inspectable
- Explainable warnings only; no unverifiable claims of compliance or certification
- Runtime dependencies limited to `pandas` and `altair`
- Simple, inspectable models and deterministic thresholds
- Explainable warnings only
- New intents require explicit data, claim, visual, test, and documentation contracts

## Verification Commands

- `python -m pip install -e ".[dev]"`
- `pytest`
- `python -m pytest`
- `python examples/bad_to_good_chart.py`
- `python examples/trend_claim.py`
- `python examples/rank_claim.py`
- `python examples/compare_claim.py`
- `python examples/distribution_charts.py`
- `python examples/statistical_diagnostics.py`
- `python examples/set_membership.py`
- `chart-contract --version`
- `git diff --check`

## Smallest Verification Demo

Run `python examples/bad_to_good_chart.py` to show a risky spec audited with failures and a corrected contract-driven chart emitted as Vega-Lite JSON.
Run `python examples/bad_to_good_chart.py` to compare a risky chart that still renders with a corrected contract-driven chart and inspect the emitted audit evidence.

For set membership, run `python examples/set_membership.py` and verify that A-only, overlap, B-only, neither, and universe counts reconcile in the generated spec metadata.

## Open Questions

- Event annotations in Altair will be kept simple in v0.1 unless implementation complexity rises.
- Subtitle/provenance handling may rely on chart metadata rather than full visual subtitles.
- What release version should carry the set-membership intent?
- Should a future many-set intent use an UpSet-style matrix rather than circles?
- Which additional external-spec shapes can be audited without inventing missing semantic evidence?
Loading
Loading