The Librarian

The Librarian is an open-source control plane for large-scale PDF accessibility remediation. It is built around a simple idea: inaccessible document collections should be treated like a corpus that can be queued, inspected, re-authored, verified, learned from, and improved over time.

Instead of treating every PDF as a one-off repair job, The Librarian keeps a persistent remediation workspace. Each document run produces structured inputs, accessible authoring artifacts, rendered outputs, verification evidence, parity metrics, review packages, and operational telemetry. The long-term goal is a system that gets better as it processes more documents because reusable layout fixes, accessibility components, and failure signatures are promoted into a shared scaffold library.

What It Does

The Librarian coordinates the document remediation loop around a Django control plane:

1. Ingest a source PDF from a local path, upload, queue, or manifest.
2. Analyze the source document and record machine-readable inspection artifacts.
3. Generate structured document inputs such as document.data.json and document.layout.json.
4. Author an accessible replacement through a scaffold-first Python report project.
5. Render and verify with Fullbleed, including tagged/PDF-UA-oriented outputs when available.
6. Evaluate quality gates for accessibility, parity, structure, text, visual similarity, and review readiness.
7. Store every attempt, artifact, issue, score, and event for auditability.
8. Let the librarian loop inspect failing documents, apply targeted remediation, and promote reusable fixes.

In practical terms, it gives you:

• A queue and dashboard for document remediation jobs.
• A repeatable job/attempt model instead of ad hoc file folders.
• Evidence bundles for each document, not just an output PDF.
• A librarian command that can walk a corpus and improve the shared scaffold over time.
• Contract-validated JSON telemetry so automation can trust the artifacts it reads.
• Optional model orchestration through Codex or Claude for inspection and authoring assistance.
• A separate engine-feedback ledger for recurring Fullbleed or platform limitations found during real remediation work.

Why It Matters

Most accessibility remediation workflows still behave like manual craft pipelines: expensive, hard to audit, hard to reproduce, and hard to scale. The Librarian turns remediation into an operations problem with durable state, machine-readable evidence, and a learning loop.

The intended outcome is not just "make this one PDF pass." The intended outcome is: process a corpus, learn recurring document patterns, preserve provenance, expose what still needs human review, and make the next thousand documents easier.

Current Status

This is an early standalone distribution extracted from an MVP. The control plane, data model, commands, contracts, tests, and dashboard are present. The core smoke/test paths run without vendored customer data.

Full remediation depends on an installed Fullbleed runtime and, for model-assisted loops, a configured Codex or Claude CLI/API. Fullbleed is intentionally not vendored in this repository.

Repository Layout

• control_plane/: Django app, API, dashboard, worker commands, librarian loop, artifact storage, and Fullbleed/Codex/Claude adapters.
• contracts/: JSON schemas for observability, review packages, quality gates, orchestrator plans, and artifact manifests.
• scripts/obs_contracts.py: contract validation and event summarization helper.
• docs/: migration plan, architecture notes, and release checklist.
• examples/: generic sample manifest and job policy templates.
• references/README.md: optional Fullbleed reference checkout instructions.

Quick Start

From a clean checkout:

cd the_librarian\control_plane
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
copy .env.example .env
python manage.py migrate
python manage.py runserver

Open:

http://127.0.0.1:8000/

The dashboard lets you enqueue a local PDF path or upload a PDF, inspect jobs, download artifacts, open review packages, and launch/stop librarian runs.

Smoke Test Without Fullbleed

Use --skip-fullbleed to exercise the queue, attempt, artifact, and dashboard plumbing without requiring a Fullbleed install:

python manage.py run_phase0_job --source-pdf C:\path\to\document.pdf --skip-fullbleed

This is useful for verifying Django, database, queue, and artifact storage behavior. It does not produce a real remediated PDF.

Running A Real Document

Install or otherwise expose the Fullbleed CLI/Python runtime, then run:

python manage.py run_phase0_job --source-pdf C:\path\to\document.pdf

Useful stricter gates:

python manage.py run_phase0_job `
  --source-pdf C:\path\to\document.pdf `
  --max-attempts 3 `
  --require-cav `
  --require-quality-passed `
  --require-visual-similarity-min 0.95 `
  --require-text-similarity-min 0.97 `
  --require-pmr-score-min 90 `
  --require-pdf-ua-seed-ok

If the gates cannot be satisfied within the attempt budget, the job is routed to needs_review with artifacts and issue context instead of silently pretending the document is complete.

Queue Workers

Enqueue documents through the dashboard or API, then process one job:

python manage.py run_worker --queue default --once

Run continuously:

python manage.py run_worker --queue default --poll-seconds 2

Run a weighted dispatcher across queues:

python manage.py run_dispatcher --queues high:3,default:1,bulk:1 --poll-seconds 2

Batch Manifests

The batch runner accepts a manifest of documents and can either enqueue or execute them.

Enqueue only:

python manage.py run_corpus_batch `
  --manifest ..\examples\sample_manifest.json `
  --corpus-root ..\examples `
  --enqueue-only `
  --queue bulk

Execute a limited batch:

python manage.py run_corpus_batch `
  --manifest C:\path\to\manifest.json `
  --corpus-root C:\path\to\corpus-root `
  --limit 25 `
  --max-attempts 3 `
  --require-quality-passed

The Librarian Loop

The run_librarian command is the corpus-level governance loop. It walks queued documents, inspects the current state, performs a bounded remediation touch, records a decision, and moves on.

Deterministic mode, useful before configuring model tooling:

python manage.py run_librarian --queue bulk --limit 100 --skip-librarian-model

Model-assisted inspection:

python manage.py run_librarian `
  --queue bulk `
  --limit 100 `
  --librarian-cli-binary codex `
  --librarian-model gpt-5.3-codex-spark `
  --librarian-effort medium

The librarian is designed to behave like a maintainer of a growing accessibility scaffold, not a one-off document generator. Shared fixes should be promoted with provenance and reviewable evidence.

Artifacts And Evidence

Runtime output is written under:

control_plane/runtime/

Important artifact classes include:

• Source PDFs and source hashes.
• Analysis JSON and source inspection payloads.
• Structured document.data.json and document.layout.json.
• Authored HTML/CSS/Python report files.
• Rendered output PDFs and page images.
• Fullbleed verify manifests, page data, glyph reports, performance reports, and traces.
• Quality gate reports and normalized issues.
• Review packages and review decisions.
• Librarian run ledgers, memory, and engine-feedback exports.

Runtime artifacts are intentionally ignored by Git.

Configuration

Start from:

control_plane/.env.example

Common environment variables:

• DJANGO_SECRET_KEY
• DJANGO_DEBUG
• DJANGO_ALLOWED_HOSTS
• DJANGO_TIME_ZONE
• THE_LIBRARIAN_PRIMARY_RENDER_MODE
• THE_LIBRARIAN_SCAFFOLD_LIBRARY_DIR
• THE_LIBRARIAN_FULLBLEED_REFERENCE_DIR
• CODEX_CLI_BIN
• CLAUDE_ORCHESTRATOR_TRANSPORT
• CLAUDE_CLI_BIN
• ANTHROPIC_API_KEY

Fullbleed source references are optional but useful. Place a checkout at:

references/fullbleed-official/

or set:

$env:THE_LIBRARIAN_FULLBLEED_REFERENCE_DIR="C:\path\to\fullbleed-official"

API Surface

Core endpoints:

• POST /api/jobs
• GET /api/jobs
• GET /api/jobs/<job_uuid>
• POST /api/jobs/<job_uuid>/resume
• POST /api/jobs/<job_uuid>/review/start
• POST /api/jobs/<job_uuid>/review/submit
• GET /api/artifacts/<artifact_uuid>/download

Example enqueue payload:

{
  "source_uri": "file:///C:/path/to/document.pdf",
  "source_sha256": "",
  "priority": 100,
  "queue": "default",
  "metadata": {
    "collection": "example"
  }
}

Development

Run checks:

cd control_plane
python manage.py check
python manage.py test

Validate observability contracts:

python ..\scripts\obs_contracts.py validate-json --schema obs.event.v1 --input C:\path\to\event.json
python ..\scripts\obs_contracts.py validate-ndjson --schema obs.event.v1 --input C:\path\to\events.ndjson
python ..\scripts\obs_contracts.py summarize-events --input C:\path\to\events.ndjson --output C:\path\to\run_summary.json

Boundaries

The Librarian is not a legal certification tool. It is an automation and evidence system for accessibility remediation workflows. Human review remains necessary for ambiguous semantics, OCR-heavy sources, complex tables/forms, and final compliance signoff.

It is also not a general-purpose PDF editor. The preferred path is accessible re-authoring through structured data, semantic components, and deterministic rendering.

License

The Librarian source is distributed under the MIT License.

Fullbleed remains a separate dependency/reference with its own AGPL-3.0/commercial license terms. This repository does not vendor Fullbleed or grant rights to Fullbleed beyond the license you separately receive for it.