From 67c8fccac1746468cacd5f6c8729e05f0cb11e1c Mon Sep 17 00:00:00 2001 From: Blake Bertuccelli-Booth <46652+bbertucc@users.noreply.github.com> Date: Thu, 24 Sep 2026 18:28:17 -0400 Subject: [PATCH 1/7] feat: optionally return the uploaded PDF, tagged, with its form filled in Iris can now call equalify-iris-pdf, a separate optional tool, to tag the source PDF from the extracted HTML. It is off unless tagged_pdf.command is set, and Iris behaves as before without it. - A session made from one PDF keeps that PDF, only when the feature is on. - GET /v1/sessions/{id}/fields returns the PDF's form fields. - POST /v1/sessions/{id}/pdf returns the tagged PDF as base64 JSON with the tagger's report. Form values are used for that request only. They are written to a 0600 scratch file that is removed afterwards, and never logged. - GET /v1/limits gains tagged_pdf: true|false. - The demo shows a "Get your PDF back, tagged" form built from the PDF's own fields, only when the deployment has the feature and the session has a PDF. Co-Authored-By: Claude Opus 5.5 --- README.md | 6 + config.example.yaml | 14 ++ docs/API.md | 65 ++++++++- public/demo.html | 187 +++++++++++++++++++++++++ src/config.ts | 3 + src/index.ts | 4 + src/routes/limits.ts | 7 +- src/routes/sessions.ts | 123 +++++++++++++++++ src/store/paths.ts | 8 +- src/util/taggedPdf.ts | 127 +++++++++++++++++ test/demo-pdf-form.test.ts | 99 ++++++++++++++ test/fixtures/fake-iris-pdf.mjs | 41 ++++++ test/tagged-pdf.test.ts | 236 ++++++++++++++++++++++++++++++++ 13 files changed, 914 insertions(+), 6 deletions(-) create mode 100644 src/util/taggedPdf.ts create mode 100644 test/demo-pdf-form.test.ts create mode 100755 test/fixtures/fake-iris-pdf.mjs create mode 100644 test/tagged-pdf.test.ts diff --git a/README.md b/README.md index e4510d0..266e9ce 100644 --- a/README.md +++ b/README.md @@ -171,6 +171,10 @@ key inline** — this section is the decisions an operator has to make, not the for a single Caddy/nginx). Without it every caller presents as the proxy's address and shares one bucket; the log warns when it sees an `X-Forwarded-For` while this is unset. Anything Iris cannot interpret warns and trusts nothing rather than failing startup. +- **Tagged PDFs — optional, off by default.** Set `tagged_pdf.command` to the `iris-pdf` command from + [equalify-iris-pdf](https://github.com/EqualifyEverything/equalify-iris-pdf), and a session made from + one PDF can return that PDF tagged, with its form filled in. The demo page shows this only when it is + on. Iris runs without it. ### One GitHub identity, and no sign-in @@ -222,6 +226,8 @@ in the other direction: it has its own shared secret and 404s unless you set it. | `POST /v1/sessions` | Create a session, upload images and/or PDFs (`multipart/form-data`) | | `GET /v1/sessions/{id}` | Poll status | | `GET /v1/sessions/{id}/output` | Fetch the HTML when ready | +| `GET /v1/sessions/{id}/fields` | The uploaded PDF's form fields (only with tagged PDFs on) | +| `POST /v1/sessions/{id}/pdf` | The uploaded PDF back, tagged, with its form filled in (only with tagged PDFs on) | | `POST /v1/sessions/{id}/feedback` | Submit feedback, trigger a re-run | | `POST /v1/sessions/{id}/close` | Finalize the session and clean tmp | | `GET /v1/sessions/{id}/logs` | Fetch the run log (ndjson) | diff --git a/config.example.yaml b/config.example.yaml index 1cc26fc..45f8256 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -93,6 +93,20 @@ storage: # alternative for multi-instance deployments (not implemented in v1). database: ./data/iris.sqlite +# Optional: give back the uploaded PDF, tagged for accessibility, with form fields filled +# in. Needs the `iris-pdf` command from +# https://github.com/EqualifyEverything/equalify-iris-pdf, which is not part of Iris. +# Blank (the default) turns it off, and Iris works as it does without it. +# +# With it on, a session made from one PDF keeps that PDF, and the demo page offers +# a form for the PDF's fields and "Download tagged PDF". Form values are personal data: +# they are used for one request, and Iris stores neither them nor the filled PDF. +tagged_pdf: + # The command to run, e.g. `iris-pdf` if it is on PATH. + command: ${IRIS_PDF_COMMAND} + # How long one PDF may take before the command is stopped. + timeout_seconds: 300 + # Iris has ONE GitHub identity: the token below. The server holds it, callers never # see it, and nobody signs in. Every session is owned by that account and every issue # Iris files is filed under it. diff --git a/docs/API.md b/docs/API.md index 9075623..8ac3d89 100644 --- a/docs/API.md +++ b/docs/API.md @@ -712,7 +712,8 @@ curl -s "$BASE/limits" | jq "upload_per_minute": 12, "max_upload_memory_mb": 256, "window_seconds": 60 - } + }, + "tagged_pdf": false } ``` @@ -740,7 +741,8 @@ are streaming: the byte total is checked before the body is read (or counted as request declares no length), while the part count is refused during parsing, once a part past `max_files` appears. `rate_limits` is how often you may ask ([Rate limits](#rate-limits-how-often-you-may-ask)), and is `null` on a deployment that does not -limit requests in the app — which means "not limiting", not "unknown". +limit requests in the app — which means "not limiting", not "unknown". `tagged_pdf` says whether +this deployment can [return a tagged PDF](#get-a-tagged-pdf-optional). A PDF's **links survive**, which rasterizing alone would not manage: a link is an annotation over the page rather than something drawn on it, so the page image carries the link text and @@ -956,6 +958,53 @@ those references 404 until a consumer supplies the files. What a screen-reader u browser will show broken images, and one that rewrites the `src`s has the log and the fragment to match them against. +## Get a tagged PDF (optional) + +A deployment that runs [equalify-iris-pdf](https://github.com/EqualifyEverything/equalify-iris-pdf) +can give you the original PDF back with tags, built from the HTML. The feature is off unless +`tagged_pdf.command` is set in the config. `GET /v1/limits` says whether it is on. + +It works only for a session made from **one PDF**. Iris keeps that PDF so it can tag it. Sessions made +from images, or from several files, get `409 no_source_pdf`. + +**Read the PDF's form fields:** + +```bash +curl -s -H "$AUTH" "$BASE/sessions/$SID/fields" | jq '.fields[0]' +``` +```json +{ "name": "applicant.name", "type": "text", "page": 1, "options": [], "required": true, + "readonly": false, "maxlen": 40, "editable": false, "multiSelect": false } +``` + +`type` is one of `text`, `checkbox`, `radio`, `combobox`, `listbox`, `button` or `signature`. A PDF with +no form gives `{"fields": []}`. + +**Get the tagged PDF:** + +```bash +curl -s -X POST -H "$AUTH" "$BASE/sessions/$SID/pdf" \ + -H 'content-type: application/json' \ + -d '{"values":{"applicant.name":"Jane Doe","applicant.consent":true}}' > tagged.json +jq -r .pdf tagged.json | base64 --decode > tagged.pdf +``` + +`values` is optional. Its keys are field names. A text field takes a string, a checkbox takes `true` +or `false`, and a multi-select list takes an array. Fields you leave out keep what the PDF had. +**The values are used for this one request and are never stored or logged.** The run log records +only which field names were given. + +The answer is `{"filename", "pdf", "report"}`: the file name to save as, the PDF in base64, and the +tagger's report. `report.warnings` lists what the tagger could not do cleanly, such as a form field +it could not find in the HTML. The demo page turns each warning into a plain sentence. + +The session must have finished (`ready_for_review` or `closed`), or you get `409 invalid_state`. The +PDF is tagged from the first pass's HTML for each page, before review. Errors from the tagger keep +its code: `400` for a bad value, `422` for a PDF it refuses (for example `encrypted`), and `504 +timeout` after `tagged_pdf.timeout_seconds` (300 by default). Two tagging requests at once is the +limit, so a third gets `503 busy` with `Retry-After`. A deployment without the feature answers +`404 tagged_pdf_unavailable`. + ## Submit feedback (re-run) Triggers a new run within the same session, with the feedback injected as a top-level @@ -1000,8 +1049,8 @@ The events worth grepping for have a section each below, and the index is a link the index when you have a `type` off a log line and want to know what it means; read a section when you want to know what the field it names is for and what it costs. -**The index is the whole log.** `src/` emits **118** event types and every one of them has a section -below — **113** sections, because a few cover a pair of events that are only read together. So a +**The index is the whole log.** `src/` emits **120** event types and every one of them has a section +below — **114** sections, because a few cover a pair of events that are only read together. So a `type` you cannot find here is not one the index skipped: it is a misread line, or a name `src/` no longer emits. @@ -1125,6 +1174,7 @@ emits fails it too. | [`agent_issue_failed`](#agent_issue_failed) | A suggestion did not get filed, and which of the two calls failed | | [`contribution_failed`](#contribution_failed) | The filing step threw, **after** `run_complete` | | [`run_failed`](#run_failed) | The run threw, so there is **no document** | +| [`tagged_pdf` / `tagged_pdf_failed`](#tagged_pdf--tagged_pdf_failed) | A tagged PDF was made, or could not be | | [`calibrate_call_failed`](#calibrate_call_failed) | One calibration verifier call threw — a tool's line, never a run's | ### `run_queued` / `run_dequeued` @@ -4278,6 +4328,13 @@ which failure it was, and this `error` is written to match: with `blank: 0` it i provider error, standing for all of them because that is the diagnosis; otherwise it counts the source pages reported blank, which an empty document could not. +### `tagged_pdf` / `tagged_pdf_failed` + +A [tagged PDF](#get-a-tagged-pdf-optional) was made, or the tagger refused. `ms` is how long it took. +`tagged_pdf` lists `fields_given`, the names of the fields that were filled in. The values are +never logged. `tagged_pdf_failed` has the tagger's `code` and `error`. An `internal_error`'s +message is left out, because it may quote a value. + ### `calibrate_call_failed` One verifier call in the calibration harness threw: `image` is the page, `defect` the seeded defect the diff --git a/public/demo.html b/public/demo.html index 339d143..c8e89b8 100644 --- a/public/demo.html +++ b/public/demo.html @@ -49,6 +49,15 @@ border: 1px solid var(--border); border-radius: 8px; background: var(--bg); color: var(--fg); } textarea { min-height: 5rem; } + select { + font: inherit; max-width: 32rem; padding: .5rem .65rem; + border: 1px solid var(--border); border-radius: 8px; background: var(--bg); color: var(--fg); + } + h3 { font-size: 1.1rem; margin: 1.5rem 0 .5rem; } + fieldset { border: 1px solid var(--border); border-radius: 8px; margin: 1rem 0; padding: .25rem 1rem 1rem; } + legend { font-weight: 600; padding: 0 .35rem; } + .choice { display: flex; gap: .5rem; align-items: center; margin: .5rem 0; } + .choice label { display: inline; margin: 0; font-weight: 400; } button { font: inherit; font-weight: 600; cursor: pointer; border: 1px solid transparent; border-radius: 8px; padding: .6rem 1.1rem; background: var(--accent); color: var(--accent-fg); @@ -157,6 +166,25 @@

Your accessible HTML

+ + +

Iris will re-run the conversion with your feedback applied (for example: “Keep footnotes separate from body text”).

@@ -195,6 +223,8 @@ // that enforces, and a page that guessed would either reject a fine file or // promise one that fails. let uploadLimits = null; + // Whether this deployment makes tagged PDFs (GET /v1/limits). False until it says so. + let taggedPdfOn = false; const $ = (id) => document.getElementById(id); const show = (id) => { $(id).hidden = false; }; @@ -396,9 +426,165 @@ // is built into the session and used, so there is no "taught itself" to // report to the person converting a document. The run log's `agent_issue` // events are where that information actually lives if it is wanted back. + // Best-effort: the HTML is the result, and the PDF offer is extra. + try { await showPdfPart(); } catch { hide('pdf-part'); } hide('status-section'); show('result-section'); focusHeading('result-h'); } + // ----- Tagged PDF (optional) ----- + // One control per settable field of the source PDF, labelled with the field's name. + // `doc` is passed in so the tests can build the form in their own document. + // Returns how many fields got a control. Read-only fields, buttons and signatures + // cannot be set, so they get none. + function renderFields(doc, list, fields) { + const el = (tag, props = {}) => Object.assign(doc.createElement(tag), props); + const req = (f) => (f.required ? ' (required on the form)' : ''); + let n = 0; + for (const f of fields) { + if (f.readonly) continue; + const id = 'pdf-f' + n; + let box; + if (f.type === 'text' || (f.type === 'combobox' && f.editable)) { + box = el('div'); + box.append(el('label', { htmlFor: id, textContent: f.name + req(f) })); + const input = el('input', { id, type: 'text' }); + if (f.maxlen) input.maxLength = f.maxlen; + if (f.options.length) { + const dl = el('datalist', { id: id + '-list' }); + for (const o of f.options) dl.append(el('option', { value: o })); + box.append(dl); + input.setAttribute('list', dl.id); + } + box.append(input); + } else if (f.type === 'checkbox') { + box = el('div', { className: 'choice' }); + box.append(el('input', { id, type: 'checkbox' }), el('label', { htmlFor: id, textContent: f.name + req(f) })); + } else if (f.type === 'radio') { + box = el('fieldset'); + box.append(el('legend', { textContent: f.name + req(f) })); + // A chosen radio cannot be unchosen, so "leave it" is a choice of its own. + ['', ...f.options].forEach((o, i) => { + const row = el('div', { className: 'choice' }); + row.append( + el('input', { id: id + '-' + i, type: 'radio', name: id, value: o, checked: i === 0 }), + el('label', { htmlFor: id + '-' + i, textContent: o || 'Leave as it is' }), + ); + box.append(row); + }); + } else if (f.type === 'combobox' || f.type === 'listbox') { + box = el('div'); + box.append(el('label', { htmlFor: id, textContent: f.name + req(f) })); + const select = el('select', { id, multiple: f.multiSelect }); + if (!f.multiSelect) select.append(el('option', { value: '', textContent: 'Leave as it is' })); + for (const o of f.options) select.append(el('option', { value: o, textContent: o })); + box.append(select); + } else { + continue; + } + box.dataset.field = f.name; + box.dataset.type = f.type; + list.append(box); + n++; + } + return n; + } + + // What was entered, as the API takes it. A blank field is left out, so the PDF keeps + // what it had. + function fieldValues(list) { + const values = {}; + for (const box of list.querySelectorAll('[data-field]')) { + const name = box.dataset.field; + if (box.dataset.type === 'checkbox') { + if (box.querySelector('input').checked) values[name] = true; + } else if (box.dataset.type === 'radio') { + const picked = box.querySelector('input:checked'); + if (picked && picked.value) values[name] = picked.value; + } else if (box.querySelector('select[multiple]')) { + const picked = [...box.querySelector('select').selectedOptions].map((o) => o.value); + if (picked.length) values[name] = picked; + } else { + const v = box.querySelector('input, select').value; + if (v) values[name] = v; + } + } + return values; + } + + // The tagger's report, as sentences a person can act on. One per kind of warning. + function pdfNotes(report) { + const count = {}; + for (const w of (report && report.warnings) || []) count[w.code] = (count[w.code] || 0) + 1; + const s = (n, one, many) => (n === 1 ? one : many.replace('#', n)); + const say = { + duplicate_text_layer: (n) => s(n, 'One page now has its text twice: screen readers read it once, but copied text may come out doubled.', '# pages now have their text twice: screen readers read it once, but copied text may come out doubled.'), + unmatched_text: () => 'Some text on the page was not in the HTML. It was kept, as plain paragraphs.', + missing_alt: (n) => s(n, 'One image has no description.', '# images have no description.'), + field_not_in_html: (n) => s(n, 'One form field was not found in the HTML, so a screen reader reaches it at the end of its page.', '# form fields were not found in the HTML, so a screen reader reaches them at the end of their page.'), + unmatched_link: (n) => s(n, 'One link was not found in the HTML, so a screen reader reaches it at the end of its page.', '# links were not found in the HTML, so a screen reader reaches them at the end of their page.'), + page_not_tagged: (n) => s(n, 'One page was left untagged.', '# pages were left untagged.'), + page_not_in_html: (n) => s(n, 'One page had no HTML and was left untagged.', '# pages had no HTML and were left untagged.'), + alignment_incomplete: () => 'On some pages the HTML and the page differed too much to match every word. The rest was kept as plain paragraphs.', + font_not_embedded: () => 'The PDF uses fonts it does not include, so it cannot claim to meet PDF/UA, the PDF accessibility standard.', + source_marked_content: () => 'The PDF has tags left over from another tool, so it cannot claim to meet PDF/UA, the PDF accessibility standard.', + repaired: () => 'The PDF was damaged. The tagged file is a repaired copy.', + link_not_on_page: (n) => s(n, 'One link in the HTML is not on the page, so it was left out.', '# links in the HTML are not on the page, so they were left out.'), + missing_glyph: () => 'Some characters could not be written in the PDF\'s fonts, so they may read wrongly.', + unmapped_element: () => 'Some HTML had no PDF tag to match, so it was tagged as a paragraph.', + field_without_name: () => 'A form control in the HTML had no name, so it was not matched to the PDF\'s form.', + no_text_positions: (n) => s(n, 'One page has no text the tagger could find, so it was left untagged.', '# pages have no text the tagger could find, so they were left untagged.'), + xfa_removed: () => 'The PDF had two versions of its form. The one most readers cannot use was removed.', + signature_invalidated: () => 'The PDF was signed. The signature no longer holds, because the file changed.', + signature_field_skipped: () => 'Signature fields cannot be filled in here, so they were left empty.', + }; + return Object.keys(count).filter((c) => c !== 'no_title').map((c) => (say[c] ? say[c](count[c]) : 'Note from the tagger: ' + c + '.')); + } + + async function showPdfPart() { + hide('pdf-part'); + if (!taggedPdfOn) return; + const res = await api('/sessions/' + sessionId + '/fields'); + if (!res.ok) return; // not one uploaded PDF, or the PDF cannot be read: no offer + const { fields } = await res.json(); + $('pdf-field-list').replaceChildren(); + $('pdf-notes').replaceChildren(); + $('pdf-fields').hidden = renderFields(document, $('pdf-field-list'), fields || []) === 0; + show('pdf-part'); + } + + $('pdf-form').addEventListener('submit', async (ev) => { + ev.preventDefault(); + setError(''); + $('pdf-btn').disabled = true; + live('Tagging your PDF. This can take a minute.'); + try { + const res = await api('/sessions/' + sessionId + '/pdf', { + method: 'POST', headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ values: fieldValues($('pdf-field-list')) }), + }); + if (!res.ok) throw new Error(await errMessage(res)); + const out = await res.json(); + const bytes = Uint8Array.from(atob(out.pdf), (c) => c.charCodeAt(0)); + const url = URL.createObjectURL(new Blob([bytes], { type: 'application/pdf' })); + const a = document.createElement('a'); + a.href = url; a.download = out.filename; + document.body.appendChild(a); a.click(); a.remove(); + setTimeout(() => URL.revokeObjectURL(url), 60000); + const notes = pdfNotes(out.report); + const list = document.createElement('ul'); + for (const n of notes) list.append(Object.assign(document.createElement('li'), { textContent: n })); + const head = Object.assign(document.createElement('p'), { + textContent: 'Downloaded ' + out.filename + '.' + (notes.length ? ' Notes on it:' : ''), + }); + $('pdf-notes').replaceChildren(head, ...(notes.length ? [list] : [])); + live('Downloaded ' + out.filename + '.' + (notes.length === 1 ? ' One note is below the button.' : notes.length ? ' ' + notes.length + ' notes are below the button.' : '')); + } catch (e) { + setError('Could not tag the PDF: ' + e.message); + } finally { + $('pdf-btn').disabled = false; + } + }); + $('restart-btn').addEventListener('click', () => { sessionId = null; setError(''); $('upload-form').reset(); @@ -548,6 +734,7 @@ const res = await fetch(API + '/limits'); if (!res.ok) return; const l = await res.json(); + taggedPdfOn = Boolean(l && l.tagged_pdf === true); const bytes = Number(l && l.image && l.image.max_bytes); if (!Number.isFinite(bytes) || bytes < 1) return; uploadLimits = { maxBytes: bytes, hint: String(l.image.hint || '') }; diff --git a/src/config.ts b/src/config.ts index 086539e..9b6f1fc 100644 --- a/src/config.ts +++ b/src/config.ts @@ -140,6 +140,9 @@ export interface IrisConfig { rate_limits?: Partial; }; storage: { data_dir: string; agents_dir: string; database: string }; + // Optional. The `iris-pdf` command that tags an uploaded PDF (util/taggedPdf.ts). + // Blank or absent means the feature is off. + tagged_pdf?: { command?: string; timeout_seconds?: number }; github: { upstream_repo: string; // Overridable for GitHub Enterprise (and for testing). Defaults below. diff --git a/src/index.ts b/src/index.ts index d3e22f4..2327ad5 100644 --- a/src/index.ts +++ b/src/index.ts @@ -18,6 +18,7 @@ import { meRouter } from "./routes/me.ts"; import { sessionsRouter } from "./routes/sessions.ts"; import { statsRouter } from "./routes/stats.ts"; import { limitsRouter } from "./routes/limits.ts"; +import { taggedPdfStatus } from "./util/taggedPdf.ts"; import { qualityRouter } from "./routes/quality.ts"; import { visionModelWarning } from "./providers/imageLimits.ts"; import { generalRateLimit } from "./util/requestLimits.ts"; @@ -245,6 +246,9 @@ const openStorage = (): { store: Store; stale: number } => { // produces it is the guarded first write above. const { store, stale } = openStorage(); if (stale > 0) console.log(`Marked ${stale} interrupted session(s) as failed on startup.`); +// Optional. Silent when tagged_pdf.command is blank; says so when it is set and does not run. +const pdfStatus = taggedPdfStatus(cfg); +if (pdfStatus) (pdfStatus.startsWith("WARNING") ? console.warn : console.log)(pdfStatus); const app = express(); // Whose address `req.ip` is. Off unless a deployment says how many proxies are in front // of it, because the rate limits below are only per-caller if this is right: unset behind diff --git a/src/routes/limits.ts b/src/routes/limits.ts index 3554ef2..595f565 100644 --- a/src/routes/limits.ts +++ b/src/routes/limits.ts @@ -2,6 +2,7 @@ import { Router } from "express"; import type { IrisConfig } from "../config.ts"; import { imageLimitsHint, resolveImageLimits } from "../providers/imageLimits.ts"; import { MAX_PDF_PAGES } from "../util/pdf.ts"; +import { taggedPdfCommand } from "../util/taggedPdf.ts"; import { MAX_UPLOAD_FILES, publishedRateLimits, uploadCeilingBytes } from "../util/requestLimits.ts"; /** @@ -23,7 +24,8 @@ import { MAX_UPLOAD_FILES, publishedRateLimits, uploadCeilingBytes } from "../ut * "pdf": { "max_pages": 25 }, * "upload": { "max_files": 25, "max_request_bytes": 134217728 }, * "rate_limits": { "general_per_minute": 240, "upload_per_minute": 12, - * "max_upload_memory_mb": 256, "window_seconds": 60 } } + * "max_upload_memory_mb": 256, "window_seconds": 60 }, + * "tagged_pdf": false } * ``` * * Not handed the auth middleware, like `GET /v1/stats`, so it answers even where the @@ -64,6 +66,9 @@ export function limitsRouter(cfg: IrisConfig): Router { // volume in the app. Same rationale as the upload limits: a budget a client can read // is one it can pace itself against, instead of discovering it by being refused. rate_limits: publishedRateLimits(cfg), + // Whether POST /v1/sessions/{id}/pdf can answer here (util/taggedPdf.ts), so a client + // offers it only where it works. + tagged_pdf: taggedPdfCommand(cfg) !== null, }; r.get("/", (_req, res) => { diff --git a/src/routes/sessions.ts b/src/routes/sessions.ts index cbf5468..4bf57c6 100644 --- a/src/routes/sessions.ts +++ b/src/routes/sessions.ts @@ -32,6 +32,7 @@ import { resolveImageLimits, } from "../providers/imageLimits.ts"; import { imageDimensions } from "../util/imageSize.ts"; +import { readFields, tagPdf, taggedPdfCommand, tagTimeoutSeconds, TaggedPdfError } from "../util/taggedPdf.ts"; // 50 MB is a memory bound, not the image limit. It stays well above what an image may // be (see imageLimits.ts) because a PDF legitimately is: 25 pages of scans is a large @@ -134,6 +135,19 @@ function ownedSession(store: Store, id: string, userId: number): SessionRecord | return s; } +// Keep the upload for a tagged PDF later, when this deployment makes them and the upload +// was one PDF on its own. Then the PDF's page N is the session's page N. +export function keepSourcePdf( + cfg: IrisConfig, + paths: Paths, + sessionId: string, + files: { originalname: string; buffer: Buffer }[], +): boolean { + if (!taggedPdfCommand(cfg) || files.length !== 1 || !PDF_EXT.test(files[0].originalname)) return false; + writeFileSync(paths.sessionSourcePdf(sessionId), files[0].buffer); + return true; +} + export function sessionsRouter(cfg: IrisConfig, store: Store): Router { const r = Router(); const paths = new Paths(cfg); @@ -383,6 +397,7 @@ export function sessionsRouter(cfg: IrisConfig, store: Store): Router { if (Object.keys(linksByOrder).length) { writeFileSync(paths.sessionLinks(sessionId), JSON.stringify(linksByOrder, null, 2)); } + keepSourcePdf(cfg, paths, sessionId, files); const record = store.createSession({ session_id: sessionId, @@ -456,6 +471,114 @@ export function sessionsRouter(cfg: IrisConfig, store: Store): Router { res.type("text/html").send(html); }); + // ----- Tagged PDF (optional; util/taggedPdf.ts) ----- + // The session's source PDF, or the reason it has none. Sends that error itself. + function sourcePdf(req: AuthedRequest, res: Response): { s: SessionRecord; command: string; pdf: string } | null { + const s = ownedSession(store, req.params.id, req.user!.github_user_id); + if (!s) { + sendError(res, 404, "session_not_found", "No such session"); + return null; + } + const command = taggedPdfCommand(cfg); + if (!command) { + sendError(res, 404, "tagged_pdf_unavailable", "This deployment does not make tagged PDFs."); + return null; + } + const pdf = paths.sessionSourcePdf(s.session_id); + if (!existsSync(pdf)) { + sendError(res, 409, "no_source_pdf", "Only a session made from one uploaded PDF can be tagged."); + return null; + } + return { s, command, pdf }; + } + // Express 4 does not catch a rejected async handler, so every error is answered here. + function taggerError(res: Response, e: unknown): void { + if (!(e instanceof TaggedPdfError)) { + sendError(res, 500, "tagger_failed", "Could not tag the PDF."); + return; + } + const status = e.code === "timeout" ? 504 : e.exit === 3 ? 400 : 422; + sendError(res, status, e.code, e.message); + } + + // GET /v1/sessions/{id}/fields — the source PDF's form fields, for a fill-in form. + r.get("/:id/fields", async (req: AuthedRequest, res) => { + const src = sourcePdf(req, res); + if (!src) return; + try { + res.json({ fields: await readFields(src.command, src.pdf) }); + } catch (e) { + taggerError(res, e); + } + }); + + // POST /v1/sessions/{id}/pdf — the source PDF, tagged from the extracted pages, with + // `values` filled in. The PDF is the answer and nothing is kept, so neither the values + // nor the filled PDF is ever stored. + let tagging = 0; + r.post("/:id/pdf", async (req: AuthedRequest, res) => { + const src = sourcePdf(req, res); + if (!src) return; + const { s } = src; + const finalPath = paths.sessionFinalFragments(s.session_id); + if ((s.status !== "ready_for_review" && s.status !== "closed") || !existsSync(finalPath)) { + sendError(res, 409, "invalid_state", "The PDF can be tagged once the session is ready_for_review."); + return; + } + const values: unknown = (req.body ?? {}).values ?? {}; + if (typeof values !== "object" || values === null || Array.isArray(values)) { + sendError(res, 400, "invalid_request", "`values` must be an object of field name to value."); + return; + } + // Each run is seconds of CPU in a child process. Two at once is enough. + if (tagging >= 2) { + res.setHeader("Retry-After", "10"); + sendError(res, 503, "busy", "Other PDFs are being tagged. Try again in a few seconds."); + return; + } + const log = new RunLog(paths.sessionLog(s.session_id)); + const started = Date.now(); + tagging++; + try { + const base = existsSync(paths.sessionSourceName(s.session_id)) + ? readFileSync(paths.sessionSourceName(s.session_id), "utf8").trim() || "document" + : "document"; + // The pages as extracted, by page order. Order is the PDF's page number, because + // the session is one PDF. + const { fragments = [] } = JSON.parse(readFileSync(finalPath, "utf8")) as { fragments?: Fragment[] }; + const outPath = paths.sessionOutput(s.session_id); + const output = existsSync(outPath) ? readFileSync(outPath, "utf8") : ""; + const input = { + lang: output.match(/]*\blang="([^"]+)"/i)?.[1], + title: base, + pages: [...fragments].sort((a, b) => a.order - b.order).map((f) => ({ sourcePage: f.order, html: f.innerHtml })), + }; + const { pdf, report } = await tagPdf(src.command, { + pdfPath: src.pdf, + input, + values: values as Record, + scratchRoot: paths.tmpDir(s.session_id), + timeoutSeconds: tagTimeoutSeconds(cfg), + }); + // Field names only. A value is never logged. + log.event("tagged_pdf", { ms: Date.now() - started, fields_given: Object.keys(values) }); + res.json({ filename: `${base}_tagged.pdf`, pdf: pdf.toString("base64"), report }); + } catch (e) { + // iris-pdf's own messages name fields and options, never values. Its + // `internal_error` passes on any exception's text, which could hold one, so that + // message is not logged. + const code = e instanceof TaggedPdfError ? e.code : "tagger_failed"; + log.event("tagged_pdf_failed", { + ms: Date.now() - started, + code, + error: code === "internal_error" ? "(not logged: it may quote a value)" : (e as Error).message, + }); + taggerError(res, e); + } finally { + tagging--; + } + }); + // GET /v1/sessions/{id}/logs — the run log as ndjson. r.get("/:id/logs", (req: AuthedRequest, res) => { const s = ownedSession(store, req.params.id, req.user!.github_user_id); diff --git a/src/store/paths.ts b/src/store/paths.ts index 1a8352c..cfb4c47 100644 --- a/src/store/paths.ts +++ b/src/store/paths.ts @@ -10,7 +10,8 @@ import type { IrisConfig } from "../config.ts"; // final.json), history/ (the PRIOR output.html, snapshotted only when a // feedback re-run is about to overwrite it — not the review loop's rounds), // output.html, log.jsonl (the run log), lint.json, unresolved.md, -// agent-updates.md, links.json, source-name.txt +// agent-updates.md, links.json, source-name.txt, source.pdf (only with +// tagged PDFs on) // fixtures// and memory/.json — keyed by agent, shared by every session // tmp// one run's scratch. `tmp//agents/` holds agents that session BUILT, // which `loadAgent` prefers over the library for the rest of it. @@ -65,6 +66,11 @@ export class Paths { sessionLinks(id: string): string { return join(this.sessionDir(id), "links.json"); } + // The uploaded PDF, kept only when tagged PDFs are on and the upload was one PDF. + // It is what `iris-pdf` tags (util/taggedPdf.ts). + sessionSourcePdf(id: string): string { + return join(this.sessionDir(id), "source.pdf"); + } // `sessionNewAgents()` and `sessionPrs()` used to sit here with zero callers, // left over from an earlier fork-and-PR design. That flow has been dropped // (contributions are issues, filed under the deployment's own GitHub account), so diff --git a/src/util/taggedPdf.ts b/src/util/taggedPdf.ts new file mode 100644 index 0000000..b641e4b --- /dev/null +++ b/src/util/taggedPdf.ts @@ -0,0 +1,127 @@ +// The optional tagged-PDF output: the uploaded PDF, tagged from Iris's HTML and with any +// form values filled in. The `iris-pdf` command from equalify-iris-pdf makes it +// (https://github.com/EqualifyEverything/equalify-iris-pdf). Iris has no PDF library of +// its own. It runs the command the way it runs poppler, and a deployment without the +// command works exactly as before. +import { execFile, spawnSync } from "node:child_process"; +import { mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import type { IrisConfig } from "../config.ts"; + +const execFileP = promisify(execFile); + +export const DEFAULT_TAG_TIMEOUT_SECONDS = 300; +// Room for a 25-page document's PDF. +const MAX_BUFFER = 64 * 1024 * 1024; + +// The first line `iris-pdf fields --help` prints, per command, or null when it did not run. +// Probed once per process: config does not change without a restart. +const probed = new Map(); + +function probe(command: string): string | null { + if (!probed.has(command)) { + const r = spawnSync(command, ["fields", "--help"], { encoding: "utf8", timeout: 30_000 }); + const first = (r.stdout ?? "").split("\n")[0].trim(); + probed.set(command, r.status === 0 && first.startsWith("iris-pdf ") ? first : null); + } + return probed.get(command)!; +} + +// The command to run, or null when the feature is off: `tagged_pdf.command` is blank, or +// it is set and does not run. +export function taggedPdfCommand(cfg: IrisConfig): string | null { + const command = cfg.tagged_pdf?.command?.trim(); + return command && probe(command) ? command : null; +} + +// For the startup log: what is on, or why it is off. +export function taggedPdfStatus(cfg: IrisConfig): string | null { + const command = cfg.tagged_pdf?.command?.trim(); + if (!command) return null; + const version = probe(command); + return version + ? `Tagged PDFs on (${version}).` + : `WARNING: tagged_pdf.command is "${command}", but "${command} fields --help" did not run. Tagged PDFs are off.`; +} + +export function tagTimeoutSeconds(cfg: IrisConfig): number { + const t = Number(cfg.tagged_pdf?.timeout_seconds); + return Number.isFinite(t) && t > 0 ? t : DEFAULT_TAG_TIMEOUT_SECONDS; +} + +// One form field, as `iris-pdf fields --json` prints it. Passed through unchanged. +export type PdfField = { + name: string; + type: string; + page: number; + options: string[]; + required: boolean; + readonly: boolean; + maxlen: number | null; + editable: boolean; + multiSelect: boolean; +}; + +// The command refused or failed. `code` is its own (`iris-pdf: : ` on +// stderr), or `timeout`, or `tagger_failed` when it printed no such line. +export class TaggedPdfError extends Error { + code: string; + exit: number | null; + constructor(code: string, message: string, exit: number | null) { + super(message); + this.code = code; + this.exit = exit; + } +} + +function failure(e: unknown): TaggedPdfError { + const err = e as { code?: unknown; killed?: boolean; stderr?: string; message?: string }; + if (err.killed) return new TaggedPdfError("timeout", "Tagging the PDF took too long.", null); + const exit = typeof err.code === "number" ? err.code : null; + const line = (err.stderr ?? "").split("\n").find((l) => l.startsWith("iris-pdf: ")); + const m = line?.match(/^iris-pdf: ([a-z_]+): (.*)$/); + if (m) return new TaggedPdfError(m[1], m[2], exit); + return new TaggedPdfError("tagger_failed", "iris-pdf failed without saying why.", exit); +} + +export async function readFields(command: string, pdfPath: string): Promise { + let stdout: string; + try { + ({ stdout } = await execFileP(command, ["fields", "--pdf", pdfPath, "--json"], { maxBuffer: MAX_BUFFER, timeout: 60_000 })); + } catch (e) { + throw failure(e); + } + try { + return JSON.parse(stdout) as PdfField[]; + } catch { + throw new TaggedPdfError("tagger_failed", "iris-pdf printed fields that are not JSON.", 0); + } +} + +export type TagInput = { lang?: string; title?: string; pages: { sourcePage: number; html: string }[] }; + +// Tag `pdfPath`. The values are personal data. They go to the child in a file only it and +// this process can read, in a scratch directory deleted before this returns. Iris keeps +// neither them nor the tagged PDF. +export async function tagPdf( + command: string, + args: { pdfPath: string; input: TagInput; values: Record; scratchRoot: string; timeoutSeconds: number }, +): Promise<{ pdf: Buffer; report: unknown }> { + mkdirSync(args.scratchRoot, { recursive: true }); + const dir = mkdtempSync(join(args.scratchRoot, "pdf-")); + const f = (name: string) => join(dir, name); + try { + writeFileSync(f("pages.json"), JSON.stringify(args.input)); + writeFileSync(f("values.json"), JSON.stringify(args.values), { mode: 0o600 }); + const argv = ["tag", "--pdf", args.pdfPath, "--pages", f("pages.json"), "--values", f("values.json"), "--out", f("out.pdf"), "--report", f("report.json")]; + try { + await execFileP(command, argv, { maxBuffer: MAX_BUFFER, timeout: args.timeoutSeconds * 1000 }); + } catch (e) { + throw failure(e); + } + return { pdf: readFileSync(f("out.pdf")), report: JSON.parse(readFileSync(f("report.json"), "utf8")) }; + } finally { + rmSync(dir, { recursive: true, force: true }); + } +} diff --git a/test/demo-pdf-form.test.ts b/test/demo-pdf-form.test.ts new file mode 100644 index 0000000..4fc0e38 --- /dev/null +++ b/test/demo-pdf-form.test.ts @@ -0,0 +1,99 @@ +// The demo's "Get your PDF back, tagged" form. It is built from the source PDF's own +// fields, so these tests build it from each kind of field, lint it with the same axe +// configuration Iris lints its output with, and check what it sends. +// +// The functions are lifted out of the page's inline script, as test/demo-tally.test.ts +// does, so a change to the page is what gets tested. +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { dirname, join } from "node:path"; +import { JSDOM } from "jsdom"; +import { runAxe } from "../src/pipeline/lint.ts"; + +const repoRoot = join(dirname(fileURLToPath(import.meta.url)), ".."); +const demoHtml = readFileSync(join(repoRoot, "public", "demo.html"), "utf8"); + +function extract(name: string): string { + const start = demoHtml.indexOf(`function ${name}(`); + assert.notEqual(start, -1, `${name} is no longer in public/demo.html`); + let depth = 0; + for (let i = demoHtml.indexOf("{", start); i < demoHtml.length; i++) { + if (demoHtml[i] === "{") depth++; + else if (demoHtml[i] === "}" && --depth === 0) return demoHtml.slice(start, i + 1); + } + throw new Error(`unbalanced braces reading ${name} from public/demo.html`); +} + +type Field = { name: string; type: string; options: string[]; required?: boolean; readonly?: boolean; maxlen?: number | null; editable?: boolean; multiSelect?: boolean }; +const renderFields = new Function(`${extract("renderFields")}; return renderFields;`)() as (doc: Document, list: Element, fields: Field[]) => number; +const fieldValues = new Function(`${extract("fieldValues")}; return fieldValues;`)() as (list: Element) => Record; +const pdfNotes = new Function(`${extract("pdfNotes")}; return pdfNotes;`)() as (report: unknown) => string[]; + +const FIELDS: Field[] = [ + { name: "name", type: "text", options: [], required: true, maxlen: 40 }, + { name: "city", type: "combobox", options: ["Chicago", "Urbana"], editable: true }, + { name: "agree", type: "checkbox", options: ["Yes"] }, + { name: "size", type: "radio", options: ["S", "M"] }, + { name: "state", type: "combobox", options: ["IL", "IN"] }, + { name: "days", type: "listbox", options: ["Mon", "Tue"], multiSelect: true }, + { name: "office", type: "text", options: [], readonly: true }, + { name: "submit", type: "button", options: [] }, + { name: "sig", type: "signature", options: [] }, +]; + +function form() { + const doc = new JSDOM('t
').window.document; + const list = doc.getElementById("f")!; + return { doc, list, n: renderFields(doc, list, FIELDS) }; +} + +test("every settable field gets one labelled control, and the rest get none", async () => { + const { doc, list, n } = form(); + assert.equal(n, 6); + assert.deepEqual([...list.querySelectorAll("[data-field]")].map((b) => (b as HTMLElement).dataset.field), ["name", "city", "agree", "size", "state", "days"]); + assert.equal(list.querySelector("input")!.maxLength, 40); + assert.match(list.textContent!, /name \(required on the form\)/); + // An editable combobox takes any text, and offers its options as suggestions. + const city = list.querySelector('[data-field="city"] input')!; + assert.equal(doc.getElementById(city.getAttribute("list")!)!.children.length, 2); + const lint = await runAxe(doc.documentElement.outerHTML); + assert.equal(lint.error, undefined, `axe-core did not run: ${lint.error}`); + assert.deepEqual(lint.violations, []); +}); + +test("an untouched form sends nothing, so the PDF keeps what it had", () => { + assert.deepEqual(fieldValues(form().list), {}); +}); + +test("what was entered is sent under each field's name", () => { + const { list } = form(); + const q = (s: string) => list.querySelector(s) as HTMLInputElement & HTMLSelectElement; + q('[data-field="name"] input').value = "Ada"; + q('[data-field="city"] input').value = "Peoria"; + q('[data-field="agree"] input').checked = true; + q('[data-field="size"] input[value="M"]').checked = true; + q('[data-field="state"] select').value = "IN"; + (q('[data-field="days"] select').options[1] as HTMLOptionElement).selected = true; + assert.deepEqual(fieldValues(list), { name: "Ada", city: "Peoria", agree: true, size: "M", state: "IN", days: ["Tue"] }); +}); + +test("the report's warnings come back as one sentence per kind", () => { + const notes = pdfNotes({ + warnings: [ + { code: "missing_alt", page: 1 }, + { code: "missing_alt", page: 2 }, + { code: "field_not_in_html", page: 1 }, + { code: "no_title" }, + { code: "something_new" }, + ], + }); + assert.deepEqual(notes, [ + "2 images have no description.", + "One form field was not found in the HTML, so a screen reader reaches it at the end of its page.", + "Note from the tagger: something_new.", + ]); + assert.deepEqual(pdfNotes({ warnings: [] }), []); + assert.deepEqual(pdfNotes(null), []); +}); diff --git a/test/fixtures/fake-iris-pdf.mjs b/test/fixtures/fake-iris-pdf.mjs new file mode 100755 index 0000000..e34d6f2 --- /dev/null +++ b/test/fixtures/fake-iris-pdf.mjs @@ -0,0 +1,41 @@ +#!/usr/bin/env node +// Stands in for the `iris-pdf` command in test/tagged-pdf.test.ts. Same arguments, exit +// codes and one-line errors, and no PDF work. Its "PDF" is JSON of what it was given. +import { readFileSync, statSync, writeFileSync } from "node:fs"; +import { parseArgs } from "node:util"; + +const [command, ...rest] = process.argv.slice(2); +const { values: a } = parseArgs({ + args: rest, + strict: false, + options: { pdf: { type: "string" }, pages: { type: "string" }, values: { type: "string" }, out: { type: "string" }, report: { type: "string" }, json: { type: "boolean" }, help: { type: "boolean" } }, +}); +const fail = (code, message, exit) => { + process.stderr.write(`iris-pdf: ${code}: ${message}\n`); + process.exit(exit); +}; + +if (a.help) { + console.log("iris-pdf 0.0.0-fake"); + process.exit(0); +} +const pdf = readFileSync(a.pdf, "utf8"); +if (pdf.includes("ENCRYPTED")) fail("encrypted", "The PDF is encrypted.", 1); + +if (command === "fields") { + console.log(JSON.stringify([ + { name: "applicant.name", type: "text", page: 1, options: [], required: true, readonly: false, maxlen: 40, editable: false, multiSelect: false }, + { name: "applicant.consent", type: "checkbox", page: 1, options: ["Yes"], required: false, readonly: false, maxlen: null, editable: false, multiSelect: false }, + ])); +} else if (command === "tag") { + const values = JSON.parse(readFileSync(a.values, "utf8")); + if ("unknown.field" in values) fail("bad_value", "No field is named unknown.field.", 3); + if ("crash" in values) fail("internal_error", `Cannot set ${values.crash}.`, 3); + if ("slow" in values) await new Promise((r) => setTimeout(r, 5000)); + const input = JSON.parse(readFileSync(a.pages, "utf8")); + const mode = (statSync(a.values).mode & 0o777).toString(8); + writeFileSync(a.out, "%PDF-fake " + JSON.stringify({ source: pdf, input, values, mode, scratch: a.values })); + writeFileSync(a.report, JSON.stringify({ tool: "iris-pdf 0.0.0-fake", warnings: [{ code: "duplicate_text_layer", page: 1 }] })); +} else { + fail("bad_arguments", `Unknown command ${command}.`, 3); +} diff --git a/test/tagged-pdf.test.ts b/test/tagged-pdf.test.ts new file mode 100644 index 0000000..431b289 --- /dev/null +++ b/test/tagged-pdf.test.ts @@ -0,0 +1,236 @@ +// The optional tagged-PDF output (src/util/taggedPdf.ts). `iris-pdf` is a separate +// project, so these tests run a stand-in for it (test/fixtures/fake-iris-pdf.mjs). +import { test } from "node:test"; +import assert from "node:assert/strict"; +import express from "express"; +import type { AddressInfo } from "node:net"; +import { existsSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; +import type { IrisConfig } from "../src/config.ts"; +import type { AuthedRequest } from "../src/auth/middleware.ts"; +import { keepSourcePdf, sessionsRouter } from "../src/routes/sessions.ts"; +import { limitsRouter } from "../src/routes/limits.ts"; +import { Store } from "../src/store/db.ts"; +import { Paths } from "../src/store/paths.ts"; +import { taggedPdfCommand, taggedPdfStatus } from "../src/util/taggedPdf.ts"; + +const FAKE = join(dirname(fileURLToPath(import.meta.url)), "fixtures", "fake-iris-pdf.mjs"); +const USER = 7; + +function cfg(dir: string, command?: string, timeout_seconds?: number): IrisConfig { + return { + server: { port: 3000, base_url: "http://localhost:3000" }, + storage: { data_dir: dir, agents_dir: "agents", database: join(dir, "iris.sqlite") }, + tagged_pdf: { command, timeout_seconds }, + github: { token: "ghp_test", upstream_repo: "https://github.com/o/r", api_base_url: "https://api.github.com" }, + providers: { default: "openrouter", openrouter: { api_key: "k", default_model: "anthropic/claude-sonnet-4.6" } }, + defaults: { max_review_iterations: 1, extraction_concurrency: 2, max_concurrent_runs: 1, recheck_sample_size: 1 }, + }; +} + +// A session made from one PDF, finished, with two extracted pages. +async function serve(opts: { command?: string; source?: string | null; status?: string; timeout?: number } = {}) { + const dir = mkdtempSync(join(tmpdir(), "iris-tagged-pdf-")); + const config = cfg(dir, "command" in opts ? opts.command : FAKE, opts.timeout); + const store = new Store(config.storage.database); + const paths = new Paths(config); + const id = "ses_pdf_1"; + store.createSession({ session_id: id, github_user_id: USER, image_count: 2, iterations_max: 1 }); + store.updateSession(id, { status: (opts.status ?? "ready_for_review") as "ready_for_review" }); + paths.initSession(id); + writeFileSync(paths.sessionSourceName(id), "permit"); + if (opts.source !== null) writeFileSync(paths.sessionSourcePdf(id), opts.source ?? "%PDF-1.7 source"); + const frag = (order: number, innerHtml: string) => ({ image: `p${order}.png`, order, agent: "page.md", region: "page", innerHtml, edges: [], log: "" }); + // Stored out of order on purpose: the pages must reach the tagger by page order. + writeFileSync(paths.sessionFinalFragments(id), JSON.stringify({ fragments: [frag(2, "

two

"), frag(1, "

one

")], body: "" })); + writeFileSync(paths.sessionOutput(id), 'x'); + + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { + (req as AuthedRequest).user = { github_user_id: USER, github_login: "t", max_review_iterations: 1 } as AuthedRequest["user"]; + next(); + }); + app.use("/v1/sessions", sessionsRouter(config, store)); + app.use("/v1/limits", limitsRouter(config)); + const server = app.listen(0); + await new Promise((r) => server.once("listening", r)); + const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}/v1`; + return { + config, + paths, + id, + fields: () => fetch(`${base}/sessions/${id}/fields`), + tag: (body: unknown) => + fetch(`${base}/sessions/${id}/pdf`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body) }), + limits: () => fetch(`${base}/limits`), + close: () => { + server.close(); + rmSync(dir, { recursive: true, force: true }); + }, + }; +} + +test("tagged PDFs are off unless the command is set and runs", () => { + const dir = tmpdir(); + assert.equal(taggedPdfCommand(cfg(dir)), null); + assert.equal(taggedPdfCommand(cfg(dir, "")), null); + assert.equal(taggedPdfStatus(cfg(dir)), null, "blank says nothing at startup"); + assert.equal(taggedPdfCommand(cfg(dir, "/nonexistent/iris-pdf")), null); + assert.match(taggedPdfStatus(cfg(dir, "/nonexistent/iris-pdf"))!, /^WARNING: .*did not run\. Tagged PDFs are off\.$/); + // A command that runs but is not iris-pdf is not taken for it. + assert.equal(taggedPdfCommand(cfg(dir, "true")), null); + assert.equal(taggedPdfCommand(cfg(dir, FAKE)), FAKE); + assert.equal(taggedPdfStatus(cfg(dir, FAKE)), "Tagged PDFs on (iris-pdf 0.0.0-fake)."); +}); + +test("GET /v1/limits says whether tagged PDFs are on", async () => { + for (const [command, on] of [[FAKE, true], [undefined, false]] as const) { + const s = await serve({ command }); + try { + assert.equal((await (await s.limits()).json()).tagged_pdf, on); + } finally { + s.close(); + } + } +}); + +test("the upload is kept only when tagged PDFs are on and it is one PDF", () => { + const dir = mkdtempSync(join(tmpdir(), "iris-keep-")); + try { + const pdf = { originalname: "a.pdf", buffer: Buffer.from("%PDF") }; + const png = { originalname: "b.png", buffer: Buffer.from("png") }; + const cases = [ + [FAKE, [pdf], true], + [undefined, [pdf], false], + [FAKE, [pdf, pdf], false], + [FAKE, [png], false], + [FAKE, [pdf, png], false], + ] as const; + cases.forEach(([command, files, kept], i) => { + const config = cfg(dir, command); + const paths = new Paths(config); + const id = `ses_keep_${i}`; + paths.initSession(id); + assert.equal(keepSourcePdf(config, paths, id, [...files]), kept, `case ${i}`); + assert.equal(existsSync(paths.sessionSourcePdf(id)), kept, `case ${i}`); + }); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + // And the upload route is what calls it, after the session directory exists. + const route = readFileSync(join(dirname(FAKE), "..", "..", "src", "routes", "sessions.ts"), "utf8"); + const handler = route.slice(route.indexOf('r.post("/", '), route.indexOf("store.createSession(")); + assert.match(handler, /paths\.initSession\(sessionId\);[\s\S]*keepSourcePdf\(cfg, paths, sessionId, files\);/); +}); + +test("GET /fields passes the PDF's fields through", async () => { + const s = await serve(); + try { + const res = await s.fields(); + assert.equal(res.status, 200); + const { fields } = await res.json(); + assert.deepEqual(fields.map((f: { name: string }) => f.name), ["applicant.name", "applicant.consent"]); + } finally { + s.close(); + } +}); + +test("a deployment without the command, or a session without a PDF, says so", async () => { + const off = await serve({ command: undefined }); + try { + const res = await off.fields(); + assert.equal(res.status, 404); + assert.equal((await res.json()).error.code, "tagged_pdf_unavailable"); + assert.equal((await off.tag({})).status, 404); + } finally { + off.close(); + } + const images = await serve({ source: null }); + try { + const res = await images.tag({}); + assert.equal(res.status, 409); + assert.equal((await res.json()).error.code, "no_source_pdf"); + } finally { + images.close(); + } +}); + +test("POST /pdf tags the pages in page order, fills the values, and keeps nothing", async () => { + const s = await serve(); + try { + const res = await s.tag({ values: { "applicant.name": "Ada Lovelace", "applicant.consent": true } }); + assert.equal(res.status, 200); + const body = await res.json(); + assert.equal(body.filename, "permit_tagged.pdf"); + assert.deepEqual(body.report.warnings, [{ code: "duplicate_text_layer", page: 1 }]); + const made = JSON.parse(Buffer.from(body.pdf, "base64").toString("utf8").replace(/^%PDF-fake /, "")); + assert.equal(made.source, "%PDF-1.7 source"); + assert.deepEqual(made.input, { + lang: "fr", + title: "permit", + pages: [{ sourcePage: 1, html: "

one

" }, { sourcePage: 2, html: "

two

" }], + }); + assert.deepEqual(made.values, { "applicant.name": "Ada Lovelace", "applicant.consent": true }); + assert.equal(made.mode, "600", "only this process can read the values file"); + // The values, and the filled PDF, are gone once the answer is sent. + assert.equal(existsSync(made.scratch), false); + const tmp = s.paths.tmpDir(s.id); + assert.deepEqual(readdirSync(tmp).filter((f) => f.startsWith("pdf-")), []); + const log = readFileSync(s.paths.sessionLog(s.id), "utf8"); + assert.match(log, /"tagged_pdf"/); + assert.match(log, /applicant\.name/, "the field names are logged"); + assert.doesNotMatch(log, /Ada Lovelace/, "the values are not"); + } finally { + s.close(); + } +}); + +test("POST /pdf passes the tagger's refusals on with its code", async () => { + const s = await serve(); + try { + const bad = await s.tag({ values: { "unknown.field": "x" } }); + assert.equal(bad.status, 400); + const { error } = await bad.json(); + assert.deepEqual([error.code, error.message], ["bad_value", "No field is named unknown.field."]); + assert.equal((await s.tag({ values: ["x"] })).status, 400); + // A crash inside the tagger can quote a value. The client sent it, so it may see it + // again, but the log does not keep it. + const crash = await s.tag({ values: { crash: "Jane Doe" } }); + assert.equal((await crash.json()).error.code, "internal_error"); + const log = readFileSync(s.paths.sessionLog(s.id), "utf8"); + assert.match(log, /"tagged_pdf_failed".*"bad_value".*No field is named unknown\.field/); + assert.doesNotMatch(log, /Jane Doe/); + } finally { + s.close(); + } + const locked = await serve({ source: "%PDF ENCRYPTED" }); + try { + const res = await locked.tag({}); + assert.equal(res.status, 422); + assert.equal((await res.json()).error.code, "encrypted"); + assert.equal((await locked.fields()).status, 422); + } finally { + locked.close(); + } +}); + +test("POST /pdf waits for the finished document and stops a run that takes too long", async () => { + const running = await serve({ status: "running" }); + try { + assert.equal((await running.tag({})).status, 409); + } finally { + running.close(); + } + const slow = await serve({ timeout: 0.5 }); + try { + const res = await slow.tag({ values: { slow: true } }); + assert.equal(res.status, 504); + assert.equal((await res.json()).error.code, "timeout"); + assert.deepEqual(readdirSync(slow.paths.tmpDir(slow.id)).filter((f) => f.startsWith("pdf-")), []); + } finally { + slow.close(); + } +}); From 5b3611f597953d80c7051be95ec012552e9466c6 Mon Sep 17 00:00:00 2001 From: Blake Bertuccelli-Booth <46652+bbertucc@users.noreply.github.com> Date: Thu, 24 Sep 2026 19:00:33 -0400 Subject: [PATCH 2/7] fix(tagged-pdf): scratch outside the session, one cap for both routes, a checkbox can be unticked Review notes on #482: - Scratch dirs go under data_dir/tmp/pdf-*, not tmp//. Closing a session removes tmp/, so a close during a tag run failed the run with a 500, and a run after close left an orphan behind. - GET /fields runs the tagger too, so it now shares the 2-run cap. - A page that failed extraction is left out of the tagger's input. It holds only its @page-failed note, so tagging it would give a blank page. - The demo asks each checkbox as leave it / checked / not checked. An unticked box used to mean "leave it", so a box the PDF had ticked could not be unticked. - The demo says the tags come from the first pass, before review. - API.md lists /fields errors and the shared cap. Co-Authored-By: Claude Opus 5.5 --- docs/API.md | 10 ++++++---- public/demo.html | 17 +++++++++-------- src/routes/sessions.ts | 36 ++++++++++++++++++++++++------------ src/store/paths.ts | 5 +++++ test/demo-pdf-form.test.ts | 11 ++++++++++- test/tagged-pdf.test.ts | 33 ++++++++++++++++++++++++++++----- 6 files changed, 82 insertions(+), 30 deletions(-) diff --git a/docs/API.md b/docs/API.md index 8ac3d89..7e8835a 100644 --- a/docs/API.md +++ b/docs/API.md @@ -978,7 +978,8 @@ curl -s -H "$AUTH" "$BASE/sessions/$SID/fields" | jq '.fields[0]' ``` `type` is one of `text`, `checkbox`, `radio`, `combobox`, `listbox`, `button` or `signature`. A PDF with -no form gives `{"fields": []}`. +no form gives `{"fields": []}`. It can answer `404`, `409 no_source_pdf`, `422` for a PDF the tagger +refuses, or `503 busy`, the same as below. **Get the tagged PDF:** @@ -999,10 +1000,11 @@ tagger's report. `report.warnings` lists what the tagger could not do cleanly, s it could not find in the HTML. The demo page turns each warning into a plain sentence. The session must have finished (`ready_for_review` or `closed`), or you get `409 invalid_state`. The -PDF is tagged from the first pass's HTML for each page, before review. Errors from the tagger keep +PDF is tagged from the first pass's HTML for each page, before review. A page that failed extraction is +left out, and the report lists it as `page_not_in_html`. Errors from the tagger keep its code: `400` for a bad value, `422` for a PDF it refuses (for example `encrypted`), and `504 -timeout` after `tagged_pdf.timeout_seconds` (300 by default). Two tagging requests at once is the -limit, so a third gets `503 busy` with `Retry-After`. A deployment without the feature answers +timeout` after `tagged_pdf.timeout_seconds` (300 by default). Two tagger runs at once, counting +both `/fields` and `/pdf`, is the limit, so a third gets `503 busy` with `Retry-After`. A deployment without the feature answers `404 tagged_pdf_unavailable`. ## Submit feedback (re-run) diff --git a/public/demo.html b/public/demo.html index c8e89b8..53b7dff 100644 --- a/public/demo.html +++ b/public/demo.html @@ -171,7 +171,7 @@

Your accessible HTML

form is built when the result is shown. -->