diff --git a/README.md b/README.md index e4510d0..266e9ce 100644 --- a/README.md +++ b/README.md @@ -171,6 +171,10 @@ key inline** — this section is the decisions an operator has to make, not the for a single Caddy/nginx). Without it every caller presents as the proxy's address and shares one bucket; the log warns when it sees an `X-Forwarded-For` while this is unset. Anything Iris cannot interpret warns and trusts nothing rather than failing startup. +- **Tagged PDFs — optional, off by default.** Set `tagged_pdf.command` to the `iris-pdf` command from + [equalify-iris-pdf](https://github.com/EqualifyEverything/equalify-iris-pdf), and a session made from + one PDF can return that PDF tagged, with its form filled in. The demo page shows this only when it is + on. Iris runs without it. ### One GitHub identity, and no sign-in @@ -222,6 +226,8 @@ in the other direction: it has its own shared secret and 404s unless you set it. | `POST /v1/sessions` | Create a session, upload images and/or PDFs (`multipart/form-data`) | | `GET /v1/sessions/{id}` | Poll status | | `GET /v1/sessions/{id}/output` | Fetch the HTML when ready | +| `GET /v1/sessions/{id}/fields` | The uploaded PDF's form fields (only with tagged PDFs on) | +| `POST /v1/sessions/{id}/pdf` | The uploaded PDF back, tagged, with its form filled in (only with tagged PDFs on) | | `POST /v1/sessions/{id}/feedback` | Submit feedback, trigger a re-run | | `POST /v1/sessions/{id}/close` | Finalize the session and clean tmp | | `GET /v1/sessions/{id}/logs` | Fetch the run log (ndjson) | diff --git a/config.example.yaml b/config.example.yaml index 1cc26fc..45f8256 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -93,6 +93,20 @@ storage: # alternative for multi-instance deployments (not implemented in v1). database: ./data/iris.sqlite +# Optional: give back the uploaded PDF, tagged for accessibility, with form fields filled +# in. Needs the `iris-pdf` command from +# https://github.com/EqualifyEverything/equalify-iris-pdf, which is not part of Iris. +# Blank (the default) turns it off, and Iris works as it does without it. +# +# With it on, a session made from one PDF keeps that PDF, and the demo page offers +# a form for the PDF's fields and "Download tagged PDF". Form values are personal data: +# they are used for one request, and Iris stores neither them nor the filled PDF. +tagged_pdf: + # The command to run, e.g. `iris-pdf` if it is on PATH. + command: ${IRIS_PDF_COMMAND} + # How long one PDF may take before the command is stopped. + timeout_seconds: 300 + # Iris has ONE GitHub identity: the token below. The server holds it, callers never # see it, and nobody signs in. Every session is owned by that account and every issue # Iris files is filed under it. diff --git a/docs/API.md b/docs/API.md index 9075623..7e8835a 100644 --- a/docs/API.md +++ b/docs/API.md @@ -712,7 +712,8 @@ curl -s "$BASE/limits" | jq "upload_per_minute": 12, "max_upload_memory_mb": 256, "window_seconds": 60 - } + }, + "tagged_pdf": false } ``` @@ -740,7 +741,8 @@ are streaming: the byte total is checked before the body is read (or counted as request declares no length), while the part count is refused during parsing, once a part past `max_files` appears. `rate_limits` is how often you may ask ([Rate limits](#rate-limits-how-often-you-may-ask)), and is `null` on a deployment that does not -limit requests in the app — which means "not limiting", not "unknown". +limit requests in the app — which means "not limiting", not "unknown". `tagged_pdf` says whether +this deployment can [return a tagged PDF](#get-a-tagged-pdf-optional). A PDF's **links survive**, which rasterizing alone would not manage: a link is an annotation over the page rather than something drawn on it, so the page image carries the link text and @@ -956,6 +958,55 @@ those references 404 until a consumer supplies the files. What a screen-reader u browser will show broken images, and one that rewrites the `src`s has the log and the fragment to match them against. +## Get a tagged PDF (optional) + +A deployment that runs [equalify-iris-pdf](https://github.com/EqualifyEverything/equalify-iris-pdf) +can give you the original PDF back with tags, built from the HTML. The feature is off unless +`tagged_pdf.command` is set in the config. `GET /v1/limits` says whether it is on. + +It works only for a session made from **one PDF**. Iris keeps that PDF so it can tag it. Sessions made +from images, or from several files, get `409 no_source_pdf`. + +**Read the PDF's form fields:** + +```bash +curl -s -H "$AUTH" "$BASE/sessions/$SID/fields" | jq '.fields[0]' +``` +```json +{ "name": "applicant.name", "type": "text", "page": 1, "options": [], "required": true, + "readonly": false, "maxlen": 40, "editable": false, "multiSelect": false } +``` + +`type` is one of `text`, `checkbox`, `radio`, `combobox`, `listbox`, `button` or `signature`. A PDF with +no form gives `{"fields": []}`. It can answer `404`, `409 no_source_pdf`, `422` for a PDF the tagger +refuses, or `503 busy`, the same as below. + +**Get the tagged PDF:** + +```bash +curl -s -X POST -H "$AUTH" "$BASE/sessions/$SID/pdf" \ + -H 'content-type: application/json' \ + -d '{"values":{"applicant.name":"Jane Doe","applicant.consent":true}}' > tagged.json +jq -r .pdf tagged.json | base64 --decode > tagged.pdf +``` + +`values` is optional. Its keys are field names. A text field takes a string, a checkbox takes `true` +or `false`, and a multi-select list takes an array. Fields you leave out keep what the PDF had. +**The values are used for this one request and are never stored or logged.** The run log records +only which field names were given. + +The answer is `{"filename", "pdf", "report"}`: the file name to save as, the PDF in base64, and the +tagger's report. `report.warnings` lists what the tagger could not do cleanly, such as a form field +it could not find in the HTML. The demo page turns each warning into a plain sentence. + +The session must have finished (`ready_for_review` or `closed`), or you get `409 invalid_state`. The +PDF is tagged from the first pass's HTML for each page, before review. A page that failed extraction is +left out, and the report lists it as `page_not_in_html`. Errors from the tagger keep +its code: `400` for a bad value, `422` for a PDF it refuses (for example `encrypted`), and `504 +timeout` after `tagged_pdf.timeout_seconds` (300 by default). Two tagger runs at once, counting +both `/fields` and `/pdf`, is the limit, so a third gets `503 busy` with `Retry-After`. A deployment without the feature answers +`404 tagged_pdf_unavailable`. + ## Submit feedback (re-run) Triggers a new run within the same session, with the feedback injected as a top-level @@ -1000,8 +1051,8 @@ The events worth grepping for have a section each below, and the index is a link the index when you have a `type` off a log line and want to know what it means; read a section when you want to know what the field it names is for and what it costs. -**The index is the whole log.** `src/` emits **118** event types and every one of them has a section -below — **113** sections, because a few cover a pair of events that are only read together. So a +**The index is the whole log.** `src/` emits **120** event types and every one of them has a section +below — **114** sections, because a few cover a pair of events that are only read together. So a `type` you cannot find here is not one the index skipped: it is a misread line, or a name `src/` no longer emits. @@ -1125,6 +1176,7 @@ emits fails it too. | [`agent_issue_failed`](#agent_issue_failed) | A suggestion did not get filed, and which of the two calls failed | | [`contribution_failed`](#contribution_failed) | The filing step threw, **after** `run_complete` | | [`run_failed`](#run_failed) | The run threw, so there is **no document** | +| [`tagged_pdf` / `tagged_pdf_failed`](#tagged_pdf--tagged_pdf_failed) | A tagged PDF was made, or could not be | | [`calibrate_call_failed`](#calibrate_call_failed) | One calibration verifier call threw — a tool's line, never a run's | ### `run_queued` / `run_dequeued` @@ -4278,6 +4330,13 @@ which failure it was, and this `error` is written to match: with `blank: 0` it i provider error, standing for all of them because that is the diagnosis; otherwise it counts the source pages reported blank, which an empty document could not. +### `tagged_pdf` / `tagged_pdf_failed` + +A [tagged PDF](#get-a-tagged-pdf-optional) was made, or the tagger refused. `ms` is how long it took. +`tagged_pdf` lists `fields_given`, the names of the fields that were filled in. The values are +never logged. `tagged_pdf_failed` has the tagger's `code` and `error`. An `internal_error`'s +message is left out, because it may quote a value. + ### `calibrate_call_failed` One verifier call in the calibration harness threw: `image` is the page, `defect` the seeded defect the diff --git a/public/demo.html b/public/demo.html index 339d143..09ffd24 100644 --- a/public/demo.html +++ b/public/demo.html @@ -49,6 +49,15 @@ border: 1px solid var(--border); border-radius: 8px; background: var(--bg); color: var(--fg); } textarea { min-height: 5rem; } + select { + font: inherit; max-width: 32rem; padding: .5rem .65rem; + border: 1px solid var(--border); border-radius: 8px; background: var(--bg); color: var(--fg); + } + h3 { font-size: 1.1rem; margin: 1.5rem 0 .5rem; } + fieldset { border: 1px solid var(--border); border-radius: 8px; margin: 1rem 0; padding: .25rem 1rem 1rem; } + legend { font-weight: 600; padding: 0 .35rem; } + .choice { display: flex; gap: .5rem; align-items: center; margin: .5rem 0; } + .choice label { display: inline; margin: 0; font-weight: 400; } button { font: inherit; font-weight: 600; cursor: pointer; border: 1px solid transparent; border-radius: 8px; padding: .6rem 1.1rem; background: var(--accent); color: var(--accent-fg); @@ -157,6 +166,25 @@

Your accessible HTML

+ + +

Iris will re-run the conversion with your feedback applied (for example: “Keep footnotes separate from body text”).

@@ -195,6 +223,8 @@ // that enforces, and a page that guessed would either reject a fine file or // promise one that fails. let uploadLimits = null; + // Whether this deployment makes tagged PDFs (GET /v1/limits). False until it says so. + let taggedPdfOn = false; const $ = (id) => document.getElementById(id); const show = (id) => { $(id).hidden = false; }; @@ -397,8 +427,182 @@ // report to the person converting a document. The run log's `agent_issue` // events are where that information actually lives if it is wanted back. hide('status-section'); show('result-section'); focusHeading('result-h'); + // Best-effort, and not awaited: the HTML is the result, and the PDF offer is extra. + // A busy tagger makes showPdfPart wait, and the result must not wait with it. + showPdfPart().catch(() => hide('pdf-part')); + } + + // ----- Tagged PDF (optional) ----- + // One control per settable field of the source PDF, labelled with the field's name. + // `doc` is passed in so the tests can build the form in their own document. + // Returns how many fields got a control. Read-only fields, buttons and signatures + // cannot be set, so they get none. + function renderFields(doc, list, fields) { + const el = (tag, props = {}) => Object.assign(doc.createElement(tag), props); + const req = (f) => (f.required ? ' (required on the form)' : ''); + let n = 0; + for (const f of fields) { + if (f.readonly) continue; + const id = 'pdf-f' + n; + let box; + if (f.type === 'text' || (f.type === 'combobox' && f.editable)) { + box = el('div'); + box.append(el('label', { htmlFor: id, textContent: f.name + req(f) })); + const input = el('input', { id, type: 'text' }); + if (f.maxlen) input.maxLength = f.maxlen; + if (f.options.length) { + const dl = el('datalist', { id: id + '-list' }); + for (const o of f.options) dl.append(el('option', { value: o })); + box.append(dl); + input.setAttribute('list', dl.id); + } + box.append(input); + } else if (f.type === 'radio' || f.type === 'checkbox') { + box = el('fieldset'); + box.append(el('legend', { textContent: f.name + req(f) })); + // A chosen radio cannot be unchosen, so "leave it" is a choice of its own. A + // checkbox is asked the same way: the page cannot show whether the PDF has it + // checked, so an unticked box would not say "leave it" or "untick it". + const choices = f.type === 'checkbox' ? ['', 'Checked', 'Not checked'] : ['', ...f.options]; + choices.forEach((o, i) => { + const row = el('div', { className: 'choice' }); + row.append( + el('input', { id: id + '-' + i, type: 'radio', name: id, value: o, checked: i === 0 }), + el('label', { htmlFor: id + '-' + i, textContent: o || 'Leave as it is' }), + ); + box.append(row); + }); + } else if (f.type === 'combobox' || f.type === 'listbox') { + box = el('div'); + box.append(el('label', { htmlFor: id, textContent: f.name + req(f) })); + const select = el('select', { id, multiple: f.multiSelect }); + if (!f.multiSelect) select.append(el('option', { value: '', textContent: 'Leave as it is' })); + for (const o of f.options) select.append(el('option', { value: o, textContent: o })); + box.append(select); + } else { + continue; + } + box.dataset.field = f.name; + box.dataset.type = f.type; + list.append(box); + n++; + } + return n; + } + + // What was entered, as the API takes it. A blank field is left out, so the PDF keeps + // what it had. + function fieldValues(list) { + const values = {}; + for (const box of list.querySelectorAll('[data-field]')) { + const name = box.dataset.field; + if (box.dataset.type === 'checkbox') { + const picked = box.querySelector('input:checked'); + if (picked && picked.value) values[name] = picked.value === 'Checked'; + } else if (box.dataset.type === 'radio') { + const picked = box.querySelector('input:checked'); + if (picked && picked.value) values[name] = picked.value; + } else if (box.querySelector('select[multiple]')) { + const picked = [...box.querySelector('select').selectedOptions].map((o) => o.value); + if (picked.length) values[name] = picked; + } else { + const v = box.querySelector('input, select').value; + if (v) values[name] = v; + } + } + return values; + } + + // The tagger's report, as sentences a person can act on. One per kind of warning. + function pdfNotes(report) { + const count = {}; + for (const w of (report && report.warnings) || []) count[w.code] = (count[w.code] || 0) + 1; + const s = (n, one, many) => (n === 1 ? one : many.replace('#', n)); + const say = { + duplicate_text_layer: (n) => s(n, 'One page now has its text twice: screen readers read it once, but copied text may come out doubled.', '# pages now have their text twice: screen readers read it once, but copied text may come out doubled.'), + unmatched_text: () => 'Some text on the page was not in the HTML. It was kept, as plain paragraphs.', + missing_alt: (n) => s(n, 'One image has no description.', '# images have no description.'), + field_not_in_html: (n) => s(n, 'One form field was not found in the HTML, so a screen reader reaches it at the end of its page.', '# form fields were not found in the HTML, so a screen reader reaches them at the end of their page.'), + unmatched_link: (n) => s(n, 'One link was not found in the HTML, so a screen reader reaches it at the end of its page.', '# links were not found in the HTML, so a screen reader reaches them at the end of their page.'), + page_not_tagged: (n) => s(n, 'One page was left untagged.', '# pages were left untagged.'), + page_not_in_html: (n) => s(n, 'One page had no HTML and was left untagged.', '# pages had no HTML and were left untagged.'), + alignment_incomplete: () => 'On some pages the HTML and the page differed too much to match every word. The rest was kept as plain paragraphs.', + font_not_embedded: () => 'The PDF uses fonts it does not include, so it cannot claim to meet PDF/UA, the PDF accessibility standard.', + source_marked_content: () => 'The PDF has tags left over from another tool, so it cannot claim to meet PDF/UA, the PDF accessibility standard.', + repaired: () => 'The PDF was damaged. The tagged file is a repaired copy.', + link_not_on_page: (n) => s(n, 'One link in the HTML is not on the page, so it was left out.', '# links in the HTML are not on the page, so they were left out.'), + missing_glyph: () => 'Some characters could not be written in the PDF\'s fonts, so they may read wrongly.', + unmapped_element: () => 'Some HTML had no PDF tag to match, so it was tagged as a paragraph.', + field_without_name: () => 'A form control in the HTML had no name, so it was not matched to the PDF\'s form.', + no_text_positions: (n) => s(n, 'One page has no text the tagger could find, so it was left untagged.', '# pages have no text the tagger could find, so they were left untagged.'), + xfa_removed: () => 'The PDF had two versions of its form. The one most readers cannot use was removed.', + signature_invalidated: () => 'The PDF was signed. The signature no longer holds, because the file changed.', + signature_field_skipped: () => 'Signature fields cannot be filled in here, so they were left empty.', + }; + return Object.keys(count).filter((c) => c !== 'no_title').map((c) => (say[c] ? say[c](count[c]) : 'Note from the tagger: ' + c + '.')); + } + + // Each call supersedes the last. A feedback re-run keeps the session id, so the id + // alone cannot tell a reply meant for a result the reader has since left. + let pdfTurn = 0; + async function showPdfPart() { + hide('pdf-part'); + if (!taggedPdfOn) return; + const id = sessionId, turn = ++pdfTurn; + const stale = () => turn !== pdfTurn || sessionId !== id || converting; + let res = await api('/sessions/' + id + '/fields'); + // 503 means other PDFs are being tagged. Wait and ask again, a few times. + let tries = 0; + for (; res.status === 503 && tries < 5; tries++) { + await new Promise((r) => setTimeout(r, (Number(res.headers.get('retry-after')) || 10) * 1000)); + if (stale()) return; // the reader has moved on: another document, or a re-run + res = await api('/sessions/' + id + '/fields'); + } + if (!res.ok || stale()) return; // not one uploaded PDF, or the PDF cannot be read: no offer + const { fields } = await res.json(); + if (stale()) return; // checked again after the last await, right before the page changes + $('pdf-field-list').replaceChildren(); + $('pdf-notes').replaceChildren(); + $('pdf-fields').hidden = renderFields(document, $('pdf-field-list'), fields || []) === 0; + show('pdf-part'); + // After a wait the reader is already on the result, so say the offer has appeared. + // Without a wait it is there before they reach it, and "ready" was just announced. + if (tries > 0) live('You can now get your PDF back, tagged. It is in a new section below the buttons.'); } + $('pdf-form').addEventListener('submit', async (ev) => { + ev.preventDefault(); + setError(''); + $('pdf-btn').disabled = true; + live('Tagging your PDF. This can take a minute.'); + try { + const res = await api('/sessions/' + sessionId + '/pdf', { + method: 'POST', headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ values: fieldValues($('pdf-field-list')) }), + }); + if (!res.ok) throw new Error(await errMessage(res)); + const out = await res.json(); + const bytes = Uint8Array.from(atob(out.pdf), (c) => c.charCodeAt(0)); + const url = URL.createObjectURL(new Blob([bytes], { type: 'application/pdf' })); + const a = document.createElement('a'); + a.href = url; a.download = out.filename; + document.body.appendChild(a); a.click(); a.remove(); + setTimeout(() => URL.revokeObjectURL(url), 60000); + const notes = pdfNotes(out.report); + const list = document.createElement('ul'); + for (const n of notes) list.append(Object.assign(document.createElement('li'), { textContent: n })); + const head = Object.assign(document.createElement('p'), { + textContent: 'Downloaded ' + out.filename + '.' + (notes.length ? ' Notes on it:' : ''), + }); + $('pdf-notes').replaceChildren(head, ...(notes.length ? [list] : [])); + live('Downloaded ' + out.filename + '.' + (notes.length === 1 ? ' One note is below the button.' : notes.length ? ' ' + notes.length + ' notes are below the button.' : '')); + } catch (e) { + setError('Could not tag the PDF: ' + e.message); + } finally { + $('pdf-btn').disabled = false; + } + }); + $('restart-btn').addEventListener('click', () => { sessionId = null; setError(''); $('upload-form').reset(); @@ -548,6 +752,7 @@ const res = await fetch(API + '/limits'); if (!res.ok) return; const l = await res.json(); + taggedPdfOn = Boolean(l && l.tagged_pdf === true); const bytes = Number(l && l.image && l.image.max_bytes); if (!Number.isFinite(bytes) || bytes < 1) return; uploadLimits = { maxBytes: bytes, hint: String(l.image.hint || '') }; diff --git a/src/config.ts b/src/config.ts index 086539e..9b6f1fc 100644 --- a/src/config.ts +++ b/src/config.ts @@ -140,6 +140,9 @@ export interface IrisConfig { rate_limits?: Partial; }; storage: { data_dir: string; agents_dir: string; database: string }; + // Optional. The `iris-pdf` command that tags an uploaded PDF (util/taggedPdf.ts). + // Blank or absent means the feature is off. + tagged_pdf?: { command?: string; timeout_seconds?: number }; github: { upstream_repo: string; // Overridable for GitHub Enterprise (and for testing). Defaults below. diff --git a/src/index.ts b/src/index.ts index d3e22f4..0c53c44 100644 --- a/src/index.ts +++ b/src/index.ts @@ -18,6 +18,7 @@ import { meRouter } from "./routes/me.ts"; import { sessionsRouter } from "./routes/sessions.ts"; import { statsRouter } from "./routes/stats.ts"; import { limitsRouter } from "./routes/limits.ts"; +import { clearPdfScratch, taggedPdfStatus } from "./util/taggedPdf.ts"; import { qualityRouter } from "./routes/quality.ts"; import { visionModelWarning } from "./providers/imageLimits.ts"; import { generalRateLimit } from "./util/requestLimits.ts"; @@ -245,6 +246,11 @@ const openStorage = (): { store: Store; stale: number } => { // produces it is the guarded first write above. const { store, stale } = openStorage(); if (stale > 0) console.log(`Marked ${stale} interrupted session(s) as failed on startup.`); +// Optional. Silent when tagged_pdf.command is blank; says so when it is set and does not run. +const pdfStatus = taggedPdfStatus(cfg); +const leftover = clearPdfScratch(join(cfg.storage.data_dir, "tmp")); +if (leftover > 0) console.log(`Removed ${leftover} tagged-PDF scratch dir(s) left by a previous process.`); +if (pdfStatus) (pdfStatus.startsWith("WARNING") ? console.warn : console.log)(pdfStatus); const app = express(); // Whose address `req.ip` is. Off unless a deployment says how many proxies are in front // of it, because the rate limits below are only per-caller if this is right: unset behind diff --git a/src/routes/limits.ts b/src/routes/limits.ts index 3554ef2..595f565 100644 --- a/src/routes/limits.ts +++ b/src/routes/limits.ts @@ -2,6 +2,7 @@ import { Router } from "express"; import type { IrisConfig } from "../config.ts"; import { imageLimitsHint, resolveImageLimits } from "../providers/imageLimits.ts"; import { MAX_PDF_PAGES } from "../util/pdf.ts"; +import { taggedPdfCommand } from "../util/taggedPdf.ts"; import { MAX_UPLOAD_FILES, publishedRateLimits, uploadCeilingBytes } from "../util/requestLimits.ts"; /** @@ -23,7 +24,8 @@ import { MAX_UPLOAD_FILES, publishedRateLimits, uploadCeilingBytes } from "../ut * "pdf": { "max_pages": 25 }, * "upload": { "max_files": 25, "max_request_bytes": 134217728 }, * "rate_limits": { "general_per_minute": 240, "upload_per_minute": 12, - * "max_upload_memory_mb": 256, "window_seconds": 60 } } + * "max_upload_memory_mb": 256, "window_seconds": 60 }, + * "tagged_pdf": false } * ``` * * Not handed the auth middleware, like `GET /v1/stats`, so it answers even where the @@ -64,6 +66,9 @@ export function limitsRouter(cfg: IrisConfig): Router { // volume in the app. Same rationale as the upload limits: a budget a client can read // is one it can pace itself against, instead of discovering it by being refused. rate_limits: publishedRateLimits(cfg), + // Whether POST /v1/sessions/{id}/pdf can answer here (util/taggedPdf.ts), so a client + // offers it only where it works. + tagged_pdf: taggedPdfCommand(cfg) !== null, }; r.get("/", (_req, res) => { diff --git a/src/routes/sessions.ts b/src/routes/sessions.ts index cbf5468..8f94d9f 100644 --- a/src/routes/sessions.ts +++ b/src/routes/sessions.ts @@ -32,6 +32,7 @@ import { resolveImageLimits, } from "../providers/imageLimits.ts"; import { imageDimensions } from "../util/imageSize.ts"; +import { readFields, tagPdf, taggedPdfCommand, tagTimeoutSeconds, TaggedPdfError } from "../util/taggedPdf.ts"; // 50 MB is a memory bound, not the image limit. It stays well above what an image may // be (see imageLimits.ts) because a PDF legitimately is: 25 pages of scans is a large @@ -134,6 +135,19 @@ function ownedSession(store: Store, id: string, userId: number): SessionRecord | return s; } +// Keep the upload for a tagged PDF later, when this deployment makes them and the upload +// was one PDF on its own. Then the PDF's page N is the session's page N. +export function keepSourcePdf( + cfg: IrisConfig, + paths: Paths, + sessionId: string, + files: { originalname: string; buffer: Buffer }[], +): boolean { + if (!taggedPdfCommand(cfg) || files.length !== 1 || !PDF_EXT.test(files[0].originalname)) return false; + writeFileSync(paths.sessionSourcePdf(sessionId), files[0].buffer); + return true; +} + export function sessionsRouter(cfg: IrisConfig, store: Store): Router { const r = Router(); const paths = new Paths(cfg); @@ -383,6 +397,7 @@ export function sessionsRouter(cfg: IrisConfig, store: Store): Router { if (Object.keys(linksByOrder).length) { writeFileSync(paths.sessionLinks(sessionId), JSON.stringify(linksByOrder, null, 2)); } + keepSourcePdf(cfg, paths, sessionId, files); const record = store.createSession({ session_id: sessionId, @@ -456,6 +471,126 @@ export function sessionsRouter(cfg: IrisConfig, store: Store): Router { res.type("text/html").send(html); }); + // ----- Tagged PDF (optional; util/taggedPdf.ts) ----- + // The session's source PDF, or the reason it has none. Sends that error itself. + function sourcePdf(req: AuthedRequest, res: Response): { s: SessionRecord; command: string; pdf: string } | null { + const s = ownedSession(store, req.params.id, req.user!.github_user_id); + if (!s) { + sendError(res, 404, "session_not_found", "No such session"); + return null; + } + const command = taggedPdfCommand(cfg); + if (!command) { + sendError(res, 404, "tagged_pdf_unavailable", "This deployment does not make tagged PDFs."); + return null; + } + const pdf = paths.sessionSourcePdf(s.session_id); + if (!existsSync(pdf)) { + sendError(res, 409, "no_source_pdf", "Only a session made from one uploaded PDF can be tagged."); + return null; + } + return { s, command, pdf }; + } + // Express 4 does not catch a rejected async handler, so every error is answered here. + function taggerError(res: Response, e: unknown): void { + if (!(e instanceof TaggedPdfError)) { + sendError(res, 500, "tagger_failed", "Could not tag the PDF."); + return; + } + const status = e.code === "timeout" ? 504 : e.exit === 3 ? 400 : 422; + sendError(res, status, e.code, e.message); + } + + // Each tagger run is a child process with seconds of CPU. Two at once, across both + // routes, is enough. + let tagging = 0; + function tooBusy(res: Response): boolean { + if (tagging < 2) return false; + res.setHeader("Retry-After", "10"); + sendError(res, 503, "busy", "Other PDFs are being tagged. Try again in a few seconds."); + return true; + } + + // GET /v1/sessions/{id}/fields — the source PDF's form fields, for a fill-in form. + r.get("/:id/fields", async (req: AuthedRequest, res) => { + const src = sourcePdf(req, res); + if (!src) return; + if (tooBusy(res)) return; + tagging++; + try { + res.json({ fields: await readFields(src.command, src.pdf) }); + } catch (e) { + taggerError(res, e); + } finally { + tagging--; + } + }); + + // POST /v1/sessions/{id}/pdf — the source PDF, tagged from the extracted pages, with + // `values` filled in. The PDF is the answer and nothing is kept, so neither the values + // nor the filled PDF is ever stored. + r.post("/:id/pdf", async (req: AuthedRequest, res) => { + const src = sourcePdf(req, res); + if (!src) return; + const { s } = src; + const finalPath = paths.sessionFinalFragments(s.session_id); + if ((s.status !== "ready_for_review" && s.status !== "closed") || !existsSync(finalPath)) { + sendError(res, 409, "invalid_state", "The PDF can be tagged once the session is ready_for_review."); + return; + } + const values: unknown = (req.body ?? {}).values ?? {}; + if (typeof values !== "object" || values === null || Array.isArray(values)) { + sendError(res, 400, "invalid_request", "`values` must be an object of field name to value."); + return; + } + if (tooBusy(res)) return; + const log = new RunLog(paths.sessionLog(s.session_id)); + const started = Date.now(); + tagging++; + try { + const base = existsSync(paths.sessionSourceName(s.session_id)) + ? readFileSync(paths.sessionSourceName(s.session_id), "utf8").trim() || "document" + : "document"; + // The pages as extracted, by page order. Order is the PDF's page number, because + // the session is one PDF. A page that failed extraction holds only its + // `@page-failed` note, so it is left out and the tagger reports it untagged. + const { fragments = [] } = JSON.parse(readFileSync(finalPath, "utf8")) as { fragments?: Fragment[] }; + const outPath = paths.sessionOutput(s.session_id); + const output = existsSync(outPath) ? readFileSync(outPath, "utf8") : ""; + const input = { + lang: output.match(/]*\blang="([^"]+)"/i)?.[1], + title: base, + pages: [...fragments] + .filter((f) => !f.innerHtml.trimStart().startsWith(""), frag(1, "

one

")]; + writeFileSync(paths.sessionFinalFragments(id), JSON.stringify({ fragments: pages, body: "" })); + writeFileSync(paths.sessionOutput(id), 'x'); + + const app = express(); + app.use(express.json()); + app.use((req, _res, next) => { + (req as AuthedRequest).user = { github_user_id: USER, github_login: "t", max_review_iterations: 1 } as AuthedRequest["user"]; + next(); + }); + app.use("/v1/sessions", sessionsRouter(config, store)); + app.use("/v1/limits", limitsRouter(config)); + const server = app.listen(0); + await new Promise((r) => server.once("listening", r)); + const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}/v1`; + return { + config, + paths, + id, + fields: () => fetch(`${base}/sessions/${id}/fields`), + tag: (body: unknown) => + fetch(`${base}/sessions/${id}/pdf`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body) }), + limits: () => fetch(`${base}/limits`), + close: () => { + server.close(); + rmSync(dir, { recursive: true, force: true }); + }, + }; +} + +test("tagged PDFs are off unless the command is set and runs", () => { + const dir = tmpdir(); + assert.equal(taggedPdfCommand(cfg(dir)), null); + assert.equal(taggedPdfCommand(cfg(dir, "")), null); + assert.equal(taggedPdfStatus(cfg(dir)), null, "blank says nothing at startup"); + assert.equal(taggedPdfCommand(cfg(dir, "/nonexistent/iris-pdf")), null); + assert.match(taggedPdfStatus(cfg(dir, "/nonexistent/iris-pdf"))!, /^WARNING: .*did not run\. Tagged PDFs are off\.$/); + // A command that runs but is not iris-pdf is not taken for it. + assert.equal(taggedPdfCommand(cfg(dir, "true")), null); + assert.equal(taggedPdfCommand(cfg(dir, FAKE)), FAKE); + assert.equal(taggedPdfStatus(cfg(dir, FAKE)), "Tagged PDFs on (iris-pdf 0.0.0-fake)."); +}); + +test("GET /v1/limits says whether tagged PDFs are on", async () => { + for (const [command, on] of [[FAKE, true], [undefined, false]] as const) { + const s = await serve({ command }); + try { + assert.equal((await (await s.limits()).json()).tagged_pdf, on); + } finally { + s.close(); + } + } +}); + +test("the upload is kept only when tagged PDFs are on and it is one PDF", () => { + const dir = mkdtempSync(join(tmpdir(), "iris-keep-")); + try { + const pdf = { originalname: "a.pdf", buffer: Buffer.from("%PDF") }; + const png = { originalname: "b.png", buffer: Buffer.from("png") }; + const cases = [ + [FAKE, [pdf], true], + [undefined, [pdf], false], + [FAKE, [pdf, pdf], false], + [FAKE, [png], false], + [FAKE, [pdf, png], false], + ] as const; + cases.forEach(([command, files, kept], i) => { + const config = cfg(dir, command); + const paths = new Paths(config); + const id = `ses_keep_${i}`; + paths.initSession(id); + assert.equal(keepSourcePdf(config, paths, id, [...files]), kept, `case ${i}`); + assert.equal(existsSync(paths.sessionSourcePdf(id)), kept, `case ${i}`); + }); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + // And the upload route is what calls it, after the session directory exists. + const route = readFileSync(join(dirname(FAKE), "..", "..", "src", "routes", "sessions.ts"), "utf8"); + const handler = route.slice(route.indexOf('r.post("/", '), route.indexOf("store.createSession(")); + assert.match(handler, /paths\.initSession\(sessionId\);[\s\S]*keepSourcePdf\(cfg, paths, sessionId, files\);/); +}); + +test("GET /fields passes the PDF's fields through", async () => { + const s = await serve(); + try { + const res = await s.fields(); + assert.equal(res.status, 200); + const { fields } = await res.json(); + assert.deepEqual(fields.map((f: { name: string }) => f.name), ["applicant.name", "applicant.consent"]); + } finally { + s.close(); + } +}); + +test("a deployment without the command, or a session without a PDF, says so", async () => { + const off = await serve({ command: undefined }); + try { + const res = await off.fields(); + assert.equal(res.status, 404); + assert.equal((await res.json()).error.code, "tagged_pdf_unavailable"); + assert.equal((await off.tag({})).status, 404); + } finally { + off.close(); + } + const images = await serve({ source: null }); + try { + const res = await images.tag({}); + assert.equal(res.status, 409); + assert.equal((await res.json()).error.code, "no_source_pdf"); + } finally { + images.close(); + } +}); + +test("POST /pdf tags the pages in page order, leaves out a failed page, fills the values, and keeps nothing", async () => { + const s = await serve(); + try { + const res = await s.tag({ values: { "applicant.name": "Ada Lovelace", "applicant.consent": true } }); + assert.equal(res.status, 200); + const body = await res.json(); + assert.equal(body.filename, "permit_tagged.pdf"); + assert.deepEqual(body.report.warnings, [{ code: "duplicate_text_layer", page: 1 }]); + const made = JSON.parse(Buffer.from(body.pdf, "base64").toString("utf8").replace(/^%PDF-fake /, "")); + assert.equal(made.source, "%PDF-1.7 source"); + assert.deepEqual(made.input, { + lang: "fr", + title: "permit", + pages: [{ sourcePage: 1, html: "

one

" }, { sourcePage: 2, html: "

two

" }], + }); + assert.deepEqual(made.values, { "applicant.name": "Ada Lovelace", "applicant.consent": true }); + assert.equal(made.mode, "600", "only this process can read the values file"); + // The values, and the filled PDF, are gone once the answer is sent. + assert.equal(existsSync(made.scratch), false); + assert.equal(dirname(made.scratch).startsWith(s.paths.pdfScratchRoot()), true); + assert.deepEqual(readdirSync(s.paths.pdfScratchRoot()).filter((f) => f.startsWith("pdf-")), []); + const log = readFileSync(s.paths.sessionLog(s.id), "utf8"); + assert.match(log, /"tagged_pdf"/); + assert.match(log, /applicant\.name/, "the field names are logged"); + assert.doesNotMatch(log, /Ada Lovelace/, "the values are not"); + } finally { + s.close(); + } +}); + +test("POST /pdf passes the tagger's refusals on with its code", async () => { + const s = await serve(); + try { + const bad = await s.tag({ values: { "unknown.field": "x" } }); + assert.equal(bad.status, 400); + const { error } = await bad.json(); + assert.deepEqual([error.code, error.message], ["bad_value", "No field is named unknown.field."]); + assert.equal((await s.tag({ values: ["x"] })).status, 400); + // A crash inside the tagger can quote a value. The client sent it, so it may see it + // again, but the log does not keep it. + const crash = await s.tag({ values: { crash: "Jane Doe" } }); + assert.equal((await crash.json()).error.code, "internal_error"); + const log = readFileSync(s.paths.sessionLog(s.id), "utf8"); + assert.match(log, /"tagged_pdf_failed".*"bad_value".*No field is named unknown\.field/); + assert.doesNotMatch(log, /Jane Doe/); + } finally { + s.close(); + } + const locked = await serve({ source: "%PDF ENCRYPTED" }); + try { + const res = await locked.tag({}); + assert.equal(res.status, 422); + assert.equal((await res.json()).error.code, "encrypted"); + assert.equal((await locked.fields()).status, 422); + } finally { + locked.close(); + } +}); + +test("POST /pdf waits for the finished document and stops a run that takes too long", async () => { + const running = await serve({ status: "running" }); + try { + assert.equal((await running.tag({})).status, 409); + } finally { + running.close(); + } + const slow = await serve({ timeout: 0.5 }); + try { + const res = await slow.tag({ values: { slow: true } }); + assert.equal(res.status, 504); + assert.equal((await res.json()).error.code, "timeout"); + assert.deepEqual(readdirSync(slow.paths.pdfScratchRoot()).filter((f) => f.startsWith("pdf-")), []); + } finally { + slow.close(); + } +}); + +test("two tagger runs at once, across both routes, and a third is told to wait", async () => { + const s = await serve(); + const slow = [s.tag({ values: { slow: true } }), s.tag({ values: { slow: true } })]; + try { + await new Promise((r) => setTimeout(r, 300)); + for (const res of [await s.tag({}), await s.fields()]) { + const body = await res.json(); + assert.equal(res.status, 503); + assert.equal(res.headers.get("retry-after"), "10"); + assert.equal(body.error.code, "busy"); + } + for (const res of await Promise.all(slow)) assert.equal(res.status, 200); + const after = await s.fields(); + await after.body?.cancel(); + assert.equal(after.status, 200, "the count goes back down"); + } finally { + await Promise.allSettled(slow.map(async (p) => (await p).body?.cancel())); + s.close(); + } +}); + +test("startup removes the scratch a killed process left, and nothing else", () => { + const root = mkdtempSync(join(tmpdir(), "iris-scratch-")); + try { + mkdirSync(join(root, "pdf-abc123")); + writeFileSync(join(root, "pdf-abc123", "values.json"), "{}"); + mkdirSync(join(root, "ses_keep")); + assert.equal(clearPdfScratch(root), 1); + assert.deepEqual(readdirSync(root), ["ses_keep"]); + assert.equal(clearPdfScratch(join(root, "missing")), 0); + } finally { + rmSync(root, { recursive: true, force: true }); + } + // And startup runs it on the root the routes use, before it listens. + const index = readFileSync(join(dirname(FAKE), "..", "..", "src", "index.ts"), "utf8"); + const call = index.indexOf('clearPdfScratch(join(cfg.storage.data_dir, "tmp"))'); + assert.ok(call > 0 && call < index.indexOf("app.listen(")); +});