Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -987,7 +987,15 @@ curl -s -H "$AUTH" "$BASE/sessions/$SID/fields" | jq '.fields[0]'
```

`type` is one of `text`, `checkbox`, `radio`, `combobox`, `listbox`, `button` or `signature`. A PDF with
no form gives `{"fields": []}`. It can answer `404`, `409 no_source_pdf`, `422` for a PDF the tagger
no form gives `{"fields": []}`.

A flat form, such as a scan, has no fields until it is tagged. If its HTML has form controls, Iris
tags it once and lists the fields it made, each with `"created": true`. `POST /pdf` fills them by
those names. This needs iris-pdf with
[equalify-iris-pdf#16](https://github.com/EqualifyEverything/equalify-iris-pdf/pull/16). An older
one makes no fields, so the list is empty.

It can answer `404`, `409 no_source_pdf`, `422` for a PDF the tagger
refuses, or `503 busy`, the same as below.

**Get the tagged PDF:**
Expand Down Expand Up @@ -1063,8 +1071,8 @@ The events worth grepping for have a section each below, and the index is a link
the index when you have a `type` off a log line and want to know what it means; read a section when
you want to know what the field it names is for and what it costs.

**The index is the whole log.** `src/` emits **125** event types and every one of them has a section
below — **117** sections, because a few cover two or three events that are only read together. So a
**The index is the whole log.** `src/` emits **127** event types and every one of them has a section
below — **118** sections, because a few cover two or three events that are only read together. So a
`type` you cannot find here is not one the index skipped: it is a misread line, or a name `src/` no
longer emits.

Expand Down Expand Up @@ -1191,6 +1199,7 @@ emits fails it too.
| [`contribution_failed`](#contribution_failed) | The filing step threw, **after** `run_complete` |
| [`run_failed`](#run_failed) | The run threw, so there is **no document** |
| [`tagged_pdf` / `tagged_pdf_failed`](#tagged_pdf--tagged_pdf_failed) | A tagged PDF was made, or could not be |
| [`tagged_pdf_fields` / `tagged_pdf_fields_failed`](#tagged_pdf_fields--tagged_pdf_fields_failed) | A flat form was tagged to list its fields, or could not be |
| [`form_fields` / `page_fields` / `page_fields_missing`](#form_fields--page_fields--page_fields_missing) | The PDF's form field names were read, shown to a page, or not used |
| [`calibrate_call_failed`](#calibrate_call_failed) | One calibration verifier call threw — a tool's line, never a run's |

Expand Down Expand Up @@ -4383,6 +4392,13 @@ A [tagged PDF](#get-a-tagged-pdf-optional) was made, or the tagger refused. `ms`
never logged. `tagged_pdf_failed` has the tagger's `code` and `error`. An `internal_error`'s
message is left out, because it may quote a value.

### `tagged_pdf_fields` / `tagged_pdf_fields_failed`

`GET /fields` tagged a flat form to list the fields it gets. `ms` is how long it took.
`tagged_pdf_fields` has `created`, how many fields were made. `tagged_pdf_fields_failed` has the
tagger's `code`. A made list is reused until the extracted pages change. A failure is reused for a
minute after it ends, so a failing form writes at most one line a minute while it is asked.

### `form_fields` / `page_fields` / `page_fields_missing`

With tagged PDFs on, the upload reads the PDF's form fields, and the page agent is asked to name
Expand Down
12 changes: 7 additions & 5 deletions public/demo.html
Original file line number Diff line number Diff line change
Expand Up @@ -167,8 +167,8 @@ <h2 id="result-h" tabindex="-1">Your accessible HTML</h2>
</div>

<!-- Only where this deployment makes tagged PDFs (GET /v1/limits `tagged_pdf`) and
the upload was one PDF. The fields come from that PDF (GET .../fields), so the
form is built when the result is shown. -->
the upload was one PDF. The fields come from that PDF, or for a flat form from
tagging it (GET .../fields), so the form is built when the result is shown. -->
<div id="pdf-part" hidden>
<h3 id="pdf-h">Get your PDF back, tagged</h3>
<p>The same PDF you uploaded, tagged so a screen reader can read it in order. The pages look the same. The tags come from Iris's first reading of each page, before review, so the HTML above is the more accurate version.</p>
Expand Down Expand Up @@ -440,14 +440,16 @@ <h2 class="visually-hidden">About</h2>
function renderFields(doc, list, fields) {
const el = (tag, props = {}) => Object.assign(doc.createElement(tag), props);
const req = (f) => (f.required ? ' (required on the form)' : '');
// A field Iris made for a flat form is named like `home-address-2`. Read it as words.
const label = (f) => (f.created ? f.name.replace(/[-_]+/g, ' ').replace(/^./, (c) => c.toUpperCase()) : f.name) + req(f);
let n = 0;
for (const f of fields) {
if (f.readonly) continue;
const id = 'pdf-f' + n;
let box;
if (f.type === 'text' || (f.type === 'combobox' && f.editable)) {
box = el('div');
box.append(el('label', { htmlFor: id, textContent: f.name + req(f) }));
box.append(el('label', { htmlFor: id, textContent: label(f) }));
const input = el('input', { id, type: 'text' });
if (f.maxlen) input.maxLength = f.maxlen;
if (f.options.length) {
Expand All @@ -459,7 +461,7 @@ <h2 class="visually-hidden">About</h2>
box.append(input);
} else if (f.type === 'radio' || f.type === 'checkbox') {
box = el('fieldset');
box.append(el('legend', { textContent: f.name + req(f) }));
box.append(el('legend', { textContent: label(f) }));
// A chosen radio cannot be unchosen, so "leave it" is a choice of its own. A
// checkbox is asked the same way: the page cannot show whether the PDF has it
// checked, so an unticked box would not say "leave it" or "untick it".
Expand All @@ -474,7 +476,7 @@ <h2 class="visually-hidden">About</h2>
});
} else if (f.type === 'combobox' || f.type === 'listbox') {
box = el('div');
box.append(el('label', { htmlFor: id, textContent: f.name + req(f) }));
box.append(el('label', { htmlFor: id, textContent: label(f) }));
const select = el('select', { id, multiple: f.multiSelect });
if (!f.multiSelect) select.append(el('option', { value: '', textContent: 'Leave as it is' }));
for (const o of f.options) select.append(el('option', { value: o, textContent: o }));
Expand Down
118 changes: 96 additions & 22 deletions src/routes/sessions.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { Router } from "express";
import type { Request, Response, NextFunction } from "express";
import multer from "multer";
import { writeFileSync, readFileSync, existsSync, rmSync } from "node:fs";
import { writeFileSync, readFileSync, existsSync, rmSync, statSync } from "node:fs";
import { extname, join } from "node:path";
import { ulid } from "ulid";
import type { IrisConfig } from "../config.ts";
Expand Down Expand Up @@ -42,7 +42,7 @@ import {
} from "../providers/imageLimits.ts";
import { imageDimensions } from "../util/imageSize.ts";
import { usableFieldName } from "../pipeline/fields.ts";
import { readFields, tagPdf, taggedPdfCommand, tagTimeoutSeconds, TaggedPdfError, type PdfField } from "../util/taggedPdf.ts";
import { createdFields, readFields, tagPdf, taggedPdfCommand, tagTimeoutSeconds, TaggedPdfError, type PdfField, type TagInput } from "../util/taggedPdf.ts";

// 50 MB is a memory bound, not the image limit. It stays well above what an image may
// be (see imageLimits.ts) because a PDF legitimately is: 25 pages of scans is a large
Expand Down Expand Up @@ -145,6 +145,13 @@ function ownedSession(store: Store, id: string, userId: number): SessionRecord |
return s;
}

// Set `key` as the newest entry, and drop the oldest past `max`. A Map keeps insertion order.
export function remember<K, V>(map: Map<K, V>, key: K, value: V, max: number): void {
map.delete(key);
map.set(key, value);
while (map.size > max) map.delete(map.keys().next().value!);
}

// Keep the upload for a tagged PDF later, when this deployment makes them and the upload
// was one PDF on its own. Then the PDF's page N is the session's page N.
export function keepSourcePdf(
Expand Down Expand Up @@ -602,14 +609,98 @@ export function sessionsRouter(cfg: IrisConfig, store: Store): Router {
return true;
}

function readyToTag(s: SessionRecord): boolean {
return (s.status === "ready_for_review" || s.status === "closed") && existsSync(paths.sessionFinalFragments(s.session_id));
}
// The download's name, and the pages as extracted, for the tagger. Order is the PDF's page
// number, because the session is one PDF. A page that failed extraction holds only its
// `@page-failed` note, so it is left out and the tagger reports it untagged.
function tagInput(s: SessionRecord): { base: string; input: TagInput } {
const finalPath = paths.sessionFinalFragments(s.session_id);
const base = existsSync(paths.sessionSourceName(s.session_id))
? readFileSync(paths.sessionSourceName(s.session_id), "utf8").trim() || "document"
: "document";
const { fragments = [] } = JSON.parse(readFileSync(finalPath, "utf8")) as { fragments?: Fragment[] };
const outPath = paths.sessionOutput(s.session_id);
const output = existsSync(outPath) ? readFileSync(outPath, "utf8") : "";
return {
base,
input: {
lang: output.match(/<html\b[^>]*\blang="([^"]+)"/i)?.[1],
title: base,
pages: [...fragments]
.filter((f) => !f.innerHtml.trimStart().startsWith("<!-- @page-failed"))
.sort((a, b) => a.order - b.order)
.map((f) => ({ sourcePage: f.order, html: f.innerHtml })),
},
};
}

// A flat form's created fields, per session. The tagger names them the same way on every
// run, so a list holds until the extracted pages change. A failure (null) holds a minute,
// so a failing form cannot keep the tagger busy. A run in progress is shared. The oldest
// of more than 200 sessions is dropped.
type Created = (PdfField & { created: true })[];
const createdCache = new Map<string, { mtimeMs: number; until: number; fields: Promise<Created | null> }>();
const FAILURE_KEPT_MS = 60_000;
const CACHED_SESSIONS = 200;

async function makeFields(src: { s: SessionRecord; command: string; pdf: string }): Promise<Created | null> {
const id = src.s.session_id;
const started = Date.now();
// An unwritable log must not cost the list it describes.
const note = (write: (log: RunLog) => void) => {
try {
write(new RunLog(paths.sessionLog(id)));
} catch {
// the list is the answer; its log line is not
}
};
try {
const { input } = tagInput(src.s);
if (!input.pages.some((p) => /<(input|select|textarea)\b/i.test(p.html))) return [];
const made = await createdFields(src.command, {
pdfPath: src.pdf,
input,
scratchRoot: paths.pdfScratchRoot(),
timeoutSeconds: tagTimeoutSeconds(cfg),
});
note((log) => log.event("tagged_pdf_fields", { ms: Date.now() - started, created: made.length }));
return made.map((f) => ({ ...f, created: true as const }));
} catch (e) {
const code = e instanceof TaggedPdfError ? e.code : "tagger_failed";
note((log) => log.event("tagged_pdf_fields_failed", { ms: Date.now() - started, code }));
return null;
}
}

// GET /v1/sessions/{id}/fields — the source PDF's form fields, for a fill-in form.
r.get("/:id/fields", async (req: AuthedRequest, res) => {
const src = sourcePdf(req, res);
if (!src) return;
if (tooBusy(res)) return;
tagging++;
try {
res.json({ fields: await readFields(src.command, src.pdf) });
const fields: PdfField[] = await readFields(src.command, src.pdf);
// A PDF with no fields, whose HTML has form controls: a flat form. Tagging gives it
// fields, so list those, marked `created`. If that fails, list none; the download
// says why.
if (!fields.length && readyToTag(src.s)) {
const id = src.s.session_id;
const mtimeMs = statSync(paths.sessionFinalFragments(id)).mtimeMs;
let hit = createdCache.get(id);
if (!hit || hit.mtimeMs !== mtimeMs || Date.now() >= hit.until) {
const entry = { mtimeMs, until: Infinity, fields: makeFields(src) };
remember(createdCache, id, entry, CACHED_SESSIONS);
void entry.fields.then((f) => {
if (f === null) entry.until = Date.now() + FAILURE_KEPT_MS;
});
hit = entry;
}
res.json({ fields: (await hit.fields) ?? [] });
return;
}
res.json({ fields });
} catch (e) {
taggerError(res, e);
} finally {
Expand All @@ -624,8 +715,7 @@ export function sessionsRouter(cfg: IrisConfig, store: Store): Router {
const src = sourcePdf(req, res);
if (!src) return;
const { s } = src;
const finalPath = paths.sessionFinalFragments(s.session_id);
if ((s.status !== "ready_for_review" && s.status !== "closed") || !existsSync(finalPath)) {
if (!readyToTag(s)) {
sendError(res, 409, "invalid_state", "The PDF can be tagged once the session is ready_for_review.");
return;
}
Expand All @@ -639,23 +729,7 @@ export function sessionsRouter(cfg: IrisConfig, store: Store): Router {
const started = Date.now();
tagging++;
try {
const base = existsSync(paths.sessionSourceName(s.session_id))
? readFileSync(paths.sessionSourceName(s.session_id), "utf8").trim() || "document"
: "document";
// The pages as extracted, by page order. Order is the PDF's page number, because
// the session is one PDF. A page that failed extraction holds only its
// `@page-failed` note, so it is left out and the tagger reports it untagged.
const { fragments = [] } = JSON.parse(readFileSync(finalPath, "utf8")) as { fragments?: Fragment[] };
const outPath = paths.sessionOutput(s.session_id);
const output = existsSync(outPath) ? readFileSync(outPath, "utf8") : "";
const input = {
lang: output.match(/<html\b[^>]*\blang="([^"]+)"/i)?.[1],
title: base,
pages: [...fragments]
.filter((f) => !f.innerHtml.trimStart().startsWith("<!-- @page-failed"))
.sort((a, b) => a.order - b.order)
.map((f) => ({ sourcePage: f.order, html: f.innerHtml })),
};
const { base, input } = tagInput(s);
const { pdf, report } = await tagPdf(src.command, {
pdfPath: src.pdf,
input,
Expand Down
19 changes: 18 additions & 1 deletion src/util/taggedPdf.ts
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,23 @@ export async function tagPdf(
command: string,
args: { pdfPath: string; input: TagInput; values: Record<string, unknown>; scratchRoot: string; timeoutSeconds: number },
): Promise<{ pdf: Buffer; report: unknown }> {
return inScratch(command, args, (out, report) => ({ pdf: readFileSync(out), report: JSON.parse(readFileSync(report, "utf8")) }));
}

// The fields a flat form gets when tagged: iris-pdf makes them from the HTML's controls, and
// names them the same way on every run. So tag it with no values and read them off the output.
export async function createdFields(
command: string,
args: { pdfPath: string; input: TagInput; scratchRoot: string; timeoutSeconds: number },
): Promise<PdfField[]> {
return inScratch(command, { ...args, values: {} }, (out) => readFields(command, out));
}

async function inScratch<T>(
command: string,
args: { pdfPath: string; input: TagInput; values: Record<string, unknown>; scratchRoot: string; timeoutSeconds: number },
read: (out: string, report: string) => T | Promise<T>,
): Promise<T> {
mkdirSync(args.scratchRoot, { recursive: true });
const dir = mkdtempSync(join(args.scratchRoot, "pdf-"));
const f = (name: string) => join(dir, name);
Expand All @@ -135,7 +152,7 @@ export async function tagPdf(
} catch (e) {
throw failure(e);
}
return { pdf: readFileSync(f("out.pdf")), report: JSON.parse(readFileSync(f("report.json"), "utf8")) };
return await read(f("out.pdf"), f("report.json"));
} finally {
rmSync(dir, { recursive: true, force: true });
}
Expand Down
15 changes: 14 additions & 1 deletion test/demo-pdf-form.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ function extract(name: string): string {
throw new Error(`unbalanced braces reading ${name} from public/demo.html`);
}

type Field = { name: string; type: string; options: string[]; required?: boolean; readonly?: boolean; maxlen?: number | null; editable?: boolean; multiSelect?: boolean };
type Field = { name: string; type: string; options: string[]; created?: boolean; required?: boolean; readonly?: boolean; maxlen?: number | null; editable?: boolean; multiSelect?: boolean };
const renderFields = new Function(`${extract("renderFields")}; return renderFields;`)() as (doc: Document, list: Element, fields: Field[]) => number;
const fieldValues = new Function(`${extract("fieldValues")}; return fieldValues;`)() as (list: Element) => Record<string, unknown>;
const pdfNotes = new Function(`${extract("pdfNotes")}; return pdfNotes;`)() as (report: unknown) => string[];
Expand Down Expand Up @@ -63,6 +63,19 @@ test("every settable field gets one labelled control, and the rest get none", as
assert.deepEqual(lint.violations, []);
});

test("a field made for a flat form is labelled in words, and sent under its own name", () => {
const doc = new JSDOM('<!DOCTYPE html><html lang="en"><head><title>t</title></head><body><form id="f"></form></body></html>').window.document;
const list = doc.getElementById("f")!;
renderFields(doc, list, [
{ name: "home-address-2", type: "text", options: [], created: true },
{ name: "pay_by", type: "radio", options: ["Cash", "Card"], created: true },
{ name: "applicant.name-x", type: "text", options: [] },
] as Field[]);
assert.deepEqual([...list.querySelectorAll("label, legend")].map((l) => l.textContent).filter((t) => !["Leave as it is", "Cash", "Card"].includes(t!)), ["Home address 2", "Pay by", "applicant.name-x"]);
(list.querySelector('[data-field="home-address-2"] input') as HTMLInputElement).value = "x";
assert.deepEqual(fieldValues(list), { "home-address-2": "x" });
});

test("an untouched form sends nothing, so the PDF keeps what it had", () => {
assert.deepEqual(fieldValues(form().list), {});
});
Expand Down
12 changes: 10 additions & 2 deletions test/fixtures/fake-iris-pdf.mjs
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
#!/usr/bin/env node
// Stands in for the `iris-pdf` command in test/tagged-pdf.test.ts. Same arguments, exit
// codes and one-line errors, and no PDF work. Its "PDF" is JSON of what it was given.
import { readFileSync, statSync, writeFileSync } from "node:fs";
import { appendFileSync, readFileSync, statSync, writeFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { parseArgs } from "node:util";

const [command, ...rest] = process.argv.slice(2);
Expand All @@ -22,7 +23,11 @@ if (a.help) {
const pdf = readFileSync(a.pdf, "utf8");
if (pdf.includes("ENCRYPTED")) fail("encrypted", "The PDF is encrypted.", 1);

if (command === "fields") {
// A "PDF" holding FLAT is a flat form: no fields, until tagging makes one from an <input>.
if (command === "fields" && pdf.includes("FLAT")) {
const made = pdf.startsWith("%PDF-fake ") && JSON.parse(pdf.slice(10)).input.pages.some((p) => p.html.includes("<input"));
console.log(JSON.stringify(made ? [{ name: "full-name", type: "text", page: 1, options: [], required: false, readonly: false, maxlen: null, editable: false, multiSelect: false }] : []));
} else if (command === "fields") {
console.log(JSON.stringify([
{ name: "applicant.name", type: "text", page: 1, options: [], required: true, readonly: false, maxlen: 40, editable: false, multiSelect: false },
{ name: "applicant.consent", type: "checkbox", page: 1, options: ["Yes"], required: false, readonly: false, maxlen: null, editable: false, multiSelect: false },
Expand All @@ -31,6 +36,9 @@ if (command === "fields") {
: []),
]));
} else if (command === "tag") {
// So a test can tell whether a tag ran: one line per run, beside the source.
appendFileSync(join(dirname(a.pdf), "tag-runs.txt"), "tag\n");
if (pdf.includes("TAGFAIL")) fail("text_lost", "Page 1 lost text.", 2);
const values = JSON.parse(readFileSync(a.values, "utf8"));
if ("unknown.field" in values) fail("bad_value", "No field is named unknown.field.", 3);
if ("crash" in values) fail("internal_error", `Cannot set ${values.crash}.`, 3);
Expand Down
Loading
Loading