Implement standard eDiscovery (ESI) production format (TIFF + text + metadata + load files)

[markkerzner]

Implement a standard eDiscovery (ESI) production format

Jonathan's item. We have a representative Stipulated Order Re: ESI (N.D. Cal., Blockchain Innovation LLC v. Franklin Resources, No. 4:21-cv-08787-HSG) — a typical court-ordered ESI production protocol. FreeEed should be able to produce in exactly this format so its output drops straight into any review platform and satisfies a standard production order.

This is core to FreeEed's mission and the 2027 Vision (legal/eDiscovery + disclosure): give users a free tool whose productions are court-ready.

Production format requirements (from the order)

Images

• Single-page, black & white, Group IV TIFF, ≥ 300 dpi, 8.5×11 (unless doc needs otherwise), page orientation preserved.
• Each image Bates-branded and named with its page-level Bates number. Bates = unique across production, 9 numeric digits, zero-padded, sequential within a doc, with a producing-party prefix + single dash (e.g. ABC-000000123).

Text

• One multipage .TXT per document (not per page), named with the doc's beginning Bates number.
• Extracted from native; OCR when redacted or hard-copy.

Native files

• Produce natively: spreadsheets (.xls/.csv/Access), audio/video (.wav/.mpeg). Provide a single-page TIFF placeholder ("Document Produced in Native") Bates-branded; native named by that Bates.
• TIFF on request when redaction needed; error/corrupt/password files → native.

Load files

• A data load file (delimited, e.g. Concordance .DAT) and an image load file (e.g. Opticon .OPT) accompanying every production.

Metadata — produce the Exhibit A field set for e-files and Exhibit B for hard copy:

• Exhibit A (e-files): PRODBEG, PRODEND, PRODBEGATT, PRODENDATT, ATTACH_COUNT, PRODVOL, PRODPARTY, CUSTODIAN, DUPCUSTODIAN, DOCTYPE, FROM, TO, CC, BCC, SUBJECT, DATE_SENT, TIME_SENT, LINK, FILE_EXTEN, FILE_NAME, FILE_PATH, AUTHOR, DATE_CREATED, DATE_MODIFIED, REDACTION, CONFIDENTIALITY, HASH, PASSWORD.
• Exhibit B (hard copy): PRODBEG, PRODEND, PRODVOL, PRODPARTY, CUSTODIAN, DOCTYPE, REDACTION, CONFIDENTIALITY.
• Date format MMDDYYYY, time HH:MM:SS; HASH = MD5 or SHA-1.

Family relationships

• Preserve parent/child (email ↔ attachments): sequential Bates within a family + accurate attachment ranges (PRODBEGATT/PRODENDATT) in metadata.

De-duplication

• Exact-dup removal by MD5, on message/family basis, with all custodians who held a copy listed (CUSTODIAN/DUPCUSTODIAN). Optional email threading; disclose method on request.

Redactions

• Labeled redaction boxes with basis (e.g. "Redacted for Privilege"); no black-box. Redacted text via OCR. Privileged-family placeholders ("Document Withheld as Privileged").

Production media / volumes

• Output organized into volumes with sequential volume numbers; deliverable on USB or via secure transfer; cover letter with volume + Bates ranges.

Gap analysis (starting point)

FreeEed already has pieces to build on: metadata/ColumnMetadata.java, MetadataWriter.java, the LoadDiscovery/ load-file code, and a UPI numbering scheme (UPIFormat = "UPI_00000"). Work is to (a) map/extend the metadata columns to the Exhibit A/B field names, (b) implement compliant Bates (prefix + 9-digit zero-pad) alongside/instead of UPI, (c) emit .DAT + .OPT load files, (d) Group IV TIFF rendering + placeholders for native, and (e) family/dedup/redaction handling per spec.

Acceptance (first cut)

• Configurable Bates prefix; 9-digit zero-padded sequential numbering; image filenames = Bates.
• Single-page Group IV TIFF (300 dpi) per page, Bates-branded; native placeholders.
• Multipage .TXT per document (extract or OCR), named by beginning Bates.
• Native production for spreadsheets/AV per rules.
• .DAT (data) + .OPT (image) load files.
• Metadata export matching Exhibit A (e-files) and Exhibit B (hard copy), correct date/time/hash formats.
• Family ranges + MD5 de-dup with custodian roll-up.
• Volume organization.
• A selectable "Standard ESI production" profile so a user produces a compliant set in one step.

Reference

Source spec: a public court order (GPO-authenticated, GovInfo USCOURTS-cand-4_21-cv-08787-11). We should capture our own abstracted production-spec doc rather than commit the case PDF (see discussion in the issue thread).

[markkerzner]

Spec doc written and committed: docs/specs/esi-production-format.md — the abstracted, court-ready production format (Bates rules, Group IV TIFF, multipage text, native handling, .DAT/.OPT load files, the Exhibit A/B metadata tables, family/dedup/redaction, and volume layout). It's written in FreeEed's own terms and cites the public GPO-authenticated order as the reference, so we don't embed the case PDF.

[markkerzner]

Note: FreeEed Viewer (#550) reuses this production format as its read-only base

Decision recorded on #550: the portable FreeEed Viewer does not define its own format — it consumes the standard ESI production specified here as its immutable base (documents, .TXT, .DAT metadata, Bates/families), and adds only a Lucene search index and a writable review.sqlite notes/tags overlay on top.

Implication for this issue: implementing the ESI production format once also delivers the Viewer's data layer. Worth keeping the metadata/.DAT output and Bates scheme stable and well-documented, since the Viewer's catalog is derived from them.