EBSCO transformer rejects 54 records on valid MARC date conventions (008 type 'q', partial years)

[kenoir]

Found by the #6464 comparison; same family as the transformer strictness rejects in #6472. A good first pipeline issue: the failing code, the failing records, and the verification path are all identified below.

What is wrong

54 EBSCO records fail the Axiell/EBSCO Python transformer deterministically and therefore have no works in the 2026-07-03 pipeline. Production still has works for them because they were transformed by the previous Scala transformer, which tolerated these values. The records themselves are live in the EBSCO adapter store and re-fail identically on every run, most recently in the changeset redrive of c2dce996-824c-11f1-afb6-0a58a9feac02 (job replay-6464-ebsco-c2dce996, 2026-08-03), which is also what fired the ebsco-transformer-failures-2026-07-03 alarm that day.

Two failure classes:

1. MARC 008 date type q is unhandled. catalogue_graph/src/adapters/transformers/ebsco/parsers/field008.py (maximal_date_range) handles date types n | c u s r t d m and raises NotImplementedError for anything else. q is a valid MARC 21 code meaning "questionable date: between date 1 and date 2", so the reasonable implementation treats it like the d/m range case. The method's docstring is doctest-driven, so add a q example there.
2. Partial or messy years crash the period parser. catalogue_graph/src/adapters/transformers/ebsco/parsers/period.py builds date ranges with int(year) and raises on values that occur in this data, such as 182- (a MARC convention for the 1820s), - 1792, and 1742 - 44. The parser needs to recover what it can (for example treat 182- as the decade) or degrade to a label-only date rather than failing the whole record.

How to work on it

All the relevant code is Python under catalogue_graph/. Install uv, then from catalogue_graph/:

uv sync
uv run pytest tests/adapters/transformers/ebsco --no-cov -q

That subset covers the EBSCO transformer and passes before any changes. Keep the behaviour of valid input unchanged: the point is tolerating these shapes, not loosening date parsing generally.

Reproducing the failures without AWS

The date parsing is pure functions. The unhandled 008 date type reproduces in one line:

uv run python -c "from adapters.transformers.ebsco.parsers.field008 import Field008; Field008('||||||q17951798').maximal_date_range"
# NotImplementedError: unexpected MARC008 date type value: q

To run a whole record through the real transformation, drive the work builder directly. Save this as repro.py in catalogue_graph/ and run uv run python repro.py:

from datetime import UTC, datetime

from adapters.transformers.builders.ebsco_work_builder import EbscoWorkBuilder
from utils.marc import parse_single_marc_record

xml = """<record>
  <leader>00000nam a2200000   4500</leader>
  <controlfield tag='001'>ebs28825414e</controlfield>
  <controlfield tag='008'>940503q17951798enk     o     000 0 eng c</controlfield>
  <datafield tag='245' ind1='0' ind2='0'><subfield code='a'>A title</subfield></datafield>
</record>"""

work = EbscoWorkBuilder(parse_single_marc_record(xml), datetime.now(UTC)).transform_work()
print(work)

The 008 here is the real one from ebs28825414e, so this currently raises the NotImplementedError above; after the fix it prints the transformed SourceWork. Swap in content fetched from the adapter store to reproduce the period parser failures; the next section shows how to get it.

The int() error messages in the table below show the fragment that reached int() after the period parser's preprocess and crack steps, not the raw subfield contents. ebs144516e fails with int('182-') but its actual 260$c is 1825-[19--?], and the int('1742 - 44') in ebs28836058e comes from M,DCC,XLII-M,DCC,XLIV. [1742-44]. Fetch the real record before writing fixtures rather than reverse-engineering inputs from the error messages.

Querying the real records from the adapter store

Use catalogue_graph/notebooks/adapter_source_tables.ipynb. It loads the EBSCO, Axiell and FOLIO Iceberg adapter tables, fetches a record by id, pretty-prints its MARC XML, and runs the matching transformer over it in-process.

1. Log in with aws sso login --profile platform-developer. The notebook sets AWS_PROFILE=platform-developer, which the S3 Tables REST API needs.
2. From catalogue_graph/, run uv run --with jupyter jupyter lab, or open the notebook in your editor against the project .venv; ipykernel is already a dev dependency.
3. In the cell that picks an example record, uncomment the override and set example_id = "ebs144516e". The example_table_name already defaults to the EBSCO table.
4. The next cells display the raw MARC XML and transform it. The transformer catches per-record exceptions and logs them as "Error transforming record" instead of raising; that same behaviour is what turns these records into failure_count in the pipeline.

Leave USE_REST_API_TABLE = True and ALLOW_REMOTE_TABLE_MUTATIONS = False as they are: reads are safe, and the destructive helpers further down the notebook are guarded behind that flag.

Where the tests live

• The parsers document their behaviour with doctests in field008.py and period.py. Run them with uv run pytest --doctest-modules src/adapters/transformers/ebsco/parsers --no-cov. They are not part of the default uv run pytest run, so new behaviour needs a doctest example and a case in the regular tests too.
• tests/adapters/transformers/ebsco/unit/test_production.py asserts on the 008 fall-back and date ranges, using the marc_record fixture from unit/conftest.py to build minimal pymarc records. Real 008 and 260$c values from the failing records make good cases here.
• tests/adapters/transformers/ebsco/test_transformer.py runs the transformer end to end against a local Iceberg table with raw MARC XML strings as input. It is a good template for a whole-record regression test using one of the failing records.

How to verify the fix in the pipeline

After merge and deploy, re-run the transformer for the changeset that contains all 54 records (any platform engineer can run this, or ask in the channel):

AWS_PROFILE=platform-developer aws stepfunctions start-execution \
  --region eu-west-1 \
  --state-machine-arn arn:aws:states:eu-west-1:760097843905:stateMachine:transformer-2026-07-03 \
  --name replay-ebsco-date-fix-verification \
  --input '{"detail": {"transformer_type": "ebsco", "job_id": "replay-ebsco-date-fix", "changeset_ids": ["c2dce996-824c-11f1-afb6-0a58a9feac02"]}}'

The execution output reports success_count and failure_count: before the fix the run ends with 54 failures, after it it should end with 0, and the 54 records should then exist in works-source-2026-07-03 (spot-check Work[ebsco-alt-lookup/ebs144516e]).

The 54 failing records

Record	Error
ebs144516e	invalid literal for int() with base 10: '182-'
ebs28825414e	unexpected MARC008 date type value: q
ebs28825932e	unexpected MARC008 date type value: q
ebs28830589e	unexpected MARC008 date type value: q
ebs28830622e	invalid literal for int() with base 10: '- 1792'
ebs28835937e	invalid literal for int() with base 10: '- 1794 - -'
ebs28836058e	invalid literal for int() with base 10: '1742 - 44'
ebs28836516e	unexpected MARC008 date type value: q
ebs28836743e	unexpected MARC008 date type value: q
ebs28837484e	unexpected MARC008 date type value: q
ebs28838441e	unexpected MARC008 date type value: q
ebs28840993e	unexpected MARC008 date type value: q
ebs28841900e	invalid literal for int() with base 10: '- 1795 - -'
ebs28841925e	unexpected MARC008 date type value: q
ebs28842707e	unexpected MARC008 date type value: q
ebs28844394e	unexpected MARC008 date type value: q
ebs28845289e	unexpected MARC008 date type value: q
ebs28845635e	invalid literal for int() with base 10: '1746 - 1747'
ebs28845878e	unexpected MARC008 date type value: q
ebs28846763e	unexpected MARC008 date type value: q
ebs28848675e	unexpected MARC008 date type value: q
ebs28849323e	unexpected MARC008 date type value: q
ebs28849684e	unexpected MARC008 date type value: q
ebs28851700e	unexpected MARC008 date type value: q
ebs28852745e	unexpected MARC008 date type value: q
ebs28864322e	unexpected MARC008 date type value: q
ebs28864387e	unexpected MARC008 date type value: q
ebs28864493e	unexpected MARC008 date type value: q
ebs375800e	invalid literal for int() with base 10: '1788 - 1789'
ebs557428e	invalid literal for int() with base 10: '186-'
ebs557433e	invalid literal for int() with base 10: '185-'
ebs557434e	invalid literal for int() with base 10: '183-'
ebs557436e	invalid literal for int() with base 10: '185-'
ebs557439e	invalid literal for int() with base 10: '188-'
ebs557444e	invalid literal for int() with base 10: '185-'
ebs557445e	invalid literal for int() with base 10: '183-'
ebs557446e	invalid literal for int() with base 10: '183-'
ebs557448e	invalid literal for int() with base 10: '182-'
ebs557451e	invalid literal for int() with base 10: '188-'
ebs557458e	invalid literal for int() with base 10: '185-'
ebs557460e	invalid literal for int() with base 10: '187-'
ebs557465e	invalid literal for int() with base 10: '182-'
ebs557468e	invalid literal for int() with base 10: '189-'
ebs557470e	invalid literal for int() with base 10: '189-'
ebs557479e	invalid literal for int() with base 10: '178-'
ebs557484e	invalid literal for int() with base 10: '184-'
ebs557487e	invalid literal for int() with base 10: '187-'
ebs557488e	invalid literal for int() with base 10: '183-'
ebs557489e	invalid literal for int() with base 10: '185-'
ebs557490e	invalid literal for int() with base 10: '184-'
ebs557491e	invalid literal for int() with base 10: '188-'
ebs557493e	invalid literal for int() with base 10: '184-'
ebs557501e	invalid literal for int() with base 10: '182-'
ebs557503e	invalid literal for int() with base 10: '188-'