Harden the feed download against paging, empty and pinless responses

[baz8080]

Ingest fixes from the retroactive review (PR F of the set). This PR changes no published number on the current data.

Changes (src/uisce/pipeline.py)

• Paging by key. download_cases pages with OBJECTID > <last seen> instead of resultOffset. With offsets, two things went wrong, and both stayed inside the 1% count tolerance:

  • one case deleted during the download pushed a live case off the next page, so that case was stamped vanished;
  • a server maxRecordCount below 1,000 silently dropped the difference on every page.

  Every page must be strictly ascending from the last id, or the build fails with the page named. That also covers a server that ignores the filter and re-serves a page, and a feature with no OBJECTID.

• Features with no pin. ArcGIS leaves out geometry for a null shape, or writes an empty point as "NaN", and one such feature used to crash every build. The coordinates are NOT NULL and key the geocode cache, so allowing NULL would mean a non-additive schema change. Instead, a new function, restore_pins, handles these cases:

  • a case the database already knows keeps the pin it last had;
  • a case that has never been pinned is left out, with a ::warning:: on the Actions run, until the feed pins it.

• Empty download. check_download_complete refuses an empty download while the database holds rows that aren't vanished. The vanish stamp touches every such row, closed ones included: 4,306 on the release, of which 498 are open. A count of 0 alongside a full download still builds.

• backfill_reduced_pressure now uses COALESCE(reduced_pressure, 0) = 0. None of the 7 rows with a NULL flag matches today.

• Read-only database access. It goes through one context manager that always closes the connection, probes the columns once, and builds the URI with as_uri().

Review

First round, high effort, 8 findings.

• Two reproduced crashes:
  • a "NaN" point;
  • a database that has no cases table yet.
• An empty-feed guard that counted only open rows.
• Page order. Paging trusted the server's order without checking, and the tests couldn't tell if orderByFields was removed.
• Never-pinned cases were only a log line.

Second round, medium effort, 10 findings.

• Connections left open when a query raised.
• The refusal was keyed on the count instead of the download.
• Missing tests:
  • both directions of the refusal;
  • a server ignoring the filter (without the check, that loops forever);
  • a missing OBJECTID.
• Smaller fixes: a doubled column probe and URI escaping.
• CLAUDE.md rows needed amending and adding.

All are addressed.

Notes

• notes/data-quality.md: a dated paragraph in the vanished-cases section, and "A feature with no pin (2026-09-24)".
• CLAUDE.md: the vanished_at row is amended, and there is a new row for features with no pin.

Tests

• New tests in tests/test_pipeline.py, each failing before its fix:
  • LiveFeed fakes a server that deletes a row mid-download, one that caps pages, one that ignores orderByFields, one with unsorted storage, and one that re-serves the first page (it fails fast instead of looping).
  • Both directions of the empty-download refusal.
  • A feature with no geometry, and a "NaN" point.
  • TestRestorePins, including a database with no cases table.
  • A NULL pressure flag.
• uv run ruff check and uv run pytest pass: 702 tests, merged with current main.

🤖 Generated with Claude Code

https://claude.ai/code/session_0168hLW2X3mkV3Jhm26LJSwQ