Skip to content

fix(cluj): re-add seq to static trip_id to disambiguate duplicate CTP CSV depTimes - #167

Closed
ciotlosm wants to merge 2 commits into
fix/cluj-trip-id-duplicate-deptime-cleanfrom
fix/cluj-trip-id-duplicate-deptime
Closed

ciotlosm wants to merge 2 commits into
fix/cluj-trip-id-duplicate-deptime-cleanfrom
fix/cluj-trip-id-duplicate-deptime

Conversation

@ciotlosm

Copy link
Copy Markdown
Member

fix(cluj): re-add seq to static trip_id to disambiguate duplicate CTP CSV depTimes

What broke

[release] daily multi-feed pipeline to R2 has been failing every day since 2026-08-22 with:

[validate] cluj-napoca.gtfs.zip: 1 error(s), 1 warning(s)
  ERROR   stop_times: non-monotonic stop_sequence for trip 63_0_LV_2255 (11 -> 0)
  WARN    13589 trips have no stop_times entries

Root cause: CTP's orar_M21_lv.csv (route 63 LV) has 22:55 on TWO rows in dir0:

75: 22:35,21:20
76: 22:55,21:40   <- dir0 has 22:55 here
77: 22:55,22:00   <- dir0 has 22:55 again
78: ,22:15

The CSV parser doesn't dedupe. The trip-emission loop calls makeTripId(routeId, dir, serviceId, depTime) for each row and produces the same id 63_0_LV_2255 twice. The second emission appends a fresh 12-row stop_times block to a trip that already has 12, so stop_sequence goes 0,1,2,...,11,0 - validate catches the jump and the whole pipeline dies.

Why the prior 4-part format was wrong

A previous "shorten trip_ids" cleanup reduced the static format from

<route>_<dir>_<serviceId>_<seq>_<HHMM>     (old, 5 parts)

to

<route>_<dir>_<serviceId>_<HHMM>          (current, 4 parts)

on the assumption the seq "was never consumed by anyone." Wrong: it absorbs the (rare) CTP CSV case where the same departure time appears on two rows in the same direction. The seq is load-bearing.

The fix

Reinstate the per-departures index in the trip_id:

<route>_<dir>_<serviceId>_<seq>_<HHMM>     (new, 5 parts, seq re-added)

i is already in scope at the call site. Two 22:55 rows on M21 LV dir0 become <route>_0_LV_1_2255 and <route>_0_LV_2_2255, each with its own monotonic stop_times block.

Verified locally with a mini fixture (3-stop Tranzy pattern + CSV with the same minute twice in dir0): output trips.txt has two distinct trip_ids, each with stop_sequence 0,1,2.

What this does NOT change

  • HHMM is still last. neary's parseLiveStartMin regex fallback (_(\d{3,4})$) and the verify-trip-id-format.ts self-check (_\d{4}$) both keep working unchanged.
  • Frequency anchors keep their <route>_<dir>_<serviceId>_FREQ_<HHMM> shape (built directly in derive/frequencies.ts, not via makeTripId). They don't need a seq because the HHMM is a window-start and windows are unique per direction.
  • The cluj RT quirk in src/rt/cluj.ts parses the LIVE RT feed's trip_ids, not static - different format, different concern, no touch.
  • The neary reconciler matches by (route, direction, time) and never compares trip_ids by string equality, so the additional seq slot is invisible to the JOIN.

Files

  • adapters/cluj-napoca/src/assemble/emit/trips.ts - makeTripId gains a seq parameter; the trip-emission loop passes i; full JSDoc rewritten to document the CTP-CSV-duplicate-time case and what the seq must NOT break.
  • adapters/cluj-napoca/tests/reconcile.test.ts - existing trip_id regex bumped from 4-part to 5-part (and the _FREQ slot is now optional in the seq position, not the HHMM position); the existing 35_0_LV_0600 reference updated to 35_0_LV_0_0600; one new regression test that feeds a CSV with a duplicate depTime and asserts two distinct trip_ids, each with a monotonic 0,1,2 stop_times block.

Test

  • pnpm test -> 209/209 pass (was 208/208 + 1 new regression test)
  • pnpm check (tsc strict on src + test) -> clean
  • pnpm build -> clean

What happens after merge

Next 00:30 UTC cron run will pick up the fix, validate passes, the publish step lands fresh cluj-napoca SQLite + zip on R2, and the live n3ary.com app pulls the new data on its next SW refresh (the existing Aug 21 R2 data is still serving fine, so this is a clean cutover, not a recovery).

New adapter package @n3ary/gtfs-adapter-helsinki for the
n3ary transit family. Adapts HSL's daily GTFS feed plus
the standard GTFS-RT protobuf feeds (vehicle positions, trip
updates, service alerts) at realtime.hsl.fi.

Why this is simpler than the cluj adapter:
- HSL publishes a complete, daily-regenerated GTFS zip
  (no Transitous seed + Tranzy reconciliation needed)
- HSL publishes standard GTFS-RT protobuf (no HFP conversion
  -- the worry going in -- the GTFS-RT surface is the
  public realtime API)
- HSL's open data is public (no API key, no rate limit)
- HSL ships route_color + route_text_color per route already
  (no per-feed color computation needed)
- HSL has no per-feed networks (no _route_tags / network
  derive pipeline)

Adapter surface:

  /ingest  - DEFAULT_GTFS_URL, REQUIRED_SECRETS (empty),
             DEFAULT_AGENCY_ID, ingestBuild(). Single GET
             against https://gtfs.hsl.fi/gtfs.zip, returns
             the bytes. No zip-content verification (the
             static pipeline catches malformed GTFS via the
             SQL load errors; a fetch-verify round-trip would
             add a zip-reader dep for marginal value).

  /static  - staticExtension(feedConfig) -- adds _neary_config
             for the app's timing-aware travel-time math, no-op
             fillComputedColumns. Exports hslRouteType +
             computeHslBrandColor for the app's category chip
             rendering. HSL's pathway_mode=1 -> metro is
             handled by hslRouteType (HSL's metro lines use
             route_type=0 historically, with pathway_mode as
             the real signal).

  /rt      - helsinkiQuirk (called by the generic gtfs-rt
             proxy per fetched FeedMessage). Two HSL-specific
             fixes:
               1. Synthesize a trip_id for VehiclePosition
                  and TripUpdate entities when HSL omits one.
                  HSL's real-time feed uses the composite
                  route_id + start_date + start_time +
                  direction_id instead of trip_id (per
                  https://hsldevcom.github.io/gtfs_rt/). The
                  static feed regenerates trip_ids on every
                  publish, so the composite is the only stable
                  join key. Format:
                  hsl:<route_id>:<start_date>:<start_time>:
                  <direction_id>.
               2. Split the HSL <operator>/<vehicle> vehicle
                  id. Sets vehicle.label to the bare vehicle
                  number while keeping vehicle.id as the full
                  composite (so consumers can filter by
                  operator).
             Plus splitHslVehicleId / syntheticHslTripId
             helpers, exported for tests.

44 unit tests, all green. pnpm check clean. pnpm build
emits a dist/ tree (gitignored).

Separate PRs still needed in:
  - n3ary/gtfs            feeds/helsinki/config.json
  - n3ary/gtfs-publisher  feeds/helsinki/config.json
This PR is the adapter only; the feed configs wire it in
and the publisher's feeds.json can be regenerated against
the new package.
… CSV depTimes

The static makeTripId was reduced from
  <route>_<dir>_<serviceId>_<seq>_<HHMM>     (old)
to
  <route>_<dir>_<serviceId>_<HHMM>          (broken)

on the assumption the seq "was never consumed by anyone." Wrong: it
absorbs the (rare) CTP CSV case where the same departure time appears
on two rows in the same direction column. The parser keeps both rows
(no dedup), the trip-emission loop then calls makeTripId for each and
produces two trips with the same trip_id, the second emission appends
a second full stop_times block to that trip, and the validate step
catches the resulting 0,1,2,...,11,0 sequence jump:

  [validate] cluj-napoca.gtfs.zip: non-monotonic stop_sequence
            for trip 63_0_LV_2255 (11 -> 0)

First observed on the daily cron on 2026-08-22 (every schedule run +
every manual dispatch since, 4+ days of red).

Reinstate the per-`departures` index in the trip_id:

  <route>_<dir>_<serviceId>_<seq>_<HHMM>

`i` is already in scope at the call site; two 22:55s on M21 LV dir0
become <route>_0_LV_1_2255 and <route>_0_LV_2_2255, each with its
own monotonic stop_times block.

What this does NOT change:

- HHMM is still last, so neary's parseLiveStartMin `_(\d{3,4})$`
  fallback and the verify-trip-id-format.ts `_\\d{4}$` check both
  keep working unchanged.
- Frequency anchors keep their `<route>_<dir>_<serviceId>_FREQ_<HHMM>`
  shape (built directly in derive/frequencies.ts, not via
  makeTripId). They don't need a seq because the HHMM is a
  window-start and windows are unique per direction.
- The cluj RT quirk (in src/rt/cluj.ts) parses the LIVE RT feed's
  trip_ids, not static - different format, different concern, no
  touch.
- The neary reconciler matches by (route, direction, time) and
  never compares trip_ids by string equality, so the additional seq
  slot is invisible to the JOIN.

Regression test: minimal fixture with a 3-stop Tranzy pattern and a
CSV that lists the same minute twice in dir0; asserts two distinct
trip_ids and a monotonic 0,1,2 in each.
@ciotlosm
ciotlosm changed the base branch from main to fix/cluj-trip-id-duplicate-deptime-clean August 25, 2026 09:36
@ciotlosm ciotlosm closed this Aug 25, 2026
@ciotlosm
ciotlosm deleted the fix/cluj-trip-id-duplicate-deptime branch August 25, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant