Skip to content

Commit e71f669

Browse files
committed
feat(worker): auto-download FL NeTEx from the Italian NAP
The CCISS NAP serves the Lazio NeTEx over an unauthenticated HTTP GET (no login, no key), so the fl-gtfs job now fetches it directly each run instead of needing a manually-placed file. NETEX_URL defaults to the RAP Lazio L2 asset (www.cciss.it/.../download/blob/Asset/663391/checkedResource); set NETEX_URL= empty to disable and use GATEWAY_NETEX_PATH. The fetch uses a generous per-request timeout (the NAP endpoint streams slowly) and is fail-soft — a download failure falls back to any existing file. Corrects ADR 0016 (the NAP is not login-gated after all) and records the data's `no_licence_no_contract` NAP terms in DATA_LICENSES (still a launch gate). Proven live: with no placed file, the worker auto-downloaded the current L2 NeTEx from the NAP and built the GTFS — 468 stops, 6 routes, 2,263 trips, 27,638 stop_times.
1 parent a9874ff commit e71f669

7 files changed

Lines changed: 44 additions & 23 deletions

File tree

‎.env.example‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,12 @@ TRENITALIA_REGION=5
3333
# RT_TRIP_UPDATES_URL=https://romamobilita.it/wp-content/uploads/shared/rome_rtgtfs_trip_updates_feed.pb
3434
RT_POLL_SECS=30
3535

36-
# FL NeTEx→GTFS (worker): the official Lazio NeTEx is placed at GATEWAY_NETEX_PATH
37-
# (the CCISS NAP is login-gated), or fetched from NETEX_URL if reachable.
36+
# FL NeTEx→GTFS (worker): auto-downloads the Lazio NeTEx from the Italian NAP
37+
# (CCISS public endpoint, no auth) by default — RAP Lazio L2. Set NETEX_URL=
38+
# (empty) to disable the fetch and use a file placed at GATEWAY_NETEX_PATH, or
39+
# point it at another NAP asset.
40+
# NETEX_URL=https://www.cciss.it/nap/mmtis/public/api/v1/download/blob/Asset/663391/checkedResource
3841
# GATEWAY_NETEX_PATH=/data/netex/trenitalia-fl.netex.xml.gz
39-
# NETEX_URL=
4042

4143
# ─── Observability (operator-local; never phones home) ───────────────────────
4244
ITER_LOG=info

‎CHANGELOG.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -62,9 +62,10 @@ Commit history.
6262
concave-hulls + per-direction platform strips + named exits).
6363
- **Worker** — a graceful-shutdown job scheduler. **FL NeTEx→GTFS** (`fl-gtfs`):
6464
a streaming `quick-xml` parser (over a `flate2` gunzip) converts the official
65-
Lazio NeTEx into a routable GTFS the OTP graph build consumes — proven on the
66-
real ~58 MB CCISS dataset (450 stops, 5 routes, 1,594 trips, 20,617 stop_times,
67-
zero loss; ADR 0016). Plus **GTFS-RT ingestion** (`rt-reliability`): polls
65+
Lazio NeTEx into a routable GTFS the OTP graph build consumes, auto-downloaded
66+
from the Italian NAP (CCISS public endpoint) — proven on the real ~58 MB
67+
dataset (450 stops, 5 routes, 1,594 trips, 20,617 stop_times, zero loss; ADR
68+
0016). Plus **GTFS-RT ingestion** (`rt-reliability`): polls
6869
ATAC's trip-updates feed,
6970
decodes it with a vendored `prost` GTFS-RT subset (no `protoc`), and derives
7071
validated stop-delay events keyed on the stable (route, direction, stop,

‎DATA_LICENSES.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ Sources: `concept doc 29` §4 and
1717
| **GTFS — ATAC / Roma Servizi** (static + realtime) | CC-BY 3.0 | `CC-BY-3.0` | `Roma Servizi per la Mobilità S.r.l.` |
1818
| **GTFS — COTRAL** (Lazio extra-urban bus) | **TBD / undeclared** 🔴 | — | — (must be cleared before public production) |
1919
| **GTFS — COTRAL-FERRO** (Lazio regional railways) | **TBD / undeclared** 🔴 | — | — (must be cleared before public production) |
20-
| **Trenitalia FL** (regional trains, NeTEx via CCISS NAP) | **TBD / undeclared** 🔴 | — | — (must be cleared before public production) |
20+
| **Trenitalia FL** (regional trains, NeTEx via CCISS NAP) | **`no_licence_no_contract`** 🔴 | — | open access via the NAP (EU MMTIS regime) but no explicit reuse/redistribution grant — clear with Regione Lazio before public production |
2121
| **House numbers (civici)** — Overture / ANNCSU | CC-BY 4.0 | `CC-BY-4.0` | double: `Overture / OpenAddresses` **and** `ISTAT / Agenzia delle Entrate — ANNCSU` |
2222
| **Glyphs (Noto Sans)** | SIL OFL 1.1 | `OFL-1.1` | bundle license + copyright with the served font files |
2323
| **Komoot Photon** Italy geocoding dump (derives from OSM) | ODbL 1.0 | `ODbL-1.0` | `© OpenStreetMap contributors` |

‎crates/iter-worker/src/jobs/fl_gtfs.rs‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,14 +33,13 @@ impl Job for FlGtfsBuild {
3333
}
3434

3535
async fn run(&self) -> anyhow::Result<()> {
36-
// Optionally fetch the NeTEx; a failure is a warning (we fall back to a
37-
// previously-placed file).
38-
if !exists(&self.netex_path).await
39-
&& let Some(url) = &self.netex_url
40-
{
36+
// When a URL is set, refresh the NeTEx on every run (the daily cadence
37+
// is the refresh) — a failure is a warning, and we fall back to a
38+
// previously-placed/fetched file.
39+
if let Some(url) = &self.netex_url {
4140
tracing::info!(url, "fl-gtfs: fetching NeTEx");
4241
if let Err(e) = download(&self.http, url, &self.netex_path).await {
43-
tracing::warn!(error = %e, "fl-gtfs: NeTEx download failed");
42+
tracing::warn!(error = %e, "fl-gtfs: NeTEx download failed; using existing file");
4443
}
4544
}
4645

@@ -79,7 +78,14 @@ async fn download(client: &reqwest::Client, url: &str, dest: &Path) -> anyhow::R
7978
let mut tmp = dest.to_path_buf().into_os_string();
8079
tmp.push(".tmp");
8180
let tmp = PathBuf::from(tmp);
82-
let mut resp = client.get(url).send().await?.error_for_status()?;
81+
// The NAP `checkedResource` is large and slow to stream; override the
82+
// client's short default with a generous per-request timeout.
83+
let mut resp = client
84+
.get(url)
85+
.timeout(Duration::from_secs(300))
86+
.send()
87+
.await?
88+
.error_for_status()?;
8389
let mut file = tokio::fs::File::create(&tmp).await?;
8490
while let Some(chunk) = resp.chunk().await? {
8591
file.write_all(&chunk).await?;

‎crates/iter-worker/src/main.rs‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,14 @@ async fn main() -> anyhow::Result<()> {
2929
// The FL GTFS lands next to the other graph inputs (steps/gtfs.rs
3030
// skips netex feeds, leaving this slot for the worker).
3131
out_path: data_dir.join("graph/TRENITALIA-FL.gtfs.zip"),
32-
netex_url: config::opt("NETEX_URL"),
32+
// The FL NeTEx auto-downloads from the Italian NAP (CCISS) public
33+
// endpoint — RAP Lazio L2 (the L1 superset). Override or set empty
34+
// (NETEX_URL=) to use a file placed at GATEWAY_NETEX_PATH instead.
35+
netex_url: Some(config::or(
36+
"NETEX_URL",
37+
"https://www.cciss.it/nap/mmtis/public/api/v1/download/blob/Asset/663391/checkedResource",
38+
))
39+
.filter(|u| !u.is_empty()),
3340
http: http.clone(),
3441
}),
3542
Box::new(jobs::rt_reliability::RtReliability::from_env(http)),

‎docs/adr/0016-fl-netex-to-gtfs-converter.md‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,10 @@ deps):
3131
- emit a referentially-complete GTFS zip (agency/stops/routes/calendar/trips/
3232
stop_times), dropping only rows that can't resolve, written atomically to
3333
`<graph>/TRENITALIA-FL.gtfs.zip` where the OTP graph build picks it up;
34-
- the FL job runs on startup + daily; `NETEX_URL`, if set, fetches the file first,
35-
else it's placed at `GATEWAY_NETEX_PATH`.
34+
- the FL job runs on startup + daily and **auto-downloads** the NeTEx from the
35+
Italian NAP (CCISS) public endpoint on each run (`NETEX_URL`, default the RAP
36+
Lazio L2 asset) — the daily cadence is the refresh; set `NETEX_URL=` empty to
37+
use a file placed at `GATEWAY_NETEX_PATH` instead.
3638

3739
## Consequences
3840

@@ -46,8 +48,12 @@ deps):
4648
window — it does **not** yet expand the `UicOperatingPeriod` bit-patterns into
4749
`calendar_dates` exceptions; for a single-week window this is faithful, but a
4850
longer feed with holiday exceptions would need that refinement.
49-
- **NAP auto-download is unsolved** (login-gated). The job fetches `NETEX_URL`
50-
when reachable; otherwise the file is placed manually.
51+
- **NAP auto-download works**: the CCISS NAP serves the NeTEx over an
52+
unauthenticated HTTP GET (no login), so the job fetches it directly. The
53+
default pins a stable asset id; if the NAP rotates ids, resolving the asset by
54+
filename from the NAP catalog JSON is the documented fallback. The data carries
55+
`no_licence_no_contract` (NAP access regime, redistribution unstated — see
56+
`DATA_LICENSES.md`).
5157
- Shapes (`shapes.txt`) are not stitched (OTP routes without them); a future
5258
refinement can derive them from OSM rail.
5359

‎docs/roadmap/README.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,10 +46,9 @@ flagged per item.
4646
- **FL NeTEx→GTFS** ✅ done (ADR 0016) — the worker streams the official Lazio
4747
NeTEx (quick-xml over gunzip) into a routable GTFS the OTP graph build consumes;
4848
proven on the real ~58 MB CCISS dataset (450 stops / 5 routes / 1,594 trips /
49-
20,617 stop_times, zero loss). **Remaining:** `UicOperatingPeriod` →
50-
`calendar_dates` exceptions, `shapes.txt` stitching from OSM rail, and **NAP
51-
auto-download** (login-gated; `NETEX_URL` fetches a reachable file, else it's
52-
placed manually).
49+
20,617 stop_times, zero loss). Auto-downloads from the Italian NAP (CCISS)
50+
public endpoint each run. **Remaining:** `UicOperatingPeriod` →
51+
`calendar_dates` exceptions and `shapes.txt` stitching from OSM rail.
5352
Design: concept doc 11 — gateway-and-external-providers · Decision: ADR 0016.
5453

5554
## 2. Remaining Rust capabilities

0 commit comments

Comments
 (0)