Skip to content

Show vessels per source and rolling usage counts in /v1/stats #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bkeepers merged 6 commits into from
Aug 23, 2026
Merged

Show vessels per source and rolling usage counts in /v1/stats #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fdb3bff
Update some docs
bkeepers Aug 23, 2026
875ba42
Report distinct and exclusive vessel counts per source in /v1/stats
bkeepers Aug 23, 2026
76855df
Count streams opened and API requests in /v1/stats
bkeepers Aug 23, 2026
43007d5
Report rolling 24 h and 7 d counts in /v1/stats instead of since-star…
bkeepers Aug 23, 2026
3110a6d
Harden the /v1/stats usage counters
bkeepers Aug 23, 2026
54bdfd3
gofmt stats_test.go
bkeepers Aug 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
.wrangler/
.env

node_modules/
signalk-plugin/dist/

# local notes with personal contact details, never committed
*.local.md
spec
6 changes: 3 additions & 3 deletions PLAN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ Kystverket allows one TCP connection per source IP (a second connection makes bo

Decision (2026-08-20): pull in sources whose terms are unclear to get coverage now, keep every one of them separable, and walk them back as volunteer and licensed data arrive. Separable means: its own `source` value on every event, its own license tag in the archive path (purgeable), its own env flag, and never in the health gate.

- [x] AISHub, reciprocal: aiscast feeds only volunteer-station events (`udp:`/`mmsi:`/`http:`/`v1:`; no public feeds, no synthesized events, per their join terms) to the assigned UDP port (`AISHUB_FEED`), and polls the aggregate snapshot once a minute (`AISHUB_USERNAME`): global terrestrial coverage (~51k vessels per snapshot, ~100k events per new snapshot), `synthesized`, source `aishub`, archive `aishub-terms/`. Measured: the world snapshot regenerates only every ~5 min, so its positions are 1–6 min old; good for "who is out there", not for close-quarters. Their current page grants "use" only (the 2021 "publish freely or commercially" sentence is gone); they can revoke at any time. Walk-back: unset the flag, purge `aishub-terms/`. [x] API username from AISHub; polling live on the box since 2026-08-20 18:09 UTC.
- [x] AISHub, reciprocal: aiscast feeds only volunteer-station events (`udp:`/`mmsi:`/`http:`/`v1:`; no public feeds, no synthesized events, per their join terms) to the assigned UDP port (`AISHUB_FEED`), and polls the aggregate snapshot once a minute (`AISHUB_USERNAME`): global terrestrial coverage (~51k vessels per snapshot, ~100k events per new snapshot), `synthesized`, source `aishub`, archive `aishub-terms/`. Measured: the world snapshot regenerates only every ~5 min, so its positions are 1–6 min old; good for "who is out there", not for close-quarters. Terms (the "Terms of Use" section of their join page is the whole contract; no separate document, no governing law, no termination clause): contributors "are allowed to use the aggregated data for free", nothing on publication, redistribution, or commercial use either way. The 2018–2021 page said "There are no restrictions on how the users will use the data. Everybody is allowed to publish the data for free or to use it for commercial purposes"; that was deleted between May and December 2021 and nothing replaced it. Same operator as VesselFinder, whose own terms do forbid redistribution, so the silence is a choice. Reading: re-serving with attribution is permitted by silence under an at-will membership; the only remedy they have is revoking the key. Confirmed in writing by AISHub on 2026-08-22: "We do not set any restrictions on how the data is used, so you are free to use it for your project, including commercial purposes and redistribution."" Walk-back: unset the flag, purge `aishub-terms/`. [x] API username from AISHub; polling live on the box since 2026-08-20 18:09 UTC.
- Not now: BarentsWatch is the same Kystverket data re-served as JSON (plus EEZ/Svalbard); only worth adding as a second path if the Kystverket TCP feed proves unreliable.
- [ ] Ask Sjöfartsverket (Sweden; 2019 price sheet: 5,000–25,000 SEK/yr distributor tiers) and DMA (Denmark live; paid subscription) for current terms.
- [ ] EuRIS inland overlay on `/v1` if the viewer wants inland Europe (anonymised; never in `/v0`).
Expand Down Expand Up @@ -177,8 +177,7 @@ Blocked until the feeder agreement (including the funding terms in Sustainabilit
- [x] Per-station stats: `GET /v1/stations/{id}` (events, duplicates heard elsewhere first, vessels in 30 min, first/last seen, coverage bbox, the station's vessels) and the viewer's `?station=` view.
- [ ] Coverage map across stations.
- [ ] Offline alerts (needs a contact address per station).
- [ ] UDP ports per station for legacy setups (anonymous hashed-IP UDP and the HTTP token path cover today's cases).
- [ ] PR to `sdr-enthusiasts/docker-shipfeeder` (`AISCAST_TOKEN` → `-H … USERPWD x:$AISCAST_TOKEN GZIP on INTERVAL 15`; patch prepared, awaiting go-ahead to open).
- [x] PR to `sdr-enthusiasts/docker-shipfeeder` (`AISCAST_TOKEN` → `-H … USERPWD x:$AISCAST_TOKEN GZIP on INTERVAL 15`; patch prepared, awaiting go-ahead to open).
- [x] Raw feed back to feeders: `/v1/nmea` WebSocket of deduped sentences with TAG blocks (`s:` station, `c:` time, `t:` license tag), bbox-filterable, feeder/peer/partner/admin tokens only. Wider access (free for all vs. free for non-commercial use) is an open question; see Sustainability.

### Stage 2: history and reporting APIs
Expand Down Expand Up @@ -247,6 +246,7 @@ Licensing is per source, carried on every reception and surfaced on every output
| EuRIS | open data with attribution | literal string `API/Service [name] incorporated from EuRIS (eurisportal.eu)` | yes, on `/v1` overlay |
| Denmark DMA archive | none stated | confirm with DMA before any redistribution | blocked |
| aisstream.io | no terms exist | ask before relying on it | unclear |
| AISHub | membership, "use for free" | keep feeding receiver-only data; credit AISHub; revocable at will | yes with attribution, confirmed in writing by AISHub 2026-08-22 (commercial use and redistribution fine); never the sole basis for an SLA |
| Volunteer stations | our feeder agreement | per agreement | ODbL for the contributed aggregate and CC0 for contributed history are the proposal, only where the feeder agreement explicitly grants it |

The feeder agreement (plain language: non-exclusive license limited to running the commons, not transferable to an acquirer, opt-in, station location never published precisely without consent) and the governance line (the project cannot be sold or unilaterally taken over) need legal review before Stage 1. Never relay paid aggregator data.
Expand Down
12 changes: 6 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,13 @@ Every event says which of these it came from. What is deliberately not pulled in

Licensing is per source. aiscast does not relicense the aggregate: each event is re-served under the terms of the source it came from, which is why `source` is on every event, every vessel, and every archived hour. If you display or redistribute the data, carry the source's attribution through.

| Source | License | What you must do |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Kystverket | [NLOD 2.0](https://data.norge.no/nlod/en/2.0) | Credit: "Contains data under the Norwegian licence for Open Government data (NLOD) distributed by the Norwegian Coastal Administration." NLOD is not sublicensable: you are bound by it directly. |
| Fintraffic Digitraffic | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | Credit: "Source: Fintraffic / digitraffic.fi, license CC 4.0 BY." |
| Source | License | What you must do |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Kystverket | [NLOD 2.0](https://data.norge.no/nlod/en/2.0) | Credit: "Contains data under the Norwegian licence for Open Government data (NLOD) distributed by the Norwegian Coastal Administration." NLOD is not sublicensable: you are bound by it directly. |
| Fintraffic Digitraffic | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | Credit: "Source: Fintraffic / digitraffic.fi, license CC 4.0 BY." |
| Volunteer receivers | beta: contributed for re-serving by this server; a written feeder agreement (open license on the aggregate, non-transferable to any acquirer, opt-in, station locations never published precisely) is the next stage and will be reviewed before volunteer data scales | Credit "aiscast volunteer receivers" for now; expect an open-data license (ODbL is the proposal) once the agreement exists. |
| AISHub aggregate | AISHub membership; their published terms grant "use" only | Treat as view-only: fine to display, do not build a product on these events alone. They may be withdrawn; `source: aishub` and the `aishub-terms/` archive tag make that a clean cut. |
| aisstream.io | no published terms | Same as AISHub: best effort, may disappear. `source: aisstream`. |
| AISHub aggregate | AISHub membership: contributors "are allowed to use the aggregated data for free"; no stated restriction on publication or commercial use | Credit "AISHub" and carry `source: aishub` through. |
| aisstream.io | no published terms | Best effort, may disappear. `source: aisstream`. |

aiscast's own code is [MIT](LICENSE). Volunteer station locations are never published with precision, UDP stations are identified by a keyed hash rather than an address, and the archive keeps raw receptions per source so any source can be purged. The full position, including the feeder agreement draft, the privacy rules, and how the project intends to fund itself without relicensing data, is in [PLAN.md](PLAN.md#licensing-and-attribution).

Expand Down
15 changes: 8 additions & 7 deletions docs/API.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Base: `https://ais.openwaters.io` (WebSocket: `wss://`). All responses are JSON;
| `GET /v1/stream` (WebSocket) | none to subscribe (anonymous tier); any token to publish | native event stream, both directions |
| `GET /v1/vessels` | none | current positions as GeoJSON |
| `GET /v1/stations`, `GET /v1/stations/{id}` | none | stations being heard, with per-station statistics |
| `GET /v1/stats` | none | usage summary: stations, vessels, event rate, clients |
| `GET /v1/stats` | none | usage summary: stations, vessels per source, event rate, streams and API requests |
| `GET /v1/nmea` (WebSocket) | feeder tier (earned or minted), peer, partner, admin | deduplicated raw NMEA back to feeders |
| `POST /v1/keys` | none | mint a personal token for a device key |
| `POST /v1/receive` | personal, feeder, peer, or admin token | AIS-catcher style HTTP ingest |
Expand Down Expand Up @@ -147,18 +147,19 @@ Every station heard since the server started:

## `GET /v1/stats`

A one-shot usage summary, for status pages and tracking growth:
A one-shot usage summary, for status pages and tracking growth. Counts are rolling windows over the last 24 hours and 7 days (hourly buckets, kept across restarts), never since-start totals:

```json
{"time": "2026-08-21T13:40:12Z", "uptime_s": 86122,
{"time": "2026-08-21T13:40:12Z",
"stations": {"total": 14, "active": 11, "by_source": {"kystverket": 1, "digitraffic": 1, "udp": 7, "http": 3, "v1": 2}},
"vessels": {"total": 4812, "with_position": 4790, "by_kind": {"vessel": 4701, "aton": 88, "base": 19, "sar": 4}},
"events": {"total": 18230411, "duplicates": 2210560, "per_second": 212.4},
"clients": 9,
"sources": {"kystverket": {"events": 9120033, "last_age_s": 0}, "udp:84a377dcf41b": {"events": 40211, "last_age_s": 3}, "...": {}}}
"events": {"per_second": 212.4, "last_24h": 18230411, "last_7d": 121004312, "duplicates": {"last_24h": 2210560, "last_7d": 15320011}},
"clients": {"streams": 9, "streams_opened": {"last_24h": 410, "last_7d": 2822}, "requests": {"last_24h": 28310, "last_7d": 190412}},
"sources": {"kystverket": {"events": {"last_24h": 9120033, "last_7d": 61233190}, "last_age_s": 0, "vessels": 2411, "vessels_exclusive": 180},
"udp:84a377dcf41b": {"events": {"last_24h": 40211, "last_7d": 281002}, "last_age_s": 3, "vessels": 61, "vessels_exclusive": 2}, "...": {}}}
```

`stations.active` counts stations heard in the last 5 minutes; `by_source` groups them by the part of `source` before `:` (`udp`, `http`, `v1`, `mmsi`, or the upstream name). `vessels` covers the 30-minute cache. `events.per_second` is the deduplicated event rate over the last 30 s; `total` and `duplicates` are since start. `clients` is open WebSocket subscriptions. `sources` has per-source event totals and seconds since each last produced an event.
`stations.active` counts stations heard in the last 5 minutes; `by_source` groups them by the part of `source` before `:` (`udp`, `http`, `v1`, `mmsi`, or the upstream name). `vessels` covers the 30-minute cache. `events.per_second` is the deduplicated event rate over the last 30 s; `last_24h`/`last_7d` count deduplicated events and `duplicates` the messages dropped as already seen. `clients.streams` is open WebSocket subscriptions; `streams_opened` counts streams accepted on `/v0/stream`, `/v1/stream` and `/v1/nmea`, and `requests` counts HTTP API requests (everything except streams, `/health` and `/metrics`). `sources` has, per source, events over the same windows, seconds since it last produced an event, `vessels` (distinct MMSIs its stations heard in the last 30 minutes, counting messages another source delivered first) and `vessels_exclusive` (those no other source heard in that window).

## `GET /v1/nmea`: raw sentences back to feeders

Expand Down
Loading
Loading