Poll AISHub every 20 seconds - #18
Merged
Merged
Conversation
AISHub lowered the rate limit on our account to 20 seconds. The default interval follows it, and snapshot ingest is paced over a fixed 45 seconds rather than a share of the poll interval so slow subscribers still see under ~1k events/s. A new snapshot only appears every ~5 minutes, so ingest outlasting a poll costs nothing.
There was a problem hiding this comment.
Pull request overview
Updates AISHub integration to comply with the new 20s account rate limit while smoothing snapshot ingest so slow subscribers are less likely to be overwhelmed.
Changes:
- Updates documentation to reflect
AISHUB_INTERVALdefault/limit of20s. - Decouples AISHub snapshot ingest pacing from the poll interval by using a fixed 45s ingest budget.
- Updates AISHub polling-related comments/docs in code and planning notes.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| server/README.md | Updates the documented AISHUB_INTERVAL default to 20s to match the new AISHub limit. |
| server/aishub.go | Adjusts AISHub snapshot ingest pacing to a fixed 45s budget and updates polling-related comments. |
| PLAN.md | Updates the AISHub polling cadence description (and related text) in the project plan. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
225
to
+227
| p.arch.write(Reception{Source: "aishub", Station: "aishub", RecvTime: start, Body: strings.TrimSpace(string(body))}) | ||
| return p.ingestAishub(body, start, st, interval*3/4) // the rest of the interval covers the fetch | ||
| // paced independently of the poll interval: a new snapshot only appears every ~5 min, so ingest can outlast a poll | ||
| return p.ingestAishub(body, start, st, 45*time.Second) |
|
|
||
| // AISHub is reciprocal: we feed them our volunteer receivers' stream over UDP (their assigned port), and poll | ||
| // their aggregate snapshot (all stations, positions downsampled to ≤60 s) once a minute. Their terms grant "use" | ||
| // their aggregate snapshot (all stations, positions downsampled to ≤60 s) every 20 s. Their terms grant "use" |
| Decision (2026-08-20): pull in sources whose terms are unclear to get coverage now, keep every one of them separable, and walk them back as volunteer and licensed data arrive. Separable means: its own `source` value on every event, its own license tag in the archive path (purgeable), its own env flag, and never in the health gate. | ||
|
|
||
| - [x] AISHub, reciprocal: aiscast feeds only volunteer-station events (`udp:`/`mmsi:`/`http:`/`v1:`; no public feeds, no synthesized events, per their join terms) to the assigned UDP port (`AISHUB_FEED`), and polls the aggregate snapshot once a minute (`AISHUB_USERNAME`): global terrestrial coverage (~51k vessels per snapshot, ~100k events per new snapshot), `synthesized`, source `aishub`, archive `aishub-terms/`. Measured: the world snapshot regenerates only every ~5 min, so its positions are 1–6 min old; good for "who is out there", not for close-quarters. Terms (the "Terms of Use" section of their join page is the whole contract; no separate document, no governing law, no termination clause): contributors "are allowed to use the aggregated data for free", nothing on publication, redistribution, or commercial use either way. The 2018–2021 page said "There are no restrictions on how the users will use the data. Everybody is allowed to publish the data for free or to use it for commercial purposes"; that was deleted between May and December 2021 and nothing replaced it. Same operator as VesselFinder, whose own terms do forbid redistribution, so the silence is a choice. Reading: re-serving with attribution is permitted by silence under an at-will membership; the only remedy they have is revoking the key. Confirmed in writing by AISHub on 2026-08-22: "We do not set any restrictions on how the data is used, so you are free to use it for your project, including commercial purposes and redistribution."" Walk-back: unset the flag, purge `aishub-terms/`. [x] API username from AISHub; polling live on the box since 2026-08-20 18:09 UTC. | ||
| - [x] AISHub, reciprocal: aiscast feeds only volunteer-station events (`udp:`/`mmsi:`/`http:`/`v1:`; no public feeds, no synthesized events, per their join terms) to the assigned UDP port (`AISHUB_FEED`), and polls the aggregate snapshot every 20 s (`AISHUB_USERNAME`): global terrestrial coverage (~51k vessels per snapshot, ~100k events per new snapshot), `synthesized`, source `aishub`, archive `aishub-terms/`. Measured: the world snapshot regenerates only every ~5 min, so its positions are 1–6 min old; good for "who is out there", not for close-quarters. Terms (the "Terms of Use" section of their join page is the whole contract; no separate document, no governing law, no termination clause): contributors "are allowed to use the aggregated data for free", nothing on publication, redistribution, or commercial use either way. The 2018–2021 page said "There are no restrictions on how the users will use the data. Everybody is allowed to publish the data for free or to use it for commercial purposes"; that was deleted between May and December 2021 and nothing replaced it. Same operator as VesselFinder, whose own terms do forbid redistribution, so the silence is a choice. Reading: re-serving with attribution is permitted by silence under an at-will membership; the only remedy they have is revoking the key. Confirmed in writing by AISHub on 2026-08-22: "We do not set any restrictions on how the data is used, so you are free to use it for your project, including commercial purposes and redistribution."" Walk-back: unset the flag, purge `aishub-terms/`. [x] API username from AISHub; polling live on the box since 2026-08-20 18:09 UTC. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AISHub lowered the rate limit on our account to 20 seconds, so
AISHUB_INTERVALnow defaults to20s(the floor in main.go already allows it).Snapshot ingest used to be spread over three quarters of the poll interval to keep slow subscribers under ~1k events/s. At 20 s that would be 15 s for a ~45k-row snapshot, about 3k events/s, which is the rate that overruns far clients. Ingest is now paced over a fixed 45 s instead. AISHub only regenerates the snapshot every ~5 minutes, so an ingest that outlasts a poll doesn't lose anything; the next poll just starts after it finishes.
No env change needed on the box unless
/etc/aiscast.envpinsAISHUB_INTERVAL.