AIS support for ground truthing app by elliott-ruebush · Pull Request #68 · dbetchkal/NPS-ActiveSpace

elliott-ruebush · 2026-06-10T22:47:24Z

Overview

Add AIS as a third --track-source for ground truthing (GPS / ADSB / AIS)
New utils/ais/ where I've moved logic that previously lived in models.py. This includes the MXAK AIS reader, query MXAK data function, and timestamp parsing.
utils/time_utils/site.py for logic around finding timezones for sites and converting UTC -> local time.
- IMO we should gradually move more central datetime utils here and also standardize towards using tz-aware datetimes or standardizing everything to UTC, but I think doing so is out of scope for this PR
TrackSource enum; track_source naming; lookup_track_vehicle() for AIS vs ADSB/GPS
- annotation UI shows vessel MMSI, name, and type
- load_tracks() functionality for importing tracks from different data sources moved to own file to split logic out more cleanly
Unit tests for AIS ingest/alignment with NVSPL, ADSB parser, and vehicle lookup

Depends on #67 (fix/split_ground_truthing_into_multiple_files).

Testing

pytest locally
Manually test making AIS annotations on a DOI windows host:

GLBALSTL 2024 annotations with AIS tracks

nps_active_space/scripts/run_ground_truthing.py -e DENA -u GLBA -s LSTL -y 2024 -t AIS

with the following config:

[database:overflights]
name = overflights
username = ... # not posting publicly
password = ... # not posting publicly
host = # not posting publicly
port = 5432

[data]
site_metadata = T:\ResMgmt\Sound\V-Drive_Transfers\Site Metadata\Complete_Metadata_AKR_2001-2025.txt
adsb = V:\Noncanonical Data\2025 - 2021 ADS-B\2025 ADS-B Healy Repeater\TSV
ais = V:\Noncanonical Data\2025 - 2020 MXAK-AIS-GLBA\MXAK_AIS_GLBALSTL_24_SAMPLE
nvspl_archive = \\inpdenagrogu\F\Sound Data
dem = T:\ResMgmt\Sound\NMSim_Partial_Extract\NMSim\03 DEMs\DENA_DEM_m_4269.tif
mennitt = T:\ResMgmt\Sound\GIS\AlaskaRegion\AK Soundscape Model\DENA_L50_Existing.tif

[project]
dir = T:\ResMgmt\Sound\NMSim_Partial_Extract\NMSim\01 SITES
nmsim = T:\ResMgmt\Sound\Applications\NMSim_2014\Nord2000batch.exe
faa_releasable_db = T:\ResMgmt\Sound\V-Drive_Transfers\ReleasableAircraft_2026\MASTER.txt
FAA_type_corrections = T:\ResMgmt\Sound\V-Drive_Transfers\ReleasableAircraft_2026\FAA_AircraftCorrections.json

Example annotation screen:

Smoke test ADS-B annotations still work

Ground truthing GUI with ADS-B data for DENATRLA 2025

python nps_active_space/scripts/run_ground_truthing.py -e DENA -u DENA -s TRLA -y 2025 -t ADSB

Smoke test GPS annotations still work

Ground truthing GUI with GPS data for DENATRLA 2023

python nps_active_space/scripts/run_ground_truthing.py -e DENA -u DENA -s TRLA -y 2023 -t GPS

Next steps

1-2 additional PRs for:

Improvements to viz.py to better display AIS [WIP] Refactor viz.py to attempt to improve code quality and better display AIS tracks #72
Updates to the actual active space generation logic to work properly with AIS annotations/tracks. Also may be worth exploring some sort of downsampling of AIS annotation data as it seems to be much larger than ADS-B data (more points since vessels move through a site area much slower than aircraft?)

dbetchkal · 2026-06-23T23:24:34Z

+}
+
+POINT_COLUMNS = ("MMSI", "TIME", "lat", "lon")
+EVENT_GAP_SECONDS = 1200


For Adsb we use 900 seconds. Let's use the same here, too...

dbetchkal · 2026-06-23T23:25:46Z

+
+        df.drop_duplicates(subset=list(POINT_COLUMNS), keep="last")
+
+        # 20-minute gap threshold splits vessel transits (mirrors ADSB event logic).


To your point a few weeks ago, this inline comment is true only when EVENT_GAP_SECONDS = 1200. Let's use more generic language or remove the comment altogether?

That sneakily stayed in there, good callout

dbetchkal · 2026-06-23T23:40:45Z

    starttime = points.point_dt.iat[0]
    endtime = points.point_dt.iat[-1]
-    flight_times = (points.point_dt - starttime).dt.total_seconds().values  # Seconds after initial point
+    flight_times = (points.point_dt - starttime).dt.total_seconds().values


Do you agree that flight_times is still the appropriate name for this variable? Something more generic seems better...

How about something quite general like elapsed_seconds? Or did you have anything else in mind?

Discussed - we confirmed elapsed_seconds seems like a clear name here

dbetchkal · 2026-06-23T23:47:48Z

+        dem_glob,
+        description=(
+            f"elevation DEM for {unit}{site} "
+            f"(expected GeoTIFF in Input_Data/01_ELEVATION/elevation_m_nad83_utm*.tif)"


Let's chat about what this line/function is doing.

Discussed - Python automatically concats strings together if they span across multiple lines.

dbetchkal · 2026-06-24T00:01:39Z

+            faa, TrackSource.ADSB, "A98046_0_20250623", _adsb_points(icao="DEADBE")
+        ) == (None, None, None)
+
+    def test_gps_no_match_returns_nones(self):


I'm not sure I understand this test. Let's chat about it.

This is verifying the logic in the vehicle info lookup that returns a tuple of None for aircraft in the event of a missing FAA DB.

This logic could arguably be changed, but this was effectively what the ground truthing app was doing before we split it into more files

NPS-ActiveSpace/nps_active_space/ground_truthing/vehicle_info.py

Lines 41 to 50 in aea2935

def lookup_aircraft(

faa: pd.DataFrame | None,

track_source: TrackSource,

track_id: str,

points: gpd.GeoDataFrame,

) -> tuple[pd.Series | None, str | None, str | None]:

"""Look up FAA row and help text for a track."""

if faa is None:

return None, None, None

faa_row = None

dbetchkal · 2026-06-24T00:07:36Z

+from nps_active_space.utils.ais.query import query_ais_mxak
+from nps_active_space.utils.ais.reader import MxakAis
+
+REPO = Path(__file__).resolve().parents[3]


Super helpful!

Yes! Ideally we move to using pathlib generally across the repo to better support potential cross-platform work.

dbetchkal · 2026-06-24T00:08:08Z

+REPO = Path(__file__).resolve().parents[3]
+AIS_FIXTURE = REPO / "example_data" / "AIS" / "MXAK-AIS-GLBA-20250107.csv"
+AIS_DIR = AIS_FIXTURE.parent
+
+METADATA_COLS = ["TIME", "MMSI", "lat", "lon", "ship_name", "shiptype", "altitude", "event_id"]
+
+
+def _tabular(gdf: gpd.GeoDataFrame, columns: list[str]) -> pd.DataFrame:
+    return pd.DataFrame(gdf)[columns].reset_index(drop=True)
+
+
+@pytest.fixture(scope="module")


A decorator, right? Let's chat

Discussed - these are pytest fixtures, which are basically a way to specify some generic setup (and/or teardown) that happens before/after a unit test is run. A common use case (and why we have them here) is to provide a generic input that is freshly initialized for each unit test that includes the fixture.

Extract MXAK AIS parsing into utils/ais, add site-local timezone conversion for NVSPL alignment, and wire AIS as a track source in the ground-truthing CLI. Co-authored-by: Cursor <cursoragent@cursor.com>

The new name states the contract: resolve a glob pattern to a file path or raise FileNotFoundError with a descriptive message. Co-authored-by: Cursor <cursoragent@cursor.com>

Restore DST-transition and UTC-offset documentation from the original models.py implementation, deduplicate AKST/AKDT handling, and rename the test module to test_timestamp_parsing.py. Co-authored-by: Cursor <cursoragent@cursor.com>

Move query_ais_mxak tests out of test_reader.py so test modules align with nps_active_space.utils.ais.reader and .query. Co-authored-by: Cursor <cursoragent@cursor.com>

Extract shared alignment helpers, add self-documenting synthetic tests for UTC-to-local CPA matching and NVSPL window peaks, and parametrize the May 24 example-data regression tests by MMSI. Co-authored-by: Cursor <cursoragent@cursor.com>

Replace per-column row asserts with an expected DataFrame comparison, matching the pattern used in AIS reader tests. Co-authored-by: Cursor <cursoragent@cursor.com>

Introduce a shared TrackSource StrEnum, move per-source track queries into ground_truthing/tracks.py, and adopt the enum in ground-truthing and metrics entry points. Co-authored-by: Cursor <cursoragent@cursor.com>

Rename tracks.py to load_tracks.py and load_ground_truthing_tracks() to load_tracks() so the module name matches its public entry point. Co-authored-by: Cursor <cursoragent@cursor.com>

Rename database_type to track_source in the GUI, unify AIS and aircraft metadata via lookup_track_vehicle, and use match/case with TrackSource. Co-authored-by: Cursor <cursoragent@cursor.com>

Cover UTC format variants, AKST/AKDT mixed rows with DST fallback, and unrecognized formats using assert_series_equal. Co-authored-by: Cursor <cursoragent@cursor.com>

Exercise lookup_aircraft and lookup_track_vehicle with synthetic FAA fixtures alongside existing AIS vessel tests. Co-authored-by: Cursor <cursoragent@cursor.com>

Use SQLAlchemy URL.create so usernames and passwords are percent-encoded before connecting for GPS track loading. Co-authored-by: Cursor <cursoragent@cursor.com>

…ll fields separately

…ble in spline fit

Ground-truthing does not depend on 3D viz behavior; the DEM sampling patch belongs in feature/vis_improvements_for_ais with the fuller package refactor and performance fixes. Co-authored-by: Cursor <cursoragent@cursor.com>

…ng paths

elliott-ruebush changed the base branch from main to fix/split_ground_truthing_into_multiple_files June 10, 2026 22:47

elliott-ruebush force-pushed the fix/split_ground_truthing_into_multiple_files branch 2 times, most recently from a016258 to 2984a6c Compare June 18, 2026 18:13

elliott-ruebush force-pushed the feature/ground_truthing_ais branch from 5d7b71c to 2e25d65 Compare June 18, 2026 18:13

elliott-ruebush mentioned this pull request Jun 18, 2026

Add example ADS-B, AIS, and NVSPL data #70

Merged

elliott-ruebush commented Jun 22, 2026

View reviewed changes

Comment thread requirements.txt

elliott-ruebush force-pushed the fix/split_ground_truthing_into_multiple_files branch from 2984a6c to eddc605 Compare June 22, 2026 19:05

elliott-ruebush force-pushed the feature/ground_truthing_ais branch 4 times, most recently from b27b335 to a2ef533 Compare June 22, 2026 20:06

elliott-ruebush marked this pull request as ready for review June 22, 2026 22:18

elliott-ruebush changed the title ~~[WIP] AIS support for ground truthing app~~ AIS support for ground truthing app Jun 22, 2026

elliott-ruebush mentioned this pull request Jun 23, 2026

[WIP] Refactor viz.py to attempt to improve code quality and better display AIS tracks #72

Draft

5 tasks

dbetchkal approved these changes Jun 24, 2026

View reviewed changes

elliott-ruebush force-pushed the fix/split_ground_truthing_into_multiple_files branch from 6fd06b1 to 558511b Compare June 24, 2026 00:59

elliott-ruebush and others added 14 commits June 25, 2026 09:13

Add AIS track support for ground truthing.

44d6054

Extract MXAK AIS parsing into utils/ais, add site-local timezone conversion for NVSPL alignment, and wire AIS as a track source in the ground-truthing CLI. Co-authored-by: Cursor <cursoragent@cursor.com>

Rename _require_glob to _glob_path_or_raise in helpers.

360234c

The new name states the contract: resolve a glob pattern to a file path or raise FileNotFoundError with a descriptive message. Co-authored-by: Cursor <cursoragent@cursor.com>

Split AIS query tests into test_query.py to mirror source layout.

93d2388

Move query_ais_mxak tests out of test_reader.py so test modules align with nps_active_space.utils.ais.reader and .query. Co-authored-by: Cursor <cursoragent@cursor.com>

Assert ADSB parser output with assert_frame_equal.

27b18c7

Replace per-column row asserts with an expected DataFrame comparison, matching the pattern used in AIS reader tests. Co-authored-by: Cursor <cursoragent@cursor.com>

Add TrackSource enum and extract ground-truthing track loading.

d6364af

Introduce a shared TrackSource StrEnum, move per-source track queries into ground_truthing/tracks.py, and adopt the enum in ground-truthing and metrics entry points. Co-authored-by: Cursor <cursoragent@cursor.com>

Rename ground-truthing tracks module to load_tracks.

6807d00

Rename tracks.py to load_tracks.py and load_ground_truthing_tracks() to load_tracks() so the module name matches its public entry point. Co-authored-by: Cursor <cursoragent@cursor.com>

Refactor ground-truthing track_source and vehicle lookup.

f5db8cc

Rename database_type to track_source in the GUI, unify AIS and aircraft metadata via lookup_track_vehicle, and use match/case with TrackSource. Co-authored-by: Cursor <cursoragent@cursor.com>

Expand MXAK AIS timestamp parsing unit tests.

ebbcd0d

Cover UTC format variants, AKST/AKDT mixed rows with DST fallback, and unrecognized formats using assert_series_equal. Co-authored-by: Cursor <cursoragent@cursor.com>

Add AIS, ADSB, and GPS coverage for vehicle_info lookups.

146614d

Exercise lookup_aircraft and lookup_track_vehicle with synthetic FAA fixtures alongside existing AIS vessel tests. Co-authored-by: Cursor <cursoragent@cursor.com>

Fix overflights DB URLs when credentials contain @.

d1fab21

Use SQLAlchemy URL.create so usernames and passwords are percent-encoded before connecting for GPS track loading. Co-authored-by: Cursor <cursoragent@cursor.com>

remove comment from template, we'll add a config with more info for a…

6b51a17

…ll fields separately

PR Feedback: change to 900s for event gap + rename elapsed time varia…

6ab0110

…ble in spline fit

elliott-ruebush and others added 2 commits June 25, 2026 09:13

Move AIS annotation viz draping to the vis improvements branch.

7db2764

Ground-truthing does not depend on 3D viz behavior; the DEM sampling patch belongs in feature/vis_improvements_for_ais with the fuller package refactor and performance fixes. Co-authored-by: Cursor <cursoragent@cursor.com>

Minor test cleanup for test_vehicle_info.py

a09b4ab

elliott-ruebush force-pushed the feature/ground_truthing_ais branch from 93f8c07 to a09b4ab Compare June 25, 2026 17:16

elliott-ruebush changed the base branch from fix/split_ground_truthing_into_multiple_files to main June 25, 2026 17:16

elliott-ruebush added 3 commits June 25, 2026 12:58

Clean up structure of AIS reader.py file + transition towards accepti…

330ea6e

…ng paths

gs

21a4a48

60*60 instead of 3600

dec82ac

dbetchkal approved these changes Jun 26, 2026

View reviewed changes

elliott-ruebush merged commit ca2849a into main Jun 26, 2026


		df.drop_duplicates(subset=list(POINT_COLUMNS), keep="last")

		# 20-minute gap threshold splits vessel transits (mirrors ADSB event logic).

	def lookup_aircraft(
	faa: pd.DataFrame \| None,
	track_source: TrackSource,
	track_id: str,
	points: gpd.GeoDataFrame,
	) -> tuple[pd.Series \| None, str \| None, str \| None]:
	"""Look up FAA row and help text for a track."""
	if faa is None:
	return None, None, None
	faa_row = None

Conversation

elliott-ruebush commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Testing

Next steps

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elliott-ruebush Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

elliott-ruebush commented Jun 10, 2026 •

edited

Loading

elliott-ruebush Jun 24, 2026 •

edited

Loading