diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index e584e9f..55bc33c 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -30,18 +30,11 @@ jobs: run: | python -m pip install --upgrade pip - # Match the GDAL Python binding to the installed system version + # Match the GDAL Python binding to the installed system version. GDAL_VERSION=$(gdal-config --version) pip install "GDAL==${GDAL_VERSION}" - pip install "numpy" "scipy" "pandas>=2.2,<3" geopandas shapely pyproj rasterio fiona - pip install matplotlib KDEpy timezonefinder tqdm - pip install sqlalchemy psycopg2-binary beautifulsoup4 requests geopy pyvista - pip install "iyore @ git+https://github.com/nationalparkservice/iyore.git" - pip install pytest - - - name: Install package - run: pip install -e . + pip install -e ".[dev]" - name: Run tests run: pytest tests/ -v diff --git a/.gitignore b/.gitignore index ff340d0..26874d5 100644 --- a/.gitignore +++ b/.gitignore @@ -14,6 +14,9 @@ # .egg-info *.egg-info/ +# macOS files +.DS_Store + # config files (environment-specific; keep credentials out of repo) nps_active_space/config/* !nps_active_space/config/template.config diff --git a/README.md b/README.md index 8bcfc06..418a6fe 100644 --- a/README.md +++ b/README.md @@ -36,94 +36,96 @@ consistent with observed audibility under specified environmental conditions. ## Installation -### Step 1: Clone the NPS-ActiveSpace repository. +The repository has been tested with Python 3.12. Runtime dependencies are declared in `pyproject.toml` and installed automatically by pip (except GDAL on macOS/Linux, which requires a system library first). -``` +Clone the repository, then follow the steps for your platform. The clone includes [`example_data/`](example_data/) (~75 MB) for local development and tests (see [`example_data/README.md`](example_data/README.md)). + +```bash git clone https://github.com/dbetchkal/NPS-ActiveSpace.git cd NPS-ActiveSpace ``` -### Step 2: Set Up a Virtual Environment. - -Install Python, either via Anaconda/Miniconda, or directly. The repository has been tested with Python version 3.12. - -You can use a Conda environment if you want, but all installation is managed by pip. +### Windows -With Conda: +GDAL is installed automatically from a pre-built wheel in `pyproject.toml`. +Create virtual environment with updated pip +```bat +python -m venv .venv +.venv\Scripts\activate.bat +python -m pip install --upgrade pip ``` -conda create --name active python=3.12.12 -conda activate active + +Install required Python packages +```bat +pip install -e ".[dev]" ``` -With venv in a Git Bash terminal: +> **Ground-truthing GUI:** `run_ground_truthing.py` uses tkinter. Include "tcl/tk and IDLE" when installing from [python.org](https://www.python.org/downloads/). Verify with `python -c "import tkinter; print('ok')"`. -``` -python -m venv .venv -source .venv/bin/activate -``` +### macOS / Linux -With venv in a Windows Command Prompt terminal: +Install [GDAL](https://gdal.org/en/stable/) as a system library first -``` -python -m venv .venv -source .venv\Scripts\activate.bat +```bash +# macOS (Homebrew) +brew install gdal + +# Linux (Debian/Ubuntu) +sudo apt-get install gdal-bin libgdal-dev ``` -### Step 3: Install Dependencies +> **Ground-truthing GUI:** also install tkinter โ€” macOS: `brew install python-tk@3.12` ยท Linux: `sudo apt-get install python3.12-tk` -Make sure you are inside your virtual environment, then: +Create virtual environment with updated `pip` +```bash +python3.12 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip +``` +Install required Python packages (including appropriate GDAL Python binding) ``` -python -m pip install --upgrade pip -pip install -r requirements.txt +GDAL_VERSION=$(gdal-config --version) +pip install "GDAL==${GDAL_VERSION}" +pip install -e ".[dev]" ``` -Historical note: -The GDAL dependency comes from a `.whl` file published [here](https://github.com/cgohlke/geospatial-wheels/releases). If the Python version is updated, the GDAL wheel URL in `requirements.txt` may need to be changed to reflect the updated version. For example, `gdal-3.11.1-cp312-cp312-win_amd64.whl` is GDAL version 3.11.1 for Python 3.12. +**Note: Using without a clone:** you can install directly from GitHub with `pip install "NPS-ActiveSpace @ git+https://github.com/dbetchkal/NPS-ActiveSpace.git"` (on macOS/Linux, complete the GDAL steps above first). Config files go in the installed package's `config/` directory โ€” find it with `python -c "import nps_active_space, os; print(os.path.join(nps_active_space.ACTIVE_SPACE_DIR, 'config'))"`. Run scripts using `python -m ...` from outside the repo so Python uses the installed package, not a local checkout. -### Step 4: Install NPS-ActiveSpace +--- -From the repository's root directory, inside the virtual environment: +### Running scripts -``` -pip install -e . +Scripts are run as Python modules from the repository root: + +```bash +python -m nps_active_space.scripts.run_ground_truthing -e production -u DENA -s MOOS -y 2018 ``` -Try importing a python module to make sure this install worked, e.g. in a python file: +See [`nps_active_space/scripts/README.md`](nps_active_space/scripts/README.md) for the full list of scripts and their arguments. -``` -from nps_active_space.active_space import ActiveSpaceGenerator -``` +**Verify the install:** -### Step 5: Create Config File +```bash +python -c "import nps_active_space, geopandas, rasterio, iyore; print('NPS-ActiveSpace OK')" +``` -All scripts require a configuration file saved in the config directory `nps_active_space/config`. Please copy the template config file, fill in the values required for the script(s) you will be running, and save it to the config directory as `.config`. For example, a configuration file for Denali National Park and Preserve might be named `DENA.config` while a configuration file for Hawaii Volcanoes National Park might be named `HAVO.config` and have a different value for where the DEM file is stored than `DENA.config` +### NMSIM (active space generation) -Currently, the template config file has the following data: +Active space generation runs the NMSIM Nord2000 physics model as an external process. NMSIM is **not** installed by pip and is **not** included in this repository. Obtain the binary + required files separately (feel free to reach out to NPS-ActiveSpace maintainers) and set the path in your config: ```text -[database:overflights] - Values required if pulling tracks from the database in run_ground_truthing.py or run_audible_transits.py -name = Database name. -username = Database credentials username. -password = Database credentials password. -port = Database port. -host = Database host. - -[data] -site_metadata = Absolute path to the the file containing site metadata. Value required for all run_ground_truthing.py and generate_active_space.py -nvspl_archive = Absolute path to the directory where all NVSPL sound data is stored. Value required for all run_ground_truthing.py and generate_active_space.py -adsb = Absolute path to the directory where ADSB track data is stored. Value required if pulling ADSB tracks in run_ground_truthing.py or run_audible_transits.py -dem = Absolute path to the DEM tif file to use for active space generation. Value required for generate_active_space.py and generate_active_space_mesh.py -mennitt = Absolute path to the mennitt ambience tif. Value required for generate_active_space.py and generate_active_space_mesh.py - [project] -dir = Absolute path to the directory where all NPS-ActiveSpace files are stored. Required for all scripts. -nmsim = Absolute path to the NMSIM Nord2000batch.exe file. Value required for generate_active_space.py and generate_active_space_mesh.py -FAA_Releasable_db = Absolute path to the FAA MASTER.txt database file downloaded from the [FAA website](https://www.faa.gov/licenses_certificates/aircraft_certification/aircraft_registry/releasable_aircraft_download). Required for run_audible_transits.py -FAA_type_corrections = Absolute path to a json file for correcting aircraft types in the FAA database. Keys are ICAO addresses, values are correct aircraft type. Required for run_ground_truthing.py and run_audible_transits.py +nmsim = C:\path\to\Nord2000batch.exe ``` +Required for `generate_active_space.py`, `generate_3d_active_space.py`, and `generate_active_space_mesh.py`. **Not** required for ground-truthing, audible transits, or validation. + +### Configuration + +All scripts require a configuration file named `.config` (e.g. `DENA.config`, `HAVO.config`). Copy [`nps_active_space/config/template.config`](nps_active_space/config/template.config), fill in the values for your deployment, and save it to `nps_active_space/config/`. + ## Toolkit Architecture ```mermaid diff --git a/nps_active_space/README.md b/nps_active_space/README.md deleted file mode 100644 index 5490b49..0000000 --- a/nps_active_space/README.md +++ /dev/null @@ -1,108 +0,0 @@ -# Installation - -### Clone the NPS-ActiveSpace repository. - -``` -git clone https://github.com/dbetchkal/NPS-ActiveSpace.git -cd NPS-ActiveSpace -``` - -### Set Up a Virtual Environment. - -Install Python, either via Anaconda/Miniconda, or directly. The repository has been tested with Python version 3.12. - -You can use a Conda environment if you want, but all installation is managed by pip. - -With Conda: - -``` -conda create --name active python=3.12.12 -conda activate active -``` - -With venv in a Git Bash terminal: - -``` -python -m venv .venv -source .venv/bin/activate -``` - -With venv in a Windows Command Prompt terminal: - -``` -python -m venv .venv -source .venv\Scripts\activate.bat -``` - -### Install Dependencies - -Make sure you are inside your virtual environment, then: - -``` -python -m pip install --upgrade pip -pip install -r requirements.txt -``` - -Historical note: -The GDAL dependency comes from a `.whl` file published [here](https://github.com/cgohlke/geospatial-wheels/releases). If the Python version is updated, the GDAL wheel URL in `requirements.txt` may need to be changed to reflect the updated version. For example, `gdal-3.11.1-cp312-cp312-win_amd64.whl` is GDAL version 3.11.1 for Python 3.12. - -### Install NPS-ActiveSpace - -From the repository's root directory, inside the virtual environment: - -``` -pip install -e . -``` - -Try importing a python module to make sure this install worked, e.g. in a python file: - -``` -from nps_active_space.active_space import ActiveSpaceGenerator -``` - -### Create Config File - -All scripts require a configuration file saved in the config directory `nps_active_space/config`. Please copy the template config file, fill in the values required for the script(s) you will be running, and save it to the config directory as `.config`. For example, a DENA configuration file might be named `dena.config` while a HAVO configuration file might be named `havo.config` and have a different value for where the DEM file is stored than `dena.config` - -Currently, the template config file has the following data: - -TODO - check this / update this for new scripts. Explaining which scripts require what is a bit messy, especially with all the new scripts we added. Consider a better way to document this. - -```text -[database:overflights] - Values required if pulling GPS tracks from the database in run_ground_truthing.py or run_audible_transits.py -name = Database name. -username = Database credentials username. -password = Database credentials password. -port = Database port. -host = Database host. - -[data] -site_metadata = Absolute path to the the file containing site metadata. Value required for all run_ground_truthing.py and generate_active_space.py -nvspl_archive = Absolute path to the directory where all NVSPL sound data is stored. Value required for all run_ground_truthing.py and generate_active_space.py -adsb = Absolute path to the directory where ADSB track data is stored. Value required if pulling ADSB tracks in run_ground_truthing.py or run_audible_transits.py -ais = Absolute path to the MXAK AIS archive directory (e.g. MXAK-AIS-GLBA). Value required if pulling AIS tracks in run_ground_truthing.py with -t AIS -dem = Absolute path to the DEM tif file to use for active space generation. Value required for generate_active_space.py and generate_active_space_mesh.py -mennitt = Absolute path to the mennitt ambience tif. Value required for generate_active_space.py and generate_active_space_mesh.py - -[project] -dir = Absolute path to the directory where all NPS-ActiveSpace files are stored. Required for all scripts. -nmsim = Absolute path to the NMSIM Nord2000batch.exe file. Value required for generate_active_space.py and generate_active_space_mesh.py -FAA_Releasable_db = Absolute path to the FAA MASTER.txt database file downloaded from the [FAA website](https://www.faa.gov/licenses_certificates/aircraft_certification/aircraft_registry/releasable_aircraft_download). Required for run_audible_transits.py -FAA_type_corrections = Absolute path to a json file for correcting aircraft types in the FAA database. Keys are ICAO addresses, values are correct aircraft type. Required for run_ground_truthing.py and run_audible_transits.py -``` - -# Data and File Directory Setup - -TODO - -Define "project directory" as the directory containing the site folders. This is consistent with how the code defines it. - -Please document: - -- acoustic data (NVSPL, SPLAT) -- ADSB data -- GPS data (database setup and usage via config file) -- project directory -- site directories, including minimal manual setup steps (what directories the user needs to make vs. what's made automatically by the code) - - mention that annotation files and clock drift correction files go in here -- NMSIM diff --git a/nps_active_space/config/template.config b/nps_active_space/config/template.config index eac3ee3..718761c 100644 --- a/nps_active_space/config/template.config +++ b/nps_active_space/config/template.config @@ -1,20 +1,51 @@ +# Copy to .config (e.g. DENA.config) and fill in paths for your deployment. +# The environment name is passed to scripts as -e / --environment. + +# Required if pulling tracks from the database in run_ground_truthing.py or run_audible_transits.py [database:overflights] +# Database name name = +# Database credentials username username = +# Database credentials password password = +# Database port port = +# Database host host = [data] +# Absolute path to the file containing site metadata. +# Required for run_ground_truthing.py and generate_active_space.py site_metadata = +# Absolute path to the directory where all NVSPL sound data is stored. +# Required for run_ground_truthing.py and generate_active_space.py nvspl_archive = +# Absolute path to the directory where ADS-B track data is stored. +# Required if pulling ADS-B tracks in run_ground_truthing.py or run_audible_transits.py adsb = +# Absolute path to the directory where AIS track data is stored. +# Required if pulling AIS tracks in run_ground_truthing.py or run_audible_transits.py ais = +# Absolute path to the DEM tif file to use for active space generation. +# Required for generate_active_space.py and generate_active_space_mesh.py dem = +# Absolute path to the Mennitt ambience tif. +# Required for generate_active_space.py and generate_active_space_mesh.py mennitt = [project] +# Absolute path to the directory where all NPS-ActiveSpace project files are stored. +# Required for all scripts dir = +# Absolute path to the NMSIM Nord2000batch.exe file. +# Required for generate_active_space.py, generate_3d_active_space.py, and generate_active_space_mesh.py nmsim = -FAA_Releasable_db = -FAA_type_corrections = +# Absolute path to the FAA MASTER.txt database file from the FAA releasable aircraft download: +# https://www.faa.gov/licenses_certificates/aircraft_certification/aircraft_registry/releasable_aircraft_download +# Required for run_audible_transits.py +FAA_Releasable_db = +# Absolute path to a JSON file for correcting aircraft types in the FAA database. +# Keys are ICAO addresses; values are correct aircraft type. +# Required for run_ground_truthing.py and run_audible_transits.py +FAA_type_corrections = diff --git a/pyproject.toml b/pyproject.toml index 207766a..eca2a16 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,23 +1,65 @@ [build-system] -requires = ["setuptools", "wheel"] +requires = ["setuptools>=61.0.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "NPS-ActiveSpace" -version = "2.0" -description = "An NPS Sound Active Space Project." -requires-python = ">=3.12" +version = "3.0" +description = "NPS software tooling for acoustic modeling of active spaces" +requires-python = ">=3.12" # Windows GDAL wheel below is pinned for cp312 โ€” update its URL when bumping Python +readme = "README.md" +license = "CC0-1.0" +license-files = ["LICENSE.md"] authors = [ {name="Kirby Heck"}, {name="Adina Zucker"}, {name="Davyd Betchkal"} ] maintainers = [ - {name="Davyd Betchkal", email="davyd_betchkal@nps.gov"} + {name="Davyd Betchkal", email="davyd_betchkal@nps.gov"}, + {name="Elliott Ruebush", email="eruebush@gmail.com"} +] + +dependencies = [ + "geopandas>=1.1.1", + "geopy>=2.4.1", + "iyore @ git+https://github.com/nationalparkservice/iyore.git@b3b1e3fd025a392bf63bf6c4a8a4eebb887291ec", + "KDEpy>=1.1.12", + "matplotlib>=3.10.3", + "numpy>=2.3.1", + "pandas>=2.2,<3", + "pillow>=11.3.0", + "pyproj>=3.7.1", + "pyvista>=0.46.3", + "rasterio>=1.4.3", + "scipy>=1.16.0", + "shapely>=2.1.1", + "SQLAlchemy>=2.0.41", + "timezonefinder>=8.2.4", + "tqdm>=4.67.1", + "fiona>=1.10.1", + "psycopg2>=2.9.10 ; sys_platform == 'win32'", + "psycopg2-binary>=2.9.10 ; sys_platform != 'win32'", + "GDAL @ https://github.com/cgohlke/geospatial-wheels/releases/download/v2025.7.4/gdal-3.11.1-cp312-cp312-win_amd64.whl ; sys_platform == 'win32'", +] + +[project.optional-dependencies] +dev = [ + "pytest>=8.0", ] [project.urls] repository = "https://github.com/dbetchkal/NPS-ActiveSpace" -[tool.setuptools] -py-modules = ['nps_active_space'] +[tool.setuptools.packages.find] +where = ["."] +include = ["nps_active_space*"] +exclude = ["legacy_code*", "tests*", "example_data*"] + +[tool.setuptools.package-data] +nps_active_space = [ + "config/template.config", + "config/*_example.config", + "data/**/*", + "img/*", +] diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index f4f6916..0000000 --- a/requirements.txt +++ /dev/null @@ -1,85 +0,0 @@ -affine==2.4.0 -asttokens==3.0.0 -attrs==25.3.0 -beautifulsoup4==4.13.4 -certifi==2025.7.9 -charset-normalizer==3.4.2 -click==8.2.1 -click-plugins==1.1.1.2 -cligj==0.7.2 -colorama==0.4.6 -comm==0.2.2 -contextily==1.6.2 -contourpy==1.3.2 -cycler==0.12.1 -debugpy==1.8.14 -decorator==5.2.1 -distlib==0.3.9 -docopt==0.6.2 -executing==2.2.0 -filelock==3.18.0 -fiona==1.10.1 -fonttools==4.58.5 -future==1.0.0 -GDAL @ https://github.com/cgohlke/geospatial-wheels/releases/download/v2025.7.4/gdal-3.11.1-cp312-cp312-win_amd64.whl#sha256=e328c144311ae5dda09805a9fd77517ed5568177a993e23aa84af05b37e140bc -geographiclib==2.0 -geopandas==1.1.1 -geopy==2.4.1 -greenlet==3.2.3 -idna==3.10 -ipykernel==6.29.5 -ipython==9.4.0 -ipython_pygments_lexers==1.1.1 -iyore @ git+https://github.com/nationalparkservice/iyore.git@b3b1e3fd025a392bf63bf6c4a8a4eebb887291ec -jedi==0.19.2 -joblib==1.5.1 -Js2Py==0.74 -jupyter_client==8.6.3 -jupyter_core==5.8.1 -KDEpy==1.1.12 -kiwisolver==1.4.8 -matplotlib==3.10.3 -matplotlib-inline==0.1.7 -mercantile==1.2.1 -nest-asyncio==1.6.0 -numpy==2.3.1 -packaging==25.0 -pandas==2.3.1 -parso==0.8.4 -pillow==11.3.0 -pipwin==0.5.2 -platformdirs==4.3.8 -prompt_toolkit==3.0.51 -psutil==7.0.0 -psycopg2==2.9.10 -pure_eval==0.2.3 -Pygments==2.19.2 -pyjsparser==2.7.1 -pyogrio==0.11.0 -pyparsing==3.2.3 -PyPrind==2.11.3 -pyproj==3.7.1 -pySmartDL==1.3.4 -python-dateutil==2.9.0.post0 -pyvista==0.46.3 -pywin32==310 -pyzmq==27.0.0 -rasterio==1.4.3 -requests==2.32.4 -scipy==1.16.0 -setuptools==80.9.0 -shapely==2.1.1 -six==1.17.0 -soupsieve==2.7 -SQLAlchemy==2.0.41 -stack-data==0.6.3 -tornado==6.5.1 -tqdm==4.67.1 -traitlets==5.14.3 -typing_extensions==4.14.1 -tzdata==2025.2 -timezonefinder==8.2.4 -tzlocal==5.3.1 -urllib3==2.5.0 -wcwidth==0.2.13 -xyzservices==2025.4.0 diff --git a/setup.cfg b/setup.cfg deleted file mode 100644 index e69de29..0000000