Skip to content

Commit b4266fa

Browse files
authored
Add openmesh (#6)
Add the OpenMesh dataset examples with OpenSense format, including wireless microwave links, personal weather station (PWS), and ASOS reference station example files, along with notebooks demonstrating data access and analysis with opensense packages. * Add OpenMesh dataset with CML, PWS, and ASOS example files - Add 1d, 1w, 20d sample files for CML, PWS, and ASOS - Add notebooks: create, analyze, and ASOS processing - Add openmesh_data_io.py and ws_opensense_conversion.py helpers - Add README for OpenMesh folder - Update main README with available datasets - Gitignore notebooks/data/ (raw downloads, not tracked) - Remove previously tracked data files from git * Fix gitignore and remove tracked data/DS_Store files * Fix notebook errors and save guard in analyze_OpenMesh_data.ipynb - Cell 7: assign known ASOS station coordinates (lon/lat were NaN in .nc file due to ID mismatch between data short IDs and K-prefixed metadata) - Cell 35: wrap interpolate_precipitation in try/except ValueError to handle pypwsqc bug with irregular time coordinates (shape mismatch) - Cell 43: remove duplicate ASOS coord fix (now handled in cell 7) - Cell 45: gate save logic under 'if save:' flag (was always executing) - README: fix double space * Fix ASOS station coordinates in openmesh_asos_ws_20d.nc lon/lat/elev were all NaN due to ID mismatch between the short station IDs used in the data (JFK, EWR, LGA, NYC) and the K-prefixed ICAO IDs in the metadata CSV (KJFK, KEWR, KLGA, KNYC). Added correct coordinates from known station locations. * Fix ASOS variable names and update notebook execution timestamps Rename precip_amount/precip_rate to rainfall_amount/rainfall_rate in openmesh_asos_ws_20d.nc to follow OpenSense naming convention; add long_name note that values may represent non-rainfall precipitation. * Rename precip_amount/rate to rainfall_amount/rate in all notebooks * Improve QC cell output and split baseline estimation into description, config, and implementation cells
1 parent e7bcf2c commit b4266fa

15 files changed

+7302
-2
lines changed

.gitignore

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
11
.ipynb_checkpoints
22
OpenMRG/notebooks/data
33
OpenRainER/notebooks/data
4-
*.pyc
4+
OpenMesh/notebooks/data
5+
*.pyc
6+
.DS_Store
7+
.idea/

OpenMesh/README.md

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# OpenMesh
2+
3+
OpenMesh is a dataset of microwave link and personal weather station (PWS)
4+
measurements over New York City, covering January 2024. It also includes ASOS reference station
5+
data for validation. The raw data is sourced from two Zenodo records (jacoby_2025_OpenMesh).
6+
7+
8+
## Example files
9+
10+
The root folder contains ready-to-use NetCDF sample files at three time windows.
11+
File names follow the convention: openmesh_{sensor}_{duration}.nc
12+
13+
CML files (75 links, received signal level in dBm):
14+
openmesh_cml_1d.nc one day
15+
openmesh_cml_1w.nc one week
16+
openmesh_cml_20d.nc 20 days
17+
18+
PWS files (35 Weather Underground stations, rainfall, temperature, humidity, wind, pressure):
19+
openmesh_wu_pws_1d.nc one day
20+
openmesh_wu_pws_1w.nc one week
21+
openmesh_wu_pws_20d.nc 20 days
22+
23+
ASOS reference station files (airport weather stations, used for comparison and validation):
24+
openmesh_asos_ws_20d.nc 20 days
25+
26+
27+
## Notebooks
28+
29+
create_OpenMesh_data.ipynb
30+
Downloads the raw CML and PWS data from Zenodo into notebooks/data/jacoby_2025_OpenMesh.
31+
Generates sample NetCDF files for any time window using the functions in openmesh_data_io.py.
32+
Also produces the root example files above.
33+
34+
analyze_OpenMesh_data.ipynb
35+
Loads CML and PWS datasets either from the root example files or by creating them on the fly
36+
from the raw data for a custom time window. Provides an overview of the dataset dimensions,
37+
link/station metadata, and basic visualizations of signal level and weather variables.
38+
39+
ASOS_opensense.ipynb
40+
Downloads and processes ASOS airport station data (JFK, EWR, LGA) for the same time window
41+
as the CML and PWS data. Converts the data to the opensense NetCDF format and saves it to
42+
the samples folder. Used to generate the reference files for validation.
43+
44+
45+
## Helper modules
46+
47+
openmesh_data_io.py
48+
Central module for all data operations. Key functions:
49+
- download_openmesh() and download_pws_wu(): download and extract Zenodo archives
50+
- create_openmesh_sample() and create_pws_wu_sample(): slice a time window from raw data
51+
- create_asos_sample(): fetch and convert ASOS data for a given time window
52+
- create_all_datasets(): run all three in one call, with save and resample options
53+
- load_cml() and load_pws_wu(): load the full raw source files into memory
54+
55+
ws_opensense_conversion.py
56+
Handles unit conversions and variable transformations for PWS data, converting raw
57+
Weather Underground fields to the opensense standard format.
58+
59+
60+
## Data folder
61+
62+
The raw downloaded data is stored in notebooks/data/jacoby_2025_OpenMesh and is not tracked
63+
by git. Running create_OpenMesh_data.ipynb will populate it automatically. The layout is:
64+
65+
raw/ full source NetCDF files downloaded from Zenodo
66+
meta/ station and link metadata CSVs, and HTML map visualizations
67+
archived/ original ZIP archives from Zenodo
68+
examples/ example notebooks included in the Zenodo release
69+
docs/ README from the Zenodo release

0 commit comments

Comments
 (0)