Visualizing and Investigating Distances to Understand the Reliability of Weather Data

[rectheworld]

Visualizing and Investigating Distances to Understand the Reliability of Weather Data

Goals

Goal: We have been able to match a subset of our volunteer submitted weather data to NOAA weather data. However, we have done no filtering on the distance between a cbc and its closest NOAA station.

Inputs

cbc_cleaned_usa_merged.csv - Contain volenteer submitted cbc data and the weather data for the closest noaa station. Distances are measured in meters.

Outputs

• A visualization of how distance affects weather matching between volunteer submitted data and noaa data
• Suggestion on what distances should be used to determine if volunteer submitted weather data is accurate at a cbc.
• ipython notebooks are preferred output file

Inspiration and Support

From Francisco Vannini:

Would be nice to get a plot of this sort
https://plot.ly/python/choropleth-maps/

If we could have some dynamic parameter so that when the parameters change(day,month,year) then so would that map that would be even better.
https://medium.com/@williamkoehrsen

From Ian Davis

Hi all, I'd like to add that we will need to consider how far away the closest NOAA station is. It turns out there is someone in "Midway Atoll" (out in the middle of the Pacific Ocean!) who collected data for a number of years. The closest station distance is 1.9e6 m (~1180 miles)
EDIT:
So I did a quick Google Search asking at what radial distance a temperature measurement would still be considered accurate. The answers seem to be quite varied and depend on the terrain, among other things. I don't think we'll be able to consider those factors. However, this one particular discussion referenced a paper from the World Meteorological Organization (WMO) about weather observations. One line states "but for small-scale or local applications the considered area may have dimensions of 10 km or less." Not sure if that's enough for use to go on, but it's a start.
https://www.researchgate.net/post/Data_and_weather_station_distance
https://www.researchgate.net/deref/https%3A%2F%2Fwww.wmo.int%2Fpages%2Fprog%2Fwww%2FIMOP%2Fmeetings%2FCB%2FEd-Board-2%2FEdBd-2_P-I_Ch-1.doc
Using the paired.csv and setting the distance limit to 10 km, the number of rows drops from 53811 to 38178. A drop of 29%. (edited)

[ijd5004]

I do not know how to assign this issue to myself, but I would like to take ownership of it.

• Ian Davis

[ijd5004]

Considering the sensitivity of the temperature to the terrain, I thought we might be able to compare the elevations of the GPS locations between "circle" and NOAA station.

Does anyone know if that data is available from NOAA?

I looked for some packages to pull elevation given GPS coordinates. There seems to be a few out there, but the user interface is not great:

• geocode: would have probably been the easiest, but built using Google maps APIs and now that costs money
• open-elevation-api: requests are timing out; not sure if it works (https://github.com/Jorl17/open-elevation/blob/master/docs/api.md)

Not sure if this is viable yet...or worth it. Input welcome.

[ijd5004]

Forgot about this one from "USGS Elevation Point Query Service"
(https://stackoverflow.com/questions/58350063/obtain-elevation-from-latitude-longitude-coordinates-with-a-simple-python-script)

I did an example request that returned data, but the values were nonsensical.

[rectheworld]

The infomation of the noaa stations came from the BigQuery ghnc database where the only info on station is Lat, Long, the measurements that station takes, Year Data started collection, year ended. T

Thus our official source has no info on elevation.

[Frankie-Figz]

Maybe here?

https://gis.stackexchange.com/questions/228920/getting-elevation-at-particular-coordinate-lat-lon-programmatically-but-offli

[Frankie-Figz]

Oh and the big query dataset has a table where information of each NOAA station is recorded; amongst that information is the elevation.

Worst case scenario you would have to manually input the elevation for each of the circles.

[rectheworld]

Is that all stations or current stations? I thought that file was only current stations, so would not have data for stations that were older.

…

On Thu, Feb 6, 2020 at 3:49 PM Francisco J Vannini R < ***@***.***> wrote: Oh and the big query dataset has a table where information of each NOAA station is recorded; amongst that information is the elevation. [image: image] <https://user-images.githubusercontent.com/39884538/73977345-31093e00-48f8-11ea-8958-8906ae69703b.png> Worst case scenario you would have to manually input the elevation for each of the circles. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#35?email_source=notifications&email_token=ACS2567FNYEYESUFH3AO2IDRBRZWLA5CNFSM4KPZE2IKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELAXLWI#issuecomment-583103961>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACS2564QXE7GVYFUG3THDVDRBRZWLANCNFSM4KPZE2IA> .

[Frankie-Figz]

I can perform that check on the table and let you know.

[ijd5004]

Thanks for taking a look @rectheworld and @Frankie-Figz !

[richard93johnson]

Hey all, I am kind of reading through the material. I am wondering if we could use the accuracy of the NOAA weather data vs the volunteer inputs and run that against the various distances to each station to establish our own threshold. So if we have an overwhelming amount of data is inaccurate beyond let's say 50 miles, then we just use 50 miles r as the cutoff. Let me know if that's the direction we are already heading?

[ijd5004]

Hi @richard93johnson, that's an interesting idea. I honestly do not know which way would be a more valid process. Do you know of a precedent for something like this? (in general terms)

We have something like...we care about the difference between A and B. There is another variable C, which may have some bearing on the relationship between A and B. How do we use the value of C to filter for valid differences between A and B?

Maybe we could make a scatter plot of the temperature difference vs. station distance. Give us a first glance at that possible relationship.

[richard93johnson]

Awesome. Let me spend a bit of time on this and I’ll get back to you. Just a heads up, I’m on this project to learn Python at the same time, so I won’t nearly as swift as other more experienced developers.

[richard93johnson]

Won’t *be

[ijd5004]

No worries. If you have any questions, let me know.