Outlier detection/Data filtering/Cleaning tool

[ecarle-metservice]

Automated tool to filter / clean CSB datasets

Data issues to address

1. Point Outliers - depths are erroneously too large/too small relative to neighbouring data points
2. Transect Outliers - a collection/transect of points are all too large/too small relative to other data/transects
3. Stalled logger - depths are recorded as single erroneous value for an extended period
4. Oversampling - remove multiple repeated depth readings that are at the same / nearby location

Potential Cleaning Processes
Note this should be modular, you can decide which of the processes to run

• Basic Checks: filter depths/lat lon/time out of some defined acceptable range
• Speed over ground checker: compute approximate speed from timestamp/lat/lon and remove points w/speed > some kt
• Point Outlier cleaning: use point cloud detection tools, e.g. Open3D to filter outliers
• Stalled logger: if point and n nearest neighbours are same value/close to same value, then remove them
• Reference dataset comparison: compare dataset with some reference bathymetry (could be point, grid, or chart data). Compare dataset points with overlapping/"nearby" reference points/grid cells. Remove points which have disparity larger than some threshold

[ecarle-metservice]

Motivated by #2 (comment)

[JeremieLemarchand]

Hello everyone,

We have tested at the Canadian Hydrographic Service a basic filtering strategy for successive redundant depths (stalled logger and/or oversampling). It seems to work from decently to better than expected on the datasets we have. The algorithm just removes successive redundant depths and then tries to remove resulting isolated group of points based on expected distance (requires logged SOG value) from previous and next group of soundings. Soundings removed seems to be often correlated with bad data. Here’s an example in the St-Lawrence river, from whale watching boat tracks.