CSB/VBI Tools Workshop at CSBWG16

[brian-r-calder]

In addition to the IHO working group meeting in Wellington, NZ (26-28 March 2025), there is going to be a practical workshop on CSB/VBI tools (24-25 March 2025) in the same venue. The first day of the workshop is going to have "state of the art" assessments on the various stages of data capture and processing by various experts, along with an expert panel and hands-on time with hardware and software data collection and processing systems. The second day of the workshop will have on-the-water demonstration time, but also a full-day development session (hackathon) for developers to work with different open source toolsets for dealing with CSB/VBI data. Details in the attached flier.

The development workshop is being supported by CCOM/JHC at the University of New Hampshire, and we're looking for input on a number of different parts of the second day's events. Particularly:

1. We want to do some actual development work, so we'll need a base library on which to work. What library do we want to use as a base? We're partial to OpenVBI (of course), and it would make a good development base, but are there other libraries that we should consider?
2. We want to build and test code. What development environment do we want? We're thinking Python 3.12, Git, and GitHub (and maybe VSCode). But should we maybe consider working in a development container, or some other structure?
3. We're likely to have some people participating remotely so we'll need some sort of collaboration tool. There are any number of collaboration platforms from Teams to Discord to Slack. Is there a favourite, or a preferred solution that would provide GitHub integration, messaging, and video conferencing?

There are probably many other things to discuss, but let's start the conversation here, and see how it goes. There's also a very simple website to provide some more details on the workshop, and links for registration, background documents, etc.

CSB_Event_NewZealand_2025.pdf

[MathieuRondeauCHS]

The Canadian Hydrographic Service (CHS) Community Hydrography Program is seeking a coding strategy for filtering/cleaning CSB datasets. Below is an image illustrating a use case of a received CSB dataset, which displays numerous erroneous soundings. These could be abnormally shallow (SW to NE purple winding line) as well as excessively deep (S to N vertical blue line).

[selimnairb]

@MathieuRondeauCHS Did you have an algorithm in mind, or do you intend to work that out during the workshop?

[MathieuRondeauCHS]

Hi Brian. I currently don't have a specific algorithm in mind. Our limited experience indicates that a high Speed Over Ground (SOG) often leads to poor soundings. That's why we believe it's crucial to include SOG as mandatory data, along with Longitude, Latitude, Depth, and Date & Time. However, the problem is more complex, and poor soundings can persist even at reasonable SOG.

[brian-r-calder]

So: different Brian, but ... There is a generic filter interface and filters in OpenVBI already for time and depth, but it might be interesting to include functionality for SOG computation (better to log, but we could approximate from position and timestamps) and then add a filter implementation that uses it for point selection. That won't get everything, naturally, but it might be a useful further tool to add. And something that would be eminently implemented in the available time.

[anthonyklemm]

We've been working on trying to use the reference bathy (like NONNA or BlueTopo) to help detect these contiguously erroneous transits (or parts of transits). We see them typically when a vessel reaches an area too deep for their echosounder's bottom tracking operational limits or settings so it tries to track bottom but it really isn't detecting all the way down, like when they cross the Mississippi River Canyon in the Gulf. We want to test methods for simple raster-to-raster comparison (using the transit_id and point uuids to link back to the actual point data for outlier flagging in the DB), which is typically less computationally expensive than sampling raster values to intersecting vector points. Although fancier vectorized libraries such as xarray and rioxarray could help.

[anthonyklemm]

Just to follow up on this a bit. Since I now sample the reference bathymetry depth value for every coterminous CSB point feature, I tried a very simple filter excluding the processed points during final product creation where the difference between the fully processed CSB depth and the reference bathmetry is greater than three times the calculated vertical uncertainty of the CSB data (not sure if that's the right multiplier number but it a start). We calculate the CSB vertical uncertainty right now using the max allowable CATZOC C definition. The results seem pretty good at getting rid of the contiguously erroneous data. It's not a perfect solution but one that is simple and quick to perform during extraction from the processed csb spatial database.
query = f""" SELECT lat, lon, outlier, depth_mod as depth FROM {table_name} WHERE lat BETWEEN {miny} AND {maxy} AND lon BETWEEN {minx} AND {maxx} AND outlier is false AND ABS(Raster_Value - depth_mod) <= (uncertainty_vert * 3) """

[heathhenley]

But should we maybe consider working in a development container

I took a first pass at setting up a python 3.12 one here: #3

[HaleyAddison]

Hi all,

I would love to help create some sort of web-based tool that generates a metadata sheet for vessel processing. I was envisioning a simple webpage with fillable boxed titled with each of the data input sections for metadata files such as “Vessel Draft” and then a little information popup that can give more info from the B-12 guidelines like “This will be used to calculate offsets by adding the draft to the overall depth” for example. This could help eliminate mistakes for things like "vessel drafts" vs "vessel offsets" vs "vertical reference", as well as creating a more streamlined and accessible process.

Then after you fill all of the boxes, click “Generate” and it spits out a “VesselName_Metadata.JSON”

[selimnairb]

@HaleyAddison Could you explain a little more about the desire to have this be a web-based tool? Would also work as a desktop tool that had a graphical user interface? I ask because there could be benefit in having a tool like you suggest be part of wibl-python or the growing OpenVBI tool suite. It could start off by just generating the vessel metadata, but then it could grow to support other functions, some of which may benefit from being able to run OpenVBI algorithms directly. Having it be a web-based tool complicates that as doing this level of integration would require more engineering and hosting all this in the web would have a non-trivial cost.

[HaleyAddison]

Hi, yes, I am more than happy for it to be a desktop-based tool. I think the GUI is the most important bit. I only thought website because I thought that'd be the simplest, but I am happy for it be whatever is the simplest!

[brian-r-calder]

I pulled together the suggestions here, and summarised them with Brian (Miles), and then started a poll (#5) so that we can vote on what topics would be of interest to the community for our precious development time. Please vote for your favourites, and encourage your colleagues to do the same! We'll close polling in a week from now (2025-03-17) so that we can refine the topics a week prior to the workshop and do some final assignments of people to projects.

[brian-r-calder]

The workshop agenda has now been finalised (attached), and can also be found at the website through the top menu, or a couple of links in the text.

2025-03-15_CSBToolsWorkshop_Agenda.pdf

[brian-r-calder]

In order to avoid confusion with future workshop idea development, this discussion is being closed; all of the ideas here were captured as work items in the issues queue.