Updated Nodebooks - use changelog to monitor any needed changes for the NMT lessons

[jdpye]

Changelog is here:

https://gitlab.oceantrack.org/otn-partner-nodes/ipython-utilities/-/blob/master/CHANGELOG.MD?ref_type=heads#20250409

A giant list of minor and major updates to go through and see if there are potential effects on the NMT lessons. Cross out items below as they are confirmed to have had no effect on lessons:

[2025.04.09]

Added

• check_nodes.py - script added to check obis.moorings transmitter records to c_vemco_tag inventory tables in another node.
• pre-commit - allows for formatting and cell output clearing to occur before code commits are made.
• Argos_FastGPS_Locations_to_Telemetries_process.ipynb - Notebook used to format a folder of argos gps data into telemetry files that can be processed in the movers pipeline.
• Argos_GPS_SSM_fitting.ipynb - R notebook to run aniMotum's SSM_fitting function.
• Auth - Create and Update Plone KDBX.ipynb - Notebook to create encrypted kdbx Plone authorization files.
• Check Sensors for Movers.ipynb - use this notebook to check for sensor tag discrepancies in legacy moving platform data sets.
• environment_r.yml - (MAC, Linux) - Added a new environment config to build a R notebook kernel.
• HR detections - 1 - load csv detections.ipynb - Loading detections from HR (High resolution) receivers into summarized database tables.
• Movers 1 through 4 Notebooks - redesigned movers worker flow notebook names have since changed.
• remora_notebook.ipynb - (R) Remora QC notebook to process OTN formatted detection extract files.
• common.py
  • CustomEngine added to workaround issues when using sqlalchemy's engine object. Adds our custom variables for better IDE code-completion.
  • drop_view function - used to drop a view
  • has_view function - returns a bool of True if the view exists
  • has_duplicate_values function added to check if a table has duplicates.
• common_extract.py - new module created to hold common detection extract functions.
  • get_repo_folder return a repository folder string if node is provided
• deploy - load deployment metadata.ipynb - added functionality to assign transmitters to receiver metadata.
• detections - 1 - load csv detections.ipynb - added a section to check for HR detectons.
• discovery_process_reload.ipynb - added cell documentation outlining the cell's operation and tables affected.
• argos_quality_control.py - ARGOS data quality control scripts.
• connect_plone.py - Plone site auth loading functions used to create PloneAuth objects needed by the plone_bridge module.
• create_plone_auth.py - business logic used by the Auth - Create and Update Plone KDBX.ipynb notebook to create and update Plone kdbx files.
• csv_to_parquet.py - Test development implementation of the database parquet conversion process. Not currently being used but exists for reference.
• database_common.py - Initial refactor of common database functions into a single file. A few functions have been currently ported over this module and more will be ported as the code is reworked.
• extract_import - imports used by the detections - create detection extracts.ipynb notebook. Used to simplify the in-notebook code.
• fetch_detecting_collections.py - helper function to pull collectioncodes for a given tracker code.
• fetch_remote_tracker_detections.py - helper function that creates the SQL to pull remote tracker detections.
• function - a common script to place all the database functions that interact with database function objects. Currently has the has_function function.
• get_pi_and_poc.py - Part of the detection extract process. It return a dataframe of the PI and POC to be joined to the extract output.
• hr_load_summarized_detections - Functions to create summarized detection tables form HR receivers.
• load_excel.py - Moving all the excel loading functions in the Load Excel.ipynb to a python module.
• moving_platform_fixes.py - functions to perform moving platform sensor value fixes.
• new_external_partner.py, new_qualified.py, new_qualified_ext.py, new_unqualified.py, new_sentinel.py, new_tracker.py, new_sentinel_tag_owner.py - Represent all the rewritten and new detection extract generating code.
• to_innovasea_conversions.csv - File created to track vendor code-space to innovasea formatted code-space conversions
• c_tag_meta_to_cache.py, cache_to_otn.py, cache_verify.py - Code rewritten to handle non acoustic tag loading.
• discovery_process.py - create_station_history_missing_columns function included to add the new columns to stations_history_rows: map_code and rcvr_serial.
• extract_dispatcher.py - Refactored to remove most of the hard-coded values and streamline the codebase.
• load_tagging.py - added more verifications on column types.
• populate_geoserver.py - Now includes geoserver species table creation and population scripts.

Changes

• Detection extracts have been totally redesigned.
  • Column names have been changed using one camelcase naming convention.
  • uncorrected datetime has been included to each extract. This time is the value before time corrections have been applied
  • A parquet file is now outputted alongside the csv output.
  • Z - Sentinel Tag Owner extract has been added as an option for detection extract to create.
  • detections - create detection extracts.ipynb documentation has been updated to add clarity.
• connect_db.py
  • CustomEngine object made to help better resolve code completion and in the future add missing functionality
  • Adds Nodebook version to the application_name value
  • Additional type hints added
• Changes to the language in the new_feature gitlab issue template
• Repopulate_geoserver_stations.sql - added rcvr_serial and map_code to the station_receivers and station_series tables. (awaiting migration before feature can be merged)
• c_detections_to_detections_yyyy.py - code added to handle sensor_match_yyyy_moves tables
• cache_to_otn.py - added check and instructions to insert_tagcahce_to_otn_transmitters function to inform on using db-fix notebooks if multiple uses found for a transmitter.
• common.py
  • Docstring added to the Color class.
  • Improved docstring for schema_to_lower
  • function get_node now has type hints
• compare_dets_telem.py - Removed hard-coded column names. Changes to prioritize platform_id
• Docker image building (Python notebooks only) has been updated to use docker compose.
• .gitlab-ci.yml - updated CI image to use OTN docker hub account - includes. Removed all unreliable notebook CI tests because execution stability could never be assured when using the playwright headless browser.
• the plonetools folder has been removed in favour of splitting the plone_bridge into a separate repository. plone_bridge is installed when the environment.yml is resolved.
• Check for Database Contacts Plone access.ipynb - Now only works for the Plone 6 version of the Plone RestAPI. Fixed the code using the external version of plone_bridge.
• Create and Update Projects.ipynb - Buffer value added to single point project locations.
• insert_vendor_sheet.ipnb - Added checkFieldNumbers function to check tags in other nodes using your connection's dblink records.
• Load Health Report.ipynb - Code added to rename columns in the health report raw data.
• Publication Control Table Update.ipynb - refined the input process to require less user interaction in the code cells.
• publish - collections to OBIS.ipynb - redesigned the OBIS publishing steps.
• plonetools functions have been moved into the dbtool folder.
• detemin_orgid.py moved to the verification methods module.
• pub_cont_update.py - removed
• animal_n_tag_verify.py - updates include new functions to process non-acoustic tags.
• mirror_schema.py - removed all create_from_model calls for database built-in model functions.
• mock.py - dataclasses added to Class definitions. Changed made for post uncorrected database migration.
• moving_platforms.py - Various changes added from mover fixes milestone.
• otn_detections_yyyy_tag_matching.py - Tag matches now can match cross all vendors using various code-space conversions.
• sensor_tag_process.py - general updates and added features for the movers pipeline.

Fixes

• active_tracking.py - removed test. schema testing code...changed to {schema}
• Animal Project Report Visualization.ipynb - fixed test string and typos
• Batch Plone downloader - updated using external plone_bridge module.
• Check Project Activity Through Plone.ipynb - updated using external plone_bridge module. Fixed any part that wasn't previously working correctly.
• Create Plone Folders and Add Users.ipynb - Updated for external plone_bridge.
• c_detections_to_detections_yyyy.py - added check from detection table before attempting to load sensor_match records from that year.
• c_events_to_events.py - Now creates missing try_cast_timestamp in the public schema instead of using temp.
• compare_dets_telem.py - Changed to use dynamic serial column names values. Read encoding set to utf-8
• missing_vrl_check.py - Fix for null end dates
• sendemail.py - Added contact project end date checks. Code updated to utilize Plone 6 api paths.

Removed

• plonetools folder has been removed in favor of using the external module plone_bridge.