Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions _episodes/01_introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ OTN partners with regional acoustic telemetry networks around the world to enabl

### How does a Node benefit its users?

OTN and affiliated networks provide automated cross-referencing of your detection data with other tags in the system to help resolve "mystery detections" and provide detection data to taggers in other regions. OTN Data Managers perform extensive quality control on submitted metadata to ensure the most accurate records possible are stored in the database and shared with researchers. OTN's database and Data Portal website are well-suited for archiving datasets for future use and sharing with collaborators. The OTN system and data workflows include pathways to publish datasets with the Ocean Biodiversity Information System, and for sharing via open data portals such as ERDDAP and GeoServer. The data product returned by OTN is directly ingestible by populare acoustic telemetry data analysis packages including `glatos`, `actel`, `remora`, and `resonATe`. In addition to the data curation materials, OTN offers continuous support and workshop materials detailing the use of these packages and tools.
OTN and affiliated networks provide automated cross-referencing of your detection data with other tags in the system to help resolve "mystery detections" and provide detection data to taggers in other regions. OTN Data Managers perform extensive quality control on submitted metadata to ensure the most accurate records possible are stored in the database and shared with researchers. OTN's database and Data Portal website are well-suited for archiving datasets for future use and sharing with collaborators. The OTN system and data workflows include pathways to publish datasets with the Ocean Biodiversity Information System, and for sharing via open data portals such as ERDDAP and GeoServer. The data product returned by OTN is directly ingestible by popular acoustic telemetry data analysis packages including `glatos`, `actel`, `remora`, and `resonATe`. In addition to the data curation materials, OTN offers continuous support and workshop materials detailing the use of these packages and tools.

Below is a link to a presentation from current Node Managers, describing the relationship between OTN and its Nodes, the benefits of the Node system as a community outgrows more organic person-to-person sharing, as well as a realistic understanding of the work involved in hosting/maintaining a Node.

<!-- check if presentation is accurate MG to Jon Y/N -->
[PowerPoint Link](https://docs.google.com/presentation/d/1ZK58KCS_Zz8qHz5IYrBGYILJ-ZQYfkeo/edit?usp=sharing&ouid=108426455008350886281&rtpof=true&sd=true)

## Node Managers
Expand Down
6 changes: 3 additions & 3 deletions _episodes/02_data_requests.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,13 @@ OTN recommends creating a "Data Request Response Policy".


## External Requests - Restricted Data

<!-- Should we update to check if a project is public by default in line with recent policy update MG to Jon Y/N -->
This is an example of how OTN handles these requests:

1. Request for data (from person other than data owner) submitted
2. Data request is scoped, in a GitLab Issue. All details from requester is included.
2. Data request is scoped, in a GitLab Ticket. All details from requester is included.
3. Impacted PIs are identified, and contacted, seeking written permission for requester to access the information.
4. Written permission is documented in GitLab Issue, to preserve the paper-trail.
4. Written permission is documented in GitLab Ticket, to preserve the paper-trail.
5. Data request report is compiled and provided to requester, once all permissions have been received.


Expand Down
8 changes: 5 additions & 3 deletions _episodes/03_OTN_System_and _structure_and_Outputs.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,16 @@ Affiliated acoustic telemetry partner Networks may become an OTN Node by deployi
# Basic Structure

The basic structural decision at the centre of an OTN-style Database is that each of a Node's projects will be subdivided into their own database `schemas`. These schemas contain only the relevant tables and data to that project. The tables included in each schema are created and updated based on which types of data each project is reporting.

<!-- Line 29: Where in project metadata is this delineated? MG to PERSON Y/N -->
<!-- Line 29: Is selected in Create and update projects.ipynb by Node Manager MG to MG Y/N -->
Projects can have the type `tracker`, `deployment`, or `data`.
- Tracker projects only submit data about tag releases and animals. They get tables based on the tags, animals, and detections of those tags.
- Deployment projects only submit data about receivers and their collected data. These projects get tables related to receiver deployments and detections on their receivers.
- Data projects are projects that deploy both tags and receivers and will submit data related tags, animals, receivers, and detections and will get all the related tables.

In addition to the project-specific `schemas`, there are some important common schemas in the Database that Node Managers will interact with. These additional schemas include the `obis`, `erddap`, `geoserver`, `vendor`, and `discovery` schemas. These schemas are found across all Nodes and are used to create important end-products and for processing.
- The `obis` schema holds summary data describing each project contained in the Node as well as the aggregated data from those projects. When data goes into a final table of the project schema it will be inherited into a table in `obis` (generally with a similar name).
<!-- Line 35: Can we clarify that 'obis' is legacy naming and not directly related to OBIS MG to PERSON Y/N -->
- The `erddap` schema holds aggregated data re-formatted to be used to serve telemetry data via an ERDDAP data portal.
- The `geoserver` schema holds aggregated data re-formatted to be used to create geospatial data products published to a GeoServer.
- The `vendor` schema holds manufacturer specifications for tags and receivers, used for quality control purposes.
Expand Down Expand Up @@ -122,7 +124,7 @@ Even though `tag`, `deployment`, and `detections` data all have their own loadin
- Their data workflows all begin with a submission of data or metadata files from a researcher.
- The Node Manager ensures there is a copy of the file on the Node's document management website.
- The Node Manager carries out visual quality control to catch any obvious errors.
- The data is then processed through the relevant OTN Nodebooks. This process is outlined by the task list associated with the GitLab Issue made for this data.
- The data is then processed through the relevant OTN Nodebooks. This process is outlined by the task list associated with the GitLab Ticket made for this data.
- The data will first be loaded into the "raw" tables. This is the table that holds the raw data as submitted by the researcher (the naming convention for raw tables is that they always have the prefix `c_` and will have a suffix indicating the date it was loaded, typically `YYYY_MM`).
- After the raw data table is verified, the data will move to the "intermediate" tables which act as a staging area for partially-processed data.
- After the intermediate table is verified, data will move to the "upper" tables, where the data is finished processing and is in its final form. This is the data that will be used for aggregation tables such as `obis` and for outputs such as Detection Extracts.
Expand All @@ -138,7 +140,7 @@ In order to create meaningful Detection Extracts, OTN and affiliated Nodes only
- Summary schemas like `discovery`, `erddap`, and `geoserver` are updated with the newly verified data.

Summary schema records can be used to create maps and other record overviews such as this map of active OTN receivers:

<!-- Still factual? MG to PERSON Y/N -->
<img src="../fig/active_receivers.JPG" alt="Summary Map" style="width:500px;"/>

# Backing Up Your Data
Expand Down
2 changes: 1 addition & 1 deletion _episodes/04_Setup_and_Installing_Needed_Software.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ In the next lesson we will practice using our database console viewer and connec
In order to work efficiently as a Node Manager, the following programs are necessary and/or useful.

## Cross-Platform

<!-- Do we want to update with Positron also? MG to PERSON Y/N -->
**Visual Studio Code** - An advanced code editing integrated development environment (IDE). Also contains extensions that can run JuPyTeR notebooks, open CSV files in a visually appealing way, as well as handle updating your Git repositories.
* [https://code.visualstudio.com/](https://code.visualstudio.com/)

Expand Down
Loading