Pkg: Include Incidental Observations

[ericnewkirk]

Mentioned by a bio a couple weeks ago - should users still enter incidental observations from their surveys into Speedgoat? If so do we include those here, potentially as Observer Activity 1 (casual observation)? Right now only observations of the target species can be included: wos_export.R#L94.

[Huh]

For future reference, the desire to only return the target species is hard coded in the wos_read function. The code can be found at wos_read line 50

[ericnewkirk]

Following up after some testing, it looks like the species filter and is_target_species filter conflict somehow. I understood species as the target species of the survey, but it looks like the API applies it to the actual species observed. Since survey types are species specific we could probably omit the species filter and just use survey type to get the incidental sightings in the results.

Filtering to non-target with no other filters gives the first 100 non-target observations:

https://counts.spdgt.com/api/aerial-surveys/entries-with-designs?filter[is_target_species]=false

{
  "schema": {...},
  "data": [
    {
      "entry_id": 148040,
      "survey_id": 3246,
      "design_id": 132320,
      "project_id": 2,
      "survey_type_id": 9,
      "species_id": 1,
      "species": "White-tailed Deer",
      "is_target_species": false,
      ...
    },
    ...
  ],
  "meta": {...}
}

Filtering to non-target with the other filters applied as in spdgt.wos::wos_read returns no data:

https://counts.spdgt.com/api/aerial-surveys/entries-with-designs?filter[species_id]=1&filter[survey_type_id]=9&filter[analysis_unit_id]=347&filter[bio_year]=2022&filter[is_target_species]=false

{
  "schema": {...},
  "data": [],
  "meta": {...}
}

Removing the species filter gets the data back:

https://counts.spdgt.com/api/aerial-surveys/entries-with-designs?filter[survey_type_id]=9&filter[analysis_unit_id]=347&filter[bio_year]=2022&filter[is_target_species]=false

{
  "schema": {...},
  "data": [
    {
      "entry_id": 148040,
      "survey_id": 3246,
      "design_id": 132320,
      "project_id": 2,
      "survey_type_id": 9,
      "species_id": 1,
      "species": "White-tailed Deer",
      "is_target_species": false,
      ...
    },
    ...
  ],
  "meta": {...}
}

Left this here for posterity, but I can work on a PR for it.

[Huh]

You are correct. I did not set the filters correctly. Should we include non-target species at this time?

BTW, you get the first 100 records everytime unless you use the page argument omit = 1, which will force all records to be returned. This should be the case in spdgt.sight, but if I missed setting this default holler.

[Huh]

We verified that is the expected behavior of the API, but I am not arguing it is great or user friendly. For a solution on the spdgt.wos side I would propose:

wos_read <- function(species, survey_type, analysis_unit, bio_year) {

  srvy <- spdgt.core::lkup_survey_types(
    species_name = species, 
    survey_type_name = survey_type
  )
  
  spdgt.sight::read_entries(
    survey_type = srvy$name,
    analysis_unit = analysis_unit,
    bio_year = bio_year,
    pages = list("omit" = 1)
  )

}

Then I will have to modify wos_format a little bit too. I already started in that direction, so just a little work to do. Should I pursue that and add non-target species? Would you like an argument to choose whether they are target or non-target?

[ericnewkirk]

Sorry, I missed this one too. I had basically the same in my fork. I ended up going way down the openxlsx2 rabbit hole without coming up with an xlsx solution yet, but I think I'm close.

Here's how I implemented the "survey type as a replacement for species" fix, I was thinking you might get cranky about abandoning lookup_by_name:

wos_read <- function(species, survey_type, analysis_unit, bio_year) {
  ids <- spdgt.sight:::lookup_by_name(
    species = species,
    analysis_unit = analysis_unit,
    survey_type = survey_type
  )

  spdgt.sight::read_entries_id(
    survey_type_id = ids$survey_type_id,
    analysis_unit_id = ids$dau_id,
    bio_year = bio_year,
    pages = list("omit" = 1)
  )
}

Seems like sixes to me.

There is an official list of what common names are allowed that we could leverage to validate incidental observations, but my strong suggestion would be to leave that up to the user to get right when entering data. I'm not sure whether incorrect records would fail on upload or would upload but then require correction before they're finalized, but neither of those would be particularly difficult to fix on the user end.

[Huh]

@ericnewkirk sounds good. I will implement this.