diff --git a/docs/http-servers.md b/docs/http-servers.md index 68d88343..d902a132 100644 --- a/docs/http-servers.md +++ b/docs/http-servers.md @@ -13,11 +13,12 @@ Biowatch uses an **HTTP-based ML model serving architecture** where each machine ### Supported Models -| Model | Focus | Species Coverage | -| ------------------------ | ------------------- | ------------------------------------ | -| **SpeciesNet** (Google) | Global wildlife | 2,000+ species worldwide | -| **DeepFaune** (CNRS) | European fauna | 34 European species | -| **Manas** (OSI-Panthera) | Central Asian fauna | Snow leopard and 11 regional species | +| Model | Focus | Species Coverage | +| ----------------------------------------- | ---------------------------------- | --------------------------------------------- | +| **SpeciesNet** (Google) | Global wildlife | 2,000+ species worldwide | +| **DeepFaune** (CNRS) | European fauna | 34 European species | +| **Manas** (OSI-Panthera) | Central Asian fauna | Snow leopard and 11 regional species | +| **MegaDetector** (Microsoft AI for Earth) | Blank filter / worldwide detection | 3 categories (animal, person, vehicle) | ### Technology Stack @@ -98,7 +99,8 @@ src/ python-environments/common/ ├── run_speciesnet_server.py # SpeciesNet LitServe implementation ├── run_deepfaune_server.py # DeepFaune LitServe implementation -└── run_manas_server.py # Manas LitServe implementation +├── run_manas_server.py # Manas LitServe implementation +└── run_megadetector_server.py # MegaDetector LitServe implementation (detection-only) ``` ## How HTTP Servers Work diff --git a/docs/specs/2026-05-16-megadetector-model-choice-design.md b/docs/specs/2026-05-16-megadetector-model-choice-design.md new file mode 100644 index 00000000..7904f42d --- /dev/null +++ b/docs/specs/2026-05-16-megadetector-model-choice-design.md @@ -0,0 +1,261 @@ +# MegaDetector as a Biowatch model choice — design + +## Background + +Biowatch currently offers three model choices in the AI Models tab: SpeciesNet (worldwide), DeepFaune (Europe), and Manas (Himalayas). All three combine an object detector with a species classifier. + +[MegaDetector](https://github.com/agentmorris/MegaDetector) is Microsoft AI for Earth's open-source detector for camera trap images. It locates animals, people, and vehicles in any image worldwide, with **no species classification**. Inside Biowatch it would fill a different role: a fast blank-filter that lets a user separate frames with subjects from empty frames, then manually annotate species on the non-blank frames using Biowatch's existing annotation UI. + +MegaDetector v6 weights are loaded via `ultralytics.YOLO` in `run_deepfaune_server.py` (the DeepFaune tarball bundles an earlier MDv6 build, `MDV6-yolov10x.pt`). The Python loading code is therefore already proven inside our `common` conda environment. The standalone MD model entry ships the current upstream variant **`MDV6-yolov10-e`** — best accuracy in the MDv6 lineup (82.8% animal recall, 92.8% mAP50) at ~30 MB. + +## Goals + +1. Ship MegaDetector v6 (`MDV6-yolov10-e`) as a standalone model choice in the AI Models tab. +2. Match the established model-integration pattern (one Python server script, one `switch` case in `server.ts`, one HF tarball, one `mlmodels.js` entry). +3. Map MD's detector labels (`animal`/`person`/`vehicle` + implicit `blank`) into Biowatch's species-centric data model with **zero changes** to the observation/annotation/export pipelines — by treating the three labels as pseudo-species. +4. Suppress UI affordances that only make sense for true species classifiers (the "View species" panel) via an opt-out flag, not a special-case in every consumer. + +## Non-goals + +- No conda env bump. MD reuses `common/0.1.4` — `ultralytics` and `torch` are already pinned for DeepFaune/Manas. +- No detector-weight sharing across models. MegaDetector ships its own tarball; the copy bundled inside DeepFaune/Manas is untouched. The marginal ~110 MB is acceptable in exchange for the model-isolation property the codebase keeps today. +- No "MD-then-classifier" workflow primitive. MD is a standalone model; chaining is out of scope. +- No MDv5a packaging. Only MDv6 (`MDV6-yolov10-e`) ships. If a user needs another MDv6 variant or MDv5 for reproducibility, that becomes a separate model entry later. +- License compatibility (MDv6 is AGPL-3.0; Biowatch is CC BY-NC 4.0) is acknowledged but out of scope for this change. Prior art: DeepFaune and Manas tarballs already bundle an MDv6 build. +- No automated reclassification of existing observations. + +## Source-of-truth artifact + +Upstream weights: `MDV6-yolov10-e-1280.pt` from the MegaDetector v6 model zoo. Downloaded from Zenodo at `https://zenodo.org/records/15398270/files/MDV6-yolov10-e-1280.pt?download=1` (the canonical hosting location linked from `https://microsoft.github.io/Biodiversity/model_zoo/megadetector/`). The `-1280` suffix denotes the model's input image size. The build script (see `scripts/build-megadetector-tarball.py` below) downloads from this URL and verifies the SHA256. + +We rehost this single file on `huggingface.co/earthtoolsmaker/megadetector` (repo already created by the maintainer), as a tarball `6.0.tar.gz` containing a top-level `6.0/` directory with the weights file inside it. This matches the layout convention of the other three model tarballs and gives us a stable, offline-deterministic download URL. The upstream `LICENSE` (AGPL-3.0) is included in the tarball next to the weights so AGPL attribution travels with the binary. + +## Tarball layout + +``` +6.0/ + MDV6-yolov10-e-1280.pt + LICENSE +``` + +Top-level directory name (`6.0/`) matches the model version. No `README.md` shipped (consistent with how `build-speciesnet-tarball.py` strips Kaggle's README). + +## Output schema + +The MD server emits one prediction object per image, matching the shape `inference-consumer.js` and `prediction.js` already consume: + +```json +{ + "filepath": "/path/to/image.jpg", + "detections": [ + { "label": "animal", "conf": 0.94, "xywhn": [0.5, 0.6, 0.3, 0.4], "xyxy": [...] } + ], + "classifications": {}, + "prediction": "animal", + "prediction_score": 0.94, + "model_version": "6.0" +} +``` + +- `prediction` is the label of the highest-confidence detection that passes `detectionConfidenceThreshold` (0.2 — MD's official default for v6). When no detection passes the threshold, `prediction` is `"blank"` and `prediction_score` is `null`. This mirrors how `run_deepfaune_server.py` emits `"blank"` when there are no detections. +- `classifications` is always an empty object — there is no classifier. Existing consumers already tolerate empty classifications (DeepFaune emits `{}` for blank frames). +- `detections[].xywhn` is the same center-format normalized bbox as DeepFaune/Manas, so the existing `xywhn` branch in `src/main/utils/bbox.js` handles it unchanged. + +## Pseudo-species mapping + +The three MD labels are stored as if they were species: + +| `prediction` field | Treated as scientific name | Common name | +|---------------------------|----------------------------|--------------------------| +| `animal` | `animal` | `Animal` | +| `homo sapiens` (mapped from MD's `person` label) | `homo sapiens` | `Human` | +| `vehicle` | `vehicle` | `Vehicle` | +| `blank` | `blank` | (existing blank handling) | + +The `person` label is the only MD category that is genuinely a species; the Python server translates `person → homo sapiens` for the top-level `prediction` field so it integrates with Biowatch's species tooltips and IUCN lookups. Per-bbox `detections[].label` stays raw (`"person"`). `animal` and `vehicle` are not species and stay as plain labels. + +A new common-names source file resolves these (schema matches `manas.json`): + +`src/shared/commonNames/sources/megadetector.json`: + +```json +{ + "modelId": "megadetector", + "modelVersion": "6.0", + "source": "MegaDetector v6 categories (animal / person / vehicle)", + "entries": [ + { "scientificName": null, "label": "animal", "commonName": "Animal" }, + { "scientificName": "homo sapiens", "label": "person", "commonName": "Human" }, + { "scientificName": null, "label": "vehicle", "commonName": "Vehicle" } + ] +} +``` + +The `person` entry has a non-null `scientificName` so the dict-build pipeline (a) registers `homo sapiens → human` in `dictionary.json` and (b) registers `person → homo sapiens` in `labelAliases.json`. The Python server emits `homo sapiens` directly for human detections; the alias map exists for downstream consistency with the LILA-style file importers that translate label → binomial at insert time. + +User-facing consequence: in the Media tab the user filters `species = animal`, finds the non-blank frames, opens each one, and assigns the real species via the existing annotation UI. Human frames already land with `scientificName = "homo sapiens"` so the species tooltip fires the same way it would for any other Homo-sapiens prediction in the system. The raw MD output stays in `model_outputs`; the user's annotation lives on the observation, same as any model. + +## Code changes + +### `python-environments/common/run_megadetector_server.py` (NEW) + +Structurally a stripped DeepFaune runner. CLI flags: `--filepath-detector-weights` and `--detection-confidence-threshold` (default 0.2). Reuses `VideoCapableLitAPI`, `safe_imread`, `to_detection_record`, and `propagate_extra_fields` from `utils.py` / `detection_utils.py`. **Does not** reuse `select_best_animal_detection`: that helper filters specifically for the `animal` class, but MD needs the best detection across all three classes (`animal`, `person`, `vehicle`). The server selects the top-confidence detection inline. No classifier import, no classifier load, no classifier inference. + +The `predict()` path: + +1. Run YOLO detector on the image. +2. Read class labels from the YOLO model's `names` attribute (same pattern as `run_deepfaune_server.py` line 440 — `class_names = detections.names`). For MDv6 these are `{0: "animal", 1: "person", 2: "vehicle"}`, but reading from the model rather than hard-coding the mapping insulates us if upstream ever renumbers. +3. Build `detection_records` from the YOLO output. +4. Pick the highest-confidence detection whose `conf >= --detection-confidence-threshold` — `prediction` becomes that detection's `label`, `prediction_score` becomes its `conf`. Default to `prediction: "blank"`, `prediction_score: null` when no detection passes. +5. Yield the dict shape shown in "Output schema" above. + +**Why the threshold appears in two places.** `mlmodels.js` carries `detectionConfidenceThreshold: 0.2` as it does for every other model — `inference-consumer.js` reads it from there (`this.model.detectionConfidenceThreshold` at server.ts/inference-consumer.js:154) to filter which detections become observations. The Python server *also* needs the threshold, because for MD the `prediction` field is itself derived from "is there a detection above threshold?" — there's no classifier to fall back on as a separate signal of "blank vs. not". The two values must agree; both are 0.2 for MD. `server.ts` passes the JS-side value through to the Python script via the CLI flag, so they stay in sync. + +### `src/main/services/ml/server.ts` + +Add `startMegaDetectorHTTPServer()` modeled on `startDeepFauneHTTPServer()`: + +```ts +interface MegaDetectorServerOptions { + port: number + detectorWeightsFilepath: string + detectionConfidenceThreshold: number + timeout: number + pythonEnvironment: { reference: { id: string; version: string } } +} + +export async function startMegaDetectorHTTPServer({ + port, + detectorWeightsFilepath, + detectionConfidenceThreshold, + timeout, + pythonEnvironment +}: MegaDetectorServerOptions): Promise<{ process: ChildProcess; shutdownApiKey: string }> { + // mirrors startDeepFauneHTTPServer; script path → run_megadetector_server.py +} +``` + +Add a new `case 'megadetector'` in the `switch` inside `startMLModelHTTPServer()`: + +```ts +case 'megadetector': { + const port = await resolveServerPort(is.dev ? 8003 : null) + const localInstallPath = getMLModelLocalInstallPath({ ...modelReference }) + const detectorWeightsFilepath = join(localInstallPath, 'MDV6-yolov10-e-1280.pt') + const model = findModel({ ...modelReference }) + const { process: pythonProcess, shutdownApiKey } = await startMegaDetectorHTTPServer({ + port, + detectorWeightsFilepath, + detectionConfidenceThreshold: model.detectionConfidenceThreshold, + timeout: 30, + pythonEnvironment + }) + registerActiveServer({ pid: pythonProcess.pid, port, shutdownApiKey, modelId: modelReference.id }) + return { port, process: pythonProcess, shutdownApiKey } +} +``` + +Dev port `8003` extends the existing fixed-port sequence (`8000` SpeciesNet, `8001` DeepFaune, `8002` Manas). + +### `src/shared/mlmodels.js` + +Add a fourth entry to `modelZoo`: + +```js +{ + reference: { id: 'megadetector', version: '6.0' }, + pythonEnvironment: { id: 'common', version: '0.1.4' }, + name: 'MegaDetector', + size_in_MB: , + files: 1, + downloadURL: + 'https://huggingface.co/earthtoolsmaker/megadetector/resolve/main/6.0.tar.gz?download=true', + description: + "MegaDetector is Microsoft AI for Earth's open-source detector for camera trap images. It locates animals, people, and vehicles in any image worldwide, without identifying species. Useful as a fast blank-filter before manual species annotation.", + website: 'https://github.com/agentmorris/MegaDetector', + logo: 'megadetector', + detectionConfidenceThreshold: 0.2, + region: 'worldwide', + species_count: 3, + detectionOnly: true +} +``` + +The new `detectionOnly: true` flag is the *only* place the rest of the app needs to special-case MD. No `species_data` key — `detectionOnly` short-circuits the species panel before it would try to load one. + +### `src/main/utils/bbox.js` + +`megadetector` joins `manas` and `deepfaune` in the existing `xywhn` case — three labels falling through to the same block: + +```js +case 'manas': +case 'megadetector': +case 'deepfaune': { + // existing xywhn → top-left conversion +} +``` + +`detectModelType()` already has a generic xywhn fallback, but for clarity we add an explicit branch: `if (version === '6.0' && prediction.detections?.[0]?.xywhn) return 'megadetector'`. + +### `src/renderer/src/models/ModelCard.jsx` + +One conditional on `model.detectionOnly`: + +- Replace the `v{version} · {size} · {species_count} species` line with `v{version} · {size} · Detection only · 3 categories`. +- Skip the `▸ View {species_count} species` toggle and the `` mount. + +### `src/renderer/src/models/ModelSelect.jsx` + +Same conditional in the rich-card dropdown rows — show "Detection only" instead of the species count. + +### `src/renderer/src/models/SpeciesPanel.jsx` + +No changes. `detectionOnly: true` means the panel is never mounted for MD. + +### Logo asset + +Add `src/renderer/src/assets/logos/megadetector.png` (maintainer provides). The existing logo resolution code is data-driven on the `logo` field — no code change needed beyond placing the file. + +## Build script + +`scripts/build-megadetector-tarball.py` — same shape as `build-speciesnet-tarball.py`, using only the Python standard library: + +1. Download `MDV6-yolov10-e-1280.pt` from the MegaDetector v6 release page on GitHub (exact URL filled in once the script's author confirms it from the upstream release manifest). +2. Verify SHA256 against a constant encoded in the script (locked after the first successful build), matching the pattern `MEGADETECTOR_SHA256` follows in `build-speciesnet-tarball.py`. +3. Download `LICENSE` (AGPL-3.0) from the MegaDetector repo at the same release tag so the attribution travels with the weights. +4. Place both files at `6.0/MDV6-yolov10-e-1280.pt` and `6.0/LICENSE`. +5. `tar -czf dist/6.0.tar.gz 6.0/`. +6. Print final size + SHA256 + file list for the `mlmodels.js` update. + +The upload step (`huggingface-cli upload …` to `earthtoolsmaker/megadetector`) stays manual — credentials are user-scoped. Maintainer (Arthur) uploads. + +## Verification + +1. **Build script self-check**: tarball contains exactly two files at `6.0/MDV6-yolov10-e-1280.pt` and `6.0/LICENSE`; SHA256 of the weights file matches the constant locked into the build script. +2. **`npm test`**: covers `mlmodels.test.js` (new entry valid), `bbox.test.js` (new megadetector case routes to xywhn), `downloadState.test.js`, common-names resolver tests (megadetector source loads). +3. **`cd python-environments/common && make lint && make format && make test`**: includes a new `tests/test_megadetector_server.py`. Test cases: + - Start the server with the bundled weights. + - POST a known-animal image → `prediction == "animal"`, `prediction_score > 0.2`, at least one detection with `label == "animal"`. + - POST a known-blank image → `prediction == "blank"`, `prediction_score is None`, `detections == []` (or all below threshold). +4. **Manual smoke test**: fresh install → download MegaDetector (model card shows "Detection only · 3 categories", no "View species" link) → import ~20 mixed images (animals + people + a vehicle + blanks) → Sources tab shows correct counts per pseudo-species → open one `animal` frame → re-annotate it with a real species via the existing annotation UI → confirm the user's annotation persists and the original MD prediction is preserved in `model_outputs`. +5. **Multi-model regression check**: with MegaDetector installed alongside SpeciesNet, kick off an import using SpeciesNet, then a second import using MegaDetector. Confirm both servers can coexist (different ports) and predictions land under the correct topic in the queue. + +## Release notes + +User-facing changelog should call out: + +1. **New model: MegaDetector v6.** Worldwide animal/person/vehicle detector. Useful as a fast blank-filter before manual species annotation. +2. MegaDetector does not identify species — it only detects whether a frame contains an animal, a person, or a vehicle. Use it to triage large folders, then annotate species manually inside Biowatch. +3. ~30 MB download. Reuses the existing `common` Python environment, so no extra environment install if you already have SpeciesNet/DeepFaune/Manas. + +## Documentation updates + +- `docs/http-servers.md` — add MegaDetector to the "Supported Models" table (Focus: "Blank filter / worldwide detection"; Species Coverage: "3 categories (animal, person, vehicle)"). Add `run_megadetector_server.py` to the project-structure listing. +- `docs/specs/2026-05-16-megadetector-model-choice-design.md` — this document (checked in). + +## Open dependencies on the user + +1. Run `python scripts/build-megadetector-tarball.py` once it lands. +2. Upload the resulting `6.0.tar.gz` to `huggingface.co/earthtoolsmaker/megadetector/main`. +3. Provide the final tarball size in MB for `size_in_MB` in `mlmodels.js`. +4. Provide a `megadetector.png` logo for `src/renderer/src/assets/logos/`. diff --git a/python-environments/common/Makefile b/python-environments/common/Makefile index f69609fb..f46227f4 100644 --- a/python-environments/common/Makefile +++ b/python-environments/common/Makefile @@ -27,6 +27,7 @@ export DEEPFAUNE_CLASSIFIER_WEIGHTS=/tmp/models/deepfaune/1.3/deepfaune-vit_larg export MANAS_DETECTOR_WEIGHTS=/tmp/models/manas/1.0/MDV6-yolov10x.pt export MANAS_CLASSIFIER_WEIGHTS=/tmp/models/manas/1.0/best_model_Fri_Sep__1_18_50_55_2023.pt export MANAS_CLASSES=/tmp/models/manas/1.0/classes_Fri_Sep__1_18_50_55_2023.pickle +export MEGADETECTOR_DETECTOR_WEIGHTS=/tmp/models/megadetector/6.0/MDV6-yolov10-e-1280.pt # Run all e2e tests (downloads models if not present) test: download-models @@ -41,3 +42,4 @@ download-models: uv run python scripts/download_model.py --model speciesnet --output /tmp/models/speciesnet uv run python scripts/download_model.py --model deepfaune --output /tmp/models/deepfaune uv run python scripts/download_model.py --model manas --output /tmp/models/manas + uv run python scripts/download_model.py --model megadetector --output /tmp/models/megadetector diff --git a/python-environments/common/run_megadetector_server.py b/python-environments/common/run_megadetector_server.py new file mode 100644 index 00000000..89908257 --- /dev/null +++ b/python-environments/common/run_megadetector_server.py @@ -0,0 +1,267 @@ +""" +CLI script to run MegaDetector v6 as a LitServer. + +MegaDetector v6 is a YOLO-based detector (animals / people / vehicles only — no +species classification). Used in Biowatch as a fast blank-filter before manual +species annotation. + +Start the server: + +``` +run_megadetector_server.py \\ + --port 8000 \\ + --filepath-detector-weights ./path/to/MDV6-yolov10-e-1280.pt \\ + --detection-confidence-threshold 0.2 +``` + +A Swagger API documentation is served at localhost:${port}/docs + +health: + +``` +$ curl http://localhost:${port}/health +"ok" +``` + +info: + +``` +$ curl http://localhost:${port}/info +{ + "model": {"type": "megadetector", "version": "6.0"}, + "server": {...} +} +``` + +predict (streaming): + +``` +$ curl -X POST http://localhost:${port}/predict \\ + -H "Content-Type: application/json" \\ + -d '{"instances": [{"filepath": "/path/to/your/image"}]}' +``` + +Output per image: + +```json +{ + "output": { + "predictions": [{ + "filepath": "/path/to/image", + "classifications": {}, + "detections": [ + {"label": "animal", "conf": 0.94, "xywhn": [...], "xyxy": [...]} + ], + "prediction": "animal", + "prediction_score": 0.94, + "model_version": "6.0" + }] + } +} +``` + +The top-level `prediction` field translates MD's `"person"` label to the +binomial `"homo sapiens"` (the only MD category that is genuinely a species), +so it integrates with Biowatch's species tooltips and IUCN lookups. Per-bbox +`detections[].label` stays raw. +""" + +import logging +from pathlib import Path + +import litserve as ls +from absl import app, flags +from fastapi import HTTPException +from ultralytics import YOLO + +from detection_utils import propagate_extra_fields, to_detection_record +from utils import VideoCapableLitAPI, is_video_file, safe_imread + +logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s") +logger = logging.getLogger(__name__) + +MODEL_VERSION = "6.0" + +# Map MegaDetector detection labels to Biowatch-canonical scientific names +# for the top-level `prediction` field. Per-bbox `detections[].label` stays raw. +# `person` is genuinely a species — emit the binomial so it integrates with +# Biowatch's species tooltips and IUCN lookups. `animal` and `vehicle` are not +# species and pass through unchanged. +LABEL_TO_PREDICTION = { + "person": "homo sapiens", +} + +_PORT = flags.DEFINE_integer("port", 8000, "Port to run the server on.") +_API_PATH = flags.DEFINE_string("api_path", "/predict", "URL path for the server endpoint.") +_WORKERS_PER_DEVICE = flags.DEFINE_integer("workers_per_device", 1, "Number of server replicas per device.") +_TIMEOUT = flags.DEFINE_integer("timeout", 30, "Timeout (in seconds) for requests.") +_BACKLOG = flags.DEFINE_integer("backlog", 2048, "Maximum number of connections to hold in backlog.") +_FILEPATH_DETECTOR_WEIGHTS = flags.DEFINE_string( + name="filepath-detector-weights", + default=None, + help="filepath for the weights of the MegaDetector detector", + required=True, +) +_DETECTION_CONFIDENCE_THRESHOLD = flags.DEFINE_float( + "detection-confidence-threshold", + 0.2, + "Confidence threshold below which a detection is ignored when computing the top prediction. " + "Detections themselves are always returned; this only controls the 'prediction'/'prediction_score' fields.", +) +_EXTRA_FIELDS = flags.DEFINE_list( + "extra_fields", + None, + "Comma-separated list of extra fields to propagate from request to response.", +) + + +def predict_one( + detector: YOLO, + filepath: Path, + confidence_threshold: float, + model_version: str = MODEL_VERSION, +) -> dict: + """Run MegaDetector on a single image and produce a Biowatch-compatible prediction dict. + + The 'prediction' field is the label of the highest-confidence detection + whose conf >= confidence_threshold, translated via LABEL_TO_PREDICTION. + When no detection passes the threshold, 'prediction' is 'blank' and + the 'prediction_score' key is omitted entirely (matches DeepFaune's + behavior and the JS-side Zod schema, which accepts an absent + prediction_score but rejects null). + """ + imagecv = safe_imread(filepath) + ultralytics_results = detector(imagecv, verbose=False) + yolo_out = ultralytics_results[0] + bboxes = yolo_out.boxes + # class_names comes from the .pt file's embedded `names` attribute — never hard-code. + class_names = yolo_out.names + + detection_records = [ + to_detection_record( + conf=conf, + class_instance=class_instance, + xywhn=xywhn, + xyxy=xyxy, + class_label_mapping=class_names, + ) + for conf, class_instance, xywhn, xyxy in zip( + bboxes.conf.cpu().tolist(), + bboxes.cls.cpu().numpy().astype(int).tolist(), + bboxes.xywhn.cpu().numpy().tolist(), + bboxes.xyxy.cpu().numpy().tolist(), + strict=True, + ) + ] + + # MD detects across animal/person/vehicle — pick top-confidence across ALL classes. + # (We don't reuse select_best_animal_detection, which filters for the animal class.) + above = [d for d in detection_records if d["conf"] >= confidence_threshold] + if not above: + return { + "predictions": [ + { + "filepath": str(filepath), + "classifications": {}, + "detections": detection_records, + "prediction": "blank", + "model_version": model_version, + } + ], + } + + top = max(above, key=lambda d: d["conf"]) + prediction_label = LABEL_TO_PREDICTION.get(top["label"], top["label"]) + return { + "predictions": [ + { + "filepath": str(filepath), + "classifications": {}, + "detections": detection_records, + "prediction": prediction_label, + "prediction_score": top["conf"], + "model_version": model_version, + } + ], + } + + +class MegaDetectorLitAPI(ls.LitAPI, VideoCapableLitAPI): + """MegaDetector API server with video support.""" + + def __init__( + self, + filepath_detector_weights: Path, + detection_confidence_threshold: float, + extra_fields: list[str] | None = None, + *args, + **kwargs, + ) -> None: + super().__init__(*args, **kwargs) + self.filepath_detector_weights = filepath_detector_weights + self.detection_confidence_threshold = detection_confidence_threshold + self.extra_fields = extra_fields or [] + + def setup(self, device): + del device # Unused. + self.detector = YOLO(self.filepath_detector_weights) + + def decode_request(self, request, **kwargs): + for instance in request["instances"]: + filepath = instance["filepath"] + if not is_video_file(filepath) and not Path(filepath).exists(): + raise HTTPException(400, f"Cannot access filepath: `{filepath}`") + return request + + def _predict_single_image(self, filepath: str, **kwargs) -> dict: + single_instances_dict = {"instances": [{"filepath": filepath}]} + single_predictions_dict = predict_one( + detector=self.detector, + filepath=Path(filepath), + confidence_threshold=self.detection_confidence_threshold, + ) + return propagate_extra_fields(self.extra_fields, single_instances_dict, single_predictions_dict) + + def predict(self, x, **kwargs): + instances = x.get("instances", []) + logger.info(f"[MegaDetector] Processing {len(instances)} instances") + try: + yield from self.predict_with_video_support(x, **kwargs) + except Exception as e: + logger.error(f"[MegaDetector] Prediction failed: {e}", exc_info=True) + raise + + def encode_response(self, output, **kwargs): + for out in output: + yield {"output": out} + + +def main(argv: list[str]) -> None: + del argv # Unused. + print("[STARTUP] Starting MegaDetector LitServer...") + api = MegaDetectorLitAPI( + filepath_detector_weights=Path(_FILEPATH_DETECTOR_WEIGHTS.value), + detection_confidence_threshold=_DETECTION_CONFIDENCE_THRESHOLD.value, + extra_fields=_EXTRA_FIELDS.value, + api_path=_API_PATH.value, + stream=True, + ) + model_metadata = {"version": MODEL_VERSION, "type": "megadetector"} + server = ls.LitServer( + api, + accelerator="auto", + devices="auto", + workers_per_device=_WORKERS_PER_DEVICE.value, + model_metadata=model_metadata, + timeout=_TIMEOUT.value, + enable_shutdown_api=True, + ) + server.run( + port=_PORT.value, + generate_client_file=False, + backlog=_BACKLOG.value, + ) + + +if __name__ == "__main__": + app.run(main) diff --git a/python-environments/common/scripts/download_model.py b/python-environments/common/scripts/download_model.py index 44481900..574113c0 100644 --- a/python-environments/common/scripts/download_model.py +++ b/python-environments/common/scripts/download_model.py @@ -5,6 +5,7 @@ uv run python scripts/download_model.py --model speciesnet --output /tmp/models/speciesnet uv run python scripts/download_model.py --model deepfaune --output /tmp/models/deepfaune uv run python scripts/download_model.py --model manas --output /tmp/models/manas + uv run python scripts/download_model.py --model megadetector --output /tmp/models/megadetector """ import argparse @@ -26,6 +27,10 @@ "repo_id": "earthtoolsmaker/manas", "filename": "1.0.tar.gz", }, + "megadetector": { + "repo_id": "earthtoolsmaker/megadetector", + "filename": "6.0.tar.gz", + }, } @@ -57,7 +62,7 @@ def main(): "--model", required=True, choices=list(MODELS.keys()), - help="Model to download (speciesnet, deepfaune, manas)", + help="Model to download (speciesnet, deepfaune, manas, megadetector)", ) parser.add_argument( "--output", diff --git a/python-environments/common/tests/test_megadetector_server.py b/python-environments/common/tests/test_megadetector_server.py new file mode 100644 index 00000000..53b84a67 --- /dev/null +++ b/python-environments/common/tests/test_megadetector_server.py @@ -0,0 +1,146 @@ +"""End-to-end tests for MegaDetector ML server.""" + +import os + +import httpx +import pytest + +from tests.conftest import ServerProcess, find_free_port, parse_streaming_response + +MEGADETECTOR_DETECTOR_WEIGHTS = os.environ.get("MEGADETECTOR_DETECTOR_WEIGHTS", "") + +pytestmark = pytest.mark.skipif( + not MEGADETECTOR_DETECTOR_WEIGHTS, + reason="MegaDetector weights not available. Set MEGADETECTOR_DETECTOR_WEIGHTS.", +) + + +@pytest.fixture(scope="module") +def megadetector_server(): + """Start MegaDetector server for the test module.""" + port = find_free_port() + command = [ + "uv", + "run", + "python", + "run_megadetector_server.py", + "--port", + str(port), + "--timeout", + "60", + "--filepath-detector-weights", + MEGADETECTOR_DETECTOR_WEIGHTS, + "--detection-confidence-threshold", + "0.2", + ] + with ServerProcess(command, port, startup_timeout=300) as server: + yield server + + +class TestMegaDetectorServer: + """Tests for MegaDetector server endpoints.""" + + def test_health_endpoint(self, megadetector_server): + resp = httpx.get(f"{megadetector_server.base_url}/health", timeout=10.0) + assert resp.status_code == 200 + assert resp.text == "ok" + + def test_info_endpoint(self, megadetector_server): + resp = httpx.get(f"{megadetector_server.base_url}/info", timeout=10.0) + assert resp.status_code == 200 + data = resp.json() + assert "model" in data + assert data["model"]["type"] == "megadetector" + assert "server" in data + + def test_predict_animal_image(self, megadetector_server, test_images): + """A clearly-animal image should produce prediction='animal'.""" + payload = {"instances": [{"filepath": str(test_images["chamois"])}]} + with httpx.stream( + "POST", + f"{megadetector_server.base_url}/predict", + json=payload, + timeout=60.0, + ) as resp: + assert resp.status_code == 200 + results = parse_streaming_response(resp) + + assert len(results) == 1 + pred = results[0]["output"]["predictions"][0] + assert pred["filepath"] == str(test_images["chamois"]) + assert pred["prediction"] == "animal" + assert pred["prediction_score"] >= 0.2 + assert pred["classifications"] == {} + assert len(pred["detections"]) >= 1 + assert pred["detections"][0]["label"] in {"animal", "person", "vehicle"} + assert "xywhn" in pred["detections"][0] + + def test_predict_human_image(self, megadetector_server, test_images): + """A human image's top prediction should be 'homo sapiens' (translated from MD's 'person' label).""" + payload = {"instances": [{"filepath": str(test_images["human"])}]} + with httpx.stream( + "POST", + f"{megadetector_server.base_url}/predict", + json=payload, + timeout=60.0, + ) as resp: + results = parse_streaming_response(resp) + pred = results[0]["output"]["predictions"][0] + assert pred["prediction"] == "homo sapiens" + # Raw bbox labels stay 'person' — MD's native output is preserved per-bbox. + person_dets = [d for d in pred["detections"] if d["label"] == "person"] + assert len(person_dets) >= 1, "raw 'person' label must be preserved per-detection" + + def test_predict_empty_image(self, megadetector_server, test_images): + """An empty (no subject) image should produce prediction='blank' with no prediction_score key. + + The JS-side Zod validator accepts an absent prediction_score but rejects null, + so the server omits the key entirely on blanks (matches DeepFaune's behavior). + """ + payload = {"instances": [{"filepath": str(test_images["empty"])}]} + with httpx.stream( + "POST", + f"{megadetector_server.base_url}/predict", + json=payload, + timeout=60.0, + ) as resp: + results = parse_streaming_response(resp) + pred = results[0]["output"]["predictions"][0] + assert pred["prediction"] == "blank" + assert "prediction_score" not in pred, "blank predictions must omit prediction_score, not emit null" + + def test_predict_streaming(self, megadetector_server, test_images): + """Streaming returns one chunk per image, all classifications empty.""" + payload = { + "instances": [ + {"filepath": str(test_images["badger"])}, + {"filepath": str(test_images["fox"])}, + {"filepath": str(test_images["empty"])}, + ] + } + with httpx.stream( + "POST", + f"{megadetector_server.base_url}/predict", + json=payload, + timeout=120.0, + ) as resp: + results = parse_streaming_response(resp) + assert len(results) == 3 + for r in results: + pred = r["output"]["predictions"][0] + assert pred["classifications"] == {} + assert pred["model_version"] == "6.0" + + def test_predict_invalid_filepath(self, megadetector_server): + payload = {"instances": [{"filepath": "/nonexistent/image.jpg"}]} + resp = httpx.post( + f"{megadetector_server.base_url}/predict", + json=payload, + timeout=30.0, + ) + assert resp.status_code in (200, 400) + + def test_docs_endpoint(self, megadetector_server): + resp = httpx.get(f"{megadetector_server.base_url}/docs", timeout=10.0) + assert resp.status_code == 200 + assert "swagger" in resp.text.lower() or "openapi" in resp.text.lower() diff --git a/python-environments/common/tests/test_utils.py b/python-environments/common/tests/test_utils.py index a3c7ec04..59b5c353 100644 --- a/python-environments/common/tests/test_utils.py +++ b/python-environments/common/tests/test_utils.py @@ -1,6 +1,10 @@ """Unit tests for shared ML server utilities.""" -from utils import VideoCapableLitAPI +from unittest.mock import MagicMock, patch + +import pytest + +from utils import VideoCapableLitAPI, get_video_metadata class TestNormalizeFailedPredictions: @@ -96,3 +100,86 @@ def test_missing_predictions_key(self): result = {} normalized = VideoCapableLitAPI._normalize_failed_predictions(result, "/x.jpg") assert normalized == {"predictions": []} + + +class TestGetVideoMetadataSanity: + """Tests that get_video_metadata rejects corrupt-file metadata. + + OpenCV can return huge negative frame_count values on broken containers + (uint64 underflow cast to int), producing absurd durations and letting a + single garbage frame slip into inference. The metadata helper rejects + these so the consumer's error-handling pipeline can flag the file. + """ + + def _mocked_capture(self, fps, frame_count, opened=True): + cap = MagicMock() + cap.isOpened.return_value = opened + cap.get.side_effect = lambda prop: { + 5: fps, # cv2.CAP_PROP_FPS + 7: frame_count, # cv2.CAP_PROP_FRAME_COUNT + }.get(prop, 0) + return cap + + def test_rejects_negative_frame_count(self): + """A corrupt MP4 with negative frame_count must raise, not return junk.""" + with ( + patch( + "utils.cv2.VideoCapture", + return_value=self._mocked_capture(fps=30, frame_count=-922337203685477), + ), + pytest.raises(ValueError, match="Invalid video metadata"), + ): + get_video_metadata("/corrupt.mp4") + + def test_rejects_zero_frame_count(self): + """A file that reports zero frames is not a usable video.""" + with ( + patch( + "utils.cv2.VideoCapture", + return_value=self._mocked_capture(fps=30, frame_count=0), + ), + pytest.raises(ValueError, match="Invalid video metadata"), + ): + get_video_metadata("/empty.mp4") + + def test_rejects_zero_fps(self): + """fps=0 means duration is meaningless.""" + with ( + patch( + "utils.cv2.VideoCapture", + return_value=self._mocked_capture(fps=0, frame_count=100), + ), + pytest.raises(ValueError, match="Invalid video metadata"), + ): + get_video_metadata("/no-fps.mp4") + + def test_rejects_excessive_duration(self): + """Duration above the 24h sanity ceiling is rejected.""" + # 30 fps × 30 hours of frames → 30h duration → over the 24h limit. + excessive_frame_count = 30 * 60 * 60 * 30 + with ( + patch( + "utils.cv2.VideoCapture", + return_value=self._mocked_capture(fps=30, frame_count=excessive_frame_count), + ), + pytest.raises(ValueError, match="Invalid video metadata"), + ): + get_video_metadata("/huge.mp4") + + def test_rejects_unopenable_video(self): + """Cannot open returns the existing 'Cannot open video' error.""" + with ( + patch( + "utils.cv2.VideoCapture", + return_value=self._mocked_capture(fps=30, frame_count=100, opened=False), + ), + pytest.raises(ValueError, match="Cannot open video"), + ): + get_video_metadata("/missing.mp4") + + def test_accepts_normal_video(self): + """Sane metadata round-trips through.""" + with patch("utils.cv2.VideoCapture", return_value=self._mocked_capture(fps=30, frame_count=300)): + metadata = get_video_metadata("/ok.mp4") + assert metadata["fps"] == 30 + assert metadata["duration"] == pytest.approx(10.0) # 300 frames / 30 fps diff --git a/python-environments/common/utils.py b/python-environments/common/utils.py index b1d96807..c02864d9 100644 --- a/python-environments/common/utils.py +++ b/python-environments/common/utils.py @@ -70,6 +70,13 @@ def get_video_metadata(video_path: str) -> dict[str, float]: Returns: Dictionary with 'fps' and 'duration' keys + + Raises: + ValueError: if the video cannot be opened, or if OpenCV reports + implausible metadata (corrupt files often surface as a hugely + negative frame_count from a uint64 underflow, which yields a + nonsense duration and would otherwise let a single garbage + frame leak through to inference). """ cap = cv2.VideoCapture(video_path) if not cap.isOpened(): @@ -80,6 +87,14 @@ def get_video_metadata(video_path: str) -> dict[str, float]: duration = frame_count / fps if fps > 0 else 0 cap.release() + # 24h ceiling is well above any realistic camera-trap clip but well below + # the absurd values OpenCV surfaces on corrupt containers. + MAX_REASONABLE_DURATION_S = 24 * 60 * 60 + if fps <= 0 or frame_count <= 0 or not (0 < duration <= MAX_REASONABLE_DURATION_S): + raise ValueError( + f"Invalid video metadata (fps={fps}, frame_count={frame_count}, duration={duration}): {video_path}" + ) + return {"fps": fps, "duration": duration} diff --git a/scripts/build-common-names-dict.js b/scripts/build-common-names-dict.js index 88ced495..74e494fe 100644 --- a/scripts/build-common-names-dict.js +++ b/scripts/build-common-names-dict.js @@ -94,7 +94,7 @@ function main() { const dictionary = {} const labelAliases = {} - const order = ['speciesnet.json', 'deepfaune.json', 'manas.json'] + const order = ['speciesnet.json', 'deepfaune.json', 'manas.json', 'megadetector.json'] for (const filename of order) { const snapshot = loadJson(path.join(SOURCES_DIR, filename)) mergeEntries(dictionary, snapshot.entries) diff --git a/scripts/build-megadetector-tarball.py b/scripts/build-megadetector-tarball.py new file mode 100755 index 00000000..26be91d5 --- /dev/null +++ b/scripts/build-megadetector-tarball.py @@ -0,0 +1,124 @@ +#!/usr/bin/env python3 +"""Build a MegaDetector model tarball ready to upload to huggingface.co/earthtoolsmaker/megadetector. + +Downloads the official MDv6 release weights and the upstream LICENSE so AGPL attribution +travels with the binary, then tar-gzips them into a directory named for the version. + +Usage: + python scripts/build-megadetector-tarball.py --version 6.0 + +Produces dist/.tar.gz alongside a summary of size + SHA256 + file list. +""" + +import argparse +import hashlib +import shutil +import sys +import tarfile +import tempfile +import urllib.request +from pathlib import Path + +WEIGHTS_URL = "https://zenodo.org/records/15398270/files/MDV6-yolov10-e-1280.pt?download=1" +WEIGHTS_FILENAME = "MDV6-yolov10-e-1280.pt" +WEIGHTS_SHA256 = "4a3a3d380ce7e151b2a8b991ab5d86f329ccd7b0b33e5d3ba0593a1166d55109" +LICENSE_URL = "https://raw.githubusercontent.com/microsoft/MegaDetector/main/LICENSE" +LICENSE_FILENAME = "LICENSE" + + +def download(url: str, dest: Path) -> None: + if dest.exists(): + print(f"[skip] {dest.name} already present ({dest.stat().st_size / 1e6:.0f} MB)") + return + print(f"[download] {url}") + is_tty = sys.stdout.isatty() + last_milestone = -1 + + def report(block_num: int, block_size: int, total_size: int) -> None: + nonlocal last_milestone + if total_size <= 0: + return + downloaded = block_num * block_size + pct = min(100, downloaded * 100 // total_size) + if is_tty: + sys.stdout.write(f"\r {pct:3d}% {downloaded / 1e6:7.1f} / {total_size / 1e6:.0f} MB") + sys.stdout.flush() + else: + milestone = pct // 10 + if milestone > last_milestone: + last_milestone = milestone + print(f" {pct:3d}% {downloaded / 1e6:7.1f} / {total_size / 1e6:.0f} MB") + + urllib.request.urlretrieve(url, dest, reporthook=report) + if is_tty: + sys.stdout.write("\n") + + +def sha256_of(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as fp: + for chunk in iter(lambda: fp.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def build(version: str, output_dir: Path, work_dir: Path) -> Path: + work_dir.mkdir(parents=True, exist_ok=True) + output_dir.mkdir(parents=True, exist_ok=True) + + model_dir = work_dir / version + if model_dir.exists(): + shutil.rmtree(model_dir) + model_dir.mkdir() + + weights_path = model_dir / WEIGHTS_FILENAME + download(WEIGHTS_URL, weights_path) + + actual_hash = sha256_of(weights_path) + if WEIGHTS_SHA256 != "TBD_LOCK_ON_FIRST_RUN" and actual_hash != WEIGHTS_SHA256: + raise RuntimeError(f"Weights SHA256 mismatch:\n expected {WEIGHTS_SHA256}\n got {actual_hash}") + print(f"[verify] {WEIGHTS_FILENAME} SHA256 = {actual_hash}") + + download(LICENSE_URL, model_dir / LICENSE_FILENAME) + + tarball = output_dir / f"{version}.tar.gz" + if tarball.exists(): + tarball.unlink() + print(f"[tar] {tarball}") + with tarfile.open(tarball, "w:gz") as tar: + tar.add(model_dir, arcname=version) + return tarball + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--version", required=True, help="MegaDetector version, e.g. 6.0") + parser.add_argument("--output-dir", type=Path, default=Path("dist")) + parser.add_argument( + "--work-dir", + type=Path, + default=Path(tempfile.gettempdir()) / "megadetector-build", + help="Working directory for downloads (retained between runs)", + ) + args = parser.parse_args() + + tarball = build(args.version, args.output_dir, args.work_dir) + + size_mb = tarball.stat().st_size / 1e6 + digest = sha256_of(tarball) + with tarfile.open(tarball) as tar: + members = sorted(m.name for m in tar.getmembers() if not m.isdir()) + + print() + print("=" * 60) + print(f" Tarball: {tarball}") + print(f" Size: {size_mb:.1f} MB") + print(f" SHA256: {digest}") + print(" Contents:") + for name in members: + print(f" {name}") + print("=" * 60) + + +if __name__ == "__main__": + main() diff --git a/scripts/lib/aliases.js b/scripts/lib/aliases.js index 82dbe808..d432ab20 100644 --- a/scripts/lib/aliases.js +++ b/scripts/lib/aliases.js @@ -1,6 +1,7 @@ import speciesnetSource from '../../src/shared/commonNames/sources/speciesnet.json' with { type: 'json' } import deepfauneSource from '../../src/shared/commonNames/sources/deepfaune.json' with { type: 'json' } import manasSource from '../../src/shared/commonNames/sources/manas.json' with { type: 'json' } +import megadetectorSource from '../../src/shared/commonNames/sources/megadetector.json' with { type: 'json' } import extras from '../../src/shared/commonNames/extras.json' with { type: 'json' } import { normalizeScientificName } from '../../src/shared/commonNames/normalize.js' @@ -14,7 +15,7 @@ import { normalizeScientificName } from '../../src/shared/commonNames/normalize. */ export function buildAliasMap() { const aliases = new Map() - const sources = [speciesnetSource, deepfauneSource, manasSource, extras] + const sources = [speciesnetSource, deepfauneSource, manasSource, megadetectorSource, extras] for (const src of sources) { for (const entry of src.entries || []) { if (!entry.scientificName || !entry.label) continue diff --git a/src/main/services/export/exporter.js b/src/main/services/export/exporter.js index bcb75892..84a79133 100644 --- a/src/main/services/export/exporter.js +++ b/src/main/services/export/exporter.js @@ -25,6 +25,7 @@ import { sanitizeDatapackage, CAMTRAP_DP_PROFILE_URL } from './sanitizers.js' +import { mapObservationType } from './observationType.js' function getStudyDatabasePath(userDataPath, studyId) { return join(getStudyPath(userDataPath, studyId), 'study.db') @@ -553,21 +554,8 @@ function inferMimeType(filePath) { return MIME_TYPES[ext] || 'application/octet-stream' } -/** - * Map internal observationType to Camtrap DP vocabulary - * Valid values: animal, human, vehicle, blank, unknown, unclassified - */ -function mapObservationType(dbType, scientificName) { - // If scientificName is present, it's an animal observation - if (scientificName) return 'animal' - if (!dbType || dbType === 'blank') return 'blank' - if (dbType === 'machine') return 'animal' - if (dbType === 'animal') return 'animal' - if (dbType === 'human') return 'human' - if (dbType === 'vehicle') return 'vehicle' - if (dbType === 'unclassified') return 'unclassified' - return 'unknown' -} +// mapObservationType is imported from ./observationType.js — see that module +// for the Camtrap DP vocabulary mapping and MegaDetector-specific routing. /** * Group observations into sequences based on deployment and timestamp gap. diff --git a/src/main/services/export/observationType.js b/src/main/services/export/observationType.js new file mode 100644 index 00000000..f96ba2bf --- /dev/null +++ b/src/main/services/export/observationType.js @@ -0,0 +1,35 @@ +/** + * Map internal observationType + scientificName to Camtrap DP vocabulary. + * + * Valid output values: `animal`, `human`, `vehicle`, `blank`, `unknown`, `unclassified`. + * + * MegaDetector's three pseudo-species labels (`animal`, `homo sapiens`, + * `vehicle`) map directly to Camtrap DP's vocabulary. We special-case + * `homo sapiens` → `human` and `vehicle` → `vehicle` BEFORE the generic + * "any scientificName → animal" fallback, otherwise the fallback would + * shadow them. The string matches also catch the same values coming from + * any other source (e.g. a SpeciesNet "homo sapiens" classification), + * which is semantically correct. + * + * Extracted from `exporter.js` into a pure module so it can be unit-tested + * without transitively loading the rest of the export pipeline (which pulls + * in `download.ts` and friends). + * + * @param {string|null} dbType - The `observationType` column value (`'machine'`, + * `'animal'`, `'human'`, `'vehicle'`, `'blank'`, `'unclassified'`, or null). + * @param {string|null} scientificName - The observation's scientific name. + * @returns {'animal'|'human'|'vehicle'|'blank'|'unknown'|'unclassified'} + */ +export function mapObservationType(dbType, scientificName) { + if (scientificName === 'homo sapiens') return 'human' + if (scientificName === 'vehicle') return 'vehicle' + // Any other scientificName (binomials, MD's literal "animal", ...) → animal. + if (scientificName) return 'animal' + if (!dbType || dbType === 'blank') return 'blank' + if (dbType === 'machine') return 'animal' + if (dbType === 'animal') return 'animal' + if (dbType === 'human') return 'human' + if (dbType === 'vehicle') return 'vehicle' + if (dbType === 'unclassified') return 'unclassified' + return 'unknown' +} diff --git a/src/main/services/ml/classification.js b/src/main/services/ml/classification.js index 376f131b..b24189ed 100644 --- a/src/main/services/ml/classification.js +++ b/src/main/services/ml/classification.js @@ -4,12 +4,21 @@ */ /** - * Select winner species from aggregated frame data using majority voting. + * Select winner species from aggregated frame data. * - * Algorithm: - * 1. Primary criteria: Species with the highest frame count wins - * 2. Tiebreaker: If multiple species have the same frame count, the one with - * the highest average confidence wins + * Two modes, controlled by the `weightedVote` option: + * + * Default (frame-count primary) — for true classifiers (SpeciesNet, + * DeepFaune, Manas) where every frame is an independent classification of + * the same scene. The species detected in the most frames wins; ties are + * broken by average confidence. + * + * `weightedVote: true` (mean-confidence primary) — for detection-only + * models (MegaDetector). MD's labels are coarse (animal/person/vehicle) + * and a video can legitimately contain multiple labels in different + * frames, so a noisy stream of low-confidence "animal" detections must + * not drown out a few high-confidence "person" detections. The label + * with the highest *mean* confidence wins; frame count is the tiebreaker. * * @param {Map} speciesMap * Map of species names to their frame data: @@ -18,31 +27,37 @@ * - firstFrame: First frame number where species appeared * - lastFrame: Last frame number where species appeared * + * @param {Object} [options] + * @param {boolean} [options.weightedVote=false] Use mean-confidence as the + * primary criterion (with frame count as tiebreaker) instead of the default + * frame-count-primary logic. + * * @returns {{ winner: string|null, winnerData: object|null }} * - winner: The winning species name, or null if no species detected * - winnerData: The winning species data with avgConfidence added, or null */ -export function selectVideoClassificationWinner(speciesMap) { +export function selectVideoClassificationWinner(speciesMap, options = {}) { if (!speciesMap || speciesMap.size === 0) { return { winner: null, winnerData: null } } + const { weightedVote = false } = options + let winner = null let winnerData = null - let maxFrameCount = 0 - let maxAvgConfidence = 0 + let bestPrimary = -Infinity + let bestSecondary = -Infinity for (const [species, data] of speciesMap) { const frameCount = data.frames.length const avgConfidence = data.scores.reduce((sum, s) => sum + s, 0) / data.scores.length - // Primary: highest frame count; Tiebreaker: highest average confidence - if ( - frameCount > maxFrameCount || - (frameCount === maxFrameCount && avgConfidence > maxAvgConfidence) - ) { - maxFrameCount = frameCount - maxAvgConfidence = avgConfidence + const primary = weightedVote ? avgConfidence : frameCount + const secondary = weightedVote ? frameCount : avgConfidence + + if (primary > bestPrimary || (primary === bestPrimary && secondary > bestSecondary)) { + bestPrimary = primary + bestSecondary = secondary winner = species winnerData = { ...data, avgConfidence } } diff --git a/src/main/services/ml/server.ts b/src/main/services/ml/server.ts index f96c87f4..1bc357cd 100644 --- a/src/main/services/ml/server.ts +++ b/src/main/services/ml/server.ts @@ -14,6 +14,7 @@ import crypto from 'crypto' import os from 'node:os' import { getMLModelLocalInstallPath, getMLModelEnvironmentLocalInstallPath } from './paths.js' +import { findModel } from '../../../shared/mlmodels.js' // ============================================================================ // Active Server Registry @@ -509,6 +510,75 @@ export async function startManasHTTPServer({ return { process: pythonProcess, shutdownApiKey } } +interface MegaDetectorServerOptions { + port: number + detectorWeightsFilepath: string + detectionConfidenceThreshold: number + timeout: number + pythonEnvironment: { + reference: { id: string; version: string } + } +} + +/** + * Starts the MegaDetector HTTP server using a specified Python environment and configuration. + * + * @async + * @param {MegaDetectorServerOptions} options + * @returns {Promise<{process: ChildProcess, shutdownApiKey: string}>} + */ +export async function startMegaDetectorHTTPServer({ + port, + detectorWeightsFilepath, + detectionConfidenceThreshold, + timeout, + pythonEnvironment +}: MegaDetectorServerOptions): Promise<{ process: ChildProcess; shutdownApiKey: string }> { + log.info('startMegaDetectorHTTPServer initiated') + log.info(pythonEnvironment) + const localInstalRootDirPythonEnvironment = join( + getMLModelEnvironmentLocalInstallPath({ + ...pythonEnvironment.reference + }), + pythonEnvironment.reference.id + ) + log.info('Local Python Environment root dir is', localInstalRootDirPythonEnvironment) + const scriptPath = is.dev + ? join(__dirname, '../../python-environments/common/run_megadetector_server.py') + : join(process.resourcesPath, 'python-environments', 'common', 'run_megadetector_server.py') + const pythonInterpreter = is.dev + ? join(__dirname, '../../python-environments/common/.venv/bin/python') + : os.platform() === 'win32' + ? join(localInstalRootDirPythonEnvironment, 'python.exe') + : join(localInstalRootDirPythonEnvironment, 'bin', 'python') + log.info('Python Interpreter found in', pythonInterpreter) + log.info('Script path is', scriptPath) + const scriptArgs = [ + '--port', + String(port), + '--filepath-detector-weights', + detectorWeightsFilepath, + '--detection-confidence-threshold', + String(detectionConfidenceThreshold), + '--timeout', + String(timeout) + ] + log.info('Script args: ', scriptArgs) + + const shutdownApiKey = crypto.randomUUID() + log.info('Generated shutdown API key for MegaDetector server') + + const pythonProcess = await startAndWaitTillServerHealty({ + pythonInterpreter, + scriptPath, + scriptArgs, + healthEndpoint: `http://localhost:${port}/health`, + env: { LIT_SHUTDOWN_API_KEY: shutdownApiKey } + }) + + return { process: pythonProcess, shutdownApiKey } +} + // ============================================================================ // Server Lifecycle Management // ============================================================================ @@ -711,6 +781,28 @@ export async function startMLModelHTTPServer({ }) return { port: port, process: pythonProcess, shutdownApiKey } } + case 'megadetector': { + const port = await resolveServerPort(is.dev ? 8003 : null) + const localInstallPath = getMLModelLocalInstallPath({ ...modelReference }) + log.info(`Local ML Model install path ${localInstallPath}`) + const detectorWeightsFilepath = join(localInstallPath, 'MDV6-yolov10-e-1280.pt') + const model = findModel({ ...modelReference }) + const { process: pythonProcess, shutdownApiKey } = await startMegaDetectorHTTPServer({ + port, + detectorWeightsFilepath, + detectionConfidenceThreshold: model?.detectionConfidenceThreshold ?? 0.2, + timeout: 30, + pythonEnvironment: pythonEnvironment + }) + log.info(`pythonProcess: ${JSON.stringify(pythonProcess)}`) + registerActiveServer({ + pid: pythonProcess.pid as number, + port, + shutdownApiKey, + modelId: modelReference.id + }) + return { port: port, process: pythonProcess, shutdownApiKey } + } default: { log.warn( `startMLModelHTTPServer: Not implemented for ${modelReference.id} version ${modelReference.version}` diff --git a/src/main/services/prediction.js b/src/main/services/prediction.js index 0136c0c2..0291fe8b 100644 --- a/src/main/services/prediction.js +++ b/src/main/services/prediction.js @@ -226,12 +226,14 @@ export async function getMedia(db, filepath) { /** * Parse scientific name from prediction based on model type * @param {Object} prediction - Model prediction output - * @param {string} modelType - 'speciesnet' | 'deepfaune' | 'manas' + * @param {string} modelType - 'speciesnet' | 'deepfaune' | 'manas' | 'megadetector' * @returns {string|null} Scientific name or null for blank predictions */ function parseScientificName(prediction, modelType) { - if (modelType === 'deepfaune' || modelType === 'manas') { - // DeepFaune/Manas: Simple label like "chamois", "panthera_uncia", "blank", "empty", "vide" + if (modelType === 'deepfaune' || modelType === 'manas' || modelType === 'megadetector') { + // DeepFaune/Manas/MegaDetector: Simple label like "chamois", "panthera_uncia", + // "animal", "vehicle", "homo sapiens" (MD's translation of "person"), or + // "blank"/"empty"/"vide"/"error" for no-detection cases. const label = prediction.prediction if (!label || label === 'blank' || label === 'empty' || label === 'vide' || label === 'error') { return null @@ -327,7 +329,12 @@ export async function insertPrediction(db, prediction, modelInfo = {}) { // Combine: best + filtered additional const validDetections = [bestDetection, ...additionalDetections] - // Create one observation per valid detection + // Create one observation per valid detection. For detection-only models + // (MegaDetector) there is no whole-image classifier, so each bbox's + // detection confidence IS the classification probability — write it + // per-row instead of copying the top-level prediction_score across all + // rows (which would render as the same number on every box in the UI). + const isDetectionOnly = modelType === 'megadetector' for (const detection of validDetections) { const bbox = transformBboxToCamtrapDP(detection, modelType) const observationData = { @@ -337,7 +344,8 @@ export async function insertPrediction(db, prediction, modelInfo = {}) { bboxY: bbox?.bboxY ?? null, bboxWidth: bbox?.bboxWidth ?? null, bboxHeight: bbox?.bboxHeight ?? null, - detectionConfidence: detection.conf + detectionConfidence: detection.conf, + ...(isDetectionOnly && { classificationProbability: detection.conf }) } await db.insert(observations).values(observationData) } @@ -451,8 +459,12 @@ export async function insertVideoPredictions(db, predictions, mediaRecord, model entry.lastFrame = Math.max(entry.lastFrame, pred.frame_number) } - // 4. Select winner using majority voting with average confidence tiebreaker - const { winner, winnerData } = selectVideoClassificationWinner(speciesMap) + // 4. Select winner. Detection-only models (MegaDetector) use mean-confidence + // voting so a noisy stream of low-conf "animal" frames can't drown out a + // few high-conf "person" frames. Classifiers keep the frame-count-primary + // majority vote, which suits per-frame classification of the same scene. + const weightedVote = modelType === 'megadetector' + const { winner, winnerData } = selectVideoClassificationWinner(speciesMap, { weightedVote }) // 5. Create exactly ONE observation (winner or blank) const fps = mediaRecord.exifData?.fps || 1 diff --git a/src/main/utils/bbox.js b/src/main/utils/bbox.js index d675dc20..9ffef372 100644 --- a/src/main/utils/bbox.js +++ b/src/main/utils/bbox.js @@ -43,6 +43,7 @@ export function transformBboxToCamtrapDP(detection, modelType) { } case 'manas': + case 'megadetector': case 'deepfaune': { // DeepFaune/Manas xywhn format: [x_center, y_center, width, height] (normalized 0-1, center format) if (!detection.xywhn || !Array.isArray(detection.xywhn) || detection.xywhn.length < 4) { @@ -94,6 +95,12 @@ export function detectModelType(prediction) { return 'manas' } + // MegaDetector v6.0 also uses xywhn (ultralytics YOLO output) and ships model_version '6.0'. + // Branch before the generic xywhn fallback so it doesn't get misrouted as 'deepfaune'. + if (version === '6.0' && prediction.detections?.[0]?.xywhn) { + return 'megadetector' + } + // DeepFaune versions typically look like "1.3" if (version.match(/^\d+\.\d+$/) && prediction.detections?.[0]?.xywhn) { return 'deepfaune' diff --git a/src/renderer/src/models/ModelCard.jsx b/src/renderer/src/models/ModelCard.jsx index a64f461c..fa86dffe 100644 --- a/src/renderer/src/models/ModelCard.jsx +++ b/src/renderer/src/models/ModelCard.jsx @@ -179,7 +179,11 @@ export default function ModelCard({
v{model.reference.version} · {formatSize(model.size_in_MB)} ·{' '} - {model.species_count} species + + {model.detectionOnly + ? `Detection only · ${model.species_count} categories` + : `${model.species_count} species`} +
{!isDownloading && ( @@ -200,17 +204,21 @@ export default function ModelCard({ )} -
{ - e.stopPropagation() - onToggleSpecies?.(model.reference.id) - }} - > - {speciesOpen ? '▾ Hide species' : `▸ View ${model.species_count} species`} -
+ {!model.detectionOnly && ( + <> +
{ + e.stopPropagation() + onToggleSpecies?.(model.reference.id) + }} + > + {speciesOpen ? '▾ Hide species' : `▸ View ${model.species_count} species`} +
- {speciesOpen && speciesPanel} + {speciesOpen && speciesPanel} + + )} ) } diff --git a/src/renderer/src/models/ModelSelect.jsx b/src/renderer/src/models/ModelSelect.jsx index c5f4c0ec..bc8dd2f4 100644 --- a/src/renderer/src/models/ModelSelect.jsx +++ b/src/renderer/src/models/ModelSelect.jsx @@ -61,7 +61,11 @@ function ModelRow({ model, status }) {
- {model.species_count} species + + {model.detectionOnly + ? `Detection only · ${model.species_count} categories` + : `${model.species_count} species`} +

{model.description}

diff --git a/src/shared/commonNames/dictionary.json b/src/shared/commonNames/dictionary.json index 97f53af5..dde6d6f5 100644 --- a/src/shared/commonNames/dictionary.json +++ b/src/shared/commonNames/dictionary.json @@ -132,6 +132,7 @@ "aneides lugubris": "arboreal salamander", "anhima cornuta": "horned screamer", "anhinga anhinga": "anhinga", + "animal": "animal", "annamite_striped_rabbit": "annamite striped rabbit", "anolis carolinensis": "green anole", "anolis sagrei": "cuban brown anole", @@ -2289,6 +2290,7 @@ "peromyscus leucopus": "white-footed mouse", "peromyscus maniculatus": "north american deermouse", "peromyscus species": "peromyscus species", + "person": "human", "petaurista elegans": "spotted giant flying squirrel", "petaurista petaurista": "red giant flying squirrel", "petaurista philippensis": "indian giant flying squirrel", diff --git a/src/shared/commonNames/labelAliases.json b/src/shared/commonNames/labelAliases.json index a15ab76e..9ea0f0ff 100644 --- a/src/shared/commonNames/labelAliases.json +++ b/src/shared/commonNames/labelAliases.json @@ -293,6 +293,7 @@ "pacific wren": "troglodytes pacificus", "painted_turtle": "chrysemys picta", "panthera_uncia": "panthera uncia", + "person": "homo sapiens", "phrynosomatidae family": "phrynosomatidae", "pig": "sus scrofa", "pig_tailed_macaque": "macaca nemestrina", diff --git a/src/shared/commonNames/sources/megadetector.json b/src/shared/commonNames/sources/megadetector.json new file mode 100644 index 00000000..61c65a0a --- /dev/null +++ b/src/shared/commonNames/sources/megadetector.json @@ -0,0 +1,22 @@ +{ + "modelId": "megadetector", + "modelVersion": "6.0", + "source": "MegaDetector v6 categories (animal / person / vehicle)", + "entries": [ + { + "scientificName": null, + "label": "animal", + "commonName": "Animal" + }, + { + "scientificName": "homo sapiens", + "label": "person", + "commonName": "Human" + }, + { + "scientificName": null, + "label": "vehicle", + "commonName": "Vehicle" + } + ] +} diff --git a/src/shared/mlmodels.js b/src/shared/mlmodels.js index 02790a5c..11ac0903 100644 --- a/src/shared/mlmodels.js +++ b/src/shared/mlmodels.js @@ -172,6 +172,23 @@ export const modelZoo = [ species_count: '2,000+', species_data: 'speciesnet' }, + { + reference: { id: 'megadetector', version: '6.0' }, + pythonEnvironment: { id: 'common', version: '0.1.4' }, + name: 'MegaDetector', + size_in_MB: 118, // tarball ~117.8 MB, weights file ~128 MB — replace with exact size once HF upload lands + files: 2, + downloadURL: + 'https://huggingface.co/earthtoolsmaker/megadetector/resolve/main/6.0.tar.gz?download=true', + description: + "MegaDetector is Microsoft AI for Earth's open-source detector for camera trap images. It locates animals, people, and vehicles in any image worldwide, without identifying species. Useful as a fast blank-filter before manual species annotation.", + website: 'https://github.com/microsoft/MegaDetector', + logo: 'megadetector', + detectionConfidenceThreshold: 0.2, + region: 'worldwide', + species_count: 3, + detectionOnly: true + }, { reference: { id: 'deepfaune', version: '1.3' }, pythonEnvironment: { id: 'common', version: '0.1.4' }, diff --git a/test/main/services/export/observationType.test.js b/test/main/services/export/observationType.test.js new file mode 100644 index 00000000..f3bc75d9 --- /dev/null +++ b/test/main/services/export/observationType.test.js @@ -0,0 +1,42 @@ +import { test, describe } from 'node:test' +import assert from 'node:assert/strict' + +import { mapObservationType } from '../../../../src/main/services/export/observationType.js' + +describe('mapObservationType — Camtrap DP vocabulary', () => { + test('MegaDetector "homo sapiens" exports as "human"', () => { + // Regression test: previously the early-return `if (scientificName) return 'animal'` + // shadowed MD's homo sapiens label, exporting humans as animals. + assert.equal(mapObservationType('machine', 'homo sapiens'), 'human') + assert.equal(mapObservationType(null, 'homo sapiens'), 'human') + }) + + test('MegaDetector "vehicle" exports as "vehicle"', () => { + // Same regression as homo sapiens — vehicles were silently mapped to animal. + assert.equal(mapObservationType('machine', 'vehicle'), 'vehicle') + assert.equal(mapObservationType(null, 'vehicle'), 'vehicle') + }) + + test('MegaDetector "animal" exports as "animal"', () => { + // MD's literal animal label still maps to animal (was coincidentally correct). + assert.equal(mapObservationType('machine', 'animal'), 'animal') + assert.equal(mapObservationType(null, 'animal'), 'animal') + }) + + test('real binomial scientific names still export as "animal"', () => { + // Locks pre-existing behavior for true classifier output. + assert.equal(mapObservationType('machine', 'Vulpes vulpes'), 'animal') + assert.equal(mapObservationType('machine', 'Panthera leo'), 'animal') + assert.equal(mapObservationType('machine', 'sciurus vulgaris'), 'animal') + }) + + test('null scientificName falls through to dbType branches', () => { + assert.equal(mapObservationType(null, null), 'blank') + assert.equal(mapObservationType('blank', null), 'blank') + assert.equal(mapObservationType('machine', null), 'animal') + assert.equal(mapObservationType('human', null), 'human') + assert.equal(mapObservationType('vehicle', null), 'vehicle') + assert.equal(mapObservationType('unclassified', null), 'unclassified') + assert.equal(mapObservationType('something-unexpected', null), 'unknown') + }) +}) diff --git a/test/main/services/ml/classification.test.js b/test/main/services/ml/classification.test.js index 452e2754..49519b24 100644 --- a/test/main/services/ml/classification.test.js +++ b/test/main/services/ml/classification.test.js @@ -174,3 +174,81 @@ describe('selectVideoClassificationWinner', () => { assert.deepEqual(winnerData.scores, [0.8, 0.9, 0.85]) }) }) + +describe('selectVideoClassificationWinner with weightedVote=true (mean-confidence primary)', () => { + test('DSCF0028.MP4 regression — strong 2-frame person beats weak 7-frame animal', () => { + // Real per-frame data from a Biowatch import that exposed the bug. + // Pure frame-count voting picked "animal" (7 frames > 2 frames), but the + // animal detections are noisy (avg 0.36) while the person detections are + // unambiguous (avg 0.83). Weighted vote correctly picks "person". + const speciesMap = new Map([ + ['homo sapiens', { frames: [0, 1], scores: [0.945, 0.708], firstFrame: 0, lastFrame: 1 }], + [ + 'animal', + { + frames: [2, 3, 4, 5, 6, 8, 9], + scores: [0.492, 0.301, 0.35, 0.36, 0.323, 0.374, 0.319], + firstFrame: 2, + lastFrame: 9 + } + ] + ]) + const { winner, winnerData } = selectVideoClassificationWinner(speciesMap, { + weightedVote: true + }) + assert.equal(winner, 'homo sapiens') + assert.ok(Math.abs(winnerData.avgConfidence - 0.8265) < 0.001) + }) + + test('with weightedVote=false (default), frame count wins (existing behavior)', () => { + // Same input, default mode. Locks the existing classifier-friendly logic. + const speciesMap = new Map([ + ['homo sapiens', { frames: [0, 1], scores: [0.945, 0.708], firstFrame: 0, lastFrame: 1 }], + [ + 'animal', + { + frames: [2, 3, 4, 5, 6, 8, 9], + scores: [0.492, 0.301, 0.35, 0.36, 0.323, 0.374, 0.319], + firstFrame: 2, + lastFrame: 9 + } + ] + ]) + const { winner } = selectVideoClassificationWinner(speciesMap) + assert.equal(winner, 'animal') + }) + + test('frame count is the tiebreaker when mean confidences match', () => { + const speciesMap = new Map([ + ['A', { frames: [1], scores: [0.8], firstFrame: 1, lastFrame: 1 }], + ['B', { frames: [2, 3, 4], scores: [0.8, 0.8, 0.8], firstFrame: 2, lastFrame: 4 }] + ]) + const { winner } = selectVideoClassificationWinner(speciesMap, { weightedVote: true }) + assert.equal(winner, 'B') // same mean (0.8), more frames → wins + }) + + test('higher mean confidence wins regardless of frame count', () => { + const speciesMap = new Map([ + ['high-conf-few', { frames: [1, 2], scores: [0.95, 0.93], firstFrame: 1, lastFrame: 2 }], + [ + 'low-conf-many', + { + frames: [3, 4, 5, 6, 7, 8], + scores: [0.4, 0.4, 0.4, 0.4, 0.4, 0.4], + firstFrame: 3, + lastFrame: 8 + } + ] + ]) + const { winner } = selectVideoClassificationWinner(speciesMap, { weightedVote: true }) + assert.equal(winner, 'high-conf-few') + }) + + test('single-species video still wins regardless of mode', () => { + const speciesMap = new Map([ + ['Vulpes vulpes', { frames: [1, 2, 3], scores: [0.7, 0.7, 0.7], firstFrame: 1, lastFrame: 3 }] + ]) + const { winner } = selectVideoClassificationWinner(speciesMap, { weightedVote: true }) + assert.equal(winner, 'Vulpes vulpes') + }) +}) diff --git a/test/main/services/predictionCommonName.test.js b/test/main/services/predictionCommonName.test.js index d72a1f36..9ee66297 100644 --- a/test/main/services/predictionCommonName.test.js +++ b/test/main/services/predictionCommonName.test.js @@ -118,4 +118,115 @@ describe('insertPrediction populates commonName via dictionary', () => { assert.equal(insertedRows[0].scientificName, null) assert.equal(insertedRows[0].commonName, null) }) + + test('MegaDetector "animal" label stores as simple label with resolved common name', async () => { + const { db, insertedRows } = makeFakeDb({ mediaRecord: baseMedia() }) + const prediction = { + filepath: '/fake/img.jpg', + prediction: 'animal', + prediction_score: 0.7, + detections: [] + } + + await insertPrediction(db, prediction, { modelID: 'megadetector' }) + + assert.equal(insertedRows.length, 1) + // Must NOT be parsed as a SpeciesNet binomial — that produced "undefined undefined" in production. + assert.equal(insertedRows[0].scientificName, 'animal') + assert.equal(insertedRows[0].commonName, 'animal') + }) + + test('MegaDetector "homo sapiens" translation lands with Human common name', async () => { + const { db, insertedRows } = makeFakeDb({ mediaRecord: baseMedia() }) + const prediction = { + filepath: '/fake/img.jpg', + prediction: 'homo sapiens', + prediction_score: 0.93, + detections: [] + } + + await insertPrediction(db, prediction, { modelID: 'megadetector' }) + + assert.equal(insertedRows.length, 1) + assert.equal(insertedRows[0].scientificName, 'homo sapiens') + assert.equal(insertedRows[0].commonName, 'human') + }) + + test('MegaDetector blank prediction leaves both scientificName and commonName null', async () => { + const { db, insertedRows } = makeFakeDb({ mediaRecord: baseMedia() }) + const prediction = { + filepath: '/fake/img.jpg', + prediction: 'blank', + // prediction_score deliberately omitted — server omits it for blanks + detections: [] + } + + await insertPrediction(db, prediction, { modelID: 'megadetector' }) + + assert.equal(insertedRows.length, 1) + assert.equal(insertedRows[0].scientificName, null) + assert.equal(insertedRows[0].commonName, null) + }) + + test('MegaDetector classificationProbability is per-detection (not the top-level prediction_score)', async () => { + // MD has no classifier — every bbox is a separate detection. Each + // observation row must carry that detection's own confidence, not the + // top-level prediction_score copied across all rows. Regression test for + // the production bug where 6 detections all showed conf=0.70 in the UI. + const { db, insertedRows } = makeFakeDb({ mediaRecord: baseMedia() }) + const prediction = { + filepath: '/fake/img.jpg', + prediction: 'animal', + prediction_score: 0.699, + detections: [ + { class: 0, label: 'animal', conf: 0.699, xywhn: [0.1, 0.5, 0.1, 0.2], xyxy: [0, 0, 0, 0] }, + { class: 0, label: 'animal', conf: 0.446, xywhn: [0.2, 0.6, 0.1, 0.1], xyxy: [0, 0, 0, 0] }, + { class: 0, label: 'animal', conf: 0.293, xywhn: [0.3, 0.7, 0.1, 0.1], xyxy: [0, 0, 0, 0] } + ] + } + + await insertPrediction(db, prediction, { + modelID: 'megadetector', + detectionConfidenceThreshold: 0.2 + }) + + assert.equal(insertedRows.length, 3) + // Sorted by conf desc per insertPrediction's existing behavior + assert.equal(insertedRows[0].classificationProbability, 0.699) + assert.equal(insertedRows[1].classificationProbability, 0.446) + assert.equal(insertedRows[2].classificationProbability, 0.293) + // detectionConfidence must match — for MD they are the same value. + assert.equal(insertedRows[0].detectionConfidence, 0.699) + assert.equal(insertedRows[1].detectionConfidence, 0.446) + assert.equal(insertedRows[2].detectionConfidence, 0.293) + }) + + test('SpeciesNet classificationProbability remains the top-level prediction_score across detections', async () => { + // For a true classifier model the classification probability IS the + // whole-image classifier output — same across all bboxes by design. + // Locks the existing behavior so the MD-specific change doesn't regress it. + const { db, insertedRows } = makeFakeDb({ mediaRecord: baseMedia() }) + const prediction = { + filepath: '/fake/img.jpg', + prediction: + '00000000-0000-0000-0000-000000000001;mammalia;rodentia;sciuridae;sciurus;vulgaris;eurasian red squirrel', + prediction_score: 0.95, + detections: [ + { category: '1', label: 'animal', conf: 0.99, bbox: [0.1, 0.1, 0.1, 0.1] }, + { category: '1', label: 'animal', conf: 0.7, bbox: [0.3, 0.3, 0.1, 0.1] } + ] + } + + await insertPrediction(db, prediction, { + modelID: 'speciesnet', + detectionConfidenceThreshold: 0.5 + }) + + assert.equal(insertedRows.length, 2) + assert.equal(insertedRows[0].classificationProbability, 0.95) + assert.equal(insertedRows[1].classificationProbability, 0.95) + // detectionConfidence remains per-bbox + assert.equal(insertedRows[0].detectionConfidence, 0.99) + assert.equal(insertedRows[1].detectionConfidence, 0.7) + }) }) diff --git a/test/main/utils/bbox.test.js b/test/main/utils/bbox.test.js new file mode 100644 index 00000000..35608623 --- /dev/null +++ b/test/main/utils/bbox.test.js @@ -0,0 +1,24 @@ +import { test, describe } from 'node:test' +import assert from 'node:assert/strict' +import { transformBboxToCamtrapDP, detectModelType } from '../../../src/main/utils/bbox.js' + +describe('MegaDetector bbox handling', () => { + test('transformBboxToCamtrapDP routes megadetector through xywhn → top-left', () => { + const detection = { xywhn: [0.5, 0.5, 0.4, 0.6] } + const result = transformBboxToCamtrapDP(detection, 'megadetector') + assert.deepEqual(result, { + bboxX: 0.3, // 0.5 - 0.4/2 + bboxY: 0.2, // 0.5 - 0.6/2 + bboxWidth: 0.4, + bboxHeight: 0.6 + }) + }) + + test('detectModelType identifies megadetector by version 6.0 + xywhn', () => { + const prediction = { + model_version: '6.0', + detections: [{ xywhn: [0.5, 0.5, 0.4, 0.6], conf: 0.9, label: 'animal' }] + } + assert.equal(detectModelType(prediction), 'megadetector') + }) +}) diff --git a/test/shared/commonNames/resolver.test.js b/test/shared/commonNames/resolver.test.js index c15f3f3a..4f41f530 100644 --- a/test/shared/commonNames/resolver.test.js +++ b/test/shared/commonNames/resolver.test.js @@ -32,4 +32,17 @@ describe('resolveCommonName', () => { test('returns null for unknown scientific name', () => { assert.equal(resolveCommonName('Foobar nonexistentium'), null) }) + + test('resolves MegaDetector pseudo-species labels (animal/vehicle)', () => { + assert.equal(resolveCommonName('animal'), 'animal') + assert.equal(resolveCommonName('vehicle'), 'vehicle') + }) + + test("resolves MegaDetector's 'person' label and its 'homo sapiens' binomial to 'human'", () => { + // MD's raw label is 'person', but the Python server emits 'homo sapiens' + // as the prediction. Both keys resolve to the same common name so either + // route (alias-translated or directly emitted) renders consistently. + assert.equal(resolveCommonName('person'), 'human') + assert.equal(resolveCommonName('homo sapiens'), 'human') + }) }) diff --git a/test/shared/mlmodels.test.js b/test/shared/mlmodels.test.js index 82217021..ba79d35c 100644 --- a/test/shared/mlmodels.test.js +++ b/test/shared/mlmodels.test.js @@ -187,6 +187,18 @@ describe('ML Model Management Tests', () => { assert.equal(result, null, 'Should return null for wrong version') }) + + test('should find megadetector entry with detectionOnly flag', () => { + const md = findModel({ id: 'megadetector', version: '6.0' }) + assert(md, 'megadetector entry should exist') + assert.equal(md.detectionOnly, true, 'megadetector should be marked detectionOnly') + assert.equal(md.region, 'worldwide', 'megadetector should be worldwide') + assert.equal(md.species_count, 3, 'megadetector should report 3 categories') + assert( + md.downloadURL.includes('earthtoolsmaker/megadetector'), + 'should download from ETM HF repo' + ) + }) }) describe('platformToKey', () => {