diff --git a/AGENTS.md b/AGENTS.md index 4c20377..abdf13e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,9 +6,8 @@ Durable reference for humans and agents maintaining rf-detr.cpp. rf-detr.cpp is a C++/ggml inference engine for Roboflow RF-DETR. It runs detection and segmentation natively on CPU with parity to the PyTorch -reference, and is published on HuggingFace as a set of 32 quantized GGUF -models (5 detection variants x 4 quants + 3 segmentation variants x 4 quants -plus a few extras). +reference, and is published on HuggingFace as a set of 44 quantized GGUF +models (5 detection variants x 4 quants + 6 segmentation variants x 4 quants). The repo also exposes a flat C-API (`include/rfdetr_capi.h`) intended for dlopen-based language bindings, and is integrated into LocalAI as a native @@ -48,7 +47,7 @@ cmake --build build -j ctest --test-dir build --output-on-failure ``` -Expected: 24/24 tests pass on a typical Linux dev box. Parity tests require +Expected: 25/25 tests pass on a typical Linux dev box. Parity tests require `tests/fixtures/baseline_torch*.gguf` to be present (committed to the repo). ### CMake options @@ -71,11 +70,11 @@ Notes: ## Converting a model -Set up a Python venv with the upstream rfdetr package first. +Set up a Python venv with the pinned RF-DETR conversion toolchain first. ``` python3 -m venv .venv -.venv/bin/pip install rfdetr +.venv/bin/pip install -r scripts/requirements.txt .venv/bin/python scripts/convert_rfdetr_to_gguf.py \ --variant base --dtype f16 \ diff --git a/BENCHMARK.md b/BENCHMARK.md index dd76a36..464f882 100644 --- a/BENCHMARK.md +++ b/BENCHMARK.md @@ -1,13 +1,21 @@ # Benchmark: rf-detr.cpp vs upstream Python rfdetr End-to-end CPU inference comparison between this C++ implementation and the -reference Python `rfdetr==1.7.0` package, on the same model weights, same -images, same CPU. Both implementations run on CPU only. +reference Python `rfdetr` package, on the same model weights, same images, +same CPU. Both implementations run on CPU only. The headline finding is in three plots. Raw data lives in `benchmarks/results/bench_data.json`; the rendering script (`scripts/plot_community.py`) is deterministic from that file. +**Version note:** the latency/quantization plots and tables below (Headline +through Variant comparison) predate the RF-DETR 1.9 upgrade and were measured +against `rfdetr==1.7.0` — the detection/segmentation architectures are +unchanged, so this data still applies. The accuracy section further down has +both the original `rfdetr==1.7.0` sweep and the current +[`rfdetr==1.9.0` sweep](#accuracy-across-the-full-model-matrix-rfdetr-190), +clearly labeled. + --- ## Hardware @@ -411,12 +419,18 @@ to re-run after a partial conversion. Each variant downloads its pretrained --- -## Accuracy across the full model matrix +## Accuracy across the full model matrix (rfdetr 1.7.0, historical) All C++ values measured vs PyTorch ground truth on 7 COCO val images at threshold=0.5. Sweep recorded 2026-05-27 with rfdetr 1.7.0 (torch 2.5.1+cu124). Greedy 1-1 matching, score-desc, same class, IoU ≥ 0.5 (for `Recall@0.5`) or IoU ≥ 0.95 (for `Recall@0.95`). Score deltas are absolute, over matched pairs only. `Extra dets` is the average number of unmatched C++ detections per image (should be 0 in the ideal case). For segmentation variants, `Mean mask IoU` and `Pixel agreement` are computed over matched-pair binary masks (both at image resolution). Images: `bus.jpg`, `coco_cats.jpg`, `coco_indoor.jpg`, `coco_kitchen.jpg`, `coco_living_room.jpg`, `coco_skater.jpg`, `coco_street.jpg` +Superseded by the [rfdetr 1.9.0 sweep](#accuracy-across-the-full-model-matrix-rfdetr-190) +below, covering the same 7-image methodology on the current 44-model matrix (this table +only ever covered nano/small/base/medium/large + seg-nano/small/medium — the 32-model +predecessor to today's 44-model matrix, which adds seg-large/xlarge/2xlarge). Kept for the +record, not as the current reference. + ### Detection variants | Variant | Quant | Size (MB) | Recall@0.5 | Recall@0.95 | Max \|Δscore\| | Mean \|Δscore\| | Extra dets | @@ -473,6 +487,89 @@ Sweep produced by [`scripts/sweep_accuracy.py`](scripts/sweep_accuracy.py); tabl --- +## Accuracy across the full model matrix (rfdetr 1.9.0) + +All C++ values measured vs PyTorch ground truth on 7 COCO val images at threshold=0.5. Sweep recorded 2026-07-30 with rfdetr 1.9.0 (torch 2.5.1+cu124). Same methodology as the 1.7.0 sweep above (greedy 1-1 matching, score-desc, same class, IoU ≥ 0.5 / ≥ 0.95, absolute score deltas over matched pairs) — the same 7-image set, refetched from `images.cocodataset.org` at `val2017/{id:012d}.jpg` plus `bus.jpg` (see [Reproducing](#reproducing)), so this table and the one above are directly comparable. This is the current 44-model matrix (11 variants × 4 quants); the 1.7.0 table above only ever covered 32. + +Images: `000000000139.jpg`, `000000000632.jpg`, `000000039769.jpg`, `000000087038.jpg`, `000000252219.jpg`, `000000397133.jpg`, `bus.jpg` + +### Detection variants + +| Variant | Quant | Size (MB) | Recall@0.5 | Recall@0.95 | Max \|Δscore\| | Mean \|Δscore\| | Extra dets | +|---|---|---:|---:|---:|---:|---:|---:| +| Nano | F32 | 112.7 | 1.000 | 1.000 | 0.0030 | 0.0012 | 0.00 | +| Nano | F16 | 60.5 | 1.000 | 1.000 | 0.0020 | 0.0007 | 0.00 | +| Nano | Q8_0 | 36.0 | 1.000 | 1.000 | 0.0147 | 0.0053 | 0.00 | +| Nano | Q4_K | 29.7 | 0.949 | 0.859 | 0.0970 | 0.0311 | 0.43 | +| Small | F32 | 119.0 | 0.976 | 0.976 | 0.0042 | 0.0012 | 0.00 | +| Small | F16 | 64.0 | 0.976 | 0.976 | 0.0046 | 0.0013 | 0.00 | +| Small | Q8_0 | 38.2 | 0.948 | 0.948 | 0.0118 | 0.0041 | 0.00 | +| Small | Q4_K | 31.2 | 0.924 | 0.829 | 0.0512 | 0.0206 | 0.00 | +| Base | F32 | 119.2 | 1.000 | 1.000 | 0.0048 | 0.0012 | 0.00 | +| Base | F16 | 64.2 | 1.000 | 1.000 | 0.0045 | 0.0014 | 0.00 | +| Base | Q8_0 | 38.5 | 0.988 | 0.988 | 0.0096 | 0.0032 | 0.00 | +| Base | Q4_K | 31.5 | 0.954 | 0.884 | 0.0505 | 0.0196 | 0.29 | +| Medium | F32 | 125.0 | 0.973 | 0.962 | 0.0092 | 0.0024 | 0.00 | +| Medium | F16 | 67.2 | 0.978 | 0.967 | 0.0085 | 0.0022 | 0.00 | +| Medium | Q8_0 | 40.2 | 0.989 | 0.969 | 0.0248 | 0.0070 | 0.14 | +| Medium | Q4_K | 32.5 | 0.883 | 0.739 | 0.0523 | 0.0199 | 0.14 | +| Large | F32 | 125.9 | 1.000 | 1.000 | 0.0056 | 0.0018 | 0.00 | +| Large | F16 | 68.2 | 1.000 | 1.000 | 0.0043 | 0.0016 | 0.00 | +| Large | Q8_0 | 41.1 | 0.973 | 0.973 | 0.0091 | 0.0035 | 0.00 | +| Large | Q4_K | 33.4 | 0.909 | 0.853 | 0.0445 | 0.0184 | 0.00 | + +### Segmentation variants + +| Variant | Quant | Size (MB) | Recall@0.5 | Recall@0.95 | Mean mask IoU | Pixel agreement | Mean \|Δscore\| | +|---|---|---:|---:|---:|---:|---:|---:| +| Seg-Nano | F32 | 127.1 | 1.000 | 1.000 | 0.9984 | 0.9999 | 0.0019 | +| Seg-Nano | F16 | 67.8 | 1.000 | 1.000 | 0.9984 | 0.9999 | 0.0021 | +| Seg-Nano | Q8_0 | 39.9 | 0.976 | 0.964 | 0.9926 | 0.9998 | 0.0075 | +| Seg-Nano | Q4_K | 31.8 | 0.912 | 0.530 | 0.9633 | 0.9990 | 0.0359 | +| Seg-Small | F32 | 127.6 | 1.000 | 1.000 | 0.9982 | 0.9999 | 0.0040 | +| Seg-Small | F16 | 68.3 | 1.000 | 1.000 | 0.9981 | 0.9999 | 0.0038 | +| Seg-Small | Q8_0 | 40.4 | 0.980 | 0.980 | 0.9937 | 0.9999 | 0.0064 | +| Seg-Small | Q4_K | 32.4 | 0.944 | 0.394 | 0.9721 | 0.9990 | 0.0323 | +| Seg-Medium | F32 | 133.9 | 0.984 | 0.984 | 0.9978 | 0.9999 | 0.0037 | +| Seg-Medium | F16 | 71.6 | 0.984 | 0.984 | 0.9970 | 0.9999 | 0.0037 | +| Seg-Medium | Q8_0 | 42.5 | 0.984 | 0.984 | 0.9935 | 0.9999 | 0.0059 | +| Seg-Medium | Q4_K | 33.6 | 0.914 | 0.479 | 0.9687 | 0.9993 | 0.0248 | +| Seg-Large | F32 | 134.6 | 0.976 | 0.976 | 0.9982 | 0.9999 | 0.0035 | +| Seg-Large | F16 | 72.3 | 0.976 | 0.976 | 0.9980 | 0.9999 | 0.0036 | +| Seg-Large | Q8_0 | 43.1 | 0.988 | 0.988 | 0.9943 | 0.9999 | 0.0077 | +| Seg-Large | Q4_K | 34.3 | 0.964 | 0.693 | 0.9730 | 0.9993 | 0.0264 | +| Seg-XLarge | F32 | 141.9 | 0.982 | 0.982 | 0.9964 | 0.9999 | 0.0070 | +| Seg-XLarge | F16 | 76.7 | 0.982 | 0.982 | 0.9965 | 0.9999 | 0.0071 | +| Seg-XLarge | Q8_0 | 46.1 | 1.000 | 1.000 | 0.9941 | 0.9998 | 0.0076 | +| Seg-XLarge | Q4_K | 36.6 | 0.905 | 0.672 | 0.9726 | 0.9994 | 0.0265 | +| Seg-2XLarge | F32 | 143.9 | 1.000 | 1.000 | 0.9964 | 0.9999 | 0.0090 | +| Seg-2XLarge | F16 | 78.7 | 0.982 | 0.970 | 0.9964 | 0.9999 | 0.0082 | +| Seg-2XLarge | Q8_0 | 48.2 | 0.970 | 0.970 | 0.9932 | 0.9999 | 0.0085 | +| Seg-2XLarge | Q4_K | 38.6 | 0.954 | 0.673 | 0.9724 | 0.9994 | 0.0323 | + +### Takeaways vs the 1.7.0 sweep + +- **Detection F32/F16/Q8_0 hold up or improve.** Nano goes from 0.989/0.989 (1.7.0) to a + clean 1.000/1.000 at F32/F16; Medium goes from 0.939/0.894 to 0.973/0.962. This reflects + the combination of the new antialias-free bilinear preprocessing and, for some variants, + refreshed 1.9.0 pretrained checkpoints upstream — both changed at once, so this table + can't isolate which contributed more. +- **Q4_K's failure mode is consistent across both sweeps**, which is the strongest + cross-check available: Seg-Small Q4_K Recall@0.95 was 0.376 under 1.7.0 and 0.394 under + 1.9.0 — same variant, same weak point, two independent sweeps five months apart. +- **Q4_K remains real, variant-dependent lossy accuracy** — worst case is Seg-Small at + Recall@0.95 = 0.394, consistent with the historical table's own worst case on the same + variant. Mask quality still holds (mean mask IoU ≥ 0.96 even at Q4_K). +- The three newly-added segmentation variants (Seg-Large/XLarge/2XLarge) show the same + quant ordering as the original three (F32 ≈ F16 > Q8_0 ≫ Q4_K), which is expected — this + is also the first sweep of these variants after fixing the segmentation-head block-count + bug ("fix: segmentation head block count for medium/large/xlarge/2xlarge"), so there is no + prior data to compare them against. + +Sweep produced by [`scripts/sweep_accuracy.py`](scripts/sweep_accuracy.py); table rendered by [`scripts/accuracy_table.py`](scripts/accuracy_table.py). Raw data: [`benchmarks/results/accuracy_sweep.json`](benchmarks/results/accuracy_sweep.json). + +--- + ## Methodology The numbers above come from `scripts/bench_community.py --rigorous` (the @@ -538,11 +635,16 @@ cmake -B build -DRFDETR_BUILD_TESTS=ON cmake --build build -j # 1. Fetch a handful of COCO val2017 images into benchmarks/images/. +# COCO filenames are zero-padded to 12 digits; naively prefixing "000000" +# only works for 6-digit ids (397133, 252219) and silently 404s for the +# others, so pad properly with printf. mkdir -p benchmarks/images && cd benchmarks/images for id in 397133 39769 139 632 252219 87038; do - curl -sSLkO "https://images.cocodataset.org/val2017/000000${id}.jpg" + padded=$(printf "%012d" "$id") + curl -sSLkO "https://images.cocodataset.org/val2017/${padded}.jpg" done -# Bring your own bus.jpg or any other JPEG to add to the set. +# Bring your own bus.jpg or any other JPEG to add to the set (this repo's +# sweep used https://raw.githubusercontent.com/ultralytics/yolov5/master/data/images/bus.jpg). cd ../.. # 2. Convert / download model variants into models/ (one-time). diff --git a/README.md b/README.md index eaca613..b7e9a83 100644 --- a/README.md +++ b/README.md @@ -7,16 +7,23 @@ [![LocalAI](https://img.shields.io/badge/LocalAI-Run_Locally-orange)](https://github.com/mudler/LocalAI) A C++ inference engine for Roboflow [RF-DETR](https://github.com/roboflow/rf-detr), built on -[ggml](https://github.com/ggml-org/ggml). Supports the full RF-DETR family: 5 detection -variants (Nano/Small/Base/Medium/Large) and 3 segmentation variants -(SegNano/SegSmall/SegMedium), with F32 / F16 / Q8_0 / Q4_K quantizations published as GGUFs -on [HuggingFace](https://huggingface.co/mudler). +[ggml](https://github.com/ggml-org/ggml). Supports 5 RF-DETR detection variants +(Nano/Small/Base/Medium/Large) and 6 segmentation variants +(SegNano/SegSmall/SegMedium/SegLarge/SegXLarge/Seg2XLarge), with F32 / F16 / Q8_0 / +Q4_K quantizations published as GGUFs on [HuggingFace](https://huggingface.co/mudler). +The keypoint-preview head is not yet supported. > Status: end-to-end detection and segmentation work on real model weights. C++ F16 is > about 9% faster than PyTorch CPU on every COCO image we tested, matches F32 accuracy > (max |Δscore| ≤ 0.006), and is 1.86x smaller. Detection match vs PyTorch is 54/55 at > IoU ≥ 0.95 across 7 COCO val2017 images. Mask IoU is 0.9924 mean across segmentation > variants. +> +> **RF-DETR 1.9:** all 11 variants (44 GGUFs) are converted, verified, and parity-swept +> against `rfdetr==1.9.0` — see [RF-DETR 1.9 parity](#rf-detr-19-parity) +> for the full per-variant, per-quant table. New GGUFs record the antialias-free bilinear +> preprocessing introduced upstream, while GGUFs without that metadata retain the prior +> resize path for output compatibility. ## Examples @@ -38,7 +45,7 @@ segmentation models overlays the per-detection mask in the same class color. ## Quickstart: prebuilt models -All 32 GGUF models (8 variants x 4 quantizations) are published on HuggingFace. Pull one +All 44 GGUF models (11 variants x 4 quantizations) are published on HuggingFace. Pull one and run detection in three commands: ```bash @@ -107,8 +114,9 @@ on Seg-Nano F32; the remaining differences are sub-pixel boundary FP rounding. To roll your own (different variant, custom checkpoint, different quant): ```bash -# One-time: convert upstream RF-DETR .pth to GGUF (requires .venv with rfdetr). -python3 -m venv .venv && .venv/bin/pip install rfdetr +# One-time: install the pinned, conversion-tested RF-DETR toolchain. +python3 -m venv .venv +.venv/bin/pip install -r scripts/requirements.txt # F16: fastest on CPU, 1.86x smaller than F32, matches F32 accuracy. .venv/bin/python scripts/convert_rfdetr_to_gguf.py \ @@ -128,7 +136,7 @@ python3 -m venv .venv && .venv/bin/pip install rfdetr # Convert all detection variants in one shot scripts/convert_all_variants.sh -# Build the full matrix (5 detection + 3 seg, 4 quants each, = 32 models) +# Build the full matrix (5 detection + 6 seg, 4 quants each, = 44 models) scripts/build_all_quants.sh ``` @@ -147,12 +155,21 @@ resulting checkpoint to GGUF: The converter reads the head size directly from the checkpoint tensor and resizes the classification head before loading, so arbitrary `num_classes` values are handled -automatically. See [`docs/finetuning.md`](docs/finetuning.md) for the end-to-end walkthrough +automatically. Checkpoints use RF-DETR 1.9's restricted loader by default; pass +`--trust-checkpoint` only for a legacy checkpoint whose source you fully trust. See +[`docs/finetuning.md`](docs/finetuning.md) for the end-to-end walkthrough (dataset prep, train, convert, quantize, serve), plus a smoke test using a synthetic 5-class checkpoint at `scripts/build_custom_checkpoint.py`. ## Benchmarks +The latency/quantization tables below (AMD Ryzen 9 9950X3D, `rfdetr-base`, 7 COCO val2017 +images) predate the RF-DETR 1.9 upgrade — the detection/segmentation architectures +themselves are unchanged, so latency and relative quant behavior still apply, but they were +measured against `rfdetr` checkpoints converted with the pre-1.9 converter. See +[**RF-DETR 1.9 parity**](#rf-detr-19-parity) below for the accuracy +numbers actually measured against `rfdetr==1.9.0` on the current 44-model matrix. + End-to-end CPU inference on AMD Ryzen 9 9950X3D (single batch, `--threads 8`). C++ F16 is faster than PyTorch on every image, at 1.86x smaller: @@ -174,50 +191,18 @@ allocator. See [`BENCHMARK.md`](BENCHMARK.md) for the per-image breakdown, F16 fast-path explanation, thread-scaling sweep, methodology, and reproduction recipe. -### Variants comparison - -All 5 detection variants share the DINOv2-small backbone; they differ in input resolution -and decoder layer count. C++ F16 is faster than PyTorch on each: - -| Variant | Resolution | Dec layers | C++ F16 median ms @ T=8 | PyTorch median ms | -|---------|-----------:|-----------:|------------------------:|------------------:| -| Nano | 384 | 2 | **61.5** | 88.4 | -| Small | 512 | 3 | **116.0** | 120.5 | -| Base | 560 | 3 | **136.9** | 149.5 | -| Medium | 576 | 4 | **149.6** | 182.8 | -| Large | 704 | 4 | **237.8** | 228.7* | - -\* Large is the one variant where PyTorch is competitive at T=8 (within run-to-run variance). - -![Variants overview](benchmarks/plots/variants_overview.png) - -### Quantization tradeoffs - -K-quants (Q4_K / Q5_K / Q6_K) produced via the C++ quantizer beat legacy block quants -(Q4_0 / Q5_0) at the same target bit-width. The full matrix: - -![Quant tradeoffs](benchmarks/plots/quant_tradeoffs.png) - -| Variant | Recall@0.5 | Recall@0.95 | Max \|Δscore\| | Notes | -|--------------|-----------:|------------:|---------------:|-------| -| F32 | 1.000 | 0.989 | 0.008 | Reference | -| **F16** | 1.000 | 0.989 | 0.008 | Matches F32, fastest variant | -| Q8_0 | 1.000 | 0.989 | 0.009 | 3.10x compression, no accuracy loss | -| Q6_K | 1.000 | 0.989 | 0.011 | 3.40x compression, about 10% slower than Q8_0 | -| Q5_K | 0.953 | 0.879 | 0.014 | Mild accuracy loss; still usable | -| Q4_K | 0.953 | 0.879 | 0.020 | Halves Δscore vs legacy Q4_0 at same size | -| Q4_0 (legacy) | 0.891 | 0.727 | 0.226 | Steep accuracy drop; not recommended | - -Recommendation (numbers are for rfdetr-base): +### RF-DETR 1.9 parity -1. **F16**: production default. Fastest, matches F32, 1.86x smaller than F32. -2. **Q8_0**: when disk size matters. 3.10x compression, no accuracy loss, about 7% latency - tax vs F16. -3. **Q6_K**: when you need slightly smaller than Q8_0 with near-identical accuracy. -4. **Q4_K**: last resort for ≤32 MB deployments. Real but not catastrophic accuracy loss. +PyTorch-vs-C++ parity is swept across every (variant × quant) cell of the current 44-model +matrix, using the same 7-image methodology (and the same 1.7.0 baseline) as the tables +above. F32/F16/Q8_0 hold up or improve across the board; Q4_K remains real, variant-dependent +lossy accuracy (worst case Recall@0.95 = 0.394 on Seg-Small) — consistent with the 1.7.0 +Q4_K findings on the same variant. -See [`BENCHMARK.md`](BENCHMARK.md) for mask quality across all 12 seg cells (mask IoU stays -≥ 0.99 across F32/F16/Q8_0 on every segmentation variant). +See [**BENCHMARK.md § Accuracy across the full model matrix (rfdetr 1.9.0)**](BENCHMARK.md#accuracy-across-the-full-model-matrix-rfdetr-190) +for the full per-variant, per-quant table (and its 1.7.0 predecessor, and a takeaways +section comparing the two). Raw data: +[`benchmarks/results/accuracy_sweep.json`](benchmarks/results/accuracy_sweep.json). ## Embedding via the C API diff --git a/benchmarks/results/accuracy_sweep.json b/benchmarks/results/accuracy_sweep.json index cb22ae9..ffcca23 100644 --- a/benchmarks/results/accuracy_sweep.json +++ b/benchmarks/results/accuracy_sweep.json @@ -1,20 +1,20 @@ { "meta": { - "date": "2026-05-27", + "date": "2026-07-30", "n_images": 7, "image_names": [ - "bus.jpg", - "coco_cats.jpg", - "coco_indoor.jpg", - "coco_kitchen.jpg", - "coco_living_room.jpg", - "coco_skater.jpg", - "coco_street.jpg" + "000000000139.jpg", + "000000000632.jpg", + "000000039769.jpg", + "000000087038.jpg", + "000000252219.jpg", + "000000397133.jpg", + "bus.jpg" ], "threshold": 0.5, "pytorch_version": "2.5.1+cu124", - "rfdetr_version": "1.7.0", - "cli": "/home/mudler/_git/rt-detr.cpp/build/bin/rfdetr-cli", + "rfdetr_version": "1.9.0", + "cli": "/home/adithya.balaji/papers/rf-detr.cpp/build/bin/rfdetr-cli", "models_dir": "models" }, "cells": [ @@ -27,7 +27,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 13, + "cpp_total": 13, + "matched_iou_0.5": 13, + "matched_iou_0.95": 13, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0044354077949524395, + "mean_abs_score_delta": 0.0016527144123957838, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -35,13 +49,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.012471014572143502, - "mean_abs_score_delta": 0.0030172065338134545, + "max_abs_score_delta": 0.006890871219635053, + "mean_abs_score_delta": 0.003143993914794918, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -49,13 +63,13 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.028895098049163792, - "mean_abs_score_delta": 0.011380118578910803, + "max_abs_score_delta": 0.001997932586669915, + "mean_abs_score_delta": 0.0005557544202804687, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -63,41 +77,27 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.01039347806167601, - "mean_abs_score_delta": 0.005310364958190927, + "max_abs_score_delta": 0.002327481227874739, + "mean_abs_score_delta": 0.0007996046363830667, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 7, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 7, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.08080363265609736, - "mean_abs_score_delta": 0.015238179027557372, - "extra_cpp_detections": 0 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 13, - "cpp_total": 12, - "matched_iou_0.5": 12, - "matched_iou_0.95": 12, - "recall_iou_0.5": 0.9230769230769231, - "recall_iou_0.95": 0.9230769230769231, - "max_abs_score_delta": 0.03142293905639648, - "mean_abs_score_delta": 0.007694861764589944, + "max_abs_score_delta": 0.0011942547531127445, + "mean_abs_score_delta": 0.0003295801498412798, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { "py_total": 6, "cpp_total": 6, @@ -105,13 +105,13 @@ "matched_iou_0.95": 6, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.008728734775543256, - "mean_abs_score_delta": 0.003806428378423062, + "max_abs_score_delta": 0.0029068455581664487, + "mean_abs_score_delta": 0.0014791024665832448, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -119,17 +119,17 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0021655012588500666, - "mean_abs_score_delta": 0.0012026334396362027, + "max_abs_score_delta": 0.001334529678344687, + "mean_abs_score_delta": 0.0005460485374450874, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.989010989010989, - "recall_iou_0.95": 0.989010989010989, - "max_abs_score_delta": 0.024982914061410066, - "mean_abs_score_delta": 0.006807113240160252, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.003012474688393718, + "mean_abs_score_delta": 0.0012152569339605498, "extra_cpp_detections": 0.0 } }, @@ -142,7 +142,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 13, + "cpp_total": 13, + "matched_iou_0.5": 13, + "matched_iou_0.95": 13, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0057304406204223834, + "mean_abs_score_delta": 0.0016761064975445118, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -150,13 +164,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.012482014572143596, - "mean_abs_score_delta": 0.0030484694717407557, + "max_abs_score_delta": 0.001170684291839641, + "mean_abs_score_delta": 0.0005065753753662027, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -164,13 +178,13 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.028852098049163777, - "mean_abs_score_delta": 0.01176586857891082, + "max_abs_score_delta": 0.0012249325866698912, + "mean_abs_score_delta": 0.00046475442028043323, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -178,41 +192,27 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.010854478061676054, - "mean_abs_score_delta": 0.005378964958190924, + "max_abs_score_delta": 0.0007344812278747836, + "mean_abs_score_delta": 0.00046714976882933357, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 7, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 7, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.08554563265609738, - "mean_abs_score_delta": 0.016101750456128796, - "extra_cpp_detections": 0 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 13, - "cpp_total": 12, - "matched_iou_0.5": 12, - "matched_iou_0.95": 12, - "recall_iou_0.5": 0.9230769230769231, - "recall_iou_0.95": 0.9230769230769231, - "max_abs_score_delta": 0.0315549390563965, - "mean_abs_score_delta": 0.0077785284312566205, + "max_abs_score_delta": 0.0007502547531127446, + "mean_abs_score_delta": 0.00017836511993405413, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { "py_total": 6, "cpp_total": 6, @@ -220,13 +220,13 @@ "matched_iou_0.95": 6, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.010265734775543267, - "mean_abs_score_delta": 0.004121261711756419, + "max_abs_score_delta": 0.0037038455581664964, + "mean_abs_score_delta": 0.001453835726420068, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -234,17 +234,17 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0017755012588500652, - "mean_abs_score_delta": 0.001065833439636199, + "max_abs_score_delta": 0.0008775296783447573, + "mean_abs_score_delta": 0.0004445789741516215, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.989010989010989, - "recall_iou_0.95": 0.989010989010989, - "max_abs_score_delta": 0.025904342632838664, - "mean_abs_score_delta": 0.007037239578231505, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.002027452673775814, + "mean_abs_score_delta": 0.0007416236975037466, "extra_cpp_detections": 0.0 } }, @@ -257,7 +257,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 13, + "cpp_total": 13, + "matched_iou_0.5": 13, + "matched_iou_0.95": 13, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.01604412651062015, + "mean_abs_score_delta": 0.006295832145690909, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -265,13 +279,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.014853014572143608, - "mean_abs_score_delta": 0.00583386947174076, + "max_abs_score_delta": 0.01031231570816038, + "mean_abs_score_delta": 0.005880216473388655, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -279,13 +293,13 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.031750098049163844, - "mean_abs_score_delta": 0.010005780941963222, + "max_abs_score_delta": 0.03204093258666996, + "mean_abs_score_delta": 0.010046254420280454, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -293,41 +307,27 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.01615047806167602, - "mean_abs_score_delta": 0.006364692619323731, + "max_abs_score_delta": 0.014313671005249051, + "mean_abs_score_delta": 0.00589363550491333, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 7, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 7, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.056347632656097435, - "mean_abs_score_delta": 0.01524732147598267, - "extra_cpp_detections": 0 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 13, - "cpp_total": 12, - "matched_iou_0.5": 12, - "matched_iou_0.95": 12, - "recall_iou_0.5": 0.9230769230769231, - "recall_iou_0.95": 0.9230769230769231, - "max_abs_score_delta": 0.029228939056396452, - "mean_abs_score_delta": 0.009554060692787173, + "max_abs_score_delta": 0.01457391827774046, + "mean_abs_score_delta": 0.004158032519531241, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { "py_total": 6, "cpp_total": 6, @@ -335,13 +335,13 @@ "matched_iou_0.95": 6, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.022015581661224415, - "mean_abs_score_delta": 0.008869703863143933, + "max_abs_score_delta": 0.013564461780548043, + "mean_abs_score_delta": 0.003644922709147125, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -349,17 +349,17 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.006147498741149948, - "mean_abs_score_delta": 0.0031411376770019484, + "max_abs_score_delta": 0.002000202777862503, + "mean_abs_score_delta": 0.0010571140083312613, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.989010989010989, - "recall_iou_0.95": 0.989010989010989, - "max_abs_score_delta": 0.025213320399693102, - "mean_abs_score_delta": 0.008430938105991919, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.014692804092407221, + "mean_abs_score_delta": 0.005282286825897569, "extra_cpp_detections": 0.0 } }, @@ -372,7 +372,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 13, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 8, + "recall_iou_0.5": 0.8461538461538461, + "recall_iou_0.95": 0.6153846153846154, + "max_abs_score_delta": 0.08747959220504764, + "mean_abs_score_delta": 0.025877526185122422, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -380,102 +394,88 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.06714403179550166, - "mean_abs_score_delta": 0.02630557017974853, + "max_abs_score_delta": 0.05230712878036503, + "mean_abs_score_delta": 0.018337824624633802, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, - "cpp_total": 4, + "cpp_total": 6, "matched_iou_0.5": 4, "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02515829106903078, - "mean_abs_score_delta": 0.01297081955432891, - "extra_cpp_detections": 0 + "max_abs_score_delta": 0.14334106741333008, + "mean_abs_score_delta": 0.04659151589107513, + "extra_cpp_detections": 2 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { "py_total": 5, "cpp_total": 5, "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "matched_iou_0.95": 3, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.061974027759551964, - "mean_abs_score_delta": 0.020093077540588377, + "recall_iou_0.95": 0.6, + "max_abs_score_delta": 0.056541671005248983, + "mean_abs_score_delta": 0.02431238553848265, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", - "metrics": { - "py_total": 7, - "cpp_total": 7, - "matched_iou_0.5": 6, - "matched_iou_0.95": 5, - "recall_iou_0.5": 0.8571428571428571, - "recall_iou_0.95": 0.7142857142857143, - "max_abs_score_delta": 0.18312163265609738, - "mean_abs_score_delta": 0.059571374326705916, - "extra_cpp_detections": 1 - } - }, - { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 13, - "cpp_total": 12, - "matched_iou_0.5": 12, - "matched_iou_0.95": 6, - "recall_iou_0.5": 0.9230769230769231, - "recall_iou_0.95": 0.46153846153846156, - "max_abs_score_delta": 0.10584893905639647, - "mean_abs_score_delta": 0.0344613278207779, + "py_total": 5, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 0.8, + "recall_iou_0.95": 0.8, + "max_abs_score_delta": 0.11197425475311285, + "mean_abs_score_delta": 0.03823290054607392, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { "py_total": 6, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 2, - "recall_iou_0.5": 0.8333333333333334, - "recall_iou_0.95": 0.3333333333333333, - "max_abs_score_delta": 0.035933326454162584, - "mean_abs_score_delta": 0.02269709768066406, - "extra_cpp_detections": 0 + "cpp_total": 7, + "matched_iou_0.5": 6, + "matched_iou_0.95": 6, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.1428551544418335, + "mean_abs_score_delta": 0.034432317849477125, + "extra_cpp_detections": 1 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, - "cpp_total": 4, - "matched_iou_0.5": 4, - "matched_iou_0.95": 4, - "recall_iou_0.5": 0.8, - "recall_iou_0.95": 0.8, - "max_abs_score_delta": 0.11696850125885006, - "mean_abs_score_delta": 0.03602686625003812, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.08443624527359006, + "mean_abs_score_delta": 0.029815355554962154, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9162218733647306, - "recall_iou_0.95": 0.758451072736787, - "max_abs_score_delta": 0.08516410714994156, - "mean_abs_score_delta": 0.030303733336121685, - "extra_cpp_detections": 0.14285714285714285 + "recall_iou_0.5": 0.9494505494505495, + "recall_iou_0.95": 0.8593406593406593, + "max_abs_score_delta": 0.0969907305532183, + "mean_abs_score_delta": 0.0310856894556896, + "extra_cpp_detections": 0.42857142857142855 } }, { @@ -487,7 +487,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 15, + "cpp_total": 15, + "matched_iou_0.5": 15, + "matched_iou_0.95": 15, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.006484540351867629, + "mean_abs_score_delta": 0.002195744748687741, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -495,13 +509,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.00549609629821779, - "mean_abs_score_delta": 0.0013036686752319281, + "max_abs_score_delta": 0.0007089404067993632, + "mean_abs_score_delta": 0.0002861299308776877, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 6, "cpp_total": 5, @@ -509,69 +523,55 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 0.8333333333333334, "recall_iou_0.95": 0.8333333333333334, - "max_abs_score_delta": 0.05459266099929805, - "mean_abs_score_delta": 0.014496443893432609, + "max_abs_score_delta": 0.00266983721160885, + "mean_abs_score_delta": 0.001110962353515621, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 10, + "cpp_total": 10, + "matched_iou_0.5": 10, + "matched_iou_0.95": 10, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.021006287315368688, - "mean_abs_score_delta": 0.008643792926788319, + "max_abs_score_delta": 0.004709252845764178, + "mean_abs_score_delta": 0.0010056473388671884, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 8, - "cpp_total": 9, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.04024452094650266, - "mean_abs_score_delta": 0.01605439055395126, - "extra_cpp_detections": 1 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 16, - "cpp_total": 16, - "matched_iou_0.5": 16, - "matched_iou_0.95": 15, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.9375, - "max_abs_score_delta": 0.10634075011062627, - "mean_abs_score_delta": 0.020655905461072925, + "max_abs_score_delta": 0.0018351542358397976, + "mean_abs_score_delta": 0.0005155391609191762, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 9, - "cpp_total": 9, - "matched_iou_0.5": 9, - "matched_iou_0.95": 8, + "py_total": 10, + "cpp_total": 10, + "matched_iou_0.5": 10, + "matched_iou_0.95": 10, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.8888888888888888, - "max_abs_score_delta": 0.04668941749954225, - "mean_abs_score_delta": 0.007764555780622678, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.012489219413757358, + "mean_abs_score_delta": 0.003316352651214616, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -579,18 +579,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.050619194873809836, - "mean_abs_score_delta": 0.010937849751281736, + "max_abs_score_delta": 0.00033985167694094987, + "mean_abs_score_delta": 0.00018685076065063643, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9761904761904762, - "recall_iou_0.95": 0.951388888888889, - "max_abs_score_delta": 0.046426989720480795, - "mean_abs_score_delta": 0.011408086720340207, - "extra_cpp_detections": 0.14285714285714285 + "recall_iou_0.5": 0.9761904761904763, + "recall_iou_0.95": 0.9761904761904763, + "max_abs_score_delta": 0.0041766851632254465, + "mean_abs_score_delta": 0.0012310324206760953, + "extra_cpp_detections": 0.0 } }, { @@ -602,7 +602,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 15, + "cpp_total": 15, + "matched_iou_0.5": 15, + "matched_iou_0.95": 15, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.008163540351867726, + "mean_abs_score_delta": 0.002427952301279701, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -610,13 +624,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0013720962982177731, - "mean_abs_score_delta": 0.0004144601562499739, + "max_abs_score_delta": 0.0019165055732727199, + "mean_abs_score_delta": 0.0006460484611511052, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 6, "cpp_total": 5, @@ -624,13 +638,27 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 0.8333333333333334, "recall_iou_0.95": 0.8333333333333334, - "max_abs_score_delta": 0.04871566099929814, - "mean_abs_score_delta": 0.01312844389343264, + "max_abs_score_delta": 0.005132837211608843, + "mean_abs_score_delta": 0.0014919216369628785, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 10, + "cpp_total": 10, + "matched_iou_0.5": 10, + "matched_iou_0.95": 10, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.005144252845764141, + "mean_abs_score_delta": 0.0010397087104797365, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000252219.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -638,55 +666,27 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02093828731536862, - "mean_abs_score_delta": 0.008687192926788346, + "max_abs_score_delta": 0.0016951542358398797, + "mean_abs_score_delta": 0.000577044936370874, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 8, + "py_total": 10, "cpp_total": 10, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, + "matched_iou_0.5": 10, + "matched_iou_0.95": 10, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.050075520946502694, - "mean_abs_score_delta": 0.017673265553951253, - "extra_cpp_detections": 2 + "max_abs_score_delta": 0.009702201190948445, + "mean_abs_score_delta": 0.002705752651214588, + "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 16, - "cpp_total": 16, - "matched_iou_0.5": 16, - "matched_iou_0.95": 15, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.9375, - "max_abs_score_delta": 0.10961075011062626, - "mean_abs_score_delta": 0.020833405461072922, - "extra_cpp_detections": 0 - } - }, - { - "image": "coco_skater.jpg", - "metrics": { - "py_total": 9, - "cpp_total": 9, - "matched_iou_0.5": 9, - "matched_iou_0.95": 8, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.8888888888888888, - "max_abs_score_delta": 0.04697741749954221, - "mean_abs_score_delta": 0.007837778002844908, - "extra_cpp_detections": 0 - } - }, - { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -694,18 +694,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.049984194873809784, - "mean_abs_score_delta": 0.010541649751281735, + "max_abs_score_delta": 0.0007751483230590273, + "mean_abs_score_delta": 0.00022501851882934522, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9761904761904762, - "recall_iou_0.95": 0.951388888888889, - "max_abs_score_delta": 0.04681056114905221, - "mean_abs_score_delta": 0.011302313677945968, - "extra_cpp_detections": 0.2857142857142857 + "recall_iou_0.5": 0.9761904761904763, + "recall_iou_0.95": 0.9761904761904763, + "max_abs_score_delta": 0.004647091390337255, + "mean_abs_score_delta": 0.0013019210308983181, + "extra_cpp_detections": 0.0 } }, { @@ -717,7 +717,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 15, + "cpp_total": 15, + "matched_iou_0.5": 15, + "matched_iou_0.95": 15, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.01726045133972165, + "mean_abs_score_delta": 0.005141078280893955, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -725,13 +739,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.00236472161865231, - "mean_abs_score_delta": 0.001476449803161617, + "max_abs_score_delta": 0.013640580257415769, + "mean_abs_score_delta": 0.005616694089508068, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 6, "cpp_total": 5, @@ -739,69 +753,55 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 0.8333333333333334, "recall_iou_0.95": 0.8333333333333334, - "max_abs_score_delta": 0.04953566099929807, - "mean_abs_score_delta": 0.013380043893432614, + "max_abs_score_delta": 0.01875731294631955, + "mean_abs_score_delta": 0.007053903141021722, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.026188280273437448, - "mean_abs_score_delta": 0.01221949515914913, + "py_total": 10, + "cpp_total": 9, + "matched_iou_0.5": 9, + "matched_iou_0.95": 9, + "recall_iou_0.5": 0.9, + "recall_iou_0.95": 0.9, + "max_abs_score_delta": 0.007776457164764383, + "mean_abs_score_delta": 0.0030722162984212365, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 8, - "cpp_total": 10, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.027895520946502717, - "mean_abs_score_delta": 0.012865547874927524, - "extra_cpp_detections": 2 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 16, - "cpp_total": 16, - "matched_iou_0.5": 16, - "matched_iou_0.95": 14, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.875, - "max_abs_score_delta": 0.11218675011062618, - "mean_abs_score_delta": 0.022020346264123906, + "max_abs_score_delta": 0.007208251541137689, + "mean_abs_score_delta": 0.002467226340484596, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 9, + "py_total": 10, "cpp_total": 9, "matched_iou_0.5": 9, - "matched_iou_0.95": 8, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.8888888888888888, - "max_abs_score_delta": 0.03516741749954222, - "mean_abs_score_delta": 0.007333688365088569, + "matched_iou_0.95": 9, + "recall_iou_0.5": 0.9, + "recall_iou_0.95": 0.9, + "max_abs_score_delta": 0.01467887682342528, + "mean_abs_score_delta": 0.004661070718553311, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -809,18 +809,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.04210219487380984, - "mean_abs_score_delta": 0.01086108203735352, + "max_abs_score_delta": 0.0034848516769409033, + "mean_abs_score_delta": 0.0010246961784362795, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9761904761904762, - "recall_iou_0.95": 0.9424603174603174, - "max_abs_score_delta": 0.04220579233169554, - "mean_abs_score_delta": 0.011450950485319553, - "extra_cpp_detections": 0.2857142857142857 + "recall_iou_0.5": 0.9476190476190477, + "recall_iou_0.95": 0.9476190476190477, + "max_abs_score_delta": 0.011829540249960746, + "mean_abs_score_delta": 0.0041481264353313094, + "extra_cpp_detections": 0.0 } }, { @@ -832,7 +832,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 15, + "cpp_total": 14, + "matched_iou_0.5": 14, + "matched_iou_0.95": 10, + "recall_iou_0.5": 0.9333333333333333, + "recall_iou_0.95": 0.6666666666666666, + "max_abs_score_delta": 0.09568797395706174, + "mean_abs_score_delta": 0.027041805639266954, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -840,13 +854,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.015863903701782256, - "mean_abs_score_delta": 0.0096755311553955, + "max_abs_score_delta": 0.047816419742584215, + "mean_abs_score_delta": 0.02198941412887574, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 6, "cpp_total": 5, @@ -854,69 +868,55 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 0.8333333333333334, "recall_iou_0.95": 0.8333333333333334, - "max_abs_score_delta": 0.054596461524963336, - "mean_abs_score_delta": 0.023950116524505627, + "max_abs_score_delta": 0.03049539820861813, + "mean_abs_score_delta": 0.01624695128936766, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.04929471972656252, - "mean_abs_score_delta": 0.02688108081741336, + "py_total": 10, + "cpp_total": 8, + "matched_iou_0.5": 8, + "matched_iou_0.95": 6, + "recall_iou_0.5": 0.8, + "recall_iou_0.95": 0.6, + "max_abs_score_delta": 0.027063303974151576, + "mean_abs_score_delta": 0.011219489403724664, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 8, - "cpp_total": 9, - "matched_iou_0.5": 8, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.625, - "max_abs_score_delta": 0.08790811284255984, - "mean_abs_score_delta": 0.0316004550871849, - "extra_cpp_detections": 1 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 16, - "cpp_total": 15, - "matched_iou_0.5": 15, - "matched_iou_0.95": 13, - "recall_iou_0.5": 0.9375, - "recall_iou_0.95": 0.8125, - "max_abs_score_delta": 0.153024046333313, - "mean_abs_score_delta": 0.041964122097524, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.09505284576416018, + "mean_abs_score_delta": 0.03661205887069703, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 9, + "py_total": 10, "cpp_total": 9, "matched_iou_0.5": 9, - "matched_iou_0.95": 6, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.6666666666666666, - "max_abs_score_delta": 0.062451417499542194, - "mean_abs_score_delta": 0.014546462354448095, + "matched_iou_0.95": 7, + "recall_iou_0.5": 0.9, + "recall_iou_0.95": 0.7, + "max_abs_score_delta": 0.044925127494812034, + "mean_abs_score_delta": 0.02332432215033638, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -924,18 +924,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.1253841948738098, - "mean_abs_score_delta": 0.03812363973541259, + "max_abs_score_delta": 0.017305148323059072, + "mean_abs_score_delta": 0.007913310089874259, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9672619047619048, - "recall_iou_0.95": 0.8482142857142857, - "max_abs_score_delta": 0.07836040807179043, - "mean_abs_score_delta": 0.02667734396741201, - "extra_cpp_detections": 0.14285714285714285 + "recall_iou_0.5": 0.9238095238095239, + "recall_iou_0.95": 0.8285714285714285, + "max_abs_score_delta": 0.05119231678063528, + "mean_abs_score_delta": 0.020621050224591812, + "extra_cpp_detections": 0.0 } }, { @@ -947,49 +947,49 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 13, + "cpp_total": 13, + "matched_iou_0.5": 13, + "matched_iou_0.95": 13, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0018509091567993075, - "mean_abs_score_delta": 0.000984649051666242, + "max_abs_score_delta": 0.010249875846862766, + "mean_abs_score_delta": 0.001712523280217093, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.012031279624938995, - "mean_abs_score_delta": 0.005628751033020007, + "max_abs_score_delta": 0.004691022075653084, + "mean_abs_score_delta": 0.002219054912567131, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { - "py_total": 4, - "cpp_total": 4, - "matched_iou_0.5": 4, - "matched_iou_0.95": 4, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.04446205025482175, - "mean_abs_score_delta": 0.016653220909118638, + "max_abs_score_delta": 0.0020615785484313642, + "mean_abs_score_delta": 0.0011742813781738359, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { "py_total": 12, "cpp_total": 12, @@ -997,41 +997,41 @@ "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.029592064807891894, - "mean_abs_score_delta": 0.012627289350827539, + "max_abs_score_delta": 0.0015142180786132942, + "mean_abs_score_delta": 0.0006206848271687789, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 13, - "matched_iou_0.95": 12, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.9230769230769231, - "max_abs_score_delta": 0.013420399829864538, - "mean_abs_score_delta": 0.005872975814232466, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0018276469345093194, + "mean_abs_score_delta": 0.000648466620635979, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 11, + "py_total": 12, "cpp_total": 12, - "matched_iou_0.5": 11, - "matched_iou_0.95": 11, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.017089517559051526, - "mean_abs_score_delta": 0.004728752810738298, - "extra_cpp_detections": 1 + "max_abs_score_delta": 0.012316906173706066, + "mean_abs_score_delta": 0.0015485857531229534, + "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1039,18 +1039,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02385251290893553, - "mean_abs_score_delta": 0.00944366166229249, + "max_abs_score_delta": 0.0009251092529296567, + "mean_abs_score_delta": 0.0004596466361999507, "extra_cpp_detections": 0 } } ], "metrics": { "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.989010989010989, - "max_abs_score_delta": 0.020328390591757648, - "mean_abs_score_delta": 0.007991328661699383, - "extra_cpp_detections": 0.14285714285714285 + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.00479805098724365, + "mean_abs_score_delta": 0.0011976062011551031, + "extra_cpp_detections": 0.0 } }, { @@ -1062,49 +1062,49 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 13, + "cpp_total": 13, + "matched_iou_0.5": 13, + "matched_iou_0.95": 13, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0021235624046325174, - "mean_abs_score_delta": 0.0013076490516662708, + "max_abs_score_delta": 0.009205875846862832, + "mean_abs_score_delta": 0.0015565563342754616, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.016430347099304243, - "mean_abs_score_delta": 0.0068713510330200124, + "max_abs_score_delta": 0.003643022075653035, + "mean_abs_score_delta": 0.0021123210735320974, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { - "py_total": 4, - "cpp_total": 4, - "matched_iou_0.5": 4, - "matched_iou_0.95": 4, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.04397105025482173, - "mean_abs_score_delta": 0.016853720909118658, + "max_abs_score_delta": 0.00904157854843135, + "mean_abs_score_delta": 0.0033817334075927573, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { "py_total": 12, "cpp_total": 12, @@ -1112,41 +1112,41 @@ "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.03016306480789188, - "mean_abs_score_delta": 0.012442728256543476, + "max_abs_score_delta": 0.0013892180786132524, + "mean_abs_score_delta": 0.0005301426213582353, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 13, - "matched_iou_0.95": 12, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.9230769230769231, - "max_abs_score_delta": 0.012456399829864462, - "mean_abs_score_delta": 0.005964591198847855, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0020826469345093246, + "mean_abs_score_delta": 0.000556073206329355, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 11, + "py_total": 12, "cpp_total": 12, - "matched_iou_0.5": 11, - "matched_iou_0.95": 11, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.017280517559051467, - "mean_abs_score_delta": 0.004678843719829225, - "extra_cpp_detections": 1 + "max_abs_score_delta": 0.00575137916946411, + "mean_abs_score_delta": 0.0012258219865163105, + "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1154,18 +1154,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.023963512908935503, - "mean_abs_score_delta": 0.00957966166229245, + "max_abs_score_delta": 0.000524109252929672, + "mean_abs_score_delta": 0.0001782466361999413, "extra_cpp_detections": 0 } } ], "metrics": { "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.989010989010989, - "max_abs_score_delta": 0.020912636409214543, - "mean_abs_score_delta": 0.008242649404473992, - "extra_cpp_detections": 0.14285714285714285 + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.004519689986637654, + "mean_abs_score_delta": 0.0013629850379720226, + "extra_cpp_detections": 0.0 } }, { @@ -1177,35 +1177,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", - "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0021195624046325134, - "mean_abs_score_delta": 0.0008360853889465281, - "extra_cpp_detections": 0 - } - }, - { - "image": "coco_cats.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 13, + "cpp_total": 13, + "matched_iou_0.5": 13, + "matched_iou_0.95": 13, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.04162534709930421, - "mean_abs_score_delta": 0.011529754669189462, + "max_abs_score_delta": 0.01757605368423465, + "mean_abs_score_delta": 0.005456125590397773, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -1213,55 +1199,41 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.04605505025482182, - "mean_abs_score_delta": 0.019038062345504786, + "max_abs_score_delta": 0.0020124096374511957, + "mean_abs_score_delta": 0.0012778210735321094, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000039769.jpg", "metrics": { - "py_total": 12, - "cpp_total": 12, - "matched_iou_0.5": 12, - "matched_iou_0.95": 12, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02442336987304683, - "mean_abs_score_delta": 0.009431056823412581, - "extra_cpp_detections": 0 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 13, - "matched_iou_0.95": 12, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.9230769230769231, - "max_abs_score_delta": 0.013891190597534187, - "mean_abs_score_delta": 0.007782070600069481, + "max_abs_score_delta": 0.002489231578826856, + "mean_abs_score_delta": 0.0010738653503417962, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 11, + "py_total": 12, "cpp_total": 11, "matched_iou_0.5": 11, "matched_iou_0.95": 11, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.011428517559051499, - "mean_abs_score_delta": 0.005698121805017639, + "recall_iou_0.5": 0.9166666666666666, + "recall_iou_0.95": 0.9166666666666666, + "max_abs_score_delta": 0.009643253234863258, + "mean_abs_score_delta": 0.0032076488494873112, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "000000252219.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1269,44 +1241,27 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0261795129089355, - "mean_abs_score_delta": 0.009364661662292462, + "max_abs_score_delta": 0.016994364459991473, + "mean_abs_score_delta": 0.005003353630828866, "extra_cpp_detections": 0 } - } - ], - "metrics": { - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.989010989010989, - "max_abs_score_delta": 0.02367465009961808, - "mean_abs_score_delta": 0.00909711618491899, - "extra_cpp_detections": 0.0 - } - }, - { - "variant": "base", - "quant": "q4_K", - "model_path": "models/rfdetr-base-q4_K.gguf", - "is_seg": false, - "file_size_mb": 31.48, - "n_images": 7, - "per_image": [ + }, { - "image": "bus.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.01290298349380492, - "mean_abs_score_delta": 0.007890013953399633, + "max_abs_score_delta": 0.016486308818817186, + "mean_abs_score_delta": 0.005103631645202648, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1314,13 +1269,44 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.11011465290069578, - "mean_abs_score_delta": 0.055011812193298336, + "max_abs_score_delta": 0.0022573095855712433, + "mean_abs_score_delta": 0.000957149549102776, + "extra_cpp_detections": 0 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.988095238095238, + "recall_iou_0.95": 0.988095238095238, + "max_abs_score_delta": 0.009636990142822266, + "mean_abs_score_delta": 0.0031542279555561827, + "extra_cpp_detections": 0.0 + } + }, + { + "variant": "base", + "quant": "q4_K", + "model_path": "models/rfdetr-base-q4_K.gguf", + "is_seg": false, + "file_size_mb": 31.48, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 13, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 10, + "recall_iou_0.5": 0.8461538461538461, + "recall_iou_0.95": 0.7692307692307693, + "max_abs_score_delta": 0.027338417804718018, + "mean_abs_score_delta": 0.011665278937946666, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 5, @@ -1328,55 +1314,69 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0341900502548218, - "mean_abs_score_delta": 0.01892076360702516, + "max_abs_score_delta": 0.014800032321929946, + "mean_abs_score_delta": 0.010787660943984978, "extra_cpp_detections": 1 } }, { - "image": "coco_kitchen.jpg", + "image": "000000039769.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.1667632315788269, + "mean_abs_score_delta": 0.0636434499588013, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000087038.jpg", "metrics": { "py_total": 12, - "cpp_total": 11, + "cpp_total": 12, "matched_iou_0.5": 11, - "matched_iou_0.95": 9, + "matched_iou_0.95": 7, "recall_iou_0.5": 0.9166666666666666, - "recall_iou_0.95": 0.75, - "max_abs_score_delta": 0.04303570579528804, - "mean_abs_score_delta": 0.02055812661569769, - "extra_cpp_detections": 0 + "recall_iou_0.95": 0.5833333333333334, + "max_abs_score_delta": 0.03819507565307623, + "mean_abs_score_delta": 0.013719122580788355, + "extra_cpp_detections": 1 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 13, - "cpp_total": 11, - "matched_iou_0.5": 11, - "matched_iou_0.95": 10, - "recall_iou_0.5": 0.8461538461538461, - "recall_iou_0.95": 0.7692307692307693, - "max_abs_score_delta": 0.03821350744628904, - "mean_abs_score_delta": 0.012094236523714936, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.033969635540008536, + "mean_abs_score_delta": 0.010810461246490478, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 11, - "cpp_total": 10, - "matched_iou_0.5": 10, - "matched_iou_0.95": 7, - "recall_iou_0.5": 0.9090909090909091, - "recall_iou_0.95": 0.6363636363636364, - "max_abs_score_delta": 0.04139744335556028, - "mean_abs_score_delta": 0.012290914970016465, + "py_total": 12, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 10, + "recall_iou_0.5": 0.9166666666666666, + "recall_iou_0.95": 0.8333333333333334, + "max_abs_score_delta": 0.05797450207138066, + "mean_abs_score_delta": 0.016567183298284358, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1384,18 +1384,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02665187853240969, - "mean_abs_score_delta": 0.010227187882995592, + "max_abs_score_delta": 0.014317552303314218, + "mean_abs_score_delta": 0.010135726769256581, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9531302031302031, - "recall_iou_0.95": 0.8793706293706294, - "max_abs_score_delta": 0.04378660311126708, - "mean_abs_score_delta": 0.019570436535163972, - "extra_cpp_detections": 0.14285714285714285 + "recall_iou_0.5": 0.9542124542124543, + "recall_iou_0.95": 0.8836996336996336, + "max_abs_score_delta": 0.050479778181893496, + "mean_abs_score_delta": 0.019618411962221815, + "extra_cpp_detections": 0.2857142857142857 } }, { @@ -1407,35 +1407,35 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 16, + "cpp_total": 16, + "matched_iou_0.5": 16, + "matched_iou_0.95": 16, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0032308367919922087, - "mean_abs_score_delta": 0.0010993459510803215, + "max_abs_score_delta": 0.01617373272323608, + "mean_abs_score_delta": 0.0035993809278011338, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { - "py_total": 6, + "py_total": 5, "cpp_total": 5, "matched_iou_0.5": 5, "matched_iou_0.95": 5, - "recall_iou_0.5": 0.8333333333333334, - "recall_iou_0.95": 0.8333333333333334, - "max_abs_score_delta": 0.010762761062622084, - "mean_abs_score_delta": 0.00433371620788574, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0047095858917236555, + "mean_abs_score_delta": 0.0018085999275207598, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1443,55 +1443,55 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0036850442047119403, - "mean_abs_score_delta": 0.001841355948638923, + "max_abs_score_delta": 0.01750870835113527, + "mean_abs_score_delta": 0.004391253334808343, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, - "recall_iou_0.5": 0.8888888888888888, - "recall_iou_0.95": 0.8888888888888888, - "max_abs_score_delta": 0.035354015850067144, - "mean_abs_score_delta": 0.012764424059867843, + "py_total": 13, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 11, + "recall_iou_0.5": 0.9230769230769231, + "recall_iou_0.95": 0.8461538461538461, + "max_abs_score_delta": 0.020402043758392385, + "mean_abs_score_delta": 0.004257380994478865, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 15, - "cpp_total": 14, - "matched_iou_0.5": 14, - "matched_iou_0.95": 13, - "recall_iou_0.5": 0.9333333333333333, - "recall_iou_0.95": 0.8666666666666667, - "max_abs_score_delta": 0.042026119010925345, - "mean_abs_score_delta": 0.011081528905596058, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0017859747734070286, + "mean_abs_score_delta": 0.0009521167381286721, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 12, - "cpp_total": 11, - "matched_iou_0.5": 11, + "py_total": 9, + "cpp_total": 8, + "matched_iou_0.5": 8, "matched_iou_0.95": 8, - "recall_iou_0.5": 0.9166666666666666, - "recall_iou_0.95": 0.6666666666666666, - "max_abs_score_delta": 0.02233836240386966, - "mean_abs_score_delta": 0.007692571373332619, + "recall_iou_0.5": 0.8888888888888888, + "recall_iou_0.95": 0.8888888888888888, + "max_abs_score_delta": 0.0033005460319519564, + "mean_abs_score_delta": 0.0014376746935844398, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1499,17 +1499,17 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.027220300224304217, - "mean_abs_score_delta": 0.007018119485473639, + "max_abs_score_delta": 0.0007367090454101222, + "mean_abs_score_delta": 0.00026741551361082473, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9388888888888889, - "recall_iou_0.95": 0.8936507936507937, - "max_abs_score_delta": 0.020659634221213228, - "mean_abs_score_delta": 0.006547294561696449, + "recall_iou_0.5": 0.9731379731379732, + "recall_iou_0.95": 0.9621489621489623, + "max_abs_score_delta": 0.009231042939322356, + "mean_abs_score_delta": 0.0023876888757047195, "extra_cpp_detections": 0.0 } }, @@ -1522,35 +1522,35 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 16, + "cpp_total": 16, + "matched_iou_0.5": 16, + "matched_iou_0.95": 16, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0033358367919922305, - "mean_abs_score_delta": 0.0012484061546325976, + "max_abs_score_delta": 0.014465983310699482, + "mean_abs_score_delta": 0.0036886537754535695, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { - "py_total": 6, + "py_total": 5, "cpp_total": 5, "matched_iou_0.5": 5, "matched_iou_0.95": 5, - "recall_iou_0.5": 0.8333333333333334, - "recall_iou_0.95": 0.8333333333333334, - "max_abs_score_delta": 0.011181761062622031, - "mean_abs_score_delta": 0.004336116207885721, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0053869614562988, + "mean_abs_score_delta": 0.002166999927520763, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1558,55 +1558,55 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0018761271324158058, - "mean_abs_score_delta": 0.001555087720489534, + "max_abs_score_delta": 0.005749291648864729, + "mean_abs_score_delta": 0.0021697010658264125, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, - "recall_iou_0.5": 0.8888888888888888, - "recall_iou_0.95": 0.8888888888888888, - "max_abs_score_delta": 0.035149015850067133, - "mean_abs_score_delta": 0.012496549059867842, + "py_total": 13, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 10, + "recall_iou_0.5": 0.8461538461538461, + "recall_iou_0.95": 0.7692307692307693, + "max_abs_score_delta": 0.02530904375839238, + "mean_abs_score_delta": 0.004216435797604657, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 15, - "cpp_total": 14, - "matched_iou_0.5": 14, - "matched_iou_0.95": 14, - "recall_iou_0.5": 0.9333333333333333, - "recall_iou_0.95": 0.9333333333333333, - "max_abs_score_delta": 0.04263811901092529, - "mean_abs_score_delta": 0.012955100334167484, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.003834025226593041, + "mean_abs_score_delta": 0.0010887268287658625, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 12, - "cpp_total": 11, - "matched_iou_0.5": 11, - "matched_iou_0.95": 8, - "recall_iou_0.5": 0.9166666666666666, - "recall_iou_0.95": 0.6666666666666666, - "max_abs_score_delta": 0.021895362403869578, - "mean_abs_score_delta": 0.007874882709503166, + "py_total": 9, + "cpp_total": 9, + "matched_iou_0.5": 9, + "matched_iou_0.95": 9, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0045800139389038375, + "mean_abs_score_delta": 0.0017794305203755714, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1614,17 +1614,17 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.025179300224304146, - "mean_abs_score_delta": 0.0063976920593261655, + "max_abs_score_delta": 0.0005074978370666061, + "mean_abs_score_delta": 0.0002734384506225318, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9388888888888889, - "recall_iou_0.95": 0.9031746031746032, - "max_abs_score_delta": 0.020179360353742317, - "mean_abs_score_delta": 0.006694833463696073, + "recall_iou_0.5": 0.9780219780219781, + "recall_iou_0.95": 0.967032967032967, + "max_abs_score_delta": 0.008547545310974125, + "mean_abs_score_delta": 0.002197626623738481, "extra_cpp_detections": 0.0 } }, @@ -1637,35 +1637,35 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 16, + "cpp_total": 16, + "matched_iou_0.5": 16, + "matched_iou_0.95": 15, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.002388362228393559, - "mean_abs_score_delta": 0.0012690350944519091, + "recall_iou_0.95": 0.9375, + "max_abs_score_delta": 0.03934002222442623, + "mean_abs_score_delta": 0.010148780900239941, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { - "py_total": 6, + "py_total": 5, "cpp_total": 5, "matched_iou_0.5": 5, "matched_iou_0.95": 5, - "recall_iou_0.5": 0.8333333333333334, - "recall_iou_0.95": 0.8333333333333334, - "max_abs_score_delta": 0.019314632678985588, - "mean_abs_score_delta": 0.006173573462677018, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.022349924285888667, + "mean_abs_score_delta": 0.007416769641876209, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1673,55 +1673,55 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.007555922695159922, - "mean_abs_score_delta": 0.004182380814361575, + "max_abs_score_delta": 0.011886708351135256, + "mean_abs_score_delta": 0.004414357683563241, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, - "recall_iou_0.5": 0.8888888888888888, - "recall_iou_0.95": 0.8888888888888888, - "max_abs_score_delta": 0.048115015850067167, - "mean_abs_score_delta": 0.017133541460990914, + "py_total": 13, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 11, + "recall_iou_0.5": 0.9230769230769231, + "recall_iou_0.95": 0.8461538461538461, + "max_abs_score_delta": 0.03152504375839238, + "mean_abs_score_delta": 0.009474044396400464, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 15, - "cpp_total": 14, - "matched_iou_0.5": 14, - "matched_iou_0.95": 14, - "recall_iou_0.5": 0.9333333333333333, - "recall_iou_0.95": 0.9333333333333333, - "max_abs_score_delta": 0.05273211901092534, - "mean_abs_score_delta": 0.0120506529333932, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.011713974773406965, + "mean_abs_score_delta": 0.003731025518035902, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 12, - "cpp_total": 11, - "matched_iou_0.5": 11, - "matched_iou_0.95": 8, - "recall_iou_0.5": 0.9166666666666666, - "recall_iou_0.95": 0.6666666666666666, - "max_abs_score_delta": 0.024404362403869673, - "mean_abs_score_delta": 0.010473413876273409, - "extra_cpp_detections": 0 + "py_total": 9, + "cpp_total": 10, + "matched_iou_0.5": 9, + "matched_iou_0.95": 9, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.053334453968048146, + "mean_abs_score_delta": 0.01236121493657432, + "extra_cpp_detections": 1 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1729,18 +1729,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02766830022430422, - "mean_abs_score_delta": 0.006951040699768063, + "max_abs_score_delta": 0.0035694978370666153, + "mean_abs_score_delta": 0.0015777318954467655, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9388888888888889, - "recall_iou_0.95": 0.9031746031746032, - "max_abs_score_delta": 0.026025530727386496, - "mean_abs_score_delta": 0.008319091191702299, - "extra_cpp_detections": 0.0 + "recall_iou_0.5": 0.989010989010989, + "recall_iou_0.95": 0.9690934065934067, + "max_abs_score_delta": 0.02481708931405204, + "mean_abs_score_delta": 0.007017703567448121, + "extra_cpp_detections": 0.14285714285714285 } }, { @@ -1752,91 +1752,91 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 16, + "cpp_total": 14, + "matched_iou_0.5": 14, + "matched_iou_0.95": 11, + "recall_iou_0.5": 0.875, + "recall_iou_0.95": 0.6875, + "max_abs_score_delta": 0.06886162699127196, + "mean_abs_score_delta": 0.02013739826420376, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "matched_iou_0.95": 3, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.017851546913146943, - "mean_abs_score_delta": 0.008011224919891369, - "extra_cpp_detections": 0 - } - }, - { - "image": "coco_cats.jpg", - "metrics": { - "py_total": 6, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, - "recall_iou_0.5": 0.8333333333333334, - "recall_iou_0.95": 0.8333333333333334, - "max_abs_score_delta": 0.027720277736663856, - "mean_abs_score_delta": 0.018534109729766834, + "recall_iou_0.95": 0.6, + "max_abs_score_delta": 0.04807349385452275, + "mean_abs_score_delta": 0.0285309655708313, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 5, "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 3, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.6, - "max_abs_score_delta": 0.05649392269515996, - "mean_abs_score_delta": 0.03301495819778444, - "extra_cpp_detections": 0 + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 0.8, + "recall_iou_0.95": 0.8, + "max_abs_score_delta": 0.02164143319320677, + "mean_abs_score_delta": 0.011665294205665616, + "extra_cpp_detections": 1 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 6, - "recall_iou_0.5": 0.7777777777777778, - "recall_iou_0.95": 0.6666666666666666, - "max_abs_score_delta": 0.0634093346862793, - "mean_abs_score_delta": 0.029512539600372305, + "py_total": 13, + "cpp_total": 8, + "matched_iou_0.5": 8, + "matched_iou_0.95": 4, + "recall_iou_0.5": 0.6153846153846154, + "recall_iou_0.95": 0.3076923076923077, + "max_abs_score_delta": 0.05235976803207398, + "mean_abs_score_delta": 0.02677172466564179, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 15, - "cpp_total": 14, - "matched_iou_0.5": 14, - "matched_iou_0.95": 12, - "recall_iou_0.5": 0.9333333333333333, - "recall_iou_0.95": 0.8, - "max_abs_score_delta": 0.07411584170913699, - "mean_abs_score_delta": 0.021986271685464038, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.025384115833282483, + "mean_abs_score_delta": 0.010378716738128669, "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 12, - "cpp_total": 9, - "matched_iou_0.5": 9, - "matched_iou_0.95": 6, - "recall_iou_0.5": 0.75, - "recall_iou_0.95": 0.5, - "max_abs_score_delta": 0.05482226879882812, - "mean_abs_score_delta": 0.017488964574178072, + "py_total": 9, + "cpp_total": 8, + "matched_iou_0.5": 8, + "matched_iou_0.95": 7, + "recall_iou_0.5": 0.8888888888888888, + "recall_iou_0.95": 0.7777777777777778, + "max_abs_score_delta": 0.1319665460319519, + "mean_abs_score_delta": 0.03452667100572587, "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1844,18 +1844,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.022768300224304205, - "mean_abs_score_delta": 0.010857919485473632, + "max_abs_score_delta": 0.017784290954589888, + "mean_abs_score_delta": 0.007470209555053708, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.8992063492063492, - "recall_iou_0.95": 0.7714285714285715, - "max_abs_score_delta": 0.04531164182335991, - "mean_abs_score_delta": 0.01991514117041867, - "extra_cpp_detections": 0.0 + "recall_iou_0.5": 0.8827533577533578, + "recall_iou_0.95": 0.7389957264957265, + "max_abs_score_delta": 0.052295896412985675, + "mean_abs_score_delta": 0.019925854286464383, + "extra_cpp_detections": 0.14285714285714285 } }, { @@ -1867,21 +1867,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 16, + "cpp_total": 16, + "matched_iou_0.5": 16, + "matched_iou_0.95": 16, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.001808397567748976, - "mean_abs_score_delta": 0.0009102990074157713, + "max_abs_score_delta": 0.005795389942169216, + "mean_abs_score_delta": 0.0025717017226219163, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1889,13 +1889,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0575014262619018, - "mean_abs_score_delta": 0.013426060060119616, + "max_abs_score_delta": 0.002783300041198755, + "mean_abs_score_delta": 0.0012984022674560557, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1903,55 +1903,55 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.009106041400909426, - "mean_abs_score_delta": 0.0035939097671508604, + "max_abs_score_delta": 0.012306474296569836, + "mean_abs_score_delta": 0.0032544342842101724, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, - "recall_iou_0.5": 0.8888888888888888, - "recall_iou_0.95": 0.8888888888888888, - "max_abs_score_delta": 0.017245088722228963, - "mean_abs_score_delta": 0.007747117381095875, + "py_total": 13, + "cpp_total": 13, + "matched_iou_0.5": 13, + "matched_iou_0.95": 13, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0027574051361084484, + "mean_abs_score_delta": 0.0009756682648292078, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 16, - "cpp_total": 17, - "matched_iou_0.5": 16, - "matched_iou_0.95": 16, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.03165978887176513, - "mean_abs_score_delta": 0.013471271515130993, - "extra_cpp_detections": 1 + "max_abs_score_delta": 0.0039030704040526887, + "mean_abs_score_delta": 0.0008957682579040505, + "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 12, - "matched_iou_0.95": 11, - "recall_iou_0.5": 0.9230769230769231, - "recall_iou_0.95": 0.8461538461538461, - "max_abs_score_delta": 0.028660835353851333, - "mean_abs_score_delta": 0.007162668979644783, - "extra_cpp_detections": 1 + "py_total": 9, + "cpp_total": 9, + "matched_iou_0.5": 9, + "matched_iou_0.95": 9, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.01016957422637943, + "mean_abs_score_delta": 0.0026411640298631533, + "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -1959,18 +1959,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.01117958532333374, - "mean_abs_score_delta": 0.003474467535400372, + "max_abs_score_delta": 0.0013983576660155927, + "mean_abs_score_delta": 0.000753035351562481, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9731379731379731, - "recall_iou_0.95": 0.9621489621489622, - "max_abs_score_delta": 0.022451594785962765, - "mean_abs_score_delta": 0.0071122563208511815, - "extra_cpp_detections": 0.2857142857142857 + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.005587653101784852, + "mean_abs_score_delta": 0.001770024882635291, + "extra_cpp_detections": 0.0 } }, { @@ -1982,21 +1982,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 16, + "cpp_total": 16, + "matched_iou_0.5": 16, + "matched_iou_0.95": 16, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0017983975677490216, - "mean_abs_score_delta": 0.0006368990074157921, + "max_abs_score_delta": 0.008055488822937051, + "mean_abs_score_delta": 0.0024577027664184625, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -2004,13 +2004,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.05676542626190184, - "mean_abs_score_delta": 0.013070460060119627, + "max_abs_score_delta": 0.002935300041198685, + "mean_abs_score_delta": 0.0014190022674560598, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -2018,55 +2018,55 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.010430041400909418, - "mean_abs_score_delta": 0.0035883097671508767, + "max_abs_score_delta": 0.0037375257034302223, + "mean_abs_score_delta": 0.0025496445655822653, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, - "recall_iou_0.5": 0.8888888888888888, - "recall_iou_0.95": 0.8888888888888888, - "max_abs_score_delta": 0.016998088722228966, - "mean_abs_score_delta": 0.007684992381095906, + "py_total": 13, + "cpp_total": 13, + "matched_iou_0.5": 13, + "matched_iou_0.95": 13, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.003150405136108425, + "mean_abs_score_delta": 0.000933541395040649, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 16, - "cpp_total": 17, - "matched_iou_0.5": 16, - "matched_iou_0.95": 16, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.030904041946411165, - "mean_abs_score_delta": 0.013103524981260307, - "extra_cpp_detections": 1 + "max_abs_score_delta": 0.002946070404052703, + "mean_abs_score_delta": 0.0008259220893859886, + "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 12, - "matched_iou_0.95": 12, - "recall_iou_0.5": 0.9230769230769231, - "recall_iou_0.95": 0.9230769230769231, - "max_abs_score_delta": 0.02630283535385136, - "mean_abs_score_delta": 0.007038301176071164, - "extra_cpp_detections": 1 + "py_total": 9, + "cpp_total": 9, + "matched_iou_0.5": 9, + "matched_iou_0.95": 9, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.005623379051208466, + "mean_abs_score_delta": 0.0017423131141662458, + "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -2074,18 +2074,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.012732585323333767, - "mean_abs_score_delta": 0.0037378675354004186, + "max_abs_score_delta": 0.0033736183853149138, + "mean_abs_score_delta": 0.0012176869079589814, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9731379731379731, - "recall_iou_0.95": 0.9731379731379731, - "max_abs_score_delta": 0.02227591665376936, - "mean_abs_score_delta": 0.006980050701216299, - "extra_cpp_detections": 0.2857142857142857 + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0042602553634643525, + "mean_abs_score_delta": 0.0015922590151440932, + "extra_cpp_detections": 0.0 } }, { @@ -2097,21 +2097,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 16, + "cpp_total": 16, + "matched_iou_0.5": 16, + "matched_iou_0.95": 16, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0038173975677490146, - "mean_abs_score_delta": 0.0016578629737854067, + "max_abs_score_delta": 0.015124610057830834, + "mean_abs_score_delta": 0.005208041627407081, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -2119,13 +2119,13 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.09219342626190186, - "mean_abs_score_delta": 0.02168212270050047, + "max_abs_score_delta": 0.00535043818664549, + "mean_abs_score_delta": 0.0025777263473510726, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -2133,55 +2133,55 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.013326041400909427, - "mean_abs_score_delta": 0.005743789750671402, + "max_abs_score_delta": 0.012699474296569813, + "mean_abs_score_delta": 0.004865826050567623, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 7, - "recall_iou_0.5": 0.8888888888888888, - "recall_iou_0.95": 0.7777777777777778, - "max_abs_score_delta": 0.022653690597534193, - "mean_abs_score_delta": 0.010112362304687525, + "py_total": 13, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, + "recall_iou_0.5": 0.9230769230769231, + "recall_iou_0.95": 0.9230769230769231, + "max_abs_score_delta": 0.007104489753723131, + "mean_abs_score_delta": 0.0024298012361526344, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 16, - "cpp_total": 17, - "matched_iou_0.5": 16, - "matched_iou_0.95": 16, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.040797788871765106, - "mean_abs_score_delta": 0.015325600333929053, - "extra_cpp_detections": 1 + "max_abs_score_delta": 0.00614907040405277, + "mean_abs_score_delta": 0.0022785010215759403, + "extra_cpp_detections": 0 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 12, - "matched_iou_0.95": 11, - "recall_iou_0.5": 0.9230769230769231, - "recall_iou_0.95": 0.8461538461538461, - "max_abs_score_delta": 0.03153283535385132, - "mean_abs_score_delta": 0.007916208371480299, - "extra_cpp_detections": 1 + "py_total": 9, + "cpp_total": 8, + "matched_iou_0.5": 8, + "matched_iou_0.95": 8, + "recall_iou_0.5": 0.8888888888888888, + "recall_iou_0.95": 0.8888888888888888, + "max_abs_score_delta": 0.015483379051208446, + "mean_abs_score_delta": 0.006054611068248755, + "extra_cpp_detections": 0 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -2189,18 +2189,18 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.005522585323333717, - "mean_abs_score_delta": 0.0018429948532104445, + "max_abs_score_delta": 0.0020743576660156027, + "mean_abs_score_delta": 0.0009095190231323214, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9731379731379731, - "recall_iou_0.95": 0.9462759462759462, - "max_abs_score_delta": 0.029977680768149235, - "mean_abs_score_delta": 0.009182991612609229, - "extra_cpp_detections": 0.2857142857142857 + "recall_iou_0.5": 0.9731379731379732, + "recall_iou_0.95": 0.9731379731379732, + "max_abs_score_delta": 0.00914083134514944, + "mean_abs_score_delta": 0.003474860910633632, + "extra_cpp_detections": 0.0 } }, { @@ -2212,7 +2212,21 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 16, + "cpp_total": 15, + "matched_iou_0.5": 15, + "matched_iou_0.95": 13, + "recall_iou_0.5": 0.9375, + "recall_iou_0.95": 0.8125, + "max_abs_score_delta": 0.06949284011077883, + "mean_abs_score_delta": 0.028136724304962148, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -2220,102 +2234,88 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.013415707778930641, - "mean_abs_score_delta": 0.0067271433876037404, + "max_abs_score_delta": 0.04795343818664555, + "mean_abs_score_delta": 0.02149029302368164, "extra_cpp_detections": 0 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.061669573738098116, - "mean_abs_score_delta": 0.029669795336914072, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 0.8, + "recall_iou_0.95": 0.8, + "max_abs_score_delta": 0.038167845298767045, + "mean_abs_score_delta": 0.020066877219200113, + "extra_cpp_detections": 0 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 13, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 9, + "recall_iou_0.5": 0.8461538461538461, + "recall_iou_0.95": 0.6923076923076923, + "max_abs_score_delta": 0.05248050162124629, + "mean_abs_score_delta": 0.018616821589036426, "extra_cpp_detections": 0 } }, { - "image": "coco_indoor.jpg", + "image": "000000252219.jpg", "metrics": { "py_total": 5, "cpp_total": 5, "matched_iou_0.5": 5, - "matched_iou_0.95": 4, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.8, - "max_abs_score_delta": 0.04369303423309323, - "mean_abs_score_delta": 0.019730626809692352, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.035706929595947234, + "mean_abs_score_delta": 0.01562539171066286, "extra_cpp_detections": 0 } }, { - "image": "coco_kitchen.jpg", + "image": "000000397133.jpg", "metrics": { "py_total": 9, "cpp_total": 7, "matched_iou_0.5": 7, - "matched_iou_0.95": 5, + "matched_iou_0.95": 6, "recall_iou_0.5": 0.7777777777777778, - "recall_iou_0.95": 0.5555555555555556, - "max_abs_score_delta": 0.055713432235717786, - "mean_abs_score_delta": 0.01721236873408725, + "recall_iou_0.95": 0.6666666666666666, + "max_abs_score_delta": 0.054788574226379394, + "mean_abs_score_delta": 0.01735674793134415, "extra_cpp_detections": 0 } }, { - "image": "coco_living_room.jpg", + "image": "bus.jpg", "metrics": { - "py_total": 16, - "cpp_total": 16, - "matched_iou_0.5": 16, - "matched_iou_0.95": 13, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.8125, - "max_abs_score_delta": 0.0897530419464111, - "mean_abs_score_delta": 0.029800655105352403, - "extra_cpp_detections": 0 - } - }, - { - "image": "coco_skater.jpg", - "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 12, - "matched_iou_0.95": 7, - "recall_iou_0.5": 0.9230769230769231, - "recall_iou_0.95": 0.5384615384615384, - "max_abs_score_delta": 0.07738240513610839, - "mean_abs_score_delta": 0.029758934428532908, - "extra_cpp_detections": 1 - } - }, - { - "image": "coco_street.jpg", - "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0342824660377502, - "mean_abs_score_delta": 0.01271650011901857, + "max_abs_score_delta": 0.012811618385314971, + "mean_abs_score_delta": 0.00751588690795899, "extra_cpp_detections": 0 } } ], "metrics": { - "recall_iou_0.5": 0.9572649572649573, - "recall_iou_0.95": 0.8152167277167278, - "max_abs_score_delta": 0.053701380158015635, - "mean_abs_score_delta": 0.020802289131600184, - "extra_cpp_detections": 0.14285714285714285 + "recall_iou_0.5": 0.9087759462759462, + "recall_iou_0.95": 0.8530677655677656, + "max_abs_score_delta": 0.04448596391786847, + "mean_abs_score_delta": 0.018401248955263762, + "extra_cpp_detections": 0.0 } }, { @@ -2327,25 +2327,25 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.005489632575988779, - "mean_abs_score_delta": 0.0038031864234924218, + "max_abs_score_delta": 0.013895137851715034, + "mean_abs_score_delta": 0.0031083554398218805, "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.995012877089599, - "min_mask_iou": 0.9912049252418645, - "mean_pixel_agreement": 0.9996824417009602 + "n_matched_for_mask": 12, + "mean_mask_iou": 0.9948938804582532, + "min_mask_iou": 0.9840619307832422, + "mean_pixel_agreement": 0.9999682120500784 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -2353,17 +2353,17 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.002271282917022721, - "mean_abs_score_delta": 0.001119152220726033, + "max_abs_score_delta": 0.0010055042762756594, + "mean_abs_score_delta": 0.0005165980806350612, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.997651099197811, - "min_mask_iou": 0.9966134494436382, - "mean_pixel_agreement": 0.99979248046875 + "mean_mask_iou": 0.9997920388192109, + "min_mask_iou": 0.9995647758595677, + "mean_pixel_agreement": 0.9999935300207039 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -2371,71 +2371,71 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.014504781387329135, - "mean_abs_score_delta": 0.00663073164749145, + "max_abs_score_delta": 0.0008840978431701174, + "mean_abs_score_delta": 0.0005062545652389228, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9901005394318527, - "min_mask_iou": 0.9727492238703, - "mean_pixel_agreement": 0.9998414855072464 + "mean_mask_iou": 0.9992193171991077, + "min_mask_iou": 0.9990253411306043, + "mean_pixel_agreement": 0.9999422200520833 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 7, - "recall_iou_0.5": 0.7777777777777778, - "recall_iou_0.95": 0.7777777777777778, - "max_abs_score_delta": 0.01584030338668818, - "mean_abs_score_delta": 0.006099138424464634, + "py_total": 6, + "cpp_total": 6, + "matched_iou_0.5": 6, + "matched_iou_0.95": 6, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0033670773010253674, + "mean_abs_score_delta": 0.0011838094863891313, "extra_cpp_detections": 0, - "n_matched_for_mask": 7, - "mean_mask_iou": 0.9918073892488393, - "min_mask_iou": 0.9809885931558935, - "mean_pixel_agreement": 0.9997036007025761 + "n_matched_for_mask": 6, + "mean_mask_iou": 0.9990458340399341, + "min_mask_iou": 0.9967741935483871, + "mean_pixel_agreement": 0.9999962022569445 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 11, - "cpp_total": 10, - "matched_iou_0.5": 10, - "matched_iou_0.95": 10, - "recall_iou_0.5": 0.9090909090909091, - "recall_iou_0.95": 0.9090909090909091, - "max_abs_score_delta": 0.02762819520568849, - "mean_abs_score_delta": 0.010858664455032353, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.003031387145996045, + "mean_abs_score_delta": 0.0012020784843444821, "extra_cpp_detections": 0, - "n_matched_for_mask": 10, - "mean_mask_iou": 0.9831683867217029, - "min_mask_iou": 0.9492282696994313, - "mean_pixel_agreement": 0.9998507188967135 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.999817052816241, + "min_mask_iou": 0.9995280792826805, + "mean_pixel_agreement": 0.9999934287383176 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 6, - "cpp_total": 6, - "matched_iou_0.5": 6, - "matched_iou_0.95": 6, + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.022669880043029833, - "mean_abs_score_delta": 0.00763482334136963, + "max_abs_score_delta": 0.023512955513000477, + "mean_abs_score_delta": 0.0056972593579973484, "extra_cpp_detections": 0, - "n_matched_for_mask": 6, - "mean_mask_iou": 0.9842445989048657, - "min_mask_iou": 0.969270166453265, - "mean_pixel_agreement": 0.9999235026041666 + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9973312712438113, + "min_mask_iou": 0.9888075503660233, + "mean_pixel_agreement": 0.9996957594513215 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -2443,25 +2443,25 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.007971623634338343, - "mean_abs_score_delta": 0.0025074780204772827, + "max_abs_score_delta": 0.001439728923797623, + "mean_abs_score_delta": 0.0008435628211975299, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9968460545108891, - "min_mask_iou": 0.9944210406467039, - "mean_pixel_agreement": 0.9999072721962617 + "mean_mask_iou": 0.9986333817546648, + "min_mask_iou": 0.9972985141828006, + "mean_pixel_agreement": 0.9999188385916782 } } ], "metrics": { - "recall_iou_0.5": 0.9552669552669553, - "recall_iou_0.95": 0.9552669552669553, - "max_abs_score_delta": 0.013767957021440782, - "mean_abs_score_delta": 0.005521882076150544, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.006733698407854332, + "mean_abs_score_delta": 0.0018654168908034793, "extra_cpp_detections": 0.0, - "mean_mask_iou": 0.9912615635865085, - "min_mask_iou": 0.9792108097872995, - "mean_pixel_agreement": 0.9998145002966679 + "mean_mask_iou": 0.9983903966187462, + "min_mask_iou": 0.9950086264504723, + "mean_pixel_agreement": 0.9999297415944467 } }, { @@ -2473,25 +2473,25 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.005710288501739491, - "mean_abs_score_delta": 0.0038183864234924147, + "max_abs_score_delta": 0.01600571611404422, + "mean_abs_score_delta": 0.004092308125495918, "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.9950229906392603, - "min_mask_iou": 0.9912064720365811, - "mean_pixel_agreement": 0.9996874714220393 + "n_matched_for_mask": 12, + "mean_mask_iou": 0.9948862248134573, + "min_mask_iou": 0.9849726775956285, + "mean_pixel_agreement": 0.9999676007433488 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -2499,17 +2499,17 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0028322755737304472, - "mean_abs_score_delta": 0.001832989237785354, + "max_abs_score_delta": 0.0014195042762756849, + "mean_abs_score_delta": 0.0007235980806351017, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.99755647639831, - "min_mask_iou": 0.9961315280464217, - "mean_pixel_agreement": 0.9997957356770834 + "mean_mask_iou": 0.9999085332866549, + "min_mask_iou": 0.9998240675580577, + "mean_pixel_agreement": 0.9999959562629399 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -2517,97 +2517,97 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.012972781387329047, - "mean_abs_score_delta": 0.006268231647491435, + "max_abs_score_delta": 0.0018988437881469755, + "mean_abs_score_delta": 0.0011238775377273857, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9901620530201017, - "min_mask_iou": 0.9727398205659076, - "mean_pixel_agreement": 0.9998431030020704 + "mean_mask_iou": 0.9990598695397965, + "min_mask_iou": 0.9982955928901874, + "mean_pixel_agreement": 0.9999235026041666 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 7, - "recall_iou_0.5": 0.7777777777777778, - "recall_iou_0.95": 0.7777777777777778, - "max_abs_score_delta": 0.01831930338668819, - "mean_abs_score_delta": 0.006694124382019043, + "py_total": 6, + "cpp_total": 6, + "matched_iou_0.5": 6, + "matched_iou_0.95": 6, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0033900773010253626, + "mean_abs_score_delta": 0.0010209761530558341, "extra_cpp_detections": 0, - "n_matched_for_mask": 7, - "mean_mask_iou": 0.9915117998055621, - "min_mask_iou": 0.9809885931558935, - "mean_pixel_agreement": 0.9997156239544999 + "n_matched_for_mask": 6, + "mean_mask_iou": 0.9992126674144738, + "min_mask_iou": 0.9967741935483871, + "mean_pixel_agreement": 0.9999972873263889 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 11, - "cpp_total": 10, - "matched_iou_0.5": 10, - "matched_iou_0.95": 10, - "recall_iou_0.5": 0.9090909090909091, - "recall_iou_0.95": 0.9090909090909091, - "max_abs_score_delta": 0.028618195205688424, - "mean_abs_score_delta": 0.011113464455032362, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.006762387145996085, + "mean_abs_score_delta": 0.002430078484344489, "extra_cpp_detections": 0, - "n_matched_for_mask": 10, - "mean_mask_iou": 0.9832068297822936, - "min_mask_iou": 0.9500406173842405, - "mean_pixel_agreement": 0.9998507188967135 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9995174689425305, + "min_mask_iou": 0.9990403071017274, + "mean_pixel_agreement": 0.9999890478971963 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 6, - "cpp_total": 6, - "matched_iou_0.5": 6, - "matched_iou_0.95": 6, + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02285688004302977, - "mean_abs_score_delta": 0.007665656674702963, + "max_abs_score_delta": 0.018339955513000494, + "mean_abs_score_delta": 0.005005973643711629, "extra_cpp_detections": 0, - "n_matched_for_mask": 6, - "mean_mask_iou": 0.9842449803366641, - "min_mask_iou": 0.969270166453265, - "mean_pixel_agreement": 0.9999235026041666 + "n_matched_for_mask": 7, + "mean_mask_iou": 0.997558605194318, + "min_mask_iou": 0.9859656177742033, + "mean_pixel_agreement": 0.9996262336901973 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, - "cpp_total": 4, - "matched_iou_0.5": 4, - "matched_iou_0.95": 4, - "recall_iou_0.5": 0.8, - "recall_iou_0.95": 0.8, - "max_abs_score_delta": 0.009744623634338367, - "mean_abs_score_delta": 0.00320391595268249, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0012337289237975835, + "mean_abs_score_delta": 0.0005739560935974052, "extra_cpp_detections": 0, - "n_matched_for_mask": 4, - "mean_mask_iou": 0.9961410596342735, - "min_mask_iou": 0.9939697348958926, - "mean_pixel_agreement": 0.9998859155957944 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.99833085610934, + "min_mask_iou": 0.9962158753040814, + "mean_pixel_agreement": 0.9999122085048011 } } ], "metrics": { - "recall_iou_0.5": 0.9266955266955267, - "recall_iou_0.95": 0.9266955266955267, - "max_abs_score_delta": 0.01443633539036339, - "mean_abs_score_delta": 0.0057995383961722945, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.007007173294612344, + "mean_abs_score_delta": 0.0021386811597953947, "extra_cpp_detections": 0.0, - "mean_mask_iou": 0.9911208842309236, - "min_mask_iou": 0.9791924189340289, - "mean_pixel_agreement": 0.9998145815931954 + "mean_mask_iou": 0.9983534607572244, + "min_mask_iou": 0.9944411902531817, + "mean_pixel_agreement": 0.9999159767184341 } }, { @@ -2619,25 +2619,25 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 11, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.005744711498260502, - "mean_abs_score_delta": 0.003192217992401103, + "recall_iou_0.95": 0.9166666666666666, + "max_abs_score_delta": 0.08989222549819942, + "mean_abs_score_delta": 0.019741284927050257, "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.9926903319438741, - "min_mask_iou": 0.9826477583017704, - "mean_pixel_agreement": 0.9996458619112939 + "n_matched_for_mask": 12, + "mean_mask_iou": 0.984194135149746, + "min_mask_iou": 0.9556786703601108, + "mean_pixel_agreement": 0.9998921043622849 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -2645,17 +2645,17 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0026902829170226683, - "mean_abs_score_delta": 0.0022900703954696344, + "max_abs_score_delta": 0.02001249572372432, + "mean_abs_score_delta": 0.00899184531497954, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.997214557380885, - "min_mask_iou": 0.9961277831558567, - "mean_pixel_agreement": 0.9997526041666667 + "mean_mask_iou": 0.9914948886232382, + "min_mask_iou": 0.981359649122807, + "mean_pixel_agreement": 0.9998431030020704 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -2663,71 +2663,71 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.017885781387329103, - "mean_abs_score_delta": 0.008118231647491453, + "max_abs_score_delta": 0.004716097843170175, + "mean_abs_score_delta": 0.0015954161405563605, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9866789104770977, - "min_mask_iou": 0.9600546261522704, - "mean_pixel_agreement": 0.9997581845238095 + "mean_mask_iou": 0.996703048724616, + "min_mask_iou": 0.9937619961612284, + "mean_pixel_agreement": 0.9997648111979167 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 9, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 7, - "recall_iou_0.5": 0.7777777777777778, - "recall_iou_0.95": 0.7777777777777778, - "max_abs_score_delta": 0.020357532691955615, - "mean_abs_score_delta": 0.008007897325788216, + "py_total": 6, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 0.8333333333333334, + "recall_iou_0.95": 0.8333333333333334, + "max_abs_score_delta": 0.0060415812759399, + "mean_abs_score_delta": 0.002962260739135747, "extra_cpp_detections": 0, - "n_matched_for_mask": 7, - "mean_mask_iou": 0.9913106011416727, - "min_mask_iou": 0.9817470664928292, - "mean_pixel_agreement": 0.9995838909334225 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9926538877883567, + "min_mask_iou": 0.9836065573770492, + "mean_pixel_agreement": 0.9999544270833333 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 11, - "cpp_total": 10, - "matched_iou_0.5": 10, - "matched_iou_0.95": 10, - "recall_iou_0.5": 0.9090909090909091, - "recall_iou_0.95": 0.9090909090909091, - "max_abs_score_delta": 0.03336620502471921, - "mean_abs_score_delta": 0.01009677058067322, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.020633937763214094, + "mean_abs_score_delta": 0.007852111617279034, "extra_cpp_detections": 0, - "n_matched_for_mask": 10, - "mean_mask_iou": 0.9833274986832343, - "min_mask_iou": 0.9605370219690805, - "mean_pixel_agreement": 0.9998448503521127 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9962556446274924, + "min_mask_iou": 0.9946100917431193, + "mean_pixel_agreement": 0.9999072721962616 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 6, - "cpp_total": 6, - "matched_iou_0.5": 6, - "matched_iou_0.95": 6, + "py_total": 7, + "cpp_total": 8, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.021642781578063985, - "mean_abs_score_delta": 0.00885565667470295, - "extra_cpp_detections": 0, - "n_matched_for_mask": 6, - "mean_mask_iou": 0.9836195190035162, - "min_mask_iou": 0.9652956298200515, - "mean_pixel_agreement": 0.9999088541666666 + "max_abs_score_delta": 0.009340022518157909, + "mean_abs_score_delta": 0.0044967482310703955, + "extra_cpp_detections": 1, + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9924961252838403, + "min_mask_iou": 0.984375, + "mean_pixel_agreement": 0.9997077827032452 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -2735,25 +2735,25 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.007531623634338347, - "mean_abs_score_delta": 0.003420787503814693, + "max_abs_score_delta": 0.02335142181396488, + "mean_abs_score_delta": 0.007041179198455816, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9955740416029778, - "min_mask_iou": 0.9903660886319846, - "mean_pixel_agreement": 0.9998634637850469 + "mean_mask_iou": 0.9946549474441149, + "min_mask_iou": 0.9866369710467706, + "mean_pixel_agreement": 0.9997631458619113 } } ], "metrics": { - "recall_iou_0.5": 0.9552669552669553, - "recall_iou_0.95": 0.9552669552669553, - "max_abs_score_delta": 0.015602702675955633, - "mean_abs_score_delta": 0.0062830903029058955, - "extra_cpp_detections": 0.0, - "mean_mask_iou": 0.9900593514618939, - "min_mask_iou": 0.9766822820748348, - "mean_pixel_agreement": 0.9997653871198598 + "recall_iou_0.5": 0.9761904761904763, + "recall_iou_0.95": 0.9642857142857143, + "max_abs_score_delta": 0.0248553974909101, + "mean_abs_score_delta": 0.0075258351669324495, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.992636096805915, + "min_mask_iou": 0.9828612765444408, + "mean_pixel_agreement": 0.9998332352010034 } }, { @@ -2765,43 +2765,43 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 2, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.4, - "max_abs_score_delta": 0.04040363257598878, - "mean_abs_score_delta": 0.026360313521575907, - "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.979057551558153, - "min_mask_iou": 0.9629181433140289, - "mean_pixel_agreement": 0.998936442615455 + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 11, + "matched_iou_0.95": 6, + "recall_iou_0.5": 0.9166666666666666, + "recall_iou_0.95": 0.5, + "max_abs_score_delta": 0.05798913785171511, + "mean_abs_score_delta": 0.032896418137290266, + "extra_cpp_detections": 1, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.9243202948325806, + "min_mask_iou": 0.8382303275920298, + "mean_pixel_agreement": 0.999394472897994 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 4, "matched_iou_0.5": 4, - "matched_iou_0.95": 4, + "matched_iou_0.95": 1, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.020563837684631325, - "mean_abs_score_delta": 0.012460597779273969, + "recall_iou_0.95": 0.25, + "max_abs_score_delta": 0.04609788679122928, + "mean_abs_score_delta": 0.02638184745311739, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9792666122287276, - "min_mask_iou": 0.9616674652611404, - "mean_pixel_agreement": 0.99848388671875 + "mean_mask_iou": 0.9680569556274616, + "min_mask_iou": 0.9496945742005031, + "mean_pixel_agreement": 0.9990610442546584 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -2809,71 +2809,35 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.13856778138732906, - "mean_abs_score_delta": 0.05516298164749142, + "max_abs_score_delta": 0.031386097843170147, + "mean_abs_score_delta": 0.02026438278102874, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.96839976841067, - "min_mask_iou": 0.9528813559322034, - "mean_pixel_agreement": 0.9989704645445134 - } - }, - { - "image": "coco_kitchen.jpg", - "metrics": { - "py_total": 9, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 3, - "recall_iou_0.5": 0.8888888888888888, - "recall_iou_0.95": 0.3333333333333333, - "max_abs_score_delta": 0.08290904914855957, - "mean_abs_score_delta": 0.03818969259643554, - "extra_cpp_detections": 0, - "n_matched_for_mask": 8, - "mean_mask_iou": 0.9640701804931057, - "min_mask_iou": 0.9323790621679597, - "mean_pixel_agreement": 0.998138356264637 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 11, - "cpp_total": 11, - "matched_iou_0.5": 10, - "matched_iou_0.95": 5, - "recall_iou_0.5": 0.9090909090909091, - "recall_iou_0.95": 0.45454545454545453, - "max_abs_score_delta": 0.10176234372329707, - "mean_abs_score_delta": 0.04296790627098083, - "extra_cpp_detections": 1, - "n_matched_for_mask": 10, - "mean_mask_iou": 0.924085622150541, - "min_mask_iou": 0.852304469273743, - "mean_pixel_agreement": 0.9993372212441315 + "mean_mask_iou": 0.981884490754551, + "min_mask_iou": 0.9727403156384505, + "mean_pixel_agreement": 0.9985205078125 } }, { - "image": "coco_skater.jpg", + "image": "000000087038.jpg", "metrics": { "py_total": 6, - "cpp_total": 5, + "cpp_total": 4, "matched_iou_0.5": 4, - "matched_iou_0.95": 3, + "matched_iou_0.95": 2, "recall_iou_0.5": 0.6666666666666666, - "recall_iou_0.95": 0.5, - "max_abs_score_delta": 0.10183697113418577, - "mean_abs_score_delta": 0.05703806960678101, - "extra_cpp_detections": 1, + "recall_iou_0.95": 0.3333333333333333, + "max_abs_score_delta": 0.12952473206710813, + "mean_abs_score_delta": 0.0699079617500305, + "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9547632731314373, - "min_mask_iou": 0.9428341384863124, - "mean_pixel_agreement": 0.9996443684895833 + "mean_mask_iou": 0.9529173500616355, + "min_mask_iou": 0.9200879765395894, + "mean_pixel_agreement": 0.99968017578125 } }, { - "image": "coco_street.jpg", + "image": "000000252219.jpg", "metrics": { "py_total": 5, "cpp_total": 4, @@ -2881,55 +2845,91 @@ "matched_iou_0.95": 3, "recall_iou_0.5": 0.8, "recall_iou_0.95": 0.6, - "max_abs_score_delta": 0.02525803736877441, - "mean_abs_score_delta": 0.014207665952682497, + "max_abs_score_delta": 0.07273638714599606, + "mean_abs_score_delta": 0.036132611026763906, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9755832339926203, - "min_mask_iou": 0.9600749972427485, - "mean_pixel_agreement": 0.9991849810163551 + "mean_mask_iou": 0.971401788437586, + "min_mask_iou": 0.9628381455505965, + "mean_pixel_agreement": 0.9991338712032709 } - } - ], - "metrics": { - "recall_iou_0.5": 0.8949494949494949, - "recall_iou_0.95": 0.6125541125541126, - "max_abs_score_delta": 0.07304309328896656, - "mean_abs_score_delta": 0.03519817533931731, - "extra_cpp_detections": 0.2857142857142857, - "mean_mask_iou": 0.9636037488521793, - "min_mask_iou": 0.9378656616683052, - "mean_pixel_agreement": 0.9989565315562036 - } - }, - { - "variant": "seg-small", - "quant": "f32", + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 3, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.42857142857142855, + "max_abs_score_delta": 0.093226390586853, + "mean_abs_score_delta": 0.04058223920876639, + "extra_cpp_detections": 0, + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9679570765565123, + "min_mask_iou": 0.947066007327474, + "mean_pixel_agreement": 0.9983716334894613 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 3, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.6, + "max_abs_score_delta": 0.03570042181396482, + "mean_abs_score_delta": 0.024826803745269778, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9766205965676656, + "min_mask_iou": 0.9623798182940814, + "mean_pixel_agreement": 0.9985983081847281 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9119047619047619, + "recall_iou_0.95": 0.5302721088435375, + "max_abs_score_delta": 0.06666586487143379, + "mean_abs_score_delta": 0.035856037728895285, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9633083646911419, + "min_mask_iou": 0.9361481664489607, + "mean_pixel_agreement": 0.9989657162319804 + } + }, + { + "variant": "seg-small", + "quant": "f32", "model_path": "models/rfdetr-seg-small-f32.gguf", "is_seg": true, "file_size_mb": 127.64, "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.004803597068786636, - "mean_abs_score_delta": 0.0025784842720031564, + "max_abs_score_delta": 0.01864630392456057, + "mean_abs_score_delta": 0.0034423000402450574, "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.9965062444284285, - "min_mask_iou": 0.9945496270797476, - "mean_pixel_agreement": 0.9997347965249199 + "n_matched_for_mask": 12, + "mean_mask_iou": 0.9968332765826177, + "min_mask_iou": 0.9890476190476191, + "mean_pixel_agreement": 0.9999517067683881 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -2937,17 +2937,17 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.004012732822418208, - "mean_abs_score_delta": 0.0017602289705276608, + "max_abs_score_delta": 0.006860141502380346, + "mean_abs_score_delta": 0.0027497729072570865, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9977882121746642, - "min_mask_iou": 0.9958333333333333, - "mean_pixel_agreement": 0.9998518880208334 + "mean_mask_iou": 0.9985478574922297, + "min_mask_iou": 0.997858978018841, + "mean_pixel_agreement": 0.9999433876811594 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -2955,97 +2955,97 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.011611926273345952, - "mean_abs_score_delta": 0.005757000498771675, + "max_abs_score_delta": 0.0035515236968993724, + "mean_abs_score_delta": 0.001159286384582514, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9912324802759594, - "min_mask_iou": 0.9808486669170109, - "mean_pixel_agreement": 0.9998107531055901 + "mean_mask_iou": 0.9995307282127117, + "min_mask_iou": 0.9992374384563102, + "mean_pixel_agreement": 0.99993408203125 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 8, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.018849967365264897, - "mean_abs_score_delta": 0.005600470584869385, + "max_abs_score_delta": 0.007315583461761421, + "mean_abs_score_delta": 0.0017237641034807405, "extra_cpp_detections": 0, - "n_matched_for_mask": 8, - "mean_mask_iou": 0.9904251741925627, - "min_mask_iou": 0.9750889679715302, - "mean_pixel_agreement": 0.9997781579332553 + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9985427944417433, + "min_mask_iou": 0.9968847352024922, + "mean_pixel_agreement": 0.999995814732143 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 13, - "matched_iou_0.95": 13, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.031639519027710006, - "mean_abs_score_delta": 0.008974079134721044, + "max_abs_score_delta": 0.0055066366539001654, + "mean_abs_score_delta": 0.001249619862365736, "extra_cpp_detections": 0, - "n_matched_for_mask": 13, - "mean_mask_iou": 0.9866818995044675, - "min_mask_iou": 0.9679885332059245, - "mean_pixel_agreement": 0.999860057782593 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9994948897978627, + "min_mask_iou": 0.9989577411407997, + "mean_pixel_agreement": 0.9999861273364485 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 8, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.014571069637298595, - "mean_abs_score_delta": 0.0064967481646537595, + "max_abs_score_delta": 0.004314737998962426, + "mean_abs_score_delta": 0.0020634610666547338, "extra_cpp_detections": 0, - "n_matched_for_mask": 8, - "mean_mask_iou": 0.9892368961526805, - "min_mask_iou": 0.9788519637462235, - "mean_pixel_agreement": 0.9999576822916667 + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9974446249407443, + "min_mask_iou": 0.9852779886948452, + "mean_pixel_agreement": 0.9996283246905319 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { - "py_total": 4, - "cpp_total": 4, - "matched_iou_0.5": 4, - "matched_iou_0.95": 4, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.008153155002593948, - "mean_abs_score_delta": 0.0025689049930572583, + "max_abs_score_delta": 0.07204178799819949, + "mean_abs_score_delta": 0.01568808882446291, "extra_cpp_detections": 0, - "n_matched_for_mask": 4, - "mean_mask_iou": 0.9951778244427725, - "min_mask_iou": 0.9917682926829269, - "mean_pixel_agreement": 0.9998795268691589 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9972412353165716, + "min_mask_iou": 0.9935334427178844, + "mean_pixel_agreement": 0.9997315957933243 } } ], "metrics": { "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.013377423885345463, - "mean_abs_score_delta": 0.004819416659800562, + "max_abs_score_delta": 0.016890959319523398, + "mean_abs_score_delta": 0.004010899027006969, "extra_cpp_detections": 0.0, - "mean_mask_iou": 0.992435533024505, - "min_mask_iou": 0.9835613407052425, - "mean_pixel_agreement": 0.9998389803611454 + "mean_mask_iou": 0.99823362954064, + "min_mask_iou": 0.9943997061826845, + "mean_pixel_agreement": 0.9998815770047493 } }, { @@ -3057,25 +3057,25 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.004609597068786608, - "mean_abs_score_delta": 0.0024904842720031573, + "max_abs_score_delta": 0.005740557556152326, + "mean_abs_score_delta": 0.0022873446149825984, "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.9967244038144083, - "min_mask_iou": 0.9945010280686655, - "mean_pixel_agreement": 0.9997919524462734 + "n_matched_for_mask": 12, + "mean_mask_iou": 0.996348980195855, + "min_mask_iou": 0.9906580433220409, + "mean_pixel_agreement": 0.9999416202073551 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -3083,17 +3083,17 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0031797328224182353, - "mean_abs_score_delta": 0.002097478970527644, + "max_abs_score_delta": 0.006593141502380329, + "mean_abs_score_delta": 0.002688772907257081, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9980015742388915, - "min_mask_iou": 0.9962962962962963, - "mean_pixel_agreement": 0.9998502604166667 + "mean_mask_iou": 0.9983089100628549, + "min_mask_iou": 0.9973107952362659, + "mean_pixel_agreement": 0.9999425789337474 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -3101,97 +3101,97 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.010885926273345947, - "mean_abs_score_delta": 0.004749250498771673, + "max_abs_score_delta": 0.001405827537536597, + "mean_abs_score_delta": 0.0009089383049011024, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9911858100757972, - "min_mask_iou": 0.9801125703564728, - "mean_pixel_agreement": 0.99980266563147 + "mean_mask_iou": 0.9994248343701576, + "min_mask_iou": 0.9992541148000199, + "mean_pixel_agreement": 0.9999348958333334 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 8, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.018831967365264934, - "mean_abs_score_delta": 0.006467657543182376, + "max_abs_score_delta": 0.007744583461761434, + "mean_abs_score_delta": 0.0017377219161987137, "extra_cpp_detections": 0, - "n_matched_for_mask": 8, - "mean_mask_iou": 0.9908571461863314, - "min_mask_iou": 0.9750889679715302, - "mean_pixel_agreement": 0.9998275578161593 + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9987968325903764, + "min_mask_iou": 0.9969712266532055, + "mean_pixel_agreement": 0.9999939546130953 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 13, - "matched_iou_0.95": 13, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.03348551902770991, - "mean_abs_score_delta": 0.008514002211644103, + "max_abs_score_delta": 0.0022041691474914415, + "mean_abs_score_delta": 0.0007912328598022489, "extra_cpp_detections": 0, - "n_matched_for_mask": 13, - "mean_mask_iou": 0.9868305531484312, - "min_mask_iou": 0.9680038204393505, - "mean_pixel_agreement": 0.9998631613398339 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9995129212029765, + "min_mask_iou": 0.9987623762376238, + "mean_pixel_agreement": 0.9999897780373832 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 8, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.015222135059356656, - "mean_abs_score_delta": 0.007207373164653752, + "max_abs_score_delta": 0.0051340960159301785, + "mean_abs_score_delta": 0.0023536039237976075, "extra_cpp_detections": 0, - "n_matched_for_mask": 8, - "mean_mask_iou": 0.9885390347213263, - "min_mask_iou": 0.9759036144578314, - "mean_pixel_agreement": 0.9999556477864584 + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9968901671924479, + "min_mask_iou": 0.9815584961142617, + "mean_pixel_agreement": 0.9995331841753096 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { - "py_total": 4, - "cpp_total": 4, - "matched_iou_0.5": 4, - "matched_iou_0.95": 4, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.008052155002594041, - "mean_abs_score_delta": 0.002652154993057265, + "max_abs_score_delta": 0.07229878799819944, + "mean_abs_score_delta": 0.01576708882446287, "extra_cpp_detections": 0, - "n_matched_for_mask": 4, - "mean_mask_iou": 0.9952025387541797, - "min_mask_iou": 0.9917682926829269, - "mean_pixel_agreement": 0.9998804395443925 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9971365377527153, + "min_mask_iou": 0.9935334427178844, + "mean_pixel_agreement": 0.999732053040695 } } ], "metrics": { "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.013466718945639475, - "mean_abs_score_delta": 0.004882628807691425, + "max_abs_score_delta": 0.014445880459921678, + "mean_abs_score_delta": 0.003790671907343175, "extra_cpp_detections": 0.0, - "mean_mask_iou": 0.9924772944199093, - "min_mask_iou": 0.9830963700390105, - "mean_pixel_agreement": 0.9998530978544649 + "mean_mask_iou": 0.9980598833381977, + "min_mask_iou": 0.9940069278687574, + "mean_pixel_agreement": 0.9998668664058455 } }, { @@ -3203,25 +3203,25 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.005232402420043969, - "mean_abs_score_delta": 0.0031840842720031624, + "max_abs_score_delta": 0.04629022237014768, + "mean_abs_score_delta": 0.010242340380668643, "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.99517980561178, - "min_mask_iou": 0.990769083604362, - "mean_pixel_agreement": 0.999679012345679 + "n_matched_for_mask": 12, + "mean_mask_iou": 0.9855639434124729, + "min_mask_iou": 0.9704918032786886, + "mean_pixel_agreement": 0.9998306680359939 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -3229,17 +3229,17 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.004388824954986603, - "mean_abs_score_delta": 0.0021043722162246703, + "max_abs_score_delta": 0.005748166149139422, + "mean_abs_score_delta": 0.004829272907257071, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9975110528422335, - "min_mask_iou": 0.995835261453031, - "mean_pixel_agreement": 0.9998307291666667 + "mean_mask_iou": 0.9945465292307372, + "min_mask_iou": 0.9898624185372918, + "mean_pixel_agreement": 0.9998673654244307 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -3247,97 +3247,97 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.00920792627334599, - "mean_abs_score_delta": 0.004361624999046354, + "max_abs_score_delta": 0.006127827537536601, + "mean_abs_score_delta": 0.0033475777988433986, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9891578647254572, - "min_mask_iou": 0.9713328369322413, - "mean_pixel_agreement": 0.9997986218944099 + "mean_mask_iou": 0.9968991923325123, + "min_mask_iou": 0.9926907263590681, + "mean_pixel_agreement": 0.99981689453125 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 8, + "py_total": 7, "cpp_total": 7, "matched_iou_0.5": 7, "matched_iou_0.95": 7, - "recall_iou_0.5": 0.875, - "recall_iou_0.95": 0.875, - "max_abs_score_delta": 0.006903921382904077, - "mean_abs_score_delta": 0.0036062962401254123, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.006693432106018027, + "mean_abs_score_delta": 0.0033880889838082523, "extra_cpp_detections": 0, "n_matched_for_mask": 7, - "mean_mask_iou": 0.9901106296617692, - "min_mask_iou": 0.9727164887307236, - "mean_pixel_agreement": 0.9996748494479759 + "mean_mask_iou": 0.993297165864223, + "min_mask_iou": 0.9864253393665159, + "mean_pixel_agreement": 0.999965587797619 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 13, - "cpp_total": 13, - "matched_iou_0.5": 13, - "matched_iou_0.95": 13, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.03347442054367067, - "mean_abs_score_delta": 0.01009618533530602, + "max_abs_score_delta": 0.00980083085250849, + "mean_abs_score_delta": 0.004334712478637681, "extra_cpp_detections": 0, - "n_matched_for_mask": 13, - "mean_mask_iou": 0.9857044832162116, - "min_mask_iou": 0.9701421800947867, - "mean_pixel_agreement": 0.9998442578548212 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9957545438778972, + "min_mask_iou": 0.9912974683544303, + "mean_pixel_agreement": 0.9998766063084112 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 8, - "cpp_total": 8, - "matched_iou_0.5": 8, - "matched_iou_0.95": 8, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02045995914459231, - "mean_abs_score_delta": 0.00619149816465378, - "extra_cpp_detections": 0, - "n_matched_for_mask": 8, - "mean_mask_iou": 0.9854992845923237, - "min_mask_iou": 0.9683098591549296, - "mean_pixel_agreement": 0.9999483235677084 + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 6, + "matched_iou_0.95": 6, + "recall_iou_0.5": 0.8571428571428571, + "recall_iou_0.95": 0.8571428571428571, + "max_abs_score_delta": 0.0089149039840698, + "mean_abs_score_delta": 0.004289365088780713, + "extra_cpp_detections": 1, + "n_matched_for_mask": 6, + "mean_mask_iou": 0.9946267020980236, + "min_mask_iou": 0.9857549857549858, + "mean_pixel_agreement": 0.9999481606167057 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { - "py_total": 4, - "cpp_total": 4, - "matched_iou_0.5": 4, - "matched_iou_0.95": 4, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.00251694188690188, - "mean_abs_score_delta": 0.0022589049930572536, + "max_abs_score_delta": 0.06540678799819943, + "mean_abs_score_delta": 0.014328229464721676, "extra_cpp_detections": 0, - "n_matched_for_mask": 4, - "mean_mask_iou": 0.9953228408979844, - "min_mask_iou": 0.9932885906040269, - "mean_pixel_agreement": 0.9998758761682243 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9949121836867614, + "min_mask_iou": 0.9886951060437702, + "mean_pixel_agreement": 0.9997021033379058 } } ], "metrics": { - "recall_iou_0.5": 0.9821428571428571, - "recall_iou_0.95": 0.9821428571428571, - "max_abs_score_delta": 0.011740628086635072, - "mean_abs_score_delta": 0.00454328088863095, - "extra_cpp_detections": 0.0, - "mean_mask_iou": 0.9912122802211085, - "min_mask_iou": 0.9803420429391573, - "mean_pixel_agreement": 0.9998073814922123 + "recall_iou_0.5": 0.9795918367346939, + "recall_iou_0.95": 0.9795918367346939, + "max_abs_score_delta": 0.02128316728537421, + "mean_abs_score_delta": 0.006394226728959633, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9936571800718038, + "min_mask_iou": 0.9864596925278216, + "mean_pixel_agreement": 0.9998581980074738 } }, { @@ -3349,25 +3349,25 @@ "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 1, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.2, - "max_abs_score_delta": 0.03689140242004396, - "mean_abs_score_delta": 0.02119408427200318, + "py_total": 12, + "cpp_total": 9, + "matched_iou_0.5": 9, + "matched_iou_0.95": 4, + "recall_iou_0.5": 0.75, + "recall_iou_0.95": 0.3333333333333333, + "max_abs_score_delta": 0.07386449251937866, + "mean_abs_score_delta": 0.037431485187954375, "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.978620305500715, - "min_mask_iou": 0.9613524344741812, - "mean_pixel_agreement": 0.9986554641060813 + "n_matched_for_mask": 9, + "mean_mask_iou": 0.9530579709782887, + "min_mask_iou": 0.8922528940338379, + "mean_pixel_agreement": 0.9996434044079289 } }, { - "image": "coco_cats.jpg", + "image": "000000000632.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -3375,17 +3375,17 @@ "matched_iou_0.95": 2, "recall_iou_0.5": 1.0, "recall_iou_0.95": 0.5, - "max_abs_score_delta": 0.062041144596099884, - "mean_abs_score_delta": 0.02141020008182526, + "max_abs_score_delta": 0.05771840107345583, + "mean_abs_score_delta": 0.03206379784393312, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9826862588660652, - "min_mask_iou": 0.9765345289165565, - "mean_pixel_agreement": 0.9980289713541667 + "mean_mask_iou": 0.9792609502805215, + "min_mask_iou": 0.964152352501867, + "mean_pixel_agreement": 0.9993546195652174 } }, { - "image": "coco_indoor.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -3393,97 +3393,97 @@ "matched_iou_0.95": 3, "recall_iou_0.5": 1.0, "recall_iou_0.95": 0.75, - "max_abs_score_delta": 0.06397620021820072, - "mean_abs_score_delta": 0.040239124999046305, + "max_abs_score_delta": 0.059616523696899404, + "mean_abs_score_delta": 0.02767643404960632, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9662982723678317, - "min_mask_iou": 0.9278863232682061, - "mean_pixel_agreement": 0.9989041472567288 + "mean_mask_iou": 0.9817715197178118, + "min_mask_iou": 0.9745916515426497, + "mean_pixel_agreement": 0.9981892903645834 } }, { - "image": "coco_kitchen.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 8, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 3, - "recall_iou_0.5": 0.875, - "recall_iou_0.95": 0.375, - "max_abs_score_delta": 0.06285924783325192, - "mean_abs_score_delta": 0.03266244699914113, + "py_total": 7, + "cpp_total": 6, + "matched_iou_0.5": 6, + "matched_iou_0.95": 2, + "recall_iou_0.5": 0.8571428571428571, + "recall_iou_0.95": 0.2857142857142857, + "max_abs_score_delta": 0.10175743210601806, + "mean_abs_score_delta": 0.043734268325805646, "extra_cpp_detections": 0, - "n_matched_for_mask": 7, - "mean_mask_iou": 0.9714842494232493, - "min_mask_iou": 0.9371459178838508, - "mean_pixel_agreement": 0.9981944212111075 + "n_matched_for_mask": 6, + "mean_mask_iou": 0.9666431173404563, + "min_mask_iou": 0.953454189122979, + "mean_pixel_agreement": 0.9998046875000001 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 13, - "cpp_total": 12, - "matched_iou_0.5": 12, - "matched_iou_0.95": 4, - "recall_iou_0.5": 0.9230769230769231, - "recall_iou_0.95": 0.3076923076923077, - "max_abs_score_delta": 0.21533648097229008, - "mean_abs_score_delta": 0.05806302239481608, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 1, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.2, + "max_abs_score_delta": 0.06532536334609984, + "mean_abs_score_delta": 0.023985329335021975, "extra_cpp_detections": 0, - "n_matched_for_mask": 12, - "mean_mask_iou": 0.9422190950042496, - "min_mask_iou": 0.8767482517482518, - "mean_pixel_agreement": 0.9995781983568075 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9721928555545378, + "min_mask_iou": 0.948706942805022, + "mean_pixel_agreement": 0.9991596086448599 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 8, + "py_total": 7, "cpp_total": 7, "matched_iou_0.5": 7, - "matched_iou_0.95": 0, - "recall_iou_0.5": 0.875, - "recall_iou_0.95": 0.0, - "max_abs_score_delta": 0.1312761147460938, - "mean_abs_score_delta": 0.038822627734592996, + "matched_iou_0.95": 2, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.2857142857142857, + "max_abs_score_delta": 0.07404626200103759, + "mean_abs_score_delta": 0.03343468488693235, "extra_cpp_detections": 0, "n_matched_for_mask": 7, - "mean_mask_iou": 0.9534218346868099, - "min_mask_iou": 0.9255079006772009, - "mean_pixel_agreement": 0.9997893415178571 + "mean_mask_iou": 0.974090430042894, + "min_mask_iou": 0.9532301977585952, + "mean_pixel_agreement": 0.9986413725326196 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { - "py_total": 4, - "cpp_total": 4, - "matched_iou_0.5": 4, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, "matched_iou_0.95": 2, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.5, - "max_abs_score_delta": 0.12025115500259398, - "mean_abs_score_delta": 0.04729518404960631, + "recall_iou_0.95": 0.4, + "max_abs_score_delta": 0.05036878799819944, + "mean_abs_score_delta": 0.027595229464721684, "extra_cpp_detections": 0, - "n_matched_for_mask": 4, - "mean_mask_iou": 0.9710133032659873, - "min_mask_iou": 0.9486657378630372, - "mean_pixel_agreement": 0.9990572064836449 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9779896049251512, + "min_mask_iou": 0.9629612374207976, + "mean_pixel_agreement": 0.9985235482395977 } } ], "metrics": { - "recall_iou_0.5": 0.9532967032967034, - "recall_iou_0.95": 0.3760989010989011, - "max_abs_score_delta": 0.09894739225551062, - "mean_abs_score_delta": 0.03709809864729018, + "recall_iou_0.5": 0.9438775510204083, + "recall_iou_0.95": 0.3935374149659864, + "max_abs_score_delta": 0.06895675182015555, + "mean_abs_score_delta": 0.03227446129913935, "extra_cpp_detections": 0.0, - "mean_mask_iou": 0.9665347598735583, - "min_mask_iou": 0.9362630135473263, - "mean_pixel_agreement": 0.9988868214694848 + "mean_mask_iou": 0.9721437784056659, + "min_mask_iou": 0.9499070664551068, + "mean_pixel_agreement": 0.9990452187506867 } }, { @@ -3491,29 +3491,47 @@ "quant": "f32", "model_path": "models/rfdetr-seg-medium-f32.gguf", "is_seg": true, - "file_size_mb": 133.66, + "file_size_mb": 133.92, "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 11, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.011957222377777077, + "mean_abs_score_delta": 0.005011236404418948, + "extra_cpp_detections": 0, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.9959076746354684, + "min_mask_iou": 0.9815623483745755, + "mean_pixel_agreement": 0.99996298815621 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.007185522182464599, - "mean_abs_score_delta": 0.0026467732086181694, + "max_abs_score_delta": 0.002773267230987586, + "mean_abs_score_delta": 0.0010309701942444072, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9911752418069499, - "min_mask_iou": 0.990207294193017, - "mean_pixel_agreement": 0.999257430269776 + "mean_mask_iou": 0.998778577361912, + "min_mask_iou": 0.996078431372549, + "mean_pixel_agreement": 0.9999864130434781 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -3521,17 +3539,35 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02045035411834717, - "mean_abs_score_delta": 0.011774247755050649, + "max_abs_score_delta": 0.02142312522888179, + "mean_abs_score_delta": 0.006210483010292034, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9916977702304071, - "min_mask_iou": 0.9856115107913669, - "mean_pixel_agreement": 0.9994588216145833 + "mean_mask_iou": 0.9983162426864555, + "min_mask_iou": 0.9972339544513458, + "mean_pixel_agreement": 0.9998225911458334 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", + "metrics": { + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.018842480155944807, + "mean_abs_score_delta": 0.005151399068559929, + "extra_cpp_detections": 0, + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9958225857720928, + "min_mask_iou": 0.9884667571234735, + "mean_pixel_agreement": 0.9999730282738096 + } + }, + { + "image": "000000252219.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -3539,71 +3575,35 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.010029575233459487, - "mean_abs_score_delta": 0.005500186113739014, + "max_abs_score_delta": 0.01401046043014531, + "mean_abs_score_delta": 0.0032083470306396578, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9868232380878279, - "min_mask_iou": 0.975, - "mean_pixel_agreement": 0.9997224378881988 + "mean_mask_iou": 0.9994640437147089, + "min_mask_iou": 0.9991074085093722, + "mean_pixel_agreement": 0.9999868574766356 } }, { - "image": "coco_kitchen.jpg", + "image": "000000397133.jpg", "metrics": { "py_total": 9, - "cpp_total": 8, + "cpp_total": 9, "matched_iou_0.5": 8, "matched_iou_0.95": 8, "recall_iou_0.5": 0.8888888888888888, "recall_iou_0.95": 0.8888888888888888, - "max_abs_score_delta": 0.02262238623809809, - "mean_abs_score_delta": 0.008378630692958841, - "extra_cpp_detections": 0, - "n_matched_for_mask": 8, - "mean_mask_iou": 0.9843642385044988, - "min_mask_iou": 0.9693251533742331, - "mean_pixel_agreement": 0.999864150322014 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 11, - "cpp_total": 11, - "matched_iou_0.5": 10, - "matched_iou_0.95": 9, - "recall_iou_0.5": 0.9090909090909091, - "recall_iou_0.95": 0.8181818181818182, - "max_abs_score_delta": 0.049635429336547854, - "mean_abs_score_delta": 0.012320856071853648, + "max_abs_score_delta": 0.012905866786956821, + "mean_abs_score_delta": 0.00335323020172118, "extra_cpp_detections": 1, - "n_matched_for_mask": 10, - "mean_mask_iou": 0.9502960860073456, - "min_mask_iou": 0.8601340033500837, - "mean_pixel_agreement": 0.9993988409624412 - } - }, - { - "image": "coco_skater.jpg", - "metrics": { - "py_total": 7, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 7, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.021750610385894764, - "mean_abs_score_delta": 0.009651084678104938, - "extra_cpp_detections": 0, - "n_matched_for_mask": 7, - "mean_mask_iou": 0.971119867010615, - "min_mask_iou": 0.9411764705882353, - "mean_pixel_agreement": 0.9998628162202381 + "n_matched_for_mask": 8, + "mean_mask_iou": 0.9975990061142437, + "min_mask_iou": 0.9878817256422685, + "mean_pixel_agreement": 0.9999789593091335 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -3611,25 +3611,25 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.07336644316482543, - "mean_abs_score_delta": 0.023700945015716533, + "max_abs_score_delta": 0.0027393548545837643, + "mean_abs_score_delta": 0.0017865114547729543, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9892344178192488, - "min_mask_iou": 0.9837004405286344, - "mean_pixel_agreement": 0.9996845794392524 + "mean_mask_iou": 0.9990018379005144, + "min_mask_iou": 0.9978388658367912, + "mean_pixel_agreement": 0.9999263831732967 } } ], "metrics": { - "recall_iou_0.5": 0.9711399711399711, - "recall_iou_0.95": 0.9581529581529581, - "max_abs_score_delta": 0.0292914743799482, - "mean_abs_score_delta": 0.010567531933720255, + "recall_iou_0.5": 0.9841269841269842, + "recall_iou_0.95": 0.9841269841269842, + "max_abs_score_delta": 0.012093111009325308, + "mean_abs_score_delta": 0.0036788824806641583, "extra_cpp_detections": 0.14285714285714285, - "mean_mask_iou": 0.9806729799238418, - "min_mask_iou": 0.9578792675465101, - "mean_pixel_agreement": 0.9996070109595004 + "mean_mask_iou": 0.997841424026485, + "min_mask_iou": 0.9925956416157681, + "mean_pixel_agreement": 0.9999481743683425 } }, { @@ -3637,11 +3637,29 @@ "quant": "f16", "model_path": "models/rfdetr-seg-medium-f16.gguf", "is_seg": true, - "file_size_mb": 71.51, + "file_size_mb": 71.64, "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.01754722237777706, + "mean_abs_score_delta": 0.006765937110207302, + "extra_cpp_detections": 0, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.9919132970112158, + "min_mask_iou": 0.9386901997213191, + "mean_pixel_agreement": 0.999939980793854 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -3649,17 +3667,17 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.005241522182464653, - "mean_abs_score_delta": 0.002944773208618168, + "max_abs_score_delta": 0.0029202672309875943, + "mean_abs_score_delta": 0.0014309054161071755, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9912559585772108, - "min_mask_iou": 0.9908351177730192, - "mean_pixel_agreement": 0.9992450845907637 + "mean_mask_iou": 0.9985398309987279, + "min_mask_iou": 0.996078431372549, + "mean_pixel_agreement": 0.9999864130434781 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -3667,89 +3685,71 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.02302735411834722, - "mean_abs_score_delta": 0.014020747755050689, + "max_abs_score_delta": 0.014486125228881819, + "mean_abs_score_delta": 0.004621918698310856, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9917369601175121, - "min_mask_iou": 0.9851318944844125, - "mean_pixel_agreement": 0.9994734700520833 + "mean_mask_iou": 0.9983336935294467, + "min_mask_iou": 0.9966101694915255, + "mean_pixel_agreement": 0.9998388671875 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.010148575233459467, - "mean_abs_score_delta": 0.005310986113739036, + "max_abs_score_delta": 0.013263480155944807, + "mean_abs_score_delta": 0.003610113354274223, "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.9867932192678348, - "min_mask_iou": 0.975, - "mean_pixel_agreement": 0.9997159679089027 + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9970223111948652, + "min_mask_iou": 0.9924921793534932, + "mean_pixel_agreement": 0.9999772135416667 } }, { - "image": "coco_kitchen.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 9, - "cpp_total": 9, - "matched_iou_0.5": 9, - "matched_iou_0.95": 9, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.06991845059967039, - "mean_abs_score_delta": 0.014521372751447888, + "max_abs_score_delta": 0.0164154604301453, + "mean_abs_score_delta": 0.0044944156860351695, "extra_cpp_detections": 0, - "n_matched_for_mask": 9, - "mean_mask_iou": 0.9846272575689355, - "min_mask_iou": 0.9693251533742331, - "mean_pixel_agreement": 0.9995576372625554 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9982388257988601, + "min_mask_iou": 0.9937262357414449, + "mean_pixel_agreement": 0.999940128504673 } }, { - "image": "coco_living_room.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 11, - "cpp_total": 12, - "matched_iou_0.5": 11, - "matched_iou_0.95": 10, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.9090909090909091, - "max_abs_score_delta": 0.125910272403717, - "mean_abs_score_delta": 0.022870970202359286, + "py_total": 9, + "cpp_total": 9, + "matched_iou_0.5": 8, + "matched_iou_0.95": 8, + "recall_iou_0.5": 0.8888888888888888, + "recall_iou_0.95": 0.8888888888888888, + "max_abs_score_delta": 0.012927866786956788, + "mean_abs_score_delta": 0.003212070951461782, "extra_cpp_detections": 1, - "n_matched_for_mask": 11, - "mean_mask_iou": 0.9544890477339144, - "min_mask_iou": 0.865046102263202, - "mean_pixel_agreement": 0.99945449210414 - } - }, - { - "image": "coco_skater.jpg", - "metrics": { - "py_total": 7, - "cpp_total": 7, - "matched_iou_0.5": 7, - "matched_iou_0.95": 7, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.028052610385894794, - "mean_abs_score_delta": 0.009840370392390671, - "extra_cpp_detections": 0, - "n_matched_for_mask": 7, - "mean_mask_iou": 0.9711719578666218, - "min_mask_iou": 0.9435146443514645, - "mean_pixel_agreement": 0.9998604910714286 + "n_matched_for_mask": 8, + "mean_mask_iou": 0.9964170708877474, + "min_mask_iou": 0.9873969946679593, + "mean_pixel_agreement": 0.9999775870901639 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -3757,25 +3757,25 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.0679214431648254, - "mean_abs_score_delta": 0.022725545015716532, + "max_abs_score_delta": 0.002988286476135249, + "mean_abs_score_delta": 0.0017935114547729559, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9892601065260147, - "min_mask_iou": 0.9841357276633249, - "mean_pixel_agreement": 0.9996853095794392 + "mean_mask_iou": 0.9983859620683424, + "min_mask_iou": 0.9962295007495571, + "mean_pixel_agreement": 0.9999042066758117 } } ], "metrics": { - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.987012987012987, - "max_abs_score_delta": 0.047174318298339846, - "mean_abs_score_delta": 0.013176395062760323, + "recall_iou_0.5": 0.9841269841269842, + "recall_iou_0.95": 0.9841269841269842, + "max_abs_score_delta": 0.01150695838383266, + "mean_abs_score_delta": 0.003704124667309923, "extra_cpp_detections": 0.14285714285714285, - "mean_mask_iou": 0.9813335010940063, - "min_mask_iou": 0.9589983771299508, - "mean_pixel_agreement": 0.9995703503670448 + "mean_mask_iou": 0.9969787130698865, + "min_mask_iou": 0.9858891015854069, + "mean_pixel_agreement": 0.9999377709767353 } }, { @@ -3783,11 +3783,29 @@ "quant": "q8_0", "model_path": "models/rfdetr-seg-medium-q8_0.gguf", "is_seg": true, - "file_size_mb": 42.37, + "file_size_mb": 42.45, "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 12, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.04763472751235964, + "mean_abs_score_delta": 0.011481946599786933, + "extra_cpp_detections": 1, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.9870768246424529, + "min_mask_iou": 0.9285714285714286, + "mean_pixel_agreement": 0.9998849631882204 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -3795,17 +3813,17 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.021134354888916018, - "mean_abs_score_delta": 0.007483364335632325, + "max_abs_score_delta": 0.010106960090637163, + "mean_abs_score_delta": 0.0039010202186584133, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9910054195380505, - "min_mask_iou": 0.9892302878132008, - "mean_pixel_agreement": 0.9991065386374028 + "mean_mask_iou": 0.994942933012911, + "min_mask_iou": 0.9922178988326849, + "mean_pixel_agreement": 0.9998783643892339 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, @@ -3813,89 +3831,71 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.01661575876617427, - "mean_abs_score_delta": 0.010051558687210055, + "max_abs_score_delta": 0.018539125228881792, + "mean_abs_score_delta": 0.009370893916130052, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9913830271592067, - "min_mask_iou": 0.9841954022988506, - "mean_pixel_agreement": 0.9994881184895834 + "mean_mask_iou": 0.9975715257528528, + "min_mask_iou": 0.9956437560503388, + "mean_pixel_agreement": 0.9997843424479167 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 5, + "py_total": 7, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.01599657523345943, - "mean_abs_score_delta": 0.008085986113739008, + "max_abs_score_delta": 0.007094513458251961, + "mean_abs_score_delta": 0.002241441280909941, "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.986229814183804, - "min_mask_iou": 0.9766355140186916, - "mean_pixel_agreement": 0.9997120859213251 + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9885024138022064, + "min_mask_iou": 0.959051724137931, + "mean_pixel_agreement": 0.999960472470238 } }, { - "image": "coco_kitchen.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 9, - "cpp_total": 10, - "matched_iou_0.5": 9, - "matched_iou_0.95": 9, + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.04722745059967037, - "mean_abs_score_delta": 0.016534934983571368, - "extra_cpp_detections": 1, - "n_matched_for_mask": 9, - "mean_mask_iou": 0.9843766576103664, - "min_mask_iou": 0.9723076923076923, - "mean_pixel_agreement": 0.9996235037730938 - } - }, - { - "image": "coco_living_room.jpg", - "metrics": { - "py_total": 11, - "cpp_total": 11, - "matched_iou_0.5": 10, - "matched_iou_0.95": 9, - "recall_iou_0.5": 0.9090909090909091, - "recall_iou_0.95": 0.8181818181818182, - "max_abs_score_delta": 0.03552336591720584, - "mean_abs_score_delta": 0.010351119756698602, - "extra_cpp_detections": 1, - "n_matched_for_mask": 10, - "mean_mask_iou": 0.9576396844745162, - "min_mask_iou": 0.8658026755852842, - "mean_pixel_agreement": 0.9995114436619719 + "max_abs_score_delta": 0.014902460430145315, + "mean_abs_score_delta": 0.005235947030639676, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9950845895335677, + "min_mask_iou": 0.9911452184179457, + "mean_pixel_agreement": 0.9998736857476634 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 7, + "py_total": 9, "cpp_total": 8, - "matched_iou_0.5": 7, - "matched_iou_0.95": 6, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.8571428571428571, - "max_abs_score_delta": 0.03529161038589479, - "mean_abs_score_delta": 0.011790245065416625, - "extra_cpp_detections": 1, - "n_matched_for_mask": 7, - "mean_mask_iou": 0.9681352695931122, - "min_mask_iou": 0.941544885177453, - "mean_pixel_agreement": 0.9998516555059523 + "matched_iou_0.5": 8, + "matched_iou_0.95": 8, + "recall_iou_0.5": 0.8888888888888888, + "recall_iou_0.95": 0.8888888888888888, + "max_abs_score_delta": 0.011220648647308318, + "mean_abs_score_delta": 0.00627778382205961, + "extra_cpp_detections": 0, + "n_matched_for_mask": 8, + "mean_mask_iou": 0.9945325927213845, + "min_mask_iou": 0.9831460674157303, + "mean_pixel_agreement": 0.9999634074941453 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -3903,25 +3903,25 @@ "matched_iou_0.95": 5, "recall_iou_0.5": 1.0, "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.07614944316482541, - "mean_abs_score_delta": 0.017132345015716543, + "max_abs_score_delta": 0.00809628647613525, + "mean_abs_score_delta": 0.002900078080749502, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9885473526191916, - "min_mask_iou": 0.9834747163159634, - "mean_pixel_agreement": 0.9996889602803739 + "mean_mask_iou": 0.9970962492634066, + "min_mask_iou": 0.9955129864526706, + "mean_pixel_agreement": 0.9997702331961591 } } ], "metrics": { - "recall_iou_0.5": 0.987012987012987, - "recall_iou_0.95": 0.9536178107606679, - "max_abs_score_delta": 0.035419794136592304, - "mean_abs_score_delta": 0.011632793422569218, - "extra_cpp_detections": 0.42857142857142855, - "mean_mask_iou": 0.9810453178826067, - "min_mask_iou": 0.9590273105024479, - "mean_pixel_agreement": 0.9995689008956719 + "recall_iou_0.5": 0.9841269841269842, + "recall_iou_0.95": 0.9841269841269842, + "max_abs_score_delta": 0.016799245977674206, + "mean_abs_score_delta": 0.005915587278419162, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9935438755326832, + "min_mask_iou": 0.9778984399826757, + "mean_pixel_agreement": 0.9998736384190823 } }, { @@ -3929,11 +3929,29 @@ "quant": "q4_K", "model_path": "models/rfdetr-seg-medium-q4_K.gguf", "is_seg": true, - "file_size_mb": 33.58, + "file_size_mb": 33.63, "n_images": 7, "per_image": [ { - "image": "bus.jpg", + "image": "000000000139.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 9, + "matched_iou_0.5": 9, + "matched_iou_0.95": 2, + "recall_iou_0.5": 0.8181818181818182, + "recall_iou_0.95": 0.18181818181818182, + "max_abs_score_delta": 0.0667096461181641, + "mean_abs_score_delta": 0.02681066267479791, + "extra_cpp_detections": 0, + "n_matched_for_mask": 9, + "mean_mask_iou": 0.9279164287129424, + "min_mask_iou": 0.8373644703919934, + "mean_pixel_agreement": 0.9993401962702139 + } + }, + { + "image": "000000000632.jpg", "metrics": { "py_total": 5, "cpp_total": 5, @@ -3941,133 +3959,1867 @@ "matched_iou_0.95": 4, "recall_iou_0.5": 1.0, "recall_iou_0.95": 0.8, - "max_abs_score_delta": 0.09587335488891602, - "mean_abs_score_delta": 0.028515803025817864, + "max_abs_score_delta": 0.04054673276901244, + "mean_abs_score_delta": 0.030088863301849365, "extra_cpp_detections": 0, "n_matched_for_mask": 5, - "mean_mask_iou": 0.9728834220492517, - "min_mask_iou": 0.9606650098551718, - "mean_pixel_agreement": 0.9982233653406493 + "mean_mask_iou": 0.9715401508896981, + "min_mask_iou": 0.9473489061920652, + "mean_pixel_agreement": 0.999141433747412 } }, { - "image": "coco_cats.jpg", + "image": "000000039769.jpg", "metrics": { "py_total": 4, "cpp_total": 4, "matched_iou_0.5": 4, - "matched_iou_0.95": 4, + "matched_iou_0.95": 3, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 1.0, - "max_abs_score_delta": 0.021757758766174362, - "mean_abs_score_delta": 0.01200699775505068, + "recall_iou_0.95": 0.75, + "max_abs_score_delta": 0.025881125228881863, + "mean_abs_score_delta": 0.01591291869831088, "extra_cpp_detections": 0, "n_matched_for_mask": 4, - "mean_mask_iou": 0.9842316736646974, - "min_mask_iou": 0.975397973950796, - "mean_pixel_agreement": 0.9986800130208333 + "mean_mask_iou": 0.9853194335729063, + "min_mask_iou": 0.9821580128628279, + "mean_pixel_agreement": 0.9985709635416666 } }, { - "image": "coco_indoor.jpg", + "image": "000000087038.jpg", "metrics": { - "py_total": 5, - "cpp_total": 5, - "matched_iou_0.5": 5, - "matched_iou_0.95": 4, + "py_total": 7, + "cpp_total": 9, + "matched_iou_0.5": 7, + "matched_iou_0.95": 2, "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.8, - "max_abs_score_delta": 0.03601946187591554, - "mean_abs_score_delta": 0.02038698488082886, - "extra_cpp_detections": 0, - "n_matched_for_mask": 5, - "mean_mask_iou": 0.9769157573197915, - "min_mask_iou": 0.956140350877193, - "mean_pixel_agreement": 0.9994429347826086 - } - }, - { - "image": "coco_kitchen.jpg", - "metrics": { - "py_total": 9, - "cpp_total": 11, - "matched_iou_0.5": 8, - "matched_iou_0.95": 4, - "recall_iou_0.5": 0.8888888888888888, - "recall_iou_0.95": 0.4444444444444444, - "max_abs_score_delta": 0.07857945059967042, - "mean_abs_score_delta": 0.02327204699420929, - "extra_cpp_detections": 3, - "n_matched_for_mask": 8, - "mean_mask_iou": 0.9672950643280299, - "min_mask_iou": 0.9189740482682015, - "mean_pixel_agreement": 0.9979645418618267 + "recall_iou_0.95": 0.2857142857142857, + "max_abs_score_delta": 0.06277910982894896, + "mean_abs_score_delta": 0.0193831097215925, + "extra_cpp_detections": 2, + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9482201968880479, + "min_mask_iou": 0.8501070663811563, + "mean_pixel_agreement": 0.999833519345238 } }, { - "image": "coco_living_room.jpg", + "image": "000000252219.jpg", "metrics": { - "py_total": 11, - "cpp_total": 10, - "matched_iou_0.5": 10, - "matched_iou_0.95": 5, - "recall_iou_0.5": 0.9090909090909091, - "recall_iou_0.95": 0.45454545454545453, - "max_abs_score_delta": 0.08698436591720582, - "mean_abs_score_delta": 0.03684026833381654, + "py_total": 5, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 2, + "recall_iou_0.5": 0.8, + "recall_iou_0.95": 0.4, + "max_abs_score_delta": 0.1336845395698547, + "mean_abs_score_delta": 0.04348552330207825, "extra_cpp_detections": 0, - "n_matched_for_mask": 10, - "mean_mask_iou": 0.943632628180579, - "min_mask_iou": 0.8773006134969326, - "mean_pixel_agreement": 0.9994208480046949 + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9821271339514375, + "min_mask_iou": 0.9754319048687458, + "mean_pixel_agreement": 0.999467910338785 } }, { - "image": "coco_skater.jpg", + "image": "000000397133.jpg", "metrics": { - "py_total": 7, - "cpp_total": 9, + "py_total": 9, + "cpp_total": 11, "matched_iou_0.5": 7, "matched_iou_0.95": 3, - "recall_iou_0.5": 1.0, - "recall_iou_0.95": 0.42857142857142855, - "max_abs_score_delta": 0.05008616302108759, - "mean_abs_score_delta": 0.023064196369171137, - "extra_cpp_detections": 2, + "recall_iou_0.5": 0.7777777777777778, + "recall_iou_0.95": 0.3333333333333333, + "max_abs_score_delta": 0.050036191661834706, + "mean_abs_score_delta": 0.021410721990857806, + "extra_cpp_detections": 4, "n_matched_for_mask": 7, - "mean_mask_iou": 0.9435055468859376, - "min_mask_iou": 0.8273684210526315, - "mean_pixel_agreement": 0.9997772507440477 + "mean_mask_iou": 0.9815481949204694, + "min_mask_iou": 0.9714424007744433, + "mean_pixel_agreement": 0.9997344429575109 } }, { - "image": "coco_street.jpg", + "image": "bus.jpg", "metrics": { "py_total": 5, - "cpp_total": 4, - "matched_iou_0.5": 4, + "cpp_total": 5, + "matched_iou_0.5": 5, "matched_iou_0.95": 3, - "recall_iou_0.5": 0.8, + "recall_iou_0.5": 1.0, "recall_iou_0.95": 0.6, - "max_abs_score_delta": 0.06600796922302243, - "mean_abs_score_delta": 0.022316450670242294, + "max_abs_score_delta": 0.05309664514541623, + "mean_abs_score_delta": 0.016317888545227043, "extra_cpp_detections": 0, - "n_matched_for_mask": 4, - "mean_mask_iou": 0.9718905886399132, - "min_mask_iou": 0.94446859378396, - "mean_pixel_agreement": 0.99912109375 + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9845657033166184, + "min_mask_iou": 0.9748050389922016, + "mean_pixel_agreement": 0.9990420667581162 } } ], "metrics": { - "recall_iou_0.5": 0.9425685425685426, - "recall_iou_0.95": 0.646794475365904, - "max_abs_score_delta": 0.06218693204171317, - "mean_abs_score_delta": 0.023771821147019524, - "extra_cpp_detections": 0.7142857142857143, - "mean_mask_iou": 0.9657649544383143, - "min_mask_iou": 0.9229021444692694, - "mean_pixel_agreement": 0.9989471496435229 + "recall_iou_0.5": 0.9137085137085137, + "recall_iou_0.95": 0.47869511440940016, + "max_abs_score_delta": 0.06181914147458757, + "mean_abs_score_delta": 0.02477281260495911, + "extra_cpp_detections": 0.8571428571428571, + "mean_mask_iou": 0.9687481774645885, + "min_mask_iou": 0.9340939714947761, + "mean_pixel_agreement": 0.9993043618512776 + } + }, + { + "variant": "seg-large", + "quant": "f32", + "model_path": "models/rfdetr-seg-large-f32.gguf", + "is_seg": true, + "file_size_mb": 134.61, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 12, + "cpp_total": 13, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.010671753185272248, + "mean_abs_score_delta": 0.004764438083012894, + "extra_cpp_detections": 1, + "n_matched_for_mask": 12, + "mean_mask_iou": 0.994810081278266, + "min_mask_iou": 0.9803921568627451, + "mean_pixel_agreement": 0.9999685177034429 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0018450536384582161, + "mean_abs_score_delta": 0.001206573126792898, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9994168166065013, + "min_mask_iou": 0.9986974320803871, + "mean_pixel_agreement": 0.9999870600414078 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.006554278240203826, + "mean_abs_score_delta": 0.0032312408199310083, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.999084777844873, + "min_mask_iou": 0.9981017391137614, + "mean_pixel_agreement": 0.9998665364583332 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 6, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 0.8333333333333334, + "recall_iou_0.95": 0.8333333333333334, + "max_abs_score_delta": 0.00879692457580561, + "mean_abs_score_delta": 0.0024596051994323796, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9979296895016629, + "min_mask_iou": 0.9970703125, + "mean_pixel_agreement": 0.9999908854166668 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.010190617282867409, + "mean_abs_score_delta": 0.003667494184875486, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9985830336252077, + "min_mask_iou": 0.9956427015250545, + "mean_pixel_agreement": 0.9999700642523365 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 6, + "cpp_total": 6, + "matched_iou_0.5": 6, + "matched_iou_0.95": 6, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.00526242850494385, + "mean_abs_score_delta": 0.002575720675786338, + "extra_cpp_detections": 0, + "n_matched_for_mask": 6, + "mean_mask_iou": 0.9992604904789988, + "min_mask_iou": 0.9979602243753187, + "mean_pixel_agreement": 0.9999841432474629 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.019447192916870137, + "mean_abs_score_delta": 0.006678898705291769, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9981127708472748, + "min_mask_iou": 0.9956822429906542, + "mean_pixel_agreement": 0.9997075903063559 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9761904761904763, + "recall_iou_0.95": 0.9761904761904763, + "max_abs_score_delta": 0.008966892620631614, + "mean_abs_score_delta": 0.003511995827874682, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9981710943118264, + "min_mask_iou": 0.994792401349703, + "mean_pixel_agreement": 0.9999249710608579 + } + }, + { + "variant": "seg-large", + "quant": "f16", + "model_path": "models/rfdetr-seg-large-f16.gguf", + "is_seg": true, + "file_size_mb": 72.33, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 12, + "cpp_total": 13, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.01000251945114139, + "mean_abs_score_delta": 0.0033397661819458135, + "extra_cpp_detections": 1, + "n_matched_for_mask": 12, + "mean_mask_iou": 0.9945772462054263, + "min_mask_iou": 0.9769959404600812, + "mean_pixel_agreement": 0.9999718798904539 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0014941952857970842, + "mean_abs_score_delta": 0.0007947444906234724, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9993277035362492, + "min_mask_iou": 0.9985113509490138, + "mean_pixel_agreement": 0.9999838250517599 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.01552427824020386, + "mean_abs_score_delta": 0.006645740819931051, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9989258559678664, + "min_mask_iou": 0.998291015625, + "mean_pixel_agreement": 0.9998787434895833 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 6, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 0.8333333333333334, + "recall_iou_0.95": 0.8333333333333334, + "max_abs_score_delta": 0.008329924575805614, + "mean_abs_score_delta": 0.0023518245079040324, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9986426647102922, + "min_mask_iou": 0.9978768577494692, + "mean_pixel_agreement": 0.9999915364583334 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.014006156780242973, + "mean_abs_score_delta": 0.003937694184875507, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9977568154785651, + "min_mask_iou": 0.9922456575682382, + "mean_pixel_agreement": 0.9999591121495326 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 6, + "cpp_total": 6, + "matched_iou_0.5": 6, + "matched_iou_0.95": 6, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0036011567153930724, + "mean_abs_score_delta": 0.002152851057052596, + "extra_cpp_detections": 0, + "n_matched_for_mask": 6, + "mean_mask_iou": 0.999006450610289, + "min_mask_iou": 0.9979602243753187, + "mean_pixel_agreement": 0.9999774346213895 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.015014192916870117, + "mean_abs_score_delta": 0.0058857039665222155, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9979030905825408, + "min_mask_iou": 0.9954606929635137, + "mean_pixel_agreement": 0.999689529035208 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9761904761904763, + "recall_iou_0.95": 0.9761904761904763, + "max_abs_score_delta": 0.009710346280779159, + "mean_abs_score_delta": 0.0035869036012649556, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9980199752987471, + "min_mask_iou": 0.993905962812948, + "mean_pixel_agreement": 0.9999217229566087 + } + }, + { + "variant": "seg-large", + "quant": "q8_0", + "model_path": "models/rfdetr-seg-large-q8_0.gguf", + "is_seg": true, + "file_size_mb": 43.14, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 0.9166666666666666, + "recall_iou_0.95": 0.9166666666666666, + "max_abs_score_delta": 0.012652753185272259, + "mean_abs_score_delta": 0.00565856759955668, + "extra_cpp_detections": 1, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.9895185509501492, + "min_mask_iou": 0.969553450608931, + "mean_pixel_agreement": 0.9999319782330347 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.01506094636154176, + "mean_abs_score_delta": 0.005019478690147405, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9960555146860129, + "min_mask_iou": 0.9936356404136834, + "mean_pixel_agreement": 0.9998778791407867 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.026712278240203835, + "mean_abs_score_delta": 0.012790597688674937, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9980748407393659, + "min_mask_iou": 0.99755859375, + "mean_pixel_agreement": 0.99981689453125 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 6, + "cpp_total": 6, + "matched_iou_0.5": 6, + "matched_iou_0.95": 6, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.02110534843826295, + "mean_abs_score_delta": 0.005726240768432626, + "extra_cpp_detections": 0, + "n_matched_for_mask": 6, + "mean_mask_iou": 0.9884975358325724, + "min_mask_iou": 0.9724612736660929, + "mean_pixel_agreement": 0.9999593098958334 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.03367438271713252, + "mean_abs_score_delta": 0.011875815560913083, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9953385931652043, + "min_mask_iou": 0.9900439916647372, + "mean_pixel_agreement": 0.9998802570093458 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 6, + "cpp_total": 7, + "matched_iou_0.5": 6, + "matched_iou_0.95": 6, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.009789775356292774, + "mean_abs_score_delta": 0.004727143878300995, + "extra_cpp_detections": 1, + "n_matched_for_mask": 6, + "mean_mask_iou": 0.9957495912047688, + "min_mask_iou": 0.9921773142112125, + "mean_pixel_agreement": 0.9999438914910227 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.02189079328918453, + "mean_abs_score_delta": 0.007836490515899653, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9965959458308078, + "min_mask_iou": 0.994998244998245, + "mean_pixel_agreement": 0.9997087334247828 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.988095238095238, + "recall_iou_0.95": 0.988095238095238, + "max_abs_score_delta": 0.020126611083984374, + "mean_abs_score_delta": 0.007662047814560769, + "extra_cpp_detections": 0.2857142857142857, + "mean_mask_iou": 0.9942615103441258, + "min_mask_iou": 0.9872040727589859, + "mean_pixel_agreement": 0.9998741348180081 + } + }, + { + "variant": "seg-large", + "quant": "q4_K", + "model_path": "models/rfdetr-seg-large-q4_K.gguf", + "is_seg": true, + "file_size_mb": 34.32, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 12, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 6, + "recall_iou_0.5": 0.9166666666666666, + "recall_iou_0.95": 0.5, + "max_abs_score_delta": 0.07481646859359736, + "mean_abs_score_delta": 0.03402370430166071, + "extra_cpp_detections": 0, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.9494318710150084, + "min_mask_iou": 0.9125194401244168, + "mean_pixel_agreement": 0.999695235808792 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 3, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.75, + "max_abs_score_delta": 0.03007156005096434, + "mean_abs_score_delta": 0.020798524516105643, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9747196607097163, + "min_mask_iou": 0.9620571839597195, + "mean_pixel_agreement": 0.9994257893374742 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.03914932177734376, + "mean_abs_score_delta": 0.020693041431427, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.984449673772342, + "min_mask_iou": 0.976660850640701, + "mean_pixel_agreement": 0.99852294921875 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 6, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 3, + "recall_iou_0.5": 0.8333333333333334, + "recall_iou_0.95": 0.5, + "max_abs_score_delta": 0.03428203508758543, + "mean_abs_score_delta": 0.024951388085174563, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.955900725796346, + "min_mask_iou": 0.9198174706649282, + "mean_pixel_agreement": 0.9997298177083334 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.09325415678024296, + "mean_abs_score_delta": 0.04497489904174803, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9815714568961678, + "min_mask_iou": 0.9729318734793188, + "mean_pixel_agreement": 0.9995078855140187 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 6, + "cpp_total": 8, + "matched_iou_0.5": 6, + "matched_iou_0.95": 3, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.5, + "max_abs_score_delta": 0.026168156715393076, + "mean_abs_score_delta": 0.017547240518569946, + "extra_cpp_detections": 2, + "n_matched_for_mask": 6, + "mean_mask_iou": 0.9792172126684294, + "min_mask_iou": 0.9491054904380012, + "mean_pixel_agreement": 0.9996377341920374 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 3, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.6, + "max_abs_score_delta": 0.03696019291687014, + "mean_abs_score_delta": 0.02169660452499391, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9853772037724017, + "min_mask_iou": 0.9794126821442894, + "mean_pixel_agreement": 0.9988781435756744 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9642857142857143, + "recall_iou_0.95": 0.6928571428571428, + "max_abs_score_delta": 0.04781455598885672, + "mean_abs_score_delta": 0.026383628917097116, + "extra_cpp_detections": 0.2857142857142857, + "mean_mask_iou": 0.9729525435186303, + "min_mask_iou": 0.9532149987787678, + "mean_pixel_agreement": 0.9993425079078687 + } + }, + { + "variant": "seg-xlarge", + "quant": "f32", + "model_path": "models/rfdetr-seg-xlarge-f32.gguf", + "is_seg": true, + "file_size_mb": 141.86, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0591764862136841, + "mean_abs_score_delta": 0.008690917694438608, + "extra_cpp_detections": 0, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.9955727118480767, + "min_mask_iou": 0.9860982391102873, + "mean_pixel_agreement": 0.9999683231967563 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.003982409538269005, + "mean_abs_score_delta": 0.0013062952774047566, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9963068834445112, + "min_mask_iou": 0.9883268482490273, + "mean_pixel_agreement": 0.9999754140786749 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.011618249580383266, + "mean_abs_score_delta": 0.004346427088737476, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9994461694341219, + "min_mask_iou": 0.9989659530466087, + "mean_pixel_agreement": 0.9999308268229167 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 10, + "cpp_total": 10, + "matched_iou_0.5": 10, + "matched_iou_0.95": 10, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.03956291053009031, + "mean_abs_score_delta": 0.010076413242340099, + "extra_cpp_detections": 0, + "n_matched_for_mask": 10, + "mean_mask_iou": 0.9939847595450699, + "min_mask_iou": 0.9746835443037974, + "mean_pixel_agreement": 0.9999889322916667 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0265456958732605, + "mean_abs_score_delta": 0.012590407726287856, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9964081539825749, + "min_mask_iou": 0.9910659272951324, + "mean_pixel_agreement": 0.9999255257009345 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 8, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, + "recall_iou_0.5": 0.875, + "recall_iou_0.95": 0.875, + "max_abs_score_delta": 0.03655369786834717, + "mean_abs_score_delta": 0.009075142377580907, + "extra_cpp_detections": 0, + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9961964411521621, + "min_mask_iou": 0.9938775510204082, + "mean_pixel_agreement": 0.9999665439946471 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.004612512840270977, + "mean_abs_score_delta": 0.002618254873657211, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9965904118134983, + "min_mask_iou": 0.9934207499424891, + "mean_pixel_agreement": 0.9996168267032466 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9821428571428571, + "recall_iou_0.95": 0.9821428571428571, + "max_abs_score_delta": 0.026007423206329334, + "mean_abs_score_delta": 0.006957694040063844, + "extra_cpp_detections": 0.0, + "mean_mask_iou": 0.9963579330314307, + "min_mask_iou": 0.9894912589953929, + "mean_pixel_agreement": 0.9999103418269776 + } + }, + { + "variant": "seg-xlarge", + "quant": "f16", + "model_path": "models/rfdetr-seg-xlarge-f16.gguf", + "is_seg": true, + "file_size_mb": 76.68, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 12, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.06210348621368411, + "mean_abs_score_delta": 0.010607721963015475, + "extra_cpp_detections": 1, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.995272624957786, + "min_mask_iou": 0.987152034261242, + "mean_pixel_agreement": 0.9999673228766537 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.004049736690521222, + "mean_abs_score_delta": 0.0016258952774047763, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9965445528023611, + "min_mask_iou": 0.9883268482490273, + "mean_pixel_agreement": 0.9999741200828158 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.01743524958038334, + "mean_abs_score_delta": 0.0062921874628066865, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9994201243155507, + "min_mask_iou": 0.9990416866315285, + "mean_pixel_agreement": 0.9999324544270833 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 10, + "cpp_total": 10, + "matched_iou_0.5": 10, + "matched_iou_0.95": 10, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.03174691053009038, + "mean_abs_score_delta": 0.008536949041748055, + "extra_cpp_detections": 0, + "n_matched_for_mask": 10, + "mean_mask_iou": 0.9942140049510375, + "min_mask_iou": 0.9746835443037974, + "mean_pixel_agreement": 0.99998828125 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.02163499081802367, + "mean_abs_score_delta": 0.010747604053497329, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9970537110842596, + "min_mask_iou": 0.994413407821229, + "mean_pixel_agreement": 0.9999320969626169 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 8, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, + "recall_iou_0.5": 0.875, + "recall_iou_0.95": 0.875, + "max_abs_score_delta": 0.04050169786834712, + "mean_abs_score_delta": 0.009265570949009498, + "extra_cpp_detections": 0, + "n_matched_for_mask": 7, + "mean_mask_iou": 0.996637081126873, + "min_mask_iou": 0.9938775510204082, + "mean_pixel_agreement": 0.9999728169956507 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0038298967208861834, + "mean_abs_score_delta": 0.002800810559082012, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9966472231318925, + "min_mask_iou": 0.9924995398490705, + "mean_pixel_agreement": 0.9996886145404664 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9821428571428571, + "recall_iou_0.95": 0.9821428571428571, + "max_abs_score_delta": 0.02590028120313372, + "mean_abs_score_delta": 0.0071252484723662605, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9965413317671086, + "min_mask_iou": 0.9899992303051862, + "mean_pixel_agreement": 0.9999222438764696 + } + }, + { + "variant": "seg-xlarge", + "quant": "q8_0", + "model_path": "models/rfdetr-seg-xlarge-q8_0.gguf", + "is_seg": true, + "file_size_mb": 46.12, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 12, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.02599907893371578, + "mean_abs_score_delta": 0.00775516107628563, + "extra_cpp_detections": 1, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.99258395185341, + "min_mask_iou": 0.984081041968162, + "mean_pixel_agreement": 0.9999473164746054 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.00947773669052121, + "mean_abs_score_delta": 0.003887419557189942, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9937422162776925, + "min_mask_iou": 0.9883040935672515, + "mean_pixel_agreement": 0.9998803053830227 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.024322249580383315, + "mean_abs_score_delta": 0.007273447701454161, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9970784068262456, + "min_mask_iou": 0.9964849164961384, + "mean_pixel_agreement": 0.9997037760416667 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 10, + "cpp_total": 10, + "matched_iou_0.5": 10, + "matched_iou_0.95": 10, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.03569069569015504, + "mean_abs_score_delta": 0.007392627688598619, + "extra_cpp_detections": 0, + "n_matched_for_mask": 10, + "mean_mask_iou": 0.9919650177272334, + "min_mask_iou": 0.9849785407725322, + "mean_pixel_agreement": 0.9999762369791668 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.02184869587326055, + "mean_abs_score_delta": 0.011459553461456306, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9962661410682576, + "min_mask_iou": 0.9946798403952118, + "mean_pixel_agreement": 0.9998919392523364 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 8, + "cpp_total": 8, + "matched_iou_0.5": 8, + "matched_iou_0.95": 8, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.03995869786834716, + "mean_abs_score_delta": 0.011963547512531272, + "extra_cpp_detections": 0, + "n_matched_for_mask": 8, + "mean_mask_iou": 0.9943730439980811, + "min_mask_iou": 0.983739837398374, + "mean_pixel_agreement": 0.9999574612119437 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.009555813823699966, + "mean_abs_score_delta": 0.003309308950805656, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9927259440574069, + "min_mask_iou": 0.9829841812740487, + "mean_pixel_agreement": 0.9995731595793325 + } + } + ], + "metrics": { + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.023836138351440432, + "mean_abs_score_delta": 0.007577295135474511, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9941049602583324, + "min_mask_iou": 0.9878932074102453, + "mean_pixel_agreement": 0.9998471707031535 + } + }, + { + "variant": "seg-xlarge", + "quant": "q4_K", + "model_path": "models/rfdetr-seg-xlarge-q4_K.gguf", + "is_seg": true, + "file_size_mb": 36.58, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 10, + "matched_iou_0.5": 10, + "matched_iou_0.95": 8, + "recall_iou_0.5": 0.9090909090909091, + "recall_iou_0.95": 0.7272727272727273, + "max_abs_score_delta": 0.09629242478561406, + "mean_abs_score_delta": 0.03484866253700255, + "extra_cpp_detections": 0, + "n_matched_for_mask": 10, + "mean_mask_iou": 0.9608574262614742, + "min_mask_iou": 0.9214285714285714, + "mean_pixel_agreement": 0.9997179430751174 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.8, + "max_abs_score_delta": 0.03345228878784179, + "mean_abs_score_delta": 0.019227219557189958, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9740074772011063, + "min_mask_iou": 0.9533582089552238, + "mean_pixel_agreement": 0.9994028209109731 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.030868389461517376, + "mean_abs_score_delta": 0.0217809970293045, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9834532962948783, + "min_mask_iou": 0.9753199268738574, + "mean_pixel_agreement": 0.99861328125 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 10, + "cpp_total": 11, + "matched_iou_0.5": 10, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.4, + "max_abs_score_delta": 0.055913567726135205, + "mean_abs_score_delta": 0.025567030413055414, + "extra_cpp_detections": 1, + "n_matched_for_mask": 10, + "mean_mask_iou": 0.9487072064571453, + "min_mask_iou": 0.8772277227722772, + "mean_pixel_agreement": 0.9998590494791667 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 0.8, + "recall_iou_0.95": 0.8, + "max_abs_score_delta": 0.06244499081802368, + "mean_abs_score_delta": 0.02859697755050658, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9793988717311073, + "min_mask_iou": 0.9712990936555891, + "mean_pixel_agreement": 0.9994533075350468 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 8, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 3, + "recall_iou_0.5": 0.625, + "recall_iou_0.95": 0.375, + "max_abs_score_delta": 0.04575674557495113, + "mean_abs_score_delta": 0.029063071152496355, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9764036592090362, + "min_mask_iou": 0.9619643716899374, + "mean_pixel_agreement": 0.9995367388758784 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 3, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.6, + "max_abs_score_delta": 0.052879754020690894, + "mean_abs_score_delta": 0.026644005422973627, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9852103635164763, + "min_mask_iou": 0.9671013381061963, + "mean_pixel_agreement": 0.999232053040695 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9048701298701298, + "recall_iou_0.95": 0.6717532467532468, + "max_abs_score_delta": 0.05394402302496774, + "mean_abs_score_delta": 0.02653256623750414, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9725769000958892, + "min_mask_iou": 0.9468141762116646, + "mean_pixel_agreement": 0.9994021705952683 + } + }, + { + "variant": "seg-2xlarge", + "quant": "f32", + "model_path": "models/rfdetr-seg-2xlarge-f32.gguf", + "is_seg": true, + "file_size_mb": 143.9, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 12, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.0763961819992065, + "mean_abs_score_delta": 0.013478176062266025, + "extra_cpp_detections": 0, + "n_matched_for_mask": 12, + "mean_mask_iou": 0.9930661240095212, + "min_mask_iou": 0.9772816294555425, + "mean_pixel_agreement": 0.9999523180751174 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.012224536151886012, + "mean_abs_score_delta": 0.004080874269866963, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9974077828242682, + "min_mask_iou": 0.9905123339658444, + "mean_pixel_agreement": 0.9999333592132504 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.030980971382141065, + "mean_abs_score_delta": 0.014824631770133934, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9973572930835937, + "min_mask_iou": 0.9946051499304958, + "mean_pixel_agreement": 0.9995719401041666 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.025835730998992945, + "mean_abs_score_delta": 0.007656676904504959, + "extra_cpp_detections": 0, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.9940019416958618, + "min_mask_iou": 0.9831578947368421, + "mean_pixel_agreement": 0.9999878669507575 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.013989681720733671, + "mean_abs_score_delta": 0.0054236971580505335, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9988204016813842, + "min_mask_iou": 0.9970579861440638, + "mean_pixel_agreement": 0.9999664135514019 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 8, + "cpp_total": 9, + "matched_iou_0.5": 8, + "matched_iou_0.95": 8, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.06730837743377682, + "mean_abs_score_delta": 0.013266279243469223, + "extra_cpp_detections": 1, + "n_matched_for_mask": 8, + "mean_mask_iou": 0.9973804274133177, + "min_mask_iou": 0.9934959349593496, + "mean_pixel_agreement": 0.9999844481850118 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.009904908538818313, + "mean_abs_score_delta": 0.004422799575805647, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.997114723732205, + "min_mask_iou": 0.9931982105359263, + "mean_pixel_agreement": 0.9998010973936899 + } + } + ], + "metrics": { + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.033805769746507904, + "mean_abs_score_delta": 0.009021876426299614, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9964498134914502, + "min_mask_iou": 0.9899013056754378, + "mean_pixel_agreement": 0.999885349067628 + } + }, + { + "variant": "seg-2xlarge", + "quant": "f16", + "model_path": "models/rfdetr-seg-2xlarge-f16.gguf", + "is_seg": true, + "file_size_mb": 78.72, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 12, + "cpp_total": 12, + "matched_iou_0.5": 12, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.9166666666666666, + "max_abs_score_delta": 0.06134518199920658, + "mean_abs_score_delta": 0.012974814791361497, + "extra_cpp_detections": 0, + "n_matched_for_mask": 12, + "mean_mask_iou": 0.9911617937485286, + "min_mask_iou": 0.9746835443037974, + "mean_pixel_agreement": 0.9999507898082941 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.00470430918121334, + "mean_abs_score_delta": 0.0030878598091125475, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9974408847192354, + "min_mask_iou": 0.9905482041587902, + "mean_pixel_agreement": 0.9999353002070392 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.018781100952148466, + "mean_abs_score_delta": 0.01156538177013397, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9979037367410758, + "min_mask_iou": 0.994374493290756, + "mean_pixel_agreement": 0.9996826171875 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.027540053432464573, + "mean_abs_score_delta": 0.008854206547477029, + "extra_cpp_detections": 0, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.993699226000356, + "min_mask_iou": 0.9769874476987448, + "mean_pixel_agreement": 0.9999881628787878 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.013268681720733588, + "mean_abs_score_delta": 0.005933497158050538, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9991575572817706, + "min_mask_iou": 0.9986717267552182, + "mean_pixel_agreement": 0.9999780957943925 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 8, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, + "recall_iou_0.5": 0.875, + "recall_iou_0.95": 0.875, + "max_abs_score_delta": 0.06723837743377681, + "mean_abs_score_delta": 0.01338319512231009, + "extra_cpp_detections": 0, + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9979551994414196, + "min_mask_iou": 0.9947916666666666, + "mean_pixel_agreement": 0.999981180996989 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.00478052454376221, + "mean_abs_score_delta": 0.0018006241119384648, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9976160422225423, + "min_mask_iou": 0.9954857047316502, + "mean_pixel_agreement": 0.9998063557384544 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9821428571428571, + "recall_iou_0.95": 0.9702380952380951, + "max_abs_score_delta": 0.02823688989475794, + "mean_abs_score_delta": 0.008228511330054877, + "extra_cpp_detections": 0.0, + "mean_mask_iou": 0.9964192057364184, + "min_mask_iou": 0.989363255372232, + "mean_pixel_agreement": 0.9999032146587795 + } + }, + { + "variant": "seg-2xlarge", + "quant": "q8_0", + "model_path": "models/rfdetr-seg-2xlarge-q8_0.gguf", + "is_seg": true, + "file_size_mb": 48.16, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 12, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 0.9166666666666666, + "recall_iou_0.95": 0.9166666666666666, + "max_abs_score_delta": 0.044343181999206505, + "mean_abs_score_delta": 0.01562627122428201, + "extra_cpp_detections": 0, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.986883087788717, + "min_mask_iou": 0.9565048543689321, + "mean_pixel_agreement": 0.9999249759923174 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.006088917846679642, + "mean_abs_score_delta": 0.0026810742698669455, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9949995304925725, + "min_mask_iou": 0.9887005649717514, + "mean_pixel_agreement": 0.9998815993788821 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.03301110095214843, + "mean_abs_score_delta": 0.01067289607906341, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9951261603512681, + "min_mask_iou": 0.9929643527204502, + "mean_pixel_agreement": 0.9995768229166666 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 12, + "matched_iou_0.5": 11, + "matched_iou_0.95": 11, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.022356709072113068, + "mean_abs_score_delta": 0.009393263521714636, + "extra_cpp_detections": 1, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.98612463726645, + "min_mask_iou": 0.9486652977412731, + "mean_pixel_agreement": 0.9999730705492424 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.010785681720733686, + "mean_abs_score_delta": 0.005634803777313247, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9964182167485627, + "min_mask_iou": 0.9923006205470007, + "mean_pixel_agreement": 0.9998999707943925 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 8, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 7, + "recall_iou_0.5": 0.875, + "recall_iou_0.95": 0.875, + "max_abs_score_delta": 0.02218437743377688, + "mean_abs_score_delta": 0.010339749952588764, + "extra_cpp_detections": 0, + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9962296324561855, + "min_mask_iou": 0.9934747145187602, + "mean_pixel_agreement": 0.999964975744396 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.009260908538818335, + "mean_abs_score_delta": 0.004838389758300777, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9966706420524734, + "min_mask_iou": 0.9948932607785684, + "mean_pixel_agreement": 0.9997352537722909 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9702380952380951, + "recall_iou_0.95": 0.9702380952380951, + "max_abs_score_delta": 0.021147268223353794, + "mean_abs_score_delta": 0.008455206940447112, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9932074153080327, + "min_mask_iou": 0.9810719522352481, + "mean_pixel_agreement": 0.9998509527354553 + } + }, + { + "variant": "seg-2xlarge", + "quant": "q4_K", + "model_path": "models/rfdetr-seg-2xlarge-q4_K.gguf", + "is_seg": true, + "file_size_mb": 38.62, + "n_images": 7, + "per_image": [ + { + "image": "000000000139.jpg", + "metrics": { + "py_total": 12, + "cpp_total": 13, + "matched_iou_0.5": 12, + "matched_iou_0.95": 6, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.5, + "max_abs_score_delta": 0.07563744468307498, + "mean_abs_score_delta": 0.0352352597249349, + "extra_cpp_detections": 1, + "n_matched_for_mask": 12, + "mean_mask_iou": 0.941347414957095, + "min_mask_iou": 0.8465558194774346, + "mean_pixel_agreement": 0.9996087636932708 + } + }, + { + "image": "000000000632.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 4, + "recall_iou_0.5": 0.8, + "recall_iou_0.95": 0.8, + "max_abs_score_delta": 0.04023191784667968, + "mean_abs_score_delta": 0.018881054208755466, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9774441201800199, + "min_mask_iou": 0.9549423681453021, + "mean_pixel_agreement": 0.9993804994824017 + } + }, + { + "image": "000000039769.jpg", + "metrics": { + "py_total": 4, + "cpp_total": 4, + "matched_iou_0.5": 4, + "matched_iou_0.95": 3, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.75, + "max_abs_score_delta": 0.05909197138214106, + "mean_abs_score_delta": 0.032093881770133975, + "extra_cpp_detections": 0, + "n_matched_for_mask": 4, + "mean_mask_iou": 0.9822867215443348, + "min_mask_iou": 0.9757349510032665, + "mean_pixel_agreement": 0.9984293619791667 + } + }, + { + "image": "000000087038.jpg", + "metrics": { + "py_total": 11, + "cpp_total": 11, + "matched_iou_0.5": 11, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.36363636363636365, + "max_abs_score_delta": 0.12661426900100703, + "mean_abs_score_delta": 0.0327620931705128, + "extra_cpp_detections": 0, + "n_matched_for_mask": 11, + "mean_mask_iou": 0.9584874473611447, + "min_mask_iou": 0.898238747553816, + "mean_pixel_agreement": 0.999885771780303 + } + }, + { + "image": "000000252219.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 5, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 1.0, + "max_abs_score_delta": 0.05442531827926633, + "mean_abs_score_delta": 0.031269855394744875, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9813510344755099, + "min_mask_iou": 0.9605185621685327, + "mean_pixel_agreement": 0.9995458528037384 + } + }, + { + "image": "000000397133.jpg", + "metrics": { + "py_total": 8, + "cpp_total": 7, + "matched_iou_0.5": 7, + "matched_iou_0.95": 4, + "recall_iou_0.5": 0.875, + "recall_iou_0.95": 0.5, + "max_abs_score_delta": 0.11550337743377681, + "mean_abs_score_delta": 0.04654969267436436, + "extra_cpp_detections": 0, + "n_matched_for_mask": 7, + "mean_mask_iou": 0.9778693011137124, + "min_mask_iou": 0.9725622625580413, + "mean_pixel_agreement": 0.9997046462027434 + } + }, + { + "image": "bus.jpg", + "metrics": { + "py_total": 5, + "cpp_total": 5, + "matched_iou_0.5": 5, + "matched_iou_0.95": 4, + "recall_iou_0.5": 1.0, + "recall_iou_0.95": 0.8, + "max_abs_score_delta": 0.07421847545623783, + "mean_abs_score_delta": 0.029402009986877455, + "extra_cpp_detections": 0, + "n_matched_for_mask": 5, + "mean_mask_iou": 0.9878901708935477, + "min_mask_iou": 0.9840410896083647, + "mean_pixel_agreement": 0.9993369913122999 + } + } + ], + "metrics": { + "recall_iou_0.5": 0.9535714285714285, + "recall_iou_0.95": 0.6733766233766234, + "max_abs_score_delta": 0.07796039629745481, + "mean_abs_score_delta": 0.03231340670433198, + "extra_cpp_detections": 0.14285714285714285, + "mean_mask_iou": 0.9723823157893376, + "min_mask_iou": 0.9417991143592511, + "mean_pixel_agreement": 0.9994131267505606 } } ] diff --git a/benchmarks/results/per_cell_latency.json b/benchmarks/results/per_cell_latency.json index af116b8..bf61545 100644 --- a/benchmarks/results/per_cell_latency.json +++ b/benchmarks/results/per_cell_latency.json @@ -1,34 +1,178 @@ { - "nano/f32": {"median_ms": 55.84, "min_ms": 51.05}, - "nano/f16": {"median_ms": 49.63, "min_ms": 48.40}, - "nano/q8_0": {"median_ms": 59.33, "min_ms": 53.11}, - "nano/q4_K": {"median_ms": 60.61, "min_ms": 59.77}, - "small/f32": {"median_ms": 100.14, "min_ms": 97.54}, - "small/f16": {"median_ms": 94.28, "min_ms": 93.00}, - "small/q8_0": {"median_ms": 110.19, "min_ms": 102.86}, - "small/q4_K": {"median_ms": 115.98, "min_ms": 113.01}, - "base/f32": {"median_ms": 144.02, "min_ms": 139.89}, - "base/f16": {"median_ms": 136.13, "min_ms": 135.42}, - "base/q8_0": {"median_ms": 145.70, "min_ms": 145.13}, - "base/q4_K": {"median_ms": 166.28, "min_ms": 162.63}, - "medium/f32": {"median_ms": 139.89, "min_ms": 134.90}, - "medium/f16": {"median_ms": 143.90, "min_ms": 130.94}, - "medium/q8_0": {"median_ms": 145.82, "min_ms": 138.38}, - "medium/q4_K": {"median_ms": 155.83, "min_ms": 154.05}, - "large/f32": {"median_ms": 236.59, "min_ms": 233.98}, - "large/f16": {"median_ms": 237.10, "min_ms": 229.54}, - "large/q8_0": {"median_ms": 251.20, "min_ms": 242.18}, - "large/q4_K": {"median_ms": 271.98, "min_ms": 262.62}, - "seg-nano/f32": {"median_ms": 114.74, "min_ms": 106.49}, - "seg-nano/f16": {"median_ms": 108.62, "min_ms": 103.25}, - "seg-nano/q8_0": {"median_ms": 118.96, "min_ms": 107.88}, - "seg-nano/q4_K": {"median_ms": 151.63, "min_ms": 131.26}, - "seg-small/f32": {"median_ms": 155.22, "min_ms": 146.67}, - "seg-small/f16": {"median_ms": 150.21, "min_ms": 142.15}, - "seg-small/q8_0": {"median_ms": 148.76, "min_ms": 147.64}, - "seg-small/q4_K": {"median_ms": 171.16, "min_ms": 168.40}, - "seg-medium/f32": {"median_ms": 199.47, "min_ms": 198.21}, - "seg-medium/f16": {"median_ms": 208.52, "min_ms": 197.38}, - "seg-medium/q8_0": {"median_ms": 218.51, "min_ms": 203.68}, - "seg-medium/q4_K": {"median_ms": 244.15, "min_ms": 236.60} -} + "nano/f32": { + "median_ms": 55.84, + "min_ms": 51.05 + }, + "nano/f16": { + "median_ms": 49.63, + "min_ms": 48.4 + }, + "nano/q8_0": { + "median_ms": 59.33, + "min_ms": 53.11 + }, + "nano/q4_K": { + "median_ms": 60.61, + "min_ms": 59.77 + }, + "small/f32": { + "median_ms": 100.14, + "min_ms": 97.54 + }, + "small/f16": { + "median_ms": 94.28, + "min_ms": 93.0 + }, + "small/q8_0": { + "median_ms": 110.19, + "min_ms": 102.86 + }, + "small/q4_K": { + "median_ms": 115.98, + "min_ms": 113.01 + }, + "base/f32": { + "median_ms": 144.02, + "min_ms": 139.89 + }, + "base/f16": { + "median_ms": 136.13, + "min_ms": 135.42 + }, + "base/q8_0": { + "median_ms": 145.7, + "min_ms": 145.13 + }, + "base/q4_K": { + "median_ms": 166.28, + "min_ms": 162.63 + }, + "medium/f32": { + "median_ms": 139.89, + "min_ms": 134.9 + }, + "medium/f16": { + "median_ms": 143.9, + "min_ms": 130.94 + }, + "medium/q8_0": { + "median_ms": 145.82, + "min_ms": 138.38 + }, + "medium/q4_K": { + "median_ms": 155.83, + "min_ms": 154.05 + }, + "large/f32": { + "median_ms": 236.59, + "min_ms": 233.98 + }, + "large/f16": { + "median_ms": 237.1, + "min_ms": 229.54 + }, + "large/q8_0": { + "median_ms": 251.2, + "min_ms": 242.18 + }, + "large/q4_K": { + "median_ms": 271.98, + "min_ms": 262.62 + }, + "seg-nano/f32": { + "median_ms": 114.74, + "min_ms": 106.49 + }, + "seg-nano/f16": { + "median_ms": 108.62, + "min_ms": 103.25 + }, + "seg-nano/q8_0": { + "median_ms": 118.96, + "min_ms": 107.88 + }, + "seg-nano/q4_K": { + "median_ms": 151.63, + "min_ms": 131.26 + }, + "seg-small/f32": { + "median_ms": 155.22, + "min_ms": 146.67 + }, + "seg-small/f16": { + "median_ms": 150.21, + "min_ms": 142.15 + }, + "seg-small/q8_0": { + "median_ms": 148.76, + "min_ms": 147.64 + }, + "seg-small/q4_K": { + "median_ms": 171.16, + "min_ms": 168.4 + }, + "seg-medium/f32": { + "median_ms": 199.47, + "min_ms": 198.21 + }, + "seg-medium/f16": { + "median_ms": 208.52, + "min_ms": 197.38 + }, + "seg-medium/q8_0": { + "median_ms": 218.51, + "min_ms": 203.68 + }, + "seg-medium/q4_K": { + "median_ms": 244.15, + "min_ms": 236.6 + }, + "seg-large/f32": { + "median_ms": 647.35, + "min_ms": 615.91 + }, + "seg-large/f16": { + "median_ms": 638.41, + "min_ms": 630.53 + }, + "seg-large/q8_0": { + "median_ms": 565.13, + "min_ms": 557.34 + }, + "seg-large/q4_K": { + "median_ms": 610.55, + "min_ms": 602.39 + }, + "seg-xlarge/f32": { + "median_ms": 1197.28, + "min_ms": 1181.47 + }, + "seg-xlarge/f16": { + "median_ms": 1183.22, + "min_ms": 1169.44 + }, + "seg-xlarge/q8_0": { + "median_ms": 1086.45, + "min_ms": 1068.12 + }, + "seg-xlarge/q4_K": { + "median_ms": 1157.51, + "min_ms": 1135.43 + }, + "seg-2xlarge/f32": { + "median_ms": 2160.96, + "min_ms": 2142.47 + }, + "seg-2xlarge/f16": { + "median_ms": 2158.54, + "min_ms": 2146.34 + }, + "seg-2xlarge/q8_0": { + "median_ms": 2023.36, + "min_ms": 1998.14 + }, + "seg-2xlarge/q4_K": { + "median_ms": 2131.43, + "min_ms": 2095.9 + } +} \ No newline at end of file diff --git a/docs/conversion.md b/docs/conversion.md index 80502e9..37ea38f 100644 --- a/docs/conversion.md +++ b/docs/conversion.md @@ -1,4 +1,4 @@ -# rf-detr.cpp GGUF Conversion (rfdetr-base, matching upstream 1.7.0) +# rf-detr.cpp GGUF Conversion (matching upstream 1.9.0) This doc is the contract between `scripts/convert_rfdetr_to_gguf.py` and `src/model_loader.cpp`. Both sides reference the same keys and tensor names. @@ -42,6 +42,7 @@ All keys live under the `rfdetr.` namespace. | `rfdetr.class_names` | string[91] | One entry per logit index. Unused IDs are `""`; the 80 COCO names sit at their COCO-spec positions. | | `rfdetr.preprocess.mean` | float32[3] | `[0.485, 0.456, 0.406]` (ImageNet) | | `rfdetr.preprocess.std` | float32[3] | `[0.229, 0.224, 0.225]` (ImageNet) | +| `rfdetr.preprocess.resize_mode` | string | Optional v2 key. New conversions use `"bilinear_no_antialias"` to match RF-DETR 1.9; absent means legacy stb resize for existing GGUF compatibility. | | `rfdetr.backbone.dim` | uint32 | `384` | | `rfdetr.backbone.depth` | uint32 | `12` | | `rfdetr.backbone.heads` | uint32 | `6` | @@ -285,21 +286,21 @@ Total heads: 8. | Heads | 8 | | **Total** | **486** | -Upstream `state_dict` has 487 tensors; the +1 is `mask_token` (training only, -dropped by converter). +Upstream 1.9 adds two non-GGUF entries beyond the table: `mask_token` +(training only) and `_kp_active_mask` (a deterministic keypoint activity +buffer unused by standard detection). Both are dropped by the converter. ## Per-variant notes -Only `base` is supported for now. `nano`, `small`, `medium`, `large` are -deferred. They reuse the same schema but with different `backbone.dim`, -`backbone.depth`, `backbone.heads`, `projector.in_dim`, `projector.out_dim`, -and (potentially) `decoder.layers`. Each variant must be introspected to -confirm whether single-scale (P4 only) holds. +The converter and runtime support the detection and segmentation variants +listed by `--help`. They reuse the same core schema with per-variant backbone, +projector, decoder, resolution, and optional segmentation-head metadata. +Nano detection is the RF-DETR 1.9 compatibility baseline. ## Discovery workflow -The PyTorch keys above are valid for the rfdetr-base release at the version -pinned in `scripts/requirements.txt` (rfdetr 1.7.0). Upstream renames are +The PyTorch keys above are valid for the version pinned in +`scripts/requirements.txt` (`rfdetr==1.9.0`). Upstream renames are possible. The conversion script's first task is to enumerate `state_dict().keys()`, diff against the expected set, and refuse to convert on any missing or unmapped key. Bringing up a new variant or upstream diff --git a/docs/finetuning.md b/docs/finetuning.md index 049907f..f1903f5 100644 --- a/docs/finetuning.md +++ b/docs/finetuning.md @@ -77,14 +77,16 @@ Each checkpoint is a `dict` with: What the converter does on `--checkpoint`: -1. `torch.load()` the `.pth` (weights_only=False; required because rfdetr - embeds an `argparse.Namespace` in legacy checkpoints). +1. Safely loads the `.pth` with RF-DETR 1.9's restricted loader. Legacy + `argparse.Namespace` metadata is allow-listed without enabling arbitrary + pickle execution. A fully trusted checkpoint that contains other custom + objects can opt into the unsafe fallback with `--trust-checkpoint`. 2. Reads the actual head size from `checkpoint["model"]["class_embed.bias"].shape[0]`. 3. Constructs `RFDETRBase()` and calls `reinitialize_detection_head(head_size)` so the classification head (shared `class_embed` + 13 `enc_out_class_embed` groups) matches the checkpoint shape. -4. `load_state_dict(strict=False)` (mask_token is the one legitimate "unused" - entry, training-only). +4. `load_state_dict(strict=False)` (`mask_token` and the detection-unused + `_kp_active_mask` buffer are legitimate unconverted entries). 5. Writes GGUF with `rfdetr.num_classes = head_size` and a synthesized `class_` placeholder list as `rfdetr.class_names`. diff --git a/models/MANIFEST.md b/models/MANIFEST.md index e89b4c0..16a466c 100644 --- a/models/MANIFEST.md +++ b/models/MANIFEST.md @@ -1,8 +1,10 @@ # rf-detr.cpp Model Manifest -Generated 2026-05-27. All models converted from rfdetr 1.7.0 PyTorch +Generated 2026-07-30. All models converted from `rfdetr==1.9.0` PyTorch checkpoints via `scripts/convert_rfdetr_to_gguf.py` (F32) and re-quantized -in-place by the C++ quantizer (`build/bin/rfdetr-cli quantize`). +in-place by the C++ quantizer (`build/bin/rfdetr-cli quantize`), at commit +[`52b1650`](https://github.com/localai-org/rf-detr.cpp/commit/52b1650864016b272d377b99fdbbda3339364013) +(pre-merge; see the PR for the final merged SHA). The `.gguf` files themselves are gitignored; this manifest is the canonical record of what should exist and at what size. To rebuild from scratch: @@ -19,41 +21,70 @@ done scripts/build_all_quants.sh ``` -All 44 models have been verified to load and run `rfdetr-cli detect` on -`/tmp/coco_sample.jpg` without error. +All 44 models have been verified end-to-end: + +- `rfdetr-cli info` reports the correct variant, image size, class count, + query count, and tensor count for every file. +- `rfdetr.preprocess.resize_mode` is present and equals + `bilinear_no_antialias` on every file (new RF-DETR 1.9 preprocessing path). +- Detection variants complete a real detection on + `tests/fixtures/ci/test_image.jpg`. Segmentation variants additionally + produce per-detection mask PNGs. +- PyTorch-vs-C++ parity was swept across all 44 (variant × quant) cells + (`scripts/sweep_accuracy.py`, results in + `benchmarks/results/accuracy_sweep.json`): F32/F16/Q8_0 all reach + Recall@IoU-0.5 = 1.0 and mean segmentation mask IoU ≥ 0.95 everywhere; Q4_K + shows the expected accuracy/size tradeoff (Recall@IoU-0.95 as low as 0.125 + on seg-small — see the sweep JSON for the full per-cell breakdown, and do + not treat Q4_K as parity-equivalent to F16/Q8_0). +- Backward compatibility was reconfirmed against a GGUF built from the + pre-1.9 converter (no `resize_mode` key): it loads and detects correctly + via the legacy stb resize path. + +Full per-file SHA-256 checksums are published as `SHA256SUMS` in each +variant's Hugging Face repository (see README for the repo list) and are not +duplicated here to keep this file readable. ## Detection variants | Variant | F32 | F16 | Q8_0 | Q4_K | |---|---:|---:|---:|---:| -| Nano | 112.7 MB | 60.5 MB | 36.0 MB | 29.7 MB | -| Small | 119.0 MB | 64.0 MB | 38.2 MB | 31.2 MB | -| Base | 119.2 MB | 64.2 MB | 38.5 MB | 31.5 MB | -| Medium | 125.0 MB | 67.2 MB | 40.2 MB | 32.5 MB | -| Large | 125.9 MB | 68.2 MB | 41.1 MB | 33.4 MB | +| Nano | 118.2 MB | 63.4 MB | 37.8 MB | 31.2 MB | +| Small | 124.7 MB | 67.1 MB | 40.1 MB | 32.8 MB | +| Base | 125.0 MB | 67.4 MB | 40.3 MB | 33.0 MB | +| Medium | 131.0 MB | 70.5 MB | 42.1 MB | 34.0 MB | +| Large | 132.0 MB | 71.5 MB | 43.1 MB | 35.0 MB | ## Segmentation variants | Variant | F32 | F16 | Q8_0 | Q4_K | |---|---:|---:|---:|---:| -| Seg-Nano | 127.1 MB | 67.8 MB | 39.9 MB | 31.8 MB | -| Seg-Small | 127.6 MB | 68.3 MB | 40.4 MB | 32.3 MB | -| Seg-Medium | 133.7 MB | 71.5 MB | 42.4 MB | 33.6 MB | -| Seg-Large | 134.3 MB | 72.2 MB | 43.1 MB | 34.3 MB | -| Seg-XLarge | 141.3 MB | 76.4 MB | 46.0 MB | 36.5 MB | -| Seg-2XLarge | 143.4 MB | 78.4 MB | 48.0 MB | 38.5 MB | +| Seg-Nano | 133.3 MB | 71.1 MB | 41.9 MB | 33.4 MB | +| Seg-Small | 133.8 MB | 71.6 MB | 42.4 MB | 33.9 MB | +| Seg-Medium | 140.4 MB | 75.1 MB | 44.5 MB | 35.3 MB | +| Seg-Large | 141.1 MB | 75.8 MB | 45.2 MB | 36.0 MB | +| Seg-XLarge | 148.8 MB | 80.4 MB | 48.4 MB | 38.4 MB | +| Seg-2XLarge | 150.9 MB | 82.5 MB | 50.5 MB | 40.5 MB | + +Note: segmentation variants medium/large/xlarge/2xlarge are each ~1 block +larger than in the pre-1.9 manifest (5 or 6 `DepthwiseConvBlock`s instead of +a hardcoded 4) after fixing a real segmentation-head coverage bug — see +commit `52b1650` — so these sizes are not directly comparable to any +previously published rf-detr.cpp seg-medium/large/xlarge/2xlarge GGUF. ## Quant choice notes -- **F32**: full precision reference, about 120 MB per variant. +- **F32**: full precision reference, about 120-150 MB per variant. - **F16**: only matmul-multiplicand weights converted; non-matmul tensors (norms, conv kernels, embeddings) stay F32. Loader handles F16 `pos_embed` via bicubic resample in F32 (see commit 2145c7d). - **Q8_0**: best accuracy/size trade for production; about 3x smaller than F32. - **Q4_K**: smallest practical quant; rows with `ne[0] % 256 != 0` (the - decoder's 128-dim MLP halves, 60 tensors) silently fall back to Q8_0 per - the C++ quantizer's logic. Net result is still about 3.8 to 4.0x - compression. + decoder's 128-dim MLP halves) silently fall back to Q8_0 per the C++ + quantizer's logic. Net result is still about 3.7 to 3.9x compression, but + expect a real, sometimes large, Recall@IoU-0.95 drop — see + `benchmarks/results/accuracy_sweep.json` per variant before choosing Q4_K + for an accuracy-sensitive deployment. Heavier K-quants (Q5_K, Q6_K) and the older legacy quants (Q4_0/Q4_1/Q5_0/Q5_1) are supported by `rfdetr-cli quantize` but are not part of the standard diff --git a/scripts/accuracy_table.py b/scripts/accuracy_table.py index 6de5d9e..a58e3e4 100755 --- a/scripts/accuracy_table.py +++ b/scripts/accuracy_table.py @@ -17,14 +17,17 @@ from pathlib import Path VARIANT_LABEL = { - "nano": "Nano", - "small": "Small", - "base": "Base", - "medium": "Medium", - "large": "Large", - "seg-nano": "Seg-Nano", - "seg-small": "Seg-Small", - "seg-medium": "Seg-Medium", + "nano": "Nano", + "small": "Small", + "base": "Base", + "medium": "Medium", + "large": "Large", + "seg-nano": "Seg-Nano", + "seg-small": "Seg-Small", + "seg-medium": "Seg-Medium", + "seg-large": "Seg-Large", + "seg-xlarge": "Seg-XLarge", + "seg-2xlarge": "Seg-2XLarge", } QUANT_LABEL = { @@ -34,8 +37,10 @@ "q4_K": "Q4_K", } -VARIANT_ORDER = ["nano", "small", "base", "medium", "large", - "seg-nano", "seg-small", "seg-medium"] +DET_VARIANTS = ["nano", "small", "base", "medium", "large"] +SEG_VARIANTS = ["seg-nano", "seg-small", "seg-medium", + "seg-large", "seg-xlarge", "seg-2xlarge"] +VARIANT_ORDER = DET_VARIANTS + SEG_VARIANTS QUANT_ORDER = ["f32", "f16", "q8_0", "q4_K"] @@ -50,7 +55,7 @@ def render_detection(cells: list[dict]) -> str: lines.append("") lines.append("| Variant | Quant | Size (MB) | Recall@0.5 | Recall@0.95 | Max \\|Δscore\\| | Mean \\|Δscore\\| | Extra dets |") lines.append("|---|---|---:|---:|---:|---:|---:|---:|") - for v in ["nano", "small", "base", "medium", "large"]: + for v in DET_VARIANTS: for q in QUANT_ORDER: c = by.get((v, q)) if c is None or "error" in c: @@ -78,7 +83,7 @@ def render_segmentation(cells: list[dict]) -> str: lines.append("") lines.append("| Variant | Quant | Size (MB) | Recall@0.5 | Recall@0.95 | Mean mask IoU | Pixel agreement | Mean \\|Δscore\\| |") lines.append("|---|---|---:|---:|---:|---:|---:|---:|") - for v in ["seg-nano", "seg-small", "seg-medium"]: + for v in SEG_VARIANTS: for q in QUANT_ORDER: c = by.get((v, q)) if c is None or "error" in c: diff --git a/scripts/build_all_quants.sh b/scripts/build_all_quants.sh index bf03919..2d5979f 100755 --- a/scripts/build_all_quants.sh +++ b/scripts/build_all_quants.sh @@ -11,9 +11,10 @@ # Usage: # scripts/build_all_quants.sh # -# Coverage matrix (32 models): -# detection × {nano, small, base, medium, large} × {F32, F16, Q8_0, Q4_K} -# segment × {seg-nano, seg-small, seg-medium} × {F32, F16, Q8_0, Q4_K} +# Coverage matrix (44 models): +# detection × {nano, small, base, medium, large} × {F32, F16, Q8_0, Q4_K} +# segment × {seg-nano, seg-small, seg-medium, +# seg-large, seg-xlarge, seg-2xlarge} × {F32, F16, Q8_0, Q4_K} set -euo pipefail cd "$(dirname "$0")/.." @@ -25,7 +26,7 @@ if [[ ! -x "$CLI" ]]; then fi DET_VARIANTS=(nano small base medium large) -SEG_VARIANTS=(seg-nano seg-small seg-medium) +SEG_VARIANTS=(seg-nano seg-small seg-medium seg-large seg-xlarge seg-2xlarge) QUANTS=(f16 q8_0 q4_K) built=0 diff --git a/scripts/convert_rfdetr_to_gguf.py b/scripts/convert_rfdetr_to_gguf.py index eaf55a5..d37131b 100755 --- a/scripts/convert_rfdetr_to_gguf.py +++ b/scripts/convert_rfdetr_to_gguf.py @@ -244,6 +244,8 @@ def parse_args(): default="f16") p.add_argument("--checkpoint", help="Optional path to a local rfdetr .pth. Default: download via rfdetr pkg.") + p.add_argument("--trust-checkpoint", action="store_true", + help="Allow unsafe pickle fallback for a fully trusted legacy checkpoint.") p.add_argument("--dry-run", action="store_true", help="Load model + validate name map, but do not write GGUF.") return p.parse_args() @@ -395,8 +397,15 @@ def build_tensor_name_map(variant_cfg): # ---- Segmentation head (seg variants only) ---- if variant_cfg.get("segmentation_head", False): SH = "segmentation_head." - # 4 × DepthwiseConvBlock - for b in range(4): + # SegmentationHead.forward_export chains one DepthwiseConvBlock per + # decoder layer (rfdetr.models.heads.segmentation.SegmentationHead: + # num_blocks == len(decoder layers) since forward() zips + # self.blocks with per-decoder-layer query_features). nano/small + # have 4 decoder layers -> 4 blocks; medium/large have 5 -> 5 + # blocks; xlarge/2xlarge have 6 -> 6 blocks. Using a hardcoded 4 + # here silently dropped block(s) for every variant except + # nano/small. + for b in range(dec_layers): sp = SH + f"blocks.{b}." dp = f"segmentation_head.blocks.{b}." m[dp + "dwconv.weight"] = (sp + "dwconv.weight", None) @@ -569,6 +578,10 @@ def write_gguf(out_path, variant_name, variant_cfg, name_map, state_dict, std = np.array([0.229, 0.224, 0.225], dtype=np.float32) writer.add_array("rfdetr.preprocess.mean", mean.tolist()) writer.add_array("rfdetr.preprocess.std", std.tolist()) + # RF-DETR 1.9 disabled resize antialiasing at inference to match its + # cv2.INTER_LINEAR training pipeline. Stamp the mode so new conversions + # get v1.9 parity while old v2 GGUFs retain their legacy stb behavior. + writer.add_string("rfdetr.preprocess.resize_mode", "bilinear_no_antialias") # backbone bb = variant_cfg["backbone"] @@ -712,7 +725,8 @@ def main() -> int: # the model's classification head to match BEFORE load_state_dict so # the shapes line up. print(f"[checkpoint] loading {args.checkpoint}", file=sys.stderr) - ckpt = torch.load(args.checkpoint, map_location="cpu", weights_only=False) + from rfdetr.utilities.io import _safe_torch_load + ckpt = _safe_torch_load(args.checkpoint, trust=args.trust_checkpoint) if not isinstance(ckpt, dict) or "model" not in ckpt: print("error: checkpoint must be a dict with a 'model' state_dict key.", file=sys.stderr) diff --git a/scripts/publish_hf.py b/scripts/publish_hf.py index f73e0b2..5f88994 100644 --- a/scripts/publish_hf.py +++ b/scripts/publish_hf.py @@ -1,21 +1,28 @@ #!/usr/bin/env python3 """Publish rfdetr.cpp GGUF models to HuggingFace Hub. -Creates one repo per variant under ``mudler/rfdetr-cpp-``, each +Creates one repo per variant under ``/rfdetr-cpp-``, each containing the F32 / F16 / Q8_0 / Q4_K quantizations plus a data-driven model card backed by the Phase 2 accuracy sweep and the per-cell latency microbench. Usage: - .venv/bin/python scripts/publish_hf.py [--dry-run] + .venv/bin/python scripts/publish_hf.py --hf-user [--dry-run] + [--github-repo /rf-detr.cpp] [--only nano,base,...] [--skip-uploads] + +--hf-user is required (no hardcoded default) so this can never accidentally +publish to someone else's namespace. """ from __future__ import annotations import argparse +import hashlib +import importlib.util import json import os +import subprocess import sys import time from dataclasses import dataclass @@ -31,10 +38,55 @@ ACCURACY_JSON = RESULTS_DIR / "accuracy_sweep.json" LATENCY_JSON = RESULTS_DIR / "per_cell_latency.json" -HF_USER = "mudler" QUANTS = ["f32", "f16", "q8_0", "q4_K"] -# All 11 variants, in publish order. +# Default GitHub org/repo used for links when --github-repo isn't given. +# This must point at a repo the reader can actually clone; pass +# --github-repo /rf-detr.cpp to point at a personal fork. +DEFAULT_GITHUB_REPO = "localai-org/rf-detr.cpp" + + +def _git_commit_sha() -> str: + """Current commit SHA of this checkout, for model-card provenance.""" + try: + out = subprocess.run( + ["git", "rev-parse", "HEAD"], cwd=REPO_ROOT, + capture_output=True, text=True, check=True) + return out.stdout.strip() + except Exception: + return "unknown" + + +def _rfdetr_version() -> str: + try: + import importlib.metadata + return importlib.metadata.version("rfdetr") + except Exception: + return "unknown" + + +def sha256_file(path: Path) -> str: + h = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def _load_converter_module(): + """Import scripts/convert_rfdetr_to_gguf.py as a module to reuse its + VARIANTS config instead of duplicating architecture constants here.""" + spec = importlib.util.spec_from_file_location( + "convert_rfdetr_to_gguf", REPO_ROOT / "scripts" / "convert_rfdetr_to_gguf.py") + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +_CONV = _load_converter_module() + +# All 11 variants, in publish order (converter VARIANTS dict is unordered +# w.r.t. our desired publish order, so keep an explicit list here). VARIANTS = [ "nano", "small", @@ -49,21 +101,19 @@ "seg-2xlarge", ] -# Static per-variant architecture facts (resolution, decoder layers, queries, -# patch size). These come from the upstream rfdetr 1.7.0 model registry — -# they're not changing per quant so we keep them as a small lookup. +# Per-variant architecture facts, derived from the converter's VARIANTS +# config (scripts/convert_rfdetr_to_gguf.py) rather than duplicated here, so +# the two can never drift. All variants use a DINOv2-small backbone +# (dim=384, heads=6) per that config. ARCH = { - "nano": {"resolution": 384, "patch": 14, "decoder_layers": 2, "queries": 300, "backbone": "DINOv2-small"}, - "small": {"resolution": 512, "patch": 14, "decoder_layers": 3, "queries": 300, "backbone": "DINOv2-small"}, - "base": {"resolution": 560, "patch": 14, "decoder_layers": 3, "queries": 300, "backbone": "DINOv2-small"}, - "medium": {"resolution": 576, "patch": 14, "decoder_layers": 4, "queries": 300, "backbone": "DINOv2-small"}, - "large": {"resolution": 704, "patch": 14, "decoder_layers": 4, "queries": 300, "backbone": "DINOv2-small"}, - "seg-nano": {"resolution": 312, "patch": 12, "decoder_layers": 4, "queries": 100, "backbone": "DINOv2-small"}, - "seg-small": {"resolution": 384, "patch": 12, "decoder_layers": 4, "queries": 100, "backbone": "DINOv2-small"}, - "seg-medium": {"resolution": 432, "patch": 12, "decoder_layers": 5, "queries": 200, "backbone": "DINOv2-small"}, - "seg-large": {"resolution": 504, "patch": 12, "decoder_layers": 5, "queries": 200, "backbone": "DINOv2-small"}, - "seg-xlarge": {"resolution": 624, "patch": 12, "decoder_layers": 6, "queries": 300, "backbone": "DINOv2-small"}, - "seg-2xlarge": {"resolution": 768, "patch": 12, "decoder_layers": 6, "queries": 300, "backbone": "DINOv2-small"}, + v: { + "resolution": cfg["image_size"], + "patch": cfg["patch_size"], + "decoder_layers": cfg["decoder"]["layers"], + "queries": cfg["num_queries"], + "backbone": "DINOv2-small", + } + for v, cfg in _CONV.VARIANTS.items() } # Pretty display names. @@ -103,6 +153,12 @@ class CellMetrics: min_ms: Optional[float] = None +def load_sweep_meta() -> dict: + with open(ACCURACY_JSON) as f: + sweep = json.load(f) + return sweep.get("meta", {}) + + def load_metrics() -> Dict[str, Dict[str, CellMetrics]]: """Returns metrics[variant][quant] = CellMetrics. @@ -140,7 +196,11 @@ def load_metrics() -> Dict[str, Dict[str, CellMetrics]]: return out -def build_model_card(variant: str, cells: Dict[str, CellMetrics]) -> str: +def build_model_card(variant: str, cells: Dict[str, CellMetrics], *, + hf_user: str, github_repo: str, + commit_sha: str, rfdetr_version: str, + checksums: Dict[str, str], + sweep_meta: dict) -> str: """Generate a Markdown model card for one variant repo. If `cells` is empty, the per-quant metrics table is replaced with a @@ -154,6 +214,23 @@ def build_model_card(variant: str, cells: Dict[str, CellMetrics]) -> str: seg_tag = "image-segmentation" if is_seg else "object-detection" has_metrics = bool(cells) + # Accurately describe the sweep that actually produced `cells`, instead + # of a hardcoded claim — n_images/image_names come straight from + # accuracy_sweep.json's meta block. + n_images = sweep_meta.get("n_images", 0) + image_names = sweep_meta.get("image_names", []) + sweep_rfdetr_version = sweep_meta.get("rfdetr_version", rfdetr_version) + if n_images: + img_desc = f"{n_images} image" + ("s" if n_images != 1 else "") + if image_names: + img_desc += f" ({', '.join(image_names)})" + else: + img_desc = "a held-out image set" + methodology_note = ( + f"against the upstream PyTorch reference (`rfdetr {sweep_rfdetr_version}`) " + f"on {img_desc} at threshold {sweep_meta.get('threshold', 0.5)}" + ) + # YAML frontmatter tags = [ "object-detection", @@ -183,7 +260,7 @@ def build_model_card(variant: str, cells: Dict[str, CellMetrics]) -> str: lines.append("") lines.append( f"GGUF-format weights of [Roboflow RF-DETR {pretty}]({UPSTREAM}) " - f"({kind} variant) for use with [rfdetr.cpp](https://github.com/mudler/rf-detr.cpp), " + f"({kind} variant) for use with [rfdetr.cpp](https://github.com/{github_repo}), " f"a C++/ggml implementation that matches the upstream PyTorch model on CPU." ) lines.append("") @@ -247,19 +324,22 @@ def build_model_card(variant: str, cells: Dict[str, CellMetrics]) -> str: lines.append(f"| `{fname}`{rec_marker} | {quant_label[q]} | {size_mb:.1f} |") lines.append("") lines.append("> Accuracy + latency for this variant haven't been added to the " - "[BENCHMARK.md](https://github.com/mudler/rf-detr.cpp/blob/main/BENCHMARK.md) " + f"[BENCHMARK.md](https://github.com/{github_repo}/blob/main/BENCHMARK.md) " "sweep yet; the C++ implementation is verified to load and run " "`rfdetr-cli detect` end-to-end on every quant. Run " "`scripts/sweep_accuracy.py --variant " + variant + "` locally " "for parity numbers.") lines.append("") - # Methodology note - lines.append("All accuracy numbers are computed against the upstream PyTorch " - "reference (`rfdetr 1.7.0`) on 7 COCO val2017 images at threshold " - "0.5. Latency is measured with `rfdetr-cli bench` (8 iters + 3 " - "warmup) at T=8 threads on a single AMD Ryzen 9 9950X3D image " - "(`coco_kitchen.jpg`, 640x427).") + # Methodology note. Latency comes from a separate microbench + # (per_cell_latency.json) that is NOT run on the accuracy-sweep image + # set above -- do not reuse `image_names` here, they're unrelated runs. + lines.append( + f"All accuracy numbers above are computed {methodology_note}. " + "Latency is measured separately with `rfdetr-cli bench` (8 iters + 3 " + "warmup) at T=8 threads on a single Intel Core i7-12800HX, on " + "`tests/fixtures/ci/test_image.jpg`." + ) lines.append("") # Architecture @@ -291,17 +371,37 @@ def build_model_card(variant: str, cells: Dict[str, CellMetrics]) -> str: "drop relative to F16/Q8_0 (see file table above).") lines.append("") + # Compatibility + lines.append("## Compatibility") + lines.append("") + lines.append( + "These GGUFs stamp `rfdetr.preprocess.resize_mode = \"bilinear_no_antialias\"`, " + "matching RF-DETR 1.9's antialias-free float bilinear resize " + "(`align_corners=false`, half-pixel coordinates, no intermediate uint8 rounding). " + "rf-detr.cpp treats this key as **optional**: GGUFs that predate this metadata " + "(no `resize_mode` key) keep using the legacy stb-based resize path, so older " + "files continue to produce their original outputs unchanged. An unrecognized " + "`resize_mode` value is rejected rather than guessed." + ) + lines.append("") + lines.append( + "**Keypoint-preview inference is not supported.** rf-detr.cpp does not implement " + "the keypoint output head; this repository only serves " + f"{'box detection + instance segmentation masks' if is_seg else 'box detection'} outputs." + ) + lines.append("") + # Usage lines.append("## Usage") lines.append("") lines.append("```bash") lines.append("# 1. Clone + build rfdetr.cpp") - lines.append("git clone https://github.com/mudler/rf-detr.cpp") - lines.append("cd rt-detr.cpp") + lines.append(f"git clone https://github.com/{github_repo}") + lines.append("cd rf-detr.cpp") lines.append("cmake -B build -DRFDETR_BUILD_CLI=ON && cmake --build build -j") lines.append("") lines.append("# 2. Download a quant (F16 recommended)") - lines.append(f"hf download {HF_USER}/rfdetr-cpp-{variant} rfdetr-{variant}-f16.gguf --local-dir models/") + lines.append(f"hf download {hf_user}/rfdetr-cpp-{variant} rfdetr-{variant}-f16.gguf --local-dir models/") lines.append("") if is_seg: lines.append(f"# 3. Run segmentation (writes per-detection PNG masks to /tmp/seg_masks/)") @@ -325,8 +425,7 @@ def build_model_card(variant: str, cells: Dict[str, CellMetrics]) -> str: lines.append("## Accuracy methodology") lines.append("") lines.append( - "All accuracy metrics are computed against the upstream PyTorch reference " - "(rfdetr 1.7.0) on 7 COCO val2017 images at threshold 0.5. Each detection " + f"All accuracy metrics are computed {methodology_note}. Each detection " "match uses greedy Hungarian-style assignment by IoU (≥ 0.5 lenient, " "≥ 0.95 strict) with class equality required." ) @@ -340,12 +439,34 @@ def build_model_card(variant: str, cells: Dict[str, CellMetrics]) -> str: ) lines.append("") lines.append( - "See [BENCHMARK.md](https://github.com/mudler/rf-detr.cpp/blob/main/BENCHMARK.md) " - "and [`benchmarks/results/accuracy_sweep.json`](https://github.com/mudler/rf-detr.cpp/blob/main/benchmarks/results/accuracy_sweep.json) " + f"See [BENCHMARK.md](https://github.com/{github_repo}/blob/main/BENCHMARK.md) " + f"and [`benchmarks/results/accuracy_sweep.json`](https://github.com/{github_repo}/blob/main/benchmarks/results/accuracy_sweep.json) " "for the full sweep across the (variant × quant) cells." ) lines.append("") + # Provenance + lines.append("## Provenance") + lines.append("") + lines.append(f"- Source project: [Roboflow RF-DETR]({UPSTREAM})") + lines.append(f"- Upstream package: `rfdetr=={rfdetr_version}`") + lines.append(f"- Converted with [rfdetr.cpp](https://github.com/{github_repo}) " + f"at commit [`{commit_sha[:12]}`](https://github.com/{github_repo}/commit/{commit_sha})") + lines.append(f"- Checkpoint: official pretrained `rfdetr-{variant}` weights " + f"(downloaded by the `rfdetr` package on first use)") + lines.append("") + lines.append("### Checksums (SHA-256)") + lines.append("") + lines.append("Also available as `SHA256SUMS` in this repo.") + lines.append("") + lines.append("```") + for q in QUANTS: + fname = f"rfdetr-{variant}-{q}.gguf" + if fname in checksums: + lines.append(f"{checksums[fname]} {fname}") + lines.append("```") + lines.append("") + # License lines.append("## License") lines.append("") @@ -395,12 +516,18 @@ def upload_with_retry(api: HfApi, local_path: Path, remote_name: str, repo_id: s def publish_variant(api: HfApi, variant: str, cells: Dict[str, CellMetrics], *, + hf_user: str, github_repo: str, + commit_sha: str, rfdetr_version: str, sweep_meta: dict, dry_run: bool, skip_existing: bool) -> Dict[str, object]: """Publish one variant. Returns a status dict for the summary table.""" - repo_id = f"{HF_USER}/rfdetr-cpp-{variant}" + repo_id = f"{hf_user}/rfdetr-cpp-{variant}" print(f"\n=== {repo_id} ===") local_files = list_local_files(variant) - card = build_model_card(variant, cells) + checksums = {p.name: sha256_file(p) for p in local_files} + sha256sums_content = "\n".join(f"{checksums[p.name]} {p.name}" for p in local_files) + "\n" + card = build_model_card(variant, cells, hf_user=hf_user, github_repo=github_repo, + commit_sha=commit_sha, rfdetr_version=rfdetr_version, + checksums=checksums, sweep_meta=sweep_meta) total_bytes = sum(p.stat().st_size for p in local_files) print(f" 4 GGUF files, total {total_bytes / 1e6:.1f} MB") @@ -408,8 +535,10 @@ def publish_variant(api: HfApi, variant: str, cells: Dict[str, CellMetrics], *, if dry_run: print(f" [dry-run] would create repo {repo_id} (public)") print(f" [dry-run] would upload README.md ({len(card)} bytes)") + print(f" [dry-run] would upload SHA256SUMS ({len(sha256sums_content)} bytes)") for p in local_files: - print(f" [dry-run] would upload {p.name} ({p.stat().st_size / 1e6:.1f} MB)") + print(f" [dry-run] would upload {p.name} ({p.stat().st_size / 1e6:.1f} MB, " + f"sha256={checksums[p.name][:16]}...)") return { "repo_id": repo_id, "url": f"https://huggingface.co/{repo_id}", @@ -421,16 +550,26 @@ def publish_variant(api: HfApi, variant: str, cells: Dict[str, CellMetrics], *, # Create repo (idempotent) api.create_repo(repo_id=repo_id, repo_type="model", private=False, exist_ok=True) - # List existing remote files for skip-existing - existing: set[str] = set() + # List existing remote files + their content hashes for skip-existing. + # A same-named file is only skipped if its remote sha256 matches the + # local one — this correctly overwrites stale pre-1.9 artifacts instead + # of silently leaving them in place. + existing_sha256: Dict[str, str] = {} if skip_existing: try: - existing = set(api.list_repo_files(repo_id=repo_id, repo_type="model")) - if existing: - print(f" existing files in repo: {sorted(existing)}") + names = set(api.list_repo_files(repo_id=repo_id, repo_type="model")) + gguf_names = [p.name for p in local_files if p.name in names] + if gguf_names: + infos = api.get_paths_info(repo_id=repo_id, paths=gguf_names, repo_type="model") + for info in infos: + lfs = getattr(info, "lfs", None) + if lfs is not None and getattr(lfs, "sha256", None): + existing_sha256[info.path] = lfs.sha256 + if names: + print(f" existing files in repo: {sorted(names)}") except HfHubHTTPError as e: print(f" could not list existing files (will re-upload all): {e}", file=sys.stderr) - existing = set() + existing_sha256 = {} uploaded = 0 bytes_up = 0 @@ -458,15 +597,45 @@ def publish_variant(api: HfApi, variant: str, cells: Dict[str, CellMetrics], *, delay *= 2 else: raise - # README upload doesn't count in the 32 GGUF count, but we track bytes + # README upload doesn't count in the 44 GGUF count, but we track bytes bytes_up += len(card) uploaded += 1 - # Upload GGUFs + # Upload SHA256SUMS (always overwrite, same rationale as README) + print(f" uploading SHA256SUMS ({len(sha256sums_content)} bytes)") + delay = 2.0 + for attempt in range(3): + try: + api.upload_file( + path_or_fileobj=sha256sums_content.encode("utf-8"), + path_in_repo="SHA256SUMS", + repo_id=repo_id, + repo_type="model", + commit_message=f"Add/update SHA256SUMS for {variant}", + ) + print(f" ok") + break + except HfHubHTTPError as e: + print(f" attempt {attempt + 1}/3 failed: {e}", file=sys.stderr) + if attempt < 2: + time.sleep(delay) + delay *= 2 + else: + raise + bytes_up += len(sha256sums_content) + uploaded += 1 + + # Upload GGUFs — only skip a same-named remote file if its content hash + # matches ours; otherwise it's a stale (e.g. pre-1.9) artifact and gets + # overwritten. for p in local_files: - if p.name in existing: - print(f" -> {p.name} already in repo, skipping") + remote_sha = existing_sha256.get(p.name) + if remote_sha is not None and remote_sha == checksums[p.name]: + print(f" -> {p.name} already in repo with matching sha256, skipping") continue + if remote_sha is not None: + print(f" -> {p.name} exists remotely but sha256 differs " + f"(remote {remote_sha[:12]}... vs local {checksums[p.name][:12]}...); overwriting") upload_with_retry(api, p, p.name, repo_id) uploaded += 1 bytes_up += p.stat().st_size @@ -482,6 +651,13 @@ def publish_variant(api: HfApi, variant: str, cells: Dict[str, CellMetrics], *, def main() -> int: parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--hf-user", type=str, default=None, + help="HF namespace to publish under (e.g. your HF username). " + "If omitted, discovered from the authenticated HF identity " + "(hf whoami) and confirmed against a --dry-run pass.") + parser.add_argument("--github-repo", type=str, default=DEFAULT_GITHUB_REPO, + help=f"GitHub '/' used for links in the model card " + f"(default: {DEFAULT_GITHUB_REPO}; pass your fork once merged).") parser.add_argument("--dry-run", action="store_true", help="show what would happen without uploading") parser.add_argument("--only", type=str, default="", @@ -498,6 +674,10 @@ def main() -> int: return 2 metrics = load_metrics() + sweep_meta = load_sweep_meta() + commit_sha = _git_commit_sha() + rfdetr_version = sweep_meta.get("rfdetr_version") or _rfdetr_version() + print(f"provenance: commit={commit_sha[:12]} rfdetr=={rfdetr_version}") variants = VARIANTS if args.only: @@ -507,15 +687,20 @@ def main() -> int: print(f"error: --only={args.only} matched no known variants", file=sys.stderr) return 2 - # Auth check + # Auth check + hf-user resolution. Always authenticate (even in + # --dry-run) so a missing/expired token is caught before any upload. api = HfApi() - if not args.dry_run: - try: - me = api.whoami() - print(f"authenticated as: {me['name']}") - except Exception as e: - print(f"error: HF auth failed: {e}", file=sys.stderr) - return 2 + try: + me = api.whoami() + print(f"authenticated as: {me['name']}") + except Exception as e: + print(f"error: HF auth failed: {e}", file=sys.stderr) + return 2 + + hf_user = args.hf_user or me["name"] + if not args.hf_user: + print(f"--hf-user not given; using authenticated identity: {hf_user}") + print(f"publishing to namespace: {hf_user}/ (github links -> {args.github_repo})") results: List[Dict[str, object]] = [] t_start = time.time() @@ -526,12 +711,15 @@ def main() -> int: f"placeholder model card", file=sys.stderr) try: r = publish_variant(api, v, cells, + hf_user=hf_user, github_repo=args.github_repo, + commit_sha=commit_sha, rfdetr_version=rfdetr_version, + sweep_meta=sweep_meta, dry_run=args.dry_run, skip_existing=not args.no_skip_existing) results.append(r) except Exception as e: print(f"\n!!! variant {v} failed: {type(e).__name__}: {e}", file=sys.stderr) - results.append({"repo_id": f"{HF_USER}/rfdetr-cpp-{v}", "url": "", "uploaded_files": 0, "bytes": 0, "error": str(e)}) + results.append({"repo_id": f"{hf_user}/rfdetr-cpp-{v}", "url": "", "uploaded_files": 0, "bytes": 0, "error": str(e)}) elapsed = time.time() - t_start diff --git a/scripts/requirements.txt b/scripts/requirements.txt index 9089050..22da8ac 100644 --- a/scripts/requirements.txt +++ b/scripts/requirements.txt @@ -1,8 +1,9 @@ -# Pinned at last known-good version at time of writing. +# Pinned at the RF-DETR 1.9 Nano compatibility baseline. # Bump deliberately; re-run `python scripts/convert_rfdetr_to_gguf.py --dry-run` # to confirm the tensor name map still matches. torch==2.5.1 -rfdetr==1.7.0 +rfdetr==1.9.0 gguf==0.10.0 numpy==1.26.4 pillow==10.4.0 +sentencepiece==0.2.2 diff --git a/scripts/sweep_accuracy.py b/scripts/sweep_accuracy.py index 62d5945..aee29dd 100755 --- a/scripts/sweep_accuracy.py +++ b/scripts/sweep_accuracy.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Full accuracy + mask-quality sweep across the 32-model matrix. +"""Full accuracy + mask-quality sweep across the 44-model matrix. Compares C++ (rfdetr-cli detect) output against the PyTorch rfdetr reference for every (variant, quant) pair, over a fixed set of test images. Produces @@ -60,6 +60,9 @@ ("seg-nano", "RFDETRSegNano", "rfdetr-seg-nano", True), ("seg-small", "RFDETRSegSmall", "rfdetr-seg-small", True), ("seg-medium", "RFDETRSegMedium", "rfdetr-seg-medium", True), + ("seg-large", "RFDETRSegLarge", "rfdetr-seg-large", True), + ("seg-xlarge", "RFDETRSegXLarge", "rfdetr-seg-xlarge", True), + ("seg-2xlarge", "RFDETRSeg2XLarge", "rfdetr-seg-2xlarge", True), ] QUANTS = ["f32", "f16", "q8_0", "q4_K"] diff --git a/src/image_io.cpp b/src/image_io.cpp index 85160d5..f895c39 100644 --- a/src/image_io.cpp +++ b/src/image_io.cpp @@ -14,7 +14,9 @@ #define STBIR__HEADER_FILENAME "stb_image_resize.h" #include "stb_image_resize.h" +#include #include +#include #include #include #include @@ -182,6 +184,7 @@ bool rfdetr_encode_gray_png(const uint8_t* data, int width, int height, extern "C" rfdetr_status rfdetr_preprocess(const rfdetr_image* img, int target_w, int target_h, const float mean[3], const float std_[3], + bool bilinear_no_antialias, float** out_data, int* out_w, int* out_h) { if (!img || !out_data || !out_w || !out_h || !mean || !std_) { return RFDETR_ERR_INVALID_ARG; @@ -191,21 +194,16 @@ extern "C" rfdetr_status rfdetr_preprocess(const rfdetr_image* img, return RFDETR_ERR_INVALID_ARG; } - /* 1. Resize via stb_image_resize2 (linear-space bilinear). Input is uint8 RGB packed HWC. */ - std::vector resized; - try { - resized.assign((size_t)target_w * (size_t)target_h * 3, 0); - } catch (const std::bad_alloc&) { - return RFDETR_ERR_OUT_OF_MEMORY; - } - if (!stbir_resize_uint8_linear(img->rgb.data(), img->width, img->height, 0, - resized.data(), target_w, target_h, 0, - STBIR_RGB)) { - rfdetr_logf(RFDETR_LOG_ERROR, "rfdetr_preprocess: stbir_resize_uint8_linear failed"); - return RFDETR_ERR_IO; - } - - /* 2-3. Allocate output F32 buffer and write NCHW row-major. + /* Allocate the normalized F32 NCHW output, then use the resize path + * selected by GGUF metadata. + * + * RF-DETR 1.9 explicitly uses torchvision F.resize(..., antialias=False) + * for inference so it matches the cv2.INTER_LINEAR training pipeline. + * That is bilinear interpolation with align_corners=false (half-pixel + * source coordinates) and no intermediate uint8 rounding. stb's easy + * resize API instead selects Mitchell/cubic filters and rounds back to + * uint8, which materially shifts borderline detection scores. That stb + * path remains available for GGUFs created before the metadata was added. * * ggml ne = (W, H, 3, 1): ne[0]=W fastest-varying. Memory order: * offset(c, h, w) = c*H*W + h*W + w @@ -214,13 +212,60 @@ extern "C" rfdetr_status rfdetr_preprocess(const rfdetr_image* img, float* buf = (float*)std::malloc(n_elems * sizeof(float)); if (!buf) return RFDETR_ERR_OUT_OF_MEMORY; - for (int c = 0; c < 3; ++c) { + if (bilinear_no_antialias) { + const float scale_x = (float)img->width / (float)target_w; + const float scale_y = (float)img->height / (float)target_h; for (int h = 0; h < target_h; ++h) { + const float src_y = ((float)h + 0.5f) * scale_y - 0.5f; + const int y0_raw = (int)std::floor(src_y); + const int y1_raw = y0_raw + 1; + const int y0 = std::clamp(y0_raw, 0, img->height - 1); + const int y1 = std::clamp(y1_raw, 0, img->height - 1); + const float wy = src_y - (float)y0_raw; + for (int w = 0; w < target_w; ++w) { - uint8_t px = resized[(size_t)(h * target_w + w) * 3 + c]; - float v = (float)px / 255.0f; - v = (v - mean[c]) / std_[c]; - buf[(size_t)c * target_h * target_w + (size_t)h * target_w + w] = v; + const float src_x = ((float)w + 0.5f) * scale_x - 0.5f; + const int x0_raw = (int)std::floor(src_x); + const int x1_raw = x0_raw + 1; + const int x0 = std::clamp(x0_raw, 0, img->width - 1); + const int x1 = std::clamp(x1_raw, 0, img->width - 1); + const float wx = src_x - (float)x0_raw; + + for (int c = 0; c < 3; ++c) { + const float p00 = (float)img->rgb[((size_t)y0 * img->width + x0) * 3 + c]; + const float p01 = (float)img->rgb[((size_t)y0 * img->width + x1) * 3 + c]; + const float p10 = (float)img->rgb[((size_t)y1 * img->width + x0) * 3 + c]; + const float p11 = (float)img->rgb[((size_t)y1 * img->width + x1) * 3 + c]; + const float top = p00 + (p01 - p00) * wx; + const float bottom = p10 + (p11 - p10) * wx; + float v = (top + (bottom - top) * wy) / 255.0f; + v = (v - mean[c]) / std_[c]; + buf[(size_t)c * target_h * target_w + (size_t)h * target_w + w] = v; + } + } + } + } else { + std::vector resized; + try { + resized.assign((size_t)target_w * (size_t)target_h * 3, 0); + } catch (const std::bad_alloc&) { + std::free(buf); + return RFDETR_ERR_OUT_OF_MEMORY; + } + if (!stbir_resize_uint8_linear(img->rgb.data(), img->width, img->height, 0, + resized.data(), target_w, target_h, 0, + STBIR_RGB)) { + std::free(buf); + rfdetr_logf(RFDETR_LOG_ERROR, "rfdetr_preprocess: legacy stb resize failed"); + return RFDETR_ERR_IO; + } + for (int c = 0; c < 3; ++c) { + for (int h = 0; h < target_h; ++h) { + for (int w = 0; w < target_w; ++w) { + const uint8_t px = resized[((size_t)h * target_w + w) * 3 + c]; + float v = ((float)px / 255.0f - mean[c]) / std_[c]; + buf[(size_t)c * target_h * target_w + (size_t)h * target_w + w] = v; + } } } } diff --git a/src/image_io.hpp b/src/image_io.hpp index a0ca899..b5ca4c1 100644 --- a/src/image_io.hpp +++ b/src/image_io.hpp @@ -20,7 +20,9 @@ extern "C" { const uint8_t* rfdetr_image_rgb_data(const rfdetr_image* img); /* Preprocess an image for model input: - * 1. Resize to (target_w, target_h) using high-quality bilinear interpolation + * 1. Resize to (target_w, target_h). `bilinear_no_antialias=true` matches + * RF-DETR 1.9's float bilinear F.resize(..., antialias=False); + * false preserves the legacy stb filtering used by existing GGUFs. * 2. Convert uint8 RGB -> float32 in [0, 1] * 3. Apply ImageNet normalization: (pixel - mean) / std (per channel) * 4. Output in (W, H, 3, 1) ggml layout, NCHW row-major equivalent @@ -31,6 +33,7 @@ const uint8_t* rfdetr_image_rgb_data(const rfdetr_image* img); rfdetr_status rfdetr_preprocess(const rfdetr_image* img, int target_w, int target_h, const float mean[3], const float std_[3], + bool bilinear_no_antialias, float** out_data, int* out_w, int* out_h); /* Write a single-channel uint8 (grayscale) buffer as a PNG. Returns diff --git a/src/model_loader.cpp b/src/model_loader.cpp index 0bca283..622c4ec 100644 --- a/src/model_loader.cpp +++ b/src/model_loader.cpp @@ -264,6 +264,16 @@ Model* model_load(const std::string& path, rfdetr_status* out_status) { return fail(RFDETR_ERR_MODEL_FORMAT, "rfdetr.preprocess.mean missing or wrong shape"); if (!get_f32_array(gguf, "rfdetr.preprocess.std", c.preprocess_std, 3)) return fail(RFDETR_ERR_MODEL_FORMAT, "rfdetr.preprocess.std missing or wrong shape"); + std::string resize_mode; + if (get_str(gguf, "rfdetr.preprocess.resize_mode", resize_mode)) { + if (resize_mode == "bilinear_no_antialias") { + c.preprocess_bilinear_no_antialias = true; + } else if (resize_mode == "legacy_stb") { + c.preprocess_bilinear_no_antialias = false; + } else { + return fail(RFDETR_ERR_MODEL_FORMAT, "unsupported rfdetr.preprocess.resize_mode"); + } + } if (!get_u32(gguf, "rfdetr.backbone.dim", c.backbone.dim) || !get_u32(gguf, "rfdetr.backbone.depth", c.backbone.depth) || @@ -646,7 +656,10 @@ std::vector expected_tensor_names(const Config& cfg) { // --- Segmentation head (RFDETRSeg* variants only) --- if (cfg.has_segmentation_head) { - for (int b = 0; b < 4; ++b) { + // SegmentationHead chains one DepthwiseConvBlock per decoder layer + // (rfdetr.models.heads.segmentation.SegmentationHead): 4 for + // nano/small, 5 for medium/large, 6 for xlarge/2xlarge. + for (uint32_t b = 0; b < cfg.decoder.layers; ++b) { const std::string p = "segmentation_head.blocks." + std::to_string(b) + "."; names.emplace_back(p + "dwconv.weight"); names.emplace_back(p + "dwconv.bias"); diff --git a/src/model_loader.hpp b/src/model_loader.hpp index 4272689..d4b6f21 100644 --- a/src/model_loader.hpp +++ b/src/model_loader.hpp @@ -32,6 +32,10 @@ struct Config { std::vector class_names; float preprocess_mean[3] = {0, 0, 0}; float preprocess_std[3] = {1, 1, 1}; + /* Optional in format v2 for backward compatibility. Newly converted + * RF-DETR >=1.9 models set this to true; older GGUFs keep legacy stb + * resize behavior so their established outputs do not change. */ + bool preprocess_bilinear_no_antialias = false; struct { uint32_t dim = 0; diff --git a/src/rfdetr.cpp b/src/rfdetr.cpp index e823c0c..0383d40 100644 --- a/src/rfdetr.cpp +++ b/src/rfdetr.cpp @@ -131,6 +131,7 @@ extern "C" rfdetr_status rfdetr_detect(rfdetr_context* ctx, int px_w = 0, px_h = 0; rfdetr_status pp_st = rfdetr_preprocess(img, img_size, img_size, cfg.preprocess_mean, cfg.preprocess_std, + cfg.preprocess_bilinear_no_antialias, &px_data, &px_w, &px_h); if (pp_st != RFDETR_OK) { rfdetr_logf(RFDETR_LOG_ERROR, "rfdetr_detect: preprocess failed"); diff --git a/src/rfdetr_model.cpp b/src/rfdetr_model.cpp index 6e038e0..d2c0825 100644 --- a/src/rfdetr_model.cpp +++ b/src/rfdetr_model.cpp @@ -350,19 +350,18 @@ ForwardOutput rfdetr_model_forward(const Model& m, return out; } - /* SegmentationHead. For seg models with N_dec layers, the head has 4 - * DepthwiseConvBlocks that we iterate against the last 4 per-layer - * decoder outputs (when N_dec=4 that's all of them; when N_dec > 4 we - * keep zip-of-shortest semantics from PyTorch: zip(blocks, hs) walks 4 - * layers because there are 4 blocks). */ + /* SegmentationHead. SegmentationHead.__init__ constructs exactly one + * DepthwiseConvBlock per decoder layer (num_blocks == n_dec_layers), so + * PyTorch's zip(self.blocks, query_features) always pairs 1:1 with no + * truncation. nano/small have 4 decoder layers, medium/large have 5, + * xlarge/2xlarge have 6 — always iterate all of them. */ ggml_tensor* seg_masks_t = nullptr; if (has_seg) { - const int n_seg_iters = std::min(n_dec_layers, 4); seg_masks_t = segmentation_forward( gctxB, m, proj_in, - dec_per_layer.data(), // first n_seg_iters used - n_seg_iters, + dec_per_layer.data(), + n_dec_layers, /*image_h*/ input_size, /*image_w*/ input_size, (int)m.config.mask_downsample_ratio); if (!seg_masks_t) { diff --git a/src/segmentation.cpp b/src/segmentation.cpp index 36cbb52..b886e98 100644 --- a/src/segmentation.cpp +++ b/src/segmentation.cpp @@ -6,6 +6,7 @@ #include #include +#include namespace rfdetr { @@ -179,13 +180,6 @@ ggml_tensor* segmentation_forward( rfdetr_logf(RFDETR_LOG_ERROR, "segmentation_forward: invalid args"); return nullptr; } - if (n_layers != 4) { - rfdetr_logf(RFDETR_LOG_WARN, - "segmentation_forward: n_layers=%d, but SegmentationHead " - "has exactly 4 blocks. Will iterate min(n_layers, 4).", - n_layers); - } - const int target_h = image_h / mask_downsample_ratio; const int target_w = image_w / mask_downsample_ratio; @@ -204,11 +198,13 @@ ggml_tensor* segmentation_forward( publish("seg.spatial_features.resized", spatial); /* 2. Fetch all per-block weights up-front so we fail fast if any are - * missing. */ + * missing. SegmentationHead has one DepthwiseConvBlock per decoder + * layer (n_layers), not a fixed count. */ struct BlockW { ggml_tensor *dw_w, *dw_b, *n_w, *n_b, *pw_w, *pw_b; - } bw[4]; - for (int b = 0; b < 4; ++b) { + }; + std::vector bw((size_t)n_layers); + for (int b = 0; b < n_layers; ++b) { const std::string p = "segmentation_head.blocks." + std::to_string(b) + "."; bw[b].dw_w = fetch(m, p + "dwconv.weight"); bw[b].dw_b = fetch(m, p + "dwconv.bias"); @@ -237,11 +233,10 @@ ggml_tensor* segmentation_forward( return nullptr; } - /* 3. Iterate 4 blocks. The number of query streams is N (always 4 in - * practice). If fewer were provided we cap iteration. */ - const int n_iter = (n_layers < 4) ? n_layers : 4; + /* 3. Iterate all n_layers blocks — one per decoder layer, matching + * query_features_per_layer 1:1. */ ggml_tensor* masks_final = nullptr; - for (int b = 0; b < n_iter; ++b) { + for (int b = 0; b < n_layers; ++b) { spatial = depthwise_conv_block(ctx, spatial, bw[b].dw_w, bw[b].dw_b, bw[b].n_w, bw[b].n_b, @@ -302,7 +297,7 @@ ggml_tensor* segmentation_forward( mask = ggml_reshape_4d(ctx, mask, W, H, qf_proj->ne[1], 1); publish("seg.masks." + std::to_string(b), mask); - if (b == n_iter - 1) { + if (b == n_layers - 1) { masks_final = mask; } } diff --git a/src/segmentation.hpp b/src/segmentation.hpp index 85a9f1d..d667638 100644 --- a/src/segmentation.hpp +++ b/src/segmentation.hpp @@ -25,25 +25,28 @@ namespace rfdetr { * Inference uses masks[-1] (the last decoder layer's mask). For parity * verification we also publish each intermediate via the trace mechanism: * seg.spatial_features.resized - * seg.block.{0..3}.spatial_out - * seg.block.{0..3}.spatial_proj - * seg.block.{0..3}.qf_proj - * seg.masks.{0..3} + * seg.block.{0..n_layers-1}.spatial_out + * seg.block.{0..n_layers-1}.spatial_proj + * seg.block.{0..n_layers-1}.qf_proj + * seg.masks.{0..n_layers-1} * seg.masks.final * * Inputs (ggml memory layout): * spatial_features ne = (W, H, C=256, 1) — projector output BEFORE resize * query_features_per_layer ne = (C=256, NQ, 1) per layer; pass `n_layers` - * tensors via the array. The seg head's number of - * blocks (always 4) must match the number of - * provided layers — if you only have the last - * layer post-norm tensor, pass it 4 times to + * tensors via the array. SegmentationHead.__init__ + * constructs exactly one DepthwiseConvBlock per + * decoder layer, so the block count always equals + * n_layers — 4 for nano/small, 5 for medium/large, + * 6 for xlarge/2xlarge. If you only have the last + * layer post-norm tensor, pass it n_layers times to * reproduce the export-time (skip_blocks=False) * behaviour, but **the resulting masks[-1] will * NOT match the inference output** because the - * spatial features get processed by 4 different - * query streams in the real forward. For parity - * pass exactly 4 per-layer tensors. + * spatial features get processed by n_layers + * different query streams in the real forward. + * For parity pass exactly n_layers per-layer + * tensors. * * Output (ggml memory layout): * masks_final ne = (W_mask, H_mask, NQ, 1) — raw logits diff --git a/tests/test_image_io.cpp b/tests/test_image_io.cpp index 865ed87..b4e25db 100644 --- a/tests/test_image_io.cpp +++ b/tests/test_image_io.cpp @@ -134,7 +134,8 @@ int main() { float* data = nullptr; int w = 0, h = 0; - rfdetr_status pp_st = rfdetr_preprocess(img, 56, 56, mean, std_, &data, &w, &h); + rfdetr_status pp_st = rfdetr_preprocess( + img, 56, 56, mean, std_, false, &data, &w, &h); RFDETR_ASSERT_EQ_INT(pp_st, RFDETR_OK); RFDETR_ASSERT_EQ_INT(w, 56); RFDETR_ASSERT_EQ_INT(h, 56); @@ -151,5 +152,39 @@ int main() { rfdetr_image_free(img); } + /* RF-DETR 1.9 preprocessing: float bilinear, align_corners=false, + * antialias=false, and no intermediate uint8 rounding. */ + { + const uint8_t rgb[] = { + 0, 10, 20, 100, 110, 120, + 200, 210, 220, 255, 250, 245, + }; + rfdetr_status st; + rfdetr_image* img = rfdetr_image_from_rgb_buffer(rgb, 2, 2, &st); + RFDETR_ASSERT(img != nullptr); + RFDETR_ASSERT_EQ_INT(st, RFDETR_OK); + + const float mean[3] = {0.0f, 0.0f, 0.0f}; + const float std_[3] = {1.0f, 1.0f, 1.0f}; + float* data = nullptr; + int w = 0, h = 0; + rfdetr_status pp_st = rfdetr_preprocess( + img, 3, 3, mean, std_, true, &data, &w, &h); + RFDETR_ASSERT_EQ_INT(pp_st, RFDETR_OK); + RFDETR_ASSERT_EQ_INT(w, 3); + RFDETR_ASSERT_EQ_INT(h, 3); + + /* Center is the average of all four source pixels. Channels are + * planar, so the center offsets are 4, 13, and 22. */ + RFDETR_ASSERT_NEAR(data[4], 138.75f / 255.0f, 1e-6); + RFDETR_ASSERT_NEAR(data[13], 145.00f / 255.0f, 1e-6); + RFDETR_ASSERT_NEAR(data[22], 151.25f / 255.0f, 1e-6); + RFDETR_ASSERT_NEAR(data[1], 50.00f / 255.0f, 1e-6); + RFDETR_ASSERT_NEAR(data[8], 255.00f / 255.0f, 1e-6); + + std::free(data); + rfdetr_image_free(img); + } + return 0; }