v2.8.5: oversized-image false positives, worker DB pool, frame check

.github/workflows/test.yml

@@ -12,10 +12,23 @@ permissions:
 jobs:
   test:
     runs-on: ubuntu-latest
+    env:
+      SECRET_KEY: test-secret-key-for-github-actions
     strategy:
       matrix:
         python-version: ["3.10", "3.11", "3.12"]
 
+    services:
+      postgres:
+        image: postgres:15
+        env:
+          POSTGRES_PASSWORD: pixelprobe_test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd pg_isready --health-interval 10s
+          --health-timeout 5s --health-retries 5
+
     steps:
     - uses: actions/checkout@v4
 
@@ -46,10 +59,15 @@ jobs:
 
     - name: Run tests with pytest
       env:
-        SECRET_KEY: test-secret-key-for-github-actions
+        PIXELPROBE_TEST_POSTGRES_URI: postgresql://postgres:pixelprobe_test@localhost:5432/postgres
       run: |
         pytest -m "not real_media" --cov=pixelprobe --cov-report=xml --cov-report=term
 
+    - name: Run real-media corruption detection tests
+      if: matrix.python-version == '3.12'
+      run: |
+        pytest -m real_media -v
+
     - name: Upload coverage to Codecov
       uses: codecov/codecov-action@v5
       with:

CHANGELOG.MD

@@ -5,6 +5,27 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0).
 
+## [2.8.5] - 2026-08-19
+
+### Fixed
+
+- **Valid oversized images are no longer reported corrupted.** Any image past Pillow's decompression-bomb threshold (~358MP at defaults) fell through to ImageMagick, whose `cache resources exhausted` exit landed in the generic corruption branch, so a valid 625MP scan was flagged the same as real pixel damage. Both guards are resource limits, not corruption evidence; they now produce a warning verdict following the existing HEIC libheif pattern.
+- **Parallel file-list scans no longer share one database connection across worker threads.** The scan-files API path (default `num_workers=4`) fanned a single PixelProbe instance across a thread pool while its engine used StaticPool, one raw psycopg2 connection that psycopg2 forbids using concurrently; the symptom was sporadic save and cache failures. The worker engine now uses a QueuePool sized to the worker count, and a postgres-backed regression test fails against the old pool.
+
+### Changed
+
+- **Frame integrity check: packets first, decode only to confirm, warning verdict.** Stage 1 paid a full sequential `-count_frames` decode on every video and then discarded the result - ffprobe's csv column order never matched what the parser expected under ffmpeg 8, so the check has been silently inert while costing minutes on large files. It now compares the near-free packet count first and runs the full decode only on a mismatch. A confirmed mismatch produces a warning, not a corruption verdict: container framerate metadata lies on sparse-video files (a 240s QuickTime fixture with 244 real frames declares 25fps), and real decode damage is caught by the deep-decode stage.
+
+### Removed
+
+- Dead `ScanExecutor`/`BatchProcessor` (`pixelprobe/services/scan_executor.py`), superseded by the Celery task path; its only callers were its own tests.
+
+### Tests
+
+- **Fixture corpus repaired.** Four committed "valid" samples (3gp, flv, mpg, wmv) were 189-byte HTML error pages from failed downloads; valid.mkv failed h264 decode under ffmpeg 8; valid.webp failed both PIL and ImageMagick; six "corrupted" samples (mp3, aiff, jpg, png, gif, bmp) decode cleanly under modern tools because the FFmpeg-bug-tracker bugs they exercised were in FFmpeg, not the files. All are replaced by a committed deterministic generator script, with per-format detection expectations documented (mpg and gif are warning-level by design).
+- **real_media suite made runnable and wired into CI.** The scan fixture rescanned every sample per test and blew each test's timeout, so the suite could never complete - and CI always deselected the marker, which is how the fixture rot went unnoticed. Scans are now session-cached, the tautological `scan_status == 'completed'` assertion clause is gone, and CI runs the suite on the Python 3.12 leg with a PostgreSQL service for the concurrency test.
+- New synthetic corruption matrix (truncation at header/mid/tail, zero-byte files, PNG-renamed-to-JPG, scattered mdat damage, SVG/PSD/progressive-JPEG/animated-WebP, symlink-loop discovery) plus oversized-image regression tests.
+
 ## [2.8.4] - 2026-08-18
 
 ### Fixed

docs/glossary.md

@@ -18,15 +18,15 @@ Every term PixelProbe uses, defined once and linked to the doc that covers it.
 ## Validation verdicts
 
 - **Healthy** - The file decoded and validated without corruption signals. Benign decoder noise (NAL unit warnings, DTS/PTS timestamp warnings, ffmpeg 8 Opus EOF parse notices) does not affect this verdict.
-- **Corrupted** - A corruption signal with a verdict fired: FFmpeg validation failure, frame integrity mismatch, JPEG pixel corruption, or a decode error flood. See [Scan Types](scan-types.md).
-- **Warning** - A signal that is informative but does not prove damage: freeze events, elevated TOUT or VREP, strict-decode notices. Warning files play back fine in most cases. See [Scan Types](scan-types.md).
+- **Corrupted** - A corruption signal with a verdict fired: FFmpeg validation failure, JPEG pixel corruption, or a decode error flood. See [Scan Types](scan-types.md).
+- **Warning** - A signal that is informative but does not prove damage: freeze events, frame-count mismatches, elevated TOUT or VREP, strict-decode notices, tool resource limits on oversized images. Warning files play back fine in most cases. See [Scan Types](scan-types.md).
 - **Marked as good** - A manual override: the file keeps its scan history but is treated as healthy in stats and filters.
 - **Error** - The file could not be read or scanned at all (permissions, I/O failure, unreadable media).
 
 ## Deep checks
 
 - **Enhanced corruption analysis** - The staged deep check for video files: Stage 1 frame integrity, Stage 2 temporal outliers, Stage 3 multi-point sampling, Stage 4 strict error detection. See [How It Works](how-it-works.md).
-- **Frame integrity check (Stage 1)** - Compares the decodable frame count against the expected count from duration and frame rate.
+- **Frame integrity check (Stage 1)** - Compares the counted packets (confirmed by a decode when they disagree by more than 5%) against the count expected from duration and frame rate. Warning-only: container metadata is unreliable on sparse-video and variable-frame-rate files.
 - **Temporal outlier check (Stage 2)** - Samples three 10-second windows at 25/50/75% of the file and computes signalstats TOUT/VREP percentages. Warning-only.
 - **TOUT (temporal outliers)** - A signalstats metric flagging pixels that differ from both temporal neighbors. Film grain triggers it on clean content, so it warns rather than condemns.
 - **VREP (vertical line repetition)** - A signalstats metric from analog-tape QC; high values are normal in flat or graphic digital content, so it warns rather than condemns.

docs/how-it-works.md

@@ -157,7 +157,6 @@ api/
 - `MaintenanceService`: Database maintenance
 - `NotificationService`: Notification provider dispatch and rule evaluation
 - `HealthcheckService`: Outbound healthcheck pings for scheduled scans
-- `ScanExecutor`: Bounded thread-pool execution for selected-file rescans
 - `scan_engine`: Celery-free core of the chunk-distributed scan engine (scan-slot claim, chunk building, finalization)
 - `scan_reporting`: Scan report creation
 
@@ -429,7 +428,7 @@ Each file is validated in `pixelprobe/media_checker.py` (`PixelProbe.scan_file`)
    - ffprobe metadata probe (stream presence, codec, duration)
    - Full remux validation: FFmpeg reads the ENTIRE file with `-map 0 -c copy -f null -` and aggressive error detection to validate container integrity across all streams
    - Enhanced corruption analysis, run for every video:
-     - **Stage 1 - Frame integrity** (always): frame count verified against duration and framerate via `ffprobe -count_frames`; this is the authoritative deep corruption check
+     - **Stage 1 - Frame integrity** (always, warning-only): the packet count from a demux-only `ffprobe -count_packets` pass is compared against duration and framerate; a mismatch above 5% is confirmed with a full `-count_frames` decode before a warning is recorded. Never a corruption verdict - container framerate metadata lies on sparse-video and VFR files
      - **Stage 2 - Temporal outlier detection** (files > 1GB): sampled decode windows checked for timing anomalies; can mark corrupt or warn
      - **Stage 3 - Multi-point sampling** (files > 5GB): decodes 10s samples at beginning, middle, and end; NEVER marks a file corrupted (seeking produces FFmpeg-version-dependent false positives), results are informational
      - **Stage 4 - Strict error detection** (warnings only): `-err_detect crccheck+bitstream+buffer+explode` over the first 30 seconds; findings are container/muxing warnings, never corruption verdicts

docs/project-structure.md

@@ -46,7 +46,6 @@ PixelProbe/
 |   |   +-- maintenance_service.py
 |   |   +-- notification_service.py
 |   |   +-- scan_engine.py        # Chunk building, scan finalization
-|   |   +-- scan_executor.py
 |   |   +-- scan_reporting.py     # Scan reports, batch file inserts
 |   |   +-- scan_service.py
 |   |   `-- stats_service.py

docs/testing-guide.md

@@ -23,9 +23,11 @@ tests/
 |-- test_migration_lock.py         # Advisory-lock migration coordination
 |-- test_performance.py            # Performance-oriented tests
 |-- test_read_timeout.py           # Unreadable-file / read timeout handling
+|-- test_postgres_concurrency.py   # Marker-gated parallel-scan test against real Postgres
 |-- test_real_media_samples.py     # Marker-gated real media parser tests
 |-- test_scheduler.py              # Scheduled scan management
 |-- test_security_fixes.py         # Security regression tests
+|-- test_synthetic_corruption.py   # Marker-gated synthetic damage matrix (truncation, mid-stream, format confusion)
 |-- unit/                          # Unit tests for individual components
 |   |-- test_bitrot_classification.py
 |   |-- test_celery_settings.py
@@ -57,10 +59,13 @@ tests/
 
 Two files deserve a call-out:
 
-- `test_real_media_samples.py` is gated behind the `real_media` marker. It
-  exercises the FFmpeg/ImageMagick stderr parsers against the real sample
-  corpus and is sensitive to tool versions, so it is excluded from the
-  default local run and executed in CI inside the Docker image instead.
+- `test_real_media_samples.py` and `test_synthetic_corruption.py` are gated
+  behind the `real_media` marker. They exercise the FFmpeg/ImageMagick
+  validation paths against the sample corpus (plus fixtures synthesized at
+  test time) and are sensitive to tool versions, so they are excluded from
+  the default local run; CI runs them on the Python 3.12 matrix leg.
+  Committed synthesized fixtures are regenerated with
+  `tests/fixtures/media_samples/generate_corrupted_fixtures.py`.
 - `test_frontend_build.py` actually runs `npm install` and `npm run build`,
   so it needs Node.js 20 and npm available.
 
@@ -107,6 +112,7 @@ Markers are declared in `pytest.ini`:
 | Marker | Meaning |
 |--------|---------|
 | `real_media` | Requires the real media sample corpus and matching tool versions; deselect locally with `-m "not real_media"` |
+| `postgres` | Requires a live PostgreSQL; set `PIXELPROBE_TEST_POSTGRES_URI`, otherwise skipped |
 | `slow` | Long-running tests; deselect with `-m "not slow"` |
 | `integration` | Integration tests |
 | `timeout` | Sets a per-test execution timeout |

pixelprobe/api/scan_routes.py

@@ -912,7 +912,14 @@ def scan_files_parallel():
 
     data = request.get_json() or {}
     force_rescan = data.get('force_rescan', False)
-    num_workers = data.get('num_workers', 4)
+    # Cap caller-supplied worker counts: num_workers sizes thread pools and
+    # the checker's DB connection pool, so an uncapped value translates
+    # directly into PostgreSQL connections
+    try:
+        num_workers = int(data.get('num_workers', 4))
+    except (TypeError, ValueError):
+        num_workers = 4
+    num_workers = max(1, min(num_workers, current_app.config.get('MAX_WORKERS', 10)))
     scan_dirs = data.get('directories', [])
     file_paths = data.get('file_paths', [])