feat: stream s3 images and allow new dataset options for inference - #16
Open
FerretCode wants to merge 24 commits into
Open
feat: stream s3 images and allow new dataset options for inference#16FerretCode wants to merge 24 commits into
FerretCode wants to merge 24 commits into
Conversation
…ttings Co-authored-by: multica-agent <github@multica.ai>
Adds a JSON parser (images/annotations/categories -> bbox records) and dispatches by file extension/content sniffing in the existing map-kwcoco-csv route so uploads of either .csv or .json are handled by the same endpoint. Co-authored-by: multica-agent <github@multica.ai>
feat: support KW COCO JSON alongside CSV in annotation pre-processing
getAllClasses/getAllImages/getMaxLabelId resolve to {success, rows}
(matching queries/client.js's db.all()/db.get() contract used
elsewhere, e.g. routes/inference/yoloInference.js), not a bare array,
so existingClassRows.map threw at runtime. The integration test mock
returned plain arrays instead of the real {rows: [...]} shape, which
is why 14/14 passed despite the bug; the mock now matches the real
client contract.
Co-authored-by: multica-agent <github@multica.ai>
…3a50e7e fix: use .rows when reading Client.all() results in mapKwCocoCsv
feat: add KW COCO CSV pre-processing annotation mapping
Lays the groundwork for serving images from S3 on demand instead of requiring a full local sync: an image row can now record where its bytes live. Existing projects are backfilled via a PRAGMA-guarded ALTER TABLE in migrateProjectDb, since CREATE TABLE IF NOT EXISTS is a no-op on tables that already exist. Co-authored-by: multica-agent <github@multica.ai>
syncS3Bucket tracked "already synced" purely by sanitized basename, so two distinct keys under different prefixes (e.g. 2024/img.jpg and 2025/img.jpg) collided and the second was silently skipped. Adds a nullable Images.SourceKey column (parallel to Source, same PRAGMA-guarded backfill) to record the literal S3 key an image came from, decoupled from its display IName. syncS3Bucket now treats "already synced" as "we've recorded this exact key" rather than "a file with this name exists," and only disambiguates (via a deterministic hash of the key) when two keys from the same bucket listing genuinely collide on basename - a name that already existed on disk before the run is still skipped as before. Co-authored-by: multica-agent <github@multica.ai>
migrateProjectDb now runs for every existing project at server boot (added alongside the Source/SourceKey columns), and every statement in it is DDL - even a no-op CREATE TABLE IF NOT EXISTS still requires SQLite to open the file for write. A read-only project database can never be migrated regardless of the reason it's read-only, so treat that specific, expected outcome as a warning instead of an error-level log repeated on every single server start. Co-authored-by: multica-agent <github@multica.ai>
Root-caused the SQLITE_READONLY errors on startup migration: the app image has no USER directive, so the container writes to the bind-mounted public/projects (and db, data) directories as root. Anything created while running via docker-compose then can't be written by a normal host user running the app outside Docker (`node .`), which is exactly what surfaced as SQLITE_READONLY for test-gcp-test/test-gcp-test3. Mapping the container to the host UID/GID prevents this going forward; already-affected directories still need a one-time chown. Co-authored-by: multica-agent <github@multica.ai>
Adds a SyncMode ("download" | "stream") on S3Buckets, chosen via a new
checkbox in the S3 project-creation form. "download" keeps today's
behavior unchanged. "stream" registers each object as a project image
without copying it to local disk, and adds an on-demand image proxy
(GET /api/v2/projects/:admin/:projectName/images/:imageName) that
serves a local file if present, or otherwise fetches the object live
from S3 for that one request - never persisting it.
getAnnotatePage (the per-image labeling/annotation view) now falls
back to that proxy and probes the object's dimensions over the same
live fetch (aborted early once the header is read, via
probe-image-size's stream support) instead of 404ing when an image
was never downloaded.
Co-authored-by: multica-agent <github@multica.ai>
… up to max limit Co-authored-by: multica-agent <github@multica.ai>
…ith max image limit Co-authored-by: multica-agent <github@multica.ai>
… options Co-authored-by: multica-agent <github@multica.ai>
Co-authored-by: multica-agent <github@multica.ai>
…c/inference run request payload Co-authored-by: multica-agent <github@multica.ai>
…re present Co-authored-by: multica-agent <github@multica.ai>
…tle safeguards Co-authored-by: multica-agent <github@multica.ai>
…erence-options Co-authored-by: multica-agent <github@multica.ai>
… sync mode selection into inference pipeline Co-authored-by: multica-agent <github@multica.ai>
…views and dropdown menus Co-authored-by: multica-agent <github@multica.ai>
…Inference to handle streaming mode images Co-authored-by: multica-agent <github@multica.ai>
…runs Co-authored-by: multica-agent <github@multica.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.