feat: SBOM attestation, dataset collection fix, migration script

.github/workflows/deploy.yml

@@ -72,6 +72,22 @@ jobs:
             --certificate-identity-regexp="github.com/Pyronewbic/casecomp" \
             "${{ env.IMAGE }}@${{ steps.digest.outputs.digest }}" || true
 
+      - name: Generate container SBOM (Syft)
+        uses: anchore/sbom-action@v0
+        with:
+          image: "${{ env.IMAGE }}@${{ steps.digest.outputs.digest }}"
+          format: spdx-json
+          output-file: /tmp/sbom.spdx.json
+          upload-artifact: false
+
+      - name: Attest SBOM to image
+        run: |
+          cosign attest --yes \
+            --oidc-issuer=https://token.actions.githubusercontent.com \
+            --predicate /tmp/sbom.spdx.json \
+            --type spdxjson \
+            "${{ env.IMAGE }}@${{ steps.digest.outputs.digest }}"
+
       - name: Attest SLSA provenance
         run: |
           cat > /tmp/provenance.json << 'PROV'

CLAUDE.md

@@ -69,7 +69,7 @@ Strict palette — no deviations:
 - **Grade probability distribution:** `gradeDistribution` field in response (e.g. `{"8": 65, "8.5": 12, "7.5": 23}`). Computed from overall + confidence.
 - **Centering hint:** POST `/api/grade` accepts optional `centeringHint` with user-measured ratios, appended to centering prompts.
 - **Shareable reports:** `GET /api/grade/report/:id` returns PNG card (SVG→sharp→PNG) with scores, distribution, limiter.
-- **ML dataset pipeline:** track-prices passively collects graded slab images (eBay sold) into `grading-dataset` Firestore collection. Parses PSA/BGS/CGC/TAG grade from listing title. `GET /api/grading-dataset/stats` (owner-only) monitors collection.
+- **ML dataset pipeline:** graded slab images passively collected into `grading-dataset` Firestore collection from eBay sold (track-prices + /api/sold), magi sold (track-prices), and search sold results (/api/search). Parses PSA/BGS/CGC/TAG grade from listing title or grade label. `GET /api/grading-dataset/stats` (owner-only) monitors collection.
 - **Roadmap:** multi-pass median for deterministic grading, defect heatmap overlay, accuracy calibration once dataset reaches ~500 images.
 
 ## Set browser
@@ -112,9 +112,9 @@ Strict palette — no deviations:
 - GCP: Cloud Run in asia-south1 + us-central1, Firestore (asia-south1), HTTPS LB (global, geo-routes), Cloud CDN, Secret Manager, Cloud Scheduler
 - Terraform: GCS state, 8 files by resource type, CI plan/apply, `for_each` over regions
 - **CI (ci.yml):** single workflow. Jobs: unit, api (demo-based, continue-on-error), smoke (non-blocking), codeql, scan (SBOM+Grype), audit (npm+lockfile-lint), secrets (gitleaks). Required: unit + codeql.
-- **Deploy:** Kaniko v1.23.2 --reproducible → cosign sign → SLSA provenance → deploy by digest → both regions → health check → OWASP ZAP DAST
+- **Deploy:** Kaniko v1.23.2 --reproducible → cosign sign → SBOM attest (Syft SPDX from container) → SLSA provenance attest → deploy by digest → both regions → health check → OWASP ZAP DAST
 - **Custom Wolfi base image:** gcr.io/casecomp-495718/casecomp-node24. Built with apko. 9 smoke tests. 0 CVEs.
-- **Supply chain:** Dependabot, lockfile-lint, Socket.dev, pre-commit hook (blocks .env, secrets, large files)
+- **Supply chain:** SBOM + SLSA attestations on image digest, Dependabot, lockfile-lint, Socket.dev, pre-commit hook (blocks .env, secrets, large files)
 - **Binary Authorization:** ENFORCED on both Cloud Run services
 - **Secret workflow:** Add to secrets.tf → CI creates → `gcloud secrets versions add` for value. Never `gcloud secrets create`.
 - Secrets: EBAY_CLIENT_ID/SECRET, ANTHROPIC_API_KEY, TOGETHER_API_KEY, PSA_AUTH_TOKEN, CASECOMP_API_KEY, CASECOMP_SANDBOX_KEY, RESEND_API_KEY, CASECOMP_JWT_SECRET, GOOGLE_OAUTH_CLIENT_ID, CASECOMP_ADMIN_SUB

README.md

@@ -122,15 +122,15 @@ curl -X POST -H "Authorization: Bearer $CASECOMP_KEY" \
 
 - **Container signing** - Sigstore cosign keyless signing via GitHub OIDC, logged to Rekor transparency log
 - **Deploy by digest** - immutable image SHA, not mutable `:latest` tag
-- **SBOM** - Syft SPDX JSON generated per deploy, uploaded as build artifact
+- **SBOM attestation** - Syft SPDX JSON generated from built container image, cosign-attested to image digest, also uploaded as build artifact
 - **Vulnerability scanning** - Grype scans SBOM for CVEs, SARIF uploaded to GitHub Security tab
 - **SAST** - CodeQL static analysis on every PR + weekly schedule
 - **Binary Authorization** - GCP policy enforced on both Cloud Run services
 - **Reproducible builds** - Kaniko `--reproducible` flag, pinned version
 - **Multi-region** - Cloud Run in asia-south1 + us-central1, global LB auto geo-routes to nearest
 - **Custom base image** - Wolfi + apko Node 24 image, manual rebuild, zero CVEs by design
 - **RASP** - runtime request inspection for SQLi, XSS, command injection, path traversal, NoSQL injection, prototype pollution. Per-IP anomaly scoring, bot fingerprinting, Firestore event logging
-- **Supply chain** - SLSA provenance, Dependabot, lockfile-lint, Socket.dev, pre-commit secret blocking
+- **Supply chain** - SLSA provenance attestation, Dependabot, lockfile-lint, Socket.dev, pre-commit secret blocking
 
 ## Claude Code Skills
 
@@ -212,12 +212,12 @@ Load unpacked from `extension/` in `chrome://extensions`.
 
 ## Tests
 
-434 tests across three layers. CI required checks: unit + codeql. Smoke is non-blocking.
+486 tests across three layers. CI required checks: unit + codeql. Smoke is non-blocking.
 
 | Suite | Count | Command | Covers |
 |-------|------:|---------|--------|
-| **Unit** | 266 | `yarn test:unit` | Filters, grading, query builder, card identity, condition detection, image preprocessing, email alerts, portfolio ROI, CSV export, autocomplete, JWT auth, price trends |
-| **API** | 97 | `yarn test:api` | Search, sold, PSA, grade, auth, admin keys, arbitrage, price history, alerts, share pages, portfolio CRUD, card view, upload-url, analytics, collection tracking |
+| **Unit** | 312 | `yarn test:unit` | Filters, grading, query builder, card identity, condition detection, image preprocessing, email alerts, portfolio ROI, CSV export, autocomplete, JWT auth, price trends, RASP detection |
+| **API** | 103 | `yarn test:api` | Search, sold, PSA, grade, auth, admin keys, arbitrage, price history, alerts, share pages, portfolio CRUD, card view, upload-url, analytics, collection tracking |
 | **Smoke** | 71 | `yarn test:smoke` | API root page, detail panel, tabs, PSA stats, arbitrage, mobile viewport, portfolio, autocomplete, search filters |
 
 ## Contributing

api.js

@@ -366,7 +366,10 @@ app.get("/api/search", apiAuthMiddleware, (req, res, next) => { req._errorType =
       result.counts.activeTotal = Object.values(result.activeByCountry).reduce((n, arr) => n + arr.length, 0);
     }
 
-    if (result.sold?.length) recordSoldPrices(q, result.sold, result.source).catch(() => {});
+    if (result.sold?.length) {
+      recordSoldPrices(q, result.sold, result.source).catch(() => {});
+      saveGradedImages(result.sold, result.source).catch(() => {});
+    }
     getOrCreateCard(q, { source: result.source, lang: config.language }).catch(() => {});
     trackSearchFrequency(q).catch(() => {});
     res.json(result);
@@ -414,6 +417,7 @@ app.get("/api/sold", apiAuthMiddleware, (req, res, next) => { req._errorType = "
       soldSource = soldRes.source;
     }
 
+    if (sold.length) saveGradedImages(sold, soldSource).catch(() => {});
     res.json({ query: q, sold, soldSource, counts: { sold: sold.length } });
   } catch (e) {
     logError(req._errorType || "api", e.message, req.originalUrl, req.requestId);
@@ -2090,6 +2094,7 @@ app.post("/api/track-prices", authMiddleware, async (req, res) => {
         magiSold = magiRes.sold || [];
         if (magiSold.length) {
           await recordSoldPrices(card, magiSold, "magi");
+          saveGradedImages(magiSold, "magi").catch(() => {});
         }
       } catch (e) {
         logError("track-prices", `Magi fetch failed for "${card}": ${e.message}`, "/api/track-prices");

docs/internals.md

@@ -26,7 +26,7 @@ lib/
     card-database.js  TCGdex card DB (29K EN+JP cards), set browser, rarity
     card-identity.js  Canonical IDs, set resolution, SET_TOTAL_MAP
     demo.js           Sample data (3 multi-source cards)
-    grading-dataset.js  ML slab image collection from eBay sold listings
+    grading-dataset.js  ML slab image collection from sold listings (eBay, magi, search)
     price-history.js  Sold comp tracking + TCGPlayer seeding
   data/
     analytics.js      Request analytics (Firestore, 30d TTL)
@@ -50,8 +50,8 @@ public/admin/         Admin panel (keys, stats, errors)
 extension/            Chrome extension: queue auto-join, drop intel
 terraform/            GCP infra (Cloud Run, Firestore, LB, CDN, Scheduler)
 test/
-  unit-test.js        266 unit tests
-  api-test.js         97 API integration tests
+  unit-test.js        312 unit tests
+  api-test.js         103 API integration tests
   smoke-test.js       71 Playwright UI smoke tests
 ```
 
@@ -82,7 +82,7 @@ Both `casecomp-api` and `casecomp-site` run in asia-south1 (Mumbai) and us-centr
 | Artifact Registry (frontend) | us (multi-region) | `us-docker.pkg.dev`, accessible from both regions |
 | GCR (API) | us (multi-region) | `gcr.io`, accessible globally |
 
-Deploy workflow: build once → cosign sign → deploy to both regions via GitHub Actions matrix (parallel, fail-fast: false).
+Deploy workflow: build once → cosign sign → SBOM attest → SLSA attest → deploy to both regions via GitHub Actions matrix (parallel, fail-fast: false) → health check → ZAP DAST.
 
 ## Caching
 
@@ -153,7 +153,7 @@ Use `--refresh` to delete all cache files before a run.
 | `api-analytics` | userId + ts desc | Per-user analytics |
 | `price-history` | cardKey + recordedAt desc | Card price history |
 
-**ML dataset pipeline**: `track-prices` passively saves graded slab images (PSA/BGS/CGC/TAG) from eBay sold listings into `grading-dataset` Firestore collection. `GET /api/grading-dataset/stats` monitors progress.
+**ML dataset pipeline**: graded slab images (PSA/BGS/CGC/TAG) are passively collected into `grading-dataset` Firestore collection from multiple sources: eBay sold (via `track-prices` and `/api/sold`), magi sold (via `track-prices`), and any search with sold results (`/api/search`). Grade is parsed from listing title or grade label. `GET /api/grading-dataset/stats` monitors collection progress.
 
 ## Security pipeline
 
@@ -163,8 +163,8 @@ Three workflows: `ci.yml` (all checks), `deploy.yml` (build + sign + deploy), `t
 
 | Job | What | Required? |
 |-----|------|-----------|
-| unit | 172 unit tests | Yes |
-| smoke | 74 Playwright smoke tests | No (continue-on-error) |
+| unit | 312 unit tests | Yes |
+| smoke | 71 Playwright smoke tests | No (continue-on-error) |
 | codeql | SAST for JavaScript/TypeScript | Yes |
 | scan | SBOM (Syft) + Grype vulnerability scan | No |
 | audit | npm audit + lockfile-lint | No |
@@ -176,7 +176,8 @@ Three workflows: `ci.yml` (all checks), `deploy.yml` (build + sign + deploy), `t
 |------|------|
 | Kaniko v1.23.2 | Build with `--reproducible`, dual tags |
 | Cosign sign | Keyless signing via GitHub OIDC → Sigstore Rekor |
-| Cosign attest | SLSA provenance attestation |
+| SBOM attest | Syft SPDX JSON from container image, cosign-attested to digest |
+| SLSA attest | Provenance attestation (builder, source, commit, entrypoint) |
 | Deploy | Matrix deploy to asia-south1 + us-central1 |
 
 **Other tools:**

lib/cards/grading-dataset.js

@@ -16,15 +16,16 @@ export async function saveGradedImages(items, source) {
   const batch = fs.batch();
 
   for (const item of items) {
-    if (!item.listingGradeLabel || !item.imageUrl) continue;
+    if (!item.imageUrl) continue;
 
-    const gradeMatch = item.listingGradeLabel.match(/(?:PSA|BGS|CGC|TAG)\s*(\d+\.?\d*)/i);
+    const gradeSource = item.listingGradeLabel || item.title || "";
+    const gradeMatch = gradeSource.match(/(?:PSA|BGS|CGC|TAG)\s*(\d+\.?\d*)/i);
     if (!gradeMatch) continue;
 
     const grade = parseFloat(gradeMatch[1]);
     if (grade < 1 || grade > 10) continue;
 
-    const provider = item.listingGradeLabel.match(/PSA|BGS|CGC|TAG/i)?.[0]?.toUpperCase() || "UNKNOWN";
+    const provider = gradeSource.match(/PSA|BGS|CGC|TAG/i)?.[0]?.toUpperCase() || "UNKNOWN";
     const docId = `${source}_${item.itemId || Date.now()}_${saved}`;
 
     batch.set(fs.collection(COLLECTION).doc(docId), {