Skip to content

Add osv.dev-compatible demo mock under /demo/osvdev - #1717

Open
yuvalk wants to merge 1 commit into
content-services:mainfrom
yuvalk:osvdev-demo-mock
Open

yuvalk wants to merge 1 commit into
content-services:mainfrom
yuvalk:osvdev-demo-mock

Conversation

@yuvalk

@yuvalk yuvalk commented Sep 9, 2026

Copy link
Copy Markdown

Summary

Adds an osv.dev-compatible mock API under /demo/osvdev that serves our Lightwell advisory data in the OSV format, so OSV tooling (e.g. osv-scanner) can consume our content by pointing at this host instead of api.osv.dev — until the data is accepted upstream by osv.dev.

The endpoint is public / unauthenticated (external scanners send no x-rh-identity), rooted at /demo/osvdev, and gated behind a new LightwellOsvDemo feature flag (off by default).

Endpoints

Query API (mirrors api.osv.dev):

  • POST /demo/osvdev/v1/query{"vulns":[…]}, or {} on no match
  • POST /demo/osvdev/v1/querybatch — index-aligned results, {id, modified} stubs
  • GET /demo/osvdev/v1/vulns/{id} — full record, 404 if unknown

Static export (mirrors storage.googleapis.com/osv-vulnerabilities):

  • GET /demo/osvdev/{ecosystem}/all.zip, GET /demo/osvdev/all.zip
  • GET /demo/osvdev/{ecosystem}/{id} (serves <id>.json)
  • GET /demo/osvdev/ecosystems.txt

How it works

lightwell_advisories rows → LightwellAdvisoryDao.ListForOsvpkg/lightwell/osv.BuildRecords (grouped by AdvisoryID, one affected[] entry per row) → serialized as OSV JSON / zipped in-memory. No files are stored on disk; records are built per request.

Files

Purpose Location
OSV schema + request/response DTOs pkg/api/osvdev.go
Advisory → OSV conversion + query matching pkg/lightwell/osv/converter.go
HTTP handler + route registration pkg/handler/osvdev.go
DAO read method (ListForOsv) pkg/dao/lightwell_advisory.go, pkg/dao/interfaces.go
Route wiring (raw engine) pkg/handler/api.go
Public-route auth skip (/demo/osvdev prefix) pkg/middleware/enforce_identity.go
Feature flag pkg/config/config.go

⚠️ Missing pieces to make the demo actually serve data

This PR wires up the API, but a few things are not yet done and are required for the endpoint to return anything useful in a demo/stage/prod environment. Flagging them explicitly so reviewers can decide what belongs in this PR vs. follow-ups.

1. The lightwell_advisories table is empty in demo/local

The mock reads lightwell_advisories, which is populated only by the sync-lightwell-advisories cronjob. That job:

  • uses the prod allowlist pkg/external_repos/lightwell_repos.json (via LoadLightwellAllowlist()), not lightwell_demo_repos.json, and
  • only ingests entries that define an osv_path (currently just lightwell/java/remediated and lightwell/java/predisclosure).

Consequences:

  • Demo/local: the table stays empty — the demo allowlist has no osv_path, and the sync ignores it anyway. /demo/osvdev returns {} / empty zips.
  • Prod: populated for the two Java/Maven OSV paths (cron runs every 10 min, SUSPEND_SYNC_LIGHTWELL_ADVISORIES defaults to false), assuming the OSV source is reachable.

Options (pick one, follow-up):

  • Add a seed for lightwell_advisories (only lightwell vulnerabilities are seeded today) so LoadLightwellDemo / local dev returns data.
  • Add osv_path entries to lightwell_demo_repos.json and teach sync-lightwell-advisories to use the demo allowlist when in demo mode.
  • Document a manual insert / one-off sync-lightwell-advisories run for demos.

2. Ecosystem is hardcoded to "Red Hat" — likely wrong

lightwell_advisories has no ecosystem column, so pkg/lightwell/osv.DefaultEcosystem currently returns a fixed "Red Hat". But the real advisories come from repos typed maven (Java) / python — so the correct OSV ecosystem is Maven / PyPI. The ecosystem is derivable from RepoName (lightwell/java/…, lightwell/python/…).
TODO: derive ecosystem from RepoName (java→Maven, python→PyPI, fallback default), or add an ecosystem column and thread it through the advisory sync.

3. Lossy OSV → advisory → OSV round-trip

Advisories are themselves ingested from OSV files (processOSVForEntry in sync_lightwell_advisories.go). We map OSV → lightwell_advisories (dropping fields not stored, e.g. severity vectors, aliases, per-range introduced) → back to OSV. Fields not persisted can't be reconstructed. If fidelity matters, consider serving from (or enriching with) the original OSV payloads.

4. Deployment/config wiring for the flag

features.lightwell_osv_demo.enabled has no ClowdApp parameter yet. To toggle it per-environment (stage/prod), add an env var + template wiring in deployments/. Not needed for local demo.

5. Regenerate mocks with the pinned mockery

The ListForOsv mock in pkg/dao/dao_mock.go was hand-added in the repo's existing generated style. make mock locally pulled a newer mockery that reformats every mock (interface{}any, adds Helper()), which was reverted as unrelated churn. Please regenerate with the CI-pinned mockery version before/at merge so the generated output is authoritative.


Testing steps

Automated:

  • go test ./pkg/lightwell/osv/... — converter unit tests (grouping, ranges, Matches, version compare)
  • go test ./pkg/handler/ -run TestOsvDevSuite — handler suite (query hit/miss, batch index alignment + stubs, vulns/{id} 200/404, all.zip, flag-off ⇒ 404, no identity header required, DAO error). Needs a Postgres (like all handler tests).
  • golangci-lint run --timeout=5m — 0 issues
  • git diff --exit-code api/openapi.json — no drift (routes intentionally omit swagger annotations)

Manual (requires data — see "Missing pieces #1"):

  1. Set features.lightwell_osv_demo.enabled: true in configs/config.yaml.
  2. Populate lightwell_advisories (seed / manual insert / sync-lightwell-advisories).
  3. Run the API server (:8000), then without any identity header:
    • curl -sXPOST localhost:8000/demo/osvdev/v1/query -d '{"package":{"name":"<pkg>"},"version":"1.0.0"}'
    • curl -s localhost:8000/demo/osvdev/v1/vulns/<AdvisoryID>
    • curl -sXPOST localhost:8000/demo/osvdev/v1/querybatch -d '{"queries":[{"package":{"name":"<pkg>"},"version":"1.0.0"},{"commit":"deadbeef"}]}'
    • curl -sO localhost:8000/demo/osvdev/all.zip && unzip -l all.zip
  4. Confirm flag off ⇒ 404.

🤖 Generated with Claude Code

Serve Lightwell advisories through an osv.dev-compatible API so OSV
tooling (e.g. osv-scanner) can consume our content until it is accepted
upstream by osv.dev. The mock is public (no x-rh-identity), rooted at
/demo/osvdev, and gated by the LightwellOsvDemo feature flag (off by
default).

Implements the query API (POST /v1/query, POST /v1/querybatch,
GET /v1/vulns/{id}) plus the GCS-style static export
(<ecosystem>/all.zip, <id>.json, ecosystems.txt). Advisories are grouped
by AdvisoryID into OSV records via pkg/lightwell/osv, backed by a new
LightwellAdvisoryDao.ListForOsv method that returns full models with
timestamps.

Note: the ListForOsv mock in pkg/dao/dao_mock.go was hand-added in the
generated style; regenerate with `make mock` before merge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@yuvalk
yuvalk requested a review from a team as a code owner September 9, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant