Context
As part of the polyglot migration (#14), load and storm now run as Rust binaries while embed / experiment orchestration stay in Python. Metrics used to be a single cross-tool Python module (supernova.metrics) — that worked when everything shared one process. It no longer does.
The tools never actually shared a metrics object at runtime anyway: each process constructs its own backend and writes to a shared Postgres/Neon DB, coordinating only through env vars (NOVA_RUN_ID, SKYPILOT_JOB_RANK) and ON CONFLICT DO NOTHING. So the thing that must stay unified is the DB contract (the runs / samples / events schema + id conventions), not shared code. The agreed model is: shared schema, one thin client per language.
The Rust client now exists (crates/nova-metrics, used by nova-storm), mirroring the Python backend's behavior (background flush thread, fail-open, secret redaction, Timescale hypertable best-effort).
The problem: the schema is now bifurcated
The DDL exists in two hand-maintained copies:
- Python — inline
_SCHEMA string in supernova/metrics/postgres.py
- Rust —
crates/nova-metrics/schema.sql (embedded via include_str! in crates/nova-metrics/src/pg.rs)
They are byte-compatible today, and create table if not exists means whichever tool's init() runs first wins and the other no-ops — so they interoperate. But there is no mechanism preventing drift. If someone adds a column to one (a new tags field, a region, an index) and not the other, the symptom is silent: writes from the lagging client still succeed against the existing table, but the new data path is missing on one side and nobody notices until a Grafana panel is empty or a fleet query returns half the nodes.
Proposed solution: centralize the schema as the single source of truth
-
One canonical file. Promote crates/nova-metrics/schema.sql (or move it to a language-neutral home, e.g. metrics/schema.sql at repo root) as the schema. Rust keeps embedding it via include_str!.
-
Python reads the same file instead of carrying _SCHEMA inline. Load it via importlib.resources with a filesystem fallback for in-repo runs, and add it to package data in pyproject.toml so installed wheels ship it. Delete the inline constant.
-
Conformance test per language, run in CI against an ephemeral Postgres (service container): apply the schema, write a run + a few samples + a summary, read them back, assert the round-tripped shape. This is the mechanical drift-catcher — a column added to the schema but not handled by a client fails its conformance test. (Both languages run against the same schema.sql, so the test also proves they agree.)
-
Future languages (a Go tool, etc.) follow the same pattern: read metrics/schema.sql, add a conformance test. No shared binary, no FFI.
Why not bind one implementation across languages (pyo3 / FFI)
Considered and rejected: the shared surface is ~250 lines of "enqueue → batch-insert into 3 tables, fail-open" — small and stable. A pyo3 binding adds a maturin build + per-platform wheels shipped to every SkyPilot worker, couples the Python embed build to a Rust toolchain (for a tool that doesn't even use metrics yet), and still doesn't solve a future Go tool. The contract is DB-shaped, so binding it at the DB is the right altitude.
Escape hatch (not now)
If we grow to 4+ languages or add traces/logs alongside metrics, revisit emitting OTLP to a collector that owns the schema — OTLP exponential histograms are mergeable across workers, which is exactly the fleet-p99 case. Not worth the infra for two languages + a stable 3-table schema.
Acceptance criteria
Context
As part of the polyglot migration (#14),
loadandstormnow run as Rust binaries whileembed/ experiment orchestration stay in Python. Metrics used to be a single cross-tool Python module (supernova.metrics) — that worked when everything shared one process. It no longer does.The tools never actually shared a metrics object at runtime anyway: each process constructs its own backend and writes to a shared Postgres/Neon DB, coordinating only through env vars (
NOVA_RUN_ID,SKYPILOT_JOB_RANK) andON CONFLICT DO NOTHING. So the thing that must stay unified is the DB contract (theruns/samples/eventsschema + id conventions), not shared code. The agreed model is: shared schema, one thin client per language.The Rust client now exists (
crates/nova-metrics, used bynova-storm), mirroring the Python backend's behavior (background flush thread, fail-open, secret redaction, Timescale hypertable best-effort).The problem: the schema is now bifurcated
The DDL exists in two hand-maintained copies:
_SCHEMAstring insupernova/metrics/postgres.pycrates/nova-metrics/schema.sql(embedded viainclude_str!incrates/nova-metrics/src/pg.rs)They are byte-compatible today, and
create table if not existsmeans whichever tool'sinit()runs first wins and the other no-ops — so they interoperate. But there is no mechanism preventing drift. If someone adds a column to one (a newtagsfield, aregion, an index) and not the other, the symptom is silent: writes from the lagging client still succeed against the existing table, but the new data path is missing on one side and nobody notices until a Grafana panel is empty or a fleet query returns half the nodes.Proposed solution: centralize the schema as the single source of truth
One canonical file. Promote
crates/nova-metrics/schema.sql(or move it to a language-neutral home, e.g.metrics/schema.sqlat repo root) as the schema. Rust keeps embedding it viainclude_str!.Python reads the same file instead of carrying
_SCHEMAinline. Load it viaimportlib.resourceswith a filesystem fallback for in-repo runs, and add it to package data inpyproject.tomlso installed wheels ship it. Delete the inline constant.Conformance test per language, run in CI against an ephemeral Postgres (service container): apply the schema, write a
run+ a fewsamples+ asummary, read them back, assert the round-tripped shape. This is the mechanical drift-catcher — a column added to the schema but not handled by a client fails its conformance test. (Both languages run against the sameschema.sql, so the test also proves they agree.)Future languages (a Go tool, etc.) follow the same pattern: read
metrics/schema.sql, add a conformance test. No shared binary, no FFI.Why not bind one implementation across languages (pyo3 / FFI)
Considered and rejected: the shared surface is ~250 lines of "enqueue → batch-insert into 3 tables, fail-open" — small and stable. A pyo3 binding adds a maturin build + per-platform wheels shipped to every SkyPilot worker, couples the Python
embedbuild to a Rust toolchain (for a tool that doesn't even use metrics yet), and still doesn't solve a future Go tool. The contract is DB-shaped, so binding it at the DB is the right altitude.Escape hatch (not now)
If we grow to 4+ languages or add traces/logs alongside metrics, revisit emitting OTLP to a collector that owns the schema — OTLP exponential histograms are mergeable across workers, which is exactly the fleet-p99 case. Not worth the infra for two languages + a stable 3-table schema.
Acceptance criteria
schema.sql; the Python_SCHEMAconstant is gone and Python reads the shared file (incl. installed-wheel packaging).nova-metricsREADME pointing new languages at the pattern.