feat(hyperliquid-outcomes): add HIP-4 outcome metadata scraper - #304
Merged
Conversation
Polls `outcomeMeta` for the live outcome universe + question groupings and recovers settled outcomes via per-id `settledOutcome` lookups, since settled outcomes drop out of the live snapshot. Writes two RMT tables: - state_outcome_meta: per-outcome name, description, side specs, status, optional settlement payload (status = live | settled) - state_question_meta: multi-outcome question grouping with named + fallback outcome ids Discovery scans distinct outcome_id from outcome_fills (substreams- written) and probes settledOutcome for ids missing from the live snapshot. Concurrency bound via HL_OUTCOMES_SETTLED_CONCURRENCY (default 4) keeps per-IP load on the upstream Info endpoint low. Wired into the CLI as `run hyperliquid-outcomes` and `setup hyperliquid-outcomes`, mirroring the existing `hyperliquid` (spot pair names) service. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/check-hyperliquid-outcomes.ts verifies schema presence + column shape, row counts by status, coverage gap between outcome_fills and state_outcome_meta, refresh-time freshness, and question reverse-map consistency. Exits non-zero on failure for use in deployment gates. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new hyperliquid-outcomes service to scrape HIP-4 outcome/question metadata from Hyperliquid’s Info API and persist it to ClickHouse so downstream Token API endpoints can join numeric outcome_ids to human-readable labels (including recovery of settled outcomes that disappear from the live snapshot).
Changes:
- Introduces
hyperliquid-outcomesscraper (Info API client + one-cycle runner) with unit tests. - Adds ClickHouse schemas for
state_outcome_metaandstate_question_meta, plus setup wiring in the CLI and setup tests. - Adds a deployment/ops validation script to verify schema shape, coverage vs
outcome_fills, and freshness.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| sql.schemas/schema.hyperliquid_outcomes.sql | Adds ClickHouse tables for outcome and question metadata (ReplacingMergeTree snapshots). |
| services/hyperliquid-outcomes/info.ts | Implements Hyperliquid Info API POST helpers and row projection utilities. |
| services/hyperliquid-outcomes/info.test.ts | Unit tests for projection helpers and Info API parsing behaviors. |
| services/hyperliquid-outcomes/index.ts | Service cycle: fetch live snapshot, discover missing IDs, probe settled outcomes, insert to ClickHouse. |
| services/hyperliquid-outcomes/index.test.ts | Integration-style unit tests for one service cycle with mocked ClickHouse and fetch. |
| scripts/check-hyperliquid-outcomes.ts | Post-deploy validator for schema presence, row counts, coverage, freshness, and reverse-map sanity. |
| lib/setup.test.ts | Ensures the new schema SQL is parsed and cluster-transformed correctly. |
| cli.ts | Registers the new service and adds setup hyperliquid-outcomes action/help text. |
The service uses `insertClient.insert` directly rather than the batch insert queue, so the queue's `getLastSuccessfulFlushAt()` never advances and the `/live` probe would return 503 after the startup grace window even on healthy cycles. Call `markServiceAlive()` after a successful insert (and not on failures) so the probe reflects real progress without masking silent upstream errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…led re-probes - check-hyperliquid-outcomes.ts: guard `Number.parseInt` results with `Number.isFinite` in row-count, coverage, and freshness checks. Without the guards a malformed CH response produced misleading output like "live=NaN, settled=X" or "NaN outcome_ids ... lack a meta row" instead of a real failure. - index.ts: warn and return early when `outcomeMeta` yields zero outcomes AND zero questions, matching the sibling spot-pair service. Avoids silent success on an empty upstream. - index.ts: skip per-id `settledOutcome` probes for outcome_ids already captured with `status='settled'`. Settled payloads are immutable, so re-probing them every cycle wastes Info API budget that grows with the cumulative settled universe. - info.ts: drop the dead `!== undefined` check on `fallbackOutcome`; the field type is `number | null`. Tests cover both new behaviors and route the mocked CH query by table name so the existing happy-path assertions are unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hyperliquid-outcomesscraper service for HIP-4 outcome + question metadataReplacingMergeTreetables:state_outcome_meta(per-outcome) andstate_question_meta(multi-outcome groupings)settledOutcomelookups since they drop out of the liveoutcomeMetasnapshotscripts/check-hyperliquid-outcomes.tsWhy
HIP-4 outcomes live in the new
outcome_fillstable (substreams-hypercore v0.4.0). The Token API needs human-readable labels (outcome name, description, side specs, question grouping) to expose/v1/hyperliquid/outcomes/*— the substreams stream only carries the numericoutcome_id. This scraper joins the gap by polling Hyperliquid's Info API.Code references
services/hyperliquid-outcomes/index.ts,services/hyperliquid-outcomes/info.tssql.schemas/schema.hyperliquid_outcomes.sqlcli.ts(run hyperliquid-outcomes,setup hyperliquid-outcomes)scripts/check-hyperliquid-outcomes.ts🤖 Generated with Claude Code