feat(hyperliquid): fold outcomes scraper into the hyperliquid service - #305
Merged
Conversation
When substreams-hypercore consolidated the outcome carve-out back into one package (v0.4.0), the scraper still ran as two separate services / two pods / two cycles. Folding them mirrors the substreams story: one HL Info client per env, one polling cadence, one set of CH credentials, one /live + /metrics endpoint. Restructured services/hyperliquid/ into two pure-function sub-cycles (`runSpotCycle`, `runOutcomesCycle`) orchestrated by `run()` in `index.ts` via `Promise.allSettled`. Failure in either surfaces to the supervisor without short-circuiting the other. Removed: - services/hyperliquid-outcomes/ (deleted entirely) - `hyperliquid-outcomes` SERVICE registration in cli.ts - `setup hyperliquid-outcomes` CLI command (folded into `setup hyperliquid` which now deploys all three tables) Tests restructured: each sub-cycle gets its own test suite over the pure cycle function; new `index.test.ts` covers the orchestrator's parallel-execution + error-surfacing semantics. End-to-end validated against dev1 v0.4.2 — single cycle inserts 307 spot pair names + 138 outcomes + 22 questions in ~1.1s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR consolidates the former hyperliquid-outcomes scraper into the main hyperliquid service, so a single service cycle polls both spot metadata and HIP-4 outcomes/questions against one HL Info endpoint and one ClickHouse config.
Changes:
- Introduces
runSpotCycle(infoUrl)and refactors outcomes intorunOutcomesCycle(infoUrl), orchestrated together viaPromise.allSettled()inservices/hyperliquid/index.ts. - Adds dedicated Info API helpers for spot (
spot-info.ts) and outcomes (outcomes-info.ts) with shared request timeout behavior. - Removes the standalone
hyperliquid-outcomesCLI service/setup command and folds table deployment intosetup hyperliquid.
Reviewed changes
Copilot reviewed 9 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| services/hyperliquid/spot.ts | New spot sub-cycle that snapshots state_spot_pair_names. |
| services/hyperliquid/spot.test.ts | Unit tests for spot sub-cycle insert + error behavior. |
| services/hyperliquid/spot-info.ts | Spot Info API fetch + @N/canonical pair-name resolution helpers. |
| services/hyperliquid/spot-info.test.ts | Updates spot-info tests to the new module path. |
| services/hyperliquid/outcomes.ts | Refactors outcomes scraper into runOutcomesCycle(infoUrl) under hyperliquid. |
| services/hyperliquid/outcomes.test.ts | Updates outcomes tests for new entrypoint + env handling removal. |
| services/hyperliquid/outcomes-info.ts | New outcomes Info API helpers + row builders. |
| services/hyperliquid/outcomes-info.test.ts | Updates outcomes-info tests to the new module path. |
| services/hyperliquid/index.ts | New combined orchestrator running spot + outcomes cycles in parallel. |
| services/hyperliquid/index.test.ts | Reworks orchestrator tests to mock the two sub-cycles. |
| cli.ts | Removes hyperliquid-outcomes service/setup; expands setup hyperliquid to deploy both schemas. |
Per the liveness contract in `lib/service-init.ts`, the wall-clock heartbeat must not advance on an errored cycle. The first cut of the combined scraper kept `markServiceAlive()` + `incrementSuccess()` inside each sub-cycle, which meant a partial success — spot insert lands but outcomes fetch fails, say — would advance the heartbeat even though the supervisor saw the throw and was retrying. `/live` reported healthy while the service was silently flapping. Move both calls to the orchestrator in `index.ts`: only fire after `Promise.allSettled` returns zero rejections. Per-sub-cycle error metrics still fire from inside each catch block before the throw propagates, so we keep per-failure visibility. Tests updated: - spot.test.ts + outcomes.test.ts assert the sub-cycle does NOT touch success / heartbeat. - index.test.ts gains coverage for partial-failure scenarios: spot-only fail, outcomes-only fail, both fail — all must leave the heartbeat unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Findings from the pre-merge code review: - outcomes-info.ts logger label was still `'hyperliquid-outcomes'`, diverging from `'hyperliquid:outcomes'` used by the wrapper. Operator log filters / Grafana panels keyed off the consolidated `service=hyperliquid` would have missed warnings from this module. - `nowRefreshTime` was inlined in spot.ts and a named helper in outcomes.ts. Both feed RMT keys so the format must stay in lockstep — lifted to a single `refresh-time.ts` and imported. - `index.ts run()` previously `throw errors[0]` when both sub-cycles rejected, silently dropping the second error. Now wraps in AggregateError so the supervisor's stack carries both reasons. Single-error case stays as a plain throw. - check-hyperliquid-outcomes.ts docstring + lib/setup.test.ts test name both still referenced the removed `hyperliquid-outcomes` service name; updated to reflect the consolidation. End-to-end re-verified against dev1 v0.4.2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
substreams-hypercoreconsolidated the outcome carve-out back into one package (v0.4.0), the scraper still ran as two separate services. This folds them: one HL Info client per env, one polling cadence, one set of CH credentials, one/live+/metricsendpoint.Structure
services/hyperliquid/now exposes two pure-function sub-cycles:runSpotCycle(infoUrl)→state_spot_pair_namesrunOutcomesCycle(infoUrl)→state_outcome_meta+state_question_metarun()inindex.tsorchestrates them viaPromise.allSettledso a failure in either surfaces to the supervisor without short-circuiting the other.Removed
services/hyperliquid-outcomes/(whole directory)hyperliquid-outcomesSERVICE registration in cli.tssetup hyperliquid-outcomesCLI command (folded intosetup hyperliquidwhich now deploys all three tables)End-to-end validation against dev1 v0.4.2
Single cycle inserts 307 spot pair names + 138 outcomes + 22 questions in ~1.1s; both polls run in parallel; failure injection (bogus info URL) surfaces the first error and doesn't leak partial state.
🤖 Generated with Claude Code