feat(driver): host the driver v3/v4 streaming controller in sentio-core#300
Merged
Merged
Conversation
Preparation for moving the streaming driver (driver v3/v4, sentio's driver/controller) into sentio-core. The controller depends on a handful of sentio-local leaf libraries that have no sentio-only dependencies, so they are vendored into sentio-core here (the sentio side will repoint its imports and delete the local copies in a paired change): - common/compress, common/window, common/contract, common/jsonutils, driver/errors It also introduces driver/controller/config, holding the driver v3/v4 per-chain config (ConfigV2 + LoadChainsConfigV2 + NewCustomizedChainConfigV2). The legacy driver v2 config (chain.Config) stays in the sentio repository. common/clickhousemanagerext is intentionally NOT vendored (it must not be open-sourced): the controller's dependency on it will be inverted on the sentio side by injecting the timeseries/entity chx.Controller into the startup config, built by sentio/driver/cmd. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
34e9be2 to
d2b07b2
Compare
The vendored driver/errors should expose just the ExitCode type and its constants, which the streaming driver (driver v3/v4) uses. The ErrXXX sentinel errors and the Halt / IsProcessorError helpers are driver v2 specific and stay in the sentio repository. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vendored package holds only the process exit/retry codes the streaming driver uses, so name it driver/exitcode (package exitcode, type Code). The ErrXXX sentinel errors and Halt/IsProcessorError helpers are driver v2 specific and stay in the sentio repository. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
a01b81c to
2135418
Compare
Move the driver v3/v4 streaming controller (~21k LOC across data / fetcher /
standard / subgraph / startup) into sentio-core. Imports are repointed to their
sentio-core homes:
- chain ConfigV2 (and loaders) -> driver/controller/config
- driver exit codes -> driver/exitcode (type Code)
- common/{compress,window,contract,gonanoid} and service/common/rpc -> sentio-core
The controller's couplings to the driver binary's metric instruments and
service clients are inverted behind controller.Notifier and
startup.ClickhouseConnector (added earlier); their implementations live in the
driver binary, which is not part of this repo. The GCP project for the webhook
pubsub topic is injected via startup.Config rather than hardcoded. Adds the
cloud.google.com/go/pubsub dependency.
The driver-binary side (remove the in-repo controller, repoint its consumers)
lands in the paired downstream change.
2135418 to
f1408a6
Compare
Now that the streaming controller lives in sentio-core, the "V2" suffix (which only disambiguated it from the legacy sentio chain.Config) is no longer meaningful here: - ConfigV2 -> ChainConfig - LoadChainsConfigV2 -> LoadChainsConfig - NewCustomizedChainConfigV2 -> NewCustomizedChainConfig Also drop the `chain` import alias on driver/controller/config across all consumers and use the package's own name (config.ChainConfig etc.). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the driver v3/v4 streaming controller to sentio-core — the engine that drives a processor over a chain block by block: fetch block data, run the processor's handlers, and persist ordered checkpoints (~21k LOC).
Design
Pipeline (
main.go).MainControllerruns a three-stage concurrent pipeline over a block stream:seqModeforces single-worker ordering);Back-pressure is bounded by the
waitingBlocks/progressNoticechannels, so a slow handler or a slow checkpoint stage throttles fetching instead of growing memory unboundedly.Run loop & resilience (
main.go,errors.go). The loop classifies failures intoExternalError(driver vs. user-runtime) carrying an exit/retryCode, retries with backoff, and dedups repeated identical errors (gives up after a cap). Two internal signals — reorg detected and new template (dynamic data source) — restart the stream cleanly rather than erroring out.Data layer (
data/). Per-chain clients that fetch and normalize raw chain data —evm,aptos,sol,sui(+sui/grpc),fuel— behind a common block/transaction abstraction, with block caching, interval sampling and subscribe support.Fetcher (
fetcher/). A generic concurrent fetch / merge / retry layer (FetchTarget/Requirement) the data layer uses to pull ranges in parallel and merge them back into ordered blocks.Two handler models.
standard/— the SDK handler model: per-chain handlers (evm log/trace/transaction, aptos, sol instruction, sui event/change/function/interval, fuel) turn block data into binding-data tasks.subgraph/— the WASM subgraph path: a wasminstanceplus block/call/event handlers and its own block-data assembly.Startup (
startup/). Builds a ready-to-run controller from a processor model, wiring the checkpoint store, entity / timeseries stores, webhook publishing and quota tracking together.Supporting libraries
driver/controller/config— the per-chainChainConfig+ loaders.common/{compress,window,contract,jsonutils}anddriver/exitcode(the process exit/retryCodetype) — leaf libraries the controller needs.cloud.google.com/go/pubsubdependency (webhook pubsub topic).Embedding seams
The controller stays free of binary-only packages and deployment values by taking its host integration through two interfaces it defines —
controller.Notifier(callbacks the controller invokes at lifecycle and data events; what an implementation does with them is up to the caller) andstartup.ClickhouseConnector(supplies the timeseries/entitychx.Controller) — plus an injectedEntityMetricsMonitorandPubSubProject.Test
bazel build //driver/...— green.bazel test //driver/controller/...— 11/11 pass.