Skip to content

feat(driver): host the driver v3/v4 streaming controller in sentio-core#300

Merged
jupiterv2 merged 5 commits into
mainfrom
dev/jupiterv2.10/move-driver-deps-from-sentio
Jun 18, 2026
Merged

feat(driver): host the driver v3/v4 streaming controller in sentio-core#300
jupiterv2 merged 5 commits into
mainfrom
dev/jupiterv2.10/move-driver-deps-from-sentio

Conversation

@jupiterv2

@jupiterv2 jupiterv2 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

What

Adds the driver v3/v4 streaming controller to sentio-core — the engine that drives a processor over a chain block by block: fetch block data, run the processor's handlers, and persist ordered checkpoints (~21k LOC).

Design

Pipeline (main.go). MainController runs a three-stage concurrent pipeline over a block stream:

  1. a producer builds per-block data and splits it into independent tasks;
  2. a worker pool runs the handler tasks (concurrency configurable; seqMode forces single-worker ordering);
  3. checkpoints are assembled strictly in block order and saved asynchronously.

Back-pressure is bounded by the waitingBlocks / progressNotice channels, so a slow handler or a slow checkpoint stage throttles fetching instead of growing memory unboundedly.

Run loop & resilience (main.go, errors.go). The loop classifies failures into ExternalError (driver vs. user-runtime) carrying an exit/retry Code, retries with backoff, and dedups repeated identical errors (gives up after a cap). Two internal signals — reorg detected and new template (dynamic data source) — restart the stream cleanly rather than erroring out.

Data layer (data/). Per-chain clients that fetch and normalize raw chain data — evm, aptos, sol, sui (+ sui/grpc), fuel — behind a common block/transaction abstraction, with block caching, interval sampling and subscribe support.

Fetcher (fetcher/). A generic concurrent fetch / merge / retry layer (FetchTarget / Requirement) the data layer uses to pull ranges in parallel and merge them back into ordered blocks.

Two handler models.

  • standard/ — the SDK handler model: per-chain handlers (evm log/trace/transaction, aptos, sol instruction, sui event/change/function/interval, fuel) turn block data into binding-data tasks.
  • subgraph/ — the WASM subgraph path: a wasm instance plus block/call/event handlers and its own block-data assembly.

Startup (startup/). Builds a ready-to-run controller from a processor model, wiring the checkpoint store, entity / timeseries stores, webhook publishing and quota tracking together.

Supporting libraries

  • driver/controller/config — the per-chain ChainConfig + loaders.
  • common/{compress,window,contract,jsonutils} and driver/exitcode (the process exit/retry Code type) — leaf libraries the controller needs.
  • Adds the cloud.google.com/go/pubsub dependency (webhook pubsub topic).

Embedding seams

The controller stays free of binary-only packages and deployment values by taking its host integration through two interfaces it defines — controller.Notifier (callbacks the controller invokes at lifecycle and data events; what an implementation does with them is up to the caller) and startup.ClickhouseConnector (supplies the timeseries/entity chx.Controller) — plus an injected EntityMetricsMonitor and PubSubProject.

Test

  • bazel build //driver/... — green.
  • bazel test //driver/controller/... — 11/11 pass.

Preparation for moving the streaming driver (driver v3/v4, sentio's
driver/controller) into sentio-core. The controller depends on a handful
of sentio-local leaf libraries that have no sentio-only dependencies, so
they are vendored into sentio-core here (the sentio side will repoint its
imports and delete the local copies in a paired change):

- common/compress, common/window, common/contract, common/jsonutils,
  driver/errors

It also introduces driver/controller/config, holding the driver v3/v4
per-chain config (ConfigV2 + LoadChainsConfigV2 + NewCustomizedChainConfigV2).
The legacy driver v2 config (chain.Config) stays in the sentio repository.

common/clickhousemanagerext is intentionally NOT vendored (it must not be
open-sourced): the controller's dependency on it will be inverted on the
sentio side by injecting the timeseries/entity chx.Controller into the
startup config, built by sentio/driver/cmd.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jupiterv2 jupiterv2 force-pushed the dev/jupiterv2.10/move-driver-deps-from-sentio branch from 34e9be2 to d2b07b2 Compare June 17, 2026 12:00
jupiterv2 and others added 2 commits June 18, 2026 16:05
The vendored driver/errors should expose just the ExitCode type and its
constants, which the streaming driver (driver v3/v4) uses. The ErrXXX sentinel
errors and the Halt / IsProcessorError helpers are driver v2 specific and stay
in the sentio repository.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vendored package holds only the process exit/retry codes the streaming
driver uses, so name it driver/exitcode (package exitcode, type Code). The
ErrXXX sentinel errors and Halt/IsProcessorError helpers are driver v2 specific
and stay in the sentio repository.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jupiterv2 jupiterv2 force-pushed the dev/jupiterv2.10/move-driver-deps-from-sentio branch 2 times, most recently from a01b81c to 2135418 Compare June 18, 2026 08:52
Move the driver v3/v4 streaming controller (~21k LOC across data / fetcher /
standard / subgraph / startup) into sentio-core. Imports are repointed to their
sentio-core homes:

- chain ConfigV2 (and loaders) -> driver/controller/config
- driver exit codes -> driver/exitcode (type Code)
- common/{compress,window,contract,gonanoid} and service/common/rpc -> sentio-core

The controller's couplings to the driver binary's metric instruments and
service clients are inverted behind controller.Notifier and
startup.ClickhouseConnector (added earlier); their implementations live in the
driver binary, which is not part of this repo. The GCP project for the webhook
pubsub topic is injected via startup.Config rather than hardcoded. Adds the
cloud.google.com/go/pubsub dependency.

The driver-binary side (remove the in-repo controller, repoint its consumers)
lands in the paired downstream change.
@jupiterv2 jupiterv2 force-pushed the dev/jupiterv2.10/move-driver-deps-from-sentio branch from 2135418 to f1408a6 Compare June 18, 2026 09:14
@jupiterv2 jupiterv2 changed the title feat(driver): vendor shared libs and driver-v3/v4 chain config into core feat(driver): host the driver v3/v4 streaming controller in sentio-core Jun 18, 2026
Now that the streaming controller lives in sentio-core, the "V2" suffix
(which only disambiguated it from the legacy sentio chain.Config) is no longer
meaningful here:

- ConfigV2 -> ChainConfig
- LoadChainsConfigV2 -> LoadChainsConfig
- NewCustomizedChainConfigV2 -> NewCustomizedChainConfig

Also drop the `chain` import alias on driver/controller/config across all
consumers and use the package's own name (config.ChainConfig etc.).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jupiterv2 jupiterv2 merged commit 77424f2 into main Jun 18, 2026
1 check passed
@jupiterv2 jupiterv2 deleted the dev/jupiterv2.10/move-driver-deps-from-sentio branch June 18, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant