Skip to content

fix: log and exit non-zero when the tycho feed stops - #409

Draft
brunoguerios wants to merge 2 commits into
mainfrom
bg/feed-stream-end-exit-nonzero
Draft

fix: log and exit non-zero when the tycho feed stops#409
brunoguerios wants to merge 2 commits into
mainfrom
bg/feed-stream-end-exit-nonzero

Conversation

@brunoguerios

Copy link
Copy Markdown
Contributor

40 of 43 fynd shutdowns in the last 48h ended the Tycho stream cleanly: TychoFeed::run broke its
loop, returned Ok(()), the spawn wrapper logged nothing, tycho_feed_failures_total did not
increment, and the process exited 0 (reason: Completed). The dominant prod failure mode was
invisible to both logs and metrics.

Now any end of the stream — clean or error — returns
StreamError("<stream> stream ended (last block: N)"). The existing spawn wrappers in solver.rs
log it at ERROR and increment tycho_feed_failures_total with no change. The feed arm of
FyndRPC::run sets fatal_error, so the process exits 1 and k8s records reason: Error. The gas
arm had the same defect and gets the same one-line fix.

Not in scope: reconnect-with-backoff inside TychoFeed, and the upstream tycho-client
RevertPositionNotFound fix (tycho repo) that causes most of these stream ends.
run_with_step_controller keeps its clean end — that end is deliberate test behavior.

Before prod rollout: dashboards or alerts that key on last_terminated_reason="Completed" for fynd
pods now see Error, and Loki queries that count the INFO line Protocol stream ended must switch
to the tycho feed error ERROR line or to the metric.

Local CI green (fmt, clippy, doc, 1270 tests). Not yet verified on staging — that needs a forced
stream end on one chain to confirm the exit code and the log line.

@brunoguerios brunoguerios self-assigned this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant