[alex-clickhouse]

What

Before a source's first run, give each consumer that reads every source a
live cursor on the source, at the source's newest message. The first
records of the new source are then unread for those consumers, and their
messages gates see them.

Why

A consumer that reads other sources does not see the first records of a
new source:

• consumer_has_unread ignores a source for which the consumer has no
  cursor. A messages gate without sources uses it, so the gate does not
  fire for the new source.
• The consumer's first poll starts its cursor at the source's newest
  message, past the first records. An expired cursor on the source starts
  again in the same way.

This affects github_repos and every source that starts after the
consumers have cursors, also the hosted sources.

How

• SourceStore.seed_consumer_cursors(source) upserts a cursor on the
  source for each consumer that reads every source. Such a consumer has a
  cursor on each other source that has messages, as poll_all_sources
  leaves it. An expired cursor that is not yet deleted also counts.
• A consumer that reads only some sources with poll_source gets no
  cursor. Its job does not move a cursor on the new source, so a messages
  gate without sources would find unread messages on every check, and the
  gate's TTL renewal would keep that cursor live.
• A consumer without a cursor on the source gets one. An expired cursor on
  the source starts again at the newest message with a fresh TTL. A live
  cursor stays as it is.
• SourceRunner calls it before the first fetch when the source has no
  sync cursor. The TTL is sync.consumer_cursor_ttl_days.
• If this step fails, the run fails before the fetch, as for a fetch
  failure. The source stores nothing and has no sync cursor, so the next
  run tries again.
• docs/sources.md describes the behavior. docs/cron.md tells a job
  that reads only some sources to name them in sources.

Based on a fix by Frank Rosner.

Tests

• A new source's first records are unread for an existing consumer, and a
  messages gate on any source and on the new source sees them.
• A baseline run without records starts the cursors, and later records
  reach poll_all_sources.
• A live cursor on the source is not changed, an expired one starts again
  at the newest message, and the seeding does not run for a source with a
  sync cursor.
• A consumer that polls only one of two sources gets no cursor on the new
  source, and its gate without sources stays false. A consumer with an
  expired cursor on one of the other sources gets a cursor.
• With an expired cursor on the new source, the first records reach the
  gate and poll_all_sources.
• A failure to start the cursors fails the run without a fetch, and the
  next run delivers the first records.
• The first drain of the hosted github_repos buffer wakes the messages
  gate of a consumer that reads github.
• Full backend suite: 8931 passed, 38 skipped, 1 failed (test_platform_activity.py::test_a_burst_of_changes_is_coalesced, a timing test, at load average 11; it passes alone, 3 runs out of 3).

🤖 Generated with Claude Code

[nerveai]

Nerve PR review · ✅ Ready to merge

Updated for 1182397: the seed now starts a cursor only for a consumer that has a cursor on every other source with messages, which is the state poll_all_sources leaves. docs/cron.md also tells poll_source jobs to name sources. That resolves my earlier blocking point, and the new test fails without the narrowing.

The PR fixes a real gap: the first records of a new source (github_repos, any hosted provider) never reached a poll_all_sources consumer such as the default inbox-processor. The runner, before the first fetch, is the right place for the seed. It needs no migration, it covers polled and hosted sources, and a failed seed stores nothing and is retried.

Completeness: tests cover the first records, a baseline run, live and expired cursors, a failed seed, the hosted drain and the narrowing. docs/sources.md and docs/cron.md are updated.

Worth considering:

• The check at nerve/db/sources.py:432 cannot tell a poll_source consumer of the only source with messages from a poll_all_sources one. Example: an install with only github gains github_repos. A poll_source job there whose gate has no sources still gets a cursor, and its gate then stays open on every check, because the gate renews that cursor (gates.py:204). docs/cron.md:259 says such a consumer "does not get one". A release note for single-source installs would cover this.

_{Checked: 7 source/gate/DB test files in an offline sandbox at 1182397: 561 passed. Backend CI still running, the other checks pass. Merges cleanly with main; conflicts with #630 (docs/cron.md) and draft #596 (4 files) · automated review by nerveai — mention @nerveai to ask a follow-up.}

[alex-clickhouse]

Thanks. Fixed in 1182397, with the first option and a docs note:

• seed_consumer_cursors now starts a cursor only for a consumer that reads every source: it has a cursor on each other source that has messages, as poll_all_sources leaves it. An expired cursor that is not yet deleted also counts. A consumer that polls only some sources with poll_source gets no cursor on the new source, so its messages gate without sources stays as it was.
• docs/cron.md (messages gate) tells a job that reads only some sources to name them in sources. docs/sources.md describes which consumers get a cursor.
• New test test_seeding_skips_a_consumer_that_reads_only_some_sources. It fails if the narrowing is removed.

One limit stays: when the consumer's sources are the only sources with messages, a consumer that polls them one by one looks the same as one that uses poll_all_sources. The docs note covers that case.