[perf] Coordinated splits churn between workers before their claims are seen

[MarcusKainth]

What kind

Throughput lower than expected

Version

main at d98aa6a. The assignment logic involved predates the polled-watch mode (#812–#815); that mode only widens the window below.

Which components

Coordination (spate-coordination)

Rust version and platform

rustc 1.96.1, macOS aarch64. The real-table runs went from a laptop to DynamoDB in eu-west-2; the local reproductions ran on the same machine.

The numbers

A fleet of equal-weight splits on the DynamoDB store (#818, #819) revokes splits faster than it completes them once the fleet grows. Ten-minute steady-state windows against a real table:

Workers	Revocations requested	Completions/s	Expected completions/s	Consumed WCU/s	Predicted WCU/s
3	2	3.04	3	27.2	24.8
10	116	8.17	10	82.4	82.6
20	8,793	11.58	20	332.7	165.2

At 20 workers the fleet completes 58% of the splits the model expects (8 lanes per worker, 6 s per split, one 2 s poll interval idle between splits) and consumes twice the predicted write capacity. Revocations are cooperative, so nothing is lost or replayed; the cost is throughput and store operations.

Cause. desired_assignment's sticky pass keeps a split with its member only when the leader can see an owner: a live lease, or failing that the progress record's owner. A split the leader assigned whose claim has not reached the leader's view has no owner, so the fill pass places it again, in id order onto the lightest member. A single completion then moves every assigned-but-unclaimed split to a different member. A worker that has already claimed one of them sees its assignment change and revokes it; each revocation brings more claims and publishes, and the effect feeds itself. The window is how long a claim takes to reach the leader: a store round trip on a push store, up to one poll interval on a polled one.

A pure repro over desired_assignment, with no store: 8 members each holding 2 of 3 lanes and 20 queued splits. Publish, remove one leased split to model a completion, publish again.

#[test]
fn one_completion_moves_unclaimed_assignments() {
    let ms = members(&["w0", "w1", "w2", "w3", "w4", "w5", "w6", "w7"]);
    let mut entries: Vec<(String, u64, Option<String>)> = Vec::new();
    for m in 0..8 {
        for k in 0..2 {
            entries.push((format!("a{m}{k}"), 1, Some(format!("w{m}"))));
        }
    }
    for q in 0..20 {
        entries.push((format!("q{q:02}"), 1, None));
    }
    let view = |e: &[(String, u64, Option<String>)]| {
        let r: Vec<(&str, u64, Option<&str>)> =
            e.iter().map(|(i, w, o)| (i.as_str(), *w, o.as_deref())).collect();
        assign_map(&r)
    };
    let p1 = assign(&ms, &view(&entries), &BTreeSet::new(), 3, 7);
    let who = |a: &BTreeMap<String, Vec<String>>, id: &str| {
        a.iter().find(|(_, v)| v.iter().any(|s| s == id)).map(|(m, _)| m.clone())
    };
    let unclaimed: Vec<String> =
        p1.values().flatten().filter(|s| s.starts_with('q')).cloned().collect();
    entries.retain(|(i, _, _)| i != "a50");
    let p2 = assign(&ms, &view(&entries), &BTreeSet::new(), 3, 7);
    let moved = unclaimed
        .iter()
        .filter(|id| who(&p1, id) != who(&p2, id))
        .count();
    assert_eq!(moved, 0, "{moved} of {} unclaimed assignments moved", unclaimed.len());
}

On main it fails: 8 of 8 unclaimed assignments moved.

Local reproductions at 20 workers, 3-minute windows:

• DynamoDB Local answers in 5–8 ms and shows no churn. With 20 ms added per operation it requested 1,979 revocations (0.89 per completion), with store-operation rates close to the real-table run.
• A push-mode MemoryStore churns at 2 ms per operation: 4.65 revocations per completion.
• A deep queue still churns, so this is not a planner running dry. The improve pass moved nothing in these runs; 71% of revocations were a fill-pass reassignment of a split the leader saw no owner for, claimed about one poll interval earlier.

The envelope

One process running every worker, each with its own store handle and coordinator. Equal-weight synthetic splits held 6 s each, progress committed every 5 s, max_in_flight 8, lease_duration 30s, DynamoDB poll_interval 2s, reconcile_interval 5m, rebalance_delay default. An open plan that keeps the queue above one replan interval of consumption.

What is saturated

Nothing on the host. The leader's publishes drive the revocation rate, and the drains leave lanes idle.

Configuration

coordination:
  lease_duration: 30s
  max_in_flight: 8
  reconcile_interval: 5m
  store:
    dynamodb:
      table: <table>
      job: <job>
      region: eu-west-2
      create_table: true
      poll_interval: 2s