TPD CXO fill-gap NOT fixed in production: ~40-transport ceiling on high-degree visors

[0pcom]

The #4009–4012 CXO fill-gap work does not hold in production. Measured 2026-08-22 (read-only, via svc tpd / tp ls):

visor	local transports	TPD knows	coverage
0323272a… (local)	~640	42–48	~7%
02f9aa58… (Frankfurt)	314	39	~12%

TPD saturates near ~40 per visor regardless of true degree (local has 638, Frankfurt 314, both land at ~40). Confirmed across all fetch paths — CXO cache (47), authoritative --http --no-cxo (42), and the dedicated /transports/stats/<pk> count endpoint (48) — so it's TPD-side stored-data truncation, not subscriber-cache lag or list pagination.

TPD's ID set is essentially a strict subset of the visor's live set (42 of 43 present locally; 598 local IDs absent from TPD; 1 stale TPD-only ID).

Why it matters: this blocks moving dmsg-disc (and TPD) registration onto CXO — the intended fix for the sequence-conflict registration storm (#4086) is 'pending it working for TPD', and TPD's CXO fill is the thing that isn't working.

Caveat to investigate: every local transport shows mode in (responder edge); if registration is initiator-driven, part of the gap may be unregistered responder-edges rather than pure CXO ingest truncation. But the ~40 ceiling on both the count endpoint and the list points at TPD-side truncation of high-degree visors.

[0pcom]

CONFIRMED LIVE (2026-08-22) — root cause is a publisher freeze + a subscriber fill failure, stacking. #4041 is insufficient.

Both ends are failing in production right now, and both builds contain #4041.

1. Publisher freeze (visor side) — #4041's self-heal runs but still fails

My visor (build 95390d35, which contains #4041) logs this every ~10s:

[stats] treestore: republish hit an evicted cached object — clearing encode cache and re-encoding  error=not found
[stats] treestore: publish failed  error=treestore encode/save: not found
[stats] treestore: heartbeat republish failed  error=treestore encode/save: not found

The #4041 self-heal ("clearing encode cache and re-encoding") executes and still returns not found, so the tp-list Root is frozen — the visor cannot publish its current ~640 transports, and TPD keeps serving the ~42 from an old sequence. The re-encode re-walks the in-memory memtree and re-Sets each object; if the evicted object's VALUE is gone from both the write-behind cache and the memtree (only the hash remains), the re-Set has nothing to write → not found again. Fix direction: pin the publisher's OWN-feed object values (never evict the tp-list objects a visor itself authored), or reconstruct the value from source (the transport manager's live list) on re-encode instead of relying on the cache.

2. Subscriber fill failure (TPD side) — separate, also live

TPD's aggregator (build 9365191, also has #4041) can't pull many visors' Roots at all:

[tpd-cxo-aggregator]: root filling broke error="no connections to fill from" seq=1 visor=<pk>
[tpd-cxo-aggregator]: root filling broke error="timeout" seq=1 visor=<pk>

Even when a visor's Root is healthy, TPD often has no live connection to fill from (or times out), so it never ingests the leaf. This compounds the publisher freeze.

Ruled out

Core CXO node/Refs replication is NOT the cause — a localhost two-node test (added as pkg/cxo/node/fillgap_repro_test.go) replicates N=512 entries fully over TCP. The cap is entirely in the treestore publisher freeze + the TPD aggregator connection layer, not object sync.

[0pcom]

Pinned: #4041's repair is on the Set path only; the Inc/reference-walk path is unrepaired

Root cause of the persisting publisher freeze, located precisely:

• fix(cxo): repair filling-item so treestore self-heal can restore evicted objects (real TPD/[stats] freeze) #4041 added the filling-invariant repair to setFilling (pkg/cxo/skyobject/cache.go:940-976) — the Cache.Set path, which carries the object value and can re-write an evicted filling placeholder.
• But incItem (cache.go:1092-1093), the path Cache.Inc uses, routes a filling item to the RAW incFilling, which only Get/Inc's and returns data.ErrNotFound with no repair.
• The treestore publisher's Save reference walk bumps refcounts via Cache.Inc → incItem → incFilling. So when the evicted-and-filling object is encountered during the reference walk (not during the re-encode's Set), it returns not found and the self-heal retry fails identically — the observed heartbeat republish failed: not found every ~10s, Root frozen.

This is why #4041's regression test passes (it drives the Set path, which now repairs) while production stays frozen (it also hits the Inc path).

Fix options

1. Prevent the trap: pin a publisher's OWN authored feed objects so peer-subscription filler-incs never turn them into val==nil placeholders (distinguish own-published from peer-filled in the cache).
2. Or make the reference walk able to recover: when incItem hits a filling item whose backing is gone, re-materialize from the in-memory publisher tree (the value IS available in memNode.leaves / re-encode) before Inc — i.e. thread a value source into the walk.

Option 1 is the more robust (avoids the trap entirely). Needs a repro that exercises the Inc path, which the existing filling-trap test does not.