exporter: fork-straddling trace ranges + /committee cardinality doc (#2968 items 1, 2) - #2975
Conversation
validateValidatorRequest evaluated the committee-duty fork gate at request.To, so a validator-traces window whose tail crossed the Boole fork was rejected wholesale - a dashboard polling a fixed window starts 400-ing the moment its tail crosses the fork, losing the still-servable pre-fork slots. Evaluate the gate at request.From instead, and report each post-fork committee-duty slot lacking pubkeys/indices as a non-fatal per-slot note (ErrPostForkCommitteeDutyNote) in the response Errors instead of silently returning nothing for it. The HTTP handler treats a response whose only errors are such notes as a partial 200 - without that, a straddling range with zero pre-fork traces would 500. Item 1 of #2968.
Without a roles filter the endpoint returns one trace per (slot, committeeID, role) - up to two rows per (slot, committeeID) post-fork, distinguished by the role field. Additive-field back-compat holds; cardinality back-compat does not, so state it in the OpenAPI description (regenerated, not hand-edited). Item 2 of #2968.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Greptile SummaryThis PR permits validator trace ranges to straddle the Boole fork and documents the committee endpoint’s role-based cardinality.
Confidence Score: 4/5The PR should not merge until the newly accepted fork-straddling path enforces a bounded slot range or avoids per-slot note amplification. A caller can select a pre-fork lower bound and an arbitrarily large post-fork upper bound, causing an unbounded slot loop and one allocated response error per post-fork slot and role; the maximum uint64 bound additionally prevents loop termination after counter wraparound. Files Needing Attention: exporter/validator.go
|
| Filename | Overview |
|---|---|
| exporter/validator.go | Changes fork gating and adds per-slot notes, but newly permits unbounded post-fork iteration and response amplification for straddling requests. |
| api/handlers/exporter/validator_http.go | Correctly distinguishes sentinel notes from genuine errors before allowing an empty partial response. |
| api/handlers/exporter/exporter_test.go | Adds HTTP coverage for note-only partial responses and mixed genuine-error behavior. |
| exporter/validator_test.go | Adds focused validation and core tests around the Boole fork boundary. |
| api/handlers/exporter/committee_http.go | Clarifies unfiltered committee response cardinality in the endpoint documentation. |
| docs/api/ssvnode.openapi.json | Regenerates the JSON OpenAPI description consistently with the handler annotation. |
| docs/api/ssvnode.openapi.yaml | Regenerates the YAML OpenAPI description consistently with the handler annotation. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
A["Validator traces request"] --> B{"from is post-Boole committee duty?"}
B -->|Yes, no filters| C["Reject request"]
B -->|No| D["Iterate from through to"]
D --> E{"Slot is post-Boole committee duty?"}
E -->|Yes, no filters| F["Append non-fatal per-slot note"]
E -->|No| G["Read validator traces"]
F --> D
G --> D
D --> H{"Only notes and no traces?"}
H -->|Yes| I["HTTP 200 with notes"]
H -->|No genuine errors| J["HTTP 500"]
H -->|Traces available| K["HTTP 200 partial response"]
Reviews (1): Last reviewed commit: "api/exporter: document the /committee pe..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
A fork-straddling range without pubkeys/indices previously appended one non-fatal note per post-fork slot per role, growing the response linearly with the range size. Fork state is monotonic in slot, so the skipped tail is contiguous: record where it starts and emit a single note per role covering the whole post-fork range instead.
iurii-ssv
left a comment
iurii-ssv
left a comment
There was a problem hiding this comment.
Check out the Greptile comment above + left two more minor nits.
ovidiu-ssv-labs
left a comment
ovidiu-ssv-labs
left a comment
There was a problem hiding this comment.
The core mechanism is sound — evaluating the fork gate at From and emitting per-slot non-fatal notes is the right shape, the sentinel/errors.Is design is clean, and the test-network cloning is genuinely leak-free. But the PR documents the smaller back-compat deviation (/committee cardinality) while leaving the larger one (/traces/validator 400 -> 200-with-empty-data) undocumented, the test guarding the new 500 escape hatch doesn't actually assert the status code, and /decideds remains fork-blind for the same two roles this PR just made fork-aware on /traces/validator. [verdict: with_fixes]
Finding 1 · [IMPORTANT] Document the /traces/validator 400 -> 200 contract change in the OpenAPI description — api/handlers/exporter/validator_http.go:15
The PR documents the smaller back-compat break and leaves the larger one undocumented.
Item 2 of this PR adds an OpenAPI note to /committee because "cardinality back-compat does not hold." That is the right instinct — but item 1 changes a status code, which is a strictly harder break for clients, and gets no doc change at all. validator_http.go:15 still reads:
// @Description Returns consensus, decided, and message traces for the requested validator duties.The behavior change, concretely. For roles=[AGGREGATOR] (or SYNC_COMMITTEE_CONTRIBUTION) with no pubkeys/indices:
| request window | before | after |
|---|---|---|
| fully pre-Boole | 200 + data | 200 + data (unchanged) |
| straddles Boole | 400 | 200, pre-fork data only, post-fork slots reported as strings in errors |
| fully post-Boole | 400 | 400 (unchanged) |
Why it matters. The 400 was a contract signal: "you must supply indices/pubkeys or this data is not retrievable." A client that treated non-2xx as "my query is wrong, fix it" now receives a 200 and, in the sparse-aggregator case that TestExporterValidatorTraces_ForkGating_ZeroPreForkTraces covers, data: [] with a populated errors array. A dashboard that only reads data will render an empty chart and report success — silently missing every post-fork aggregator duty in the window.
There is also an undocumented cliff: a window starting one slot before the fork returns 200-with-notes, while the same window shifted one slot forward returns 400. Nothing tells an API consumer that.
Additional wrinkle: ValidatorTracesResponse.Errors is a flat []string (validator_model.go:63). Post-fork notes and genuine store failures land in the same untyped array, so a client cannot programmatically distinguish "partial coverage, expected" from "the store is broken." The sentinel exists in Go (ErrPostForkCommitteeDutyNote) but is flattened to a string at the API boundary.
Suggested fix: Extend the godoc description and regenerate via make openapi, mirroring what item 2 did for /committee:
// @Description Returns consensus, decided, and message traces for the requested validator duties.
// @Description For AGGREGATOR and SYNC_COMMITTEE_CONTRIBUTION the fork state is evaluated at 'from': a range whose
// @Description 'from' is post-Boole and that supplies no 'pubkeys'/'indices' is rejected with 400. A range whose
// @Description 'from' is pre-Boole is accepted and served partially — the post-Boole slots are omitted from 'data'
// @Description and reported per slot in 'errors' with the text "committee duty post-fork". Such a response is a 200
// @Description even when 'data' is empty; clients must inspect 'errors' to detect partial coverage, and should
// @Description supply 'indices'/'pubkeys' or use /traces/committee to retrieve the post-fork portion.If you want the stronger fix, promote the notes out of the flat errors array into a typed field (e.g. "partial": [{"slot": N, "role": "AGGREGATOR", "reason": "post_fork_committee_duty"}]) so clients can branch on it without string matching. Given this is a new field, additive back-compat holds.
Finding 2 · [IMPORTANT] /decideds is still fork-blind for AGGREGATOR / SYNC_COMMITTEE_CONTRIBUTION — exporter/decided.go:41
Adjacent gap — pre-existing, not introduced here, but this PR makes it the last fork-blind routing site in the package.
After this PR, isCommitteeDutyAtSlot (exporter/validator.go:278) is the only Boole-aware routing point in exporter/.
TraceDecidedsCore routes on a hardcoded role switch with no slot/fork input:
// exporter/decided.go:41-46
switch role {
case spectypes.BNRoleAttester, spectypes.BNRoleSyncCommittee:
roleParticipantsIdx, roleErrs = e.getCommitteeDecidedsForRole(slot, indices, role)
default:
roleParticipantsIdx, roleErrs = e.getValidatorDecidedsForRole(slot, indices, role)
}Mechanism. Post-Boole, aggregator and sync-committee-contribution duties execute under the RoleAggregatorCommittee runner (protocol/v2/ssv/runner/aggregator_committee.go:86), and the observer stores their traces on the committee path (protocol/v2/ssv/validator/committee_observer.go:117 gates RoleCommittee || RoleAggregatorCommittee onto the committee-ID branch). So post-fork there is no ValidatorDutyTrace for BNRoleAggregator to find. Yet the default arm sends it to getValidatorDecidedsForRole -> GetAllValidatorDecideds(BNRoleAggregator, slot) -> c.store.GetValidatorDuties(role, slot) (exporter/dutytracer/store.go:421-424), which reads the validator-duty index and returns nothing.
The strongest evidence that this is a bug and not intended: the store layer is already built to serve these roles from the committee side. committeeRunnerRolesForBeaconRoles explicitly maps aggregator-family beacon roles to RoleAggregatorCommittee, and GetCommitteeDecideds defaults to []RunnerRole{RoleCommittee, RoleAggregatorCommittee}. There is even a dedicated test, TestCollector_GetCommitteeDecideds_RoleFiltering, asserting GetCommitteeDecideds(slot, index, spectypes.BNRoleAggregator) works. Nothing in production calls it — the decided.go:42 switch never routes BNRoleAggregator to the committee path.
Impact. Post-Boole, /v1/exporter/decideds with roles=AGGREGATOR or SYNC_COMMITTEE_CONTRIBUTION returns an empty participants list with a 200. Silent data loss — no error, no note, no 400. Worse than the /traces/validator case this PR just fixed, because there the user at least gets a note. Any duty-syncer or dashboard consuming decideds for these roles will show zero participation post-fork and conclude the operators stopped performing the duty.
Scope. This is outside items 1-2 of #2968 and not in this diff, so not blocking — but it is the natural item 3 of the same umbrella and should be filed before Boole activates, not after.
Suggested fix: Route the decideds switch through the same fork-aware predicate the traces path now uses, so there is exactly one place that knows about Boole:
for _, role := range request.Roles {
for s := request.From; s <= request.To; s++ {
slot := phase0.Slot(s)
var roleParticipantsIdx []dutytracer.ParticipantsRangeIndexEntry
var roleErrs *multierror.Error
if e.isCommitteeDutyAtSlot(role, slot) {
roleParticipantsIdx, roleErrs = e.getCommitteeDecidedsForRole(slot, indices, role)
} else {
roleParticipantsIdx, roleErrs = e.getValidatorDecidedsForRole(slot, indices, role)
}
...This is behavior-preserving pre-Boole (isCommitteeDutyAtSlot returns true unconditionally for ATTESTER/SYNC_COMMITTEE and false for the aggregator family pre-fork), and correct post-Boole. Note the loop nesting in decided.go is role-outer/slot-inner, the inverse of validator.go, so isCommitteeDutyAtSlot must be evaluated inside the slot loop as above. Add a straddling-range test mirroring TestValidatorTracesCore_StraddlingFork.
If you'd rather keep this PR tight, file it as item 3 of #2968 and land it separately — but before Boole activates on mainnet.
3 findings approved — 2 could not be inlined (target line not in this diff), 1 inlined below.
…error test The fork gate moved from the range's upper bound to the lower bound; two test doc comments still described the old behavior. Also assert the exact status code and surfaced message in the 'genuine error alongside notes' subtest, so a future misroute to the 400 branch can't pass silently (require.Error alone was satisfied by either branch).
A fork-straddling range without pubkeys/indices used to be rejected with 400; it now returns 200 with the pre-fork portion and per-role notes in 'errors'. That is a status-code contract change for clients, so spell it out in the OpenAPI description like the /committee cardinality note.
|
@ovidiu-ssv-labs re the two findings without inline threads: Finding 1: fixed in 32c9d96 — extended the /traces/validator OpenAPI description with the partial-coverage contract, mirroring the /committee note. One wording change vs. the suggestion: since 82535ff the notes are aggregated into one per role covering the post-fork tail, rather than one per slot. The typed Finding 2: agreed on all points — filed as #2987 (item 3 of the #2968 umbrella) so it lands before Boole activates. |
iurii-ssv
left a comment
iurii-ssv
left a comment
There was a problem hiding this comment.
Couple more things to check out.
Moving the fork gate from 'to' to 'from' removed the 400 that shielded unfiltered committee-duty requests from the inclusive per-slot loop's uint64 wraparound hang. Reject the guaranteed-hang value in validation; the endpoint-wide range bound remains tracked in #2986.
…coexisting The straddling-range 200 tests only asserted empty data with notes; add an unfiltered case whose pre-fork slots return real traces so a non-empty 'data' and the post-fork note are proven to coexist in one response, and pin that the unfiltered validator path is never consulted post-fork.
|
@ovidiu-ssv-labs please lets report here when we tested this PR and exporter data against pre and post fork operators. when you approve lets merge |
Test report: build with the PR against build without the PRAll tests PASSED - with mentionsI tested this PR on the hoodi-stage exporter. I used two builds on the same pod and the same volume. Both builds use Boole epoch 117422. The fork slot is 3757504.
The two builds use the same database. Therefore the row counts are comparable. Result: 41 test cases. 3 cases differ but expected. 38 cases are the same. The three cases that differ
Without the PR, the exporter refuses the request: With the PR, the exporter serves the rows before the fork. It adds one note for each role: The note gives the exact slots that the exporter did not serve. VAL-03 shows that the exporter groups the notes. The range holds 7 slots after the fork and the request holds 2 roles. The exporter returns 2 notes. It does not return 14 notes. One control case shows the value of the change. Without the PR, the same range with The 38 cases that are the sameNo behaviour changed outside the three cases above. These cases include:
Status of each change in the PR
No regression found. Note on additional testsThe 41 cases also cover the three exporter routes for their own sake, before the fork and after the fork. These tests found four items that this PR does not cause and does not correct. The most important item: the |
iurii-ssv
left a comment
iurii-ssv
left a comment
There was a problem hiding this comment.
Just couple minor nits
…decideds validators The 'to == MaxUint64' guard that prevents the inclusive per-slot loop from wrapping only covered /traces/validator; /traces/committee and /decideds run the identical loop and could still be pinned forever. Hoist the range check into validateSlotRange and call it from all three validators. The range-size cap stays in #2986.
…ork() The clone-and-pin of TestNetwork.SSV.Forks.Boole was inlined four times in this file; the exporter package's pre/post/straddling helpers are not importable here, so add one local helper and use it at all four sites.
e22aeb9
The surfaced note wrapped a sentinel that already read 'committee duty post-fork requires pubkeys or indices' and then repeated both halves in the format string, so a client saw 'committee duty post-fork' twice and 'pubkeys or indices' three times. Trim the sentinel to the bare marker and let the format string carry the actionable hint once. errors.Is matching and the 'committee duty post-fork' assertions are unaffected.
5 participants
Items 1 and 2 of #2968, one commit each.
Item 1 —
validateValidatorRequestevaluated the committee-duty fork gate atrequest.To, so a validator-traces window whose tail crossed the Boole fork was rejected wholesale: a dashboard polling a fixed window starts 400-ing the moment its tail crosses the fork, losing the still-servable pre-fork slots. The gate now evaluates atrequest.From, and each post-fork committee-duty slot lacking pubkeys/indices is reported as a non-fatal per-slot note (ErrPostForkCommitteeDutyNote,errors.Is-able) in the responseErrorsinstead of being silently empty. The HTTP handler treats a response whose only errors are such notes as a partial 200 — review caught that without this, a straddling range with zero pre-fork traces (common: aggregator duties are sparse) would have 500'd.from-already-post-fork and unfiltered ATTESTER/SYNC_COMMITTEE requests reject exactly as before.Item 2 — docs-only: the
/committeeOpenAPI description now states that absent arolesfilter the endpoint returns one trace per (slot, committeeID, role) — up to two rows per (slot, committeeID) post-fork, distinguished by therolefield. Additive-field back-compat holds; cardinality back-compat does not, hence the note. Regenerated viamake openapi, diff limited to description text.Covered by fork-phase tests (per-test
TestNetworkcopies withForks.Boolepinned), including HTTP-level cases for the straddling 200-partial path and the genuine-error-still-500 path.