Skip to content

Exporter API: four defects found in the trace and decideds routes #2996

Description

@ovidiu-ssv-labs

These four items come from tests on the hoodi-stage exporter on 2026-08-21. The tests used build 1c85cf90 and build f77f5095. Both builds show the same behaviour for all four items. PR #2975 does not cause these items. PR #2975 does not correct them. But they are all Boole Fork relevant and we'll need to decide whether they require fixing or not.


1. The /v1/exporter/decideds route ignores the Boole fork

The route selects the data path from the role only. It does not use the slot. See exporter/decided.go:39-45:

switch role {
case spectypes.BNRoleAttester, spectypes.BNRoleSyncCommittee:
    roleParticipantsIdx, roleErrs = e.getCommitteeDecidedsForRole(slot, indices, role)
default:
    roleParticipantsIdx, roleErrs = e.getValidatorDecidedsForRole(slot, indices, role)
}

After the Boole fork, the AGGREGATOR role and the SYNC_COMMITTEE_CONTRIBUTION role are committee duties. The runner is AGGREGATOR_COMMITTEE. This switch sends them to the single-validator store. The store does not hold them. The route then returns no rows.

The route returns HTTP 200. The route returns an empty errors array. Measured results for the same role on one exporter:

Slot range Rows
Before the fork 126
After the fork 0
Across the fork 58 (only slots before the fork)

A client cannot see the difference between two conditions. The first condition is "no aggregator duty occurred". The second condition is "this route cannot read aggregator duties after the fork".

The /v1/exporter/traces/validator route gives an error for the same condition. The /v1/exporter/decideds route gives no error.


2. The /v1/exporter/decideds route has no limit on the slot range

The function validateDecidedRequest tests two conditions only. It tests that from is not more than to. It tests that the request has one role or more. See exporter/decided.go:121-131.

The route then starts a loop for each slot. See exporter/decided.go:35:

for _, role := range request.Roles {
    for s := request.From; s <= request.To; s++ {

PR #2975 adds a limit to exporter/validator.go. That limit refuses a to value that is equal to MaxUint64. The limit applies to one value only. A to value below MaxUint64 is still permitted. PR #2975 does not add a limit to exporter/decided.go. Therefore this route accepts a very large slot range.

We did not test this condition on a shared exporter. Item 3 gives the reason.


3. A large slot range holds a handler goroutine. The client cannot stop it

The file exporter/validator.go does not use a context. The loop for each slot does not test for cancellation. Go does not stop a handler when the client closes the connection.

One request caused this condition on 2026-08-21. The request used a to value of MaxUint64 - 1. The new limit does not refuse this value:

  • 16 goroutines stopped in ValidatorTracesCore at exporter/validator.go:61. The data comes from /debug/pprof/goroutine.
  • The process used approximately 344% of one core. The limit for the container is 3 cores.
  • The counter nr_throttled increased from 465 to 485 in 15 seconds. Necessary work competed with these goroutines.
  • The client stopped. The goroutines continued. Only a restart of the pod removed them.

One request can hold a handler and use CPU until an operator restarts the pod. Issue #2986 records the missing limit on the slot range. This behaviour makes the missing limit more serious.

Recommendation: test this condition on a temporary environment only.


4. After the fork, a validator trace for an aggregator duty has no message data

The type ValidatorCommitteeTrace includes the full trace type. See exporter/models.go:74-77:

type ValidatorCommitteeTrace struct {
	traces.ValidatorDutyTrace          // holds Rounds, Decideds, Pre, Post
	CommitteeID *spectypes.CommitteeID
}

The function getValidatorCommitteeDutiesForRoleAndSlot builds this type from a committee duty. See exporter/validator.go:167. The function sets the committee ID. The function does not set the message fields. Measured results for the same role:

Slot range Rows Rows with message data
Before the fork 1 1
After the fork 1 0

After the fork, the row shows that the validator had the duty. The row shows the committee. The row shows no messages.

Part of this behaviour is correct. After the fork, consensus belongs to the committee. One validator cannot own a QBFT round. But the partial signatures in pre and post can possibly show the validator. The signer data for the aggregator committee holds ValidatorIdx. The membership test in the same function reads this field.

Question for the exporter owner: is this behaviour correct? If it is correct, remove the message fields from the response for this path, or record the behaviour in the API documentation.


Test data

Happy to provide test harness details and results if necessary.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions