Skip to content

Include compRefTests - #11134

Open
Matilda-Clerke wants to merge 14 commits into
Consensys:masterfrom
Matilda-Clerke:integrate-comp-ref-tests
Open

Include compRefTests#11134
Matilda-Clerke wants to merge 14 commits into
Consensys:masterfrom
Matilda-Clerke:integrate-comp-ref-tests

Conversation

@Matilda-Clerke

@Matilda-Clerke Matilda-Clerke commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Integrate new compliance reference tests into teku's CI process

https://github.com/Consensys/teku-internal/issues/277


Note

Medium Risk
Production fork-choice timeliness is tied to block import outcomes; incorrect confirm/invalidate logic could affect timely/late head signaling, though changes are scoped to retry and rejected-gossip paths.

Overview
Adds consensus-specs compliance reference tests to CI by downloading comptests.tar.gz (tagged releases only; skipped on nightly) and extracting just the fork_choice_compliance subtree so the full archive is not expanded.

Updates ForkChoiceTestExecutor for those vectors: attestations deferred by fork choice are no longer treated as successful, and viable_for_head_roots_and_weights can assert optional payload status per head root.

Fixes block timeliness so observations stay unconfirmed until a block is successfully imported: failed or rejected import attempts (including gossip REJECT) clear stale unconfirmed entries, later arrivals can refresh the value, and confirm on successful import. BlockTimelinessTracker tests reflect the new refresh/invalidate/confirm behavior.

Reviewed by Cursor Bugbot for commit a1128ad. Bugbot is set up for automated code reviews on this repo. Configure here.

@Matilda-Clerke

Copy link
Copy Markdown
Contributor Author

According to claude, this change will not take effect until the CI process switching to a new reference test version (i.e. ${refTestVersion} is updated in build.gradle) due to caching of downloaded test artifacts.

@Matilda-Clerke
Matilda-Clerke marked this pull request as ready for review August 19, 2026 02:33
Comment thread build.gradle
@rolfyone

Copy link
Copy Markdown
Contributor

According to claude, this change will not take effect until the CI process switching to a new reference test version (i.e. ${refTestVersion} is updated in build.gradle) due to caching of downloaded test artifacts.

did it run locally?

@lucassaldanha

Copy link
Copy Markdown
Member

I remember talking to Mikhail and Alex about this, and one of our concerns was how long executing the tests would take. We need some sort of comparison to understand the impact this will have on our reference tests run to understand the tradeoff of adding these new tests to the current CI pipeline.

If they aren't too expensive to run, it is likely fine. But if they add too much time, we might need to consider some alternative (e.g. run it nightly).

@Matilda-Clerke

Matilda-Clerke commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Looks like the full change in previous commits made by Claude caused significant regression. I'll force push a simpler change that doesn't.

Edit: Actually, it looks like someone already a fix for this issue on master, so I'll just include that instead.

@Matilda-Clerke
Matilda-Clerke force-pushed the integrate-comp-ref-tests branch from 425aa46 to b81364b Compare August 25, 2026 00:09
@Matilda-Clerke
Matilda-Clerke force-pushed the integrate-comp-ref-tests branch from b81364b to d041005 Compare August 25, 2026 00:20

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 2aa5421. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants