feat(ci): support exhaustive tests for PRs (code changes or GH comment) by v1v · Pull Request #18327 · elastic/logstash

v1v · 2025-10-21T14:30:31Z

Release notes

rn:skip

What does this PR do?

This pull request introduces a new "smart exhaustive tests" pipeline for Logstash, integrating it into the Buildkite CI configuration and documenting it in the service catalog. The main goal is to trigger exhaustive test runs automatically for pull requests that modify files in the qa/acceptance/ directory or when specific GitHub comments are made, improving test coverage and automation for critical changes.

Why is it important/What is the impact to the user?

Test the code that will rarely impact the standard PR test suite, but will exercise the acceptance tests.

Checklist

My code follows the style guidelines of this project
~~[ ] I have commented my code, particularly in hard-to-understand areas~~
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files (and/or docker env variables)~~
~~[ ] I have added tests that prove my fix is effective or that my feature works~~

How to test this PR locally

Add a path to always_require_ci_on_changed in pull-request.json for logstash-smart-exhaustive-tests-pipeline, then apply changes in that directory, commit and push. That will trigger the exhaustive tests pipeline in Buildkite.

Caveats

The existing BK PR Bot does not support for code changes and GitHub comments, without running the first buidl when a PR is created.

That's the reason for creating the logstash-smart-exhaustive-tests-pipeline to actually filter:

code changes in the qa/acceptance folder
github comment

This should help with running faster builds and reduce the overhead of running the exhaustive tests for the first build of each new PR.

Duplicates #18202, that I messed up with the rebase.

github-actions · 2025-10-21T14:30:48Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2025-10-21T14:31:15Z

This pull request does not have a backport label. Could you fix it @v1v? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

fr4nc1sc0-r4m0n

LGTM in general. Just a little question

fr4nc1sc0-r4m0n · 2025-10-22T08:45:44Z

+                    - ELASTIC_SLACK_NOTIFICATIONS_ENABLE=false
+
+  - label: "Trigger logstash-exhaustive-tests-pipeline for GitHub comments"
+    if: build.env("GITHUB_PR_TRIGGER_COMMENT") != ""


What happens if a file has changed in the qa/acceptance/ and a comment in the PR is made?
will be the pipeline logstash-exhaustive-tests-pipeline executed twice?

Two different events:

push event with the file changeset

github comment matches the values in https://github.com/elastic/logstash/pull/18327/files#diff-646855f9c36bcc3f7a5c49c1fa33be9b7771d99e5e2219332b215eaac6e51537R20-R21

Each event should be treated independently.

Otherwise, what do you mean by your question?

Understood. We want to execute the tests in both cases and every time when push and the right files have changed or when a comment that matches has made.
Thanks!

fr4nc1sc0-r4m0n

LGTM

v1v · 2025-10-29T14:38:45Z

@@ -0,0 +1,33 @@
+steps:


As long as the BK pr bot does not support running the very first build for PRs. We use this intermediate pipeline. This will help with running faster builds regardless, although it adds the complexity for maintaining a new pipeline.

donoghuc

As someone who often uses buildkite UI to go and kick off a PR build, i'm a HUGE fan of the GH comment trigger. That will be a huge help.

I'm a bit more neutral on the file watch trigger, but i dont think it will hurt anything and it establishes a nice pattern for doing that for the future.

Have a few non-critical clarifying questions and suggestions but i'm a massive plus one on this. Thanks!

donoghuc · 2025-10-29T17:25:41Z

+				"dependabot[bot]",
+				"mergify[bot]",
+				"github-actions[bot]",
+				"elastic-vault-github-plugin-prod[bot]"


Why is "elastic-vault-github-plugin-prod[bot]" included here byt not for logstash-smart-exhaustive-tests-pipeline?

It was already there, no idea the rationale, but I think we don't need it in this case in the new pipeline.

That's the GH ephemeral token app. I don't think PRs created by that user affect any files that could trigger lostash-smart-exhaustive-tests-pipeline.

Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>

donoghuc

🚀

elasticmachine · 2025-10-29T19:00:33Z

💚 Build Succeeded

Buildkite Build
Commit: 806cb8f

History

💛 Build #3727 was flaky 0fb75c5

cc @v1v

…t) (elastic#18327)

* feat(ci): support exhaustive tests for PRs (code changes or GH comment) (#18327) * bk: fix regex and add new file (#18382) * bk(smart exhaustive tests): tune the steps to support GH comments and changesets (#18383) * chore: remove test file for exhaustive tests and normalise github commands (#18384) --------- Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* feat(ci): support exhaustive tests for PRs (code changes or GH comment) (#18327) * bk: fix regex and add new file (#18382) * bk(smart exhaustive tests): tune the steps to support GH comments and changesets (#18383) * chore: remove test file for exhaustive tests and normalise github commands (#18384) --------- Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> (cherry picked from commit 906d5bd)

* feat(ci): support exhaustive tests for PRs (code changes or GH comment) (#18327) * bk: fix regex and add new file (#18382) * bk(smart exhaustive tests): tune the steps to support GH comments and changesets (#18383) * chore: remove test file for exhaustive tests and normalise github commands (#18384) --------- Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> (cherry picked from commit 906d5bd) # Conflicts: # .buildkite/pull-requests.json

* feat(ci): support exhaustive tests for PRs (code changes or GH comment) (#18327) * bk: fix regex and add new file (#18382) * bk(smart exhaustive tests): tune the steps to support GH comments and changesets (#18383) * chore: remove test file for exhaustive tests and normalise github commands (#18384) --------- (cherry picked from commit 906d5bd) Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

feat(ci): support exhaustive tests for PRs (code changes or GH comment)

0fb75c5

v1v requested review from a team and donoghuc October 21, 2025 14:30

v1v self-assigned this Oct 21, 2025

v1v added the backport-skip Skip automated backport with mergify label Oct 21, 2025

v1v mentioned this pull request Oct 21, 2025

[ci] Allow triggering exhaustive tests from PRs depending on path #18202

Closed

1 task

fr4nc1sc0-r4m0n reviewed Oct 22, 2025

View reviewed changes

fr4nc1sc0-r4m0n approved these changes Oct 22, 2025

View reviewed changes

v1v commented Oct 29, 2025

View reviewed changes

v1v mentioned this pull request Oct 29, 2025

fix(bk): serverless-beats-tests start failing after merging #10734 elastic/elastic-agent#10906

Merged

8 tasks

donoghuc approved these changes Oct 29, 2025

View reviewed changes

v1v and others added 2 commits October 29, 2025 19:35

Update .buildkite/pull-requests.json

dae7fb6

Co-authored-by: Cas Donoghue <cas.donoghue@gmail.com>

Update .buildkite/smart_exhaustive_tests_pipeline.yml

806cb8f

donoghuc approved these changes Oct 29, 2025

View reviewed changes

v1v merged commit fc9ad7f into elastic:main Oct 30, 2025
11 checks passed

v1v mentioned this pull request Oct 30, 2025

bk: fix regex and add new file #18382

Merged

5 tasks

donoghuc pushed a commit to donoghuc/logstash that referenced this pull request Oct 31, 2025

feat(ci): support exhaustive tests for PRs (code changes or GH commen…

f41001a

…t) (elastic#18327)

donoghuc mentioned this pull request Oct 31, 2025

Backport smart exhaustive tests #18386

Merged

This was referenced Nov 1, 2025

[9.1] (backport #18386) Backport smart exhaustive tests #18388

Merged

[8.19] (backport #18386) Backport smart exhaustive tests #18389

Merged

Conversation

v1v commented Oct 21, 2025

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

How to test this PR locally

Caveats

Uh oh!

github-actions Bot commented Oct 21, 2025

🤖 GitHub comments

Uh oh!

mergify Bot commented Oct 21, 2025

Uh oh!

fr4nc1sc0-r4m0n left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fr4nc1sc0-r4m0n Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

fr4nc1sc0-r4m0n Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

fr4nc1sc0-r4m0n left a comment

Choose a reason for hiding this comment

Uh oh!

v1v Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

donoghuc left a comment

Choose a reason for hiding this comment

Uh oh!

donoghuc Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

v1v Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

donoghuc left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 29, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fr4nc1sc0-r4m0n left a comment •

edited

Loading