Skip to content

pre-merge-basic: move hotplug test to end of the test list - #67

Open
Anil (qcom-anilyada) wants to merge 1 commit into
qualcomm-linux:masterfrom
qcom-anilyada:test/reorder-hotplug
Open

Anil (qcom-anilyada) wants to merge 1 commit into
qualcomm-linux:masterfrom
qcom-anilyada:test/reorder-hotplug

Conversation

@qcom-anilyada

Copy link
Copy Markdown

Running hotplug immediately after boot causes intermittent failures with the error "[hotplug] CPU1 offline returned EBUSY" as the system has not fully settled down before the hotplug operation is issued. Moving hotplug to the end of the test list gives the system enough time to stabilize before the test runs.

Signed-off-by: Anil Yadav anilyada@qti.qualcomm.com

Running hotplug immediately after boot causes intermittent failures with
the error "[hotplug] CPU1 offline returned EBUSY" as the system has not
fully settled down before the hotplug operation is issued. Moving hotplug
to the end of the test list gives the system enough time to stabilize
before the test runs.

Signed-off-by: Anil Yadav <anilyada@qti.qualcomm.com>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but this is not the way to go. You need to find the root cause of the issue, not change the order of the tests. In fact it should be possible to run tests in a random order.

@smuppand

Srikanth Muppandam (smuppand) commented Jun 12, 2026

Copy link
Copy Markdown

Sorry, but this is not the way to go. You need to find the root cause of the issue, not change the order of the tests. In fact it should be possible to run tests in a random order.

I agree that ideally hotplug tests should be order-independent and runnable in any sequence. We are not trying to mask the issue via ordering. The failure is due to boot-time stabilization gap in LAVA execution model. However, what we’re observing in LAVA is not just a logical dependency on test order, but a platform readiness timing issue immediately after boot.
In our setup:

  • LAVA starts executing tests serially as soon as the shell becomes available, which does not necessarily guarantee that

  • all CPUs are fully brought up and stabilized background kernel workqueues / init flows related to CPU bring-up have completed

As a result, invoking CPU hotplug very early intermittently returns:
CPU1 offline returned EBUSY

@mwasilew

Copy link
Copy Markdown

This sounds like either bug in the kernel, init process or the test.

@smuppand

Copy link
Copy Markdown

This sounds like either bug in the kernel, init process or the test.

hotplug is a disruptive CPU-topology mutation test, so it should not be the first post-boot readiness test in a pre-merge sequence.

Anil (@qcom-anilyada) Do you have any observations on whether moving the test sequence has resulted in any reported test failures?

@mwasilew

Copy link
Copy Markdown

This sounds like either bug in the kernel, init process or the test.

hotplug is a disruptive CPU-topology mutation test, so it should not be the first post-boot readiness test in a pre-merge sequence.

What should happen if I only run hotplug test in a test job?

@qcom-anilyada

Copy link
Copy Markdown
Author

Here are the jobs to compare all requested scenarios:

@mwasilew

Copy link
Copy Markdown

As you demonstrated, there is a problem with a test script.

@smuppand

Copy link
Copy Markdown

As you demonstrated, there is a problem with a test script.

I will address this in test by deferring CPUs that return EBUSY after the normal retry window, continuing with the remaining CPUs, and retrying the deferred CPUs at the end. That keeps CPU0 covered, avoids adding a larger fixed boot sleep. If a CPU still returns EBUSY during the deferred pass, the test will still fail with the existing dmesg evidence.

With that script-side fix, the lava-test-plans reordering becomes optional rather than required for correctness.

@mwasilew

Copy link
Copy Markdown

Sounds good to me. Are there any other signals that could be used to determine that the "initialization" phase is completed at the boot time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants