Skip to content

balloons: Synchronize() may not finish within timeout with 100 containers and 600+ IRQs #746

Description

@askervin

Bug Description

Starting balloons pod may fail due to exceeding container runtime's plugin_request_timeout that defaults to 2 seconds. The reason is that synchronization may take very long on a large server that has already 100+ containers running. Noticed when enabling IRQ controls in such a system.

Expected behavior

Synchronization should handle 100 containers within the default timeout.

Experienced behavior

It did not.

Steps to reproduce the behavior

Deploy the balloons plugin with IRQ configuration enabled on a server with 600+ IRQs to be managed and 100 containers running. Optionally include CPU class configurations that configure frequencies with cpufreq or SST.

Environment

  • nri-plugins version: e8e29bf (v0.13.1 + balloons IRQ support)
  • nri plugin(s) running: balloons
  • container runtime: containerd

Additional context

Perhaps Synchronize should not apply system configurations (cpufreq, IRQ, ...) for each container separately, as long as there are containers left that may immediately make previous configuration invalid at least in some parts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions