High CPU usage

[slalomsk8er]

I tested a set of Service Checks and got a 18% increase in CPU usage on a 2 core virtual server.

A 18%+ increase in CPU usage for monitoring over the whole cluster is not acceptable for us.

The configured services

Icinga 2 + Icinga PowerShell Service enabled

Icinga 2 + Icinga PowerShell Service disabled

Any Ideas on reducing the impact of the checks?

[LordHepipud]

Hello and thank you for the report. Indeed on machines with less cores the CPU impact might be higher.
In general the performance impact should only be a short peek during the call and not exceed a longer period.

Is the reported high CPU usage only present in short "bursts" or is the entire CPU usage constantly higher? We are currently investigating different solutions to decrease the overall impact of the Framework.
Right now it would be important for me to understand the impact on the system itself, if the load is in general higher or if it only increases during the exection of plugins.

[slalomsk8er]

The high CPU usage happens in bursts but I think they aren't so short.

Icinga2 & Icinga PowerShell Service disabled

I measured on a different server that had no service checks activated yet and it's flat same as if I disabled the services.

Also I see the peaks coresponding to the PowerShell process

[vr255]

Hello,

we can also see this behaviour in our environment.
Mostly when the CPU check is executed.

This has the unpleasant side effect that we get a lot of false alarms.

[LordHepipud]

Thank you for all the detailed reports. We are already taking a look on this to figure out how we can reduce the overall impact during plugin execution.

The biggest "issue" is that plugins do not remember their last state, which means all Performance Counter and internal objects have to be re-initialised.

[drapiti]

Hi, we also have this problem and on some critical systems we have had to disable the agent and service, if there is anything we can do to help we are available.

[LordHepipud]

To get a better understanding of the current impact and to provide possible solutions in the future (and for internal testing) it would be helpful get some additionals data:

• List of checks being assigned to the system
• Check interval for each check
• Total amount of CPU cores availables
• Frequenzy of the CPU cores (or the hypervisor below, in case it's limited the assigned max frequency

This will help a lot to build proper test environments to see where we can reduce overall impact.

[drapiti]

Here is the info of the applied service set:

These checks are all using the latest powershell framework. This is the current base service set applied to all windows machines. We have many VMs with 1 to 2 vcpu machines which are often at high load therefore these checks really kill the system.
Cpu's speed is typicall between 2.1-2.6 mhz but cpu usage is apparent on all systems.

The below graph is quite explicit, the cpu and memory initially at 1 minute intervals, then at 5m and finally the last step in the graph we removed the entire above service set from the specific machine. This machine has 2 vcpu running at 2.6mhz.

Note. Currently our systems are running the ms scom agent which is consuming 1/3 of the resources while collecting much more data including the information provided in the above graph.

Other details of the specific machine without any of the above icinga services:

I will add that we have necessarily needed to suspend all monitoring on windows systems, so this is quite critical. Our users will not accept this type of impact on their systems.

[slalomsk8er]

the requested list with &addColumns=service_check_interval

2 vCPUs @ 2.1 GHz

[LordHepipud]

Thanks a lot for the input. We will dig into this and see if we can find a long-term solution to reduce the impact.

[LordHepipud]

I created the linked PR #142 which addresses this issue a little by adding the experimental feature for caching the entire Framework code. In case possible, it would be great if you could test this and see if it mitigates the current issue you are having.