Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High CPU Load for Windows Servers with icinga2 2.12.1 #8507

Closed
MohamedRedaLyoubi opened this issue Nov 25, 2020 · 2 comments · Fixed by #8488
Closed

High CPU Load for Windows Servers with icinga2 2.12.1 #8507

MohamedRedaLyoubi opened this issue Nov 25, 2020 · 2 comments · Fixed by #8488
Assignees
@julianbrost
Labels
area/windows Windows agent and plugins bug Something isn't working ref/NC
Milestone
2.11.8

Comments

@MohamedRedaLyoubi
Copy link

MohamedRedaLyoubi commented Nov 25, 2020 •
edited
Loading

Describe the bug

Hello,

Most of our windows 2012/2016/2019 servers with only 1 vcpu are constantly overloaded in cpu because of icinga2 process.

The logs on the client & satellites look fine, The config from director is pushed successfully to the agent at this time. (we even checked in debug.log)

[2020-11-25 10:19:57 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-centos-check_log-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:19:57 +0100] warning/ApiListener: Removing API client for endpoint 'sat2.host.net'. 0 API clients left.
[2020-11-25 10:19:57 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-certificate-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:19:58 +0100] warning/ApiListener: Removing API client for endpoint 'sat1.host.net'. 0 API clients left.
[2020-11-25 10:20:03 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_apache_status-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:08 +0100] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2020-11-25 10:20:08 +0100] information/ApiListener: Reconnecting to endpoint 'sat1.host.net' via host 'x.x.x.1' and port '5665'
[2020-11-25 10:20:09 +0100] information/ApiListener: Reconnecting to endpoint 'sat2.host.net' via host 'x.x.x.2' and port '5665'
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_apache_tomcat-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_cassandra_backup-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_certificate_ssl_file-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_dir-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_disk-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_dns-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_elasticsearch_backup-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_http-remote-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_http-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_iostat-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_load-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_log-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_mariadb-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_mariadb_backup-service-template.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_mem-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_memcached-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_mongodb-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_mongodb_backup-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_mysql-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_mysql_backup-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_nginx-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_ntp-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_oracle_backup-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_partition_readonly-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_ping_vpn-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_postgres-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_postgres_backup-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_procs-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_snmp-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:14 +0100] information/ApiListener: New client connection for identity 'sat2.host.net' to [x.x.x.2]:5665
[2020-11-25 10:20:14 +0100] information/ApiListener: New client connection for identity 'sat1.host.net' to [x.x.x.1]:5665
[2020-11-25 10:20:19 +0100] information/ApiListener: Stage: Updating received configuration file 'C:\ProgramData\icinga2\var\lib\icinga2/api/zones-stage/global-templates//_etc/linux-check_snmp-v3-service-apply.conf' for zone 'global-templates'.
[2020-11-25 10:20:19 +0100] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'sat1.host.net'.
[2020-11-25 10:20:19 +0100] information/ApiListener: Finished reconnecting to endpoint 'sat2.host.net' via host 'x.x.x.2' and port '5665'
[2020-11-25 10:20:19 +0100] information/ApiListener: Sending config updates for endpoint 'sat1.host.net' in zone 'SAT-PAR1'.
[2020-11-25 10:20:19 +0100] information/ApiListener: Finished sending config file updates for endpoint 'sat1.host.net' in zone 'SAT-PAR1'.
[2020-11-25 10:20:19 +0100] information/ApiListener: Syncing runtime objects to endpoint 'sat1.host.net'.

Expected behavior

Normal CPU load consumed by icinga2

Screenshots

high_load
icinga2_process

Your Environment

  • Icinga version 2.12.1-1
  • OS at the Windows Agents: W2012R2, W2016, W2019
  • PowerShell 5.1 or later
  • .NET Framework 4.7 or later

Additional context

To resolve temporarily

  • If we add another vcpu to the VM we no longer have the problem
  • If we set manually the Icinga 2 process priority to low it also fixes the problem
  • If we set the Icinga 2 process priority to low by a registry key, the cpu load remains at 100% but if we set back the priority to normal and then to low manually it fixes the issue
@Al2Klimov Al2Klimov self-assigned this Nov 25, 2020
Al2Klimov added a commit that referenced this issue Nov 25, 2020
@Al2Klimov
Windows agents: run with low priority
bf13755 
... not to let the monitoring block the actual services.

refs #8507
@Al2Klimov Al2Klimov added area/windows Windows agent and plugins bug Something isn't working labels Nov 25, 2020
@Al2Klimov Al2Klimov mentioned this issue Nov 25, 2020
Closed
@lippserd
Copy link
Member

ref/NC/696161

@Al2Klimov Al2Klimov mentioned this issue Nov 26, 2020
Merged
This was referenced Nov 27, 2020
Merged
Merged
@icinga-probot icinga-probot bot added this to the 2.11.8 milestone Nov 27, 2020
@icinga-probot icinga-probot bot closed this as completed Nov 27, 2020
@icinga-probot
Copy link
Contributor

icinga-probot bot commented Nov 27, 2020

Closed this issue because #8510 has been merged.

@Al2Klimov Al2Klimov assigned julianbrost and unassigned Al2Klimov Nov 27, 2020
@julianbrost julianbrost mentioned this issue Aug 29, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@julianbrost julianbrost

Labels
area/windows Windows agent and plugins bug Something isn't working ref/NC
Projects
None yet
Milestone
2.11.8
Development

Successfully merging a pull request may close this issue.

Improve config sync locking Icinga/icinga2
Windows agents: run with low priority Icinga/icinga2
4 participants
@julianbrost @lippserd @Al2Klimov @MohamedRedaLyoubi