auditd is not restarted properly

[stejoo]

Describe the bug

The handler that is supposed to restart auditd does not work. The auditd service is not restarted.

We added auditd_log_group: splunk setting and ran the playbook to implement the configuration change. The role ran without problems. Including the run of the handler at the end (restart auditd) which returns a changed status, as expected. However, the configuration change did not take effect. The log file was still owned by root:root. Looking at the service status showed the auditd daemon was not restarted.

Playbook

- hosts: all
  become: true
  vars:
    auditd_manage_rules: false
    auditd_log_format: ENRICHED
    auditd_space_left: 100
    auditd_space_left_action: syslog
    auditd_admin_space_left: 50
    auditd_admin_space_left_action: rotate
    auditd_disk_full_action: suspend
    auditd_disk_error_action: suspend
    auditd_log_group: splunk

  roles:
    - role: robertdebock.auditd
      tags: ['audit', 'auditd']

Output

Relevant output of the playbook run:

...
 _______________________________________________ 
< TASK [robertdebock.auditd : configure auditd] >
 ----------------------------------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Tuesday 10 January 2023  08:09:13 +0100 (0:00:03.114)       0:00:14.201 ******* 
--- before: /etc/audit/auditd.conf
+++ after: /home/stefan/.ansible/tmp/ansible-local-7906_k3_yaj9/tmpw0q9f6us/auditd.conf.j2
@@ -5,7 +5,7 @@
 local_events = yes
 write_logs = yes
 log_file = /var/log/audit/audit.log
-log_group = root
+log_group = splunk
 log_format = ENRICHED
 flush = incremental_async
 freq = 50

changed: [myhost] => changed=true 

...inbetween tasks ran OK...
 ________________________________________________________ 
< RUNNING HANDLER [robertdebock.auditd : restart auditd] >
 -------------------------------------------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Tuesday 10 January 2023  08:09:16 +0100 (0:00:00.781)       0:00:17.309 ******* 
changed: [myhost] => changed=true 
  name: auditd
  state: started

However, the file on the system was still owned by the (default) root user:

# ls -lh /var/log/audit/ 
total 12M
-rw-------.  1 root root 3,3M Jan 10 08:08 audit.log
-r--------.  1 root root 8,1M Jan  6 21:38 audit.log.1

Further inquiry showed the auditd daemon was not restarted on the system. Performing the restart manually, using the service command instead of systemctl (as specified by https://access.redhat.com/solutions/2664811), resulted in the auditd daemon being properly restarted and the expected change to the log file's group ownership made:

# service auditd restart
Stopping logging:
Redirecting start to /bin/systemctl start auditd.service
# ls -lh /var/log/audit/
total 12M
-rw-r-----.  1 root splunk 3,4M Jan 10 08:11 audit.log
-r--r-----.  1 root root   8,1M Jan  6 21:38 audit.log.1

Finding (and TL;DR): The role does not restart the auditd daemon properly.

Debugging the issue

The role's restart auditd handler (at tag 3.2.4) looks like this:

- name: restart auditd
  ansible.builtin.service:
    name: "{{ auditd_service }}"
    state: restarted
    use: service  # systemctl can't restart auditd, service can: https://access.redhat.com/solutions/2664811
  when:
    - not ansible_check_mode | bool

The use argument gives one the impression Ansible might use the service command here. It is apparent that is not the case. Reading the documentation one sees this option in fact influences the module choice Ansible makes, because ansible.builtin.service is a abstraction module (like package). Instead of choosing a module based on Ansible fact ansible_service_mgr the use: option forces Ansible to choose a specific module. Setting it to service makes Ansible call the, and I quote, "'old" service module'.

Whatever goes on behind the scenes: the end result does not have the desired effect: auditd is not restarted. My theory at this point was: Ansible is still calling systemctl restart auditd; which is incorrect for this service. The auditd daemon demands to be restarted with service auditd restart by design (more info at: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-starting_the_audit_service). To test my theory of systemctl is being called I decided to debug the module on the node side of the affair. I commanded Ansible to keep the (temporary) generated Python code on the node with:

ANSIBLE_KEEP_REMOTE_FILES=1 ansible myhost -b -m ansible.builtin.service -a 'name=auditd state=restarted use=service'

After "exploding" the resulting AnsiballZ file on the node I had a look at the module code. To test my theory I added some debug code to the resulting debug_dir/ansible/modules/service.py module to have it display the commands it executed. This resulted in:

[root@myhost ansible-tmp-1673336562.8091812-11481-93647040079523]# /usr/libexec/platform-python AnsiballZ_service.py execute 
Platform: Linux
Distribution: None
Params: {'name': 'auditd', 'state': 'restarted', 'runlevel': 'default', 'arguments': '', 'sleep': None, 'pattern': None, 'enabled': None}
Service: <__main__.LinuxService object at 0x7f6de09dc0f0>
Service command (svc_cmd): /bin/systemctl
Perform execute_command (cmd): /bin/systemctl show 'auditd'
Perform execute_command (cmd): /bin/systemctl stop auditd 
Perform execute_command (cmd): /bin/systemctl start auditd 

{"name": "auditd", "state": "started", "changed": true, "invocation": {"module_args": {"name": "auditd", "state": "restarted", "runlevel": "default", "arguments": "", "sleep": null, "pattern": null, "enabled": null}}}

The use: option in handler task does influence the module chosen by Ansible as expected. In this case it tells Ansible to use the ansible.legacy.service module instead of ansible.builtin.systemd. Looking at the code of this module: the ansible.legacy.service performs it's own discovery of the init system on the node. If it finds systemctl on the host it decides to use that. As a last resort, if it doesn't find any supported init system it falls back to using the service command. In my/our case it ends up calling systemctl, which results in auditd not restarting... 😞

Ansible bug report 22171 describes this issue/behavior in some more depth. A comment there recommends calling service directly with the command module:

- command: /sbin/service auditd restart

Perhaps the role should do something like that. The exact path might be an issue, maybe call it with a relative path and rely on it being in $PATH? Or figure out if the supported distributions by this role share a common absolute path to service.

I am somewhat surprised to have hit this bug. Does auditd restart properly for others using this role? Or did the use: service have the desired effect on other systems or other (older?) Ansible versions? Or was this somehow not noticed? 😇

Environment

• Control node OS: Fedora 36
• Control node Ansible version: 2.13.6
• Managed node OS: RHEL 8.7

[jwarnier]

I commented on the PR you submitted.
The issue is on your target (or the ansible/module version you're using not supporting it correctly).

[stejoo]

Sorry for leaving this be. My fork has been working great for us internally. I welcome any suggestions what to try next/instead.

Auditd does not want to be restarted by systemctl. It reports back as such:

# systemctl restart auditd.service 
Failed to restart auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).
See system logs and 'systemctl status auditd.service' for details.

It's documentation is also clear on why this is and how you should restart it using # /sbin/service auditd restart.
Current ansible-lockdown code for RHEL9-CIS (as do 8 and 7) provides the same solution: https://github.com/ansible-lockdown/RHEL9-CIS/blob/4fb533bcbe8a253d3e8dd13117641f1017e4de56/handlers/main.yml#L104

I believe I have described that issue in some detail as well. I have reviewed the Python code of the service module and have shown what command the module generates and that those will not work for this purpose: it will never restart auditd properly (on RHEL8 and probably other systems as well).

I am open to your ideas.

[oferchen]

Sorry for leaving this be. My fork has been working great for us internally. I welcome any suggestions what to try next/instead.

Auditd does not want to be restarted by systemctl. It reports back as such:

# systemctl restart auditd.service 

Failed to restart auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).

See system logs and 'systemctl status auditd.service' for details.

It's documentation is also clear on why this is and how you should restart it using # /sbin/service auditd restart.

Current ansible-lockdown code for RHEL9-CIS (as do 8 and 7) provides the same solution: https://github.com/ansible-lockdown/RHEL9-CIS/blob/4fb533bcbe8a253d3e8dd13117641f1017e4de56/handlers/main.yml#L104

I believe I have described that issue in some detail as well. I have reviewed the Python code of the service module and have shown what command the module generates and that those will not work for this purpose: it will never restart auditd properly (on RHEL8 and probably other systems as well).

I am open to your ideas.

The commands used for start,stop,reload can be mapped at the unit service file using ExecStart, ExecStop, ExecReload, etc..

https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html