RPM package sets deprecated logging.dest cli flag

[rudolf]

Description:
The RPM package service description sets the deprecated --logging.dest flag which causes the following problems:

1. Kibana generates a deprecation warning which users are unable to resolve
   @mshustov suggested setting deprecation.skip_deprecated_settings='logging.dest' to suppress the warning
2. RPM package keeps writing to rotated log files when using the new logging configuration
   When using the RPM package and the new logging configuration, Kibana will keep writing legacy request logs to log files that have already been rotated (exceeded size or interval duration).

This is because legacy logging is enabled in the service description by setting --logging.dest=
https://github.com/elastic/kibana/blob/7.15/src/dev/build/tasks/os_packages/service_templates/systemd/etc/systemd/system/kibana.service#L18

Versions affected: 7.15, 7.14, 7.13 and probably all of 7.x

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[rudolf]

To fix this in 7.x we will probably have to set logging.dest=/var/log/kibana/kibana.log in kibana.yml instead of doing this in the service file. That way the legacy logging system is still the default when installing the RPM, but it's a bit more obvious to users that Kibana is using the legacy logging configuration and that they should remove these legacy configuration options.

[jbudz]

We merged a change for this recently - #98213

[rudolf]

Closed in #98213

Any reason we're not backporting that to 7.15.1?

[rudolf]

#98213 has been reverted in #111966

[rudolf]

This would affect users in 7.16 because they will get a deprecation warning for using logging.dest but not be able to resolve it without changing the service definition manually.

[LeeDr]

FYI, on my CentOS 7 install it's here;

[root@localhost vagrant]# grep -n logging.dest /etc/init.d/kibana 
38:args="--logging.dest=/var/log/kibana/kibana.log"

[rashmivkulkarni]

++, FYI: after having changed the logging.dest=/var/log/kibana/kibana.log in my CentOS7_rpm, and upon Kibana restart, it fails to come up.

[akash1810]

I've come up against a problem that I believe is caused by this too.

When running Kibana (v7.16.2) via systemd logs are written in json format to /var/log/kibana/kibana.log via the service file.

I've recently had an incident where Kibana became unavailable due to disk exhaustion - /var/log/kibana/kibana.log was consuming a quarter of all disk space. I'd like to configure log rotation as described here, my kibana.yml looks like this:

logging:
  appenders:
    rolling-file:
      type: rolling-file
      fileName: /var/log/kibana.log
      layout:
        type: pattern
      policy:
        type: size-limit
        size: 100mb
      strategy:
        type: numeric
        pattern: '-%i'
        max: 2
  root:
    appenders:
      - default
      - rolling-file

However, this is not having the desired effect. Whilst /var/log/kibana.log is rotating as desired, /var/log/kibana/kibana.log continues to be written to too, and w/out rotation. In fact I'm now in a worse situation as more disk space is being used with two log files!

Attempts to disable the root logger (logging.root.level: off) has no effect either.

How can one stop Kibana writing logs to /var/log/kibana/kibana.log when running Kibana via systemd? Alternatively, how can one rotate /var/log/kibana/kibana.log using the logging configuration syntax introduced in v7.13?

[jbudz]

Hey @akash1810 sorry you ran into this. I'd recommend removing logging.dest from ExecStart in /usr/lib/systemd/system/kibana.service followed by systemctl daemon-reload.

The logging.dest setting in the service file wasn't implemented as well as it could have been, but we ended up stuck with it for the 7.x release line due to the fix being a breaking change. In 8.0+ we've removed logging settings from the systemd service file and use only kibana.yml.

[jbudz]

Related to the deprecated logging.dest flag in use - this still exists in 7.17 due to the new logger outputting logs in a different format. We added more documentation to 7.17 to address this and avoid a breaking change.

#120306
#120273

Closing this out.

[akash1810]

@jbudz thanks for the reply, it's come at a great time too as we have been upgrading our cluster this week! Great to hear the issue is resolved in the v8 release.