3cd751de2f
So currently the logrotate_crond container has a few issues issues:
A) In the postrotate it matches pids multiple times and sends SIGHUPs multiple time to processes:
======== /var/log/messages =====
Jun 3 09:01:15 overcloud-controller-0 logrotate-crond: kill -HUP 1575
Jun 3 09:01:15 overcloud-controller-0 rsyslogd: [origin software="rsyslogd" swVersion="8.24.0" x-pid="1575" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jun 3 09:01:15 overcloud-controller-0 logrotate-crond: kill -HUP 1575
Jun 3 09:01:15 overcloud-controller-0 rsyslogd: [origin software="rsyslogd" swVersion="8.24.0" x-pid="1575" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jun 3 09:01:15 overcloud-controller-0 logrotate-crond: kill -HUP 1575
Jun 3 09:01:15 overcloud-controller-0 rsyslogd: [origin software="rsyslogd
...
Adding sort -u in the pipeline of the postrotate script takes care of
that.
B) The logrotate_crond container should not rotate logs for services
running on the host outside of containers (i.e. rsyslog has its own
/etc/logrotate.d/rsyslog rules). Doing so violates the principle of
least surprise.
Using 'lsof ..+D /var/log/containers' takes care of this as we won't
match any non containerized processes
C) The find command matches older files to be deleted but the SIGHUP is
never sent so we actually can end up in a situation where we remove a
file but the new one never gets created because the service does not get
a SIGHUP signal:
ls -la /var/log/containers/httpd/*/*
-rw-r--r--. 1 root root 52046652 May 29 14:10 /var/log/containers/httpd/aodh-api/aodh_wsgi_access.log.1
-rw-r--r--. 1 root root 0 May 24 19:14 /var/log/containers/httpd/aodh-api/aodh_wsgi_error.log
-rw-r--r--. 1 root root 5894 May 24 19:14 /var/log/containers/httpd/aodh-api/error_log
-rw-r--r--. 1 root root 50755274 May 29 14:10 /var/log/containers/httpd/cinder-api/cinder_wsgi_access.log.1
-rw-r--r--. 1 root root 4138 May 25 11:58 /var/log/containers/httpd/cinder-api/cinder_wsgi_error.log
-rw-r--r--. 1 root root 5894 May 24 19:13 /var/log/containers/httpd/cinder-api/error_log
Using 'lsof ..+D /var/log/containers' fixes this case as well because
now we correctly match the processes that have a deleted file that is
open and we send a proper SIGHUP to them.
Tested by doing the following:
1) Logging rotation of containerized services (B, C)
1.1) Stopped the keystone container
1.2) Made the /var/log/container/keystone/keystone.log file 21M large
1.3) Started the keystone container and observed that it was logging
correctly to /var/log/container/keystone/keystone.log
1.4) Inside the logrotate_crond container we ran the following:
/usr/sbin/logrotate -s /var/lib/logrotate/logrotate-crond.status /etc/logrotate-crond.conf
1.5) We observed correct log rotation and keystone was notified via
SIGHUP and started logging correctly:
-rw-r--r--. 1 42425 42425 21628706 Jun 13 08:43 keystone.log.1
-rw-r--r--. 1 42425 42425 999 Jun 13 08:43 keystone.log
2) No SIGHUP to host processes (A)
2.1) stopped rsyslog on the host and made one of its log files > 10M:
-rw-r--r--. 1 root root 28M Jun 13 08:59 /var/log/messages
2.2) restart rsyslog
2.3) Ran the logrotation inside the container
/usr/sbin/logrotate -s /var/lib/logrotate/logrotate-crond.status /etc/logrotate-crond.conf
2.4) Observed that no SIGHUP was sent to rsyslog on the host
Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com>
NB: Cherry-pick to pike omitted the part around finding files older
than X days as that feature did not exist
Change-Id: I5029a4b9c76268455812696290aaf82f1a0c2c23
Closes-Bug: #1776533
(cherry picked from commit
|
||
|---|---|---|
| .. | ||
| docker_distribution | ||
| logrotate | ||
| redis | ||
| securetty | ||
| selinux | ||
| stunnel | ||
| ui | ||