openstack
/
puppet-tripleo
Code Issues Proposed changes

Fix postrotate to notify holders of rotated logs 

Lsof +L1 locates unlinked and open files and does not work for
logrotate, neither with copyteuncate not w/o that option.

Instead, find *.X (X - number) files held and notify the processes
owning those to make an apropriate actions and reopen new log files to
stop writing to the rotated files.

The actions to be taken by such processes are:

* For httpd processes, use USR1 to gracefully reload
* For neutron-server, restart the container as it cannot process
  HUP signal well (LP bug #1276694, LP bug #1780139).
* For nova-compute, restart the container as it cannot process
  HUP signal well (LP bug #1276694, LP bug #1715374).
* For other processes, use HUP to reload

This also fixes the filter to match logfiles ending with *err,
like rabbitmq startup errors log.

Closes-Bug: #1780139
Closes-Bug: #1785659
Closes-Bug: #1715374

Change-Id: I5110426aa26e5fce7ebb4d80d8a2082cbf80519c
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
This commit is contained in:
Bogdan Dobrelya 2018-08-24 12:15:51 +02:00 committed by Emilien Macchi
parent 7edd77522a
commit d37c74d638
1 changed files with 30 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
templates/logrotate/containers_logrotate.conf.erb
View File

@ -1,4 +1,4 @@
/var/log/containers/*/*log /var/log/containers/*/*/*log {
/var/log/containers/*/*log /var/log/containers/*/*/*log /var/log/containers/*/*err {
<%= @rotation %>
rotate <%= @rotate %>
maxage <%= @purge_after_days %>
@ -22,12 +22,35 @@
\( -mtime +<%= @purge_after_days %> -or \
-atime +<%= @purge_after_days %> -or \
-ctime +<%= @purge_after_days %> \) -exec rm -f {} \;;
/sbin/lsof -nPs +L1 +D /var/log/containers 2>/dev/null|\
grep -v /var/log/httpd/ |\
awk '/\S+\s+[0-9]+\s.*\/var\/log\/.*\(deleted\)/ {print $2}' |\
# Send HUP to gracefully restart processes owning rotated files
# Httpd, nova-compute and neutron-server processes need to take another approach
/sbin/lsof -nPs +D /var/log/containers 2>/dev/null |\
grep -v -e /var/log/containers/httpd/ \
-e /var/log/containers/neutron/server \
-e /var/log/containers/nova/nova-compute |\
awk '/\S+\s+[0-9]+\s.*\/var\/log\/containers\/\S+\.[0-9]+\S+?$/ {print $2}' |\
sort -u | /bin/xargs -n1 -r -t kill -HUP;
/sbin/lsof -nPs +L1 +D /var/log/containers 2>/dev/null|\
awk '/\S+\s+[0-9]+\s.*\/var\/log\/httpd\/.*\(deleted\)/ {print $2}' |\
sort -u | /bin/xargs -n1 -r -t kill -USR1
# USR1 to gracefully restart httpd processes owning rotated files
/sbin/lsof -nPs +D /var/log/containers/httpd 2>/dev/null |\
awk '/\S+\s+[0-9]+\s.*\/var\/log\/containers\/httpd\/\S+\.[0-9]+\S+?$/ {print $2}' |\
sort -u | /bin/xargs -n1 -r -t kill -USR1;
# FIXME(bogdando): Unhealthy containers can't be restarted, until
# --exit-on-unhealthy supported. Therefore we cannot send TERM to self-heal.
# LP(#1780139): HUP does not work with neutron server API process.
# Restart the neutron API container instead, if it is
# holding rotated server.log.X file.
/sbin/lsof -nPs +D /var/log/containers/neutron 2>/dev/null |\
awk '/\S+\s+[0-9]+\s.*\/var\/log\/containers\/neutron\/server\S+\.[0-9]+\S+?$/ {print $1}' |\
grep -q neutron-s && docker restart neutron_api ||:;
# LP(#1276694): HUP does not work well with nova compute process.
# Restart the nova compute container instead, if it is
# holding rotated nova-compute.log.X file.
/sbin/lsof -nPs +D /var/log/containers/nova 2>/dev/null |\
awk '/\S+\s+[0-9]+\s.*\/var\/log\/containers\/nova\/nova-comp\S+\.[0-9]+\S+?$/ {print $1}' |\
grep -q nova-comp && docker restart nova_compute ||:
endscript
}