opendev
/
system-config
Code Issues Proposed changes

Run htcacheclean under lock 

Users reported timeouts in citycloud lon1 jobs, upon logging into
mirror.lon1.citycloud.openstack.org I found it with a load average of
> 100 and htcacheclean going bananas

ianw@mirror:~$ ps -aef | grep htcacheclean | wc -l
172

I/O on this node seems extremely slow.  My guess is that the
htcacheclean's started piling up creating a big negative feedback loop.

Run this under a lock so at least we only have one running at a time.

Change-Id: I584ff99bfbb08a4c0d8ae7c8531b7c3357ebcd43
This commit is contained in:
Ian Wienand 2017-08-10 21:29:09 +10:00
parent f332f9d811
commit 1a63647da6
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
modules/openstack_project/manifests/mirror.pp
View File

@ -296,7 +296,7 @@ class openstack_project::mirror (
# Clean apache cache once an hour, keep size down to 80GiB.
minute => '0',
hour => '*',
command => 'htcacheclean -n -p /var/cache/apache2/proxy -t -l 81920M > /dev/null',
command => 'flock -n /var/run/htcacheclean.lock htcacheclean -n -p /var/cache/apache2/proxy -t -l 81920M > /dev/null',
environment => 'PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin',
require => [
File['/var/cache/apache2/proxy'],