Remove the separate "mirror_opendev" group and rename it to just
"mirror". Update various parts to reflect that change.
We no longer deploy any mirror hosts with puppet, remove the various
configuration files.
Depends-On: https://review.opendev.org/728345
Change-Id: Ia982fe9cb4357447989664f033df976b528aaf84
Ceph Nautilus is released and the official mirror
is available. This adds the Ceph Nautilus mirror
so we can sync it for Stretch and Bionic.
Based on the same change that was done when Mimic
was released [1]
[1] https://review.opendev.org/#/c/571989/
Change-Id: I9424d1f4df58acde8ea70dc16283d4de89189bae
There are many references to review.openstack.org, and while the
redirect should work, we can also go ahead and fix them.
Change-Id: I28f398796a6392a3dffea1d25cfe2ae3a36a3589
Remove the pypi symlink from the mirrors in preparation for volume
removal
Change-Id: I0f1265857e204616e5e342afe764ff3a0215c4fe
Needed-By: https://review.openstack.org/618326
As a follow-on to Ic33bca431ae19b44e5bc2938de425cc68785b331
... (reprepro configuration really is quite inscrutable outside "try
it and see if it works")
The update is just called "debian-security", no codename.
We want to use the debian-security key to validate the gpg signatures,
update that.
Use the "updates" suite, which seems to be where debian-security is
Add the symlink for mirror
Change-Id: Ice2a06f4274cdad766f427df70d9fe3e92f62e8c
In I30bd6b658d87abc765b5caee5003e387981850b4 I missed that this one
symlink should be removed by puppet as it is on all mirror hosts. We
can then remove this after puppet has run.
Change-Id: I37f52b0ed012ad680eb86f8905764a1c78212148
The mirror was put in place for OpenStack Ansible however the project
is not consuming it anymore. In the parent change of this commit, the
addition of a reverse proxy cache will be used instead in the project
and it will allow usage across all distributions.
Change-Id: I30bd6b658d87abc765b5caee5003e387981850b4
We expect at least 100GB of disk available for the apache mirror caches
we run in each region. However, due to htcacheclean lagging apache's
ability to cache we cleaned up to 50GB to give us breathing room. Since
then we have reorganized the cache layout so that htcacheclean can
operate more quickly and don't need as much breathing room.
Increase this cache size value up to 70GB which still gives us a
conservative ~30GB or so of breathing room. We can continue to increase
this number if htcacheclean keeps up.
Change-Id: I8a13a5876cea930c5d981af2c7cd6ab36ba867fb
Ceph Mimic is now release and the official mirror
is available. This adds the Ceph Mimic mirror so we
sync the version for Stretch, Xenial and Bionic.
Change-Id: Id358d657b6d1daed3bc72447fa94aef198928e50
We are in the process of remove tripleo-test-cloud-rh1 and no longer
running jobs there. So we can start to increase our proxy cache again.
This reverts commit 5fed4c4c4c.
Change-Id: I94ec544c9a8ef3e4eed77635e736031207401be0
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The npm mirror was removed with
Id539d336814cce2ce18898526e561b8b6977f62f. This change is "inspired"
by a proposed puppet-nodejs update in
Ia7966fb9578d0d79f3a7f9480e3a956555737dc8. Rather than fixing it up
for the new version, remove it (also, puppet is failing trying to
access /afs/.openstack.org/mirror/npm).
I believe the npmrc.erb file is actually a vestige of prior release
methods and is also no longer required.
Change-Id: I6fa48e4700779d2c90194f0129c770bf2d6d865f
This reverts commit 1eb7cc27c3.
Switch back to the RO mirror as we believe it to be in a good state,
and it has been released.
Change-Id: Id850ab5c06dc237395bf3d5e008c5c23c3d470c1
We only have 100GB disk for tripleo-test-cloud-rh1, between AFS and
apache cache, we've filled the disk. Drop down to 40GB until
tripleo-test-cloud-rh1 is removed, then we can revert.
Change-Id: I6e3d1b6f3202b375853919d1920bd945fe035855
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This is a new Xenial-based mirror in the RAX IAD region
Add munging to have a serveralias for numerically named servers
Change-Id: I5690e231a53a45e33bd925cd1a4a87ffa025af04
This folder still exists on some of our mirrors, force delete it since
puppet complains.
Aug 12 15:42:47 mirror puppet-user[14127]: (/Stage[main]/Openstack_project::Mirror/File[/opt/apache_cache]) Not removing directory; use 'force' to override
Change-Id: I02240c3bcd315e206f22351cbe40ea4fd34fb3c0
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Right now rax-ord is are busiest mirror, however it appears to be on a
slow cinder volue. Right now, our proxy cache is full, because we don't
have enough buffer between min / max.
Change-Id: I07117808e47fec5b7dd9ab385ace9dc1f3abe261
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Users reported timeouts in citycloud lon1 jobs, upon logging into
mirror.lon1.citycloud.openstack.org I found it with a load average of
> 100 and htcacheclean going bananas
ianw@mirror:~$ ps -aef | grep htcacheclean | wc -l
172
I/O on this node seems extremely slow. My guess is that the
htcacheclean's started piling up creating a big negative feedback loop.
Run this under a lock so at least we only have one running at a time.
Change-Id: I584ff99bfbb08a4c0d8ae7c8531b7c3357ebcd43
We want to start caching Kolla registry tarballs, but these
presently range upward of 4GiB each and need around 50GiB of
aggregate cache space. As a result, we'll increase the max cached
file size from 100MiB to 5GiB and the max cache content from 2GiB to
80GiB to support this.
Change-Id: Ic6e4ae6b44f134bce8a8873f6e4d4517f9d30e14
New volumes have been mounted at /var/cache/apache2 on our
per-region mirror instances and proxy sibdirectories warmed with
content from the old /opt/apache_cache directories. Switch to using
the newer locations and remove the old ones to free up space on root
filesystems.
Change-Id: I8834ee2098ed4d2b2bfb7d08784db5ba9f034efa
Add the bits needed to mirror apt.dockerproject.org into
openstack-infra.
Change-Id: Id8b34407fcc5b1705f3db17a5fc26a4a79034baa
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Sadly, RDO changes too much for us to mirror. However, we have setup
a reverse proxy cache, so jobs are now using it.
Once landed, I'll remove the volume from AFS.
Change-Id: I249ba3719096aa24fab00fe6c660409a78168f23
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Now that we have reverse proxy caching on our mirror nodes set up
htcacheclean to run their hourly and reduce the size of the cache to
2GB. Based on using the cache lightly for a few days the max size of the
cache has been 648MB so 2GB should be plenty of headroom while still
being well under available disk on our mirror nodes.
Change-Id: I3fe41a3de669d08966357e10d4c121caa9bab6d4
Recently https://trunk.rdoproject.org enabled rsync, which allows us
to now mirror them. Start with the centos7 repo before dealing with
stable branches.
Change-Id: I942ad30b1026554b1a1c8309b1725d76e7870c7f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
As not to affect rewrite rules for our AFS mirrors, move caching to
port 8080. Also means we can more easily identify traffic.
Add missing setting for SSL support and http proxy. Note, mod ssl is
already enabled by puppet-httpd.
Change-Id: I4b3a13ff3ec2e1045ade18fc21741cba9e28f117
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We can't sanely mirror the rdo packages beacuse they are constantly
updating. Instead put a caching reverse proxy in front of the repos in
each region on our existing mirror servers. This should allow us to get
region local data that is "hot" and hopefully improve reliability to the
rdo repos.
Change-Id: I6ab810f7b6788459da1cee00baf6b053f31843cf
This has been on my list for some time, since we run fedora jobs in
the gate, we should also mirror this infra too.
Change-Id: I523bf263b5f9455ee51a712fc97cde3f8daeba80
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This patch adds a static, read-only gem mirror to openstack-infra's regional
mirrors under the /gem path.
Change-Id: I2f67fe01d32c4472ff56862b9dc25b9915a695c0
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Since this is currently unmanaged by openstack-infra, the default is
to rotate logs every week, 52 times. Needless to say, we are running
out of HDD space. Now we rotate every day, 7 times. This will cut
down on the amount of HDD space apache2 logs require.
Change-Id: I7baf62a36432f5d322e7f63e8c89bd6205974ca8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This patch adds a mirror for MariaDB packages. The releases
mirrored is selective, only covering the currently supported
Ubuntu and MariaDB versions for OpenStack-Ansible.
Having this mirror available aids deployment projects which
consume these packages by ensuring that all apt packages needed
are available in mirror local to the nodepool cloud provider.
Change-Id: Ic549bbbd4f5dafc6555174f3ef3bd713b9a727d4
As more teams will be switching to jewel from hammer, we also need to
provide a debian mirror for it.
Change-Id: I0f5097672c0afd092380d841c665c3fdfa25f280
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
So project using debian-jessie DIB can benefit from our AFS infra.
Change-Id: I30206d6802120f5b89f58f1c1dc3548515fcef76
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This patch adds a mirror for Ubuntu Cloud Archive. The releases
mirrored is selective, only covering the currently supported
Ubuntu and OpenStack versions.
Having this mirror available aids deployment projects which
consume these packages by ensuring that all apt packages needed
are available in mirror local to the nodepool cloud provider.
Note:
I have no real understanding of how puppet/system-config works
and don't know how to test this patch. If there's no
system-config gate check to validate that this works, please can
someone assist by testing and providing feedback.
Change-Id: Id658f2f29ce4a42eee2ef1ecdf935ada882da51e
Ian Wienand