Commit Graph

188 Commits

Author SHA1 Message Date
Ian Wienand 67cbecdfb4 Restore blank nodepool/scripts
I was a little too hasty in I76acbd08acda20c88ff9fd9148e3918b78d6c6c6
removing the scripts/ directory.  It has broken the puppeting of old
hosts.

Restore the directory with a blank file explaining the situation.

Also, we don't need to copy this in the nodepool elements; remove that.

Change-Id: I8b82950237ef69c4941866900cac9bda42f58ca2
2020-03-03 15:01:23 +11:00
Ian Wienand e674d5b4b1 Remove nodepool/scripts
None of these are relevant any more, remove them.

Change-Id: I76acbd08acda20c88ff9fd9148e3918b78d6c6c6
2020-03-02 16:23:02 +11:00
Andreas Jaeger 91400af8ba Good bye jenkins/scripts
We now only have afs-slug.sh in jenkins/scripts/ and it is only
used by nodepool/scripts/configure_mirror.sh. Move the file into the
nodepool/scripts directory and adjust paths.

Remove copying of jenkins/scripts from nodepool, the directory does not
exist anymore.

Remove 89-jenkins-scripts, we don't need it anymore.

Depends-On: https://review.openstack.org/548393
Change-Id: I13cac791d70a2e5bcb0de1ba0adcd79e1b0aee9f
2018-03-05 14:28:42 +01:00
Paul Belanger 70eb9ea584
Remove unused nodepool ready-scripts
We no longer have the ability to run ready-scripts in nodepool, so
let's delete / clean them up.

Change-Id: I2551037dcfba2362d23401536295eb8f8f5d3e0d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-02-27 06:59:54 -05:00
Paul Belanger c39c5b8744 NODEPOOL_DOCKER_REGISTRY_PROXY needs trailing slash for docker
We likely want to change all our variables to have trailing slashes,
but need to confirm other clients will be fine first.  I've already
tested docker, so make the change.

Change-Id: I2c3fcb0b99b45b04068ef9be6a2887adf1f0ebab
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-15 17:22:36 -04:00
Paul Belanger 8f04a70bda Update NODEPOOL_RUGYGEMS_PROXY / add bundle client support
Both gem install and bundle install will use the
NODEPOOL_RUGYGEMS_PROXY variable. However, each client has a different
configuration file.

Change-Id: I329d871d6dff5b8df0b3066363c52b56f1f6fc7f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-14 17:15:23 -04:00
Paul Belanger 751e55909b Add support for NODEPOOL_RUGYGEMS_PROXY
Now that we have a working reverse proxy cache for api.rubygems.org,
add the .gemrc file and start using it.

Change-Id: I6d8cd1d1353fc28469aceb16d487cd9b39eda73b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-11 10:39:27 -04:00
Jenkins d04602839b Merge "Add mirror host for images.linuxcontainers.org reverse proxy" 2017-08-11 12:51:39 +00:00
Paul Belanger 32e85d95a2 Revert "Remove NPM mirror settings"
This is a partial revert and update to our NPM mirror settings. We now
have a reverse proxy change enabled on our mirrors. Update .npmrc to
use them now.

This reverts commit 0b00bd6abd.

Change-Id: Ic73c72627811d2f92ef08478edbd2282d23feb4f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-10 15:59:29 -04:00
Jesse Pretorius a5601b2fa8 Add mirror host for images.linuxcontainers.org reverse proxy
In I8336aeb85689cfaac8a794a5c3f5c62273d8ded0 a reverse proxy
for images.linuxcontainers.org was added to the infra services.

This adds a record in the mirror host env vars so that it's
easy to know which host name to use when executing lxc-create.

eg: lxc-create --template download --name test1 -- --server ${NODEPOOL_LXC_IMAGE_PROXY} --dist ubuntu --release xenial --arch amd64

Change-Id: I457fb98c0aa4e4b4a77acbc77a927a94643b5d65
2017-08-09 16:22:38 +00:00
Paul Belanger 2d743fb855
Add buildlogs.centos to nodepool variables
All jobs to reference buildlogs.centos from /etc/ci/mirror_info.sh.

Change-Id: Ic0e8008261f47ac9496c2a3d8696c81475556bd3
Depends-On: I9d9d51971e52e8dc594d9d46bafc60147ad3c96b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-01 13:09:10 -04:00
Paul Belanger 17fc37d06f
Add NODEPOOL_TARBALLS_PROXY to configure_mirror.sh
We now proxy tarballs.o.o via apache reverse proxy, add a variable to
make it easier for jobs to use it.

Change-Id: I6d8084c0d0443d9e499215acd06d24484d22f91a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-12 11:13:07 -04:00
Clark Boylan 5213751af6 Set PATH when running configure mirror script
The configure mirror script is trying to run utilities like ip and
restorecon which at least on some distros are hidden away in /usr/sbin
and not where nodepool would find them otherwise. Explicitly add /sbin,
/usr/sbin, /bin, /usr/bin, and /usr/local/bin to the end of PATH to list
all of the various places we might find things. With this in place we
should be able to find ip and restorecon whereever they are.

Change-Id: I800cd69e47b4b099b453e47f1b579ab015ed628f
2017-06-21 15:11:29 -07:00
Clark Boylan 17d470f60c Restore selinux context on unbound forwarding.conf
When we modify the forwarding.conf in our ready script we have to
restore the selinux contexts if on centos and fedora. If we don't the
unbound services fails to read the file (despite normal file system
permissions being fine). Address this by restorecon'ing the file if
restorecon is present (implying this is an selinux enabled host).

Change-Id: Ib8a64b0ba9ee4144a6123d7b8f7c1c8fc9501112
2017-06-21 11:41:47 -07:00
Paul Belanger 6fb7a47873
Use sudo for unbound forwarding.conf in configure_mirror.sh
We need sudo permissions to write the forwarding.conf file.

Change-Id: I788915fccc5ac92b57b6a0815b90c40e32948bed
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-06-21 05:31:17 -04:00
Clark Boylan 023f9f19ac Use ipv6 resolvers only with ipv6 networking
If we have ipv6 networking that is routable then only use ipv6 name
resolvers to avoid traversing potential NAT with ipv4. If we don't have
ipv6 available and routable then only use ipv4 resolvers to avoid
timeouts and failovers.

Change-Id: I885418e07265002b09592c3cf3aff9712c6e001f
2017-06-20 13:05:18 -07:00
Jenkins 2f3963546e Merge "Remove 99-cache-testrepository-db from cache-devstack element" 2017-05-26 07:48:21 +00:00
Dirk Mueller 07e15b14a4 explicitly configure the openSUSE mirror paths
Rather than a sophisticated sed explicitely set the repository
mirros similarly like for all other distributions.

Change-Id: I846cd43b1f9c5949b656738959b74d60c21c98ef
2017-05-25 21:02:42 +02:00
Paul Belanger 3ef635e004
Remove 99-cache-testrepository-db from cache-devstack element
It seems this optimization is no longer needed. Once we address:

  https://bugs.launchpad.net/testrepository/+bug/1416512

We could consider adding it back. However, this also means we stop
hitting logstash.openstack.org each time we build an image. So, we
should maybe consider caching that data some place over having our
image builds do it.

Change-Id: I16b56a13028c0f228d21dde423db3f70f07f4e1a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-05-25 11:45:17 -04:00
Clark Boylan 0b00bd6abd Remove NPM mirror settings
The NPM mirror has unfortunately become unuseable as NPM itself has
grown to many terabytes in size and we haven't been able to keep up
downstream. Stop using the mirror by default as its current state is
causing some jobs to fail.

Change-Id: Iab9ece625ca71bb02c9c9b77ca7be9c89bcc47b9
2017-04-18 09:28:11 -07:00
Jenkins b5f3a0eab7 Merge "Stop enabling EPEL mirror by default" 2017-04-12 21:31:45 +00:00
Paul Belanger 8460475981 Update APT indexes before bringing node online
It is possible for our APT lists on disk to be out-of-date. Be sure to
update them, so we don't leave jobs in a broken state.

Change-Id: I019c1832fcaa0a6f162f71e9a8168e2b202f9a79
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-09 10:42:31 -04:00
Paul Belanger c26f59580b Remove precise references from configure_mirror.sh
Change-Id: I2a9e935c68fdfa974886bb727ec3706faebb4338
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-07 15:06:14 -04:00
Jenkins 375471856a Merge "Add reverse proxy caches to configure_mirror.sh" 2017-04-07 17:23:07 +00:00
Jenkins 764f11e59c Merge "Move nodepool mirror vars into seperate filename" 2017-04-07 17:23:00 +00:00
Jenkins df7ff2c65a Merge "Revert "Write mirror host location if mirror is in use"" 2017-04-07 16:54:33 +00:00
Paul Belanger e7184f1b89 Add reverse proxy caches to configure_mirror.sh
We'll we don't populate files on disk, give a way for projects to
source them first.

Change-Id: I58380647ad16cdffe5ced78c632caf2b05ddc5f3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-07 09:50:38 -07:00
Paul Belanger 5b193c7a10 Move nodepool mirror vars into seperate filename
Provide a simple way for projects to source nodepool mirror
variables. We construct the new file in such a way that it doesn't
depend on nodepool, but existing nodepool scripts use it just as
downstream consumers would so that we are sure that everything lines up
without double accounting.

Change-Id: I2642659406e72090cffe5163783290e22d6ef3c2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-07 09:50:36 -07:00
Paul Belanger 8ef7e61afa Revert "Write mirror host location if mirror is in use"
I think we want to go with: https://review.openstack.org/#/c/339261/

This reverts commit 5151d13338.

Change-Id: I2e26ce98d82c18009e237c6ae9042411e43f8959
2017-04-07 16:01:51 +00:00
Jenkins ec1b84ca40 Merge "Configure the openstack mirroring for openSUSE based images" 2017-04-07 08:04:41 +00:00
Dirk Mueller a777c761a4 Configure the openstack mirroring for openSUSE based images
Change-Id: I176902782e812362d09e1fe2852024aaa80a2e4f
2017-04-06 11:41:01 +02:00
Clark Boylan 5151d13338 Write mirror host location if mirror is in use
In various places we have wanted to have just the mirror host location
and not specific mirror config like for pip or apt or yum. For example
devstack-gate attempts to use this information to do simple networking
ping tests, dib tests construct yum repo locations on ubuntu and apt
repo locations on centos, and so on. Have a clear file that is set to
the root mirror location if a mirror is in use to make this simpler.

Change-Id: I306467f1870e5ee823b8b5c98d81917e607348ef
2017-04-05 11:44:12 -07:00
Monty Taylor befcdfacd7
Stop enabling EPEL mirror by default
Change-Id: I714cd44afb77191c89ac82f79cf57aa85a794e6d
2017-04-04 10:26:01 -05:00
Clark Boylan 018ee3e899 Add ocata and pike UCA repos
These are xenial only repos. Add them alongside the existing UCA setup.

Change-Id: Ib081d30e44e5517f5713f50e856c213dab981ba8
2017-03-29 10:23:04 -07:00
Paul Belanger bbbfaacd81 Disable deltarpms for fedora mirrors
Until now, this wasn't an issue. However it seems fedora 25 started
looking for a drpm file today:

  [MIRROR] pcre-8.40-5.fc25_8.40-6.fc25.x86_64.drpm: Status code: 404 for
  http://mirror.iad.rax.openstack.org/fedora/updates/25/x86_64/drpms/pcre-8.40-5.fc25_8.40-6.fc25.x86_64.drpm

Because we don't mirror drpms (to save space) our fedora-25 nodes are
failing.  So, just disable drpms for our mirrors.

Change-Id: I35b2f3f638f4551d5f169d1d6e081f29377ccb85
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-03-29 07:56:27 -04:00
Paul Belanger 062fa627f6 Add ubuntu-mariadb to configure_mirror.sh
Change-Id: I20d9091f6540dd8074820963132233cb1d2cd1c3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-03-02 14:13:31 -05:00
Paul Belanger 6f3bb9d7f5 Add nl01.o.o support for configure_mirror.sh
Because we are using the 'zuul' user to SSH into our zuul workers, we
need to add it to configure_mirror.sh and still support the jenkins
user.

Change-Id: I79eff670f56d06c49a16e1330a73889e7b8482f1
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-02-22 10:07:21 -05:00
Paul Belanger 736cee2421 Remove fedora-24 from openstack-infra
Now that we have migrated all our jobs to fedora-25, we can remove
fedora-24 from project-config.

Change-Id: I3a58c785c12f779060a6dca5cd40ddcc46429971
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-01-22 12:02:33 -05:00
Paul Belanger 25968cc69a Add support for fedora (25+) AFS mirrors
Now that fedora-25 is online, we can start using the new AFS mirror for
fedora. Keep in mind, we didn't mirror fedora-24, since fedora-25 will
be replacing it.

Once fedora-24 has been removed from nodepool.yaml, we can remove our
conditional check.

Change-Id: I1bd510c6a568b71b604c27d6cae16d2654063d0c
Depends-On: I762e55fb7fe0c5ebfb7810545ddb4d5ccfedf2ff
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-01-13 16:11:37 -05:00
Jenkins 1b4bc250cc Merge "cleanup: remove unused scripts used for snapshots" 2016-11-15 15:25:59 +00:00
Paul Belanger e2cec702be Support all 3 repos for UCA mirrors
We incorrectly setup our mirrors for UCA. Now we support 3 openstack
releases.

Change-Id: Ie3a05398d9fdc7a7f7c0c74366a7c4dd70200a2b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-11-08 12:22:13 -05:00
Isaac Beckman 3f04bd8071 cleanup: remove unused scripts used for snapshots
This review cleanup scripts that were exclusively used by
snapshot scripts for nodepool which were removed in:
https://review.openstack.org/#/c/331494/

Change-Id: I37900a5d698a43bc0d510100289a5a7ec527021d
2016-11-06 11:43:16 +02:00
Jenkins 5d5fa20a57 Merge "Remove snapshot scripts for nodepool" 2016-11-03 14:45:01 +00:00
Paul Belanger d9ce152ced
Use Debian AFS mirror for debian-jessie nodes
Change-Id: I16f4232570975ee668d52785de0276c4a60539a7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-08-26 22:47:24 -04:00
Paul Belanger d623c337ba
Include dib-builddate.txt for configure_mirror.sh
Update our script to provide additional debug information, such as the
builddate of our DIB images.

Change-Id: Ie399b1ac9cd6c6ea4372378ae7b7bf930fac16a3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-08-23 11:24:58 -04:00
Paul Belanger a44be1116e
Run host lookup first for configure_mirror.sh
A cosmetic change to reduce the debug output on DNS failures. First
check for valid DNS, then proceed with configuration.

Change-Id: I3370a155bf6b49088c398dc9d4d572b6210b84b2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-08-23 11:21:49 -04:00
Jenkins a286f8c8d4 Merge "Add smarter dns checking for configure_mirror.sh" 2016-08-22 18:08:58 +00:00
Jenkins 64e0f4dba7 Merge "Resolve DNS soon in configure_mirror.sh" 2016-08-18 13:43:31 +00:00
Paul Belanger 79120078ea
Use ip6tables if nodepool is using IPv6
Change-Id: Ifebc8061dbbc99eed47938c7401b5220fd62d19a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-08-17 13:41:47 -04:00
Paul Belanger fce3a76c22
Add smarter dns checking for configure_mirror.sh
After noticing ubuntu-xenial launch failures in osic-cloud1, it looks
like our unbound service is taking up to 1 minute to start properly.
So wait 30 seconds for host to timeout, and try 10 times.  This gives
us a 300 second timeout window to configure DNS properly.

Change-Id: Id0432c91cc853fb4ecab43da991948c2e9d84b7d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-08-15 22:20:17 -04:00