Commit Graph

272 Commits

Author SHA1 Message Date
Clark Boylan cfc83807b7 Organize zuul jobs in zuul.d/ dir
Our .zuul.yaml file has grown quite large. Try to make this more
manageable by splitting it into zuul.d/ directory with jobs organized by
function.

Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 17:30:48 -05:00
Zuul 99f809ccc5 Merge "Use zuul checkouts of ansible roles from other repos" 2020-05-07 18:41:21 +00:00
Zuul b59c9d8a2b Merge "Retire nb01/02.openstack.org" 2020-05-07 18:00:10 +00:00
Zuul 502ddff9b3 Merge "Test zuul-executor on focal" 2020-05-07 17:53:20 +00:00
Zuul 351264c919 Merge "Run cloud_launcher from zuul" 2020-05-07 15:31:29 +00:00
Zuul 1d8bb10ffe Merge "Add system-config-run-base-arm64" 2020-05-07 03:54:05 +00:00
Ian Wienand e400865dd0 Retire nb01/02.openstack.org
Remove references to these older builders.  We thank them for their
service.

Change-Id: I1f48f070406bee79ac0d1de61beb44eb7d58d605
2020-05-07 13:06:26 +10:00
Monty Taylor 39495ffdd5 Test zuul-executor on focal
We want to replace the current executors with focal executors.
Make sure zuul-executor can run there.

Kubic is apparently the new source for libcontainers stuff:

  https://podman.io/getting-started/installation.html

Use only timesyncd on focal

ntp and timesyncd have a hard conflict with each other. Our test
images install ntp. Remove it and just stay with timesyncd.

Change-Id: I0126f7c77d92deb91711f38a19384a9319955cf5
2020-05-06 18:00:29 -05:00
Zuul e56cbdcee3 Merge "Run nodepool launchers with ansible and containers" 2020-05-06 14:21:53 +00:00
Zuul 4f53da0f11 Merge "Add focal integration tests" 2020-05-05 20:13:43 +00:00
Monty Taylor 4b9d1a88bd Use zuul checkouts of ansible roles from other repos
We have two standalone roles, puppet and cloud-launcher, but we
currently install them with galaxy so depends-on patches don't
work. We also install them every time we run anything, even if
we don't need them for the playbook in question.

Add two roles, one to install a set of ansible roles needed by
the host in question, and the other to encapsulate the sequence
of running puppet, which now includes installing the puppet
role, installing puppet, disabling the puppet agent and then
running puppet.

As a followup, we'll do the same thing with the puppet modules,
so that we arent' cloning and rsyncing ALL of the puppet modules
all the time no matter what.

Change-Id: I69a2e99e869ee39a3da573af421b18ad93056d5b
2020-04-30 12:39:12 -05:00
Monty Taylor 1b126ef48a Run cloud_launcher from zuul
This is running on a cron right now, let's run it from zuul.

This moves the contents from clouds_layouts into the hostvars
for bridge and changes the playbook to run against bridge
instead of localhost. This lets us not pass in the variables
on the CLI, which we don't have support for in the apply job.
It also is made possible by the lack of all-clouds.yaml.

Change-Id: If0d2aacc49b599a0b51bf7d84f8367f56ed2d003
2020-04-30 12:37:38 -05:00
Monty Taylor e0619f17f1 Run nodepool launchers with ansible and containers
We don't run start in prod normally but we do need to run
it in the gate.

Change-Id: Iec50684280409eb978bf5638bf74ae16fad8aa26
2020-04-30 17:37:22 +00:00
Ian Wienand 18ed6f3364 Add system-config-run-base-arm64
We have a mirror job for arm64, but it runs infrequently and can get
broken by base changes (as described inline we can't currently have a
mixed environment).  Let's run the base playbook too to give more
visibility.

Change-Id: I557bdadf7fe09463b4bb51130df6b88737bb9a46
2020-04-30 09:04:21 +10:00
Zuul fdfbc3d0b9 Merge "Run zookeeper cluster in nodepool jobs" 2020-04-29 22:22:37 +00:00
Monty Taylor 8d7075b02f Run zookeeper cluster in nodepool jobs
Rather than running a local zookeeper, just run a real zookeeper.
Also, get rid of nb01-test and just use nb04 - what could possibly
go wrong?

Dynamically write zookeeper host information to nodepool.yaml

So that we can run an actual zk using the new zk role on hosts in
ansible inventory, we need to write out the ip addresses of the
hosts that we build in zuul. This means having the info baked in
to the file in project-config isn't going to work.

We can do this in prod too, it shouldn't hurt anything.

Increase timeout for run-service-nodepool

We need to fix the playbook, but we'll do that after we get the
puppet gone.

Change-Id: Ib01d461ae2c5cec3c31ec5105a41b1a99ff9d84a
2020-04-29 16:18:25 -05:00
Clark Boylan 7c92589f3a Increase timeout on system-config-run-zuul
This job compiles openafs with dkms among other things that cause it run
over the default half hour timeout occasionally. Bump the timeout to an
hour to deal with that.

Change-Id: I8a56a7f42ce2ee8331befb45aceb1d511a33d9e6
2020-04-29 17:03:56 +00:00
Ian Wienand f1e5c5d34e Add focal integration tests
Change-Id: Ida7215569a0a5960961706583095322f6418c263
2020-04-29 12:22:05 +10:00
Zuul 81d9dc44ab Merge "Use the sync-project-config role in service-zuul" 2020-04-28 14:28:36 +00:00
James E. Blair 913ea11a86 Clean up some job descriptions
Also remove the unused infra-prod-zuul_reconfigure job.

Change-Id: I67272b35a9c40ba70bba028cbd671faa852d1d19
2020-04-27 14:40:18 -07:00
Clark Boylan 18568882b8 Fix zuul.conf jinja2 template
This adds a necessary newline, removes port numbers, and sets the
executor ssh key to the correct path.

Change-Id: I6b4afa876b6cd7d8f87cc35bc51b4e9d6e31ee2b
2020-04-24 18:15:31 -07:00
Monty Taylor b7c0c0f8ce The package is libjemalloc1
When we install packages on ubuntu, we should use their actual
package names rather than incorrect or otherwise fictional
package names.

Also, fix the hostname in the test job - because when we don't
do that, we don't run all of the roles, and thus we don't
catch these things.

Change-Id: I18e676ef0fe343513db4c8ad7e340ee45092c0a3
2020-04-24 18:15:31 -07:00
Monty Taylor c1714ef918 Use the sync-project-config role in service-zuul
We can sync the project-config repo instead of cloning it on the
hosts.

Change-Id: Ic238d743f4dda0a3c04a7e3618a9d86d3c3e5371
2020-04-24 14:43:35 -05:00
Zuul 1febd9a0f2 Merge "Add service-zookeeper to nightly jobs" 2020-04-24 17:58:21 +00:00
Zuul 8bd18e9089 Merge "Actually run service-zuul playbook in prod" 2020-04-24 17:34:26 +00:00
Monty Taylor d4922ce29d Add service-zookeeper to nightly jobs
We missed adding a periodic job for it when we added it.

Change-Id: I560bf11a56ffb1cf67f89f3bc2749758385dd37e
2020-04-24 16:59:39 +00:00
Monty Taylor 535e1602c0 Actually run service-zuul playbook in prod
We added the playbook but not so much a job to run it.

Change-Id: I385b57f6a4499ab0c6b6874eced7e9fd34e71bc7
2020-04-24 11:39:56 -05:00
Zuul b21a8e58cf Merge "Run Zuul using Ansible and Containers" 2020-04-24 16:31:42 +00:00
Zuul 3b48da5a74 Merge "Stop logging puppet to syslog" 2020-04-24 15:02:36 +00:00
Zuul 1b2d113c0f Merge "Split eavesdrop into its own playbook" 2020-04-24 15:02:34 +00:00
Monty Taylor f0b77485ec Run Zuul using Ansible and Containers
Zuul is publishing lovely container images, so we should
go ahead and start using them.

We can't use containers for zuul-executor because of the
docker->bubblewrap->AFS issue, so install from pip there.

Don't start any of the containers by default, which should
let us safely roll this out and then do a rolling restart.
For things (like web or mergers) where it's safe to do so,
a followup change will swap the flag.

Change-Id: I37dcce3a67477ad3b2c36f2fd3657af18bc25c40
2020-04-24 09:18:44 -05:00
Zuul e044023dc9 Merge "Split codesearch into its own playbook" 2020-04-24 13:57:03 +00:00
Monty Taylor 99aa528c83 Stop logging puppet to syslog
We run puppet with ansible now pretty much all the time. It's not
helpful for the puppet output to go to syslog on the remote host.
What's more helpful is for it to come back to the stdout in the
ansible playbook so that we can see it.

Also turn off ansi color from the output.

Depends-On: https://review.opendev.org/721732
Change-Id: I604081d5400bd53b8dda5a3a7685323c1443991b
2020-04-23 19:38:51 +00:00
Monty Taylor 9fd2135a46 Split eavesdrop into its own playbook
Extract eavedrop into its own service playbook and
puppet manifest. While doing that, stop using jenkinsuser
on eavesdrop in favor of zuul-user.

Add the ability to override the keys for the zuul user.

Remove openstack_project::server, it doesn't do anything.

Containerize and anisblize accessbot. The structure of
how we're doing it in puppet makes it hard to actually
run the puppet in the gate. Run the script in its own
playbook so that we can avoid running it in the gate.

Change-Id: I53cb63ffa4ae50575d4fa37b24323ad13ec1bac3
2020-04-23 14:34:28 -05:00
Monty Taylor 08a807e188 Add a uwsgi-base container image
For things using the builder/base pattern providing uwsgi services.

Change-Id: Iaf6d31a3d119f6b7e87b54cda1969a9994110dad
2020-04-23 13:11:44 +00:00
Monty Taylor d5c68c5131 Split codesearch into its own playbook
Make a service playbook, manifest and jobs for codesearch.

Remove openstack_project::server - it doesn't do anything.

Change-Id: I44c140de4ae0b283940f8e23e8c47af983934471
2020-04-21 13:18:28 -05:00
Zuul fe7d38af63 Merge "Remove system-config-puppet-beaker-rspec-puppet-4-centos-7-infra" 2020-04-21 18:09:45 +00:00
Zuul fbcc45401b Merge "Use real passwords for meetpad" 2020-04-20 18:29:43 +00:00
Zuul 5c1c0d4a56 Merge "Move puppet apply jobs to system-config repo" 2020-04-20 17:30:56 +00:00
Zuul f5cb448351 Merge "Remove puppet-beaker-rspec-puppet-4-infra-system-config" 2020-04-20 17:13:33 +00:00
Andreas Jaeger ef2a21cc4d Remove system-config-puppet-beaker-rspec-puppet-4-centos-7-infra
This job is not used, remove it. OpenDev doesn't run CentOS 7 anymore.

Change-Id: I6865f24a6670d0c545a1e1014f7dd954d324b00d
2020-04-20 19:05:37 +02:00
James E. Blair f7bf07a03d Use real passwords for meetpad
The docker containers expect this now and refuse to start with
fake passwords.

Change-Id: I4c4bd243c9684e3987eeb99e4c66d31a882336a0
2020-04-20 09:05:51 -07:00
Monty Taylor fedd6c42f8 Fix remote_puppet playbook names
They have underscores, not dashes.

Change-Id: I6e3e0e96e38f24484e6038e502dfcc279e8e4fbf
2020-04-20 07:44:52 -05:00
Monty Taylor ca684c775b Move puppet apply jobs to system-config repo
These use legacy-base, which sucks, but what sucks even more is
that they are in openstack-zuul-jobs, which makes them extra
awkward to try to adjust.

Change-Id: I87b3d56de41f0ba5658c1240ddfc7ecf1c3c43af
2020-04-19 10:57:22 -05:00
Monty Taylor 1a39133661 Remove puppet-beaker-rspec-puppet-4-infra-system-config
This doesn't actually do anything useful any more, but it spends
a lot of time not doing it.

Basically, this is only testing the things in
modules/openstack_project/spec/acceptance/basic_spec.rb, which
are things we install and test in ansible now.

There are related jobs, puppet-beaker-rspec-puppet-4-infra,
which are run on puppet- repos and run their rspec tests, but
that won't be affected by this.

Change-Id: I21b01d360b50dba10673c2986e8a2868b8747522
2020-04-19 10:57:22 -05:00
Zuul faccef4594 Merge "Add install-docker and pip3 to files triggers" 2020-04-17 22:19:46 +00:00
Monty Taylor 7f5faf8eec Add install-docker and pip3 to files triggers
The jobs which use install-docker and pip3 should be triggered
by changes to install-docker or pip3.

Change-Id: Ia6ec8da72fee38377760cb27dd7df26fa169760b
2020-04-17 20:31:33 +00:00
Monty Taylor f378793ac5 Remove semaphore from service-bridge
This is not intended to have its own semaphore.

Change-Id: I587b22f099c6031177e590dbb8dabd8c2ee58295
2020-04-17 15:29:06 -05:00
Zuul b0ab2f37c5 Merge "Run ZK from containers" 2020-04-17 16:49:34 +00:00
Clark Boylan e7878b1ddb Simplify .zuul.yaml regexes
Zuul uses an re.match() check on file list regexes. This means that the
leading ^ is redundant as is a trailing .*

Attempt to make this more clear by dropping those leading and trailing
regex operators to be consistent across the file. This makes the rules
easier to read and should make them easier to reason about.

Change-Id: Id4cd17d816c9af023a655bdadeedb9421e51cdca
2020-04-17 08:54:27 -07:00