Commit Graph

126 Commits

Author SHA1 Message Date
Clark Boylan 2e9cd0c803 Retire this repo
The opendev project has been moving away from puppet and this is one of
the puppet modules that is no longer used. To simplify things for us we
are taking the extra step of retiring this repo.

Change-Id: I83b9374c66f13d672800e76e8c10903f82479471
2022-02-17 12:42:55 -08:00
David Shrewsbury bb0f56209d Support aarch64 builders for jemalloc usage
Change-Id: I0af7a221c02615aa81eb5c90bbafdd36b37d25e2
2019-05-06 13:29:24 -04:00
Zuul 56f8af9a88 Merge "Enable jemalloc on nodepoolv3 builders + launchers" 2019-05-06 14:50:33 +00:00
David Shrewsbury d503ffbcdd Enable jemalloc on nodepoolv3 builders + launchers
Depends-On: https://review.opendev.org/656986
Change-Id: I721326d75771198cce0289b6ee21596f0b761467
2019-05-06 09:48:52 -04:00
David Shrewsbury f5ae4069d5 Install jemalloc on nodepool servers
We already use this library on the Zuul executors to help with
memory issues. The nodepool builders and launchers are seeing
memory issues as well, and it is believed/hoped that using jemalloc
will help reduce some of the memory pressure.

Change-Id: I02129bb33baf00fedb5d1a6f6b82944a05c84963
2019-05-03 13:19:44 -04:00
David Shrewsbury 5eb147f7d9 Install yappi and objgraph packages
If yappi is installed, we can get nodepool to dump thread info.
If objgraph is installed, we can get object counts to help find
memory leaks (which we seem to have).

Change-Id: I88e79da77fb03c6d522f48baa19d55fd69968528
2019-04-22 13:57:44 -04:00
OpenDev Sysadmins c3b5a3497e OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:26:10 +00:00
Ian Wienand c7f47f956e Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: Ie8487699fafa714d74df40b0bb955eaef63ee9ce
2019-03-24 20:35:42 +00:00
Clark Boylan 71fe2002e5 Make builder ssl vhost futureparser safe
The builder ssl vhost template looks up vars as if they are local.
Problem is with futureparser and puppet4 these vars don't make it into
the global scope so they aren't valid here. Instead we do explicit
lookup of the values in the correct scope.

Change-Id: I92a6533d47075d8dac404999a6e3b5c4ccea84ed
2019-01-04 16:52:13 -08:00
Zuul 1114e062f2 Merge "Increase timeout on nodepool install" 2018-09-12 06:50:02 +00:00
David Shrewsbury fe2f7811f3 Add openstacksdk logging to nodepool builder
Nodepool now uses openstacksdk in place of shade. Add a logger
for that.

Also, this makes the builder logging config look like the one in
system-config which is, accidentally, the actual logging config
being used right now. I'm going to put up another change to use
the logging config in this repo, which is the correct thing to do
for v3 of nodepool.

Change-Id: I5e6ab0a34e9e5df0ee152b33b01e5266691e6fc6
2018-07-26 10:00:20 -04:00
David Shrewsbury 845ebb93e3 Update nodepool launcher logging
We've moved from shade to openstacksdk.

Change-Id: I13df848ac742a9c2a48b57ef128482bb24637767
2018-07-25 16:25:37 -04:00
Zuul 671ee78949 Merge "Fix name of libxslt dev package" 2018-07-20 18:15:51 +00:00
Colleen Murphy 502580dd25 Update Gemfile for Zuulv3
The logic in the Gemfile was relying on Zuulv2 variables to find out
whether the spec helper gem was already available on disk, and since
Zuulv3 has changed things it was failing to find it and downloading the
master version instead. This patch ensures the Gemfile looks for the gem
in the right place when running in CI.

Change-Id: Ic85e06a90f464e34df1d0ead6e2973926809b532
2018-07-12 09:57:45 +02:00
Colleen Murphy c945f0e3f1 Fix name of libxslt dev package
`libxslt-dev` is the virtual package. Puppet can install it fine, but
can't detect that it's already installed and will attempt to reinstall
it on every run, breaking idempotency tests. Use the real name of the
package, `libxslt1-dev`, to avoid this. This name is the same for
Trusty, Xenial, and Bionic.

Change-Id: If65865c5d895559f48db325b65f3e72445ef9d4b
2018-06-28 10:13:43 +02:00
Ian Wienand 7cd8a2d4a3 Fix ssl template logic
If we *have* a value for the ssl_cert_file, we want to use the SSL
template.

Change-Id: I470642d6308b4cb59deacf8b7c8e849d0a8efb0d
2018-05-21 15:51:36 +10:00
Ian Wienand 0fb8056471 Fix ssl key lookup
It seems the scope.lookupvar() doesn't work with non-global variables;
use the @ syntax.  This is copied from puppet-cgit

Change-Id: I38eb50157d25c6c7900a8455eddedb4655c4637d
2018-05-21 14:47:55 +10:00
Ian Wienand c1ca2795c2 Add SSL support in builder
We wish to export the build logs and artifacts via https; not so much
for any security reasons but for transparent-proxy-busting effects.

Add SSL arguments and a template that redirects 80->443 for hosts if
they're passing in key contents.

Change-Id: I8a15333a7c662f3d32fa4045785498dafc87ad53
2018-05-17 13:46:58 +10:00
Zuul f4f6314397 Merge "Add bindep.txt file" 2018-04-20 19:33:40 +00:00
Clark Boylan 2d269e75a1 Serve qcow2 images from builder http servers
Let people download our qcow2 images to aid in local reproduction of
issues. We only serve the qcow2 files as they are smallest.

Note that this should probably be updated to use a negative match
against qcow2 so that we can ensure only qcow2 is served regardless of
what image formats we are using. Unfortuantely I can't get that to work
with autoindex. If you have ideas feel free to share them.

Change-Id: Ia23e68e87d2126eac2e22d9c3fed8406e2f447cb
2018-04-20 11:28:27 -07:00
Paul Belanger f5dfa52d44
Add bindep.txt file
Add support for bindep.txt as we are wanting to remove
bindep-fallback.txt.

Change-Id: I161ca1c8b1d9644d2a5b2900de48f2d2df2f60a3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-20 14:05:07 -04:00
Zuul 026afd3957 Merge "Enable build logs on builder, web app on launchers" 2018-02-25 23:41:50 +00:00
Ian Wienand 0be57c44c3 Increase timeout on nodepool install
This can require building a few binary packages, which can take a
while on a slower cloud.  Increase timeout over default 300s

Change-Id: I4b50c1af1f3fa4afb2f28e14f699fa8c3166e01e
2018-02-22 10:37:09 +11:00
Ian Wienand 97b05c1432 Disable vhd on aarch64
The ppa with the special vhd packages doesn't have arm support;
disable this.

Change-Id: I214b2d3a317a3ef072a87393f178a441a85881ec
2018-02-21 10:48:59 +11:00
Ian Wienand a8682baa18 Enable build logs on builder, web app on launchers
There is currently a confusing array of exporting options for
exporting build and upload logs.  The upload logs have never really
worked, and the info is better given from image-list end-points.  The
build logs since Ia7415d2fbbb320f8eddc4e46c3a055414df5f997 are logged
into separate files into /var/log/nodepool/builds.  With a separate
builder, it can not access the webapp ports provided by the launcher,
so the redirects there are wrong -- and we don't deploy apache on the
launcher to expose them.  If you're using single node anyway, it's
more than likely you have zuul which installs the main website so
can't enable this.

For these reasons deprecate and remove these old settings.

Instead, add a separate flag to each of the builder and launcher.

builder::enable_build_log_via_http will deploy apache and a config
file that exposes /var/log/nodepool/builds

launcher::enable_webapp will deploy apache and redirects to the
internal webapp

Note this does not handle a launcher and builder on the same host;
since the webapp, and indeed refactoring this module in general for
multiple daemons, is under some development, we leave this as a future
task.

Depends-On: https://review.openstack.org/543667
Change-Id: I447886dd32f7f3bc6758ffd7a1b725689d04ee68
2018-02-13 14:15:03 +11:00
Paul Belanger dae2c5809f
Add kazoo logging to nodepool
Both launcher and builder now use kazoo, we should at least log INFO
logging. This will help see what is going on when we loose connection
to zookeeper.

Also remove gear logger from launcher, it is no longer needed.

Change-Id: I7024e6f5994d8f79bae853a642ada3b4b52bf6e6
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-01-28 11:36:47 -05:00
Paul Belanger 70e0e5f1c6
Fix typo in nodepool-builder.init
Change-Id: I5902595d1aa6ee4e7141b58b51ce6913b73df627
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-01-19 11:32:17 -05:00
Paul Belanger aaa485a777
Support systemdctl daemon-reload for nodepool services
Otherwise, when we modify systemd unit files, we won't be able the use
them.

Change-Id: Ie2a383e771a4f6fe606614ed09134576ca5b6a66
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-01-18 09:05:39 -05:00
Paul Belanger 078f336a01
Don't assume pidfile location
When configuring the dir for the pidfiles in our init scripts don't
assume the path is /var/run/$NAME. The defaults are now /var/run/nodepool
now and could be set to other values as well. Instead of assuming
/var/run/$NAME we instead take the dirname of $PIDFILE and configure
whatever that dir is instead.

Also expose zuulv3 flag to user to toggle the location of pidfile,
this changes in nodepool > 0.5.0.

Change-Id: I7fff2565f9e77d737c247f204925f66a0e4aabe8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-01-17 14:33:32 -05:00
Colleen Murphy e2b8c27908 Depend on helper gem for spec_helper_acceptance
Instead of keeping a local copy of spec_helper_acceptance.rb and
requiring updates to all modules for any change, we can move it into the
common helper gem and require it from there. This will make it easier to
create and review changes that affect all puppet modules. Also change
the Gemfile to look for the gem in the local workspace if running in a
zuul environment.

Change-Id: I17643139de1e79403dce09a5c5d5e66908b5c8d9
2017-08-18 10:41:44 +02:00
Paul Belanger c65c6f84f4
Use double quotes for $pip_command
Fix syntax error with pip install command.

Change-Id: Ib42c6a4fbc5cef8cf937ee36227136e6de7152ed
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-07-26 13:13:44 -04:00
David Shrewsbury 07e39e7bdc Add support for python 3
Change-Id: Ie2c223d74b32798e5f0cb0ca8b76d9fa6a4f3095
2017-07-19 11:52:30 -04:00
Colleen Murphy f7202f7743 Update beaker setup for xenial
Add a xenial nodeset and update the spec helper to install puppet 3 from
the Ubuntu repos instead of from puppetlabs.

Change-Id: I4442570d78d6ccf98a9718dab5f3bcf40c3dbd42
2017-06-11 20:54:00 +02:00
Paul Belanger f88e7f75e7 Rename nodepoold to nodepool-launcher for zuulv3
Change-Id: I83a27cff4e998c165f534c1111d537f9d45b2ac2
Depends-On: Ic04e3cf2dbdaf914bf8f92d073acb972380708f1
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-03-28 13:12:14 -04:00
Paul Belanger 8d91d91108 Add statsd_prefix to launcher.pp
Expose the ability to override our statsd_prefix, which defaults to
none.

Change-Id: I8bef6aae5a566ff3a34e816ece14f9cfc1ef289d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-03-10 10:49:57 -05:00
Paul Belanger c283f5975e Switch back to nodepoold for nodepool-launcher service
We haven't yet renamed nodepoold to nodepool-launcher, so revert this
change.

Change-Id: I976b13fc2cfee4b6bf253077e6b27d2de9419895
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-02-21 08:43:50 -05:00
Jenkins d62777026c Merge "Add redirects to json status output" 2017-02-16 21:08:33 +00:00
Paul Belanger c68c8fcede Create nodepool::launcher
Move nodepool-launcher into its own class. This allows us to scale out
nodepool-launcher for our feature/zuulv3 effort.

Change-Id: I393fa1d8ff080260af772a2f020cca9b9e49b173
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-02-14 13:27:10 -05:00
Ian Wienand ef52d4952a Add redirects to json status output
These were added to nodepool with
I3410a4e4efd649732747b0502d66d212f50fa1bb

Change-Id: I5828a3eb94232596c1ad9f081d53e412e944bff3
2017-02-03 11:05:19 +11:00
Paul Belanger 90929f8848 Remove --no-images from nodepool
Now that the feature/zuulv3 branch of nodepool has been merged into
master, remove deleted args from our default files.

Change-Id: Icd595ae3311fa8428b812292c8f40c290bde5e3b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-12-01 12:44:44 -05:00
Monty Taylor 89789b79c0
Ensure python-openssl is not present
python-openssl should be installed from pip along with the rest of the
requirements. Not doing so leads to issues with versions of pyOpenSSL
that are too old.

Change-Id: I22a7d6e4080efbfcd1f86083505d015cf33ea885
2016-11-29 16:30:30 -06:00
Jenkins ad163ceb22 Merge "Add SSH public key to nodepool-builder" 2016-11-29 17:58:54 +00:00
Paul Belanger c2e9f66d73 Add SSH public key to nodepool-builder
Since we use DIB elements to create our zuul user, we also need to
have our SSH public key on disk. Other wise, diskimage builds will
fail.

Change-Id: I6879d095941fe76d151d3bd9e590b1f691c146e2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-11-29 12:24:53 -05:00
James E. Blair 3a2e9c0d34 Conditionally set mysql max connections
Only install the max_connections config file for the mysql server
if we are installing the mysql server.

Change-Id: I0c9beb54d3e6982fc4c1b4de8ce1f81eb40c654d
2016-11-23 12:50:38 -08:00
Paul Belanger 799dbee521 Add option not to install nodepool-builder from init.pp
Give an option to users to not install nodepool-builder from init.pp.

Change-Id: I387de90147a8d0bc1b50dfd29ada1015542a6c57
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-11-18 14:49:29 -05:00
Paul Belanger 8f8753d90e include ::diskimage_builder in builder.pp too
Since it is possible for ::nodepool::builder to only be called, we
need to make sure diskimage-builder is also installed.

Change-Id: Ibbb49bbb6a28ee402dfb60ca5ca1fa2ae8f2f7da
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-11-18 14:20:47 -05:00
Ian Wienand cd086492a4 Add image upload log arguments
Add image upload log directory and arguments.  This is where provider
upload logs configured in Ic80e532891f039e0f835847ddfbae38f1ce3cd8aIf
can be stored.

Note http export (which we do not enable in infra at this time) would
put logs under /upload ; it might be more logical to move the build
logs to be a sibling at /build but that would be backwards
incompatbile.

Change-Id: I0cc9b59063fb14920f0da581b8036c2664d5d5f1
2016-11-08 13:36:05 +11:00
Clark Boylan d70a85b686 Properly set pidfile for nodepoold
Nodepoold wants to know what its pidfile is too so we need to set that
in the init scripts after we set DAEMON_ARGS. Without this it will
fallback to its default which causes problems when running more than one
nodepoold process.

Change-Id: I5cc73542ae2499f13fadcfad8f692f5123f62630
2016-10-06 11:44:54 -07:00
Paul Belanger 39e8bb392c
Add missing default files for launcher / deleter
Change-Id: If0a0c69dfb8979c657edaf99de49761c43a8a108
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-10-06 14:24:51 -04:00
Paul Belanger 65d3344f81
Add nodepool-(builder|deleter) services
We've been running 3 nodepool services for a week now, lets properly
template them. nodepool uploads images, nodepool-launcher creates
nodes, nodepool-deleter deletes them.

This is mean to be a short term solution for zuulv25, and shouldn't
affect existing nodepool installations.

Change-Id: Iaee3148d6710f796e9f1ea7cba2962cfbb530fa2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-10-05 17:27:35 -04:00