The opendev project has been moving away from puppet and this is one of
the puppet modules that is no longer used. To simplify things for us we
are taking the extra step of retiring this repo.
Change-Id: I83b9374c66f13d672800e76e8c10903f82479471
We already use this library on the Zuul executors to help with
memory issues. The nodepool builders and launchers are seeing
memory issues as well, and it is believed/hoped that using jemalloc
will help reduce some of the memory pressure.
Change-Id: I02129bb33baf00fedb5d1a6f6b82944a05c84963
If yappi is installed, we can get nodepool to dump thread info.
If objgraph is installed, we can get object counts to help find
memory leaks (which we seem to have).
Change-Id: I88e79da77fb03c6d522f48baa19d55fd69968528
`libxslt-dev` is the virtual package. Puppet can install it fine, but
can't detect that it's already installed and will attempt to reinstall
it on every run, breaking idempotency tests. Use the real name of the
package, `libxslt1-dev`, to avoid this. This name is the same for
Trusty, Xenial, and Bionic.
Change-Id: If65865c5d895559f48db325b65f3e72445ef9d4b
We wish to export the build logs and artifacts via https; not so much
for any security reasons but for transparent-proxy-busting effects.
Add SSL arguments and a template that redirects 80->443 for hosts if
they're passing in key contents.
Change-Id: I8a15333a7c662f3d32fa4045785498dafc87ad53
This can require building a few binary packages, which can take a
while on a slower cloud. Increase timeout over default 300s
Change-Id: I4b50c1af1f3fa4afb2f28e14f699fa8c3166e01e
There is currently a confusing array of exporting options for
exporting build and upload logs. The upload logs have never really
worked, and the info is better given from image-list end-points. The
build logs since Ia7415d2fbbb320f8eddc4e46c3a055414df5f997 are logged
into separate files into /var/log/nodepool/builds. With a separate
builder, it can not access the webapp ports provided by the launcher,
so the redirects there are wrong -- and we don't deploy apache on the
launcher to expose them. If you're using single node anyway, it's
more than likely you have zuul which installs the main website so
can't enable this.
For these reasons deprecate and remove these old settings.
Instead, add a separate flag to each of the builder and launcher.
builder::enable_build_log_via_http will deploy apache and a config
file that exposes /var/log/nodepool/builds
launcher::enable_webapp will deploy apache and redirects to the
internal webapp
Note this does not handle a launcher and builder on the same host;
since the webapp, and indeed refactoring this module in general for
multiple daemons, is under some development, we leave this as a future
task.
Depends-On: https://review.openstack.org/543667
Change-Id: I447886dd32f7f3bc6758ffd7a1b725689d04ee68
Otherwise, when we modify systemd unit files, we won't be able the use
them.
Change-Id: Ie2a383e771a4f6fe606614ed09134576ca5b6a66
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
When configuring the dir for the pidfiles in our init scripts don't
assume the path is /var/run/$NAME. The defaults are now /var/run/nodepool
now and could be set to other values as well. Instead of assuming
/var/run/$NAME we instead take the dirname of $PIDFILE and configure
whatever that dir is instead.
Also expose zuulv3 flag to user to toggle the location of pidfile,
this changes in nodepool > 0.5.0.
Change-Id: I7fff2565f9e77d737c247f204925f66a0e4aabe8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Expose the ability to override our statsd_prefix, which defaults to
none.
Change-Id: I8bef6aae5a566ff3a34e816ece14f9cfc1ef289d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Move nodepool-launcher into its own class. This allows us to scale out
nodepool-launcher for our feature/zuulv3 effort.
Change-Id: I393fa1d8ff080260af772a2f020cca9b9e49b173
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
python-openssl should be installed from pip along with the rest of the
requirements. Not doing so leads to issues with versions of pyOpenSSL
that are too old.
Change-Id: I22a7d6e4080efbfcd1f86083505d015cf33ea885
Since we use DIB elements to create our zuul user, we also need to
have our SSH public key on disk. Other wise, diskimage builds will
fail.
Change-Id: I6879d095941fe76d151d3bd9e590b1f691c146e2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Only install the max_connections config file for the mysql server
if we are installing the mysql server.
Change-Id: I0c9beb54d3e6982fc4c1b4de8ce1f81eb40c654d
Give an option to users to not install nodepool-builder from init.pp.
Change-Id: I387de90147a8d0bc1b50dfd29ada1015542a6c57
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Since it is possible for ::nodepool::builder to only be called, we
need to make sure diskimage-builder is also installed.
Change-Id: Ibbb49bbb6a28ee402dfb60ca5ca1fa2ae8f2f7da
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Add image upload log directory and arguments. This is where provider
upload logs configured in Ic80e532891f039e0f835847ddfbae38f1ce3cd8aIf
can be stored.
Note http export (which we do not enable in infra at this time) would
put logs under /upload ; it might be more logical to move the build
logs to be a sibling at /build but that would be backwards
incompatbile.
Change-Id: I0cc9b59063fb14920f0da581b8036c2664d5d5f1
We've been running 3 nodepool services for a week now, lets properly
template them. nodepool uploads images, nodepool-launcher creates
nodes, nodepool-deleter deletes them.
This is mean to be a short term solution for zuulv25, and shouldn't
affect existing nodepool installations.
Change-Id: Iaee3148d6710f796e9f1ea7cba2962cfbb530fa2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
When installing nodepool on single_node_ci context, it
gives an error for build-essential duplicated declaration
(nodepool and zuul). Zuul is protecting that with an if !defined
declaration, but nodepool is not doing that.
Enclose all the package list in ensure_packages, to protect
from duplication errors.
Change-Id: Ibfc7af81ff8429d592d7d9fe70b06e1f5be4e77f
So that the nodepool and zuul modules can be coinstalled, each needs
a conditional wrapper around the build-essential package to avoid a
duplicate resource conflict. Corresponding change is:
I17c84c21bea9a93000bc3e34f6c004c58c04c10f
Change-Id: I3f230c3a5c245fa0c2503836b5cc17b2b1106f87
Because we are using the devuser element for zuulv2.5
(zuul-worker element). We also need to pass in the public
SSH key for nodepool.
Change-Id: Idda577e5cb210ca2018f4a9eb497457026e53cdb
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
- Created nodepool::mysql class, which includes mysql stuff from
init.pp. This allows to host database on separated host than
the one running nodepool.
- Discarded creation of 'max_connections.cnf' file. Instead, use
'max_connections' from '::mysql::server' config_hash/override_options.
Change-Id: Ic745875b69563ed3834e99533227ca3d1150156b
This allows a user to tweak nodepool-builder image workers over using
static settings.
Change-Id: I54c879e1607d960629a2ed64efddff99491b6440
Depends-On: I5268e0ca34ab91ffae7a203ca710eecf571585e5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Puppet tries to parse its own escape, so there is a warning:
Warning: Unrecognised escape sequence ( in file
Adding extra escape for escape fixes the warning, and stil does produce
desired cronscript.
Change-Id: I01fbd8d646820b2d2d71cf98be61a7b72198b234
'*.log' does not match filename postfixed with date, as most of them
are, e.g. dib.devstack-fedora21-dib.log.2016-01-11_14. Add an
additional filter to match that case.
The cleanup was added in Iee55717bc5fbf88d739a4124f62f76d193cf3b75.
Change-Id: I2a796ee1e9a9cb7f7ea83312227311c9515f09d1
When images are removed and no longer being built, their log files
hang around forever. Add a simple cron job to clean them up after 7
days, but hide it behind a default-off flag just in case people really
want their old logs.
Change-Id: Iee55717bc5fbf88d739a4124f62f76d193cf3b75
This is an intermediate step before builders are completely broken out
into their own hosts and nodes.
As part of this work we split up the logging configs for the two daemons
which allows them to be colocated and write to different log files on
the same host.
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>
Change-Id: I990e96dde352fb7d01cc4b89d0f4f02de166b943
This patch uses the load_module_metadata function to check whether the
mysql module has a metadata.json file. If it does, we can assume it's
new enough to support the new interface. If not, use the old one.
Change-Id: I3ff754b15eef51c3c86c188647353a4a1d3bfea0
Depends-On: Ia366c0f7f1bfbfa843071e733a5b0a96873a60d8
Nodepool depends on PyMySql and installs it from pip as part of its
requirements. Therefore it is no longer necessary to include the
mysql::python class, which installs the python-mysqldb package from
Ubuntu packages. Removing this class inclusion will make transitioning
to an up-to-date version of the mysql puppet module easier, as the
current version moves the mysql::python class to
mysql::bindings::python.
Change-Id: I6ac8d7433d4a0a2c591f718e3b4579b27de67e24
In anticipation of puppet 4, start trying to deal with puppet 4 things
that can be helpfully predicted by puppet lint plugins. Also fix lint
errors caught by the puppet-lint-absolute_classname-check check as well
as arrow alignment errors that were not being caught by the
system-installed version of puppet-lint before.
Change-Id: I4e6b0df76c0e1b1e9faa9e8884a78ab1e9ab38d7
The nodepool image log folder needs to be created even if
not accessible via http because it is being used in the
nodepool logging configuration. Otherwise, nodepool fails to start.
Change-Id: I7ed76bd89e1164a26cf7d7f43e19dab7b38ff14f
This patch makes the following changes to make the parameter defaults
more sensible:
- Makes $nodepool_ssh_private_key required. It is not sensible to
create a private key file that is empty, so we opt to fail fast in
puppet rather than confuse nodepool. The system-config repository
passes this value so this change will not affect Infra.
- Make $statsd_host default to undef. The nodepool.default template
where this value is used now needs to check for both nil and ""
because system-config passes "" directly to it and we want to
continue the same behavior.
- Make $scripts_dir and $elements_dir default to undef and update the
logic in the manifest to handle this.
Change-Id: Ic5e9af720c334d2cf2f7a9abe7c12fb81873086c
This will be a new file that will store all the secrets needed
by nodepool at this stage: mysql password, and jenkins masters
credentials.
Following there will be a patch in nodepool, to use that file
to retrieve mysql and jenkins settings. By this way,
nodepool.yaml can be a plain file, and not a template, and can
be created into project-config.
Change-Id: Ie9381740e3644feaee1f1b201499e3a253677f39
puppet-httpd is the openstack-infra version of puppetlabs-apache
(0.0.4) release.
This patchset will remove the puppetlabs-apache namespace from -infra
allowing for possible future patchsets to use newer puppetlabs-apache
modules.
Change-Id: Iedd42f9fb628f1fbf6c4916a5811fd02860f389f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Rather than having the diskimage-builder config sprinkled in here,
it's been split into its own module. Consume that module.
Depends-On: Idaac8b1e2ff3e3705bd50b76e48f5d12b743244c
Change-Id: I822ddd1041c1207a3d82d4864cf020d4db7ccc43
Clark Boylan