The opendev project has been moving away from puppet and this is one of
the puppet modules that is no longer used. To simplify things for us we
are taking the extra step of retiring this repo.
Change-Id: I83b9374c66f13d672800e76e8c10903f82479471
The builder ssl vhost template looks up vars as if they are local.
Problem is with futureparser and puppet4 these vars don't make it into
the global scope so they aren't valid here. Instead we do explicit
lookup of the values in the correct scope.
Change-Id: I92a6533d47075d8dac404999a6e3b5c4ccea84ed
Nodepool now uses openstacksdk in place of shade. Add a logger
for that.
Also, this makes the builder logging config look like the one in
system-config which is, accidentally, the actual logging config
being used right now. I'm going to put up another change to use
the logging config in this repo, which is the correct thing to do
for v3 of nodepool.
Change-Id: I5e6ab0a34e9e5df0ee152b33b01e5266691e6fc6
It seems the scope.lookupvar() doesn't work with non-global variables;
use the @ syntax. This is copied from puppet-cgit
Change-Id: I38eb50157d25c6c7900a8455eddedb4655c4637d
We wish to export the build logs and artifacts via https; not so much
for any security reasons but for transparent-proxy-busting effects.
Add SSL arguments and a template that redirects 80->443 for hosts if
they're passing in key contents.
Change-Id: I8a15333a7c662f3d32fa4045785498dafc87ad53
Let people download our qcow2 images to aid in local reproduction of
issues. We only serve the qcow2 files as they are smallest.
Note that this should probably be updated to use a negative match
against qcow2 so that we can ensure only qcow2 is served regardless of
what image formats we are using. Unfortuantely I can't get that to work
with autoindex. If you have ideas feel free to share them.
Change-Id: Ia23e68e87d2126eac2e22d9c3fed8406e2f447cb
There is currently a confusing array of exporting options for
exporting build and upload logs. The upload logs have never really
worked, and the info is better given from image-list end-points. The
build logs since Ia7415d2fbbb320f8eddc4e46c3a055414df5f997 are logged
into separate files into /var/log/nodepool/builds. With a separate
builder, it can not access the webapp ports provided by the launcher,
so the redirects there are wrong -- and we don't deploy apache on the
launcher to expose them. If you're using single node anyway, it's
more than likely you have zuul which installs the main website so
can't enable this.
For these reasons deprecate and remove these old settings.
Instead, add a separate flag to each of the builder and launcher.
builder::enable_build_log_via_http will deploy apache and a config
file that exposes /var/log/nodepool/builds
launcher::enable_webapp will deploy apache and redirects to the
internal webapp
Note this does not handle a launcher and builder on the same host;
since the webapp, and indeed refactoring this module in general for
multiple daemons, is under some development, we leave this as a future
task.
Depends-On: https://review.openstack.org/543667
Change-Id: I447886dd32f7f3bc6758ffd7a1b725689d04ee68
Both launcher and builder now use kazoo, we should at least log INFO
logging. This will help see what is going on when we loose connection
to zookeeper.
Also remove gear logger from launcher, it is no longer needed.
Change-Id: I7024e6f5994d8f79bae853a642ada3b4b52bf6e6
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
When configuring the dir for the pidfiles in our init scripts don't
assume the path is /var/run/$NAME. The defaults are now /var/run/nodepool
now and could be set to other values as well. Instead of assuming
/var/run/$NAME we instead take the dirname of $PIDFILE and configure
whatever that dir is instead.
Also expose zuulv3 flag to user to toggle the location of pidfile,
this changes in nodepool > 0.5.0.
Change-Id: I7fff2565f9e77d737c247f204925f66a0e4aabe8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Expose the ability to override our statsd_prefix, which defaults to
none.
Change-Id: I8bef6aae5a566ff3a34e816ece14f9cfc1ef289d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Move nodepool-launcher into its own class. This allows us to scale out
nodepool-launcher for our feature/zuulv3 effort.
Change-Id: I393fa1d8ff080260af772a2f020cca9b9e49b173
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Now that the feature/zuulv3 branch of nodepool has been merged into
master, remove deleted args from our default files.
Change-Id: Icd595ae3311fa8428b812292c8f40c290bde5e3b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Add image upload log directory and arguments. This is where provider
upload logs configured in Ic80e532891f039e0f835847ddfbae38f1ce3cd8aIf
can be stored.
Note http export (which we do not enable in infra at this time) would
put logs under /upload ; it might be more logical to move the build
logs to be a sibling at /build but that would be backwards
incompatbile.
Change-Id: I0cc9b59063fb14920f0da581b8036c2664d5d5f1
We've been running 3 nodepool services for a week now, lets properly
template them. nodepool uploads images, nodepool-launcher creates
nodes, nodepool-deleter deletes them.
This is mean to be a short term solution for zuulv25, and shouldn't
affect existing nodepool installations.
Change-Id: Iaee3148d6710f796e9f1ea7cba2962cfbb530fa2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
- Created nodepool::mysql class, which includes mysql stuff from
init.pp. This allows to host database on separated host than
the one running nodepool.
- Discarded creation of 'max_connections.cnf' file. Instead, use
'max_connections' from '::mysql::server' config_hash/override_options.
Change-Id: Ic745875b69563ed3834e99533227ca3d1150156b
This allows a user to tweak nodepool-builder image workers over using
static settings.
Change-Id: I54c879e1607d960629a2ed64efddff99491b6440
Depends-On: I5268e0ca34ab91ffae7a203ca710eecf571585e5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This is an intermediate step before builders are completely broken out
into their own hosts and nodes.
As part of this work we split up the logging configs for the two daemons
which allows them to be colocated and write to different log files on
the same host.
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>
Change-Id: I990e96dde352fb7d01cc4b89d0f4f02de166b943
This patch makes the following changes to make the parameter defaults
more sensible:
- Makes $nodepool_ssh_private_key required. It is not sensible to
create a private key file that is empty, so we opt to fail fast in
puppet rather than confuse nodepool. The system-config repository
passes this value so this change will not affect Infra.
- Make $statsd_host default to undef. The nodepool.default template
where this value is used now needs to check for both nil and ""
because system-config passes "" directly to it and we want to
continue the same behavior.
- Make $scripts_dir and $elements_dir default to undef and update the
logic in the manifest to handle this.
Change-Id: Ic5e9af720c334d2cf2f7a9abe7c12fb81873086c
Since all the variables accessed in this template are within the
current scope, there is no need to use scope.lookupvar here. Using it
adds complications, especially when passing undefined values. This
patch removes unnecessary uses of scope.lookupvar and replaces them
with instance variables.
This patch leaves the vhost template alone since the template is not
evaluated in the same scope. The nodepool.logging.conf template was
already properly using scoped instance variables.
Change-Id: Ie7d0a45905103c81d0c1822d89e2731e7395a3e3
This will be a new file that will store all the secrets needed
by nodepool at this stage: mysql password, and jenkins masters
credentials.
Following there will be a patch in nodepool, to use that file
to retrieve mysql and jenkins settings. By this way,
nodepool.yaml can be a plain file, and not a template, and can
be created into project-config.
Change-Id: Ie9381740e3644feaee1f1b201499e3a253677f39
puppet-httpd is the openstack-infra version of puppetlabs-apache
(0.0.4) release.
This patchset will remove the puppetlabs-apache namespace from -infra
allowing for possible future patchsets to use newer puppetlabs-apache
modules.
Change-Id: Iedd42f9fb628f1fbf6c4916a5811fd02860f389f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
According to https://wiki.apache.org/httpd/CommonMisconfigurations
is is wrong to specify server name in opening tag. ServerName
should be used instead.
Indentations also fixed in some files.
Change-Id: Id9d20a672103221efa01be61a174b62706036e57
Individual nodepool .log files have now grown to being > 250mb. This
is a bit much to download and go through when you're trying to debug
an issue with a particular image.
8-hour rotations seems a bit more useful than 24-hour ... and perhaps
even that is a bit long. I don't think we need to extended the number
of rotations, I'm not sure really old logs are that helpful.
Change-Id: Ia788db809afe470bb9241f17411e4fa97d09ea11
In Ie0b269835ebb8effbac0285b782d8add7b47db32 I didn't consider how
difficult it is to get puppet to put in AllowOverride directives to
allow .htaccess to work. The puppetlabs-apache version we use does
not support "override" as an argument, so you can't easily add this to
allow apache to read htaccess. Upgrading is a big task because we are
so far behind (see I6fa5f3275a84ec4480169de562d1d4a656111814).
So deploy a full template for nodepool vhost that includes the config
options we need.
Additionally, the existing mimetype isn't set for the log-files
---
$ curl -I http://nodepool.openstack.org/image.log.2014-10-14
HTTP/1.1 200 OK
Date: Fri, 17 Oct 2014 00:02:39 GMT
Server: Apache/2.2.22 (Ubuntu)
Last-Modified: Tue, 14 Oct 2014 23:59:56 GMT
ETag: "f89bb-d458315-5056acfe33700"
Accept-Ranges: bytes
Content-Length: 222659349
---
I'm pretty sure this means it defaults to text/plain, but this makes
it explicit. This should also matches on the timestampped log files
per the rules of multiple extensions [1]
[1] http://httpd.apache.org/docs/2.2/mod/mod_mime.html#multipleext
Change-Id: I7fa2603f4160b06af71a515e655d4a705fa0c768
The patch to add nodepool to jenkins-dev (https://review.openstack.org/#/c/57333)
did not work.
There were a few issues with it:
1. jenkins-dev.pp was passing literal strings to the nodepool module, instead it
should be passing in the variables.
2. jenkins-dev.pp was calling ::nodepool but puppet seems to think that it wants
::openstack_project::nodepool due to puppet's scoping weirdness :(
3. The script to build nodepool machines needed the jenkins_dev_ssh_key.
Fixes to above issues:
1. This is trivial, just passed the variables thru instead of literal strings.
2. The nodepool.pp module is renamed to nodepool_prod.pp to prevent the scoping problem.
3. We use the dev jenkins ssh key with dev nodepool by allowing the nodepool module
to pass arbitrary env settings through the defaults file.
Change-Id: Id91053212f088079ff1b0f06ebdce5c381f5cd19
Nodepool logs the creation of nodepool images to an image.log which
are only accessible to root users. This change updates the nodepool
puppet configuration to install apache and allows users to access
these image logs.
Change-Id: I867030c258d00ce017c69812c133f3419215d045
Clark Boylan