Commit Graph

91 Commits

Author SHA1 Message Date
Clark Boylan d82984ddf2 Retire this project
We've shutdown the logstash service and don't need to manage it with
puppet any more.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/839235
Change-Id: I2b5d3ac705d246f705b4ef91f12a4895e6854fc1
2022-04-25 09:50:21 -07:00
OpenDev Sysadmins 46f9ce10b7 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:26:03 +00:00
Ian Wienand 32e9f596f4 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I649dc632aeca673a6d09db3d3d426fdf16eaf984
2019-03-24 20:35:40 +00:00
Colleen Murphy ace749ebbc Update Gemfile for Zuulv3
The logic in the Gemfile was relying on Zuulv2 variables to find out
whether the spec helper gem was already available on disk, and since
Zuulv3 has changed things it was failing to find it and downloading the
master version instead. This patch ensures the Gemfile looks for the gem
in the right place when running in CI.

Change-Id: I6efe6cc34950e382656e39b579ff57566ed1e91a
2018-07-12 09:57:44 +02:00
Colleen Murphy b7032eb94b Depend on helper gem for spec_helper_acceptance
Instead of keeping a local copy of spec_helper_acceptance.rb and
requiring updates to all modules for any change, we can move it into the
common helper gem and require it from there. This will make it easier to
create and review changes that affect all puppet modules. Also change
the Gemfile to look for the gem in the local workspace if running in a
zuul environment.

Change-Id: I20bf542923065f7790993e29938c71bfc189e447
2017-08-18 10:41:43 +02:00
Jenkins 515357a301 Merge "Remove the mqtt options from logstash::indexer" 2017-08-02 18:20:16 +00:00
Colleen Murphy 38c66fc220 Remove unnecessary beaker tests
We do not need to check every package and file with beaker. It is safe
to assume that if puppet ran successfully then it also was able to find
and install every package and was able to apply every config file.

We still keep the service checks because it is possible for puppet to
declare a service to be running successfully only for it to die later.

Change-Id: I313fe7907b165e4c064cc37cf0c75051aabf103e
2017-06-14 14:38:27 +02:00
Colleen Murphy 399a2ca633 Parameterize JRE package
Add a params class so that puppet can distinguish between JRE packages
on different Ubuntu releases.

Change-Id: Ia1d22379d915d347c549c8cb8785920d97470f06
2017-06-14 14:38:27 +02:00
Colleen Murphy f3e089077f Update beaker setup for xenial
Add a xenial nodeset and update the spec helper to install puppet 3 from
the Ubuntu repos instead of from puppetlabs.

Change-Id: I2a590ac0d8eecfcd964cd7ffd1a6bdb540815cb9
2017-06-14 14:38:27 +02:00
Colleen Murphy 567ae88a96 Fix curator cron command string
The puppet syntax check complains about the '\%' in the cron command
string, but the % still does need to be escaped for the cron to work
properly. We double escape it here in the puppet string which will
result in a properly single-escaped command string in the crontab.

Change-Id: I3d97463a611c43ded12ddb788e9da5e13466e0c5
2017-06-14 00:32:56 +02:00
Matthew Treinish f183b53d30
Remove the mqtt options from logstash::indexer
The logstash::indexer no longer sets up the mqtt output plugin for
logstash. So we don't need to keep the options around to enable it and
write the create the cert file. This commit removes those pieces.

Change-Id: I6ebdb5dc8fea8309609d837a758eeaf1dc5ee760
Depends-On: I6becec12604c61fe50d3e6b9c7ed9f9e9be311ae
2017-05-03 15:58:57 -04:00
Matthew Treinish 2f689ee899
Remove logstash mqtt plugin install
The logstash mqtt plugin caused issues in production and we're not going
to use it. We'll be emitting mqtt notifications directly from the
gearman worker instead. So this commit removes the plugin install so we
can set enable_mqtt to true to install the cert.

Change-Id: I0a17444cce18dd4b63f1f924e393483f6d8fe8eb
2017-04-27 15:57:33 -04:00
Clark Boylan 0e3d046cd6 Update to latest 2.x release of logstash
Hopes are this fixes the bug where logstash stops processing events and
ends up stalled out requiring forceful stoppage. Has been running
successfully on logstash-worker01 for almost a day now.

Change-Id: I62f311c33d8f31f26baee887881d6b672b0f2916
2017-01-03 08:45:14 -08:00
Matthew Treinish ce51abae9c
Switch onlyif to unless
We want unless not onlyif so we actually run the install command

Change-Id: I7ae6b05e38e2234b51f6263134a77d305fa46648
2016-09-21 08:18:25 -04:00
Matthew Treinish 616156f9a1
Add mqtt output plugin to logstash indexers
This commit adds support for configuring the mqtt logstash output
plugin. This will enable us to push events from our logstash workers
to firehose.o.o. The plugin can be found here:

https://github.com/kompa3/logstash-output-mqtt (althought it's README
is from a template and you have to look at the inline docs for how
to use it)

and the gem is published on rubygems.org here:

https://rubygems.org/gems/logstash-output-mqtt/

Change-Id: I7255f6c256ab3e3ca33caf69b71bf4ffab02c1bb
2016-09-20 05:57:34 -04:00
Clark Boylan f671fcf2c9 Pin curator to 3.5.1
elasticsearch-curator 4.0 introduced completely new everything. It will
take some work to support it and all of the new features so pin to 3.5.1
for now. This will keep index deletes working while we add the support.

Change-Id: Iea4e8794d2b953cfdd59ff7907420dffb3de4dff
2016-07-08 15:23:52 -07:00
Spencer Krum 3a198f3a9d Use new infra_spec_helper for gem dependencies
Change-Id: Iccd29fa410ef5a3c16d8c055ea75bb384bc17466
2016-06-21 18:39:01 -07:00
Philip Marc Schwartz 38571dfae5 Update pip provider to use openstack_pip provider
Change-Id: Ib7a8740b7e7f83b1e93e54dc0e05b0d5e8ed61a9
Signed-off-by: Philip Marc Schwartz <philip@progmad.com>
2016-06-16 10:20:00 -04:00
Clark Boylan eb1fad4e80 Cleanup apache configuration
There is a puppet kibana module now that configures apache for us so we
don't need to do that here. Remove the redundant config.

Change-Id: I9e25c637b645f28b167bf248ee213608a1dfe615
Depends-On: Ic05f26d61f2ac4b5adffee98782d1a02931d5731
2016-05-26 10:41:47 -07:00
K Jonathan Harker c55836e555 Remove old config when switching to split configs
When we switch from a single config to split configs, make sure the old
config is removed. Leaving it in place prevents logstash from starting,
since we have multiple inputs attempting to listen on the same port.

Change-Id: I12e9da295e892ec70bcd9717fbdc0526b0822b86
2016-05-12 14:46:34 -07:00
Clark Boylan 049a6e2b87 Properly support logstash conf.d
Add in proper support for conf.d configs. Create input at 00-input,
output at 99-output, and filters at user specifiable levels in between
using a new defined type for filters.

Co-Authored-By: Jonathan Harker <code@gentlydownthe.net>
Change-Id: Icbca7a6ba0c5a94a273ef158f707311b588483fd
2016-04-29 14:15:29 -07:00
Jonathan Harker b91deb2b0c Fix beaker tests
Now that we install logstash 2.0, and its package manages service files
and user creation, update our beaker tests to reflect these changes.

Also, the logstash deb depends on the logrotate package, so make sure that's
installed.

Depends-On: Id50f98459b677460c28772c406c86482dcd0e9e4
Change-Id: I4c062c82d8d0aae5dd27036021a9464d788244d1
2016-04-29 12:41:02 -07:00
Spencer Krum ced76655f7 Pin google-api-client; sanitize Gemfile
Change-Id: I38f06d44ec57ea0802e94e5a909f6e41ab4566eb
2016-04-14 15:20:40 -07:00
Andrey Nikitin 04f3167e9a Order of the classes parameters is refactored
Order and intendation of those parameters are changed
to follow Puppet Style Guide recommendation [0].
Moreover, it will allow to an user to find much faster
a variable in a list of variables.

[0]. https://docs.puppetlabs.com/guides/style_guide.html

Change-Id: I4457f7c6dd3cfca24084a252f360c45d1aa2b2d7
2016-03-21 13:13:04 +03:00
Jenkins ee59b4dae7 Merge "Install logstash 2.0" 2016-03-10 22:23:42 +00:00
Clark Boylan f326d9af28 Install logstash 2.0
This makes a whole host of changes. First we install logstash from the
deb packages that are now available. This package sets up the logstash
user, group, logrotate for log files, a conf.d dir, and an init script
for a service for us. This means all of that stuff can be removed from
puppet and be provided by the package instead.

In addition to cleanup related to the package we remove things that no
longer make sense with newer logstash. There is no more logstash web
service for example use kibana instead. Also the distinction between
agent and indexer is no longer strong and logstash has a separate
shipper tool which should be used in place of the agent (though backward
compat for the agent class is preserved by setting it up as an indexer
using an agent config).

Depends-On: I2354fbe9d3ab25134c52bfe58f562dfdf9ff6786
Change-Id: If3fa423a80a03d09993e58f6d68a3a212b457468
2016-03-10 09:24:45 -08:00
K Jonathan Harker 03844fb02d Escape % in cron command
The % character needs to be escaped in cron commands as it has a special
meaning.

Change-Id: Iab028133276c8186458c3f80098c9ef2ae58e9a9
2015-12-09 08:47:52 -08:00
K Jonathan Harker 4a8f77c2fb Look for curator in both /usr/bin and /usr/local/bin
Change-Id: I7adacd984aa8175ad991e0b0e42765ddcb4b8306
2015-11-17 09:43:49 -08:00
Jenkins 9779295da6 Merge "Upgrade from kibana2 to kibana3" 2015-10-15 21:01:13 +00:00
K Jonathan Harker 5f9717b803 Upgrade from kibana2 to kibana3
Change-Id: I176148c3bdaf603a1703d215ac2086101ecabe21
Depends-On: I13f9dff0bbd6498a36dc75b026c9042a9bb05e8f
2015-10-14 14:02:19 -07:00
Jenkins 90ea634692 Merge "Create a class to install and run curator" 2015-10-01 07:47:22 +00:00
jesusaurus 3c904c866e Create a class to install and run curator
The curator project manages elasticsearch indices, with knowledge of the
naming format used by logstash and the ability to delete indicies
matching a given pattern.

Needed-By: Iba3c0726b789dcbcf58f2da071e6fa93d4cd67ae
Change-Id: Ic49c8649a0ad849a8d72900425e589e40b82a37f
2015-09-28 22:56:27 -07:00
Danilo Ramalho be01b8866c Add acceptance tests for puppet-logstash.
Add acceptance tests for puppet-logstash module so that once the module
is applied we check if files were created, packages were installed and
users configured.

Co-Authored-By: Bruno Tavares <btavare@thoughtworks.com>
Change-Id: I0332e9caf732ab8874cd444a9faa42143d40e335
2015-09-24 19:27:25 -03:00
Spencer Krum c5013c79c5 Fix target path for regular git clone during tests
Use same target directory for zuul-cloner and
the regular git command.

Change-Id: I5f7667a2bede9ed5eb3979103891587d3df1ae12
Co-Authored-By: Fabien Boucher <fabien.boucher@enovance.com>
2015-08-19 16:39:32 -07:00
Colleen Murphy 466d44326b Add Gemfile and puppet 4 checks
In anticipation of puppet 4, start trying to deal with puppet 4 things
that can be helpfully predicted by puppet lint plugins. Also fix lint
errors caught by the puppet-lint-absolute_classname-check gem and arrow
alignment errors not caught before.

Change-Id: Icb76cc5019c378524344640e9670229e142a9627
2015-08-04 13:18:30 -07:00
Spencer Krum ea531ba7f8 Boilerplate beaker-rspec files
Change-Id: I12789c34c18d3a4c7ff7ee76fca43f8dad7ca1b5
2015-07-28 02:04:56 -07:00
Paul Belanger 60d1e2b295 Migrate to puppet-httpd module
puppet-httpd is the openstack-infra version of puppetlabs-apache
(0.0.4) release.

This patchset will remove the puppetlabs-apache namespace from -infra
allowing for possible future patchsets to use newer puppetlabs-apache
modules.

Change-Id: Idff8228de0a5c309b02d95feb6e6f25aafaa5f2a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-07-16 15:38:11 -04:00
Jenkins db58618c89 Merge "Start logstash indexer service after networking" 2015-06-01 15:51:44 +00:00
Spencer Krum 0af7937e45 Flesh out the readme
Add boiler plate from puppet-httpd and some basic examples.

Change-Id: If2e303198e2f7f67253a729ef51e12b5a75085e5
2015-05-31 17:52:13 -04:00
Jenkins 584d0f3479 Merge "Logrotate the indexer.log logfile." 2015-05-15 01:23:55 +00:00
Jeremy Stanley 0c90ff0bb8 Replace ci.o.o links with docs.o.o/infra
The http://ci.openstack.org/ documentation site has been deprecated,
replaced by redirects to corresponding paths within
http://docs.openstack.org/infra/ where other Project Infrastructure
documentation already resides.

Change-Id: Ic9151804ffca5903969c44ca0465645b4d6fc1ea
2015-05-14 21:38:18 +00:00
Clark Boylan 79d4e6bbf6 Logrotate the indexer.log logfile.
Over time this logfile can grow quite large filling disk. Rotate it
daily and keep one week worth of logs to keep a reasonable amount of
logs.

Change-Id: Ic5f3acf290d3febc8af83773eb120df774937f1b
2015-05-14 10:31:51 -07:00
Jenkins b12ff0223a Merge "Add missing vhost_name" 2015-05-06 20:57:19 +00:00
Jenkins b825c78301 Merge "Modernize kibana vhost template" 2015-04-28 17:14:56 +00:00
Yolanda Robla 9d3ba41410 Add missing vhost_name
Add a missing parameter into apache inclusion,
that was causing vhost_name to don't be passed properly
and generating the vhost incorrectly.

Change-Id: Ia39c005fffb82b5f1d75ac704cf24a1a2a978d6d
2015-04-23 08:20:25 +01:00
Ramy Asselin 41318c5a84 Rename openstackci to openstackinfra
Change-Id: If6f1025758f85f0daee3e85761ca939e149e223d
2015-04-20 13:45:13 -07:00
Clark Boylan dae3541847 Start logstash indexer service after networking
We need networking to be up before the indexer service starts or it dies
a quiet death.

Change-Id: Ic32196c59f8c418c9a29ccdd35706f2d4c87eb67
2015-03-05 11:01:03 -08:00
Spencer Krum 14a6ec4c99 Modernize kibana vhost template
In the past it was common to do scope.lookupvar but now it is only
required when explicitly grabbing a different scope. All the
variables in kibana.vhost.erb are available from the class that
calls the template() function.

Change-Id: I100bb764b5742b1f3068e5b86813c825f9f71868
2015-02-11 21:54:00 -08:00
Spencer Krum f76780d6cb Locking down apache module version
We need puppetlabs-apache 0.0.4 for this module. We will, hopefully,
soon move this and other modules on to openstackci-httpd.

Change-Id: I27546066cb071eb942bd669434483c5bdd77fb24
2015-02-11 21:54:00 -08:00
Spencer Krum fd00522597 Enhance and refactor head-of-file docs
The strange looking syntax the head-of-file docs are in is in
compliance with the new puppet-strings system for inline documentation

This enables neato stuff like 'puppet string <modulename>' to get docs
on the command line, and generating html, which we could then ship to
somewhere.

Change-Id: Icb0be2c733e4d3ee6beff32780c171b9f750b06f
2015-02-11 21:53:56 -08:00