Commit Graph

131 Commits

Author SHA1 Message Date
Ian Wienand e5e1127122 Add delete to synchronize
Since I53cb63ffa4ae50575d4fa37b24323ad13ec1bac3 some puppet runs are
failing because system-config:playbooks/roles/set-hostname moved to
roles/set-hostname and became a symlink.  rsync refuses to overwrite
the directory with a symlink because it seems to not be empty, because
we never removed the files in it.

Handle the possibility of things moving around like this by
specifying delete so old files are cleared out.

Finally set manifest_base dir ownership and mode to address linter E208.

Depends-On: https://review.opendev.org/762181
Change-Id: I249d713214d8383a3ccfe1a400404cbdcb97dcc6
2020-11-10 09:48:56 -08:00
Monty Taylor eafdbf9593 Add flag to control logging the rsyncs
rsyncing the puppet modules can be super chatty and not super
interesting. Add a flag to turn it off.

Change-Id: I7206955aeb335adafb8625c89d8b8a70078d666c
2020-04-30 08:45:59 -05:00
Monty Taylor 0505f79daa Add color parameter
When we're logging inside of ansible, turning off ansi colors can
be nice. Especially inside of Zuul.

Change-Id: I719ebb106fbb31795cc02c9504507bdae607bcdd
2020-04-21 16:04:48 -05:00
Zuul 7570d49d8e Merge "Revert "Don't log puppot module sync"" 2019-07-18 16:37:41 +00:00
Ian Wienand e9fc2df1bc Clear out old facts
In debugging why I631e3e3592a9be5c7da642cf34eec6d9758bf161 was not
rolling out to the zuul executors, I eventually found
/etc/facter/facts.d/ansible.json was present on a number of hosts and
effectively pinning the project-config checkout to an old reference.

Given the dates around file creation and the pinned change number, it
seems very likely something happened during the openstack->opendev
transition where some hosts had ansible run with a specific
project_config_ref set as a fact.  Although the fact is now unset,
nothing will ever clear the old ansible.json which keeps applying.

We could remove all stale files by hand; but a) it's not clear where
it ran (although likely only hosts starting with z*) and b) we leave a
chance for it to happen again.

We could clear this after every puppet run; but it seems like this
role should leave the system largely in the state it ran puppet in.

Thus this clears out the file before the puppet run if no facts are
given.  This should fix our immediate issue on all hosts.

Change-Id: I96adf4fb77f06b18fe73b6f10d1b07b7fe44385f
2019-05-30 07:35:21 +10:00
Andreas Jaeger 7a9fd23801 Fix ansible-lint
The current tests never fail, change ansible-lint invocation so that it
fails - and fix problems.

Change-Id: I6aeb1e3a4c28af71a858ec83c21f4dbb5e7ef97f
2019-05-13 18:30:12 +02:00
Andreas Jaeger e42e0856b5 Switch to linters job
Prepare deprecation of ansible-lint job and use linters job instead.
Create a new tox environment for this.

Add job to run linters job.

Needed-By: https://review.opendev.org/658682
Change-Id: Ib2c7bdc66077442993d017cf97a4a8116799d130
2019-05-12 20:20:42 +02:00
OpenDev Sysadmins 734695ffab OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:25:37 +00:00
Colleen Murphy fe5255b56f Fix hieradata path for puppet 4
The datadir symlink points to /opt/system-config, but the private
hieradata actually lies within /opt/system-config/hieradata.

Change-Id: Iefedf1d733701c7abebd5022ca98993c5fcbd8a3
2019-02-03 15:45:20 +01:00
Colleen Murphy ac51f0245b Revert "Don't log puppot module sync"
This reverts commit 7b47277e3e.

It turns out that for some reason this boring operation does actually go
wrong[1], and if logging doesn't happen then we don't know why.

[1] http://logs.openstack.org/00/616000/6/check/infra-puppet-apply-4-ubuntu-xenial/7379d8c/applytest/puppetapplytest44.final.out.FAILED

Change-Id: I2ce9c3d79385fc0a01b3a5fc85fa4bf4128c25bc
2019-01-03 15:09:50 +01:00
Clark Boylan 98b7ea710b Fix ansible block, always, when syntax
Newer ansible (2.7) is far more picky about always blocks being a part
of block blocks. This means you cannot have a set of when conditions
apply to a standalone always block. Fix our use of this by putting our
always block in the run puppet block then move the always tasks into a
block of their own with a condition set.

Change-Id: I50988b6b312e4d00b73ca4454e0420913d4ae181
2018-09-12 13:08:28 -07:00
James E. Blair 7b47277e3e Don't log puppot module sync
This is boring.  The output is voluminous.

Change-Id: Ib905311a2b3c4eb9a0316f3863aa688b9ed07c55
2018-08-17 14:02:48 -07:00
Zuul 3bf4b22034 Merge "Make sure target manifest directory exists" 2018-08-17 20:44:43 +00:00
Zuul a0ce4c7317 Merge "Allow overiding the remote hieradata link dest" 2018-08-17 20:44:43 +00:00
Monty Taylor ffbb43a218
Make sure target manifest directory exists
On a bare system, bridge.o.o will be trying to rsync /opt/system-config/
to /opt/system-config/production - but if /opt/system-config doesn't
exist on the remote host, the rsync will fail. Make sure we create the
directory.

Change-Id: Id56a537fe2352c099b81559eb4032138ba108bed
2018-08-17 13:58:25 -05:00
Monty Taylor 7fc585e1ab
Allow overiding the remote hieradata link dest
In order to support the layouts being different on the management server
and the remote nodes, we need to also be able to override the location
of the link that gets made. Add a variable that defaults to the old
value.

Change-Id: I7434c1b1a9d79f04b3e872e750057d69490e80e0
2018-08-17 13:43:46 -05:00
Zuul 6e8b8a56b9 Merge "Add ability to override mgmt manifest path" 2018-08-17 17:45:44 +00:00
Monty Taylor 0a3c32b5ac
Add ability to override mgmt manifest path
We have it currently hardcoded that the manifests on the management
server are going to have an environment path. Add a variable to allow
override of that idea.

Change-Id: I226c030332ff93c3ee74650c7ab725994971f007
Needed-By: https://review.openstack.org/592763
2018-08-17 05:21:53 -05:00
Zuul 05fb2ce4e4 Merge "Allow setting puppet_version explicitly" 2018-08-16 16:22:01 +00:00
Monty Taylor d8a1943081
Change include_tasks back to include
The ansible on puppetmaster is too old for this.

Change-Id: I49458188e5ddd61bba95c9eb28792364008389c6
2018-08-15 18:32:38 -05:00
Monty Taylor 745913a3c2
Allow setting puppet_version explicitly
If we set puppet_version as an inventory variable, then we can drive
behavior from that variable, including updating puppet. Add the ability
to set it here. A followup change in system-config will start setting
it, but starting off it should be a no-op.

Change-Id: I773ba3a2137b62ae2b3440dd02b3cdfb2099a62d
2018-08-15 14:24:33 -05:00
Monty Taylor 92d5d596b9
Update yaml style to be consistent with infra yaml
Infra has adopted the approach of indenting lists an extra level
instead of starting their '-' at the same level as the parent element.
Additionally, Infra uses .yaml extensions instead of .yml extensions.
This role predates Infra having enough ansible to have opinions, so
update it to match current practice.

Also, while we're in there, update when clauses that have an 'and' in
them to just use the list form of when, and change 'include:' to
'include_tasks:'.

Change-Id: Icbeaf99c4d103091ee094e2fa219d7e16229b998
2018-08-09 09:04:06 -05:00
Monty Taylor 57873cfa78
Allow explicit override for mgmt_hieradata
ansible-role-puppet attempts to infer where it should copy hieradata
from based on puppet3 or puppet4. However, for hosts that are not
running puppet themselves this isn't going to work very well. Allow a
user to set mgmt_hieradata explicity.

Change-Id: Ie533da9af71ab84ce6e95dc63ceb8426740f53a3
2018-08-07 17:23:43 -05:00
Monty Taylor 24eb60b34d
Reorganize hieradata to match ansible variables
We are already organizing our hiera content basically the same way as
ansible needs for variables to be provided. If we reorganize the
directories slightly (to be coordinated with dirs on puppetmaster)
then we can have a single directory do double-duty.

Change-Id: I6ac90a7439ed8a5d9433d9526f37e44668b360ff
2018-08-02 13:11:29 -05:00
Colleen Murphy f0cd27f060 Add parser = future config option to puppet.conf
Enable testing puppet 3 with the future parser[1] by adding the boolean
parameter 'futureparser' to the role, which will turn on `parser =
future` in puppet.conf if set.

[1] https://docs.puppet.com/puppet/3.8/experiments_future.html

Change-Id: Ia65948ba954457d873758ed08502e844151f5cf5
2018-06-05 22:38:48 +02:00
Clark Boylan 7fbe25598b Fix hieradata lookup paths
This requires we get the magical symlink between hieradata in
/opt/system-config to line up with the actual on disk environment
specific hiera location which differs between puppet 3 and 4.

Change-Id: Id4d7a0af72efddb758d94c6437bb1c331d12f406
2018-04-25 12:25:59 -07:00
Clark Boylan c95f026c14 Add some missing /'s and +'s to fix hieradata pathing
These were missed when the puppet4 support was added.

Change-Id: I7f7153093aaf46d5bbdb620228cf1c7fce3650e1
2018-04-25 11:32:57 -07:00
Zuul d5dbfc3a62 Merge "Don't hardcode puppet-3-specific config paths" 2018-04-25 17:43:26 +00:00
Colleen Murphy 4d64868882 Don't hardcode puppet-3-specific config paths
This patch ensures that hiera data and puppet modules, and puppet config
are copied to the right directory depending on the current puppet
version. Since it's possible for the ansible management server and the
managed nodes to have different puppet versions, we need to account for
the possibility that the source and destination paths might be
different. We also don't need to hardcode the various config paths in
config or manage environments since we're using the defaults and
hardcoding them would make them incorrect for one or the other puppet
versions.

Change-Id: I164f91f9a7942e8c5f059652634ec1078ae41aae
2018-04-22 00:13:21 +02:00
Paul Belanger 6adb0c7632
Add bindep.txt file
Bindep is a tool for checking the presence of binary packages needed
to use an application / library. It started life as a way to make it
easier to set up a development environment for OpenStack projects.

Change-Id: If95270b9fac64fd21d279954eecc3e627be2f078
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-20 14:34:02 -04:00
Paul Belanger edb0e245d5 Expose puppet_timeout variable
It is possible we want our puppet role to run longer then 30mins,
expose the ability for a user to change it.

Change-Id: Id42ba80a5118a9f93e45619ac6ecc5baa774549a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-01-12 14:14:56 -05:00
James E. Blair 8d0f0bfd0a Set facts type to dict
With newer versions of ansible, module arguments are assumed to
be strings unless otherwise specified.  Our 'facts' argument is
expected to be a dictionary, so tell ansible that.

Without this, the argument will arrive as a string and be written
to the facter file inside string quotes.  Facter will produce the
following error:

  undefined method `each' for #<String:0x000000016ee640>

Change-Id: Iea626466d0920349f434f678fe2f71cbee259383
2017-01-12 09:55:37 -08:00
Jenkins 3fe078c070 Merge "Update README with info about puppet apply" 2016-10-21 15:16:58 +00:00
Monty Taylor d666d6c0e6 Update README with info about puppet apply
The README currently says you have to pass puppetmaster, but this is
untrue for puppet apply installs. Add an example playbook to run puppet
without a puppet master.

Change-Id: I6dfb5b97cbca42b83f790310b12a3d49f7dae8a0
2016-10-20 15:48:40 -07:00
James E. Blair 117fd2ed43 Revert "Remove basemodulepath from environment config"
This reverts commit e7a06ed55a.

This caused the ansible apply test to fail (and therefore
potentially puppet runs from our puppetmaster).

Change-Id: I0db01d128258f78e9b196354799a80774f6c6808
2016-10-04 22:36:16 +00:00
Monty Taylor e7a06ed55a
Remove basemodulepath from environment config
We want the env specific modules to be used first then fall back onto
the normal base module path for any modules missing from the
environment. This means you can test local changes without having a
complete set of modules in your env.

Change-Id: I2f0311400a8cf76b9793f50453e818ed4dd7e5a2
2016-10-01 08:11:13 -07:00
Monty Taylor 1c69c305e7
Sync puppet module changes from upstream ansible
We keep a local copy of this file for operational purposes, but the
upstream one has grown a few things. Sync to make it easier to submit
our local changes upstream.

Change-Id: I84927750fce1a7c6101e8d5c131c0b13020bb6aa
2016-10-01 08:11:13 -07:00
James E. Blair e8632f41a4
Don't set detailed-exitcodes when noop
Change-Id: I3819ad5920a0a88b0b52ecb9b77a58c6d588fbdc
2016-10-01 08:11:13 -07:00
James E. Blair fccf47a713
Add debug argument
To run puppet in debug mode.

Change-Id: I00baef254db2f9eaee23fb9eacded8efc5a2afc4
2016-10-01 08:11:13 -07:00
Clark Boylan 8cc1cd9126
Simplify puppet cmd generation
We were copy pastaing code in the puppet cmd generation for noop and
environment handling. Collapse that to make code easier to read and
hopefully avoid future bugs where only one side is edited.

Local testing shows that puppet parses commands like:

  puppet apply test.pp --noop

just fine.

Change-Id: Ie7665f72b9327b6b834d358699addf2c60a95ec0
2016-10-01 08:11:12 -07:00
James E. Blair b4f591b72d
Add noop argument
check_mode causes other things that we may want to actually happen
(such as setting up hiera) not to run.  This way we can do all of
those but still run puppet in noop.

Change-Id: I32f07a032d28f5c5d04e9eeb50b94f7c0a57ce53
2016-10-01 08:11:12 -07:00
James E. Blair 1a20bc89eb
Allow any logdest
Puppet's logdest option can also point to a file, so allow any
string through here.

Change-Id: I36fc99fa7e6169463acd24a5a6f4b54cf23849b7
2016-10-01 08:11:12 -07:00
Monty Taylor e46f449ca5
Pass environment to puppet module
Change-Id: I6be88234eab32844f395df7658952b1cef944bd8
2016-10-01 08:11:12 -07:00
Monty Taylor a3cc1c5c9a
Make sure environment winds up in basemodulepath
Change-Id: I7a80158fef3fd764000f7d57a56ebf0df83347ea
2016-10-01 08:11:12 -07:00
Jenkins ede55a8691 Merge "Honor show-diff options with puppet apply" 2016-08-05 20:03:36 +00:00
Jenkins 77bde8611e Merge "Remove connection: localhost" 2016-07-25 19:27:16 +00:00
Jeremy Stanley 10e657d56d Correct AnsibleModule string type names to str
The valid type names can be found in
AnsibleModule._CHECK_ARGUMENT_TYPES_DISPATCHER, and any deviation
from those causes us to no longer be able to puppet any hosts. The
type name for strings is, courtesy of Ansible's implementation being
Python, "str".

Change-Id: Ia4d4cde9926f7506a864739b1b616438c5fe6f43
2016-07-23 22:53:06 +00:00
Ramy Asselin 92737dfe27 Honor show-diff options with puppet apply
puppet apply ignores the show-diff option.
Update the code so that both puppet apply and puppet agent
support the show-diff option.

Change-Id: I5b59b8ba8e50a44458d01ec97113a2ce313c1d9a
2016-07-19 09:18:39 -07:00
Paul Belanger b8e3bdd1fd
Be explicit with AnsibleModule types
When moving to ansible 2.1.0.0, we'll need to be more explicity about
our AnsibleModule parameters. Ansible now defaults them to strings.

This will ensure groups is a list.

Change-Id: Ic126c200a432277c0ef406d4406f527d6a3c9fa7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-07-18 14:35:12 -04:00
Jenkins b5fc21ca37 Merge "Call main() from a conditional" 2016-06-06 13:02:06 +00:00