Commit Graph

26 Commits

Author SHA1 Message Date
Tobias Henkel cd9827e664
Manage ansible installations within zuul
As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.

The actual usage of multiple ansible versions will be done in
follow-ups.

For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.

Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
Depends-On: https://review.openstack.org/623927
2019-03-15 09:09:16 +01:00
Tobias Henkel 0ae7a157e2
Don't do live streaming in loops
Currently we do live streaming of command and shell tasks in
loops. However this is severely broken as we only get the output of
the first task that ran. This is hard to solve as a per task streaming
in loops is not possible because of missing start events. Further
streaming several tasks at once is difficult too because we don't know
how many tasks we need to stream upfront.

So the intermediate solution until the whole log streaming is reworked
is to just not live stream tasks in loops.

While we're at it also fix a broken log statement and remove duplicate
exit code prints of action tasks.

Change-Id: Ic1358d5b9f939549ffdd5d770fe4eafc047be6f1
2018-11-05 22:21:38 +01:00
Tobias Henkel bfcdff9118 Fix broken command tasks in handlers
With the update to Ansible 2.5 command and shell tasks used in
handlers are broken and fail with [1]. The reason for this is that the
callback v2_playbook_on_task_start is not called anymore for
handlers. Instead the callback v2_playbook_on_handler_task_start is
called for them. This leads to a missing zuul_log_id in the handler
task and trying to log to /tmp/console-None.log. In case this file was
already created by a handler using sudo it may not be accessible which
leads to the exception.

This can be fixed by also defining v2_playbook_on_handler_task_start
in zuul_stream.

Also add a validation of zuul_log_id in the command module. This
should make it easier to spot such errors next time.

[1] Trace
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/tmp/ansible_X_j_M1/ansible_module_command.py", line 185, in follow
    with Console(log_uuid) as console:
  File "/tmp/ansible_X_j_M1/ansible_module_command.py", line 162, in __enter__
    self.logfile = open(self.logfile_name, 'ab', buffering=0)
IOError: [Errno 13] Permission denied: '/tmp/console-None.log'

Change-Id: Ib9dd7fe09e4e7734f7a9ada876e6ce450ebc5038
Story: 2002528
Task: 22067
2018-06-12 08:53:59 -07:00
James E. Blair 896df11638 Revert callback fixes
We suspect there are more fixes required, and the additional check
these changes added may cause more harm in the interim.

This reverts commit 3512a5608c.
This reverts commit d94a0d6f06.

Change-Id: I6e874cd68a1cf6f974e36ab1870573bddf4e647b
2018-06-12 08:53:19 -07:00
Tobias Henkel d94a0d6f06
Fix broken command tasks in handlers
With the update to Ansible 2.5 command and shell tasks used in
handlers are broken and fail with [1]. The reason for this is that the
callback v2_playbook_on_task_start is not called anymore for
handlers. Instead the callback v2_playbook_on_handler_task_start is
called for them. This leads to a missing zuul_log_id in the handler
task and trying to log to /tmp/console-None.log. In case this file was
already created by a handler using sudo it may not be accessible which
leads to the exception.

This can be fixed by also defining v2_playbook_on_handler_task_start
in zuul_stream.

Also add a validation of zuul_log_id in the command module. This
should make it easier to spot such errors next time.

[1] Trace
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/tmp/ansible_X_j_M1/ansible_module_command.py", line 185, in follow
    with Console(log_uuid) as console:
  File "/tmp/ansible_X_j_M1/ansible_module_command.py", line 162, in __enter__
    self.logfile = open(self.logfile_name, 'ab', buffering=0)
IOError: [Errno 13] Permission denied: '/tmp/console-None.log'

Change-Id: I1978aa8faa488fec87406e1481d455d49731f867
Story: 2002528
Task: 22067
2018-06-12 12:30:49 +02:00
David Shrewsbury 7bced14e8c Update to Ansible 2.5
Our custom command.py Ansible module is updated to match the
version from 2.5, plus our additions.

strip_internal_keys() is moved within Ansible yet again.

Change-Id: Iab951c11b23a24757cf5334b36bc8f7d12e19db0
Depends-On: https://review.openstack.org/567007
2018-05-08 21:20:13 +02:00
David Shrewsbury 8391f8d16b Update to Ansible 2.4
This updates both the dependency to ansible 2.4 and also ports in the
needed changes to the command module.

Version 2.4.0 definitely does not work for us because YAML
hosts file parsing is broken, but 2.4.1 and greater should
be fine.

Change-Id: I63f72b45ecb9533eac5ba9eb0eef426beec905e3
2018-04-04 07:58:42 +00:00
Tristan Cacqueray b84197e5d7 ansible: honor command no_log module attribute
The Zuul implementation of the command module doesn't need to stream
the console when the no_log attribute is enabled.

Change-Id: I04856c194d815855ff098a1259db16d7e0098a65
2018-01-19 00:32:10 +00:00
James E. Blair d7f164293e Be explicit about byte and encoding in command module
This module reads output from a command (via a pipe) one line at
a time.  The only input we should receive from it is either:

* a byte string from the command output terminating with a \n
* a python "string" terminating with a \n
  * that means in python2, a bytestring with a \n
  * and in python3, a unicode string with a \n

For now, we only need to focus on python2 because we explicitly run
this code under python2, however, it's wise to be forward-compatible
with python3.

The error in the previous version of this code is to assume that the
value we read from the command was a unicode string which needed to
be encoded in order to be written to the log file.  That's incorrect;
what we receive from the command should already be encoded according
to the system locale.

This change no longer encodes the lines received from the command
(because they are always bytestrings, in python2 or python3, they
will follow the code path where no further encoding happens).  Of course,
if it turns out not to be encoded in utf-8, then zuul_stream is likely
going to bomb because it assumes everything it reads is utf-8, but
that's a different problem.  In practice, we have utf-8 or C locales
universally at the moment.

Finally, there is a bunch of explicit encoding and bytestring handling
added to this method.  That is mostly in service of the future codepath
under python3; elsewhere in this file we call ".addLine('[Zuul] ...')".
Under python2, that's a bytestring so no further work is necessary.  In
python3, that's a unicode string, so we need to encode it.

We should never hit the exception handler, however, if somehow we manage
to, it should at least be able to write some data to the log file which
approximates what it was given.

Change-Id: Iae2f3ee012d914454c335184a8ec7c7ecb924ec7
2017-09-05 16:51:56 -07:00
Tobias Henkel 542f948952 Check ret for None in zuul_run_command
The check 'if not ret' not only matches a missing return code but also
the value 0 which is actually a successful run. Thus successful
commands end with an error 'Something went horribly wrong during task
execution'. This can be fixed by explicitly checking for None.

Also adds a successful shell task to the test_playbook test case which
fails without this patch.

Change-Id: If1d0721574a82e247659ab0f865ae6acfe12a6be
2017-07-11 10:28:09 +02:00
Monty Taylor d635d2a31c
Make sure we always log the exit line
If there's an exception, we might not write the final lines to the log.
Put those in a finally block.

Change-Id: I0a294af3874c53543176a7bd9c4b89253717b83c
2017-07-10 15:01:29 -05:00
James E. Blair 3fabd44226 Fix py3 issue with command module
This caused us to fail to log lines when running commands on
localhost (because in that situation, we use python3).

Change-Id: If9f5cf6ed62cb24a6fc16ebf189d36a752fa66d5
2017-06-30 17:29:33 -07:00
Jenkins 547af8e8fd Merge "Write logfile as binary encoded utf-8" into feature/zuulv3 2017-06-30 20:33:02 +00:00
Monty Taylor 1a3bb28e47
Write logfile as binary encoded utf-8
Change-Id: I08e3e2fbfb629a6b68b80ddaadbef71038b3eda0
2017-06-30 15:17:44 -05:00
James E. Blair c9003740b8 Fix exception handler in command module
Change-Id: Ie7f860e25a533620b263d3f1f7f127605991fa11
2017-06-30 13:15:44 -07:00
Clint Byrum b159de7881 Revert "Revert sync from latest command from ansible"
It also fixes the bug of executing the command twice.

This reverts commit 1cfc946297.

Change-Id: Ia0b4f0b7d1bea5bba74f518a1b2cb7b9d84eba50
2017-06-15 17:21:16 -07:00
Monty Taylor 1cfc946297
Revert sync from latest command from ansible
These are causing WILDLY strange issues on ze01. Revert them until we
understand.

Revert "Update run_command to latest ansible"

This reverts commit 1dbf5f96b5.

Revert "Sync command from ansible"

This reverts commit 429428c020.

Change-Id: Ib481160312216a4dbc9c3184d31cbc35b5b36371
2017-06-15 18:30:40 -05:00
Jamie Lennox 1dbf5f96b5 Update run_command to latest ansible
Update the run_command from the basic module_utils that zuul overwrites.
This has a whole bunch of python3 changes that we can simply port in.
I've tried to just go around the zuul changes.

Change-Id: Ifca6b345f633add8e8bd136bd133ad62d1b92169
2017-06-14 22:00:08 +10:00
Jamie Lennox 429428c020 Sync command from ansible
Sync the base of the command module from the latest ansible. There are a
few fixes in here that relate to python 3.

Change-Id: I474c2dd82bb11c43c52e6bba0507539951579780
2017-06-14 22:00:06 +10:00
Monty Taylor b115358259
Make log streaming to executor per-task
Spawning a single reader at the top isn't actually working. In cases
where there are multiple playbooks per host (literally every zuul v3 job
given pre playbooks for git repos), the stamp file was preventing following
playbook from spawning a daemon, but the daemon was only persistent in
the context of a single playbook.

We can't just spawn a new one per playbook without some modifications,
as otherwise the existing already-streamed content would get streamed
again.

Grab the finger streaming code, which accepts an argument as to what to
stream, and re-use it in zuul_console. Combine this with adding a unique
id to each task. That way each task on a host will log to a distinct
logfile, and each callback will stream only that task's log output.

This allows us to join the reader as well, so that we won't get
streaming overlap across tasks.

Change-Id: Ic5eb6c38af698f4ba8b4504aa69170834ec4036a
2017-06-07 11:12:53 -05:00
Monty Taylor 9b2007a220
Also send stdout back in the Result object
It's a legitimate usecase in ansible to register the results of a shell
command and then test its value and make decisions. Not sending stdout
back in the result object would make playbooks that do that break.

There is a risk that we die under the load of really long log files
though - so let's keep track of that.

Change-Id: I2c2fe558d6ec93cef7bb4b1c5abd983488edd745
2017-04-27 16:22:38 -05:00
Monty Taylor d969430747
Minor cleanups in on-node logging
We don't need zuul_log anymore. People can write ansible themselves,
which means they can write debug statements, which will go into the
collected log.

Also, put the port and the filename into constants so that they don't
seem like magic numbers as much.

Change-Id: I8020cc3e841617366831e80fe92fc477452d6da2
2017-02-23 10:25:37 -05:00
James E. Blair 280cd8d062 Ansible launcher: don't close stdout in command module
If the job fails to close stdout/stderr, then this will cause
an exception.

Change-Id: I994c0a0c09bf2ec316ee7c7c51a18fa22372f152
2016-10-19 22:25:21 -07:00
James E. Blair fa3df0bf1f Ansible launcher: import get_exception in ansible command
The ansible command module was missing this import statement.

Change-Id: I456512a2a7f2138ac78eb8594c8808f664a9506f
2016-10-19 16:37:11 -07:00
Monty Taylor d1ddd284b8
Use command module instead of zuul_runner
Having a modified command module with the zuul_runner logic allows us to
use normal command and shell entries in the playbooks. (shell is just a
wrapper around command)

At this moment in time it's an invasive fork of the run_command method
on AnsibleModule. That's not optimal for long term, but should get us
closer to being able to discuss appropriate hook points with upstream
ansible.

Use environment task parameter instead of parameters

ansible has a structure for passing in environment variables which we
can use. We did not use it before due to a behavior in ansible from
pre-2.2 that set LANG settings in the environment in a way that caused
us to need to clean things in zuul_runner. The module_set_locale
variable defaults to False in 2.2, but to True in 2.1 (which was the
regression) Set the config value explcitly just to be sure.

Change-Id: Iae4769f923ecf74462e1fe43168ea93ff1c61d6e
2016-09-29 18:17:56 -05:00
Monty Taylor 331c3de4a7
Rename zuul_runner to command
In the next patch, we're going to change the body of zuul_runner. But,
in order to render that diff well, do the rename in this patch.

Change-Id: I3727f506cae5da561948869bd8f8daaf42e4dc0d
2016-09-29 18:17:56 -05:00