As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.
The actual usage of multiple ansible versions will be done in
follow-ups.
For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.
Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
Depends-On: https://review.openstack.org/623927
flake 3.6.0 introduces a couple of new tests, handle them in the zuul
base:
* Disable "W504 line break after binary operator", this is a new warning
with different coding style.
* Fix "F841 local variable 'e' is assigned to but never used"
* Fix "W605 invalid escape sequence" - use raw strings for regexes.
* Fix "F901 'raise NotImplemented' should be 'raise
NotImplementedError'"
* Ignore "E252 missing whitespace around parameter equals" since it
reports on parameters like:
def makeNewJobs(self, old_job, parent: Job=None):
Change "flake8: noqa" to "noqa" since "flake8: noqa" is a file level
noqa and gets ignored with flake 3.6.0 if it's not at beginning of line
- this results in many warnings for files ./zuul/driver/bubblewrap/__init__.py and
./zuul/cmd/migrate.py. Fix any issues there.
Change-Id: Ia79bbc8ac0cd8e4819f61bda0091f4398464c5dc
The console daemon has, thus far, primarily been tested on python2 for
various reasons.
However, when forcing it to python3 by setting
ansible_python_interpreter=/usr/bin/python3, we find that the console
daemon explodes because Python 3 does not allow unbuffered I/O on
non-binary files.
It should be fine to have /dev/null be binary mode, as it is just being
used to do low-level fileno() operations, the file ojbect is discarded
shortly thereafter.
Change-Id: Ib030863c2de17825e29874733fc5a9b023f7a601
There was an occurrence to fail_json where the message was put in via
a positional argument instead of the msg keyword.
Change-Id: Ie4b9935869fab01e598fd7f34a5245515152c09b
Currently when ansible has an error when trying to run a command or
shell task the command module has no chance to send any console
log. Thus zuul_stream doesn't terminate by itself and gets killed
after a timeout of 30s. As a try to fix this zuul_console sends
periodically a notice about the not found logfile. When requesting the
streamer to exit it can check then if there is still no console file
and exit by itself without needing to timeout.
Change-Id: I42bc05b0d2c530fbfc00c6295da24d18a6ec6435
It gets decoded as utf-8 on the other side of the socket, and it's being
written to the file as utf-8 - so there's not reason to read it in to a
string before sending it over the wire.
Change-Id: Iad3d33108835fad3ceae0eec38985659449a6452
This should look for the process holding open the port, and then delete
all of the remaining files.
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>
Change-Id: Iba3eda63e84c4357a121a1782b97a12232e1b8ce
Spawning a single reader at the top isn't actually working. In cases
where there are multiple playbooks per host (literally every zuul v3 job
given pre playbooks for git repos), the stamp file was preventing following
playbook from spawning a daemon, but the daemon was only persistent in
the context of a single playbook.
We can't just spawn a new one per playbook without some modifications,
as otherwise the existing already-streamed content would get streamed
again.
Grab the finger streaming code, which accepts an argument as to what to
stream, and re-use it in zuul_console. Combine this with adding a unique
id to each task. That way each task on a host will log to a distinct
logfile, and each callback will stream only that task's log output.
This allows us to join the reader as well, so that we won't get
streaming overlap across tasks.
Change-Id: Ic5eb6c38af698f4ba8b4504aa69170834ec4036a
We don't need zuul_log anymore. People can write ansible themselves,
which means they can write debug statements, which will go into the
collected log.
Also, put the port and the filename into constants so that they don't
seem like magic numbers as much.
Change-Id: I8020cc3e841617366831e80fe92fc477452d6da2
Now that osic-cloud1 is providing IPv6 only, we need to update
zuul_console to also include ipv6 support. Previously, our logic
would only bind to 1 address, even if the host supported both
ipv4/ipv6.
With help from fungi, we can simplify our logic binding to both ipv4
and ipv6 at the same time.
Change-Id: Ia0286e017f14eab77c5d60333ee092c87bb1a84b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Some of our tooling expects console.html; the easiest thing to do
at this point is just to call it that everywhere.
Change-Id: I2ac3d03bf7e263e4f30d6f0d94974f1699ec8f24
These were developed together in situ:
* Fix tailing the console log
* Change console log name from console.log to console.txt
(for better auto content typing)
* Expand JJB macros for builders and publishers
* Use a two-stage SCP copy (worker -> controller; controller -> site);
a one-stage copy is possible but will require installing a key
on the remote site
* Substitute parameters (eg $LOG_SERVER) into scp/ftp site paths
* Better worker logging (use the worker name in the logger name)
Change-Id: I98e5603f7a3210c1322640a66ecdeadb24ce74fe
Soon-to-be-released Ansible 2.1 can sometimes hate you for not
surrounding your call to main() with the proper conditional check.
Let's fix that.
Change-Id: If9eca7866d8d2057d200110186e03772cd77a8fb
Tobias Henkel