Change I316e9587b6e290cd421b47f506c91dbebe0975c0 had a rather
embarrasing oversight in that it copied the /usr/bindep-env/bin/pip
invocation for upgrading pip to all the other venv's.
i.e. we were upgrading hte bindep-env pip over and over, and not
actually the pip in the working venv. The os-testr install on older
platforms has now broken because it still tries to install with the
ancient inbuilt pip -- local testing has confirmed that it works with
the updated pip.
Change-Id: I22c549b5f9b9e3882fcd2340946d2850b0b2f86b
The pip installed in the venv with "python3 -m venv" on Xenial is 8.X
-- this does not understand python_requires metadata on packages and
can thus pull in requirements that won't actually run inside the
virtualenv.
Avoid this by upgrading pip in the venv before installing.
While this is the immediate need; do the same on the other venv's we
create for general sanity.
Change-Id: I316e9587b6e290cd421b47f506c91dbebe0975c0
This reverts commit 6f992efbc5.
Setuptools 50.1.0 has been released which reverts the breaking behavior.
https://review.opendev.org/#/c/749766/ tests that these fixes work in
other venvs that exhibited the same problems. That change looks happy so
I think this revert is ready to go.
Change-Id: I31b62be4f85f40f4d99e463cd961dec0a3542f47
Since all platforms have Python 3, use the new ensure-venv element
from the dependent change to install bindep and os-testr.
Since we are no longer using pip to install anything during the
builder, this drops the dependency on pip-and-virtualenv from
nodepool-base. Avoiding this element is our long-term goal, as it's
modification to system state are problematic in a number of ways. To
maintain the status-quo, the pip-and-virtualenv element is added
explicitly to each build's element list, with a note on it's future.
The current plan for backwards compatability is to replicate the
environment pip-and-virtualenv provides in a base role/job that can be
optionally included. To test this, provide a new node type
"ubuntu-bionic-plain" that will not include the pip-and-virtualenv
element. This is put on just one provider (rax) to minimise impact.
The dependent-change (and a dib release) is required before merge so
the ensure-venv element is available.
Depends-On: https://review.opendev.org/707513
Change-Id: I85438baf5bb31790a56fe5b38327361f0a2398e9
--seeder=pip is breaking images where pip is not installed from source.
New virtualenv upstream release 20.0.2 has fixed the issue seen when
using sudo [1] by copying instead of symlinking and does not need this
fix anymore.
[1] f4fd6a0991
This reverts commit be9530ae16.
Change-Id: I799982d9c023141cf612901084d4ecbd4447e969
New virtualenv will by default use a common location for seed libs like
setuptools, wheel, and pip. Unfortunately this breaks if root installs
the virtualenv and other users are expected to use it because these
other users cannot access /root/.local (where the files are stashed).
We fix this by using --seeder=pip which will install all of those libs
into the virtualenv itself.
Change-Id: I4922ea50e31dceda96f545a0d409c0d7dc022e19
On platforms such as CentOS 8 which are python3 first, "virtualenv"
and "pip" may not exist (removed to avoid any confusion over them
being v2 or v3 commands).
The referenced dib change exports new variables that should be correct
on all platforms for creating virtualenv's and pip installs.
(note will require DIB release to be active on builders)
Depends-On: https://review.opendev.org/684462
Change-Id: I3414fb9e503f94ff744b560eff9ec0f4afdbb50e
In openSUSE Tumbleweed, the SuSEfirewall2 package was removed in favor
of firewalld[1]. This commit updates the openSUSE nodeset to use plain
iptables rather than injecting iptables rules into the SuSEfirewall2
service. This will work on both Tumbleweed and Leap nodesets.
openSUSE provides no iptables-service package the way the RHEL family
does, so we can't fall back to that. Rather than try to convert iptables
rules to firewalld syntax, this change leverages init.d to ensure
iptables rules are loaded at boot. The 89-unbound script has been
coopted for this purpose since it already creates
/etc/init.d/boot.local. Switched from `dd` to `cat` which makes
conditionally composing the file more natural.
[1] https://lists.opensuse.org/opensuse-factory/2019-01/msg00490.html
Change-Id: Ia2b72e25078efa68019f1bf7c7a0b77e6ff702fd
Add rhel7 distrib in unbound and iptables script for nodepool-base, and
add haveged installation for infra-package-needs to ensure users could
use these elements to build rhel based image.
Change-Id: Ib0ad877369bafab64a1fd25cc331363d771d5753
This is applying the iptables rules during the build on the builder.
This will need to be converted into a script to run on boot or
similar.
Change-Id: Icc753013f218c3e4f04031c2fdbc7b32a534d887
There is an issue with newer kernels (it seems to happen with 4.15.x)
that when conntrack is reloaded while a connection has packets in
flight, this connection going forward is neither considered INVALID
nor RELATED nor ESTABLISHED by the stateful tracking. While this is
certainly a bug somewhere in the kernel, we can be easily avoiding
this by just not using stateful filtering for ssh connections, as
we accept any connection from anywhere anyway.
Change-Id: I1b20644ce888930cd28d6eaf2c23787315e8199c
We do not use the jenkins user anymore, so can remove the sudo settings
that allows jenkins to become root.
Change-Id: I69d58c2d10a2f48406cf2991fe83ce0e64851c8c
Remove jenkins-sudo-grep and use zuul-sudo-grep instead in the only user
which is run-tox.sh.
This allows to cleanup the legacy pre.yaml file in openstack-zuul-jobs.
that file replaces already jenkins-sudo-grep with zuul-sudo-grep.
Remove also now unneeded jenkins-sudo-grep settings.
Change-Id: I73e57eef72b1e5a0ec7be9e57160b24c6076e710
Needed-By: https://review.openstack.org/538344
We've updated the fetch-zuul-cloner role to properly create the
virtualenv and pyyaml dependency now.
This reverts commit 132cdf30d5.
Change-Id: I3fee0144eef4adeeb2b1f905c1f58c5bb4a6e554
Depends-On: I3e55a3a1582aa7dea21c7de67260c2c906c2192b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This reverts commit b770db0c21.
Turns out the zuul cloner shim in v3 does run out of the virtualenv's
python afterall. It appears to do this because it needs pyyaml
installed.
Change-Id: Idfa7a54e7c54b9068dc3d8ca9ff820c7aa47e8b4
Now that we have decided not to rollback to zuulv2, we can start
removing legacy code from our images. In this case, there is no more
need to be using zuul-cloner.
Change-Id: I6138482cb9a787792c09c44bb6235797cc046262
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
When we run heat api services with apache+uwsgi(mod_proxy_uwsgi),
we would need the services to be reachable from the neutron public
net on port 80.
Change-Id: I7c227b3fe580b2a60aa50a043200f4e4f270a5a3
Break the dependency on openstack-repos as we are considering removing
this element for zuulv3. Also, this is a requirement if for us maybe
building our project-config nodepool images in the gate.
Change-Id: I3bddca68385dc6c765fefb01e3042a48c4df5b3d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Move non-puppet-related parts of the puppet element into other elements,
remove the puppet element, and stop running the prepare-node script.
\o/
Change-Id: Ib241ea976ca0a3d661599f36e3f5a2d4eb023c08
puppet-iptables was creating /etc/iptables/rules and linking it to
rules.v4. We have removed this with
I03db58441674a3f3eea86165c949a7d14425a0b7 so this can simply make
rules.v4 and rules.v6 files, as required by
[netfilter|iptables]-persistent
Change-Id: I7e65793ba7513ac54ae701cc29c7c5ff6e33410d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
openSUSE Leap uses its own firewall manager called SuSEfirewall2, which
is capable of loading custom iptables rules. This patch adds the
necessary configuration to tell SuSEfirewall2 where to look for custom
firewall rules so that we can manage openSUSE firewall rules in the same
way we manage firewall rules for other images.
Change-Id: Ifaebda6c7775244668710340831e12aabf9e86bc
Instead of managing firewall rules with puppet we can statically copy
them into the image at build time.
Change-Id: I3ee306e46747b77499ff8975cd3d842b09ec2937
Switch to how we setup sudoers files, like we did in the zuul-worker
element. This stops us from possibility affecting the permissions on
/etc/sudoers.d with the install-static element.
Change-Id: Idaf2f4c582f333fd9acf4e4a08e5ade6fba61947
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Because we also check sudoers files in our zuul-worker element, we
need to make sure our permissions are set in install.d not
postinstall.d.
This was also not an issue locally, since our puppet master seems to
chmod 0755 our elements directory on nodepool-builders.
Change-Id: Ibcfd2741263889ef2dce8e04237537a6d83de301
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Part of our effort to migrate from puppet to dib elements.
Change-Id: Ie1b4a29a7496c862c14417344b8d51aef779e104
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Remove the unused builders and publishers for zuul-swift, remove unused
script to upload, and remove unused swift-logs nodepool element.
Change-Id: Idfa83d435a951dcfe6f6545a2515aec281ed43cc
This seems to have been here since the beginning of time (with time
being the project-config split).
However, since then, Openstack_project::Slave_common as acquired the
ability to install this same environment
(I290a695c697fb456bee6f8212ba50b6c1b4533fc).
The difference was that this file installed zuul from git, while
puppet was installing from pip. However, that changed in
If07b31f3a735cf7bcf6bfb8591ed37577f5ae201 and puppet is now installing
zuul from git. Thus since this now duplicates what's happening in
puppet, I do not believe want or need this.
This element is now breaking the Fedora 24 build when it tries to
overwrite the existing virtualenv already created by puppet.
python::virtualenv creates the virtualenv with a "-p python" argument,
while the second call doesn't, which creates issues with the symlink
layout (see [1] for details).
[1] https://github.com/pypa/virtualenv/issues/976
Change-Id: I7963630c699eaa4984adc6a155bea8f74280cd80
The workaround in my custom kernel (from
Iafe6d88e3ac7a2ea23553a5011df920a2ee3317d and
I0769f005da1931658a5fb9e627983ed30c11d212) are incorporated in the
latest upstream release.
Change-Id: Ibb2e2045ce813b4e69447fb5c896a2e0dfd4b1ec
IPv6 privacy extensions can cause issues by preferring a temporary
network over a public one. This preference may limit connectivity
in certain situations. An example of a connectivity issue can be
seen where the command ``traceroute6`` fails or misses all hops
while other traffic to a given domain with a "AAAA" record may
succeed. To resolve this issue the IPv6 privacy extensions have
been disabled.
Related-Bug: #1068756
Change-Id: If3bb0fd690673a6d93114e6aebddb5985344b437
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
As described, I missed that we only keep *one* kernel during dib
build, so as soon as the upstream package updates, it suddenly becomes
the latest kernel and kicks our custom version out.
Guess what happened in the hours between me committing
I0769f005da1931658a5fb9e627983ed30c11d212 and the next dib build.
This will install the current latest kernel with the required patch.
As described in the comment, I have the fix committed upstream so we
can remove this whole thing when fedora rebuilds for the next stable
release (even if the patch isn't in the official stable tree yet).
Change-Id: Iafe6d88e3ac7a2ea23553a5011df920a2ee3317d
At the end of gate jobs, we get warnings from zuul_swift_upload that
are not actionable for us.
Use requestsexceptions to silence these warnings. Add requestsexceptions
to the venv that is created for zuul_swift_upload to run in.
Example file
http://logs.openstack.org/48/298048/1/check/gate-ha-guide-tox-checkniceness/62e2d16/console.html
The example shows 6 warnings starting with:
/usr/zuul-swift-logs-env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:315:
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject
Name Indication) extension to TLS is not available on this platform.
This may cause the server to present an incorrect TLS certificate, which
can cause validation failures. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
/usr/zuul-swift-logs-env/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:120:
InsecurePlatformWarning: A true SSLContext object is not available. This
prevents urllib3 from configuring SSL appropriately and may cause
certain SSL connections to fail. For more information, see
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
...
Change-Id: I02b4b6c7b426f3e9aa498941e4d75f67066d4d27
Add a build-date timestamp file to the nodepool-base element and
output that as part of the "network-info" macro that we run at the
start of most tests.
This will allow non-priviledged users to quickly see the date the node
running their test was built on, which can be correlated to the logs
on nodepool.openstack.org to help debug issues that might have to do
with the underlying image build.
Change-Id: Id0c9f6203ed487350285031d3965bc6290370a27
Since I6c5a962260741dcf6f89da9a33b96372a719b7b0 dib has had a
standardised method for ensuring consistency of tracing and error
detection. Bring the tracing for these elements up to that standard,
but maintain the status-quo of flags such as "-e" and "pipefail" by
adding ignore flags where appropriate (we can update these separately
to avoid breakage)
Other minor changes are alphabetical-ordering in the element-deps
files and permissions on prepare-node script
With this, "tox -edib" passes
Change-Id: Ibba1dadb9e819f94294c9d583b83ff698252f93f
This commit creates a venv for installing os-testr which will enable
all test jobs to have access to the subunit2html utility which has
been moved to live inside the os-testr package instead of as a slave
script.
Change-Id: I2050b54eb2def10438764f3eeb55ecf9caa874dc
When we're building nodepool images on top of minimal elements, there
will be no cloud-init, and therefore no need to disable cloud-init
datasources. In fact, trying to do so will be an error.
Change-Id: I98887c43566e07f2be9d2dc5fae6538078c7348e
Rather than deleting cloud-init, which is going to take longer, just
disable ec2 metadata service. This will be a no-op on rackspace, which
already does this.
Change-Id: I5e8baee50800f7aae474288a914333c21466855a
Allow supplying filename and paths with '**' recursive glob matches
to zuul-swift-upload. Since bash (or shell etc) will expand on any
filenames provided to the program this needs to be used in quotes.
Usage example:
./zuul_swift_upload.py my_results.txt '**/sdist/*.zip' output.log
The hierarchy is always flattened meaning the supplied list is
placed in the topmost generated index.html. Sub-folders still keep
their hierarchy.
Change-Id: I9ba04f7e46b579dcf3f8ad0bd188f41fa5dbcad9
Since grub isn't installed by DIB until finalise.d/51 we need to
make sure the restrict-memory element happens after that, so it can
find and alter the grub configs accordingly.
Also make it apply the updated configuration, similar to how
nodepool/scripts/restrict_memory.sh does it.
Change-Id: I854f3bd1850594811cc8957f7a9263c33dfe6826
These are no longer being used to run jobs, so clean up references
to them.
Change-Id: Iac7ffde66a2d5d1a1361ed9bcadb9144e034f10f
Depends-On: I40d5d09f28ef53583d239d2e852e9c50b5962cf8
The zuul swift log uploader script needs argparse on python2.6 and
requests on all python versions to work properly. Install these two
packages to the virtualenv configured for this script at image build
time.
Note that `pip install argparse` on python >= 2.7 is not an error
it just reports that argparse is installed as part of the python
installation and moves on.
Change-Id: Ie79b59e1c687cce2b435c9704959b02911f94778
DIB needs its scripts to be executable or they are not run. Update
permissions on two files that were missing the executable bit.
Change-Id: Ie172e50de57b5168264964644cd28530f023542a
Set up a venv for zuul-swift-logs and install its dependencies
(python-magic) there instead of as system packages.
For DIB use 90 as uploading logs may be considered more important
than installing zuul.
Change-Id: Id0fc01729853e65c81cdd50e4ffa4d0d6de00ae9
Ian Wienand