Commit Graph

149 Commits

Author SHA1 Message Date
Ghanshyam Mann cbf9d36df6 Retire Tripleo: remove repo content
TripleO project is retiring
- https://review.opendev.org/c/openstack/governance/+/905145

this commit remove the content of this project repo

Change-Id: Ib5a8155d76652044701ae05e5dbcc7078f41d863
2024-02-24 11:41:18 -08:00
Cédric Jeanneret b6eedb6fb1 Ensure update_yum.sh has correct SELinux labels
On recent, enforcing systems, the update_yum.sh can't be executed,
because SELinux prevents container_t to open user_tmp_t:

type=AVC msg=audit(1674049913.380:22858): avc:  denied  { open } for
    pid=70472 comm="bash" path="/tmp/yum_update.sh" dev="vda4" ino=218200014
    scontext=system_u:system_r:container_t:s0:c65,c705
    tcontext=unconfined_u:object_r:user_tmp_t:s0
    tclass=file permissive=0

This patch ensures it gets properly relabelled when bind-mounted during
the image build. Using the "z" will also ensure it's still usable even
when running multiple builds at the same time.

Change-Id: I4085865965f48c9fa6a88cde7010a51cd8c653d8
2023-02-09 09:27:48 +00:00
Sandeep Yadav d023d54eb6 Moves the shared zuul queue to the project level
See [1] for more information

[1] https://lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html

Depends-On: https://review.opendev.org/c/openstack/tripleo-quickstart/+/873213
Change-Id: I3c56720df32bf5b227c450ee25972c723d34f4d8
2023-02-09 07:47:18 +00:00
Zuul aaa89b23c7 Merge "Add some notes about SELinux and limitations" 2022-08-02 17:43:45 +00:00
Cédric Jeanneret a09c8af2fb Add some notes about SELinux and limitations
With newer podman and container-selinux, we may face some issues
depending on where directories are located. The way we're running the
role is also important - being launched as root or as a user makes a big
difference, especially with the user's home content.

It also updates the setup.cfg content to match new format requirements

Change-Id: Ib2b7fd2f557d4743efd2eaca18474fb45b91cbcf
2022-07-14 15:46:51 +02:00
Luigi Toscano c9831b59fd Disable setup.py auto discovery
Tripleo-ci jobs are broken after latest release of setuptools 61.0
because of breaking changes which are not backwork compatible,
details in related bug and [1].

[1] https://github.com/pypa/setuptools/issues/3197

Related-Bug: #1966382
Change-Id: Id60e8bc882f6c7944f89333fb539f6800ef43725
2022-07-11 18:09:52 +02:00
Zuul 30d23d5152 Merge "Move zuul jobs layout to centos9 only for master branch" 2022-02-15 12:43:28 +00:00
Pooja Jadhav 8360ef2d5d Move zuul jobs layout to centos9 only for master branch
See topic branch [1] for related reviews.

[1] https://review.opendev.org/q/topic:c8_teardown_master
Depends-On: https://review.opendev.org/c/openstack/tripleo-ci/+/826317

Change-Id: I310aed164deece24fc2adbe197562ef66403abf1
2022-02-09 14:21:40 +05:30
Bhagyashri Shewale af582e0628 Move zuul jobs layout to centos9 only for master branch
See topic branch [1] for related reviews.

[1] https://review.opendev.org/q/topic:c8_teardown_master
Depends-On: https://review.opendev.org/c/openstack/tripleo-ci/+/826317

Change-Id: I6b49ef5639b092166f171a43f3587ae83332c881
2022-02-08 18:22:32 +05:30
Sagi Shnaidman 0b9fdccb8b Remove config of container_build_tool
Because there is no docker supported anymore, remove all custom
config of "container_build_tool", only "buildah" is supported.

Change-Id: I57720cc1d7da96f9277b62298cabce9ff7765f47
2021-08-11 10:24:31 +00:00
Sagi Shnaidman 81ca6b4dc0 Remove docker modules from the role
Remove docker modules usage from the role since we
use buildah/podman everywhere.
Change-Id: Ic2a46c0a55781041016f42bdc1af686055af9fa2
2021-08-09 20:55:41 +03:00
Ronelle Landy e4c84adc36 Extract updated rpms from buildah images
When a gating, component or delorean current repo
is present, containers should be updated with
the latest rpms.

This review checks the installed rpms and
greps for the repos where the updated rpms
are sourced.

Change-Id: Ie29c7c33c8d66bc3729c03c2d72cbdbf85ad443a
2021-07-21 22:14:02 -04:00
Ronelle Landy b304c8979c Only mount /etc/{{ pkg_mgr_suffix }}/vars if it exists
/etc/{{ pkg_mgr_suffix }}/vars exists on stream and
some other platforms but is missing from RHEL 8.x.

This patch checks that that directory exists first
before mounting it.

Change-Id: I048434b38eb1d6b1c83a89d90e01f71d467d7fb7
2021-05-20 14:08:42 -04:00
yatinkarel 2c53d70612 Use dnf/yum vars based on what is available
With [1] it used yum/vars but in CentOS8-Stream
that is not available, so detect and use dnf or yum
vars.

Closes-Bug: #1927302
Change-Id: Idda53d1b68b97e5bb65314b1e07d507736932531
2021-05-06 12:21:26 +05:30
Zuul 6ee663848e Merge "Mount yum/vars along with yum repos" 2021-05-05 18:03:55 +00:00
Zuul 38691f5ef4 Merge "Fix git lookup" 2021-04-23 11:57:36 +00:00
yatinkarel 98fa494f5c Mount yum/vars along with yum repos
Since repos may rely on yum vars, it's required
to mount yum/vars along with yum repos.
Container builds already mount yum/vars along
with yum yum. Molecule jobs also adding it with [1].

Since the repo is branchless and used in CentOS7 too,
so using yum/vars instead of dnf/vars.

[1] https://review.opendev.org/c/openstack/tripleo-ansible/+/787423

Change-Id: I36f175d97a86d4221b09dadf62f64a16b5c527e2
2021-04-23 11:06:43 +05:30
Alex Schultz 9f3298c75e Fix git lookup
In CentOS8, git-core is the package instead of just git.

Depends-On: https://review.opendev.org/c/openstack/tripleo-common/+/786053
Change-Id: I5746a7230f307e59d93401372033ae45a65ba191
2021-04-13 16:01:14 +00:00
Hervé Beraud 12655c7e73 Replace deprecated UPPER_CONSTRAINTS_FILE variable
UPPER_CONSTRAINTS_FILE is deprecated and TOX_CONSTRAINTS_FILE is
the new environment variable name that replaces it [1].

This allows to use upper-constraints file as more
readable way instead of UPPER_CONSTRAINTS_FILE=<lower-constraints file>.

[1] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file
[2] https://review.opendev.org/#/c/722814/

Change-Id: I089ea79398f71bd841992eacfd088d831ce0d085
2020-11-04 11:14:26 +01:00
Amol Kahat 9961cf84a3
Migrate to content provider jobs/templates
This change migrate c8 jobs/templates to content provider
jos so consumer jobs can use resources built by provider
jobs.

Change-Id: I17a40cbf6fb978bec17ab21fb2296ad3db46b6b3
Signed-off-by: Amol Kahat <amolkahat@gmail.com>
2020-10-09 15:51:27 +05:30
Bhagyashri Shewale 1dffa21831 Port jobs from centos7 to centos8
All the tripleo-ci jobs running on greater than train branch should run
on Centos8 only. Porting of this job was skipped earlier.

This change ports train jobs from centos7 to centos8.

Change-Id: I1eb2954ffcc582117b27fa3cf4d3fac4789d01b0
Signed-off-by: Bhagyashri Shewale <bshewale@redhat.com>
2020-07-30 16:10:38 +00:00
Emilien Macchi affe2c3e71 rpm_install: return 0 if package was already installed
Currently, if the package was already installed, the script would return
2, or 6 or the package would be upgraded.
It causes the script to fail (we use set -e); so to avoid that we added
--replacepkgs --replacefiles and --oldpackage arguments to rpm.

- Install the packages even if some of them are already installed on
  this system.
- Install the packages even if they replace files from other, already
  installed, packages.
- Allow an upgrade to replace a newer package with an older one.

Change-Id: I8391e02456a304d4e29256cbbf5433879c6c42b1
2020-07-03 08:54:41 -04:00
Julie Pichon 4130a447d4 Add retries on image pull ahead of updating
We are seeing occasional failures to pull due to intermittent registry
errors downstream. A retry should help to avoid the whole update
failing because of this.

Change-Id: Ib9415e46a52cc6ad6459ec3f170d7e23aa9aca03
2020-06-11 14:55:24 +01:00
Kevin Carter bc21900d06 Fix become and remote_src when modifying an image
This change updates the copy task to assume the source is remote. This
will allow the module to use any path provided and not assume the file
is within the relative path from the role.

The become jinja used in the task "Modify image from {{ modify_dir_path }}"
was attempting to match on the full command executed, which would always
evaluate to `false`. This change corrects the condition so that it can
evaluate to true, when "buildah" is used.

Change-Id: I348dbf91b5f2e0dcdfeffd336782c24499d7a569
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2020-05-26 12:27:33 -05:00
Julie Pichon bb6f78deca Add double quotes around the printf statements in yum_update.sh
This is consistent with the other printf statements in the file. This
keeps the sorted data with the newlines as they are, and avoids
breaking down the data again. Otherwise, the sorting order can be
altered which breaks comm comparison and causes the "Run yum_update.sh"
task to fail with the following error:

    "stderr_lines": [
        "comm: file 2 is not in sorted order",
        "error while running runtime: exit status 1"
    ],

This is particularly important for the $available variable which
contains the repoquery output, often very large and with unusual
characters. The error happens consistently when including RHEL 7 server
repos or RHEL 8 BaseOS repo and prevents the update from working at
all.

This patch also adds -u when sorting the $available output. Once the
versions are removed with cut, there can be a tremendous amount of
duplication in the data obtained from repoquery. That makes verbose
mode even more difficult to navigate when debugging.

Change-Id: Id64a8c2e395de28945d5d6b4370c44b80392e543
2020-01-30 09:28:23 +00:00
Bogdan Dobrelya eff235ee72 Ensure the yum cache has at most one writer
If the yum cache path exists and already mounted by someone,
do not attempt writing to it, use the overlay mode instead.

This still leaves a window of opportunity for another workers to
RW mount the cache after the ansible check has reported a stale
fact about there was no other mounts found. But this is unlikely
to happen.

Also, if it has to be retried in the rescue block, do not use the yum
cache for the maximum data safety and clean (a scratch) cache state
reasons.

This drastically reduces chances to have multiple writers for the
cache.

Closes-bug: #1860804

Change-Id: I19491a162e5bf6d6517fd343d675aff12bdc9719
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2020-01-29 09:33:07 +01:00
Alex Schultz 1e10b2294c Force docker format
The undercloud registry currently doesn't handle OCI formated images
correctly. We need to ensure that when buildah is run, we specify that
we want the docker format until we correctly support the OCI metadata.

Change-Id: Icf1a1c8f3a353239f2d244aa0bc811f8f86f6867
Related-Bug: #1860585
2020-01-22 10:39:05 -07:00
David Hill 58d7a5bc9d Dockerfile-yum.j2 doesn't copy the RPMs so yum_update.sh fails
After implementing "yum localinstall", a critical part of the
process wasn't merged in the code which consists of copying
the actual RPMs through the Dockerfile-yum.j2 file.

Change-Id: Ifbcaa07bb80dd612e85f0d7cf8d99131fb739c84
Closes-bug: #1860184
2020-01-17 17:47:17 -05:00
Zuul 3c3aa68778 Merge "Add the possibility to update packages from local rpms using yum." 2020-01-14 15:33:57 +00:00
David Hill 22374eb422 Add the possibility to update packages from local rpms using yum.
This is needed if we want to only update installed packages and not
hit depenency issues encountered when updating packages with rpm_install.sh

Change-Id: I5095d7b04cb10fde1bd82afd1bc406445b7595fd
Closes-bug: #1858837
2020-01-13 20:38:19 -05:00
Julie Pichon 8820e9493d When updating, set PKG variable to dnf or yum at runtime
When updating RHEL7 containers from a RHEL8-based environment, the
update fails with the following messages:

    "stderr": "/tmp/yum_update.sh: line 18: /bin/dnf: No such file or directory",
    "stdout": "No packages were found for update...",

This patch reverts the PKG variable assignment to the way it used to be
so that the decision to use dnf or yum is made at runtime, within the
container.

Change-Id: I6a248b5277efd8707fc744292075949d94e3a05f
2019-12-20 15:42:42 +00:00
Zuul c2a428db12 Merge "Mount /etc/pki during buildah yum update for RHUI" 2019-12-03 23:27:23 +00:00
Chandan Kumar (raukadah) 2dd5a11c55 Mount /etc/pki during buildah yum update for RHUI
In order to make sure RHUI repos works with in a container,
PKI certs dirs needs to be mounted to container so that
RHUI repo solves and download the content.

Related-Bug: #1854685

Change-Id: Id09059559b5c207ef6f604e4bb999528118ae096
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
2019-12-03 18:29:13 +05:30
Lance Bragstad 970e479c48 Use length to avoid unsupported operations with '>'
The dev_install.yaml script breaks when running python 3 because of
unsupported comparisions between 'AnsibleUnsafeText' and 'int' with the
'>' operator.

This is very similar to a bug opened against ansible upstream [0]. We
can fix the issue by using length explicitly, which ensures we're
comparing integers to integers since we're dealing with a list.

[0] https://github.com/ansible/ansible/issues/50388

Change-Id: I100639982b75c9d345269ba9bf10defb7e96d774
2019-11-27 07:22:49 -06:00
Emilien Macchi 66a92a458e Python3 support for dev_install
Change-Id: Ib893946407c2fd79e498b0fae2f3862e75e1348f
2019-10-29 14:42:38 +00:00
Sorin Sbarnea 75ee474fe1 fixed and bumped linters
- avoids ansible-lint installation bug
- bumps linters to their current versions
- adds missing passenv which prevented run with SSL proxies
- removes tox environments to ease maintenance, user can use posargs
  to run a single linter (hook).

Bug: https://bugs.launchpad.net/tripleo/+bug/1848512
Change-Id: I7657bb829a2928a8310c1758e6934b2f2ddef5c9
2019-10-29 10:58:22 +00:00
Alex Schultz 589486e4f2 Add retires/ignore errors for removing buildah containers
We want to try a best effort to remove the buildah containers but since
it's run with multiple processes we occassionally get layer conflicts.
Let's add a bit of a retry and ultimately skip the error since that was
the previous behavior.

Change-Id: I75a85745aed652a85f4c143c987cd5cccbf31cac
Related-Bug: #1846413
2019-10-02 15:05:07 -06:00
Zuul bf745be97f Merge "Enable saving of rpms" 2019-10-02 15:47:10 +00:00
Zuul 42a263210d Merge "Fix exists check" 2019-10-02 00:02:45 +00:00
Zuul 85ae72f6af Merge "Cleanup buildah working container" 2019-10-01 23:39:30 +00:00
yatinkarel 65e18ace49 Add missing package manager detect task
https://review.opendev.org/#/c/685466/ switched
to yum_update script to template but missed
adding tasks for detecting package manager.

Change-Id: I1acbbe418cba9e9f363ce4fdf4314985b52a2dd6
2019-10-01 11:34:44 +05:30
Zuul 8d218521cf Merge "Fix tasks names for the yum update" 2019-10-01 03:11:24 +00:00
Alex Schultz 2261d850a7 Enable saving of rpms
We currently map in the cache dir, but it only saves the metadata. The
improvements would come with the rpm saving as well. In order to have
that, we need to set keepcache=1 which is disabled by default. We can
enable this on the command line via --setopt so we don't have to touch
the container yum configuration.

Change-Id: Ia274dbbad9a220e52b2cfa554dede3bf40055f98
2019-09-30 15:54:57 -06:00
Alex Schultz d533efdfa3 Cleanup buildah working container
Currently in the yum update output, we see buildah trying to rmi the
image we were working with however it is currently in use. After we
commit our changes we need to cleanup our working container so we can
remove the image we were using (if not used by another process
elsewhere).

Change-Id: I54e37b43346b97be0a7cfab12e6cac9809537c83
2019-09-30 12:53:58 -06:00
yatinkarel 20e69383e1 Switch to template for yum_update.sh
Followup of [1] which switched to template for
yum_update.sh but missed updating it for
yum_update_docker.yml.

[1] https://review.opendev.org/#/c/683100/

Change-Id: Ibf5c9f0e07ff141a6c879fcda71be8119ae29b38
2019-09-28 00:45:09 +05:30
Zuul d28803f403 Merge "Add support for yum caching when buildah updates" 2019-09-27 04:26:12 +00:00
Bogdan Dobrelya f594406d7e Fix tasks names for the yum update
Change-Id: If66423850ade2782e8a4b3b973d9b4d343644b23
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2019-09-26 18:04:00 +02:00
Bogdan Dobrelya 908875eeab Add support for yum caching when buildah updates
When yum_cache is set, that directory will be automatically
picked as either the source or destination for the containers
being updated as the following:

* when that host directory is missing (or empty), the container
  under update will start populating it, while it gets updated.
  That path going to become the lower overlay FS layer for future
  use among other containers under concurrent yum update executions.

* when the yum_cache directory exists and is not empty, it will be
  bind-mounted as an upper overlay FS layer for other containers under
  update. So those can benefit from some of the already prefetched
  contents in its yum cache without data races or conflicts when
  concurrently accessing the cached data.

Overlaying ensures data safety as each container can only see the lower
layer of the overlay, while storing its local changes on top of it as
an ephemeral. The yum_cache directory existance & non-emptiness facts
act as a single mutex, which only grants a dedicated writing access to
the lower layer to a single "populating" container at a time. This
behavior may be forcefully reset via the force_purge_yum_cache flag.
The container update playbook invoked with it, instantly creates a
new populator and creates a fresh yum cache.

Note that the 100% saturation of the cache is only expected, when the
populating container finishes its execution.

The feature can be used only for buildah in yum update scenarios using
yum or dnf.

Change-Id: I30c6dd12454a0b1781803ab16ef79b5914178114
Related-bug: #1844446
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2019-09-25 14:01:15 +02:00
Alex Schultz e19509f90c Fix exists check
The |exists is currently deprecated.

[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using
`result|exists` use `result is exists`. This feature will be removed in version
 2.9. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.

Change-Id: I0c32686062e79142aa5c664a4a42ac590263b64e
2019-09-23 15:11:39 -06:00
Bernard Cafarelli 8c20273098
Update README examples on yum update/install
Drop compare_host_packages parameter and description from examples
This was dropped some time ago in
c9d085729f

Also highlight that only file repositories will be used in yum update
task

Change-Id: If9942c75ea786304ffe3aa6e28939708f8597ef3
2019-09-16 16:17:54 +02:00