Commit Graph

35 Commits

Author SHA1 Message Date
Ghanshyam Mann cbf9d36df6 Retire Tripleo: remove repo content
TripleO project is retiring
- https://review.opendev.org/c/openstack/governance/+/905145

this commit remove the content of this project repo

Change-Id: Ib5a8155d76652044701ae05e5dbcc7078f41d863
2024-02-24 11:41:18 -08:00
Alex Schultz 9f3298c75e Fix git lookup
In CentOS8, git-core is the package instead of just git.

Depends-On: https://review.opendev.org/c/openstack/tripleo-common/+/786053
Change-Id: I5746a7230f307e59d93401372033ae45a65ba191
2021-04-13 16:01:14 +00:00
Emilien Macchi affe2c3e71 rpm_install: return 0 if package was already installed
Currently, if the package was already installed, the script would return
2, or 6 or the package would be upgraded.
It causes the script to fail (we use set -e); so to avoid that we added
--replacepkgs --replacefiles and --oldpackage arguments to rpm.

- Install the packages even if some of them are already installed on
  this system.
- Install the packages even if they replace files from other, already
  installed, packages.
- Allow an upgrade to replace a newer package with an older one.

Change-Id: I8391e02456a304d4e29256cbbf5433879c6c42b1
2020-07-03 08:54:41 -04:00
Emilien Macchi 66a92a458e Python3 support for dev_install
Change-Id: Ib893946407c2fd79e498b0fae2f3862e75e1348f
2019-10-29 14:42:38 +00:00
Bogdan Dobrelya 908875eeab Add support for yum caching when buildah updates
When yum_cache is set, that directory will be automatically
picked as either the source or destination for the containers
being updated as the following:

* when that host directory is missing (or empty), the container
  under update will start populating it, while it gets updated.
  That path going to become the lower overlay FS layer for future
  use among other containers under concurrent yum update executions.

* when the yum_cache directory exists and is not empty, it will be
  bind-mounted as an upper overlay FS layer for other containers under
  update. So those can benefit from some of the already prefetched
  contents in its yum cache without data races or conflicts when
  concurrently accessing the cached data.

Overlaying ensures data safety as each container can only see the lower
layer of the overlay, while storing its local changes on top of it as
an ephemeral. The yum_cache directory existance & non-emptiness facts
act as a single mutex, which only grants a dedicated writing access to
the lower layer to a single "populating" container at a time. This
behavior may be forcefully reset via the force_purge_yum_cache flag.
The container update playbook invoked with it, instantly creates a
new populator and creates a fresh yum cache.

Note that the 100% saturation of the cache is only expected, when the
populating container finishes its execution.

The feature can be used only for buildah in yum update scenarios using
yum or dnf.

Change-Id: I30c6dd12454a0b1781803ab16ef79b5914178114
Related-bug: #1844446
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2019-09-25 14:01:15 +02:00
Alex Schultz 3a4caf4587 Implement a yum install command
Yum update works if you want to update existing packages, but sometimes
you want to install a package that's available via the yum repositories
that is extra. This implements a yum install action similiar to the yum
update action except it takes package names instead of repos to install.

Change-Id: Ia47a1ea9eb51a37f6d75d39c524e97fd4ec94fba
2019-07-03 17:45:21 -06:00
Zuul 1415b042f5 Merge "Allow devs to modify images with Python directories" 2019-06-21 15:56:09 +00:00
Sorin Sbarnea 6dab9dda02 Fix dnf repoquery regression
Fixed issue identified at https://review.opendev.org/#/c/664872/1 which
prevented full execution of yum_update.sh on dnf platforms because
`dnf repoquery` returned exit code 1 when repo did not exist.

Change-Id: I86918f292b6863d6cb0e4fd16e62409a7f9fa80a
2019-06-21 03:19:56 +00:00
Emilien Macchi b25fe2590c Allow devs to modify images with Python directories
This will allow dev to update their container images from a local Python
directory, example in /home/joe/git/openstack/heat.

The new parameter python_dir is a list of directories.

To use it, your playbook must be like:

    - hosts: localhost
      connection: local
      tasks:
      - name: dev install heat-api
        import_role:
          name: tripleo-modify-image
        vars:
          tasks_from: dev_install.yml
          source_image: docker.io/tripleomaster/centos-binary-heat-api:current-tripleo
          modified_append_tag: -devel
          python_dir:
            - /home/joe/git/openstack/heat

Change-Id: I182c3fa58dc9af870e0da9f51ae1e22aa90d03e5
2019-06-21 01:31:01 +00:00
Zuul 8befbdbe5a Merge "dev_install: fetch git/gerrit from opendev" 2019-06-18 02:52:50 +00:00
Emilien Macchi 9f112a5693 dev_install: fetch git/gerrit from opendev
To avoid warning about the redirection, let's use the new URL.

Change-Id: I053ce25b7531fbec8bbca6dfb087acf13afcf511
2019-06-14 13:44:32 -04:00
Julie Pichon 60cb00de9a yum_update.sh: don't fail when plugin package is unavailable for install
Change-Id: Id1534e9d1005d48fc3893182e156503b703af405
2019-06-12 12:59:55 +01:00
Julie Pichon 8fa6dd7dde Use 'dnf repoquery' when system uses DNF
repoquery is part of dnf so we can use 'dnf repoquery' directly rather
than install dnf-utils, which isn't installed by default on e.g. RHEL8.

When using yum, command -v repoquery will still be used as it was
before this update.

Change-Id: Iea8822084988b23652e6d3f8b5828c05734158af
2019-06-12 11:14:07 +01:00
Julie Pichon 367f3e032a Small yum_update.sh improvements
Following the revert due to issue with repoquery in
Id12743c0cf83d054189df10cad2c8abec57961e8, re-submitting the two
unrelated changes. Follow-up patch to come for handling repoquery more
sanely.

* Use the --quiet flag with repoquery

  This prevents notices such as subscription-related stderr warnings
  from being added to the output and messing with the sorted order.

* Remove the : after else

  It seems using else without a colon is the bash standard. This gets
  rid of the following error on RHEL8:
  "/tmp/yum_update.sh: line 34: else:: command not found"

Change-Id: I8e52f239fcf962c17543ba56d3653de0016ed47f
2019-05-31 10:43:00 +01:00
yatin f33cad7443 Revert "yum_update.sh improvements"
Containers no longer updated in CentOS7 now,
as yum repoquery is not a command.

This reverts commit c475c8394a.

Change-Id: Id12743c0cf83d054189df10cad2c8abec57961e8
2019-05-31 09:03:54 +00:00
Julie Pichon c475c8394a yum_update.sh improvements
* Use <yum|dnf> repoquery instead of repoquery directly

  "repoquery" is part of <yum|dnf>-utils which may not be installed, in
  which case no packages would end up updated.

* Use the --quiet flag with repoquery

  This prevents notices such as subscription-related stderr warnings
  from being added to the output and messing with the sorted order.

* Remove the : after else

  It seems using else without a colon is the bash standard. This gets
  rid of the following error on RHEL8:
  "/tmp/yum_update.sh: line 34: else:: command not found"

Depends-On: https://review.opendev.org/662317
Change-Id: Id03405d49dc213d018941c44129521cca845e7d6
2019-05-30 20:17:55 +00:00
Steve Baker c0bcc3c438 Don't print commands during yum update
The commands contain package lists, so having "set -x" makes the
logged result very very verbose.

Change-Id: I8ffda90eb0633b6b46e29c5af7ed2ff3571c14a2
Related-Bug: #1813684
2019-03-22 11:30:56 +13:00
Steve Baker 9014df9472 Speed up execution time of yum_update.sh
Doing a yum update with an explicit list of packages where many of
those packages are already up-to-date has a significant time penalty.
This change includes version comparison to remove packages from
packages_for_update where the installed and available versions are
identical.

This change also only installs yum-plugin-priorities if it is not
already installed, for some extra time saving.

For a specific example of time saved, I had a
centos-binary-rsyslog-base with the following timings.

Without this change
-------------------
packages passed to yum update: 87
packages actually updated:     11
time taken                     35s

With this change
-------------------
packages passed to yum update: 11
packages actually updated:     11
time taken                     10s

This time saved should add up over all of the container images during
image prepare in CI.

Change-Id: Ie282acd1cdc0de8d875959ec9578122fe34f98ac
2019-02-25 04:44:21 +00:00
Chandan Kumar 410408ef1b Ported tripleo-modify-image to py3
It makes yum and pip installs work with py3 packages. The package manager
dnf/yum and pip binary are determined at run time from the OS.

Use dnf-plugins-core for dnf with respect to
yum-plugin-priorities.

Use pipefail with set -ex to fix linter errors.

Change-Id: I997509204e30abb8b21ef936ea44440fbaa5a0e4
Closes-Bug: #1813546
2019-02-18 18:22:40 +05:30
Chandan Kumar b541c5ef99 Run all linters via pre-commit
Executes all linters via pre-commit, which is much faster, guarantees
their version locking and allows upgrading them with a single command.

Before this change the only linter running via pre-commit was
ansible-lint.

Now we also run bashate, flake8 and yamllint via pre-commit.

For developer convenience we still keep the old tox environments
which allow running a single linter.

Added long_description_content_type to fix twine check failure

Change-Id: I037eae61921b2a84aa99838804f70e96ee8d8b13
2019-02-15 18:22:10 +05:30
Bogdan Dobrelya 8b82f78e35 Track obsoleted packages replacements for updates
Co-authored-by: Yatin Karel <ykarel@redhat.com>
Change-Id: I5dfe91238898208aedfeb6623c62b97a9398cecb
Closes-bug: #1813533
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2019-01-29 12:15:41 +01:00
Dan Prince 618d3ab83c Add dev mode install playbook for source installs
The following playbook will produce a modified image with Python source
code installed via pip. To minimize dependencies within the container
we generate the sdist locally and then copy it into the resulting
container image as an sdist tarball to run pip install locally.

Useful for local ad-hoc testing of upstream patches without
having to build and RPM in those cases.

Change-Id: I93c0e86b2e421b0bcb777d536fcb0d30e0ee3b68
2018-11-05 09:33:43 -05:00
Steve Baker c9d085729f Remove compare_host_packages strategy
Now that the undercloud is containerized, there will be very few host
packages to compare to, so there is a high risk that required package
updates will be skipped.

This is a strategy inherited from container-update.py that was
intended to avoid unnecessary calls to yum update, however we now have
a better approach using the repoquery, so host package comparison is
no longer required, and probably causing some of the instances of bug

Change-Id: Iab7b9d6377494001d904bb84b058ea293d73110c
Partial-Bug: #1786764
2018-09-17 12:17:46 +00:00
Steve Baker b58b916dbc Don't yum update files which are not installed
This change builds the packages_for_update list by intersecting
installed packages with the packages available, which avoids passing
uninstalled packages to the yum update call.

For some reason doing a yum update with packages which are not
installed causes a delay for each uninstalled package.

For my test case this gave a speed improvement of 35s vs 45s for the
whole image build. This would accumulate to decent time savings for
all images in CI.

Change-Id: I338d9881d5a05dd7b1fbc5efea1145349f0997a6
2018-09-10 10:22:26 -06:00
Sagi Shnaidman 678b93fe30 Fix 'too many arguments' error
Fix error:
"/tmp/yum_update.sh: line 17: [: too many arguments"
when upgrading packages in container
Change-Id: I159ee948b541adf281263696447608f3b6286505
2018-09-03 12:57:04 +00:00
Zuul c79d9a35b7 Merge "Install yum priorities to containers before update" 2018-08-31 20:36:36 +00:00
Sagi Shnaidman 9d4b320222 Install yum priorities to containers before update
When installing packages the package which was built by DLRN should
be installed even it has lower version. Let yum priorities to
ensure that.

Change-Id: Id1bb77550596c517e684a80a73cf0bee6d26e958
2018-08-30 21:54:48 +03:00
Sagi Shnaidman bca80b37cd Fix type in variable which blocked updates
Containers haven't been updated because of typo in variable name

Change-Id: Ie3d82368be4cf8030fe7393f7f70d44cd7bd77f7
2018-08-30 11:50:42 +03:00
Alex Schultz 7b587fe0f5 Only do yum update when needed
Currently the logic for this results in a full yum update when no
package updates are found in the provided repository. This can lead to
job timeouts when nothing was built in CI because it effectively does a
yum update on every container and applies other system packages rather
than ones that actually changed.

There is a larger issue in that we can still get out of sync with the
host OS.

Change-Id: Iaf41691ea3cb6e78186741ac5e15614fb73f89ff
Related-Bug: #1786764
2018-08-17 14:37:29 -06:00
Dan Prince 7178d84867 Add rpm_install tasks
This patch adds a new rpm_install tasks which can be used
to build a new container layer from a set of local RPMs.
This is useful for building contains locally with no network
connectivity.

As part of the change the existing Yum update Dockerfile.j2 was
renamed to Dockerfile-yum.j2 to make it more specific.

Change-Id: I951c976ebb84c28cecefd2f5753a8232f46ad4aa
2018-07-17 12:17:08 -04:00
Martin André 45ec917d1d Do not fail when repoquery is not installed
A good chunk of container images try to be as slim as possible and as
a consequence do not have repoquery binary which is used to limit the
packages to a given repo.

This commit makes the playbook more against these containers and
installs all available updates when repoquery is not installed, the
same as when we don't restrict the list of repositories to get updates
from.

Change-Id: I81e5cd0edae56c1535a2e4dcc639ec71055b10d2
2018-06-21 14:23:08 +02:00
Steve Baker 1f31f485ba Add tox checks, pbr packaging 2018-05-23 11:57:41 +12:00
Steve Baker 262e5497e0 update_repo to limit to packages in repo
Rather than updating every package, allow to limit to a single repo
2018-05-14 10:43:34 +12:00
Steve Baker e5c5b29e15 Don't yum clean all 2018-05-10 14:14:49 +12:00
Steve Baker cf858399d5 task to yum update in the image 2018-05-02 17:05:59 +12:00