Commit Graph

67 Commits

Author SHA1 Message Date
Michal Nasiadka fa2591d327 kolla-toolbox: Use py311 on rpm distros
ansible-core 2.16 requires py3.10+ on the host running
ansible-playbook - let's use Python 3.11 on RPM distros
to have the same version of ansible-core used on all
distributions.

Change-Id: I2caf2c972cd0283889fadde01bdd6a53e5f66ed9
2024-04-18 09:19:16 +00:00
Sven Kieske c0aee4adb6 bump td-agent lts from v4 to v5
notice that the td-agent command got renamed to fluentd.
see also the upgrade notes at[0].

for more details, refer to the changelog[1].

[0]: https://web.archive.org/web/20231006084137/https://www.fluentd.org/blog/upgrade-td-agent-v4-to-v5
[1]: b4b7017c09/CHANGELOG.md (release-v500---20230729)

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/901509
Change-Id: I5e95363c6c180e330945fe3d98e97c7049bbd6f7
Signed-off-by: Sven Kieske <kieske@osism.tech>
2023-11-30 07:07:31 +00:00
Michal Arbet 4871ca4d6b Lower limit for open files when installing fluent gem
Podman has default ulimit set to 1024, because
of that fluent build is failing.

Change-Id: I105351b58c530841e17fcba331fc4300eb8296f2
2023-09-21 11:01:21 +02:00
Zuul 64098b209c Merge "Revert "macros/pip: revert to old setuptools way"" 2023-09-13 22:33:41 +00:00
Michal Nasiadka f778b61c5c Revert "macros/pip: revert to old setuptools way"
This reverts commit 497b40fe5a.

Reason for revert: Should be fixed?

Change-Id: I524a4e1e34473fed523070eeeb9d0f3d0b1de26c
2023-09-12 19:41:27 +00:00
Marcin Juszkiewicz 19e40f1608 docker: implement macros to alter upper-constraints.txt
upper_constraints_remove() macro allows to remove line
upper_constraints_version_change() allows to change versions

This way we have cleaner way to alter u-c in those images which need it.

Change-Id: I8fc354b8aa4d03fcd3ecfb9cbfe75de67492a0e3
2023-04-21 14:13:24 +02:00
Marcin Juszkiewicz 3915785884 drop add_binary_source_envs() macro
With binary install type dropped there is no need for this macro.

Change-Id: I7bdd29eb48a23faa04a98e785129d97258f05e64
2022-04-11 12:09:29 +02:00
Marcin Juszkiewicz 497b40fe5a macros/pip: revert to old setuptools way
AArch64 builds fail with "AttributeError: install_layout". This is due
to setuptools issue in Debian:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1003252

Switching setuptools to use distutils from stdlib makes error go away.

Backports: Xena, Wallaby (this is where it was first reported).
Change-Id: I97102e18e4ec47bca94d17d964936f039580b06b
2022-03-30 09:52:58 +02:00
Radosław Piliszek 79b74ff220 Clean up fluentd image recipe
Since Xena, there is no arch supported other than x86_64 and
aarch64.

Additionally, the conditional for x86_64 Debian does not seem to be
needed now with current td-agent.

Moreover, drop two plugins that are relevant only to fluentd 0.12,
one of which (parser) causes fluentd 0.12 (!) to be installed.
Modern fluentd has both built-in.

Finally, drop the labels as they are no longer useful.

Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/823093
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/823094
Change-Id: I7326cd8fcfa459dedcb6404e5287f0f5aeeed6cb
2021-12-29 10:39:56 +00:00
Marcin Juszkiewicz a11815cd96 Do not use distro_python_version for interpreter
We can just use python3 - not longer need to tell which exactly like it
was in py2/py3 times.

Change-Id: Ib02f13cd2ad0037228e8975bd3fc37f41f23846f
2021-09-16 16:28:00 +02:00
Marcin Juszkiewicz bd90b02708 drop leftovers of RHEL support
As a project we never built RHEL based images so support for them
was not tested.

Change-Id: I27dfe34ade088228d71f6857ec4f3ab28ee63915
2021-07-28 16:04:21 +00:00
Marcin Juszkiewicz 87730f8f7c create 'add_binary_source_envs' macro and use it
There is small subset of images where we need to know which install_type
or install_metatype is used. So add them only there (and their
children).

Change-Id: Ib7d5e36b958d6c8daf2989df32e29fa24b46c62a
Implements: blueprint infra-images
2020-09-21 12:10:57 +00:00
Gaël THEROND (Fl1nt) b93c40a363 Improve pip install process for offline deployment.
* "Use distribution-provided pip"
    * "Use python's pip module invocation method"
    * "Install pip earlier in order to avoid multiple installation"
    * "Remove pip_version variable requirement and call"

Change-Id: Id0e738044a1931f9d611a7281a48ea4a593f1cf1
Closes-bug: #1893204
2020-09-17 13:50:15 +00:00
Marcin Juszkiewicz 53443c5c71 Remove support for CentOS 7
With the move to RHEL/CentOS 8 we no longer have Python 2 in our images
so there is no need for checking which Python version (2.x or 3.x) is
used inside of containers.

We also no longer have to support yum as a value for
distro_package_manager.

Partially-Implements: blueprint centos-rhel-8

Change-Id: Ie45cf3465fedddbde7856961527421883ba3d5c9
2020-04-15 09:32:06 +00:00
Dincer Celik f1f1d85459 Upgrade RabbitMQ to 3.8
Upgrade and pin RabbitMQ to 3.8.

Upgrade Erlang to 22 to meet requirements:
https://www.rabbitmq.com/which-erlang.html

Start using bintray repository provided by RabbitMQ.

Linaro builds will be used for aarch64.

Implements: blueprint rabbitmq-version-upgrade

Closes-Bug: #1848452

Change-Id: I3d775fd832f2603181da89009cebc1370a61a023
2020-04-06 15:14:02 +03:00
Zuul 7bec0316a8 Merge "CentOS 8: base and openstack-base images" 2020-01-29 11:55:19 +00:00
Mark Goddard 3c618a1f6f CentOS 8: base and openstack-base images
Adds support to the base and openstack-base images for CentOS 8.
These images have been removed from the list of unbuildable images, and
replaced with all dependent images.

Change-Id: I18fba8a5c2f99b700990cbf41c76077a1ec2408a
Partially-Implements: blueprint centos-rhel-8
2020-01-09 17:51:42 +00:00
Marcin Juszkiewicz 2c2951b59b centos: disable EPEL by default
Our images use 'scsi-target-utils' from EPEL and then can be built
without it. So let's disable this repository right after installing
package from it.

Images can enable EPEL with 'enable_extra_repos' macro.

Also added 'disable_extra_repos' macro as we have some images in a need.

Change-Id: Id70474e7602b531b63fdc11ae690fd9ed33def74
2020-01-02 09:58:58 +01:00
Mark Goddard 0ff14f85e4 Revert "Introduce macro to remove Py3 only plugins on CentOS 7"
This reverts the following commits:
df987c9d3f
d866c63a32
6b40b655f1

These are no longer necessary as we are using stable/train branch for
source images until CentOS 8 / python 3 support is available and
working.

Change-Id: I5aedf1141862d51c5bb676a7393fc131c452c914
Related: blueprint centos-rhel-python-3
2019-12-06 15:41:32 +00:00
Marcin Juszkiewicz df987c9d3f Introduce macro to remove Py3 only plugins on CentOS 7
As CentOS 7 is Python 2 we need a way to remove plugins used in
OpenStack components once they switch to be Python 3 only.

'remove_py3_only_plugins_for_py2' macro does exactly that.

This change also disables networking-bgpvpn for python 2.

Change-Id: Ib90aabc485f1c831d3d41cc0f70052f8fffc3fe6
Related: blueprint drop-py2-support
2019-12-05 14:48:05 +01:00
Zuul fb74313d7d Merge "Enable repos only when needed" 2019-12-04 17:45:50 +00:00
Marcin Juszkiewicz 70ef9a4fee drop *_existing_user_fix macros
Commit 109706aa8b bumped 'system' users
beyond range used by Kolla. So Debian/Ubuntu does not complain that
system users created by packages exist already on package install.

Change-Id: I9bf4b240839d46088ac668f26cf065dd5e3775c2
2019-11-29 17:20:14 +01:00
Marcin Juszkiewicz cf11cd6f3d Enable repos only when needed
Disable external repositories by default and enable only when needed.

Depends-on: https://review.opendev.org/696480

Implements: blueprint repos-off-by-default

Change-Id: Icf2a8397a8349e0fe849d88d160409fd234480a9
2019-11-29 11:38:06 +01:00
Radosław Piliszek a20fa84e29 Remove deprecated images
Change-Id: I2fe57f7cd79724fa8a36bd702d75b013a68ecd1b
2019-11-13 09:17:34 +01:00
Radosław Piliszek 76fbf2c08b Debian: switch fluentd to td-agent on x86_64
This patch uses the newly added td-agent repo on Debian.

It also makes code more robust regarding different distro handling.

We now have td-agent everywhere on x86_64.

Change-Id: I4d2449d7d100c68084f937ef9981596fc1625375
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-09-25 08:43:30 +00:00
Mark Goddard fa8e174be6 Remove support for OracleLinux
We have agreed to remove support for Oracle Linux.

http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006896.html

Change-Id: I961dfba1aeef2c509ecdaf376dfe6928926ffb39
Depends-On: https://review.opendev.org/682858
2019-09-18 13:42:47 +01:00
Michal Nasiadka ef8d7a27ab Change fluentd to td-agent on CentOS
Fluentd packages in CentOS Opstools SIG repos is very old,
because of old-ish ruby (2.0) packaged with CentOS 7.

On the other hand - TreasureData packages for fluentd are packaged
with embedded ruby as dependencies - so this change moves to those.
Thanks to that we are able to move from fluentd 0.12 to 1.4.2.

Still we need ruby-devel for non-td-agent distros,
because we download monasca output plugin as source and build it
during installation.

* Bump up td.repo to version 3
* Merge sudoers into one file
* Change fluentd plugin install macro to reflect changes

Depends-On: https://review.opendev.org/679209/
Depends-On: https://review.opendev.org/679885/
Change-Id: I40a81103317ac044d9311b2505593056cf313bd2
2019-09-04 09:10:28 +00:00
Marcin Juszkiewicz f4e0bbd3fb fluentd: there is no 'td-agent' for Debian/x86-64
As there is no 'td-agent' package image build fails on 'x86-64'.

We have to wait for TD Agent 4.

Change-Id: I03d38628eee0f0823a58cba3a70df22f29022956
2019-08-19 18:17:06 +02:00
Mark Goddard c585ca5055 Use td-agent-gem to install fluentd monasca plugin on ubuntu
If monasca is enabled on ubuntu, we see the following error in fluentd:

[error]: config error file="/etc/td-agent/td-agent.conf"
error_class=Fluent::ConfigError error="Unknown output plugin 'monasca'.
Run 'gem search -rd fluent-plugin' to find plugins"

This is because we use 'fluent-gem install' to install the gem, which
works on CentOS but on Ubuntu it is necessary to use 'td-agent-gem
install'.

Change-Id: I515f9764ca93a41a871e520696193a9298856d08
Closes-Bug: #1830147
2019-05-29 14:41:52 +00:00
Zuul 3e9cf08ac5 Merge "Install fluentd gem on debian/ubuntu for non x86_64" 2019-05-28 19:04:17 +00:00
chenming 841961d4a7 Update kubectl v1.5.4 to v1.11.0
Change-Id: I51c4a4a139e87fd82f411d07683f62453889de9a
2019-05-07 20:30:45 +02:00
Pedro Alvarez 35394e25a3 Install fluentd gem on debian/ubuntu for non x86_64
The package used to install fluentd on debian/ubuntu is not
available for architectures other than x86_64. For those cases
install fluentd as a gem until the package 'td-agent' is available
for more architectures.

Change-Id: Ic1b25b37d06b429a112f1418a79d385d505bb1e9
2019-03-28 16:26:22 +00:00
Jon Schlueter 95f38fb0f6 cleanup in base for yum install entries
- introduce install_packages(..., chain=True) to allow chaining on
  onto existing RUN commands
- introduce install_packages(..., clean) which defaults correctly
  but allows you to override if you want to avoid doing extra
  package manager cleanup
- refactored several calls in base to make use of these new features

In general this allows for macros.install_packages to be chained
after other calls

for example

RUN foo && {{ macros.install_packages(..., chain=True) }}

Change-Id: I4b2fe8e13ee3c0fffe77a1179a7af5b534496996
2019-03-21 13:20:48 -04:00
Jon Schlueter cd307fb1ab refactor to pull out rpm_security_update macro
- pulled out the repeated code from 3
  paths in base and moved it into docker/macros
- created macros.rpm_security_update
- made sure the new macro could be customized for cleanup caches
- update macro to use distro_package_manager

Change-Id: Ic79ff42d913e1cbb68be66743e4f79f0bcff47f5
2019-03-21 12:48:53 -04:00
Emilien Macchi d170256896 Introduce clean_package_cache
Default to true (to maintain backward compatibility), this parameter
will allow to disable the Package cache cleaning when building the Kolla
images.

In TripleO, we are investigating how we can accelerate the builds by not
cleaning the Yum cache and this patch will help us to do so.

Also included same for deb package_install method.

Co-Authored-By: Jon Schlueter <jschluet@redhat.com>
Change-Id: I0adc85302dfe0411d362c899f3e0d6a0019680d4
2019-03-17 17:39:52 +00:00
Alex Schultz 78c819d623 Add distro_package_manager configuration
Add configuration to allow the switch between dnf/yum or provide an
alternative package manager.

Change-Id: I913769fcf5875a12b4561958cf7c90b74cd116ea
2019-03-12 18:24:29 +00:00
Alex Schultz 3e5d8e2653 Add python3 packages for RHEL systems
RDO is currently working on python3 support for the next version of
CentOS/RHEL based systems.  This package uses the distro_python3 flag
that was added as part of I4028991bad92c0e8e21066cc4173c06ce5eba393 to
use the python3 specific package names.  This change only adds python3
package names for RHEL systems.

Conflicts-With: https://review.openstack.org/#/c/636457/
Change-Id: Iad6b70b433a0dd1b0f8ae6790fd280594517661a
Related-Blueprint: python3-support
2019-03-12 17:48:18 +00:00
Marcin Juszkiewicz 8df6eb0118 replace 'distro_python3' variable with 'distro_python_version'
RHEL7 derived distros use Python 2.7
RHEL8 uses Python 3.6
Ubuntu 18.04 uses Python 3.6, but we haven't enabled it yet.
Debian:buster uses Python 3.7, but we haven't enabled it yet.

We have several places in code where "/lib/pythonX.Y/site-packages"
is used so we need to know value of X.Y rather than py2/py3 switch.

How to check for Python 2:

{% if distro_python_version == '2.7' %}

Breaks: Iad6b70b433a0dd1b0f8ae6790fd280594517661a
Change-Id: Ie24f109af19ae0438c21e0d43cb3cc947812b4ce
2019-03-12 17:13:46 +00:00
Marcin Juszkiewicz ef83ff13fe Introduce 'get_pip' macro
We have code for it in four places so it is a good candidate for macro.

Change-Id: Icadcbbe5b386bdf688fd67e8160b40974f047377
2019-03-11 16:52:09 +01:00
Marcin Juszkiewicz 43b74ccc15 debian/ubuntu: make use of Python3 based packages
Both Ubuntu Stein UCA and Debian 'buster' migrated their OpenStack
packages to Python 3.

Note that Debian 'buster' is not released yet and contains Rocky
packages. Stein ones will be available later.

Co-Authored-By: Lee Yarwood <lyarwood@redhat.com>
Co-Authored-By: Eduardo Gonzalez <dabarren@gmail.com>

Change-Id: I160f79cc57f54ec3eac857c5babd1a6e2656d228
2019-02-28 13:06:24 +01:00
Marcin Juszkiewicz ab679adcbf debian/ubuntu: introduce 'debian_arch' variable
Many images have 'if x86_64 then amd64 elif aarch64 then arm64' check to
comply with Debian like architecture names in several projects.

This patch creates 'debian_arch' variable which can be used in all
Docker files (similar to 'base_arch' one).

All required images got converted to use it.

Change-Id: I9c5e6f13d6c9b24fe323408512bd5aef290111ad
2019-02-19 11:24:45 +00:00
zhubingbing 4113483354 Add support rabbitmq promethues plugin
Change-Id: I1b2b3d6708bfcd58864e26e6c1fc618fb421e9a2
2019-01-04 15:46:22 +08:00
Martin André 27bab79096 Download binaries more securely
Obtain binaries from encrypted source when we're unable to check for
their signatures. This should provide better security than downloading
the files over HTTP but does not replace signature verification or file
integrity check.

Related-Bug: #1791674
Change-Id: I7d6eed9ab14ceb130ea4f5f03d893ddaaa0a7acd
2018-09-10 19:19:16 +02:00
Martin André 0d4760eaac Use absolute path in install_pip macro
This lets us use the macro in the case there was a WORKDIR instruction
that changed the working directory from / earlier in the dockerfile.

Change-Id: I2369c9997ef189e7084fac7b5ace014f6f76d698
Closes-Bug: #1746682
2018-02-01 09:30:32 +01:00
Marcin Juszkiewicz b6162f9f16 mongodb: handle "user already exists" error on Debian
When we install 'mongodb-server' package in Debian/stable it errors out
because 'mongodb' user already exists.

It was reported [1] and got resolved in Debian/testing.

1. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858903

Change-Id: Iabfcbf68a3aea9d7b09b72eb3e8325f09c2f9d31
2017-12-14 11:24:52 +00:00
Martin Mágr dd4fe190d0 Clean fluentd_plugins_install block
fluentd_plugins_install should contain only plugin installation operations,
so that the override does not have to handle irrelevant operations

Change-Id: I8f810d194a1e638683df1f445c187b8d98629730
2017-11-24 15:14:32 +01:00
Christian Berendt 4de0706190 Cleanup APT lists after installing packages
/var/lib/apt/lists/ can take several MBs so get rid of them after
packages are installed.

Change-Id: Icfff55d92fa5c865f09db5112da28bf452a5a221
2017-11-02 13:55:49 +01:00
Eduardo Gonzalez 47e794dfdc Remove yum cache
Change-Id: I92c8626b8ee4d5d3dd102a9fe378c4f41a7f2154
2017-09-19 11:50:17 +02:00
Sean Mooney 4f5bac1089 introduces support for the OVS DPDK dataplane
- DPDK is a  userspace library for high-performance networking.
- This change enable support of Open vSwitch with the DPDK
  accelerated netdev datapath.
- This change provided binary and source support via
  a template-override.
- This change provides an example template override file
  and corresponding documentation.

Co-Authored-By: Mauricio Lima <mauriciolimab@gmail.com>
Partial-Implements: blueprint ovs-dpdk
Change-Id: I7c3a5dddeca1886fe1c7681ce8d5bebde08f3102
2017-07-05 11:34:51 +00:00
Eduardo Gonzalez 8080531823 Add opendaylight install fix
Opendaylight package fails to install with existing
odl user.
This change adds a macro to fix the package install. Similar
to haproxy install fix.

Change-Id: Ie646769fe12b8f3f4c758b678b23c2d98c2e8027
Closes-Bug: #1700502
2017-06-27 09:44:35 +01:00