Commit Graph

108 Commits

Author SHA1 Message Date
Dr. Jens Harbott 0b79f6f769 Add debian-bookworm job
Change-Id: Id5e54775e2be38a75db0bd1f55d1d3b5ae7ef71f
2023-08-02 05:11:03 +00:00
Zuul 2e607b0cbd Merge "Modify devstack-base to allow for fips" 2023-04-20 23:55:40 +00:00
Zuul b33ec4bf1b Merge "Remove support for opensuse" 2023-04-18 17:15:32 +00:00
Ade Lee 15b2e42968 Modify devstack-base to allow for fips
devstack-base is changed to descend from
openstack-multinode-fips which is defined in
project-config.

This allows jobs to execute the enable_fips playbook
to enable FIPS mode on the node, but only if they
opt-in by setting enable_fips to True.  Otherwise,
this is a no-op.

Change-Id: I5631281662dbd18056ffba291290ed0978ab937e
2023-04-17 08:43:22 +00:00
Dan Smith 7567359755 Try to reduce mysql memory usage
These are a few tweaks I applied to my own memory-constrained cloud
instances that seemed to help. I have lower performance requirements
so this may make things worse and not better, but it's worth seeing
what the impact is. I'll admit to not knowing the full impact of these
as they're mostly collected from various tutorials on lowering memory
usage.

Enable this for now on devstack-multinode

Change-Id: I7b223391d3de01e3e81b02076debd01d9d2f097c
2023-02-16 12:52:50 -08:00
Martin Kopec ec07b343d2 Remove support for opensuse
We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.

Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc
2023-02-16 12:01:39 +01:00
Miguel Lavalle 0a40648b38 Fix the db user for mariadb in ubuntu 22.04
In Ubuntu 22.04, mariadb version 10.6 is installed. Per [0] and [1]
authentication management was changed in version 10.4. This change
adapts the way the db user is created to the new rules in versions
10.4 and later.

[0] https://mariadb.com/kb/en/authentication-from-mariadb-104/
[1] https://mariadb.org/authentication-in-mariadb-10-4/

Closes-Bug: #1999090
Change-Id: I77a699a9e191eb83628ad5d361282e66744b6e4a
2022-12-07 17:05:35 -06:00
Dan Smith fe52d7f0a8 Change DB counting mechanism
The mysql performance_schema method for counting per-database queries
is very heavyweight in that it requires full logging (in a table) of
every query. We do hundreds of thousands in the course of a tempest
run, which ends up creating its own performance problem.

This changes the approach we take, which is to bundle a very tiny
sqlalchemy plugin module which counts just what we care about in
a special database.

It is more complex than just enabling the features in mysql, but it
is a massively smaller runtime overhead. It also provides us the
opportunity to easily zero the counters just before a tempest run.

Change-Id: I361bc30bb970cdaf18b966951f217862d302f0b9
2022-05-12 07:55:02 -07:00
Dan Smith c2772c2984 Gather performance data after tempest
This makes us gather a bunch of consistent statistics after we run
tempest that can be use to measure the impact of a given change. These
are stable metrics such as "number of DB queries made" and "how much
memory is each service using after a tempest run."

Note that this will always run after devstack to generate the JSON
file, but there are two things that control its completeness:

 - MYSQL_GATHER_PERFORMANCE must be enabled to get per-db stats
 - Unless tls-proxy is enabled, we will only get API stats for keystone

Change-Id: Ie3b1504256dc1c9c6b59634e86fa98494bcb07b1
2022-04-20 13:07:22 -07:00
Brian Haley 7943a92bdb Do not use hardcoded IPv4 localhost value
There are a couple of places that still use a hardcoded
127.0.0.1 value, even if devstack is run with
SERVICE_IP_VERSION=6 in local.conf. While things still
work, SERVICE_LOCAL_HOST should be used instead since
everything else could be using IPv6.

Change-Id: I2dd9247a4ac19f565d4d5ecb2e1490501fda8bca
2022-03-14 16:04:08 -04:00
Carlos Camacho cc6e20b24d Allow skip the database server installation
This patch allows to skip the installation
of the database backend packages (MySQL or Postgres)
with the introduction of the INSTALL_DATABASE_SERVER_PACKAGES
variable (defaulted to True).
This is useful in such environments that do not require
to install the MySQL/Postgres server packages directly but using
a container serving that purpose, for those cases all the
remaining steps should be executed just skipping the
packages install.

Change-Id: I26628a31fdda3ce95ed04a2b7ae7b132c288581f
2022-01-10 09:26:56 +01:00
Miguel Lavalle c1a75c6a50 Fix mysqladmin failure for Fedora 34 and mariadb
mysqladmin is incorrectly installed in Fedora 34 with mariadb. This
causes the failure of Zuul Fedora based jobs. The issue is a conflict
between mariadb and community mysql that is described in [1] and [2].

The workaround is to explicitly install package "mariadb"

Also configure an increased swap size like for the other platform jobs
in order to avoid OOM issues.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2026933
[2] https://lists.launchpad.net/maria-discuss/msg06179.html

Closes-Bug: #1956116
Change-Id: Icf6d7e1af5130689ea10b29d37cc9b188b2c9754
2022-01-03 15:33:35 +01:00
Zuul 71c215ac29 Merge "Revert "Generate deprecation warning for postgresql"" 2021-12-30 10:29:00 +00:00
Jens Harbott 959a7c262a Enable running on Debian Bullseye
Some adaption in database handling is all that is missing. Also add a
platform job that tests this.

Co-Authored-By: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Change-Id: I6dd3e48444dd415d84df5e7f5c74540847cdd6db
2021-10-04 17:27:33 +02:00
Ian Wienand 812e7846c9 Cleanup rpm-distro mariadb install
We have lib/databases/mysql which is installing databases, remove it
from the bulk package lists.

Split is_fedora (fedora & centos8 -- soon) to install mariadb-server
and mariadb-devel to retain status-quo.

On suse this seems to be a meta-package

 'mariadb-server' not found in package names. Trying capabilities.

so split that out.  It seems it has never been installing the -devel
package, and things work (presumably clients are coming from wheels so
don't need to build against it).

Change-Id: I86433318e8f76c40c5c792b795411a5c9d8351d3
2020-04-22 14:02:10 +10:00
Dr. Jens Harbott 08d84bc47f Make database code work with mysql8
The GRANT command in mysql8 can no longer create a user implicitly.
Split that part into a dedicated CREATE USER command.

Also drop disabling the query_cache, it is off by default for some time
and the option got removed in mysql8.

Change-Id: I31bcc285ff8e373abbacb303c1269857c9cfa9ed
2020-02-17 17:15:13 +00:00
YAMAMOTO Takashi ede8b1269c mysql: Don't bother to change auth plugin on centos
This partially reverts the previous change [1], which
broke networking-midonet jobs.

[1] https://review.opendev.org/#/c/681201/

Closes-Bug: #1855516
Change-Id: I0255c6acce72a8376dbc6d8f8d0314a7dabf019c
2019-12-09 14:23:59 +09:00
Slawek Kaplonski d54a1c6869 Add possibility to configure manually MYSQL_SERVICE_NAME
This variable can be now set in Devstack's config file and in
such case Devstack will not set it automatically to value most
likely correct for the distro.
By default this value is empty string and in such case Devstack
will work in exactly same way as it was before this patch and
will determine automatically what name should be used there.

In addition in case of Ubuntu package $MYSQL_SERVICE_NAME-server
will be now installed instead of mysql-server always.
This will allow to easy configure e.g. CI job which will run using
Mariadb instead of Mysql on Ubuntu.

Change-Id: I25af0b54ad235b08c6c399b4125c737acf57ee2e
2019-12-04 19:58:31 +00:00
Armando Migliaccio a676c4029e Revert "Generate deprecation warning for postgresql"
Based on resolution [1], there's no clear indication that next
steps involve the removal of the DB from Devstack or from the gate.

[1] I332cef8ec4539520adcf37c6d2ea11488289fcfd

This reverts commit d9aaae95f2.

Change-Id: I8410d65c0e0b24035aa035fac7560a686d53ec50
2019-10-17 15:58:34 -04:00
Matt Riedemann b14665f0dd Revert "Remove deprecated PostgreSQL database driver"
This reverts commit 168ca7f0a4.

Removing postgresql support from devstack was unnecessary
since it's not broken and not causing maintenance issues
as far as I know. The commit being reverted said that pg
support was deprecated in Pike but nothing in the docs or
commit message refer to official deprecation of postgres
support in devstack or openstack in general. Not to mention
that there are still postgres-based jobs that will no
longer work *and* the notification to the mailing list about
doing this happened *after* it was already done [1] leaving
stakeholders with no time to reply.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010196.html

Change-Id: Ie7036d37d79e6aba462b7c97f917e2e7aed108f9
2019-10-17 15:58:34 -04:00
Stephen Finucane 168ca7f0a4 Remove deprecated PostgreSQL database driver
This was deprecated for removal in Pike. It's probably time to drop it.
Note that the 'postgresql-devel'/'postgresql-server-dev-all' packages
are retained since some packages still include 'psycopg2' in their
general requirements.

Change-Id: I51e8354e99972757253ce259e6c03c91da24398c
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2019-08-26 10:16:06 +01:00
Adam Spiers bc2a88d1f4 On SUSE-based systems, check whether we have mariadb or mysql service
Older mariadb packages on SLES 12 provided mysql.service.  The newer
ones on SLES 12 and 15 use mariadb.service; they also provide a
mysql.service symlink for backwards-compatibility, but let's not rely
on that.

Change-Id: Ife6bd007ba30af0b77d44832b19d518034bdb12b
2019-01-25 00:57:41 +00:00
Dirk Mueller 297a50ac86 Convert to openSUSE Leap 15.0 platform testing
Leap 15.0 has been released May 25th, 2018 (see
https://en.opensuse.org/Portal:15.0 ) and we'd like to
transition devstack against it and remove Leap 42.3 from
the testing matrix. Leap 15.0 is newer than Leap 42.3 as
the numbering schema of openSUSE was changed.

Co-Authored-By: Antonio Ojea <itsuugo@gmail.com>

Change-Id: I078f9a2580160c564c33e575008516f5e92239d6
2018-10-08 07:09:21 +00:00
Jens Harbott dc7b429463 Fix running with SERVICE_IP_VERSION=6
- There are some locations where we need the raw IPv6 address instead of the
  url-quoted version enclosed in brackets.
- Make nova-api-metadata service listen on IPv6 when we need that.
- Use SERVICE_HOST instead of HOST_IP for TLS_IP.

Change-Id: Id074be38ee95754e88b7219de7d9beb06f796fad
Partial-Bug: 1656329
2018-03-11 08:53:41 +00:00
Dirk Mueller 1d968d7a54 Switch to mariadb on openSUSE
The mysql-community-server is a compat provide, openSUSE uses
mariadb for quite some time. Make it futureproof in case
the compat provide goes away in the future. Cleanup
mysql service name to MYSQL_SERVICE_NAME and consistently
use it.

Change-Id: I2df7b8d8b798dfa7ceade90e0c127e0609524a8b
2017-11-08 17:03:18 +11:00
Yuval Brik 13e81ad1cf Fedora mariadb: disable cracklib
In Fedora mariadb, cracklib has been enabled [0] in order to verify the
password strength.
Disable cracklib in Fedora devstack in order to allow simple passwords
in dev environments.

[0] https://src.fedoraproject.org/cgit/rpms/mariadb.git/
    commit: 9442da192282aa74f43e86c96202109a173bbaba

Change-Id: I2d5e965f0f19f86992794eec78134e862899c931
2017-06-23 10:32:16 +03:00
Jens Rosenboom 4b59fbb857 Revert "put mysql on a memory diet"
The diet seems to be too strict, jobs failing with "out of sort memory". Needs more investigation before resubmitting.

This reverts commit 1e66388c5f.

Change-Id: Ic10effaaf047eb3527082baab889772c5e57fa90
2017-03-15 21:58:48 +00:00
Amrith Kumar 1e66388c5f put mysql on a memory diet
We propose several MySQL configuration parameter changes (with
explanations) to reduce the memory footprint of MySQL. A demonstration
of the improvement is provided in
https://etherpad.openstack.org/p/change-438668.

As Clint provided some of the descriptions that I've used, I have
listed him as a co-author (thanks Clint). Let this serve as a warning
to all that commetors may be enlisted :)

Change-Id: Icb2d6ea91d3d45a68ce99c817a746b10039479cc
Co-Authored-By: Clint 'SpamapS' Byrum <clint@fewbar.com>
2017-03-02 09:07:12 -05:00
Jenkins cdf79c9620 Merge "mysql: set default sql_mode to TRADITIONAL" 2017-02-24 00:59:06 +00:00
Sean Dague d9aaae95f2 Generate deprecation warning for postgresql
Change-Id: I599e6d84b70bb6a7718ae48dd0cfc91796af189e
2017-02-08 07:49:26 -05:00
Roman Podoliaka 88b84094ec mysql: set default sql_mode to TRADITIONAL
We currently use a more permisive STRICT_ALL_TABLES mode, but that's
not what modern MySQL versions default to (i.e. TRADITIONAL):

https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html#sql-mode-changes

(non-Devstack deployments will most likely use TRADITIONAL as well)

Due to the fact that we default to TRADITIONAL in oslo.db, this
produces annoying warnings on MySQL 5.7 versions we use in the gate:

    Warning: (3090, u"Changing sql mode 'NO_AUTO_CREATE_USER' is
              deprecated. It will be removed in a future release.")

https://git.openstack.org/cgit/openstack/oslo.db/tree/oslo_db/options.py#n49

Unlike STRICT_ALL_TABLES, TRADITIONAL mode includes NO_AUTO_CREATE_USER,
and MySQL emits this warning on switching it on:

https://dev.mysql.com/worklog/task/?id=8326

So we have two options here:

1) make oslo.db default to STRICT_ALL_TABLES
2) make Devstack default to TRADITIONAL

The latter seems to be more appropriate as:

1) it's what modern MySQL versions default to
2) it's what people are actually using, if they do not override the
   oslo.db default
3) it's more strict

Closes-Bug: #1652452

Change-Id: Ie6d823c9f8465ac9f2ce4825929d1a50438fab45
2017-02-07 18:01:25 +02:00
Jenkins 72af0d1cab Merge "Postgres: fix detection of existing roles" 2016-12-21 11:22:15 +00:00
Jens Rosenboom 9abb26d696 Try to set initial mysql root password also on Ubuntu
On Ubuntu nodes, devstack tries to predefine the initial mysql root
password by doing some debconf-set-selections, but these will not take
effect if the corresponding package has been installed earlier. So
just try to set it every time, like we do on other distros.

Change-Id: I2c167051fc5e53dd0ccf82a60ab085cd9cdea28d
2016-12-07 21:12:55 +01:00
Anton Merzlyakov ae61e6f3d2 Postgres: fix detection of existing roles
Role "root" it is hardcode.
In general case role name comes from  local.conf: string  "DATABASE_USER="

Change-Id: Iedfca48e04d23c313851f48d68ac40ba29340805
2016-12-05 11:10:11 +03:00
Yalei Wang 174986db21 Add pg_createcluster creating process for psql version after 9.3
postgresql 9.3 don't create /etc/postgresql and related conf file by
default. So we need start the pg_createcluster in devstack if has not
started after package installed.

Change-Id: I2b348658d79b23b5f21871b33d8023499b2fb956
Close-bug: #1552051
2016-03-03 04:08:01 +00:00
gordon chung b9201ccbbf fix postgres service name
in fedora postgresql is the service name and postgresql-server is
the package.[1]

os: Fedora release 23 (Twenty Three)
psql: psql (PostgreSQL) 9.4.5

i'm not entirely sure when this changed, but it's devstack is broken
in above environment.

[1]https://fedoraproject.org/wiki/PostgreSQL

Change-Id: Id940fed2a777ca469ce77402e1136251ba572359
2016-02-18 15:50:01 -05:00
Jenkins ddeaaf2044 Merge "Enable some serivce when on boot" 2016-01-07 01:38:08 +00:00
Zhang Jinnan 4d8c03a377 Enable some serivce when on boot
Solve the devstack ./rejoin-stack.sh when is reboot-safe in RHEL 7.
Enable mysql, postgresql, rabbitmq-server, openvswitch service when on boot.

Change-Id: I3ce9fc58ccc76092ad08314de1c3c9339ebfb3b5
Related-Bug: #1486833
2016-01-06 16:40:11 +00:00
Ian Wienand 523f488036 Namespace XTRACE commands
I noticed this when debugging some grenade issues failures.

An include of grenade/functions stores the current value of XTRACE
(on) and disables xtrace for the rest of the import.

We then include devstack's "functions" library, which now overwrites
the stored value of XTRACE the current state; i.e. disabled.

When it finishes it restores the prior state (disabled), and then
grenade restores the same value of XTRACE (disabled).

The result is that xtrace is incorrectly disabled until the next time
it just happens to be turned on.

The solution is to name-space the store of the current-value of xtrace
so when we finish sourcing a file, we always restore the tracing value
to what it was when we entered.

Some files had already discovered this.  In general there is
inconsistency around the setting of the variable, and a lot of obvious
copy-paste.  This brings consistency across all files by using
_XTRACE_* prefixes for the sotre/restore of tracing values.

Change-Id: Iba7739eada5711d9c269cb4127fa712e9f961695
2015-11-27 15:36:04 +11:00
Johan Pas 199d857442 Remove brackets from IPv6 address in mysql cfgfile
stack.sh creates a user-specific configuration file ~/.my.cnf for mysql.
If devstack is installed with SERVICE_IP_VERSION=6 option in local.conf,
the IPv6 host address was stored in the ~/.my.cnf file with square
brackets. However mysql does not use bracketing for IPv6 addresses,
resulting in 'Unknown MySQL server host' error when 'mysql' command is
run. With this patch IPv6 host address is written to ~/.my.cnf without
brackets.

Closes-Bug: #1516776
Change-Id: I27a7be8c75cf6b09b4a75dc4c9d09cd36bc5ac81
2015-11-17 01:02:54 +01:00
Julien Danjou 0eec4f86c1 database: fix PostgreSQL connection string
If all databases drivers are loaded, MySQL SQLAlchemy driver
overrides all the other one that might not have set one.
This patches fixes that.

Change-Id: If6d8d08e5b7b7c48ca012677b536d71058def6fd
Closes-Bug: #1493304
2015-09-08 16:15:33 +00:00
Ian Wienand 9c0b9f3024 Use sudo iniset to modify /etc files
The existing mysql code is wrong and not detected as failing [1], and
boto config requires work-arounds [2,3] that are all fairly ugly.  Use
-sudo argument to iniset to handle this.

[1] I24388b5de777995f92d73076524122cf599d6371
[2] I5f4c43bbbe477c570936e2e40ac05cc38febbb3f
[3] Ib7556dac9aaaf2f3c96237e0ca28ed6ae1b1b7ac

Change-Id: Iaceb8d42ce37be728adae6fd0a30a1f9d33d4029
2015-08-31 10:56:32 +10:00
Marian Horban ea21eb4f69 Remove non-ASCII characters
Change-Id: If1c68e5aab6990617519150d8aeb3f073df2ad17
2015-08-18 19:53:39 +00:00
Tony Breeds 0294ddc735 mysql: Fix mysql config
devstack attempts to set bind-address, sql_mode, default-storage-engine,
max_connections, query_cache_type and query_cache_size.

However the bash command is missing some '&&'s and was omiting
max_connections, query_cache_type and query_cache_size.

Change-Id: I24388b5de777995f92d73076524122cf599d6371
2015-07-21 14:18:38 -05:00
Brian Haley 180f5eb652 Add IPv6 support to devstack infrastructure
By default, most Openstack services are bound to 0.0.0.0
and service endpoints are registered as IPv4 addresses.
With this change we introduce two new variables to control
this behavior:

SERVICE_IP_VERSION - can either be "4" or "6".

When set to "4" (default if not set) devstack will operate
as today - most services will open listen sockets on 0.0.0.0
and service endpoints will be registered using HOST_IP as the
address.

When set to "6" devstack services will open listen sockets on ::
and service endpoints will be registered using HOST_IPV6 as the
address.

There is no support for "4+6", more work is required for that.

HOST_IPV6 - if SERVICE_IP_VERSION=6 this must be an IPv6
address configured on the system.

Some existing services, like the Openvswitch agent, will continue
to use IPv4 addresses for things like tunnel endpoints.  This is
a current restriction in the code and can be updated at a later
time.  This change is just a first step to supporting IPv6-only
control and data planes in devstack.

This change is also partly based on two previous patches,
https://review.openstack.org/#/c/140519/ and
https://review.openstack.org/#/c/176898/

Change-Id: I5c0b775490ce54ab104fd5e89b20fb700212ae74
Co-Authored-By: Sean Collins <sean@coreitpro.com>
Co-Authored-By: Baodong Li <baoli@cisco.com>
Co-Authored-By: Sridhar Gaddam <sridhar.gaddam@enovance.com>
Co-Authored-By: Adam Kacmarsky <adam.kacmarsky@hp.com>
Co-Authored-By: Jeremy Alvis <jeremy.alvis@hp.com>
2015-07-02 15:16:34 -06:00
Clint Byrum d16bfa48ee Tune mysql a bit better for concurrent operations
With PyMySQL in the projects we can expect things to happen more
concurrently now. The query cache is a hinderance to concurrency, and
more connections will be required.

Change-Id: Icfb8cdbb9ed39cfd7732ad05fe740e01c767af7b
2015-06-18 13:22:35 -07:00
Jenkins 8cd5576a17 Merge "Revert "Revert "change the default to PyMYSQL""" 2015-06-16 11:31:55 +00:00
Jenkins aadcde06a2 Merge "Install PyMySQL if used" 2015-06-12 18:36:50 +00:00
armando-migliaccio b3d8822ec8 Revert "Revert "change the default to PyMYSQL""
Some projects (Neutron) seem to be affected more than others, so we should revert this
to allow for a more selective choice of the DB driver on a per project basis.

We can re-enable the use MySQL-python just for Neutron.

This reverts commit de8d29ed8c.

Related-Bug: #1464612

Change-Id: I889f4f8b116c413b300ab9eecc7b428a9a4afb1a
2015-06-12 16:42:32 +00:00
Sean Dague de8d29ed8c Revert "change the default to PyMYSQL"
The failure rate with neutron is too high to keep this
as the default.

Related-Bug: #1464612

This reverts commit b3798af474.

Change-Id: Ie9550aeb25d472a38e3d3ef6f3711622c9221c46
2015-06-12 10:43:28 +00:00