Commit Graph

131 Commits

Author SHA1 Message Date
Ghanshyam Mann a2447137d9 Retire TripleO: remove the TripleO jobs/ref
TripleO is retired so removing its jobs/ref

- https://review.opendev.org/q/topic:%22retire-tripleo%22

Change-Id: Id6fc86b570627bc6863dde330877966da3126fe6
2024-03-20 08:23:44 +00:00
wu.chunyang 65bcf6d6e2 Set swift_api_insecure to false in trove-guest
As the devstack uses the self-signed certificate , we should
set swift_api_insecure to false by default to access the
swift service.

Change-Id: Idc2b8127dcfd589d48eb26d7624709a9fdc64029
2024-02-23 10:37:57 +08:00
wu.chunyang 48439619b5 [CI]Add postgresql jobs in zuul
This commit adds postgresql jobs in zuul and cleans up the outdated
jobs.

Change-Id: I538042576702c8ea4b83f85b08dccbd7d204e9fc
2023-11-14 13:02:30 +08:00
wu.chunyang 603a2f48f0 [CI] Add mariadb10.4 jobs in zuul.
Add mariadb10.4 tests in zuul.

Depends-On: https://review.opendev.org/c/openstack/trove-tempest-plugin/+/900498

Change-Id: I5635a7e5fb2fe8bbd6e6dfa56fe7969f3a03cf9c
2023-11-12 22:43:49 +08:00
wu.chunyang aaaad924b0 [CI] Improve CI tests for mysql
As the previous jobs always timeout in the CI system, This commit
separates the task to multiple tests to prevent timeout.

This commit also adds replication tests for mysql

Depends-On: https://review.opendev.org/c/openstack/trove-tempest-plugin/+/899777

Change-Id: Ie555be434f773ac9e48022dd2ff763a16c2aaee7
2023-11-01 14:54:07 +08:00
wu.chunyang d438839cf9 [CI]: test mysql 5.7 version instead of 5.7.29
Change-Id: I46902522645fc9b2697072dffe6373b527a2d962
2023-10-25 01:12:21 +00:00
Zuul 36f23fbd87 Merge "Allows users to specify a MariaDB version in Dockerfile" 2023-10-07 11:23:16 +00:00
Hirotaka Wakabayashi 1062b035ce Allows users to specify a MariaDB version in Dockerfile
This PR allows users to select MariaDB versions.

The original problem is that Trove fails to create backup of MariaDB-
-10.7 instances because Trove's backup script is from MariaDB-10.4[1],
which is generally incompatible with MariaDB-10.7[2].

1: https://opendev.org/openstack/trove/src/branch/master/backup/install.sh#L26
2: https://mariadb.com/kb/en/mariabackup-overview/#compatibility-of-mariabackup-releases-with-mariadb-server-releases

Story: 2010566
Task: 47275
Change-Id: I03e6b58a7b1746fef8c08d92acd7b1ccac9a46eb
2023-09-18 06:20:23 +00:00
wu.chunyang e1b72b5115 [CI]: Add support for Ubuntu Jammy
Run tests on Ubuntu Jammy in zuul.
refer to:https://governance.openstack.org/tc/reference/runtimes/2023.2.html

Change-Id: Ib12b92ed4ddd9133ad7a3d079534d02f403dbdc0
2023-09-01 15:29:04 +08:00
wu.chunyang d29efdf1a6 [CI]: fix trove api started failed
this may be caused by: https://review.opendev.org/c/openstack/devstack/+/558930

Change-Id: I7af019fc3c4ce295e4e4b4ccafae5a170a7b7833
2023-08-18 02:52:33 +00:00
wuchunyang 2f755b64b3 Add network isolation for trove
this PR adds a network_isolation config option for trove,
with network_isolation enabled, trove guest agent will
plug the user-defined port to database container by
docker host_nic network driver which is implemented in this PR.

docker host_nic network driver is a simple driver to plug host
nic to a container. this driver supports ipv4,ipv6 and dual-stack.

for more details please see the story.

story: 2010733
task: 47957

Change-Id: I35d6f8b81a2c5e847cbed3f5bc6095dc1d387165
2023-08-03 11:35:04 +00:00
wu.chunyang 7e5b54e9a3 [CI] fix devstack install failed
As we disabled docker bridge mode when install docker,
trove starts to failed to build the backup images.
because the docker build command uses the default network mode
which is bridge.  this commit changes the network mode to host when
build the backup images.

Depends-On: https://review.opendev.org/c/openstack/trove/+/888640

Change-Id: Icfb9f1cef3a7e669cdeb2a7b501c1e991c2f0ffd
2023-07-18 13:53:14 +08:00
Zuul 904e28a1bd Merge "Build backup images during the devstack installation" 2023-07-17 06:13:55 +00:00
wu.chunyang caf06bc4f7 Prevent docker from manipulating iptables
by default, Docker sets the policy for the FORWARD chain to DROP.
this behavior will block our public network connectivity.

for more details: https://docs.docker.com/network/packet-filtering-firewalls/#docker-on-a-router

Change-Id: I66408c9e65f07c3c96cabb1f7f55a312f6dc9f36
2023-07-06 06:58:11 +00:00
wu.chunyang 6f81ed8f42 Build backup images during the devstack installation
this PR builds the images during the devstack installation instead
of importing the registry tarball. so that we can test the backup
codes in zuul jobs.

Change-Id: I95c152a7abd715df72a6ac6604d7b5d9c6774b51
2023-07-02 19:18:46 +08:00
wu.chunyang 09fd606163 synchronize guest logs to controller node
use rsync to synchronize guest logs to the controller node so that we can
find the logs even after the instance is deleted

Add a guest-log-collection timer, this timer triggers
guest-log-collection service every 10 seconds

Change-Id: I264235ce7b7df585358654c46bdac4dcd743fb3d
2023-06-13 10:39:25 +08:00
wu.chunyang 06c8ed94ab Revert "Add debug log for CI tests"
This reverts commit 49c16b889b.

Reason for revert: this patch doesn't work as expected

Change-Id: I398dd7ac8c628d19b75693dcac0521346e8f5d64
2023-06-07 06:25:10 +00:00
wu.chunyang 49c16b889b Add debug log for CI tests
When the CI fails, specifically related to the Trove guest agent,
we are unable to find a way to debug it.
this patch synchronizes the guest agent logs to controller nodes.

Change-Id: I24695238075f9b1354ee4c54673d021de481a868
2023-06-06 15:43:44 +08:00
Takashi Kajinami f62d3197c4 Do not configure removed [DEFAULT] rpc_backend
This option was removed in oslo.messaging 9.0.0[1].

[1] https://docs.openstack.org/releasenotes/oslo.messaging/stein.html#relnotes-9-0-0-stable-stein-upgrade-notes

Change-Id: I11660751532dde117bbf509302fbde594e3911b1
2023-03-20 03:29:35 +00:00
wu.chunyang 1f1e2becfe Using local registry in devstack.
This patch runs a registry container during the installation of
devstack

Change-Id: I5e5a89cd9c65d1ea5b47fece985dbc044bfdec5e
2023-02-26 17:27:17 +08:00
Hirotaka Wakabayashi 6e4dd51c02 Rename api_wsgi.py to app_wsgi.py
This PR changes wsgi script filename to app_wsgi.py.

Original problem is that mod_wsgi script not found after devstack
installation. This bug is created by #854356[1].

--
[1]: https://review.opendev.org/c/openstack/trove/+/854356

Story: 2010264
Task: 46163
Change-Id: I22883c8a8bb7e2818679d667b6868587a040a602
2022-09-02 11:43:00 +09:00
wu.chunyang 44c3cdef5c Rename app.wsgi to app_wsgi.py
1. app.wsgi is a python script and should be end with py extension.
2. Generate trove-wsgi script which is needed by kolla to support
https and ipv6.
3. Ignore build directory in git
4. update docs

Change-Id: Ibbd6ecb8db25aa3a443d3f526f4cf6d418eb26f3
2022-08-25 11:10:33 +08:00
wu.chunyang 0e44c623ea Set os release to focal in devstack
trove built image failed in devstack due to the bionic
os release. in 'Z' cycle, openstack doesn't support
bionic any more.

Change-Id: I1138c4dc346d4b4bbe4d72e72ff8ae59f2fce606
2022-08-11 20:47:42 +08:00
wu.chunyang a8d9a82df1 Add a check before executing ip replace
sometimes, ROUTER_GW_IP or IPV4_ADDRS_SAFE_TO_USE
may not defined in localrc, we add a check before
executing ip replace command.

Change-Id: I48ca40cc2259e21b8a8f0a9fa763746c30f0e8ab
2022-07-19 22:33:21 +08:00
Dylan McCulloch f6327f0668 Use os-cloud trove creds
Task: 44414
Story: 2009826
Change-Id: I382b86c341de2d5c4687fbb4cc9cc48b17a03081
2022-02-03 20:00:23 +11:00
Lingxian Kong 5590ecdce0 Show user network ID for getting instance
Change-Id: Ia1e9112ae69e04f8c3e9e9d1b4a0189c743d7448
2021-08-02 14:25:54 +12:00
Lingxian Kong 02971d850b Add periodic task to remove postgres archived wal files
* Added a periodic task for postgresql datastore to clean up the
  archived WAL files.
* Added a check when creating incremental backups for postgresql.
* A new container image ``openstacktrove/db-backup-postgresql:1.1.2``
  is uploaded to docker hub.

Story: 2009066
Task: 42871
Change-Id: I235e2abf8c0405e143ded6fb48017d596b8b41a1
2021-07-23 22:16:20 +12:00
Lingxian Kong 03e5512695 Support customized database container images in DevStack
Story: 2008990
Task: 42643
Change-Id: Ie62e9aa1863d705a60b244981c9dee160d50d2dc
2021-06-19 16:47:31 +12:00
Ghanshyam Mann bcb8992f99 [goal] Deprecate the JSON formatted policy file
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc and tests.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: I790409da69df8479ad2fe152b15c32ba45067c23
2020-12-26 05:30:18 +00:00
Lingxian Kong cc57e32c48 Use nested virt node for trove functional test
Enable the jobs trove-functional-mysql and trove-tempest.

Change-Id: I930a1468fec5922916af0819698fceb94bddb2f3
2020-11-30 09:48:09 +13:00
Zuul d38defed9b Merge "Remove use of removed tail_log command" 2020-11-26 01:58:52 +00:00
Lingxian Kong da2da6969a Support --image-tags for trove-manager datastore_version_update command
image-tags was introduced in commit 1d24b65 but the implementation was
backward incompatible for the CLI `trove-manager datastore_version_update`.

This patch changes that by supporting an optional parameter `--image-tags`.

Change-Id: I583c296f15a453bdfadf10de5c678ac483cbd4dd
2020-11-26 11:06:55 +13:00
Sam Morrison c615f7ead9 Remove use of removed tail_log command
This was deprecatd in queens and removed recently from devstack

Change-Id: Iac9f6e4cff6780a3983cfb814d3745d9ac43acbb
2020-11-25 10:14:00 +11:00
Lingxian Kong 33c0b64051 Add 'version' to datastore version
This patch is part of story 2008358 implementation.

1. DB schema change
2. trove-manage datastore_version_update subcommand change
3. API change for creating and listing datastore version.

Story: 2008358
Task: 41264

Change-Id: I8069e6f4b972497f2b8be540ea35047d5fb2f9a5
2020-11-17 22:10:56 +13:00
Lingxian Kong d1af33f17b Support mysql 8.0
* MySQL 5.7 and MySQL 8.0 need different percona-xtrabackup package version.
  Added Percona XtraBackup 8 support for MySQL 8.x backup and restore.
* Construct different backup container image names for MySQL 5.7 and MySQL 8.0
  based on the default option value.
* Two docker images are uploaded for backup/restore:
  openstacktrove/db-backup-mysql5.7:1.0.0 and
  openstacktrove/db-backup-mysql8.0:1.0.0. Trove guest agent can automatically
  choose the approriate one based on the datastore version.
* Added option "secure-file-priv=NULL" in MySQL config template to fix
  https://github.com/docker-library/mysql/issues/541.
* Stop using IDENTIFIED BY in GRANT clause (also REVOKE). Starting with MySQL 8
  creating a user implicitly using the GRANT command is not supported.

Story: #2008275
Task: #41143

Change-Id: Ibdec63324b1b39ba9b8a38dbe529da17bbb06767
2020-10-23 23:50:39 +13:00
Lingxian Kong 1d24b65052 Image tags support in datastore version
Change-Id: I0e51d08515c121e3a7b0e82b6e4c4161bb4fbc4a
2020-10-09 11:43:48 +13:00
Lingxian Kong cf3e9a6e74 Support to rebuild instance by admin
Change-Id: I48f8c6f997daeb6d82ff62b3325277d86fee2147
2020-08-06 20:38:11 +12:00
Lingxian Kong aa1d4d2246 Datastore containerization
Significant changes:

* Using docker image to install datastore.
* Datastore image is common to different datastores.
* Using backup docker image to do backup and restore.
* Support MariaDB replication
* Set most of the functional jobs as non-voting as nested
  virtualization is not supported in CI.

Change-Id: Ia9c97a63a961eebc336b70d28dc77638144c1834
2020-05-27 10:31:50 +12:00
Lingxian Kong 429c39890e Support ip address type for instances
Include address type in getting instance response.

* Deprecate confip option network_label_regex as we don't reply on Nova
  to get addresses, network names don't make any sense.
* Add 'addresses' in instance API response, keep 'ip' as is but mark
  it deprecated in API doc, python-troveclient shouldn't break.

Story: 2007562
Task: 39445

Change-Id: Ia0458b5ddae8959ce29c17e444e1a51a026283cd
2020-04-26 09:56:50 +12:00
Lingxian Kong c952838a9b Support to test non-dev guest image in CI
Make the job non-voting because it may fail when the guest agent code
changes.

Change-Id: Ibd595b5b5c02cf72f1e5f05c79469cd4cb15a3ca
2020-03-16 11:10:05 +13:00
Lingxian Kong 602c4d42de Improve the function tests
- Execute test groups in serial to make sure no more than 2 database
  instance are created at the same time.
- Remove some unneccesary tests
- Remove unneeded datastore, e.g. 'Test_Datastore_1'
- Remove unsupported trovestack subcommands
- Move unsupported DIB elements to the 'deprecated-elements' folder
- Decrease default value of 'agent_call_high_timeout' to 5min
- Add initial_deplay for pooling task
- Use socket file to connect with database instead of using localhost
  IP

Change-Id: Ie5030a671fbeb453eafa6cbe04e08da7b52e33c9
2020-01-28 14:27:52 +13:00
Lingxian Kong dfe826402b Config admin clients as default
Change-Id: Ib4dc349ab7ed4193189ed6bc2bae699d51064998
2020-01-19 16:11:41 +13:00
Lingxian Kong 72524f9d85 Support PostgreSQL 12
Change-Id: Ic1d4982a479c2154d7ddcc807d863e2a37fea506
2019-12-18 22:13:32 +00:00
Lingxian Kong a0a10f0b94 Support HEALTHY status for db instance
- 'HEALTHY' means the db service is responsive, 'ACTIVE' means the db
  service is alive.
- Remove the CI job fakemodetests, but will add similar testing task in
  the future.
- Fix the periodic CI job
- Remove MongoDB and  related jobs

Change-Id: I5abe9091ba203297dc87db5fba139179166321f7
2019-12-14 12:55:56 +13:00
Lingxian Kong d93a410943 Fix delete instance
Deal with the situation when Trove fails to look for the Nova server
when waiting for the db instance ACTIVE

Change-Id: I484d45f24176c89d999864d3eb1c48860b3038bd
2019-12-10 12:18:32 +13:00
Zuul b2a31c5ac5 Merge "Support incremental backup for MariaDB" 2019-12-08 23:17:08 +00:00
Lingxian Kong 605ff34608 Support incremental backup for MariaDB
Enable the tests in CI as well.

Change-Id: Ie9706d26355bd325baf50ec874f05e6904768a1a
2019-12-08 22:27:36 +13:00
Lingxian Kong 4c551947c9 Move the iptable change from trovestack to devstack
Change-Id: I6e10744f86f19f3c29babc07fda77936c95022ff
2019-12-08 18:50:42 +13:00
Lingxian Kong 8fc0b7695d Build reusable Trove guest image for dev
When building the guest image for dev_mode=true, the controller IP
address is injected to the image in order for the guest-agent to
download Trove code during initialization. As a result, we have to build
guest image each time in the Trove CI because the image build relies on
the devstack host IP address.

If we could remove the dependency, we can build the image(in
devmode=true) for a specific datastore once and use that image for all
related CI jobs, which could save some time for Trove CI.

Fix the current CI issue as well.

Change-Id: If23f4f179a6ab72cfb35e4c45d55142fedb76498
2019-12-05 00:45:56 +13:00
Lingxian Kong fc0c4fff4c Rename devstack variable TROVE_DISABLE_IMAGE_SETUP
Rename TROVE_DISABLE_IMAGE_SETUP to TROVE_ENABLE_IMAGE_BUILD to avoid
double negative definition.

Change-Id: Iede80c39bb2a7aaf3c0453bd67f85375fa5ee76c
2019-10-11 14:35:12 +13:00