Commit Graph

206 Commits

Author SHA1 Message Date
Takashi Kajinami 3a69bd353b Remove extras and enum34
Trove does not use extras and it was removed from the top-level
requirements by 2cc36175c8 .

Also enum34 is no longer required because now the minimum supported
python version is 3.8 .

Change-Id: I62c04ccc3c8157a86fa8b5a307983ba0796111b0
2024-05-07 12:54:12 +09:00
wu.chunyang 93f35c7f04 Get rid of nose and Proboscis
they are not maintained for a long time, and trove already switch
the tests to templest.
This commit removes the usage of them. more details see story.

Story: 2010882
Task: 48606

Depends-On: https://review.opendev.org/c/openstack/trove/+/914228

Change-Id: Ie021e0bd708cf8ed13853dea632312af38190205
2024-03-28 07:07:15 +00:00
Sean Mooney 3367c25e77 stop using passlib
passlib is unmaintained and has not had a release since 2020
a recent bcrypt release just broke passlib
see https://github.com/pyca/bcrypt/issues/684

trove's use of passlib is pretty tirval so this change
just removes it as a depency and delegate the random password
generation in trove.common.utils to the generate_random_key
function in the trove.common.crypto_utils module

Change-Id: I6b6c64147c627025d5f89db6032d1c54445df94f
2024-03-26 14:46:55 +00:00
Ghanshyam Mann a2447137d9 Retire TripleO: remove the TripleO jobs/ref
TripleO is retired so removing its jobs/ref

- https://review.opendev.org/q/topic:%22retire-tripleo%22

Change-Id: Id6fc86b570627bc6863dde330877966da3126fe6
2024-03-20 08:23:44 +00:00
Hirotaka Wakabayashi ab3563d621 Fixes the DIB script
This PR adds the "-y" option to the "add-apt-repository" script to
prevent from asking the yes/no question while installing openstack
automatically.

Change-Id: Ic6c86b95da28c4a3f2ab0a4c918e291a7e1b9bb3
2024-01-30 03:06:44 +00:00
wu.chunyang e1b72b5115 [CI]: Add support for Ubuntu Jammy
Run tests on Ubuntu Jammy in zuul.
refer to:https://governance.openstack.org/tc/reference/runtimes/2023.2.html

Change-Id: Ib12b92ed4ddd9133ad7a3d079534d02f403dbdc0
2023-09-01 15:29:04 +08:00
wuchunyang 2f755b64b3 Add network isolation for trove
this PR adds a network_isolation config option for trove,
with network_isolation enabled, trove guest agent will
plug the user-defined port to database container by
docker host_nic network driver which is implemented in this PR.

docker host_nic network driver is a simple driver to plug host
nic to a container. this driver supports ipv4,ipv6 and dual-stack.

for more details please see the story.

story: 2010733
task: 47957

Change-Id: I35d6f8b81a2c5e847cbed3f5bc6095dc1d387165
2023-08-03 11:35:04 +00:00
wu.chunyang caf06bc4f7 Prevent docker from manipulating iptables
by default, Docker sets the policy for the FORWARD chain to DROP.
this behavior will block our public network connectivity.

for more details: https://docs.docker.com/network/packet-filtering-firewalls/#docker-on-a-router

Change-Id: I66408c9e65f07c3c96cabb1f7f55a312f6dc9f36
2023-07-06 06:58:11 +00:00
wu.chunyang 09fd606163 synchronize guest logs to controller node
use rsync to synchronize guest logs to the controller node so that we can
find the logs even after the instance is deleted

Add a guest-log-collection timer, this timer triggers
guest-log-collection service every 10 seconds

Change-Id: I264235ce7b7df585358654c46bdac4dcd743fb3d
2023-06-13 10:39:25 +08:00
wu.chunyang 06c8ed94ab Revert "Add debug log for CI tests"
This reverts commit 49c16b889b.

Reason for revert: this patch doesn't work as expected

Change-Id: I398dd7ac8c628d19b75693dcac0521346e8f5d64
2023-06-07 06:25:10 +00:00
wu.chunyang 49c16b889b Add debug log for CI tests
When the CI fails, specifically related to the Trove guest agent,
we are unable to find a way to debug it.
this patch synchronizes the guest agent logs to controller nodes.

Change-Id: I24695238075f9b1354ee4c54673d021de481a868
2023-06-06 15:43:44 +08:00
wu.chunyang 8ea76a145a Drop bionic support
drop the support of bionic.
ref:
https://governance.openstack.org/tc/reference/runtimes/zed.html

Change-Id: I6dd126d49196af30bd5578c6aefa90fff9f1d5c1
2022-08-12 06:51:00 +00:00
wu.chunyang 3285b6dc8a Increase guest image size to 5G
trovestack built image failed due to the less disk space.
this ps increases the image size from 3 to 5G
ref:
https://zuul.opendev.org/t/openstack/build/fedba359fbc8427c886ff2b2f0767d37/console

Change-Id: I7e48d74e2e03fe168a185949b162b3dfcc6fb7d5
2022-08-12 10:19:53 +08:00
wu.chunyang 2f8f2f8d92 Add ubuntu focal support for trove guest image
story: 2009996
task: 45072
Change-Id: Ib3613119dc136c8a0d96a27c52e0bc1874065775
2022-04-19 09:49:03 +08:00
wu.chunyang 4d8e55c5fe Add image build test on Centos8 stream
build image on Centos8 stream

Change-Id: I759d482430eaf0e9304986c9fad047cbd44c09ca
2022-04-17 16:53:52 +08:00
wu.chunyang c0ad3f8c64 WIP: add guest image build check
This PR adds trove-guest-image-build check in ci.

Change-Id: If512cb7c094d14850555215d412b8e7b418c35ce
2022-04-13 22:51:20 +08:00
Lingxian Kong 1e4405c60b Fix trove guest agent logrotate
* The logrotate file owner should be root
* Log file path is /var/log/trove/trove-guestagent.log

Change-Id: I669fedc76aa79d7bfc1e4e64684adc4a89a3a15d
2021-09-30 22:27:53 +13:00
Lingxian Kong 4ce59ace17 Add package iptables-persistent inside guest instance
iptables-persistent is needed to persists the iptable settings inside
guest instance.

Change-Id: I96bb9f948dda22f50ec2a4fb5710ad9778e16b9d
2021-07-09 11:27:40 +12:00
Lingxian Kong b050996b9f Use bridge network for db container
- Changed the network mode of database container to "bridge" and exposed
  the service ports.
- Use socket file to connect with the database.
- Upgrade the backup container image for postgressql.

Change-Id: Id5b119f8a474befc3a2cd6e061bbffc4ae5f7bb6
2021-06-22 19:10:14 +12:00
Lingxian Kong 1e04b269ca Stop using pip-and-virtualenv diskimage element
ALthough diskimage-builder will solve the issue eventually, but we
still prefer to put the package installation under trove's own control.

Story: 2008559
Task: 41697

Change-Id: Ie704db8233c4b797eeefa140c4a11427280dffd9
2021-01-27 11:43:16 +13:00
Zuul 774cb66671 Merge "Improve trove guest image build script" 2020-11-29 10:49:01 +00:00
Lingxian Kong 9dd8e31516 Remove the experimental claim of dev_mode false
I am using the guest image built by dev_mode=false for some time, it's stable.

Change-Id: Ic74d10647f82044038ea9a4d99cb9a0320459675
2020-11-27 11:30:38 +13:00
Lingxian Kong a97ff84c19 Improve trove guest image build script
Also using ubuntu-minimal to decrease the image size a little bit, from 963M
to 815M in the test.

Change-Id: I6d70add380f4b61ebe89c72765612d8cb6127ccb
2020-11-26 14:17:57 +13:00
Lingxian Kong da2da6969a Support --image-tags for trove-manager datastore_version_update command
image-tags was introduced in commit 1d24b65 but the implementation was
backward incompatible for the CLI `trove-manager datastore_version_update`.

This patch changes that by supporting an optional parameter `--image-tags`.

Change-Id: I583c296f15a453bdfadf10de5c678ac483cbd4dd
2020-11-26 11:06:55 +13:00
Lingxian Kong 33c0b64051 Add 'version' to datastore version
This patch is part of story 2008358 implementation.

1. DB schema change
2. trove-manage datastore_version_update subcommand change
3. API change for creating and listing datastore version.

Story: 2008358
Task: 41264

Change-Id: I8069e6f4b972497f2b8be540ea35047d5fb2f9a5
2020-11-17 22:10:56 +13:00
Lingxian Kong d1af33f17b Support mysql 8.0
* MySQL 5.7 and MySQL 8.0 need different percona-xtrabackup package version.
  Added Percona XtraBackup 8 support for MySQL 8.x backup and restore.
* Construct different backup container image names for MySQL 5.7 and MySQL 8.0
  based on the default option value.
* Two docker images are uploaded for backup/restore:
  openstacktrove/db-backup-mysql5.7:1.0.0 and
  openstacktrove/db-backup-mysql8.0:1.0.0. Trove guest agent can automatically
  choose the approriate one based on the datastore version.
* Added option "secure-file-priv=NULL" in MySQL config template to fix
  https://github.com/docker-library/mysql/issues/541.
* Stop using IDENTIFIED BY in GRANT clause (also REVOKE). Starting with MySQL 8
  creating a user implicitly using the GRANT command is not supported.

Story: #2008275
Task: #41143

Change-Id: Ibdec63324b1b39ba9b8a38dbe529da17bbb06767
2020-10-23 23:50:39 +13:00
wangzihao e954184693 Remove six usage and basestring check
Remove basestring check.
Remove six Replace the following items with Python 3 style code.

- six.string_types
- six.int2byte
- six.indexbytes
- six.add_metaclass
- six.StringIO
- six.text_type
- six.integer_types
- six.binary_type
- six.BytesIO
- six.reraise

Change-Id: I4fb9033d152963c504ceb4d5c4d08f934ee4accb
2020-10-16 10:40:22 +08:00
Lingxian Kong 1d659d0342 Fix the image CI job
Change-Id: I08a3a5873bf883c453e86e662296f7d0c7fbaf49
2020-10-10 00:13:22 +13:00
Lingxian Kong 1d24b65052 Image tags support in datastore version
Change-Id: I0e51d08515c121e3a7b0e82b6e4c4161bb4fbc4a
2020-10-09 11:43:48 +13:00
Lingxian Kong 7aa269583c Install pip3 for image build
Change-Id: I3117cc56aa0edf2f4345784291ae8294a8ffc07a
2020-09-22 00:01:38 +12:00
Zuul 768ec34dfe Merge "Use sudo when deleting image cache dir" 2020-09-04 07:14:16 +00:00
Zuul 488d8a71e2 Merge "Remove six.PY3" 2020-09-04 04:21:02 +00:00
Lingxian Kong ec8a10e36b Use sudo when deleting image cache dir
Change-Id: I8a9fd906d6b70dfad1c38e751cadc004a5aeb13f
2020-09-04 13:58:56 +12:00
likui 5bf159de36 Remove six.PY3
The Python 2.7 Support has been dropped since Ussuri.
So remove hacking rules for compatibility between python 2 and 3.

Change-Id: Id3fd78b5d3e21579975ea0d0fc65e4f66ea4582e
2020-09-03 18:30:24 +08:00
Lingxian Kong 00a07f2423 Use pip3 instead of pip in trovestack
Change-Id: Icbbfaa07a699e1280b0d39dfbae87d7468cbb260
2020-09-02 10:11:12 +12:00
Lingxian Kong 9914bd6aad Allow to specify root password when building guest image
For testing purpose.

Change-Id: I8193f4262156ed9b0784666f5981ebd6df4d525b
2020-08-12 21:36:00 +12:00
Lingxian Kong cf3e9a6e74 Support to rebuild instance by admin
Change-Id: I48f8c6f997daeb6d82ff62b3325277d86fee2147
2020-08-06 20:38:11 +12:00
Lingxian Kong 428e3644f8 Support image type for guest image building
Change-Id: I8d640c47f00446d7f912ece9892b17585cde3add
2020-06-12 22:41:23 +12:00
Hervé Beraud 816a7bca58 Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: I0adf51fb6a54c323c3d7a26ea62c23184a99933d
2020-06-02 21:06:08 +02:00
Lingxian Kong e2b26fdd7d Trove doc and CI
Change-Id: I49bd6c824b837d22d9f3945a3e8c2343c831b352
2020-05-28 22:53:17 +12:00
Lingxian Kong aa1d4d2246 Datastore containerization
Significant changes:

* Using docker image to install datastore.
* Datastore image is common to different datastores.
* Using backup docker image to do backup and restore.
* Support MariaDB replication
* Set most of the functional jobs as non-voting as nested
  virtualization is not supported in CI.

Change-Id: Ia9c97a63a961eebc336b70d28dc77638144c1834
2020-05-27 10:31:50 +12:00
Lingxian Kong 523d66e8fd Fix CI job
* Fix pep8: ambiguous variable name
* Install Python 3.7 for trove guest image. oslo.concurrency requires
  python 3.6 or newer. See
  https://bugs.launchpad.net/tripleo/+bug/1861803
* Mark tempest job non-voting temporarily because of some tempest bugs.

Change-Id: I6d316779cc7220a855ce187437056b667bbe1f75
2020-05-25 17:37:38 +12:00
Lingxian Kong 7d78c74158 Add python-troveclient in requirements.txt
Change-Id: I2bd0d81171df85c05423e19ab2e4afa82fcf880e
2020-03-15 20:57:39 +13:00
Lingxian Kong 602c4d42de Improve the function tests
- Execute test groups in serial to make sure no more than 2 database
  instance are created at the same time.
- Remove some unneccesary tests
- Remove unneeded datastore, e.g. 'Test_Datastore_1'
- Remove unsupported trovestack subcommands
- Move unsupported DIB elements to the 'deprecated-elements' folder
- Decrease default value of 'agent_call_high_timeout' to 5min
- Add initial_deplay for pooling task
- Use socket file to connect with database instead of using localhost
  IP

Change-Id: Ie5030a671fbeb453eafa6cbe04e08da7b52e33c9
2020-01-28 14:27:52 +13:00
Lingxian Kong 72524f9d85 Support PostgreSQL 12
Change-Id: Ic1d4982a479c2154d7ddcc807d863e2a37fea506
2019-12-18 22:13:32 +00:00
Lingxian Kong 4c8482095d Fix Trove periodic CI jobs
Change-Id: I11456aaeb2d417dbae6bdb0e51944aa122c521e0
2019-12-08 21:11:44 +00:00
Lingxian Kong 4c551947c9 Move the iptable change from trovestack to devstack
Change-Id: I6e10744f86f19f3c29babc07fda77936c95022ff
2019-12-08 18:50:42 +13:00
Zuul 94331620a3 Merge "Support pip3 and run on guest-agent service for redis" 2019-12-06 09:57:17 +00:00
Lingxian Kong 637c900b87 Mark CI job trove-scenario-mariadb-single voting
Fix the current CI issue as well.

Change-Id: I7daafa68a03a3bf88c3651b6cab8c58cf5256b29
2019-12-06 11:21:45 +13:00
Nguyen Van Trung d5549f30cf Support pip3 and run on guest-agent service for redis
This patch will solve some following items:
- Support python3/ pip3
- Permit redis is running with guest-agent service

Tested with standalone, replicate and cluster successfully.

Change-Id: I81b72fae1c31ce2995232b52460d0ad0d6f894be
2019-12-05 09:35:12 +00:00