As the previous jobs always timeout in the CI system, This commit
separates the task to multiple tests to prevent timeout.
This commit also adds replication tests for mysql
Depends-On: https://review.opendev.org/c/openstack/trove-tempest-plugin/+/899777
Change-Id: Ie555be434f773ac9e48022dd2ff763a16c2aaee7
this PR adds a network_isolation config option for trove,
with network_isolation enabled, trove guest agent will
plug the user-defined port to database container by
docker host_nic network driver which is implemented in this PR.
docker host_nic network driver is a simple driver to plug host
nic to a container. this driver supports ipv4,ipv6 and dual-stack.
for more details please see the story.
story: 2010733
task: 47957
Change-Id: I35d6f8b81a2c5e847cbed3f5bc6095dc1d387165
trove built image failed in devstack due to the bionic
os release. in 'Z' cycle, openstack doesn't support
bionic any more.
Change-Id: I1138c4dc346d4b4bbe4d72e72ff8ae59f2fce606
* Added a periodic task for postgresql datastore to clean up the
archived WAL files.
* Added a check when creating incremental backups for postgresql.
* A new container image ``openstacktrove/db-backup-postgresql:1.1.2``
is uploaded to docker hub.
Story: 2009066
Task: 42871
Change-Id: I235e2abf8c0405e143ded6fb48017d596b8b41a1
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also replace policy.json to policy.yaml ref from doc and tests.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Change-Id: I790409da69df8479ad2fe152b15c32ba45067c23
Significant changes:
* Using docker image to install datastore.
* Datastore image is common to different datastores.
* Using backup docker image to do backup and restore.
* Support MariaDB replication
* Set most of the functional jobs as non-voting as nested
virtualization is not supported in CI.
Change-Id: Ia9c97a63a961eebc336b70d28dc77638144c1834
- Execute test groups in serial to make sure no more than 2 database
instance are created at the same time.
- Remove some unneccesary tests
- Remove unneeded datastore, e.g. 'Test_Datastore_1'
- Remove unsupported trovestack subcommands
- Move unsupported DIB elements to the 'deprecated-elements' folder
- Decrease default value of 'agent_call_high_timeout' to 5min
- Add initial_deplay for pooling task
- Use socket file to connect with database instead of using localhost
IP
Change-Id: Ie5030a671fbeb453eafa6cbe04e08da7b52e33c9
Allow the cloud admin to control the security groups on the management
port of Trove instance, a new config option `management_security_groups`
is introduced for that purpose.
Change-Id: I4b22b87d37792be700d4ec7f78a7ea479ddb5814
Story: 2006466
Task: 36395
Since Trove already supports to specify a Nova keypair when creating
instance for management convenience, devstack needs to be changed to
create the management keypair and add to Trove config file.
One extra change in this patch is to use a single config file for Trove
API, task-manager and conductor.
Change-Id: I1e6c4f4305104815bdf89b31776a4955de61bc89
Story: 2005429
Task: 30463
Changes to get jobs working
1) After [1] devstack no longer changes the ownership of the whole
/opt/stack tree to the stack user unconditionally. Switch to the
stack user when running integration test.
2) Add bindep.txt file[2]. The default fallback file is not installed
anymore and therefore a bindep.txt file is needed to add install
additional packages.
3) Use trovestack script rather than devstack to build image so many
global variables could be used for consistency. By default, devstack
won't build image.
4) Remove the tools/test-setup.sh as it is not used any more.
5) Instance upgrade test keeps failing in CI for some reason, although
it's always passed on my local environment. In order not to block
other patches, skip the instance upgrade tests temporarily.
[1] https://review.opendev.org/203698
[2] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006888.html
Change-Id: I35e17afb9e827b1fead9d28dbf32f11ce4034a9b
The service tenant deployment model means Trove creates most of the
resources(vm, volume, security group, etc.) relating to a database
instance in the Trove service tenant rather than the end user.
With this deployment model, most of the related resources behind the
scenes are invisible to the user, which is a more secure deployment
model for either private or public cloud provider.
DevStack should follow this model as it will be recommended for Trove
deployment in production.
Changes included in this patch that are necessary in order to make that
happen:
- Add 'admin' role to Trove service user(username: trove, project:
service) in DevStack.
- Create Trove management network resources for Trove service user in
DevStack.
- Enable Trove remote client configuration by default in DevStack.
- Mainly use alt_demo user in alt_demo project for integration tests,
config trove user as the admin role user in integration tests.
- Disable the module related tests(module_groups) for now because of no
use cases but need effort to fix all the failed tests in the service
tenant model.
Story: #2005445
Task: #30489
Change-Id: I2efb69d3d50344914a875b773f62a227dba2ccaf
Trove implies strict network requirements. All Trove instances must be
launched in a network that is connected to the management network so
that Guest Agents residing in the data plane could communicate with
control plane services, e.g. RabbitMQ and Trove Conductor. Trove
Devstack plugin creates a management network bound to management physnet
(trove-mgmt), as well as tenant network (alt-private) connected to it
via a router. Consequently, Trove instances deployed in the tenant
network can easily access the management services mentioned above.
However, these networks are visible only from the perspective of the
tenant that created the networks.
Since Tempest does not provide an easy way to create a similar
management network, and as such does not enable creating tenant network
for Trove instances automatically, this change adds --share flag to
networks created by Devstack so that Tempest could reuse them.
Change-Id: I8c642ab6150858478c48291a51675ef9d15dafaa
Signed-off-by: Bartosz Zurkowski <b.zurkowski@samsung.com>
Main goal is to have a working vm image registered in
trove after devstack installation.
It is a first step towards trovestack-redesign as described
in https://etherpad.openstack.org/p/trovestack-redesign
It is enabled by default during devstack installation with
trove enabled plugin, but it can be skipped by setting
DISABLE_TROVE_IMAGE_SETUP=TRUE
Implementation details:
* export environmental variables for diskimage-builder
* build an image with disk-image-create script
* register the newly created image in trove
* add tripleo-elements information to default settings
Change-Id: I6e57890c5b6fb65e9b8c63363f92e5a7c70e523e
Signed-off-by: Dariusz Krol <d.krol@samsung.com>
TEMPEST_SERVICES global variable is not supported
by devstack since long back.
- I380dd20e5ed716a0bdf92aa02c3730359b8136e4
- I9c24705e494689f09a885eb0a640efd50db33fcf
Service availability of tempest known services will be
set by devstack with local check.
- I02be777bf93143d946ccbb8e9eff637bfd1928d4
This commit removes the unused TEMPEST_SERVICES setting
Related-Bug: #1743688
Change-Id: Iccbc833d821a2d7b2bc34860170ce1817fbecead
Ubuntu Trusty continues to get MySQL 5.6.
The selection of the MySQL version is based on the version
of Ubuntu that is installed. If the script trovestack is
invoked on Xenial, the version that is chosen is 5.7. If
it is invoked on Trusty, then version 5.6 is chosen.
The only thing that is eliminated is the dubious combination
of MySQL 5.6 on Ubuntu Xenial. This combination is probably
not supported by Ubuntu as it is down level from their
standard offering is probably not tested anywhere.
Note Xenial does not provide the 5.6 version of mysql. This
was accomplished by initializing the Apt repo with the
Trusty software repository.
The size of the root file system is increased from 3 to 4 GBs
as the Mysql Xenial image does not fit in 3 GBs. This has an
impact on the flavors that are used by Trove for testing as the
name of the flavor includes the size of the root file system.
This is turn caused a change to each of the db specific test
config files as the trove falvors are referenced in them.
Change-Id: I4f4e497208b8f4728580e48239a8ae208e0a96dd
We already had default rule in code, so we should not
still define all of them again in policy file.
Besides, we should you yaml format for now instead json.
Another thing, we don't need to config policy file in
Devstack enviroment.
Change-Id: I783ba51695271d358764557899fe91e84620556d
Increase the timeout for out of process commands called
by the guest agent in the devstack plugin.
Change-Id: Ic8fec30a829ca828bd7fdef18e65fa2cb59f73c2
Re-enable the integration tests.
Integration tests were temporarily disabled under
9ec134996a to avoid the
circular dependency with python-troveclient and allow
the merging of the python-troveclient changes necessary
to complete the fixing of the gates.
This also increases the TROVE_AGENT_CALL_HIGH_TIMEOUT to
help the upgrade test pass. The upgrade test has an agent call
that takes longer than the current timeout when running in
the gate on Nova compute with virt_type set as qemu.
Change-Id: I8459cd64cdf6107e537f4e7c793c3626fa4bf9f9
Issue 1:
The os-testr 1.0.0 release had a couple of required config changes due
to it's internal usage of stestr.
This change to stestr changed the way tests were discovered by
os-testr and as a result the unit test run was picking up tempest
tests.
A regex is added to the py3base environment call of ostestr because
the use of --serial and --blacklist-file together is broken in stestr
and adding the regex parameter allows the blacklist-file to be
processed. The stestr issue is documented here [1].
Issue 2:
Cache dirs for PKI tokens have been removed for all services in
devstack under I5680376e70e74882e9fdb87ee1b95d5f40570ad7.
We must also remove the use here to pass the right parameters to
configure_auth_token_middleware.
Issue 3:
Keystone V2 APIs have been removed. When creating Nova and Glance
clients, the test code was either hard coding v2 Keystone or not
providing enough information for the V3 auth.
Issue 4:
Oslo context has deprecated parameters such as 'tenant', has removed
them from its constructor and is using a rename decorator to handle
them. As such, the code and test case to check for unrecognized
parameters to TroveContext and Context is erroneously removing the
tenant parameter. Oslo context has also changed the from_dict method
since the original code to remove parameters was introduced into
Trove. The new method signature and code should already provide most
or all of the protections against incompatibility the original code
was attempting to provide. The fix for this issue is to change
TroveContext's from_dict method to use the kwargs to handle its own
__init__ parameters and be more in line with what Nova is doing in
its RequestContext subclass.
Issue 5:
Jobs run as jenkins on Zuul v2 but run as user zuul on Zuul v3.
Issue 6:
Ignore one case of pylint E1101 in the Ceilometer notification
code base.
[1] https://github.com/mtreinish/stestr/issues/103
Change-Id: Ic55187b0d73d4c572d7f8332882b4f455a6177c8
Change I3f020b6bcb1b9bf6d18a3b4f738c13cccd1bbff8 in
python-troveclient 2.11.0 changed the
troveclient.compat.client._logger variable to be a LOG
variable.
I have no idea how this hasn't been breaking the Trove
API examples CI job since python-troveclient 2.11.0 was
released. Maybe it has an no one has noticed or cared to
fix it.
Anyway, this adds hasattr checking in the test code to
set the log level on the correct variable based on which
version of troveclient is being used.
Also - no idea why setting the log level in the client
for these API tests is even necessary, but this dates back
to a change in 2014 so who knows. Not me.
--
This also fixes bug 1708171 by making nova run in
singleconductor mode so the affinity/anti-affinity
scheduling tests work. Trove CI is blocked by both
changes so they have to go together.
Change-Id: Iaf00fc55336a8049c8303b8fa2849df2366115e6
Closes-Bug: #1708190
Closes-Bug: #1708171
This change enables behind mod-wsgi as part of the community pike goal
goal-deploy-api-in-wsgi.
The change includes:
- the wsgi script files to run trove api under apache
- updates to the devstack plugin
- a basic deploy doc which explains this change
Change-Id: Icdd39b47a1be426e87a5d09f9e9d567af1974a9c
Depends-On: I3d6f6649430ee40879de15fee0b215dc32e8b666
Closes-Bug: #1681478
The Oslo Policy library provides support for RBAC policy
enforcement across all OpenStack services.
Update the devstack plugin to copy the default policy file
over to /etc/trove in the gate environments.
Note: Not adding a rule for 'reset-password' instance
action as that API was discontinued years ago
and is now just waiting for removal (Bug: 1645866).
DocImpact
Co-Authored-By: Ali Adil <aadil@tesora.com>
Change-Id: Ic443a4c663301840406cad537159eab7b0b5ed1c
Implements: blueprint trove-policy
Now that devstack uses Neutron by default (and so does the gate)
we need to create a private network and subnet in the alt_demo
tenant for the int-tests to work.
(see https://review.openstack.org/#/c/355492 )
Note that we will not use this network to start as some issues
still need to be sorted out.
The tcp_ports configuration was also moved to facilitate both
network styles.
The 'PUBLIC_INTERFACE=eth0' line was also moved to be in just
nova networking as it is not necessary for Neutron (and caused
the network to become unstable). This should allow it to
continue working for backwards compatibility if required.
A number of changes to code and/or tests were also required.
Since these are specifically tied to the switch to neutron
(as they are equally valid for nova-net) they have been
made separately and referenced below in the 'dependencies' list.
This should also facilitate easier reviews and make it
possible to backport any of the fixes if required.
The cluster tests for PXC and Redis were turned off until the
networking issues with these two datastores can be sorted out.
Merged dependencies: I136141b3db78b7a0a1804bb5e3d4ca5cf4bc3bb3
I47cbe9e899cb0767c377d184f123e38eda5b6a05
I2ede074eb9bdf26420750f19f3aa4b8d057c5d7d
I6aff776680c9c0632d8965c3ed70ed26db0a63f2
I54edc4acac09ca2278f525c08ad0d87576f0549e
Iab530fe503336c83e9b948c2c008d371a8030a1e
I6464c0031ba5b504ff103d8f62093d252deb87bb
We need to have the new variable IPV4_ADDRS_SAFE_TO_USE defined:
Depends-On: Ic6720677a4a496d246a8f9be5ede1a53821f56dc
Change-Id: I5c1714b7839b2736c50f2daa2f4506c4006815a1
Closes-Bug: #1613889
At present if the Trove plugin is enabled and Horizon is enabled,
the Trove dashboard still needs to be enabled (which is currently
done by redstack). This should be handled by the Trove plugin
itself.
With this fix, if Trove and Horizon are installed the dashboard
will be automatically set up.
Some variable names were modified slightly as well to bring
them in line with the rest.
The cleanup code was also moved to the 'stop' section, which is
where it probably belonged right from the beginning.
Depends-On: I71cf27a57e8d2cd141928f26ac01bd5ca14c723f
Change-Id: I6586801bff4282e3bbf11e45e4f171fffcab285e
Partial-Bug: #1593806
Many configuration values are hard-coded in the Trove devstack
plugin, however they would be better served letting the
user/developer decide what limits to impose.
The following variables were added to facilitate this:
TROVE_MAX_ACCEPTED_VOLUME_SIZE
TROVE_MAX_INSTANCES_PER_TENANT
TROVE_MAX_VOLUMES_PER_TENANT
TROVE_AGENT_CALL_LOW_TIMEOUT
TROVE_AGENT_CALL_HIGH_TIMEOUT
TROVE_RESIZE_TIME_OUT
TROVE_USAGE_TIMEOUT
TROVE_STATE_CHANGE_WAIT_TIME
These values are only inserted if they are set. If not set,
the Trove defaults are used instead.
Change-Id: I2c568fdfa05064682f372b996a5aebcdd4e93ef3
Using the Trove devstack plugin was dependent on having the
python-troveclient also enabled. This change modifies the plugin.sh
file to install the client correctly, thus removing the need of
separately enabling a plugin for it.
Variables were set up in the devstack settings file, which can be
overridden as desired.
A README.rst file was added with instructions on how to set up the
plugin in devstack.
Change-Id: If5a76360249827c19759a5c90dc00f70c59c32cc
Ghanshyam Mann