This ensures that the options from the oslo.db library are picked up
by oslo-config-generator.
Also, update .gitignore so that the sample config file is ignored by
git.
Change-Id: I6ab051fd1c8034d9697b1caab9c1fda64a3275c3
The healthcheck middleware from oslo.middleware library is now commonly
enabled in OpenStack services, for service healthcheck (often used by
load balancers such as haproxy). This enables the middleware so that
operators can use the healthcheck endpoint more easily. This also
replaces the oslo.middleware config entry point by more specific sub
entry points, to avoid rendering options from unused middlewares such
as sizelimit.
Change-Id: If3296e651e7f20a3f5314c085cc99ce4b004c065
Python 2.y and Python <= 3.2 are no longer supported, so we no longer
need to maintain logics for such old versions.
Also the check_python_version method is removed by this change, because
the required python version is now enforced by setup.cfg.
Change-Id: Ifc5e618a791bdf591e0ec61075089dd992c73e0c
Config reference has been dropped for some reason several releases before.
We return configuration reference along with policy to the docs.
Change-Id: I0927466d18eca5572932fcf9e2ca6349b11ca847
Significant changes:
* Using docker image to install datastore.
* Datastore image is common to different datastores.
* Using backup docker image to do backup and restore.
* Support MariaDB replication
* Set most of the functional jobs as non-voting as nested
virtualization is not supported in CI.
Change-Id: Ia9c97a63a961eebc336b70d28dc77638144c1834
* Hard delete the datastore_configuration_parameters table record.
* Make 'datastore_version_id' nullable for 'instances' table.
* Check if the datastore version is still being used before removal.
Story: 2007563
Task: 39451
Change-Id: I84e4a31f14f9327cc01ff2d699167d91112e1565
Changes to get jobs working
1) After [1] devstack no longer changes the ownership of the whole
/opt/stack tree to the stack user unconditionally. Switch to the
stack user when running integration test.
2) Add bindep.txt file[2]. The default fallback file is not installed
anymore and therefore a bindep.txt file is needed to add install
additional packages.
3) Use trovestack script rather than devstack to build image so many
global variables could be used for consistency. By default, devstack
won't build image.
4) Remove the tools/test-setup.sh as it is not used any more.
5) Instance upgrade test keeps failing in CI for some reason, although
it's always passed on my local environment. In order not to block
other patches, skip the instance upgrade tests temporarily.
[1] https://review.opendev.org/203698
[2] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006888.html
Change-Id: I35e17afb9e827b1fead9d28dbf32f11ce4034a9b
The older version of pylint being used does not work correctly under
python 3. In order to be able to update the pylint job to run under
python 3, we need to update the tool.
This patch updates to the latest version at this time. It also updates
and pins astroid, which was previously capped. Using a pin instead of
a cap should let us avoid issues with new releases while being
specific about which version to actually use.
Disable not-callable because that appears to be a new rule that is
confused by the use of properties to access things that are set to
callables.
Co-Authored-By: Fan Zhang <zh.f@outlook.com>
Co-Authored-By: Marcin Piwowarczyk <m.piwowarczy@samsung.com>
Change-Id: I65705804b222dcd30a653fe10be3d823fa6143ff
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Starting with Python 3.6, invalid escape sequences in string literals
are now deprecated[1]. This influence also automatic style checkers like
pycodestyle which starting with 2.4.0 complains about invalid escape
sequences (W605)[2].
Let's fix all those warnings at once by using raw strings where possible
and adding additional \ where not.
Footnotes:
1 - https://docs.python.org/3/whatsnew/3.6.html#deprecated-python-behavior
2 - https://github.com/PyCQA/pycodestyle/pull/676
Change-Id: I009a366fd8342edfd30890df6fe8e1fca88bf3cc
Signed-off-by: Krzysztof Opasiak <k.opasiak@samsung.com>
The messages for many of the errors reported by pylint have been
extended to include "suggestions" (especially in the case of undefined
variables or members). Rather than update all of the individual
messages in the configuration file, just treat the messages as
prefixes.
Change-Id: I1f11c50e39dbd43c4a9b54ccaaccf03f5ff07aca
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
When disabling root, there is no need to call guestagent to check
whether the root user is ever enabled. Root hisotry table should
be used for this purpose.
As datastore specific root controller of MySQL/Cassandra/PostgreSQL
were created only for the '_find_root_user' which were calling
guestagent to find root user, these controllers are removed and
'DefaultRootController' is used instead.
RedisRootController is also updated as it didn't do this check
previously.
Unittests directory structure is also slightly changed. It's more
clear to use similar directory hierarchies for testing and source
code, e.g.
trove/extensions/common/service.py
trove/tests/unitests/extensions/common/test_service.py
Change-Id: I9faac61d9650347b51f23e8fcaf5a92aed5fbf93
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
A new entrypoint in setup.cfg and a config file are added for
using olso.policy helper script to generate the sample file.
A new tox target also is added to simplify the environment
setting up. Now policy sample file can be generated
automatically, so the in-repo sample file is no longer needed.
Co-Authored-By: Andrew Laski <andrew@lascii.com>
Partial-Implements: blueprint policy-in-code
Change-Id: Ic336fa154ccc05b5e9db3a8e751a484b1cc5aa9c
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
In py3.4 notice method readfp will removed in the future,so use
method readfile to replace it.This warning is only in py3 and
above.Py2,configparse do not have.Actually in py3,This function
is finally pointed to the readfile function
warnings.warn(
"This method will be removed in future versions. "
"Use 'parser.read_file()' instead.",
DeprecationWarning, stacklevel=2
)
self.read_file(fp, source=filename)
Change-Id: I22a2dca71008469a8c8b7afcc0934f3b4adcdae4
Fixed the module-instances command to return a paginated
list of instances. Also added a --count_only flag to the
command to return a summary of the applied instances
based on the MD5 of the module (this is most useful
for live_update modules, to see which ones haven't been
updated).
Also cleaned up the code a bit, putting some methods
into files where they made more sense (and would cause
less potential collisions during import).
Change-Id: I963e0f03875a1b93e2e1214bcb6580c507fa45fe
Closes-Bug: #1554900
Earlier code introduced in[1] aimed to handle the situation where a
context object serialized and sent over a message queue as a
dictionary is received by a version of code that doesn't necessarily
understand that context object.
That code relied on the fact that vars(TroveContext).keys() was a
dictionary that matched the full set of parameters that you can send
to the initializers for TroveContext() and RequestContext().
Recent changes in oslo.context [2] and [3] broke that
assumption. vars() on TroveContext provided a dictionary with the
internal representation(s) that are not the same as the kwargs in
RequestContext() and the params that TroveContext used to pop.
To get around this, new code introduced here uses
oslo_utils.reflection to determine all the possible kwargs that can be
sent to TroveContext, some of which are consumed by TroveContext and
the rest of which are sent along to RequestContext().
The construct in the earlier _remove_incompatible_context_args() also
modified kwargs.keys() by pop'ing values from it while iterating over
this. Python 3 takes a dim view of this and therefore some changes
have been made to accomodate this.
A unittest has been added to ensure that the from_dict() method
properly dumps stuff it doesn't know about. The test explicitly
verifies the warning generated when a bogus argument is eliminated.
[1] 24c5e8e244
[2] 2394cff0631944a9259bfe04925e444d9f817758
[3] f25543fcc792ebf155728a91fde06e8dc4e96cea
Change-Id: I477dd29e034295e770925091c4ac6268c22ae59b
Related-Bug:#1661790
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests.
This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.
Update CONTRIBUTING for this.
See also
http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html
Change-Id: I3463e75057d0d4544f6a0212da888759ab5e171b
Needed-By: I92e6e6502c2c516babf2bf66f3514875f77c460e
Implement configuration attach and detach API for clusters.
Implement rolling strategy for applying configuration changes
(both attach and detach follow the same pattern).
1. Persist the changes on all nodes (leaving nodes in RESTART_REQUIRED state).
2. Update Trove records.
3. Apply changes dynamically via one or all node(s) if possible
(and remove RESTART_REQUIRED flag from all nodes).
Notes:
The single instance implementation has been restructured (similar to above)
such that it always leaves the instance in one of the three states:
a) Unchanged
b) Changes persisted but not applied
(Instance has configuration attached but requires restart.
It is safe restart manually or detach the group to avoid
any changes)
c) Changes persisted and applied (if possible)
This implemenation should always leave the cluster (and each instance)
in a consistent state.
Runtime configuration will not be changed until it is first persisted
on all nodes.
If there is a failure during step 1) the cluster is still running
the old configuration. Some instances may have new configuration
persisted, but not applied.
The cluster will not have configuration attached unless it can
be applied to all nodes.
The individual nodes will have configuration attached as soon as it is
persisted on the guest.
It is safe to retry, reapplying the same configuration on a node is
noop.
It is safe to detach. Removing configuration from nodes without one
is a noop.
It is safe to detach the configuration from individual nodes via
single-instance API.
It is safe to attach the configuration to remaining nodes via
single-instance API and rerun cluster attach to update Trove records.
If 3) fails for whatewer reason the instances are left
in RESTART_REQUIRED state.
It is safe to retry or detach configuration or restart the
instances manually.
Also fixed various minor cluster issues.
Implements: blueprint cluster-configuration-groups
Change-Id: I7c0a22c6a0287128d0c37e100589c78173fd9c1a
This is an interim commit of the changes for secure
oslo-messaging.rpc. In this commit we introduce the code for
serializers that will encrypt all traffic being sent on
oslo_messaging.rpc.
Each guest communicates with the control plane with traffic encrypted
using a per-instance key. This includes both traffic from the
taskmanager to the guest as well as the guest and the conductor.
Per-instance keys are stored in the infrastructure database. These
keys are further encrypted in the database.
Tests that got annoyed have been placated.
Upgrade related changes have been proposed. If an instance has no key,
no encryption is performed. If the guest gets no key, it won't
encrypt, just pass through. When an instance is upgraded, keys are
added.
The output of the trove show command (and the show API) have been
augmented to show which instances are using secure RPC communication
** if the requestor is an administrator **.
A simple caching mechanism for encryption keys has been proposed; this
will avoid the frequent database access to get the encryption
keys. For Ocata, to handle the upgrade case, None as an encryption_key
is a valid one, and is therefore not cached. This is why we can't use
something like lrucache.
A brief writeup has been included in dev docs
(dev/secure_oslo_messaging.rst) which shows how the feature can be
used and would help the documentation team write up the documentation
for this capability.
Change-Id: Iad03f190c99039fd34cbfb0e6aade23de8654b28
DocImpact: see dev/secure_oslo_messaging.rst
Blueprint: secure-oslo-messaging-messages
Related: If0146f08b3c5ad49a277963fcc685f5192d92edb
Related: I04cb76793cbb8b7e404841e9bb864fda93d06504
A method for specifying 'priority' modules plus a way to rank the order
in which modules are applied has been added. Two new attributes
'priority_apply' and 'apply_order' are available in the payload
on create and update. In addition, an is_admin flag was added as an
automatic attribute, to be set when someone with admin credentials
creates a module or updates an existing module with 'admin-only'
options. This allows better control on the driver plugin
side with regards to security concerns, etc. The attribute is
now passed in to the guest 'apply' interface for use by the driver.
All three of these attributes are stored in the Trove database.
An admin can create a 'non-admin' module by passing in --full_access
on the command line (or python interface). This will cause an
error if any admin-only options are selected.
Scenario tests have been added to verify that the modules are
applied in the correct order. The timestamp for the 'updated'
field on the guest was also enhanced to allow for fractional
seconds, since most applies take less than a second.
The issue where modules were allowed to be applied even if
they belonged to a different datastore has been fixed and
scenario tests added to check for this case.
Change-Id: I7fcd0cf12790564ba62e7d6451fff96f763e539d
Implements: blueprint module-management-ordering
This commit includes two cleanups in trove-pylint.
First, the deprecation warning is because the text.ParseableOutputFormat is
being deprecated[1] and produces a deprecation warning as shown in the
bug.
Second, while looking into 1650816, I stumbled upon 1650888 which is
also fixed. A block of code was wrongly indented and when I disabled
"-E" to generate warnings, produced a crash in pylint. If the line
generated doesn't match the regular expression, the various parameters
to the next call (to ignore()) are not set and you get unbound
variable errors.
[1] https://github.com/PyCQA/pylint/blob/master/pylint/reporters/text.py#L163
Change-Id: I43bd6569a98eeb1b326e4a0ca1addaf97c0830a5
Closes-Bug: #1650530
Closes-But: #1650888
When someone has a pylint config file that has an error such as when a
merge goes bad, loading the file fails, and pylint motors along
happily with the default file (i.e. nothing ignored) and this is
confusing. Unless you happen to catch the little error that it prints
out.
Instead, just bomb out with an error.
Change-Id: Ie94dda35652f2a8f06502af72b68be73d61fc94a
Closes-Bug: 1639040
This is an initial attempt at supporting multiple regions. It should
handle the mechanics of deploying an instance/volume to a remote
region. Additional changes may be required to allow the guest
agent on the instance to connect back to the originating region.
Co-Authored-By: Petr Malik <pmalik@tesora.com>
Change-Id: I780de59dae5f90955139ab8393cf7d59ff3a21f6
Removed downgrade from all existing migrations.
Modified a test that verifies that no migrations have a downgrade.
Modified test _walk_versions to only do upgrades.
Removed exceptions from pylint.config
Related cross-project spec: https://review.openstack.org/152337
Closes-Bug: #1434103
Change-Id: I9a7c87ae3f4e2eff3a4a6df881d086d52062dbba
Postgresql has been implemented differently from
all other guest agents.
This has been causing difficulties with maintenance
and future development of Trove design.
It has been decided (Newton midcycle) to bring the guest agent
in line with others.
This patch set merges all Postgres service modules into
App/Admin classes.
This also resolves 20 pylint issues related to mixin usage.
Change-Id: I1b12a5296c59e9d3b08bcf34ac196d16189d525e
Since the config is not in a deterministic order makes it hard to
compare two config's and see what changed. Personally, I'm not
positive I understand this use-case; i.e. you have an existing config
file, you save it, and then rebuild and then diff the two files. I'd
have thought you'd just run check and the output of the tool was the
diff.
I however do see the value in sorting the file so that when someone
submits a change that includes a change to the config, reviewers can
see more easily what the change is doing.
Similarly, the output from pylint (errors) are generated one file at a
time and os.walk makes no guarantee of deterministic order. So we
should collect all errors (across all files) and then print an ordered
list for human consumption.
The intent is also to make pylint voting soon (in master). the changes
to contributing.rst and tox.ini are to make that easier. The config
file has also been sorted in place.
This change was motivated by an email exchange with Peter so I am
marking him as a co-conspirator.
The line numbers were removed from the tools/trove-pylint.config file
as these would change whenever the line numbers in the file changed
(since they are currently not being used in the comparison; they can
be re-added if deemed necessary at the cost of having every 'rebuild'
run create a different file).
The tools/trove-pylint.config was regenerated as well, since the
remaining two errors seem to be innocuous:
ERROR: trove/taskmanager/manager.py 392: E1101 no-member,
Manager.upgrade: Instance of 'BuiltInstance' has no 'upgrade' member
(new method introduced by instance upgrade; other BuiltInstance
member errors are already ignored.)
and
ERROR: trove/guestagent/datastore/experimental/postgresql/service/
access.py 80: E1101 no-member, PgSqlAccess.list_access: Instance ofi
'PgSqlAccess' has no '_find_user' member
(this is due to the fact that PostgreSQL is spread over multiple
files and pylint should cease to complain once
https://review.openstack.org/#/c/346082/ lands.)
Change-Id: I910c738d3845b7749e57910f76523150ec5a5bff
Closes-Bug: #1625158
Closes-Bug: #1625245
Co-Authored-By: Peter Stachowski <peter@tesora.com>
this is a pylint wrapper for Trove's tox tests. This commit includes a
basic infrastructure for running pylint through tox.
It also fixes some very obvious import errors that are flagged by the
tool. One is to handle missing imports for _ and _LE.
There is one instance where an exception is being raised but
trove.common.exception isn't imported, and another where an exception
is being incorrectly thrown.
A short readme is also provided.
Change-Id: I0a38f5efde3cb491f1f6c27f6c6500ab29987968
Partial-Bug: #1621636
In PY3, print should used as a function instead of a statement.
This is PY27 compatible as well.
For example, instances of the first case were replaced by the
second case:
>>> print 'help'
File "<stdin>", line 1
print 'help'
^
SyntaxError: Missing parentheses in call to 'print'
>>> print('help')
help
Partially implements: blueprint trove-python3
Change-Id: I21de65bc874d0fcfb51714857f8eaa97acb2f51d
Closes-bug: #1594741
Trove does not support Python 2.6 anymore starting with Kilo and might
not work correctly with it, so remove the classifier.
Change-Id: Ida17ce3ce76258442222b591846e1e1717119eed
* rename db_wipe to db_recreate
* make --repo-path an optional argument for db_wipe and use it
* add command line help strings for db_wipe
Closes-Bug: #1279734
Change-Id: I1a1d3ad56989026fb10f6c57bb479814dc51328e
Remove vim setting, Remove line containing:
comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4
at the top of source code files, except for files in
openstack/common.
Change-Id: I5e2aad6dca8407890fc3c2afbda541bd4f124ea8
Closes-Bug: #1229324
Zuul