passlib is unmaintained and has not had a release since 2020
a recent bcrypt release just broke passlib
see https://github.com/pyca/bcrypt/issues/684
trove's use of passlib is pretty tirval so this change
just removes it as a depency and delegate the random password
generation in trove.common.utils to the generate_random_key
function in the trove.common.crypto_utils module
Change-Id: I6b6c64147c627025d5f89db6032d1c54445df94f
H306: import in alphabeical order
Imports should be grouped in the following order:
standard library imports
related third party imports
local application/library specific imports
put a blank line between each group of imports.
https://docs.openstack.org/hacking/latest/user/hacking.html#imports
Change-Id: I2c188d27f0595ac3b2d71f4612d93829915e389c
This change checks if the user input is legal.
if illegal, we raise an InvalidValue excepiton.
Story: 2010004
Task: 45128
Change-Id: Ib81646b8f8a01fcbc31d033ec205491b76a7b755
ABCs in collections should be imported from collections.abc and direct
import from collections is deprecated since Python 3.3.
Change-Id: Ic4c1cb4d9ce7595d1e684c26229a5148b5f4befa
Significant changes:
* Using docker image to install datastore.
* Datastore image is common to different datastores.
* Using backup docker image to do backup and restore.
* Support MariaDB replication
* Set most of the functional jobs as non-voting as nested
virtualization is not supported in CI.
Change-Id: Ia9c97a63a961eebc336b70d28dc77638144c1834
- Execute test groups in serial to make sure no more than 2 database
instance are created at the same time.
- Remove some unneccesary tests
- Remove unneeded datastore, e.g. 'Test_Datastore_1'
- Remove unsupported trovestack subcommands
- Move unsupported DIB elements to the 'deprecated-elements' folder
- Decrease default value of 'agent_call_high_timeout' to 5min
- Add initial_deplay for pooling task
- Use socket file to connect with database instead of using localhost
IP
Change-Id: Ie5030a671fbeb453eafa6cbe04e08da7b52e33c9
- The users need to specify the network to create Trove instance, but
trove-taskmanager will create port in that network for Nova instance
creation. Using port gives Trove more capabilities to define how the
database service is exposed.
- Deprecate ICMP protocol for the instance.
- Restrict 'nics' parameter for creating instance.
- Add 'access' parameter for creating instance.
- Add 'public_network_id' option in order to create floating IP for the
instance.
- Do not create records for security groups, but Trove can still delete
existing instances for backward compatibility.
- Delete unreasonable Host, Account, Storage API.
Story: 2006500
Task: 36468
Task: 36466
Change-Id: I80827e1ad5e6b130cbf94c2bb7a909c44d5cf1e5
This patch adds catching loopingcall.LoopingCallTimeOut and raising
PollTimeOut exceptions.
Without this change we didn't know which exception would be thrown in
case of task timeout in function poll_until. As in the rest of the
trove code we catch only PollTimeOut exception it was possible to miss
loopingcall.LoopingCallTimeOut.
Change-Id: Ic3f8bdf99e82ac5c438354a4f3af7c6856d14f34
Functional tests are executed by trovestack script, which uses python
in default OS version which is 2.7.
This change wraps python in tox and executes trovestack int_tests in
virtual enviroment, starting them in python3.
Any future python version change could be managed by tox framework.
Change-Id: I3a849978241d3d0669ef9e1c802ff504ac3c32cb
Signed-off-by: Marcin Piwowarczyk <m.piwowarczy@samsung.com>
According to discussions on the ML, log messages should not be
translated any more. This patch also:
* removes all usage of _LI, _LW, _LE, _LC;
* updates log translation hacking rule.
ML discussions:
http://lists.openstack.org/pipermail/openstack-i18n/2016-November/002574.htmlhttp://lists.openstack.org/pipermail/openstack-dev/2017-March/113365.html
The original commit messages are kept as following to make a historic
note:
remove unwanted log translations
recent discussions on the ML have led to the decision to eliminate all
translations for messages that are destined for log files but retain
them for messages that are going to be returned as exceptions and
potentially shown to end users via an API. see [1], [2].
This change does that as follows.
1. If a string is being used to generate an exception, it will still
be translated. Still Trove used both _LE and _ for these translations,
there are some cases where _LE has been simply changed to _, and not
removed.
2. If a string is used for a logging message, remove the use of _,
_LE, _LW, _LI.
Also, I have long felt that we have had a creep of checks in the pep8
tests that apparently make the code easier to read. I strongly believe
that these kinds of "easier to read" things make sense if they are
followed across all projects and not just gratuitously added one
project at a time.
I've taken this opportunity to reduce the merge mess caused by this
change, to sync up our ignore flags with a long list of ignores from
Nova. When they made the change for removing log translation, they
could do it in an automated-way like I have because they didn't have
to deal with under and overindented lines for visual edit (E127). Same
for Cinder.
Changes 448443 [3] and 447839 [4] were inadequate because they only
addressed a little part of the problem, namely removing the use of
_LE, _LI, and _LW, and I think this is a change we don't need to
dribble in a few files at a time. The changes are straightforward and
should be taken in a single lump to make it easy to deal with the
merges coming.
[1] http://lists.openstack.org/pipermail/openstack-operators/2017-March/012887.html
[2] http://lists.openstack.org/pipermail/openstack-dev/2017-March/113365.html
[3] https://review.openstack.org/448443
[4] https://review.openstack.org/447839
Co-Authored-By: Amrith Kumar <amrith@amrith.org>
Co-Authored-By: Valencia Serrao <vserrao@us.ibm.com>
Change-Id: I5f86c982469e625997fc8bd15c6fae0fc77a8c64
There are a lot request debug logging in Trove, when some values of
headers are encoded in utf8, UnicodeEncodeError will be raised by
webob.Request. Override how webob.Request is represented will fix.
Closes-Bug: #1720121
Change-Id: I91683b8dd24262b0f643e8d2bc7886a7c03be40a
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
oslo_utils.timeutils is deprecating isotime(). In reality they are
deprecating some other things as well but Trove doesn't (currently)
use any of those things.
Much has been written on the subject of this deprecation. I think the
proposal to merely replace isotime with datetime.datetime.isoformat()
is a little simplistic. Well intentioned, but nonetheless I believe
that it is simplistic.
The primary issue I could find with oslo_utils.timeutils.isotime() was
the fact that it was naive. I think it could well have been fixed in
oslo_utils but for whatever reason(s) oslo decided not to want to go
that route.
The primary challenge from Trove's perspective is that I want to
respect the existing API contract while at the same time get an
implementation of time handling that is not identical in its flaws
with oslo_utils.timeutils.isotime().
This change set attempts to address that by making
trove.common.timeutils.isotime() that is aware. It also implements a
utcnow_aware() function that is aware.
ISO 8601 allows for four representations of timezone and those are
<time>Z
<time>[+-]hh:mm
<time>[+-]hhmm
<time>[+-]hh
Trove conventionally used the first one, even if the time wasn't
really a UTC time. That's one of the things being fixed here.
In review cp16net asked whether this change removes the 'Z' at the end
of time strings generated by the isotime() function. The answer is
NO. The new isotime() function performs identical to the old and now
deprecated function in oslo_utils.timeutils for UTC (Z) times.
There was a utcnow() function in trove.common.utils which just wrapped
datetime.datetime.utcnow(). That has been moved now to
trove.common.timeutils with the other new time related functions.
There were a couple of places in Trove where code was using
datetime.now() which was not ideal. Those have been corrected now as
well.
Unit tests have been proposed for the new routines.
Closes-Bug: #1532120
Change-Id: Ic5abf6669edd4f1a9fd62e61f437565aa887aebe
List options tcp_ports and udp_ports are lists of strings and some
with a '-' in the middle to indicate a range. To help validate the
options better a new type was introduced to oslo.config called Range.
oslo.config version 3.18.0 merged this Range type which will no
longer require the following in our project:
* utility function gen_ports because a Range of ints are returned
* test to check for proper from-to format, the type is smart so
it flips the numbers around for us. So 63000-300 returns
Range(300, 63001) 63001 because inclusion=True is set by default.
Change-Id: I63b6a865a3f3c79202dd299f6cd25dd59e182252
Closes-Bug: #1500141
For avoiding XSS vulnerabilities, bandit suggests to set
autoescape=True.
After this change the bandit issues no longer appears.
Change-Id: Ic47dadef49b4504b3bcfbdc63ea85c937aabf334
Closes-Bug: #1663417
After looking at the periodic py27 job with oslo master I found a
deprecation warning that states the following:
trove/common/utils.py:290: DeprecationWarning: the function
passlib.utils.generate_password() is deprecated as of Passlib 1.7,
and will be removed in Passlib 2.0, use passlib.pwd.genword() /
passlib.pwd.genphrase() instead.
I simply replaced the function call and arguments with genword.
Change-Id: Id3cc81678ff856b8a11ea96242b958fa0e5cef87
The current implementation of poll_until will attempt the retriever
every second and given that the call itself may take some time, this
effectively ends up being a non-existant sleep. On the under-powered
CI systems, this has an unwanted bad effect with polls running
continuously.
Changing poll_until to use the BackOffLoopingCall instead.
Change-Id: I9487d753e65c4de7753d9db19626497217c71f63
Depends-On: Id5be526a0418db5d00cbf5cbceb4993274989e72
Currently Trove supports full offline backups for DB2. In this
implementation, we have added support for full online backups for DB2
using archival logging.
Change-Id: I30b5b0b85120fd105cc3db57983b062fad5fab5a
Implements: blueprint db2-online-backup
When an error occurs in Trove, it is very difficult
to determine the cause without access to the
server logs. To make these errors available to
the end user, they are now persisted in the database
and can be viewed using the standard 'show' command.
Also fixed TESTS_USE_INSTANCE_ID test path, as it
somehow got broken over time.
Change-Id: I84ed28ee73a24a2dd6bdbf895662d26e406e9fae
Depends-On: I5d3339e9cbfd6aeb0c3ff6936fefa8dbe9e841f8
Implements: blueprint persist-error-message
Make 'default_password_length' per-datastore-property.
Set max length to 24 in Couchbase config.
Change-Id: I391bc68b44028412ed25490264cbb1e396d540d2
Closes-Bug: 1572230
The Trove code is full of numerous cases where kwargs are
defaulted to CONF values. This is an error-prone
way of setting up a default value, as python sets these one
time only. This fix accounts for that.
Change-Id: Icc0858ccb2d3e2584bf6f3d1542a7d631b251ac8
Closes-Bug: 1571076
This changeset handles the details of applying,
removing, listing and retrieving 'modules'
from Trove instances.
See https://review.openstack.org/#/c/290177 for
the corresponding troveclient changes.
Scenario tests have been extended to cover the
new functionality. These tests can be run by:
./redstack int-tests --group=module
A sample module type 'driver' - ping - is included
that simply parses the module contents for a
message=Text string and returns the 'Text' as the
status message. If no 'message=' tag is found, then
the driver reports an error message.
Due to time constraints, a few unimplemented
parts/tests of the blueprint have been triaged as bugs
and are scheduled to be fixed before mitaka-rc1.
These include:
Vertica license module driver:
https://bugs.launchpad.net/trove/+bug/1554898
Incomplete module-instances command:
https://bugs.launchpad.net/trove/+bug/1554900
Incomplete 'live-update' of modules:
https://bugs.launchpad.net/trove/+bug/1554903
Co-Authored-by: Peter Stachowski <peter@tesora.com>
Co-Authored-by: Simon Chang <schang@tesora.com>
Partially Implements: blueprint module-management
Change-Id: Ia8d3ff2f4560a6d997df99d41012ea61fb0096f7
Depends-On: If62f5e51d4628cc6a8b10303d5c3893b3bd5057e
This changeset handles the details of creating,
updating, listing and deleting Trove 'modules.'
Two new tables have been added to the Trove database:
modules
instance_modules
although the instance_modules table is at present unused.
Scenario tests have been written as well, to exercise the
new functionality. These tests can be run by:
./redstack int-tests --group=module_create
In the follow-up changeset, all module tests can be run
by:
./redstack int-tests --group=module
Since module support is available for all datastores
(controlled by a CONF option) the module test has been
added to the common modules group.
Note: Trying to do admin tasks with none admin
credentials results in an Unauthorized exception
being thrown, instead of Forbidden. This
is due to the fact that Forbidden is in the
HTTPUnauthorized section of wsgi.py instead of
the HTTPForbidden section. Moving the exception
caused too many failures, so I created a 'Module'
Forbidden exception and put it in the right section.
Change-Id: I755b0431b33b870ae02d903527f071fd8e23130d
Depends-On: I54d37025275dee4731ad49ebbd21612c4464e4c4
Depends-On: I779c24472d3d96a7b2fe4ed0284fd5869cdef93b
Partially-Implements: blueprint module-maintenance
Implements log file retrieval from the guest agent. The contents
of the log file are pushed up to a swift container as a series of
objects that represent a subset of the lines in the log.
The following trove CLI commands are now supported:
trove log-list <instance> : lists log files available on guest
trove log-enable <instance> <log> : enables writing to log file
trove log-disable <instance> <log>: disables writing to log file
trove log-publish <instance> <log>: publishes updates to swift container
trove log-discard <instance> <log>: discards published logs
trove log-tail <instance> <log> : displays last lines of log
trove log-save <instance> <log> : saves the entire log to a file
Log declarations and scenario tests were added for MySQL and
PostgreSQL.
Co-Authored-By: Morgan Jones <morgan@tesora.com>
Co-Authored-By: Alex Tomic <atomic@tesora.com>
Co-Authored-By: Peter Stachowski <peter@tesora.com>
Implements: blueprint datastore-log-operations
Change-Id: I16c3bba4a3183d05af2971be6ba56110105797a6
Implement configuration groups for PostgreSQL.
Notes:
- Improved the PropertiesCodec to handle (strip) in-line comments.
Also fix the codec so that it preserves quotes around string values.
- Registered min/max functions with JINJA environment.
Python min() and max() can be used in configuration templates.
- Fixed the file-existence check in operating_system.read_file()
to also work with files that are not readable by the Trove user.
- Extended the operating_system.list_files_in_directory() to handle
paths not readable by the Trove user (e.i. add 'as root' flag).
- Pass 'requires_root' flag on the read_file() in the config manager.
- Improved the PropertiesCodec to remove white-spaces around the
property name (first item).
Also add a missing string conversion when properties with
just a single item did not get serialized to the proper
string value.
Implements: blueprint postgres-configuration-groups
Change-Id: Ieff1669b0ae5542b72cd7dce8921ee0c01e0cd58
In function object.__get__(self, instance, owner), owner is always
the owner class, while instance is the instance that the attribute
was accessed through, or None when the attribute is accessed through
the owner. Setting owner default value makes no sense.
refer to:
http://python-reference.readthedocs.org/en/latest/docs/dunderdsc/get.html#get
Change-Id: I5b84c77a52e5b26cb76d70ac738e1caef6e74c09
Closes-Bug: #1531395
The logic to find the pagination limit is
repeated several times and is not in one
function. This patch creates a function that
is called in various locations replacing
repetitive code.
Change-Id: I3eced6a4236e4189e57ea5a1589e8f0d411cc4e3
Closes-Bug: #1308285
The int-tests in Trove are very MySQL specific, which makes it difficult
to reuse code for other datastores. This changeset breaks them down
into 'groups' and 'runners.' Runners can be over-ridden to add
datastore specific handling/tests. This should allow most generic
code to be reused across datastores, while also providing for datastore
specific enhancements.
Runner implementations are stored in a new package
'trove.tests.scenario.runners'. A datastore-specific implementation can
be added to the appropriate runner module file. Its name has to match
'PrefixBaseRunnerClassName' pattern, where 'BaseRunnerClassName' is the
name of the default runner and 'Prefix' is the datastore's manager
name with the first letter capitalized.
Example:
Given the default implementation for negative cluster tests in
'trove.tests.api.runners.NegativeClusterActionsGroup'. One can
provide a custom implementation for MongoDB (with manager mongodb)
in 'trove.tests.api.runners.MongodbNegativeClusterActionsRunner'
This initial changeset adds tests for basic actions on instances
and clusters. Some basic replication tests were also migrated.
The concept of a helper class for datastore specific activies
was also created. This makes it easy to have tests use standard
methods of adding data and verifying that the datastore behaves
as it should.
Vertica was refactored to use the new infrastructure.
Running the tests can be accomplished by specifying one of the
new groups in int-tests (see int_tests.py for the complete list):
./redstack kick-start mongodb
./redstack int-tests --group=instance_actions --group=cluster
or
./redstack int-tests --group=mongodb_supported (to run all
tests supported by the MongoDB datastore)
As with the original int-tests, the datastore used is the one
referenced in test configuration file (test.conf) under the
key dbaas_datastore. This key is automatically set when
kick-start is run.
Additional Notes:
Also temporarily disabled volume size check in
instances tests.
It is supposed to assert that the used space on the
Trove volume is less that the size of the volume.
It however often fails because 'used' > 'size'.
From inspection of the instance it appears that the reported
'used' space is from the root volume instead of the
attached Trove volume. Plus it sometimes returns int instead of float.
Change-Id: I34fb974a32dc1b457026f5b9d98e20d1c7219009
Authored-By: Petr Malik <pmalik@tesora.com>
Co-Authored-By: Peter Stachowski <peter@tesora.com>
Remove the log module of oslo-incubator.
Move the WritableLogger wrapper to the base_wsgi module.
Add oslo.log to the requirements.
Change-Id: I724fa6090cebf40e7d7c78cc6b8458dfba9508a8
oslo.service has graduated, so trove should consume it.
Remove a "ticks_between_runs" parameter from the periodic tasks.
All periodic_tasks will be executed with the same interval with the value of
the "report_interval" option.
According to the old default value of the "report_interval" option and
the value of the ticks_between_runs paramentr (3), increase
the "report_interval" option to the 30 seconds.
Rename the "exists_notification_ticks" option to the "exists_notification_interval"
and increase the default value of it to 3600, according to the old value of
the "report_interval" option.
DocImpact
The option name and default value of exists_notification_ticks was changed to
exists_notification_interval with 3600 sec. And the new default value of
the report_interval option is 30 sec. The deployers need to make sure that
the new values in the seconds, not in the ticks.
Change-Id: Id14d28146f677faf017160ac93289fd119674fc4
Depends-On: Ia5b887e69853f39b387d309831fb7ce51b881149
Closes-Bug: #1466851
Facilitate code reuse by implementing a manager class
that could be used by all guestagents to manage their configuration
files and overrides.
ConfigurationManager is responsible for management of
datastore configuration.
Its base functionality includes reading and writing configuration files.
It is responsible for validating user inputs and requests.
When supplied an override strategy it allows the user to manage
configuration overrides as well.
ConfigurationOverrideStrategy handles configuration files.
The strategy provides functionality to enumerate, apply and remove
configuration overrides (revisions).
The patch set also includes functionality for reading and writing
files and implements codecs for serialization and
deserialization of common configuration formats. The implemented
codecs get reused in the existing configuration parsers.
Includes a couple of little fixes to the taskmanager that will
be required by other datastores.
- Do not validate value ranges if min/max is not specified in the
validation rules.
- Do not attempt to parse non-string configuration values in the
taskmanager.
Implements: blueprint guestagent-configuration-manager
Change-Id: I1c940c96deb20ca722d9fd400a6ef757b2ba249f
This patchset helps reducing the ignored pep8 errors.
Fixed some ignored hacking rules, as listed follows:
- E111 indentation is not a multiple of four
- E122 continuation line missing indentation or outdented
- E123 closing bracket does not match indentation of opening bracket's line
- E128 continuation line under-indented for visual indent
- E251 unexpected spaces around keyword / parameter equals
- E265 block comment should start with '# '
- E713 test for membership should be 'not in'
- H105 Don't use author tags
- H306 imports not in alphabetical order
Change-Id: Iadf2af4f6ec90420153ad63b5a41650392ef2cbd
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: I9d3e80d94795060d375aae30ce249513aae3fd97
This commit syncs the following oslo incubator modules used by Trove
and all dependent changes.
__init__.py
6b048e79 Let oslotest manage the six.move setting for mox
context.py
411ba356 Simplify is_user_context method
9b73877b Add a RequestContext.from_dict method
85d1ce6e Python 3: enable tests/unit/middleware/test_request_id.py
c0d357bb Add model_query() to db.sqlalchemy.utils module
eventlet_backdoor.py
a3220c51 add list_opts to all modules with configuration options
5d40e143 Remove code that moved to oslo.i18n
90ae24bf Remove redundant default=None for config options
fcf517d7 Update oslo log messages with translation domains
ad17a697 Fix filter() usage due to python 3 compability
8b2b0b74 Use hacking import_exceptions for gettextutils._
12bcdb71 Remove vim header
log.py
943cb94a Merge "Make use_syslog=True log to syslog via /dev/log"
8345204c Merge "add list_opts to all modules with configuration options"
ac4330dd Make use_syslog=True log to syslog via /dev/log
df774ff4 Import PublishErrorsHandler from oslo.messaging
a3220c51 add list_opts to all modules with configuration options
6c706c5c Delete graduated serialization files
5d40e143 Remove code that moved to oslo.i18n
6ff6b4b4 Switch oslo-incubator to use oslo.utils and remove old modules
aa744115 log: add missing space in error message
037dee00 Set stevedore log level to WARN by default
759bd879 Merge "Set keystonemiddleware and routes.middleware to log on WARN level"
71d072f1 Merge "Except socket.error if syslog isn't running"
37c00918 Add unicode coercion of logged messages to ContextFormatter
66144135 Correct coercion of logged message to unicode
loopingcall.py
5d40e143 Remove code that moved to oslo.i18n
e3773930 Changes calcuation of variable delay
ab5d5f1c Use timestamp in loopingcall
bc48099a Log the function name of looping call
fb4e863c Remove deprecated LoopingCall
fcf517d7 Update oslo log messages with translation domains
8b2b0b74 Use hacking import_exceptions for gettextutils._
12bcdb71 Remove vim header
service.py
5d40e143 Remove code that moved to oslo.i18n
6ede600f rpc, notifier: remove deprecated modules
threadgroup.py
1523f000 threadgroup: don't log GreenletExit
5a1a0166 Make stop_timers() method public
fdc88831 Add graceful stop function to ThreadGroup.stop
5f8ace05 Merge "threadgroup: use threading rather than greenthread"
2d06d6ca Simple typo correction
4d18b57a threadgroup: use threading rather than greenthread
25ff65e9 Make wait & stop methods work on all threads
12bcdb71 Remove vim header
9d3c34b5 Add a link method to Thread
versionutils.py
5d40e143 Remove code that moved to oslo.i18n
1c3ecfcd Enhance versionutils.deprecated to work with classes
7d42c107 Merge "Add Kilo release name to versionutils"
9a462718 Add Kilo release name to versionutils
a2ad3a25 Allow deprecated decorator to specify no plan for removal
05ae498b Add JUNO as a target to versionutils module
de4adbc4 pep8: fixed multiple violations
Closes-Bug: #1366189
Change-Id: I2289452a9b838e22bb6d4dd1a854fbeb326042d2
Sean Mooney