hacking 3.1.0 is too old.
Note:
We can't directly bump hacking to 6.x.0 (which is the latest major
version) because of the existing cap by diskimage-builder. The cap is
now being updated by [1].
[1] https://review.opendev.org/c/openstack/diskimage-builder/+/909336
Change-Id: I8778a7decc6669b4d95d6886c971433e7c34c5c8
Python 2.y and Python <= 3.2 are no longer supported, so we no longer
need to maintain logics for such old versions.
Also the check_python_version method is removed by this change, because
the required python version is now enforced by setup.cfg.
Change-Id: Ifc5e618a791bdf591e0ec61075089dd992c73e0c
H306: import in alphabeical order
Imports should be grouped in the following order:
standard library imports
related third party imports
local application/library specific imports
put a blank line between each group of imports.
https://docs.openstack.org/hacking/latest/user/hacking.html#imports
Change-Id: I2c188d27f0595ac3b2d71f4612d93829915e389c
The older version of pylint being used does not work correctly under
python 3. In order to be able to update the pylint job to run under
python 3, we need to update the tool.
This patch updates to the latest version at this time. It also updates
and pins astroid, which was previously capped. Using a pin instead of
a cap should let us avoid issues with new releases while being
specific about which version to actually use.
Disable not-callable because that appears to be a new rule that is
confused by the use of properties to access things that are set to
callables.
Co-Authored-By: Fan Zhang <zh.f@outlook.com>
Co-Authored-By: Marcin Piwowarczyk <m.piwowarczy@samsung.com>
Change-Id: I65705804b222dcd30a653fe10be3d823fa6143ff
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We're updating a dict while iterating in PropertiesCodec, this leads to
random failures of the py35 unittest gate job.
As the logs of a zuul gate job won't be permanent, I wrote a simple
script(though may be ugly) to testing this:
$ python --version
Python 3.5.2
$ cat /tmp/python35_dict_updating.py
data_dict = {}
def init_dict():
global data_dict
for x in range(11):
key = 'key' + str(x+1)
data_dict[key] = x * 10
def update_dict():
global data_dict
for k, v in data_dict.items():
data_dict.update({k: v+1})
def checking_failed(round):
global data_dict
FAILED = False
for x in range(11):
key = 'key' + str(x+1)
origin = x * 10
current = data_dict[key]
if (current - origin) != round:
print('%s is %s, expecting %s' %
(key, current, origin + round))
FAILED = True
break
return FAILED
TEST_TIMES=5
PASSED = True
for x in range(TEST_TIMES):
init_dict()
for round in range(1, 100):
update_dict()
if checking_failed(round):
print("Failed at round %s" % round)
PASSED = False
break
if PASSED:
print("No failures in 5 100-round tests")
$ python /tmp/python35_dict_updating.py
key9 is 82, expecting 81
Failed at round 1
$ python /tmp/python35_dict_updating.py
key2 is 12, expecting 11
Failed at round 1
$ python /tmp/python35_dict_updating.py
key9 is 82, expecting 81
Failed at round 1
$ python /tmp/python35_dict_updating.py
No failures in 5 100-round tests
From the above testing results, we could see it's quite often that one
of the item will be updated twice during the dict iteration. This is
the reason why test_properties_file_codec failed sometimes.
This fix is manually converting the result of dict.items to a list:
$ sed -i 's/data_dict.items()/list(data_dict.items())/' \
/tmp/python35_dict_updating.py
$ python /tmp/python35_dict_updating.py
No failures in 5 100-round tests
$ python /tmp/python35_dict_updating.py
No failures in 5 100-round tests
$ python /tmp/python35_dict_updating.py
No failures in 5 100-round tests
$ python /tmp/python35_dict_updating.py
No failures in 5 100-round tests
Closes-Bug: #1764321
Change-Id: Ia9fcfc6519b29f1a9508b79614c5e81456ad57b6
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
* Tests for _write_file_as_root failed because tempfile.NamedTemporaryFile
use binary mode by default, changing to text mode fixes.
* Migrate to olso_serialization.base64 for Base64Codec. Also change the
return value from bytearray to bytes for Base64Codec.deserialize.
Base64Codec supports reverse encoding(i.e. binary data will be written
to the dest file), in this situation, the dest file should be opened
with binary mode(and when reading from the file, binary mode should
also be used).
* stream_codecs.StringConverter converts iterable objects with map
function. The behavior of map function is different under Python 2.x
and 3.x. However csv.writerows(before Python 3.5) and unpack_singleton in
trove.common.utils both need a list object. Converting the return
value of map function to a list explicitly before passing to these
functions fixes.
guestagent.test_operating_system is the last blacklist regex pattern for
py3 unittests. With the above problems fixed, blacklist-py3.txt is not
needed any more, and tox.ini is also updateted.
Migrating from ostestr to stestr is also done while updating tox.ini.
Here is the ML post for more information about os-testr and stestr:
http://lists.openstack.org/pipermail/openstack-dev/2017-September/122135.html
Partially implements: blueprint trove-python3
Change-Id: I31f1f97901d6ebff8a91c1b70a343e724ab806eb
Signed-off-by: Zhao Chao <zhaochao1984@gmail.com>
In py3.4 notice method readfp will removed in the future,so use
method readfile to replace it.This warning is only in py3 and
above.Py2,configparse do not have.Actually in py3,This function
is finally pointed to the readfile function
warnings.warn(
"This method will be removed in future versions. "
"Use 'parser.read_file()' instead.",
DeprecationWarning, stacklevel=2
)
self.read_file(fp, source=filename)
Change-Id: I22a2dca71008469a8c8b7afcc0934f3b4adcdae4
SafeConfigParser is deprecated in Python 3.2 and log warning
like " DeprecationWarning: The SafeConfigParser class has
been renamed to ConfigParser in Python 3.2. This alias will be
removed in future versions. Use ConfigParser directly instead."
So use ConfigParser in Python 3.2+.
Closes-Bug: #1618666
Change-Id: I30fe51324ffcc0afbd02799449daee8f628634b6
Support for standard WAL based streaming replication for
postgresql guests. Sets up read-only hot standby servers.
Basic replication is supported for 9.3, but promote
requires pg_rewind which is only supported for 9.4 and
greater. Promote currently only works with single
master-replica configurations (see TODO and associated
bug 1553339 for details)
Test changes:
- moved the replication promote tests to a separate group
- skipped known issues with bug reference
Change-Id: I519fa58d786edddb09bf14ce7629e1be51b62600
Implements: bp/pgsql-replication
Add system.log retrieval.
The Cassandra's system/general log is located in the cassandra
log directory and is by default called system.log.
Cassandra 2.1 and higher uses 'SLF4J' with 'logback' backend.
Logging properties can be configured at runtime via the nodetool utility
(setlogginglevel) and/or persisted in 'logback.xml' configuration file.
Logging can be enabled on per-module/class basis with configurable
verbosity levels.
The logging in Trove is configured globally (on the root level)
- i.e. for all application modules.
A new configuration property 'system_log_level' was added to control the
log verbosity. It allows the operator define the log level that will be
set when the logging is enabled on an instance.
All logging will be turned off when disabled.
Note: The log level is not changed during prepare. The instances will be
provisioned with the default logging setting (INFO level).
Note: An XML codec was added to allow parsing XML files into/from Python
dicts.
Change-Id: Ia57587bf557771ea6d7f00e2fe9d674789b98559
Depends-On: I4a088b7a541c35a6c32d6985727bad65ff491815
Depends-On: I254b81b45e44aeda3049865bee76cd9075312761
Closes-Bug: 1550557
The data type information was lost in conversions.
IniCodec should deserialize Python objects (like other codecs).
guestagent_utils.to_bytes should return byte values as ints.
* The IniCodec is also used by Cassandra.
Tested with both MySQL and Cassandra scenario tests.
Change-Id: Ibb703b3db6814fc0c9ea4c6d96399f6c881cea03
Closes-Bug: 1599656
* Fix test_template: replace server_id > 1 with len(server_id) > 1,
server_id is a string. On Python 3, str cannot be compared to int,
a TypeError is raised.
* Replace ConfigParser import with six.moves.configparser
* Replace basestring with six.string_types
* Replace dict.iteritems() with dict.items(). The iteritems() was
removed in Python 3.
* Replace a/b with a//b to get integer on Python 3.
* tox.ini: add common/test_template.py to Python 3.4
Partially implements: blueprint trove-python3
Change-Id: Ibe2ccdcba7d55edcc014ea00c3b927d8201d6c1b
This changeset handles the details of applying,
removing, listing and retrieving 'modules'
from Trove instances.
See https://review.openstack.org/#/c/290177 for
the corresponding troveclient changes.
Scenario tests have been extended to cover the
new functionality. These tests can be run by:
./redstack int-tests --group=module
A sample module type 'driver' - ping - is included
that simply parses the module contents for a
message=Text string and returns the 'Text' as the
status message. If no 'message=' tag is found, then
the driver reports an error message.
Due to time constraints, a few unimplemented
parts/tests of the blueprint have been triaged as bugs
and are scheduled to be fixed before mitaka-rc1.
These include:
Vertica license module driver:
https://bugs.launchpad.net/trove/+bug/1554898
Incomplete module-instances command:
https://bugs.launchpad.net/trove/+bug/1554900
Incomplete 'live-update' of modules:
https://bugs.launchpad.net/trove/+bug/1554903
Co-Authored-by: Peter Stachowski <peter@tesora.com>
Co-Authored-by: Simon Chang <schang@tesora.com>
Partially Implements: blueprint module-management
Change-Id: Ia8d3ff2f4560a6d997df99d41012ea61fb0096f7
Depends-On: If62f5e51d4628cc6a8b10303d5c3893b3bd5057e
Implements log file retrieval from the guest agent. The contents
of the log file are pushed up to a swift container as a series of
objects that represent a subset of the lines in the log.
The following trove CLI commands are now supported:
trove log-list <instance> : lists log files available on guest
trove log-enable <instance> <log> : enables writing to log file
trove log-disable <instance> <log>: disables writing to log file
trove log-publish <instance> <log>: publishes updates to swift container
trove log-discard <instance> <log>: discards published logs
trove log-tail <instance> <log> : displays last lines of log
trove log-save <instance> <log> : saves the entire log to a file
Log declarations and scenario tests were added for MySQL and
PostgreSQL.
Co-Authored-By: Morgan Jones <morgan@tesora.com>
Co-Authored-By: Alex Tomic <atomic@tesora.com>
Co-Authored-By: Peter Stachowski <peter@tesora.com>
Implements: blueprint datastore-log-operations
Change-Id: I16c3bba4a3183d05af2971be6ba56110105797a6
Implement configuration groups for PostgreSQL.
Notes:
- Improved the PropertiesCodec to handle (strip) in-line comments.
Also fix the codec so that it preserves quotes around string values.
- Registered min/max functions with JINJA environment.
Python min() and max() can be used in configuration templates.
- Fixed the file-existence check in operating_system.read_file()
to also work with files that are not readable by the Trove user.
- Extended the operating_system.list_files_in_directory() to handle
paths not readable by the Trove user (e.i. add 'as root' flag).
- Pass 'requires_root' flag on the read_file() in the config manager.
- Improved the PropertiesCodec to remove white-spaces around the
property name (first item).
Also add a missing string conversion when properties with
just a single item did not get serialized to the proper
string value.
Implements: blueprint postgres-configuration-groups
Change-Id: Ieff1669b0ae5542b72cd7dce8921ee0c01e0cd58
* Convert configuration casts into blocking calls and move them
from the task manager to the API. Update the Trove records
only after a successful update on the guestagent.
* Change the way how we apply the configuration changes.
- On attach: Apply the values dynamically only if all of them
can be applied at once. Put the instance into
the 'RESTART_REQUIRED' state otherwise.
- On detach: Apply the values dynamically only if:
a) Default values for all of them are
available in the configuration template.
b) All values can be applied at once.
Put the instance into the 'RESTART_REQUIRED'
state otherwise.
* Remove override template resolution. Pass override values in a
Python dict (like it is for 'apply_overrides').
The user-provided configuration values do not get resolved and
are applied as supplied by the user - hence no need for a template.
It also avoids the need to double-parse the overrides in
guestagents that do not support configuration imports.
* Moved MySQL-specific value conversions (K, M, G suffixes)
down to the MySQL guestagent.
* Update the MySQL 'update_overrides' methods to accept both
Python dicts and rendered strings. This is for backwards
compatibility with older task managers that send overrides as
a string.
* Remove deprecated methods from the taskmanager.
Closes-Bug: 1468488
Change-Id: Ie125131945ad82afe6663e6c5a5d70c6e949c60b
Also includes:
* Use pymongo to determine database status.
* JsonCodec for writing/reading JSON files.
Implements blueprint: mongodb-configuration-groups
Change-Id: I64f3fdd05b2c320613cbd1c394e6d3a88e09363b
Facilitate code reuse by implementing a manager class
that could be used by all guestagents to manage their configuration
files and overrides.
ConfigurationManager is responsible for management of
datastore configuration.
Its base functionality includes reading and writing configuration files.
It is responsible for validating user inputs and requests.
When supplied an override strategy it allows the user to manage
configuration overrides as well.
ConfigurationOverrideStrategy handles configuration files.
The strategy provides functionality to enumerate, apply and remove
configuration overrides (revisions).
The patch set also includes functionality for reading and writing
files and implements codecs for serialization and
deserialization of common configuration formats. The implemented
codecs get reused in the existing configuration parsers.
Includes a couple of little fixes to the taskmanager that will
be required by other datastores.
- Do not validate value ranges if min/max is not specified in the
validation rules.
- Do not attempt to parse non-string configuration values in the
taskmanager.
Implements: blueprint guestagent-configuration-manager
Change-Id: I1c940c96deb20ca722d9fd400a6ef757b2ba249f
Takashi Kajinami