When using Galera, the wsrep_sync_wait option [1]
can change the behavior of a variety of Galera DQL/DML statements
such that a particular operation will pause until outstanding
write-sets are fully persisted to the local node. The setting
supersedes the previous boolean parameter wsrep_causal_reads
which only affected SELECT, with an updated approach that allows
for fine-grained control of so-called "causality checks"
on individual statement types. The legacy-compatible setting
of '1' indicates that READ/SELECT/BEGIN operations should
proceed only after any pending writesets are fully available.
The use case for this setting is for an application that
is running operations on multiple Galera nodes simultaenously.
An application that commits data on one node, and then immediately
uses a different connection (on a potentially different node)
to SELECT that data, may fail to see those changes if
"causality checks" for SELECT are not enabled. While
a COMMIT operation in Galera will block locally until all other
nodes approve of the writeset, the operation does not block
for the subsequent period of time when other nodes are actually
persisting that writeset. Setting up "causal reads"
in this case indicates that a SELECT operation will wait until
any writesets in progress are available, thus maintaining
serialization between the COMMIT and subsequent SELECT.
As the name implies, wsrep_sync_wait adds...waiting! to the
operation, and thus directly impacts performance by adding
latency to SELECT operations or to the operations that have
been selected for causality checks, to the degree that
concurrent writesets are expected to be present.
Since it's not expected that most if any Openstack applications
actually need this setting in order to be effective with
Galera multi-master operation, and as the setting is available
within client session scope and also impacts performance,
making it available on a per-application basis means that
specific applications which may see issues under load can
choose to enable this setting, much in the way any other
"transaction isolation" settings might be made, without having
to add a cluster-wide performance penalty by setting it at the
Galera server level.
[1] https://mariadb.com/docs/ent/ref/mdb/system-variables/wsrep_sync_wait/
Change-Id: Iee7afcac8ba952a2d67a9ad9dd0e4eae3f42518e
Enable the 'CaptureOutput' fixture provided by oslotest by default.
Change-Id: Ib34d8ab411a67816db2e26b49bec75993b5bed56
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Well, sort of. We enable them but immediately filter out the ones we're
actually seeing, the rationale being that we can address these in a
piecemeal fashion without the risk of introducing new issues.
Change-Id: I32cad5fd9ddbfd2ff6b62e5e2f87fad6ece7a4c1
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We were actually running tests four times. Twice with eventlet disabled
and twice with it enabled. Correct this.
Change-Id: Idb3926d73ff3c11abe9c273062aca1b299b9f72e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Introduced changes:
- pre-commit config and rules
- Add pre-commit to pep8 gate, Flake8 is covered in the pre-commit
hooks.
- Applying fixes for pre-commit compliance in all code.
Also commit hash will be used instead of version tags in pre-commit to
prevend arbitrary code from running in developer's machines.
pre-commit will be used to:
- trailing whitespace;
- Replaces or checks mixed line ending (mixed-line-ending);
- Forbid files which have a UTF-8 byte-order marker
(check-byte-order-marker);
- Checks that non-binary executables have a proper
shebang (check-executables-have-shebangs);
- Check for files that contain merge conflict strings
(check-merge-conflict);
- Check for debugger imports and py37+ breakpoint()
calls in python source (debug-statements);
- Attempts to load all yaml files to verify syntax (check-yaml);
- Run flake8 checks (flake8) (local)
For further details about tests please refer to:
https://github.com/pre-commit/pre-commit-hooks
Change-Id: Id03c1ea4752db3236c36521ca9c67125d5ba2526
Co-authored-by: Moisés Guimarães de Medeiros <moguimar@redhat.com>
Python 3.8 is now our highest level supported python runtime.
This updates the default tox target environments to swap out
py37 for py38 to make sure local development testing is
covering this version.
This does not impact zuul jobs in any way, nor prevent local
tests against py37. It just changes the default if none is
explicitly provided.
Change-Id: Id0107e9cdabbc1435c709641b93cb5c90cdf6907
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
blacklist:
W504 line break after binary operator
Fix:
E305 expected 2 blank lines after class or function definition, found 1
Change-Id: I56d62f67914f07e7aef74f857c4b42e3bfd3416e
Sphinx 1.8 introduced [1] the '--keep-going' argument which, as its name
suggests, keeps the build running when it encounters non-fatal errors.
This is exceptionally useful in avoiding a continuous edit-build loop
when undertaking large doc reworks where multiple errors may be
introduced.
[1] https://github.com/sphinx-doc/sphinx/commit/e3483e9b045
Change-Id: Idaca0bc97e33e0a080de612420177b5e102c628c
As it was announced [1] global bindep-fallback.txt was removed and now
projects need to have a local bindep.txt to be able to install binary
dependencies for testing.
In test jobs the script tools/test-setup.sh is called which requires
mysql and postgres servers and clients to be installed.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007272.html
Change-Id: I6b237ecf55ff440b5f5c956ee849a1927bb15441
Quotes around {posargs} cause the entire string to be combined into one
arg that gets passed to stestr. This prevents passing multiple args
(e.g. '--concurrency=16 some-regex')
Change-Id: Id8b5f9e1e334609d39a58706ac2039240f0ce9cb
According to Openstack summit session [1],
stestr is maintained project to which all Openstack projects should migrate.
Let's switch to stestr as other projects have already moved to it.
[1] https://etherpad.openstack.org/p/YVR-python-pti
Change-Id: Ifd51e726de277d33ff61db47110bf8b5563fcda7
We no longer support SQLAlchemy 0.9.0, so remove the test environment.
Change-Id: I614945b178ee09fe3c4c6c61c9356d072d776639
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.
We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.
We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.
Change-Id: I31a207868c07e08ae27c14635d52506021032cb7
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Set the default python to python3 except for the py27 environment. We
have to set that explicitly to override the new default.
Change-Id: I8557195409a559950da6c95aece5787195f31ad6
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Create a tox environment for running the unit tests against the lower
bounds of the dependencies.
Create a lower-constraints.txt to be used to enforce the lower bounds
in those tests.
Add openstack-tox-lower-constraints job to the zuul configuration.
See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
for more details.
Change-Id: I33404e115161a430ce0587bf203450c8145db0a0
Depends-On: https://review.openstack.org/555034
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
As part of removing reliance on the old and deprecated zuul-cloner,
we need to shift constraints declaration to the deps line. This
means we unfortunately have to duplicate the extras declarations
into test-requirements - because otherwise the contraints for
oslo.db conflicts with the installation of itself.
Change-Id: I2b79444a58a9144993131f9cba7494b236d9e950
In order to fade out MySQL-python from OpenStack we need to stop
testing it. Add release notes entry and adjust documentation
references accordingly.
Depends-On: Ie1f07062ed18350bcbb9e7b5e33c7ab2390be9ab
Change-Id: Ie17f4543fa3d72b507d88d2c4023a9a5a430419e
Now that there exists only a gate job for Python 3.5 and not 3.4,
we should remove those references to the 3.4 that is untested.
Change-Id: Ic2e61f28bc82f64814a528d210d1dee8d79b49e5
Adding constraints support to libraries is slightly more complex than
services as the libraries themselves are listed in upper-constraints.txt
which leads to errors that you can't install a specific version and a
constrained version.
This change adds constraints support by also adding a helper script to
edit the constraints to remove oslo.db.
Change-Id: I31febfe00698e36f4c2e0e41c3f7bf64acf47adf
Removes the separate per-Python-interpreter
setup for the pifpaf run and consolidates necessary
variables in one place to be passed into a single command
line.
Change-Id: I2f34642f16d6dba5b01500fcdc89e88b93642cd3
py35 jobs are going to be run in the gate soon. Add py35 to the list
of defaut envs and add corresponding pifpaf environments for running
tests locally.
All py35 tests currently pass - we can effectively declare py35
compatibility via a new classifier in setup.cfg.
Change-Id: Iea7e286062f95d5c61b9ec1e7f6af4138fe3ce99
This leverage pifpaf to start MySQL or PostgreSQL temporarily and allow to test
provisioning directly without being on OpenStack CI by adding new tox
targets:
- py27-mysql
- py27-postgresql
- py34-mysql
- py34-postgresql
- py27-all
- py34-all
These targets will start MySQL or PostgreSQL before running the tests,
while exporting the connection string to $PIFPAF_URL. The `all' target
will start both MySQL and PostgreSQL and will run the tests against
these backends, plus SQLite.
Also, this patches add OS_TEST_DBAPI_ADMIN_CONNECTION to be allowed to
be passed through tox. This allows to run the opportunistic tests on a
more persistent pifpaf database, for example by running tox with:
$ eval `pifpaf -g OS_TEST_DBAPI_ADMIN_CONNECTION run postgresql`
$ echo $OS_TEST_DBAPI_ADMIN_CONNECTION
postgresql://localhost/postgres?host=/var/folders/7k/pwdhb_mj2cv4zyr0kyrlzjx40000gq/T/tmpMGqN8C&port=9824
$ tox -e py27
[…]
$ tox -e py34
[…]
$ kill $PIFPAF_PID
Change-Id: I1ee582e6f96e98378f02be79f4aaff0f447a062a
Depends-On: Id3e6b694bb186724517599cd9875ad80ceeee053
To solve the problem of "db type could
not be determined" on py34 we have to run first the py34 env to, then, run
py27. This patch puts py34 first on the tox.ini list of envs to avoid this
problem to happen.
Change-Id: I2365f66eee43cd36e62fe05e582c328787a0a112
Closes-bug: #1489059
Adds extras for e.g. nova to use for testing, so that oslo.db can
control the libraries used.
Removes test-requirements.txt - it becomes a single env-marker
controlled list, and shows how tox will access those dependencies.
Co-Authored-By: Roman Podoliaka <rpodolyaka@mirantis.com>
Change-Id: I97ef5bad215abe60d92c3fda4474c48ccb13eab0
* Remove support for python 2.6
* Remove skipsdist : Needs to be set only if
sdist is expensive
* Remove usedevelop : only needed when skipsdist
is set to True
* Remove install_command : We can just use the
default, we don't need to override
* sitepackages : default is already false
Change-Id: I12de2f2175fce3a0b24c16a1fc55100c2927883f
testscenarios and testresources are only needed if you're using the
oslo.db testing fixtures downstream, otherwise you don't need this at
runtime in a deployment, so move them into the setup.cfg [extras] group.
If a consumer of oslo.db wants to use the db test fixtures then they
install oslo.db like 'sudo pip install oslo.db[fixtures]'.
This should trigger a major version bump for the library when it lands.
Closes-Bug: #1429233
Change-Id: I1a7c5b6a7b29cd3a50655f60606cc0b025400a96
Neutron (and presumably other projects) expect to be able to have
overlapping transactions open from two eventlets at once, without
deadlock.
Note that the default mysql sqlalchemy engine (mysqldb) fails this test.
This change modifies py27 tox env to run the full test suite separately
with and without TEST_EVENTLET environment variable set. When set,
tests are run with eventlet imported and monkey_patched.
Change-Id: Ib35c95defea8ace5b456af28801659f2ba67eb96
Related-Bug: #1350149
The global requirements require now SQLAlchemy 0.9.7 or newer,
so there is no choice to test in a SQLAlchemy 0.8 environment
anymore. Just remove the code related to that.
Change-Id: Id4be73285a1f4b4ca70a3cbb0904e8d4f577aaa4
As discussed in the Liberty Design Summit "Moving apps to Python 3"
cross-project workshop, the way forward in the near future is to
switch to the pure-python PyMySQL library as a default.
Added a special test environment to keep MySQL-python support.
Documentation modified.
https://etherpad.openstack.org/p/liberty-cross-project-python3
Change-Id: I12b32dc097a121bd43991bc38dd4d289b65e86c1
Takashi Kajinami