Commit Graph

61 Commits

Author SHA1 Message Date
Zuul eedd2d842b Merge "pre-commit: Integrate bandit" 2024-02-21 06:08:38 +00:00
Takashi Kajinami 8ff86619bb Display coverage report
... for easy reference. Also make sure old data is purged.

Change-Id: I90bdc8e5eaf738dcbf05715436c7ba140e6f5dfb
2024-02-07 03:10:50 +00:00
Takashi Kajinami 797f40aa43 pre-commit: Integrate bandit
We also remove these unnecessary linter dependencies from
test-requirements.txt.

Co-Authored-By: Stephen Finucane <sfinucan@redhat.com>
Change-Id: Id17f4865b2da9f4a5c1cd5ea360cd007315d1c0e
2024-02-02 21:46:30 +09:00
Hervé Beraud 0af59425ad Fix issues related to tox4
Related to https://lists.openstack.org/pipermail/openstack-discuss/2023-February/032247.html

Change-Id: I38b5e7fe25820ffda712bbc983e4ea3d6c3bf17d
2023-02-17 13:16:04 +00:00
Balazs Gibizer 796203c948 Prove that spawn_n with fair lock is broken
The fasteners lib in version 0.15.0 removed the
threading.current_thread workaround for eventlet[1] because eventlet
seemed to fixed the current_thread issues tracked in [2]. However the
fix for [2] only fixed half of the problem. The threading.current_thread
call works if it is called from thread created by eventlet.spawn.
However if the thread is created with eventlet.spawn_n then
threading.current_thread is still broken and returns the ID of the
python native thread.

The fasteners' ReaderWriterLock depends heavily on
threading.current_thread to decide which thread holds a lock and to
allow re-entry of that thread. This leads to the situation that
multiple threads created from spawn_n could take the same
ReaderWriterLock at the same time.

The fair internal lock in oslo.concurrency uses ReaderWriterLock and
as a result such lock is broken for threads created with spawn_n.

Note that this issue was raised with eventlet in [3] when the nova team
detected it via a direct usage of ReaderWriterLock in the nova test
code. As [3] did not lead to a solution in eventlet nova implemented a
nova local fix for the test code in [4].

However now we detected that oslo.concurrency is affected by this issue
as well.

This patch adds tests to show the scope of the problem.

Note that the coverage tox target is changed to explicitly enable native
threading otherwise it runs eventlet specific tests in a native
environment.

Also note that [5] was opened to reintroduce the workaround[1] in fasteners.

[1] 467ed75ee1
[2] https://github.com/eventlet/eventlet/issues/172
[3] https://github.com/eventlet/eventlet/issues/731
[4] https://review.opendev.org/c/openstack/nova/+/813114
[5] https://github.com/harlowja/fasteners/issues/96

Related-Bug: #1988311
Change-Id: Ibc193c855b49b95b46ebd2aac82ea89e33f885f0
2022-09-05 13:27:37 +02:00
Zuul 4da91987d6 Merge "Use py3 as the default runtime for tox" 2021-04-16 17:21:33 +00:00
wu.shiming 5e6ab754f7 Remove lower-constraints remnants
These were missed in change: Ic086f5e9a7a8b9c9050ef27993f339d52ad1e236

Change-Id: Ifbfb33e18c95a3bf01dd522835d1246989e78242
2021-02-19 14:43:13 +08:00
Hervé Beraud 07cf5c0fc6 Use TOX_CONSTRAINTS_FILE
UPPER_CONSTRAINTS_FILE is old name and deprecated
This allows to use upper-constraints file as more
readable way instead of UPPER_CONSTRAINTS_FILE=<lower-constraints file>.

[1] https://review.opendev.org/#/c/722814/
[2] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file

Change-Id: Ib833cd6818e7071cf3e85f77966e9c02a4026efd
2020-11-04 10:46:42 +01:00
Hervé Beraud aa1be28473 Use py3 as the default runtime for tox
Moving on py3 as the default runtime for tox to avoid to update this at
each new cycle.

Wallaby support officially the following runtimes [1]:
- Python 3.6
- Python 3.8

During Victoria Python 3.7 was used as the default runtime [2] however this
version isn't longer officially supported.

[1] https://governance.openstack.org/tc/reference/runtimes/wallaby.html#python-runtimes-for-wallaby
[2] https://governance.openstack.org/tc/reference/runtimes/victoria.html#python-runtimes-for-victoria

Change-Id: Id19b6e38400f7b644faeabcac6d8037603cac459
2020-11-04 10:01:14 +01:00
Hervé Beraud b8f713bc79 Adding pre-commit
Introduced changes:
- pre-commit config and rules
- Add pre-commit to pep8 gate, Flake8 is covered in the pre-commit hooks.
- Applying fixes for pre-commit compliance in all code.

Also commit hash will be used instead of version tags in pre-commit to
prevend arbitrary code from running in developer's machines.

pre-commit will be used to:
- trailing whitespace;
- Replaces or checks mixed line ending (mixed-line-ending);
- Forbid files which have a UTF-8 byte-order marker (check-byte-order-marker);
- Checks that non-binary executables have a proper
  shebang (check-executables-have-shebangs);
- Check for files that contain merge conflict strings (check-merge-conflict);
- Check for debugger imports and py37+ breakpoint()
  calls in python source (debug-statements);
- Attempts to load all yaml files to verify syntax (check-yaml);
- Run flake8 checks (flake8) (local)

For further details about tests please refer to:
https://github.com/pre-commit/pre-commit-hooks

Change-Id: I6ae02d4e52fd812360414e7b8c1c9c4f17230812
Signed-off-by: Moisés Guimarães de Medeiros <moguimar@redhat.com>
2020-09-03 14:25:24 +02:00
Sean McGinnis 1cb17464c4
Bump default tox env from py37 to py38
Python 3.8 is now our highest level supported python runtime.
This updates the default tox target environments to swap out
py37 for py38 to make sure local development testing is
covering this version.

This does not impact zuul jobs in any way, nor prevent local
tests against py37. It just changes the default if none is
explicitly provided.

Change-Id: Ia2304240ac09a756249b5689e7fff9c829c7bc9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-04-24 10:25:48 -05:00
Andreas Jaeger 0a0c99a42a Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Blacklist: W504 line break after binary operator

Change-Id: I8bca364fdc4c09cbd34127de10fa31da47078b4f
2020-03-30 11:56:23 +00:00
Hervé Beraud 4af4e0271d trivial: Cleanup tox.ini
Update the minversion parameter to use the python -m pip to install
python packages:

https://tox.readthedocs.io/en/latest/changelog.html#id185

Also Inherit the jobs requirements to simplify maintainance.

Change-Id: Icb5116908148dcbb58a7531bbed498d29ca1d2ea
2020-02-11 15:25:49 +01:00
Stephen Finucane cfc27ef497 Drop python 2.7 support and testing
Ussuri Communtiy-wide goal:
https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html

Change-Id: I94e709093e01825069b44b6c485e49a81f8f14c1
Sem-Ver: api-break
2020-01-30 11:37:22 +00:00
Hervé Beraud 485fe226ec Stop configuring install_command in tox.
Currently, we are overriding 'install_command' to use 'pip'. This is
considered poor behavior and 'python -m pip' should be used instead:

https://snarky.ca/why-you-should-use-python-m-pip/

It turns out that this is the the default value provided by tox:

https://tox.readthedocs.io/en/latest/config.html#conf-install_command

So we can remove the line and simply use the default value.

Also these changes introduce a new explicit env var related to eventlet
testing to avoid misunderstand.

Change-Id: I03b9b179c4cb6ea5a9a8d8c74af3502a0243ace6
2020-01-15 09:51:30 +01:00
Hervé Beraud fc21061460 tox: Trivial cleanup
Move 'basepython' to the top-level 'testenv'.

Change-Id: I2ab1eb99a61695bfbc7ecc66f215b1cb991007dd
2020-01-08 20:16:07 +01:00
caoyuan ec52c3b5dc tox: Keeping going with docs
Sphinx 1.8 introduced [1] the '--keep-going' argument which, as its name
suggests, keeps the build running when it encounters non-fatal errors.
This is exceptionally useful in avoiding a continuous edit-build loop
when undertaking large doc reworks where multiple errors may be
introduced.

[1] https://github.com/sphinx-doc/sphinx/commit/e3483e9b045

Change-Id: Id12730272782a7cfda89cba79d40919e2a5ecd24
2019-10-24 13:58:31 +08:00
pengyuesheng 941cf048f0 Update the constraints url
For more detail, see http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html

Change-Id: I2e16a74f601a36ecc71b8bda74ded42dac99bf52
2019-09-29 10:14:24 +08:00
Eric Fried fe86f5e4ae Some test cleanup
- Tests are based on oslotest.base.BaseTestCase, which uses the
  NestedTempFile fixture, which uses the TempDir fixture, which adds a
  cleanup routine to remove the base temporary directory it creates.
  There's therefore no need for tests to clean up this directory, so all
  the code that does that is removed.
- The eventlet incarnation of tests was trying to make use of the `env`
  external without whitelisting it, resulting in an ugly red deprecation
  warning. This commit adds `env` to whitelist_externals in [testenv].
- A handful of typos in the Beowulf quote are corrected.

Change-Id: I91cc52e00e0a918dadd2a3a771bd322b0f165ed2
2019-09-17 16:13:30 -05:00
Corey Bryant 610df387b6 Add Python 3 Train unit tests
This is a mechanically generated patch to ensure unit testing is in place
for all of the Tested Runtimes for Train.

See the Train python3-updates goal document for details:
https://governance.openstack.org/tc/goals/train/python3-updates.html

Change-Id: Icdd6929f479b54599ddf19a8092947272d5c8b68
Story: #2005924
Task: #34234
2019-07-05 14:42:45 -04:00
Zuul a57f84cda6 Merge "Replace git.openstack.org URLs with opendev.org URLs" 2019-05-19 06:14:49 +00:00
jacky06 443ed81b43 Replace git.openstack.org URLs with opendev.org URLs
1. Thorough replacement of git.openstack.org URLs with their
opendev.org counterparts.
2. dict_object.keys() is not required for *in* operator

Change-Id: Ie1d0c828522ae92b01f28e7017fd7fd42c2e91e7
2019-05-05 13:01:11 +08:00
Zuul 3280b697ac Merge "Dropping the py35 testing" 2019-05-03 18:05:40 +00:00
Ghanshyam Mann 7dd0913194 Dropping the py35 testing
All the integration testing has been moved to
Bionic now[1] and py3.5 is not tested runtime for
Train or stable/stein[2].

As per below ML thread, we are good to drop the py35
testing now:
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005097.html

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004647.html
[2]
https://governance.openstack.org/tc/reference/runtimes/stein.html
https://governance.openstack.org/tc/reference/runtimes/train.html

Change-Id: Idd5ceaf4d50d7181bce2859011b2a0c8206044d5
2019-04-15 19:00:38 +00:00
ZhongShengping 7274bc23f1 Follow the new PTI for document build
For compliance with the Project Testing Interface as described in:

https://governance.openstack.org/tc/reference/project-testing-interface.html

For more detials information, please refer to:

http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html

Change-Id: I4b1b023ff2c40f5b31429584fc1a77c0315ed2d9
Co-Authored-By: Stephen Finucane <stephenfin@redhat.com>
2019-04-02 14:28:24 +01:00
Ben Nemec b8bf7b2810 Stop using setup.py build_sphinx
This is deprecated in pbr and the preferred way to do doc builds now
is to use sphinx-build directly.

Change-Id: Id5081ffb9f5b5e6ce608d2f10e9ef89a8b40eeed
2019-01-09 23:11:16 +00:00
Vieri 5b284480c4 Don't quote {posargs} in tox.ini
Quotes around {posargs} cause the entire string to be combined into one
arg that gets passed to stestr. This prevents passing multiple args
(e.g. '--concurrency=16 some-regex')

Change-Id: I62860fe0f6cfea902c60a251408577bbe0ea894f
2018-10-09 13:34:01 +00:00
Andreas Jaeger 7d4433eef5 Use templates for cover and lower-constraints
Use openstack-tox-cover template, this runs the cover job as
non-voting in the check queue only.

Use openstack-lower-constraints-jobs template

Remove jobs that are part of the templates.

Sort template list alphabetically.

Fix cover tox.ini setting so that this works.

Change-Id: I6e7d197b780e79f22ad936fe63da13ca48098764
2018-09-08 15:38:12 +02:00
Vu Cong Tuan a9d728b71e Switch to stestr
According to Openstack summit session [1],
stestr is maintained project to which all Openstack projects should migrate.
Let's switch to stestr as other projects have already moved to it.

[1] https://etherpad.openstack.org/p/YVR-python-pti

Change-Id: I753f32ecf3275cf49d8c93bf648a6a26bc6da8e7
2018-07-04 08:18:31 +07:00
Doug Hellmann 31d31ab699 fix tox python3 overrides
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.

We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.

We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.

Change-Id: Ia0bccf6ed1b28f64622217d88dc6401ba6bd4406
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-06-06 14:57:00 -04:00
Kenneth Giusti 8a0fd6c3c7 Remove stale pip-missing-reqs tox test
pip-check-reqs tool is no longer maintained and has broken with
release 10 of pip

Refer to:
 http://lists.openstack.org/pipermail/openstack-dev/2018-April/130027.html

Change-Id: Id4217fce07cdf836df9c44dc81246154e5389150
2018-05-01 09:15:35 -04:00
Doug Hellmann b2c131076a set default python to python3
Set the default python to python3 except for the py27 environment. We
have to set that explicitly to override the new default.

Change-Id: I3fa047b21fa6bc6e5bdf3222287fdce31c5ccef2
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-04-12 16:20:18 -04:00
Ben Nemec ca3c45044b Switch pep8 job to python 3
pep8 under python 3 is more strict than under python 2, so to make
sure our projects meet the more strict standards we want to run the
pep8 jobs using python 3 by default.

Change-Id: Ice3cffa36e9167f2f1f30337372cf9187af5e02e
2018-04-12 16:28:07 +00:00
Doug Hellmann e55a18c5e0 add lower-constraints job
Create a tox environment for running the unit tests against the lower
bounds of the dependencies.

Create a lower-constraints.txt to be used to enforce the lower bounds
in those tests.

Add openstack-tox-lower-constraints job to the zuul configuration.

See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
for more details.

Change-Id: I6183925f1acc8bb020caa099c60b6baee61e579a
Depends-On: https://review.openstack.org/555034
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-03-24 21:02:16 -04:00
ChangBo Guo(gcb) 86fb3dcc27 add bandit to pep8 job
Add the bandit security scanner to the pep8 job.
Note: we skip some cases which we can't fix now.

Change-Id: Iad8b31d1c214376cbdca39fa28afa7f46af6ccca
2017-12-12 10:49:20 +08:00
Andreas Jaeger 9584101c99 Remove -U from pip install
'pip install -U' ugrades specified packages, this is not necessary
since we use constraints, remove the parameter '-U' from the line.

With tools/tox_install.sh - which a previous change of mine removed -
the -U was not harmful, but with the current set up, it might cause
upgrades, so remove it.

Change-Id: I3174ca15402723da2eea20b17d7ce1cc1ed18e41
2017-12-02 19:29:49 +01:00
Andreas Jaeger d3abe4a67e Avoid tox_install.sh for constraints support
We do not need tox_install.sh, pip can handle constraints itself
and install the project correctly. Thus update tox.ini and remove
the now obsolete tools/tox_install.sh file.

This follows https://review.openstack.org/#/c/508061 to remove
tools/tox_install.sh.

Change-Id: Ice6d75bcf236713aaad4dfbfa51e6b886555f517
2017-12-01 09:41:01 +01:00
gengchc2 cf3f64152e Remove support for py34
The gating on python 3.4 is restricted to <= Mitaka. This is due
to the change from Ubuntu Trusty to Xenial, where only python3.5
is available. There is no need to continue to keep these settings.

Change-Id: I69fedf01df329ab2f60c4b2583ac5541debeeee4
2017-02-08 16:48:09 +08:00
Tony Breeds cda2474b8f Add Constraints support
Adding constraints support to libraries is slightly more complex than
services as the libraries themselves are listed in upper-constraints.txt
which leads to errors that you can't install a specific version and a
constrained version.

This change adds constraints support by also adding a helper script to
edit the constraints to remove oslo.concurrency.

Change-Id: Ic887ce8e220df6b87d823490dfb14894935cac61
2016-12-20 14:35:31 +11:00
ChangBo Guo(gcb) e9a0914493 Add Python 3.5 classifier and venv
Now that there is a passing gate job, we can claim support for
Python 3.5 in the classifier. This patch also adds the convenience
py35 venv. This commit also removes py34 specific venv.

Change-Id: I50ddda7d96c77e7db60aa0b42e8e3b701dadf404
2016-07-14 12:23:14 +08:00
Kirill Bespalov 9a36c18b12 Add reno for releasenotes management
An initial patch to add reno and create a base directory for
release notes

Change-Id: Ie3f6653a27a49d8b2fced3f399b1d4c8f0c95b7c
2016-05-20 17:56:23 +03:00
Yuriy Taraday 7e47d32161 Add doc/ to pep8 check
We had unused imports there, this would prevent us from letting more of
such errors in.

Change-Id: Ie78c18dd3b9ab34370e47eba98b32740e0c83e1d
2016-05-12 14:05:20 +03:00
ChangBo Guo(gcb) 45cf9f0a9e Trivial: ignore openstack/common in flake8 exclude list
The directory openstack/common doesn't exist any more.
So remove it from flake8 exclude list.

Change-Id: Ieb4a5aac33b69f4cd7a4c0761836cb6bee087430
2016-05-11 19:19:39 +08:00
Davanum Srinivas e5baaf8a4c Remove python 2.6 and cleanup tox.ini
* Remove support for python 2.6
* Remove skipsdist : Needs to be set only if 
  sdist is expensive
* Remove usedevelop : only needed when skipsdist 
  is set to True
* Remove install_command : We can just use the 
  default, we don't need to override
* Remove setenv : We can just use the default as
  we don't need any extra environment variables
* Remove requirements.txt from deps, as this is
  already added automatically


Change-Id: I5c4addd8be5797fec439b011d99c00a7801704cd
2015-11-16 00:29:59 +00:00
Ronald Bradford 7057ed4e45 Fix coverage configuration and execution
A number of configuration errors prevent the successful creation of code
coverage. This corrects the .coveragerc source/omit setup and the tox
package name generation.

http://lists.openstack.org/pipermail/openstack-dev/2015-October/076193.html

Change-Id: If85385ac8b0817e9e5d167940ace37e733cacd41
2015-10-05 17:01:16 -04:00
Davanum Srinivas ca1a4c217a flake8 - remove unused rules
Change-Id: Icbf1ddd8d9d6c8b24a9938f3d31a59ea70360074
2015-08-08 11:37:30 +00:00
Davanum Srinivas eecaaeddaa Add tox target to find missing requirements
Change-Id: I2cfa6c2f5f763d710b2ac2f07bdf5aa80bc4a408
2015-06-26 07:18:38 -04:00
Davanum Srinivas 9433f0d394 Advertise support for Python3.4 / Remove support for 3.3
Change-Id: Ib6382d1ef3b3f336f1c83390b544d7e659c88511
2015-05-10 19:13:42 -04:00
Doug Hellmann 94624a77fe Remove py33 env from default tox list
We no longer need to test on Python 3.3, so remove that environment from
the default list in the tox configuration.

Change-Id: Icb3f22224678f9dad5de610a40d43fbef47be7ef
2015-03-11 19:51:58 +00:00
Jenkins e5a357dd14 Merge "Activate pep8 check that _ is imported" 2014-12-12 15:39:10 +00:00