Commit Graph

687 Commits

Author SHA1 Message Date
Stephen Finucane e3e35ba2d0 pre-commit: Add mypy
Add mypy so we can run these checks as part of our linting step. Note
that mypy is the last item in the list of checks since flake8 is going
to catch e.g. syntax issues which should be addressed first.

This also allows us to drop use of mypywrap: rather than wrapping mypy,
we can make use of mypy's configuration to specify the files we wish to
run checks on.

Change-Id: Ie51401ead4e31ccd70ab833c59cb397f802ff188
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-12-20 18:31:47 +00:00
Takashi Kajinami 7a832ea91c Fix python shebang
The current shebang requires python 2 instead of python 3.

Trivial-Fix

Change-Id: I9172b43648c28e51edfd5d8695e750c18292ff90
2023-10-18 00:43:20 +09:00
Elod Illes fe125da63b CI: fix backport validator for new branch naming
validate-backport job started to fail as only old stable branch naming
is accepted. This patch extends the script to allow numbers and dot as
well in the branch names (like stable/2023.1).

Change-Id: Icbdcd5d124717e195d55d9e42530611ed812fadd
2023-05-11 16:23:53 +02:00
Ade Lee 9a7028f02c Test setting the nova job to centos-9-stream
Depends-On: https://review.opendev.org/c/openstack/tempest/+/831607

Change-Id: Ic8da6ee0313a911d742190ea5b0d4362cb6aef2f
2022-07-13 14:08:03 +00:00
Stephen Finucane e07bb310b6 tools: Ignore bot-generated branch creation patches
Don't force infra to change their bot because of our weird script. If
it's from the OpenStack Release Bot, it's probably good and we can skip
the backport check.

Change-Id: I682bd198a74fc252e47e8bdfb33bd21782b63ff9
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2021-09-27 10:20:34 +01:00
Stephen Finucane eb728e877a db: Final cleanups
Some things that were missed in previous patches and are thrown together
here:

- Add alembic as an explicit dependency (we were getting it transitively
  from oslo.db). We also bump the sqlalchemy dependency to a 1.4.x
  release, which is the minimum supported by our chosen version of
  alembic (more on this below)
- Remove tooling related to the old migrations
- Fix the tox whitelisting of the flaky MySQL tests

On the SQLAlchemy front, we opt for 1.4.13. Technically alembic should
support anything from 1.4.0, however, with SQLAlchemy >= 1.4.0, < 1.4.13
we see errors like the following in some tests:

  sqlalchemy.exc.InvalidRequestError: Entity namespace for
  "count(instance_mappings.id)" has no property "queued_for_delete"

There's nothing specific about this in the release notes for 1.4.13 [1]
but it definitely fixes things.

[1] https://docs.sqlalchemy.org/en/14/changelog/changelog_14.html#change-1.4.13

Change-Id: I4c8eb13f11aa7471c26a5ba326319aef245c9836
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2021-08-17 13:50:19 +01:00
Stephen Finucane bf8b5fc7d0 db: Move remaining 'nova.db.sqlalchemy' modules
The two remaining modules, 'api_models' and 'api_migrations', are
moved to the new 'nova.db.api' module.

Change-Id: I138670fe36b07546db5518f78c657197780c5040
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2021-08-09 15:34:40 +01:00
Stephen Finucane 100b9dc62c db: Unify 'nova.db.api', 'nova.db.sqlalchemy.api'
Merge these, removing an unnecessary layer of abstraction, and place
them in the new 'nova.db.main' directory. The resulting change is huge,
but it's mainly the result of 's/sqlalchemy import api/main import api/'
and 's/nova.db.api/nova.db.main.api/' with some necessary cleanup. We
also need to rework how we do the blocking of API calls since we no
longer have a 'DBAPI' object that we can monkey patch as we were doing
before. This is now done via a global variable that is set by the 'main'
function of 'nova.cmd.compute'.

The main impact of this change is that it's no longer possible to set
'[database] use_db_reconnect' and have all APIs automatically wrapped in
a DB retry. Seeing as this behavior is experimental, isn't applied to
any of the API DB methods (which don't use oslo.db's 'DBAPI' helper),
and is used explicitly in what would appear to be the critical cases
(via the explicit 'oslo_db.api.wrap_db_retry' decorator), this doesn't
seem like a huge loss.

Change-Id: Iad2e4da4546b80a016e477577d23accb2606a6e4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2021-08-09 15:34:40 +01:00
Stephen Finucane bcf225daf4 db: Move main DB migrations
We place these in a 'legacy_migrations' directory, as we will soon be
adding alembic-based migrations in a 'migration' directory.

Change-Id: Ib927e4c48f59a467a913875111ffbf64ffe0de90
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2021-07-05 11:05:04 +01:00
Lee Yarwood 085de864db tests: Allow bindep and test-setup.sh to run on EL distros
Only install mariadb on EL based hosts. Also when using mariadb and
postgresql on EL based distros we need to ensure each service is
configured and actually started before using either.

Co-Authored-By: Ade Lee <alee@redhat.com>
Change-Id: I7122933d85bd7d0333c2c35e0f1a8414c1baa6d5
2021-06-18 12:34:27 +01:00
Stephen Finucane 98b01c9a59 Move 'check-cherry-picks' test to gate, n-v check
This currently runs in the 'check' pipeline, as part of the pep8 job,
which causes otherwise perfectly valid backports to report as failing
CI. There's no reason a stable core shouldn't be encouraged to review
these patches: we simply want to prevent them *merging* before their
parent(s). Resolve this conflict by moving the check to separate voting
job in the 'gate' pipeline as well as a non-voting job in the 'check'
pipeline to catch more obvious issues.

Change-Id: Id3e4452883f6a3cf44ff58b39ded82e882e28c23
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2021-06-16 15:51:38 +01:00
Stephen Finucane 74f5eb2ce9 Add generate schemas tool
This tool was used while building the compact-db-migrations-wallaby
series during Wallaby [1]. It might be helpful in the future, so commit
it now.

Change-Id: Idd556fac96ce621ad095862597c4f11851852fed
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2021-01-18 16:27:00 +00:00
Lee Yarwood 610396f8ad tools: Allow check-cherry-picks.sh to be disabled by an env var
The checks performed by this script aren't always useful to downstream
consumers of the repo so allow them to disable the script without having
to make changes to tox.ini.

Change-Id: I4f551dc4b57905cab8aa005c5680223ad1b57639
2020-12-07 12:53:42 +00:00
melanie witt 1e10461c71 Follow up for cherry-pick check for merge patch
This is a follow up to change
I8e4e5afc773d53dee9c1c24951bb07a45ddc2f1a which fixed an issue with
validation when the topmost patch after a Zuul rebase is a merge
patch.

We need to also use the $commit_hash variable for the check for
stable-only patches, else it will incorrectly fail because it is
checking the merge patch's commit message.

Change-Id: Ia725346b65dd5e2f16aa049c74b45d99e22b3524
2020-10-08 05:45:24 +00:00
Stephen Finucane 45c0ea4a3e tools: Remove xenserver tooling
These will not be used in a world without the XenAPI driver.

Change-Id: I5bc3c7855b817c4ce2b8919c76be80cceabeec9e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2020-08-31 15:53:35 +01:00
Zuul 43802e5170 Merge "Fix user creation with GRANT in MySQL 8.0(Ubuntu Focal)" 2020-07-02 18:40:46 +00:00
Ghanshyam Mann e0d541073d Fix user creation with GRANT in MySQL 8.0(Ubuntu Focal)
In Ubuntu Bionic (18.04) mysql 5.7 version used to create
the user implicitly when using using the GRANT.

Ubuntu Focal (20.04) has mysql 8.0 and with mysql 8.0 there
is no implicit user creation with GRANT. We need to
create the user first before using GRANT command.

Nova unit and functional tests job using tools/test-setup.sh
script start failing when running on Ubuntu Focal

https://zuul.opendev.org/t/openstack/build/8b0f4fcc21854655a638c413b6fe1a91

Error log:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1064 (42000) at line 4: You have an error in your SQL syntax; check the manual
that corresponds to your MySQL server version for the right syntax to use near
'identified by 'openstack_citest' WITH GRANT OPTION' at line 2

Story: #2007865
Task: #40200

Change-Id: I97b0dcbb88c6ef7c22e3c55970211bed792bbd0d
Closes-Bug: #1885825
2020-06-30 21:21:39 -05:00
Elod Illes c7c48c6f52 Fix cherry-pick check for merge patch
Cherry-pick check script validates the proposed patch's commit message.
If a patch is not on top of the given branch then Zuul rebases it to
the top and the patch becomes a merge patch. In this case the script
validates the merge patch's commit message instead of the original
patch's commit message and fails.

This fix selects the parent of the patch if it is a merge patch.

Change-Id: I8e4e5afc773d53dee9c1c24951bb07a45ddc2f1a
2020-06-26 20:31:20 +02:00
Dan Smith aebc829c4e Check cherry-pick hashes in pep8 tox target
This adds a tools/ script that checks any cherry-picked hashes
on the current commit (or a provided commit) to make sure that
all the hashes exist on at least master or stable/.* branches.
This should help avoid accidentally merging stable backports
where one of the hashes along the line has changed due to conflicts.

Change-Id: I4afaa0808b75cc31a8dd14663912c162281a1a42
2020-06-15 11:15:31 -07:00
Stephen Finucane 34ecf5ab91 tox: Integrate mypy
mypy is an experimental optional static type checker for Python that
aims to combine the benefits of dynamic (or "duck") typing and static
typing. While still in development, most features are supported and it's
already being used by real world projects like Sphinx. Let's start small
by integrating it into some of the interfaces that nova exposes. We can
further build upon this if it works out.

This change sets up the boilerplate necessary to use mypy in nova. Type
annotations are not included for any module - these will be added
separately. We're calling mypy by way of a script, as this allows us to
store a list of files that we have converted while we're in the process
of adding type annotations where necessary.

Change-Id: I75ab46a6768c4ca2050fdde2b7f8eeb90724c8c6
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2020-05-15 15:59:53 +01:00
Stephen Finucane 1515a229e3 Remove future imports
These particular imports are no longer needed in a Python 3-only world.

Change-Id: Ia1b60ce238713b86f126e2d404199d102fdbc5bc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2020-03-24 15:05:36 +00:00
Monty Taylor e7cc5efd20 Keep pre-commit inline with hacking and fix whitespace
The flake8 hook for pre-commit installs its own flake8 which is
not what hacking is pinned to. This results in a bunch of errors.
By doing a local hook we can cause these to be the same.

There are flake8 exclusions in tox.ini that need to be used
in pre-commit. Also we should exclude svg files from the whitespace
modifications, because who cares.

Once those are fixed, running pre-commit -a fixes some whitespace
issues. While the normal use isn't to run -a - if someone does make
a patch that touches any of these they'll get erroneous unrelated
errors compared to their lower work - so fix all of the existing
issues, which are mostly space-at-end-of-line issues it seems.

Change-Id: I78e753492e99c03b30a0a691db3bd75ee3d289c9
2019-12-12 14:56:39 +00:00
Stephen Finucane 5f8dd65f6c Integrate 'pre-commit'
This nifty tool provides a extensible way to configure pre-commit hooks
to do everything from running linters to spellchecking [1] and has
already been adopted by a few OpenStack projects [2]. Let's start using
it as a faster, more automatic alternative to the 'fast8' tox target and
iterate on it as we find more things to use it for. The 'fast8' target
is not deprecated though a note is included for end-users to alert them
to this great new tool.

Note that the 'check-executables-have-shebangs' actually failed and
highlighted that the 'contributor/development-environment.rst' document
was marked as executable when it shouldn't have been. This is corrected.

[1] https://pre-commit.com/
[2] https://review.opendev.org/#/q/file:.pre-commit-config.yaml

Change-Id: I256e75eceeb3495dce04885ca42d16adb4160cf3
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2019-11-28 12:46:55 +00:00
Adam Spiers 5df748b2ed Make it easier to run a selection of tests relevant to ongoing work
During development of a new git commit, locally running a whole unit
or functional test suite to check every minor code change is
prohibitively expensive.  For maximum developer productivity and
happiness, it's generally desirable to make the feedback loop of the
traditional red/green cycle as quick as possible.

So add run-tests-for-diff.sh and run-tests.py to the tools/
subdirectory, using a few tricks as explained below to help with this.

run-tests.py takes a list of files on STDIN, filters the list for
tests which can be run in the current tox virtualenv, and then runs
them with the correct stestr options.

run-tests-for-diff.sh is a simple wrapper around run-tests.py which
determines which tests to run using output from "git diff".  This
allows running only the test files changed/added in the working tree:

    tools/run-tests-for-diff.sh

or by a single commit:

    tools/run-tests-for-diff.sh mybranch^!

or a range of commits, e.g. a branch containing a whole patch series
for a blueprint:

    tools/run-tests-for-diff.sh gerrit/master..bp/my-blueprint

It supports the same "-HEAD" invocation syntax as flake8wrap.sh (as
used by the "fast8" tox environment):

    tools/run-tests-for-diff.sh -HEAD

run-tests.py uses two tricks to make test runs as quick as possible:

  1. It's (already) possible to speed up running of tests by
     source'ing the "activate" file for the desired tox virtualenv,
     e.g.

        source .tox/py36/bin/activate

     and then running stestr directly.  This saves a few seconds by
     skipping the overhead introduced by running tox.

  2. When only one test file needs to be run, specifying the -n option
     to stestr will skip the costly test discovery phase, saving
     several more valuable seconds.

Future commits could build on top of this work, harnessing a framework
such as watchdog / watchmedo[0] or Guard[1] in order to automatically
run relevant tests every time your editor saves changes to a .py file.

[0] https://github.com/gorakhargosh/watchdog - Python-based
[1] https://guardgem.org - probably best in class, but Ruby-based so
    maybe unacceptable for use within Nova.

Change-Id: I9a9bda5d29bbb8d8d77f769cd1abf7c42a18c36b
2019-08-19 17:48:39 +01:00
ZhongShengping 7ecaa3fcf8 Replace git.openstack.org URLs with opendev.org URLs
Thorough replacement of git.openstack.org URLs with their opendev.org
counterparts.

Change-Id: I3e0af55e0707f04428a422b973d016ad30c82a12
2019-04-24 13:59:57 +08:00
Matthew Edmonds d7ed9a550b Make Xen code py3-compatible
A couple places in the Xen code have been using syntax that is
python2-specific. This replaces those instances with code that will
work in both python2 and python3.

PEP 3108 [1] moves urllib.urlopen to urllib.request.urlopen. The six
module is used in order to work in both python2 and python3 [2].

PEP 3110 [3] changed the grammar for except clauses, such that the
'as' keyword is now required in place of a comma when specifying an
exception variable.

[1] https://www.python.org/dev/peps/pep-3108/
[2] https://pythonhosted.org/six/#module-six.moves.urllib.request
[3] https://www.python.org/dev/peps/pep-3110/

Change-Id: I1235d767718a4207f4cef3e5b140319d003ad7b0
2018-08-10 20:04:19 +00:00
Stephen Finucane a34731dad1 Convert 'placement_api_docs' into a Sphinx extension
This ensures it will get run in the gate.

Change-Id: I923c39d3115b595b8de94e85d2977eb7782d98e2
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2018-07-02 11:21:47 +01:00
Stephen Finucane de4ab973d5 trivial: Remove 'tools/releasenotes_tox.sh'
This is not needed since reno 2.1.0 (commit f8fc8f97) [1].

[1] f8fc8f97ff

Change-Id: I66ba0e0835970b965acce00ba69d143bd4471037
2018-06-28 15:06:46 +01:00
Matt Riedemann 9f48aee9b0 Remove deprecated monkey_patch config options
These were deprecated in Queens:

  Id793ac2c5cdc5dc473f95eac53b77617a1e389da

And can now be removed.

Change-Id: I27518d1154ec09c57700905bc58ae6a3ce1d8049
2018-05-16 11:40:41 -04:00
Matthew Treinish 9c56c78513
Finish stestr migration
Nova switched to using stestr for all it's in tree testing except for
the coverage job. This commit updates that job so it also uses stestr
and cleans up the last bits of testr setup left in tree. As part of
this change this moves all the one off per job configuration into the
tox job definitions instead of hiding it in either the .stestr.conf or
in a wrapper script.

Change-Id: I36e0e791d485b44641a1ed8770eca8e95aca19be
2017-11-24 16:51:12 -05:00
naichuans 693ace79fb xenapi: cached images should be cleaned up by time
For xenapi driver, there needs to be some way to delete cached
images based on when they were created. add an optional arg to
control delete operation.

Change-Id: I24fc45e989aa951aab55a261fce77f7e3667d988
Closes-bug: 1481689
2017-10-23 16:29:16 +00:00
melanie witt 53f244da12 Make setenv consistent for unit, func, and api-samples
Since ostestr switched to running stestr under the covers, we lost the
old magic setting of the environment variables via .testr.conf for
capturing stderr/stdout and the test timeout. This makes the unit,
functional, and api-samples envs consistent with the py27 env that was
already updated to set those variables.

This also updates the pretty_tox3.sh script to run stestr directly.

Change-Id: I27fa9b7e25c1a1dc921653eec84864423f898a85
2017-10-05 21:45:46 +00:00
Jenkins 823b118b8c Merge "trivial: Remove files from 'tools'" 2017-09-06 18:31:35 +00:00
Chris Dent c324551015 Prevent blank line at start of migration placeholders
Because those blank lines sure are annoying.

Change-Id: I7b73536a89d18e0752651c198018eff6cc8b0e57
2017-08-25 18:43:19 +01:00
Andrey Volkov fa5749727f [placement] Make placement_api_docs.py failing
If there is no api reference for some placement API method
tox -e placement-api-ref will fail now.

Change-Id: Ifefedcd51f1f8d5e741ef6d02b7a0879a185e05f
2017-08-10 11:27:13 +00:00
Stephen Finucane 017b5b869a trivial: Remove files from 'tools'
Two files are removed.

'ebtables.workaround' was required for libvirt < 1.2.11 for the reasons
described in the file itself. We no longer support this version of
libvirt, thus, the workaround can be removed.

'regression_tester' is a tool for running tests in a patch against old
code, to ensure the tests validate what the patch is supposed to be
fixing. This is unmaintained and is not referenced anywhere, and should
therefore be removed as it's likely bitrotted (or will do so).

Change-Id: Iac7f16ff4c178b5e71b786c332ca46a78bef5e3c
TrivialFix
2017-08-07 16:01:43 +01:00
Stephen Finucane 21d5dfacb4 tools/xenserver: Remove 'cleanup_sm_locks'
XenServer 6.0 has long since been replaced and the bug was fixed in 6.1
[1]. No need to keep this legacy script around.

[1] https://bugs.launchpad.net/nova/+bug/1103158/comments/1

Change-Id: I8c41e88281aaf4250859622142a2f03f4452e1be
2017-08-07 12:26:53 +00:00
Andrey Volkov 90cabd0c8c [placement] Fix placement-api-ref check tool
Some time ago the placement-api-ref source structure was changed
(Ia2fd62ae7f401cad34ee7c2b355c9a5ab1c93f6b) to use inc-files istead of
putting all in index.rst, but that change broke placement_api_docs.py.

This change modifies placement_api_docs.py to go through all *.inc files
in placement-api-ref directory and scan for the "rest method" header.

Change-Id: I60150ab51173934f848662499dcf6af18625697e
2017-05-24 12:15:28 +03:00
Jenkins 4613b33674 Merge "Structure for simply managing placement-api-ref" 2017-03-24 20:17:52 +00:00
Chris Dent 8c59991363 Structure for simply managing placement-api-ref
This consists of a duplicate of the [nova-]api-ref setup and
conf.py along with tooling to fail the tox -edocs target when
a route that is defined in
nova.api.openstack.placement.handler.ROUTE_DECLARATIONS is not
present in placement-api-ref/source/index.rst.
tools/placement_api_docs.py will report which routes are missing.

Though completely gameable (as demonstrated in the current lame
index.rst) it's better than nothing and provides some useful
structuring on what to do next. It's also the case that the 'docs'
target in tox is not part of gating.

The response for GET / is in place with the necessary
parameters.yaml for it to be correctly described. The 'get-root.json'
file provides the JSON of the expected response. The expectation is
that later commits will add information for other urls and their
JSON files will be named method-path-separated-by-dash.json with a
request/response qualifier as necessary.

Followup patches will add other routes.

A new parameters.yaml is used instead of reusing the one from
api-ref as there isn't a lot of expected overlap and having a
separate file will ease eventual extraction.

Running tox -eplacement-api-ref will generate the docs for review,
with output in placement-api-ref/build/html/index.html.

This will be hooked up with CI to deploy the generated docs,
eventually.

Change-Id: Ifb4d91d39db0e49b55952e37cdfc9f63dcd37aa3
2017-03-24 15:15:53 +03:00
Jenkins e3eac21382 Merge "More usage of ostestr and cleanup an unused dependency" 2017-03-20 16:04:47 +00:00
ChangBo Guo(gcb) 6b73c281bc Remove usage of config option verbose
oslo.log deprecate config option verbose since aug 1, 2015 in review:
in https://review.openstack.org/#/c/206437/

That was a long time ago so it should be possible to remove it now. This
was already merged once but had to be reverted because some projects
were still relying on it. Oslo team plan to remove it, so we need clean
up its usage before removing it.

Closes-Bug: #1663570

Change-Id: I072e1078486f50cebea92695ff46c1af879eb23d
2017-02-27 10:02:45 +08:00
Luigi Toscano 8e34aece1a More usage of ostestr and cleanup an unused dependency
Use ostestr as runner instead of the custome pretty_tox.sh script
(pretty much removed everywhere). pretty_tox3.sh is just a wrapper
for ostestr but it could be probably removed as well going forward.
Also remove the tempest-lib dependency, which originally provided
subunit-trace (now from os-testr).

Change-Id: I4390964ce83167d634b80f696e82a0e199582ce6
2017-02-23 18:25:08 +01:00
Jenkins a026af51dc Merge "tools: Remove 'colorizer'" 2017-02-23 12:42:44 +00:00
Jenkins 71f92d4f3a Merge "tools: Remove 'with_venv'" 2017-02-23 12:42:17 +00:00
Jenkins b6c32ade17 Merge "tools: Remove 'install_venv', 'install_venv_common'" 2017-02-23 12:41:45 +00:00
Jenkins 3cd849a162 Merge "tools: Remove 'clean-vlans'" 2017-02-23 12:41:19 +00:00
Jenkins 1b5cbe3c95 Merge "tools: Remove 'enable-pre-commit-hook'" 2017-02-23 12:37:32 +00:00
Andreas Jaeger d60dffc6be Prepare for using standard python tests
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests. This is exactly what is
currently done by OpenStack CI in project-config.

This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.

See also
http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html

Update devref for this.

Needed-By: Iea42a0525b2c5a5cdbf8604eb23a6e7b029f6b48
Change-Id: Ie9bae659077dbe299eea131572117036065bdccf
2017-02-09 17:52:55 +00:00
Huan Xie 062065ca9e XenAPI Use os-xenapi lib for nova
XenServer has released os-xenapi lib on pypi, this patch is to
use os-xenapi in nova project. In this patch, we only change
the usage of XenAPI to os-xenapi and fix unit tests. In the next
patch, we will delete those unused files.

Partially-Implements: blueprint add-os-xenapi-library

Depends-On: Ic327135b893a77672fd42af919f47f181e932773

Change-Id: I424dfcd507c4b7fbeea5245cc1c234dec01d5781
2017-01-10 17:26:23 -08:00