This chnage adds the pre-commit config and
tox targets to run codespell both indepenetly
and via the pep8 target.
This change correct all the final typos in the
codebase as detected by codespell.
Change-Id: Ic4fb5b3a5559bc3c43aca0a39edc0885da58eaa2
This patch introduces the new table and associated migrations to support
share mapping between shares and instances.
Manila is the OpenStack Shared Filesystems service.
These series of patches implement changes required in Nova to allow the shares
provided by Manila to be associated with and attached to instances using
virtiofs.
Implements: blueprint libvirt-virtiofs-attach-manila-shares
Change-Id: Id2956188eef2160e718f08d63ced81e9a25e47af
This is being reverted because it's overly strict and complaining
that upgrade-related work has not been done before it should have or
needs to have been done. This may be re-added later when we start
depending on these linkages.
Closes-Bug: #2039597
This reverts commit 27f384b7ac.
Change-Id: Ifa5b82ca3b83d0ba481aa7a062827bd8e838989a
this is the inital patch of applying codespell to nova.
codespell is a programing focused spellchecker that
looks for common typos and corrects them.
i am breaking this into multiple commits to make it simpler
to read and will automate the execution of codespell
at the end of the series.
Change-Id: If24a6c0a890f713545faa2d44b069c352655274e
Per the SQLAlchemy docs [1]:
The relationship.backref keyword should be considered legacy, and use
of relationship.back_populates with explicit relationship() constructs
should be preferred.
A number of the relationships defined here don't have foreign keys (long
live mordred?) so their conversion is slightly more difficult than would
otherwise be the case. A blog post is available to explain what's going
on [2] and might be worth a read. The learnings from that blog post do
have the benefit of allowing us to simplify some existing relationships
that had unnecessary arguments defined.
[1] https://docs.sqlalchemy.org/en/14/orm/backref.html
[2] https://that.guru/blog/sqlalchemy-relationships-without-foreign-keys/
Change-Id: I5a135b012dabdff7cf06204fc3c5438aaa0985c9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
About two thirds of our online data migrations are using 'count' as
the name for the limit argument, where the others (and the runner)
call it max_count. This makes them all consistent with the original
intent.
Change-Id: I33a4afdef0f19bf667af616b1ba2b032d166b546
Previously, we archived deleted rows in batches of max_rows parents +
their child rows in a single database transaction. Doing it that way
limited how high a value of max_rows could be specified by the caller
because of the size of the database transaction it could generate.
For example, in a large scale deployment with hundreds of thousands of
deleted rows and constant server creation and deletion activity, a
value of max_rows=1000 might exceed the database's configured maximum
packet size or timeout due to a database deadlock, forcing the operator
to use a much lower max_rows value like 100 or 50.
And when the operator has e.g. 500,000 deleted instances rows (and
millions of deleted rows total) they are trying to archive, being
forced to use a max_rows value several orders of magnitude lower than
the number of rows they need to archive was a poor user experience.
This changes the logic to archive one parent row and its foreign key
related child rows one at a time in a single database transaction
while limiting the total number of rows per table as soon as it reaches
>= max_rows. Doing this will allow operators to choose more predictable
values for max_rows and get more progress per invocation of
archive_deleted_rows.
Closes-Bug: #2024258
Change-Id: I2209bf1b3320901cf603ec39163cf923b25b0359
This adds a compute_id column to the instances table in anticipation
of later patches to link instances to computes by row id instead of
name. It also adds a dest_compute_id to migrations, so that we can
positively track where the instance is to end up.
Related to blueprint compute-object-ids
Change-Id: Ia9c367c08604926598b3c1c21bd8dbd2ba0bfbc6
Since [1] (release 1.11.0), the Alembic operations "now enforce
keyword-only arguments as passed as keyword and not positionally"
(from the commit message).
This change is compatible with the previous versions (as confirmed
in the CI).
[1]df75e85489
Closes-Bug: #2019948
Change-Id: I020585a192ec5bcfb92cf92ef235da379ccb985e
If you load a table schema using the 'autoload_with' parameter, a
connection will be auto-begun. We were previously getting away with this
because we were relying on nested connections, but these are no longer a
thing in SQLAlchemy 2.0. Explicitly manage connections when loading the
table schema.
Change-Id: I1a37deef449f72b54cb3cecf0c677203256c0443
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The function signature for this method is change in SQLAlchemy 2.x:
TypeError: insert() takes 1 positional argument but 2 were given
We missed this previously as the deprecation checks in SQLAlchemy were
presumably checking for non-None values.
Change-Id: I4e366a0dbad56e739370b8bec0b84f6162f3d7a0
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We were previously calling 'connect()' on the 'connectable' object in
'run_migrations_online', regardless of whether it was an 'Engine' or
'Connection' object. This worked because, as noted in an inline comment,
"when connectable is already a Connection object, calling 'connect()'
gives us a *branched connection*." This is no longer the case. From the
SQLAlchemy docs [1]:
The Connection object does not support "branching", which was a
pattern by which a sub "connection" would be used that refers to this
connection as a parent.
Update our code to reflect this change, using the newly updated example
from the SQLAlchemy cookbook doc [2] as inspiration.
[1] https://docs.sqlalchemy.org/en/14/core/future.html#sqlalchemy.future.Connection
[2] https://alembic.sqlalchemy.org/en/latest/cookbook.html#connection-sharing
Change-Id: I79f98930017f26492055c1295fc7900a2257ac68
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
sqlalchemy-migrate does not (and will not) support sqlalchemy 2.0. We
need to drop these migrations to ensure we can upgrade our sqlalchemy
version.
Change-Id: I7756e393b78296fb8dbf3ca69c759d75b816376d
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
* There were two unique constrains on the same column uuid of instances.
This change drops one of them. The second constraint was introduced with
https://review.opendev.org/c/openstack/nova/+/97946, but was pending cleanup
since.
* In console_auth_tokens there was a unique constraint and another index on
column token_hash.
Closes-Bug: #1641185
Change-Id: I0ffa47d2afbfbfa63651991b3791dfad3e1832e1
This change adds the `encrypted`, `encryption_secret_uuid`,
`encryption_format` and `encryption_options` to the BlockDeviceMapping
object and associated table in the database.
Change-Id: I6178c9bc249ef4d448de375dc16d82b2d087fc90
During review of change Ic43c21038ee682f9733fbde42c6d24f8088815fc, we
noticed that we were leaking connections if we had an early return from
'_archive_deleted_rows_for_table'. Correct this.
Change-Id: I748d962b6c7012e9bc2b8c91519da99d2d4bd240
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolving the following SAWarning warnings:
Coercing Subquery object into a select() for use in IN(); please pass
a select() construct explicitly
SELECT statement has a cartesian product between FROM element(s)
"foo" and FROM element "bar". Apply join condition(s) between each
element to resolve.
While the first of these was a trivial fix, the second one is a little
more involved. It was caused by attempting to build a query across
tables that had no relationship as part of our archive logic. For
example, consider the following queries, generated early in
'_get_fk_stmts':
SELECT instances.uuid
FROM instances, security_group_instance_association
WHERE security_group_instance_association.instance_uuid = instances.uuid
AND instances.id IN (__[POSTCOMPILE_id_1])
SELECT security_groups.id
FROM security_groups, security_group_instance_association, instances
WHERE security_group_instance_association.security_group_id = security_groups.id
AND instances.id IN (__[POSTCOMPILE_id_1])
While the first of these is fine, the second is clearly wrong: why are
we filtering on a field that is of no relevance to our join? These were
generated because we were attempting to archive one or more instances
(in this case, the instance with id=1) and needed to find related tables
to archive at the same time. A related table is any table that
references our "source" table - 'instances' here - by way of a foreign
key. For each of *these* tables, we then lookup each foreign key and
join back to the source table, filtering by matching entries in the
source table. The issue here is that we're looking up every foreign key.
What we actually want to do is lookup only the foreign keys that point
back to our source table. This flaw is why we were generating the second
SELECT above: the 'security_group_instance_association' has two foreign
keys, one pointing to our 'instances' table but also another pointing to
the 'security_groups' table. We want the first but not the second.
Resolve this by checking if the table that each foreign key points to is
actually the source table and simply skip if not. With this issue
resolved, we can enable errors on SAWarning warnings in general without
any filters.
Change-Id: I63208c7bd5f9f4c3d5e4a40bd0f6253d0f042a37
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warning:
The current statement is being autocommitted using implicit
autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
method of Engine or Connection in order to use an explicit transaction
for DML and DDL statements.
I genuinely expected this one to be more difficult to resolve, but we
weren't using this as much as expected (thank you, non-legacy
enginefacade).
With this change, we appear to be SQLAlchemy 2.0 ready.
Change-Id: Ic43c21038ee682f9733fbde42c6d24f8088815fc
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warning:
The Column.copy() method is deprecated and will be removed in a future
release.
The recommended solution here (by zzzeek himself) is to use the private
method. This method isn't perfect (hence why the public version was
deprecated) but it's more than okay for what we want. The alternative is
to effectively vendor a variant of the 'Column.copy()' code, which means
we'll lose out on any future bug fixes.
Change-Id: Ia663251dfa7cf8f7d33f19902a92bcc586ae9f43
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warning:
Implicit coercion of SELECT and textual SELECT constructs into FROM
clauses is deprecated; please call .subquery() on any Core select or
ORM Query object in order to produce a subquery object.
This one was easy.
Change-Id: Ifeab2aa8cef7ad151d5d5f92937e90ab34b96e8a
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warnings:
Using strings to indicate column or relationship paths in loader
options is deprecated and will be removed in SQLAlchemy 2.0. Please
use the class-bound attribute directly.
Using strings to indicate relationship names in Query.join() is
deprecated and will be removed in SQLAlchemy 2.0. Please use the
class-bound attribute directly.
This is rather tricky to resolve. In most cases, we can simply make use
of getattr to fetch the class-bound attribute, however, there are a
number of places were we were doing "nested" joins, e.g.
'instances.info_cache' on the 'SecurityGroup' model. These need a little
more thought.
Change-Id: I1355ac92202cb504a7814afaa1338a4a511f9b54
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warnings:
The MetaData.bind argument is deprecated and will be removed in
SQLAlchemy 2.0.
The ``bind`` argument for schema methods that invoke SQL against an
engine or connection will be required in SQLAlchemy 2.0.
We must retain a couple of workarounds to keep sqlalchemy-migrate happy.
This shouldn't be an issue since this library is not compatible with
SQLAlchemy 2.x and will have to be removed (along with the legacy
migration files themselves) when we eventually support 2.x.
Change-Id: I946b4c7926ba2583b84ab95a1fe7aedc6923d9f8
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following RemovedIn20Warning warnings:
The Row.keys() method is considered legacy as of the 1.x series of
SQLAlchemy and will be removed in 2.0. Use the namedtuple standard
accessor Row._fields, or for full mapping behavior use
row._mapping.keys()
An additional warning that appears to have been resolved in the interim
is also removed.
Change-Id: I0c33130a745b986f1bcd2ec177f78e3bb68d2271
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following RemovedIn20Warning warning:
Passing a string to Connection.execute() is deprecated and will be
removed in version 2.0. Use the text() construct, or the
Connection.exec_driver_sql() method to invoke a driver-level SQL
string.
Change-Id: I44d6bf1ebfaf24f00a21389364456bceaae7c4d1
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warning:
The insert.inline parameter will be removed in SQLAlchemy 2.0. Please
use the Insert.inline() method.
Change-Id: Ib5fcb841294b4e20fe085e8603d4132e97be7db9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warning:
The legacy calling style of select() is deprecated and will be removed
in SQLAlchemy 2.0. Please use the new calling style described at
select().
Change-Id: I36e43e30e07f4904c7b49925cefe804af45cff6c
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warnings:
The autoload parameter is deprecated and will be removed in version
2.0. Please use the autoload_with parameter, passing an engine or
connection.
Change-Id: I10e9bbf14706aece2a59ebb5861004c5b852a592
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
autopep8 is a code formating tool that makes python code pep8
compliant without changing everything. Unlike black it will
not radically change all code and the primary change to the
existing codebase is adding a new line after class level doc strings.
This change adds a new tox autopep8 env to manually run it on your
code before you submit a patch, it also adds autopep8 to pre-commit
so if you use pre-commit it will do it for you automatically.
This change runs autopep8 in diff mode with --exit-code in the pep8
tox env so it will fail if autopep8 would modify your code if run
in in-place mode. This allows use to gate on autopep8 not modifying
patches that are submited. This will ensure authorship of patches is
maintianed.
The intent of this change is to save the large amount of time we spend
on ensuring style guidlines are followed automatically to make it
simpler for both new and old contibutors to work on nova and save
time and effort for all involved.
Change-Id: Idd618d634cc70ae8d58fab32f322e75bfabefb9d
There are a number of nova-network tables which we can now drop.
nova-network
Feature removed entirely in Ussuri.
- dns_domains
- fixed_ips
- floating_ips
- networks
- provider_fw_rules
- security_group_default_rules
Unfortunately we can't get rid of the security group-related entries due
to the unfortunate presence of the 'security_groups' attribute on the
'Instance' object and corresponding table, which in turn brings in a
load of other tables. We'll address that separately. For now, just drop
what we can easily drop.
Change-Id: I8858faa14119f4daa9630b0ff6dcf082d0ff8fba
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We've built up a large amount of tables that are no longer used for
anything, given the removal of their users in past releases. We'd like
to remove those, but before we do that we need to drop the models. This
means there are no references to the tables come N+2, at which point we
can remove the tables themselves.
XenAPI virt driver
Feature removed entirely in Victoria.
- agent_builds
- bw_usage_cache
- console_pools
- consoles
Cells v1
Feature removed entirely in Train.
- cells
Volume Snapshots
Feature removed entirely in Liberty.
- snapshots
EC2 API
Feature removed entirely in Mitaka. Note that these tables are *not*
used by the separate ec2-api project.
- snapshot_id_mappings
- volume_id_mappings
There are still some tables related to nova-network left here. Those are
unfortunately referenced from elsewhere, so we need to clean them up
separately.
Change-Id: I5e3d022fdf7328a1132f6e00998a3286b19be69a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Finish up removing these entries from the versioned instance
notifications. They're useless since we dropped support for the XenAPI
virt driver. The underlying model is retained for now: that will be
handled separately.
Change-Id: I774c50fca99bc655ca5010e3b9d8247b739293b3
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Remove all of the models that were moved to the API database many many
cycles ago.
Change-Id: Ib327f47b889dbccd5279f43c39203ed27689748b
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Zuul