The two remaining modules, 'api_models' and 'api_migrations', are
moved to the new 'nova.db.api' module.
Change-Id: I138670fe36b07546db5518f78c657197780c5040
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change If90d9295b231166a28c2cc350d324691821a696b kicked off this effort
but only change the migrations. This change completes the job.
Change-Id: Ic0f2c326ebce8d7c89b0debf5225cbe471daca03
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Picked up with vulture [1].
[1] https://pypi.org/project/vulture/
Change-Id: I0b318ea5309d201499f0a9ef6f205425c89e6946
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This migration was added in Ocata and backported to Newton in change
I8a05ee01ec7f6a6f88b896f78414fb5487e0071e to deal with Mitaka-era
build_requests records that would not have an instance_uuid value
and thus raise a ValueError in BuildRequest._from_db_object (because
BuildRequest.instance_uuid is not nullable).
This is essentially a revert of that change now since operators
have had long enough to run the migration. If anyone were to skip
level upgrade from Mitaka to Train (which we don't support, we require
you to roll through), and hit an issue with this they could simply
execute this on their nova_api DB:
DELETE FROM build_requests WHERE instance_uuid IS NULL;
Change-Id: Ie9593657544b7aef1fd7a5c8f01e30e09e3fcce6
The instance_mappings table already contains the project_id for an
instance. This adds the corresponding user_id. This also adds an index
on (user_id, project_id) because later patches in this series will be
using these columns for quota counting.
Part of blueprint count-quota-usage-from-placement
Change-Id: Id9eef7a58f66c73cd638c6c3e228447b7ab81e34
This adds an online data migration for the queued_for_delete flag
on InstanceMapping. It does this by processing instance mappings that
do not have a value set for the field (i.e. are set to NULL) and queries
instances in each cell that need to be updated.
Note that this has what looks like a scary schema change in it. When the
column was added, it was specified as default=False and this changes that
value. This is not actually a schema change, but rather just changes what
we are asking SQLAlchemy to do for us when we do an INSERT of that object
if/when a value for that field is not specified. If we had used the
'server_default' flag, that would have been encoded into the schema and
thus would require a migration to change. This patch removes it because
we need to actually be able to see those NULL values in the database in
order to efficiently use them to query for records needing to be migrated.
Related to blueprint handling-down-cell
Change-Id: I0bdadec6f6b9dd7382247213a08fa139329a0683
The change items in this patch as below:
1. Add policy and rules to InstanceGroup, deprecate
the policies field.
2. Also, make _from_db_object and _create_in_db method to
support new "policy" field.
The internal usage of InstanceGroup.policies is converted to
use the new model in the REST API and RequestSpec compat code.
Related to blueprint complex-anti-affinity-policies
Change-Id: Ib33719a4b9599d86848c618a6e142c71ece79ca5
This column is being added to have information regarding an instance
being active/deleted in the nova-api database as a part of the handling
a down cell spec.
Related to blueprint handling-down-cell
Change-Id: Ic6253acab2ae08025d65add9a5be9ed463457290
A group can only have 1 policy associated it, but now we have
ONE to MANY model in group model, this patch change it to ONE
to ONE Model.
We also adapt create method to new model, and remove policy
update support in group.save because now we don't support to
update a policy of the group.
Related to blueprint complex-anti-affinity-policies
Change-Id: I95eb1c8470a567408696132ae656305be53dc5bb
This adds the rules column to instance_group_policy table.
The ''Text'' column "rules" which is a dict, it can be
applied to the policy, and represents the rules for
specific policy.
Related to blueprint complex-anti-affinity-policies
Change-Id: I61ffb5ddb2d808bfef4e60088f4524bd98e0474e
In order to prevent races where two different processes can try to write
the allocations for a consumer, we are adding a 'generation' column to
the consumers table. This will be incremented each time a consumer's
allocations change, and allocation requests whose generation doesn't
match the current value will be rejected.
Blueprint: add-consumer-generation
Change-Id: Ia247f0783c5bbd172aafbfd86372fabeeef557c3
This column has been added to the cell_mappings table since it is
required to represent if a cell is enabled or disabled as a part of the
cell-disable spec.
Related to blueprint cell-disable
Change-Id: Ibeddcb9a93c180f74727431b0579308c400436db
request_spec.instance_group.members is a list of instance UUIDs. It
can get so long that it overflows its TEXT column. This patch changes
request_spec.spec to MEDIUMTEXT.
Change-Id: I6bf0fa19b72887803e77b66698587c2108c9372a
Closes-bug: 1738094
Adds initial support for storing the relationship between parent and
child resource providers. Nested resource providers are essential for
expressing certain types of resources -- in particular SR-IOV physical
functions and certain SR-IOV fully-programmable gate arrays. The
resources that these providers expose are of resource class
SRIOV_NET_VF and we will need a way of indicating that the physical
function providing these virtual function resources is tagged with
certain traits (representing vendor_id, product_id or the physical
network the PF is attached to).
The compute host is a resource provider which has an SR-IOV-enabled
physical function (NIC) as a child resource provider. The physical
function has an inventory containing some total amount of SRIOV_NET_VF
resources. These SRIOV_NET_VF resources are allocated to zero or more
consumers (instances) on the compute host.
compute host (parent resource provider)
|
|
SR-IOV PF (child resource provider)
:
/ \
/ \
VF1 VF2 (inventory of child provider)
The resource provider model gets two new fields:
- root_provider_uuid: The "top" or "root" of the tree of nested
providers
- parent_provider_uuid: The immediate parent of the provider, or None
if the provider is a root provider.
The database schema adds two new columns to the resource_providers
table that contain the internal integer IDs that correspond to the
user-facing UUID values:
- root_provider_id
- parent_provider_id
The root_provider_uuid field is non-nullable in the ResourceProvider
object definition, and this code includes an online data migration to
automatically populate the root_provider_id field with the value of the
resource_providers.id field for any resource providers already in the
DB.
The root_provider_id field value is populated automatically when a
provider is created. If the parent provider UUID is set, then the
root_provider_id is set to the root_provider_id value of the parent. If
parent is unset, root_provider_id is set to the value of the id
attribute of the provider being created. The corresponding UUID values
for root and parent provider are fetched in the queries that retrieves
resource provider data using two self-referential joins.
The root_provider_id column allows us to do extremely quick lookups of
an entire tree of providers without needing to perform any recursive
database queries.
Logic in this patch ensures that no resource provider can be deleted if
any of its children have any allocations active on them. We also check
to ensure that when created or updated, a resource provider's parent
provider UUID actually points to an existing provider.
It's important to point out that qualitative trait information is only
associated with a resource provider entity, not the resources that
resource provider has in its inventory. This is the reason why nested
resource providers are necessary. In the case of things like NUMA nodes
or SRIOV physical functions, if a compute host had multiple SRIOV
physical functions, each associated with a different network trait,
there would be no way to differentiate between the SRIOV_NET_VF
resources that those multiple SRIOV physical functions provided if the
containing compute host had a single inventory item containing the
total number of VFs exposed by both PFs.
Change-Id: I2d8df57f77a03cde898d9ec792c5d59b75f61204
blueprint: nested-resource-providers
Co-Authored-By: Moshe Levi <moshele@mellanox.com>
This adds the description attribute to the Flavor object
and the accompanying database schema upgrade.
This is the first field on a flavor that we've allowed
to be updated, so it requires some new plumbing for doing
an update on the flavor record itself during save(). The
updateable fields are whitelisted so we don't leak other
updates in here, like name, flavorid, vcpus, etc.
As part of this change, we also have to be sure to not
persist the description in the embedded instance.flavor
since (1) it's a large field and (2) we are not going
to expose the instance.flavor.description out of the
servers API.
Versioned notifications will be handled in a later change.
Part of blueprint flavor-description
Change-Id: I6db4eb46df0d7ec025b969a46621823957503958
We were a bit naughty in Iecbe0eb5717afb0b13ca90d4868a3ca5f9e8902b in
that we didn't normalize the storage of Keystone project and user
identifiers in the API DB's new consumers table. This means that we
will use a whole lot more storage for what ends up being very
repetitive data.
This patch changes the consumers DB table schema's project_id and
user_id columns from VARCHAR(255) to INT data type. This should result
in significantly faster queries for usage information since 9X the
index records can fit into a single index block in memory (36-byte
UUIDs stored in VARCHAR(255) versus 4-byte integers. The more index
records we can fit into a single page of memory, the faster both scans
and seeks will be.
Let's address this now before anything uses the consumers table.
Change-Id: I1b7357739f2a7e55c55d3acb9bd604731c4a2b32
blueprint: placement-project-user
Originally it was felt that we would need this column to distinguish
between compute nodes and non-compute providers. With the advent of
traits, though, this column is no longer used or needed.
Closes-Bug: #1648197
Change-Id: I614db98727f4737deb6728ee874ab0f68024ebe5
This is the first patch of the series,
this patch adds tags to build_request
table and corresponding db update
script.
Change-Id: I01e1973449a572ecd647ede0a70d274eac1583bd
Part of blueprint support-tag-instance-when-boot
The name of model object should be singular. This patch corrects it.
Partial implement blueprint resource-provider-traits
Change-Id: I7600d7e3775d237e0593ec52fbb474e1dad079c1
This patch adds DB table `traits` and `resource_provider_traits` into
the api database.
The traits table is used to represent the trait. The resource_provider_traits
table is used to represent the relationship between trait and resource provider.
Part of implementation blueprint shared-resources-pike
Change-Id: I5c34bdd1423beab53cc4af45e016d9a9bba5ffda
In order to facilitate a future extraction of the placement service
we want to record the association between a resource provider and an
arbitrary aggregate uuid in its own table.
A PlacementAggregate model is joined from ResourceProvider via
ResourceProviderAggregate. Note that this structure is used so we can
join on ids instead of strings (the uuids). A direct mapping between
ResourceProvider uuid and Aggregate uuid was mooted earlier in the year
but was determined to be suboptimal.
The name 'placement_aggregates' is used as the least problematic of
several choices after discussion amongst several parties.
The data will be used by the forthcoming get_ and set_aggregates
methods on the ResourceProvider object.
Change-Id: Id0355cb022f68e962af306ff04cf724d22b68d19
Partially-Implements: blueprint generic-resource-pools-ocata
The build_requests.instance column is a serialized
instance object, and the instances.user_data column
is MediumText, so the build_requests.instance column
itself needs to be at least MediumText in size for MySQL.
Change-Id: I7d65df37c02750593037744543ad15e5bc64e913
Closes-Bug: #1635446
Quotas are required to exist in the API database as we need to enforce
quotas across cells.
blueprint cells-quota-api-db
Change-Id: I52fd680eaa4880b06f7f8d4bd1bb74920e73195d
We will store custom resource classes in the new resource_classes table.
These custom resource classes represent non-standardized resource
classes. Followup patches add the plumbing to handle existing
standardized classes currently using the fields.ResourceClass field type
and to perform CRUD operations against a /resource-classes REST API
endpoint.
Change-Id: I60ea0dcb392c1b82fead4b859fc7ed6b32d4bda0
blueprint: custom-resource-classes
When deploying with a very large number of volumes the
block_device_mappings column is not sufficient. The column
needs to be increased to MEDIUMTEXT size to support this use case.
Change-Id: Ia34d06429c1f8f0a8259616bcba0c349c4c9aa33
Closes-Bug: #1621138
This reverts commit 1b5f9f8203.
On IRC I we decided that we agreed no migrations should be placement
specific, we should just use the API table migrations to generate the
schema for both DBs.
There is also a separate debate around the alias for the aggregates
table, but that is not really a reason to revert, its just things in
here that will need rework.
Change-Id: I275945aee9d9be8e35d6ddc05515df39d559457a
If 'connection' is set in the 'placement_database' conf group use
that as the connection URL for the placement database. Otherwise if
it is None, the default, then use the entire api_database conf group
to configure a database connection.
This works by:
* adding a 'placement sync' and 'placement version' command to
nova-manage
* adding placement_migrations that that sync will run
* adding a placement_database config group with the relevant
database configuration settings
* adding a placement_context_manager. If
CONF.placement_database.connection is None this is the same as
the api_context_manager, otherwise it is a new one from its own config
* adjust nova/tests/fixtures to be aware of a 'placement' database
and the placement_context_manager
This version of this change differs from others by not requiring
separate placement commands for migration, instead using existing
tooling, which makes the size of the change a bit smaller and also
addresses problems with the test fixtures needing to be too aware of
what migration code to run. Now it runs the same code.
This functionality is being provided to allow deployers to choose
between establishing a new database now or requiring a migration
later. The default is migration later.
This is a modification of Id93cb93a0f4e8667c8e7848aa8cff1d994c2c364
and I3290e26d0a212911f8ef386418b9fa08c685380b.
Change-Id: Ice03144376c9868c064e4393d531510615fc6fc1
Co-Authored-By: Chris Dent <cdent@anticdent.org>
Partially-Implements: blueprint generic-resource-pools
Make aggregate.create() and destroy() use the API rather than cell database.
Also block aggregate creation until main database empty. This makes
Aggregate.create() fail until the main database has had all of its aggreagtes
migrated. Since we want to avoid any overlap or clashes in integer ids we
need to enforce this.
Note that this includes a change to a notification sample, which encodes
the function and module of a sample exception (which happens to be during
an aggregate operation). Since the notifications are encoding internal
function names, which can and will change over time, this is an expected
change.
blueprint cells-aggregate-api-db
Co-Authored-By: Dan Smith <dansmith@redhat.com>
Change-Id: Ida70e3c05f93d6044ddef4fcbc1af999ac1b1944
Due to the os-extended-volumes API it is necessary to be able to
retrieve block device mapping info for an instance at any time. In order
to do so it needs to be stored with the build request.
This also makes it available during instance deletion where it may be
useful to look up whether delete_on_termination is set on a bdm so that
it can be cleaned up.
Change-Id: Ib774a43e49b7153b3f7b099a59483c62003ee7a8
Partially-Implements: bp add-buildrequest-obj
An Allocation represents a use of some class of resource from a
particular resource provider, by a consumer. They can be created
and destroyed.
It is also possible get a list of all allocations associated with a
particular resource provider.
This also includes: a small correction to functional tests of Inventory
so that its structure (where resource_class is defined) is aligned
with the Allocation tests.
Co-Authored-By: Dan Smith <dansmith@redhat.com>
Partially-Implements: blueprint resource-providers-allocations
Change-Id: Iaec8a8e318adfd8feae6496ad962bc02d13adebd
CellsV2 requires that instance groups be available in the API database.
Create the 'instance_groups', 'instance_group_policy', and
'instance_group_member' tables in the API database.
The instance_id column of instance_group_member has been renamed to
instance_uuid.
Part of blueprint cells-instance-groups-api-db
Change-Id: Id8efd701c9a8e142688ecb276ade41e92ae1b936
DB call method get_all_by_resource_provider_uuid queries all
Inventories and filters them by provider's uuid from ResourceProvider,
but query list is not valid (contains inventories for all providers),
the correct way to load ResourceProvider is to use query join. Replace
joinedload with contains_eager to avoid redundant joins.
As result last updated provider overwrites all provider's inventories
with wrong data.
Closes-Bug: #1572555
Change-Id: I49fb1bf63400280635c202dea8f870d727a91e81
CellsV2 requires that aggregates be available in the API db. Create
the 'aggregates', 'aggregate_metadata' and 'aggregate_hosts' tables in the
API db.
blueprint cells-aggregate-api-db
Change-Id: I19fa3a28181831c5cd7b19cd2a5a2ea0d40e45f8
This just mirrors the DB schema baked in the cell DB up to the API DB.
Related to blueprint generic-resource-pools
Change-Id: I8324a8b9c5ba640e48b6d6e426f7d8073a1b962b
Now that there's a serialized instance object on the BuildRequest object
and in the build_requests table a lot of fields/columns are no longer
necessary. To prepare for removal of the columns from the db at a later
time references to them have been removed. This starts a deprecation
timer so that they can be removed in Ocata at the earliest.
Note that the BuildRequest object was never passed over the wire, and is
just created and immediately destroyed at this point so there is no need
to handle backwards compatibility for these changes.
Change-Id: Ie86ba66d5f73c153b68bef83661d4c8bdb0ae9dc
Now that a serialized Instance object is being stored in the
build_requests table most of the other columns are no longer necessary
and should be deprecated. By first allowing them to be nullable they can
then be removed from the BuildRequest object and then removed in Ocata.
Also remove a UniqueConstraint that is no longer necessary.
Change-Id: I92fa2381c43e5e05d95bbcf9a7bb4306d3b3b856
Partially-implements: bp add-buildrequest-obj
This adds the keypairs table to the API database and the corresponding
model.
Related to cells-keypairs-api-db
Change-Id: I5f6d88fee47dd87de2867d3947d65b04f0b21e8f
New fields keep needing to be added to build_requests in order to
capture all of the relevant instance information necessary. At this
point it's simpler to just embed a serialized Instance object in the db.
A column for that is added, as well as an instance_uuid column for
indexed lookups.
Change-Id: I1628ddd8cb75b4d0a7bbcd2720536c2e0bfc6043
Partially-implements: bp add-buildrequest-obj
Right now InstanceMapping exposes the cell_mappings.id db field which is
just an implementation detail of the db relationship and should not be
exposed in the object. It is also useless as a way to look up the actual
CellMapping through that object interface. And InstanceMapping is only
looked up in order to find a CellMapping.
So the CellMapping should be joined during the db query and loaded on
the InstanceMapping object by default.
Change-Id: Ia8691b76bba310327bfe0995964525409794d1af
Some api database models did not have backrefs setup for their
relationships which affected how queries were done and the result sets
were handled. This diverged from how it's done elsewhere in Nova which
led to many questions on reviews. So backrefs have been added to bring
this in line with the rest of the models.
back_populates is used in one spot which is functionally equivalent to
backref except that it requires the relationship to be defined on both
sides. Since one side was already defined I used back_populate here to
minimize changes.
Change-Id: If653143738c91771e845ceb127e60cead8d40fae
Stephen Finucane