We're going to be extensively reworking these to handle the alembic
switchover in a future change. Ahead of this rework, rework things
somewhat to simplify things and remove unnecessary noise.
Change-Id: Ie9d00e25b7e99104155466060a2b6f8a884a5e0a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Rewrap some code just to make it a little less fugly to read.
Change-Id: If78bbd578bbba73fc85446ad55d34d3addd6c4af
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The two remaining modules, 'api_models' and 'api_migrations', are
moved to the new 'nova.db.api' module.
Change-Id: I138670fe36b07546db5518f78c657197780c5040
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
At this point is nova's lifecycle, we're not going to change our ORM,
making the layer of indirection that is 'nova.db.sqlalchemy' is a
useless one. Start removing it with the 'migration' module.
Change-Id: I5fd6c6f74b7de2b422359cbe72defb07252e6b1e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Nothing but placeholders to be removed here.
Change-Id: I065e3a950e79b0754a2589232ef8a46bb4e83e7f
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Specific changes include:
- Add 'user_id' column to 'instance_mappings' table (062)
- Add index covering 'user_id' and 'project_id' columns to
'instance_mappings' table (062)
Note that we forgot to add placeholder migrations in Rocky, meaning
there is significantly less to clean up here.
Change-Id: I38a6f4d1c3d2a9797ba543694401b7b821a79805
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Specific changes include:
- Add 'disabled' column to 'cell_mappings' table (058)
- Add 'generation' column to 'consumers' table (059)
- Add 'rules' column to 'instance_group_policy' table (060)
- Add 'queued_for_delete' column to 'instance_mappings' table (061)
Note that we need to explicitly disable constraints for 'Boolean' fields
sqlalchemy since these weren't added by sqlachemy-migrate.
Change-Id: Id1c970c3d52c7cc672d00cb0486bfe49d69531b7
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Specific changes include:
- Add 'description' column to 'flavors' table (050)
- Add 'root_provider_id' and 'parent_provider_id' columns to
'resource_providers' table (051)
- Add indexes covering 'root_provider_id' and 'parent_provider_id'
columns of 'resource_providers' table (051)
- Alter type of 'spec' column of 'request_specs' table to MediumText
(052)
Change-Id: Iafccbb52cba0c45b5159984d6560b822d9405d5f
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Specific changes include:
- Add 'resource_classes' table (026)
- Add 'quota_classes', 'quota_usages', 'quotas', 'project_user_quotas',
'reservations' and 'resource_classes' tables (027)
- Alter type of 'instance' column of 'build_requests' table to
'MediumText' (028)
- Add 'placement_aggregates' table (029)
Note that migration 030 included a check to ensure cell mappings were
created when upgrading existing deployments. Since users will now be
required to upgrade to at Ocata before running migrations, this is no
longer necessary. The check was never executed for new deployments,
as noted in the migration itself:
NOTE(danms): We need to be careful here if this is a new
installation, which can't possibly have any mappings. Check
to see if any flavors are defined to determine if we are
upgrading an existing system. If not, then don't obsess over
the lack of mappings.
Change-Id: Ibf82b5e8f46b89f8c7dcdc1925bf48d581cdc3f6
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The NoDBTestCase base class does a lot of env setup that is not needed
for these tests. As we have issues with these tests (see bug 1823251)
this patch tries to decrease the complexity of them to help the
troubleshooting.
Change-Id: I46fafa470b9d378d96c0f364e9b487742bf58cb2
Related-Bug: #1823251
Seeing as these were missed for Rocky, we really ought to get them in
for Stein.
Change-Id: I29401773a0686e7be5d7d1cbb5ec82ffbb16fb4a
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
The instance_mappings table already contains the project_id for an
instance. This adds the corresponding user_id. This also adds an index
on (user_id, project_id) because later patches in this series will be
using these columns for quota counting.
Part of blueprint count-quota-usage-from-placement
Change-Id: Id9eef7a58f66c73cd638c6c3e228447b7ab81e34
Make use of the newer fixtures in oslo_db.sqlalchemy.test_fixtures
which is what was intended to supersede fixtures
in oslo_db.sqlalchemy.test_base.
Change-Id: Icb88c13336fffb499083197ed864b8e4e45dc241
Closes-Bug: #1797102
This column is being added to have information regarding an instance
being active/deleted in the nova-api database as a part of the handling
a down cell spec.
Related to blueprint handling-down-cell
Change-Id: Ic6253acab2ae08025d65add9a5be9ed463457290
This adds the rules column to instance_group_policy table.
The ''Text'' column "rules" which is a dict, it can be
applied to the policy, and represents the rules for
specific policy.
Related to blueprint complex-anti-affinity-policies
Change-Id: I61ffb5ddb2d808bfef4e60088f4524bd98e0474e
In order to prevent races where two different processes can try to write
the allocations for a consumer, we are adding a 'generation' column to
the consumers table. This will be incremented each time a consumer's
allocations change, and allocation requests whose generation doesn't
match the current value will be rejected.
Blueprint: add-consumer-generation
Change-Id: Ia247f0783c5bbd172aafbfd86372fabeeef557c3
This column has been added to the cell_mappings table since it is
required to represent if a cell is enabled or disabled as a part of the
cell-disable spec.
Related to blueprint cell-disable
Change-Id: Ibeddcb9a93c180f74727431b0579308c400436db
request_spec.instance_group.members is a list of instance UUIDs. It
can get so long that it overflows its TEXT column. This patch changes
request_spec.spec to MEDIUMTEXT.
Change-Id: I6bf0fa19b72887803e77b66698587c2108c9372a
Closes-bug: 1738094
Adds initial support for storing the relationship between parent and
child resource providers. Nested resource providers are essential for
expressing certain types of resources -- in particular SR-IOV physical
functions and certain SR-IOV fully-programmable gate arrays. The
resources that these providers expose are of resource class
SRIOV_NET_VF and we will need a way of indicating that the physical
function providing these virtual function resources is tagged with
certain traits (representing vendor_id, product_id or the physical
network the PF is attached to).
The compute host is a resource provider which has an SR-IOV-enabled
physical function (NIC) as a child resource provider. The physical
function has an inventory containing some total amount of SRIOV_NET_VF
resources. These SRIOV_NET_VF resources are allocated to zero or more
consumers (instances) on the compute host.
compute host (parent resource provider)
|
|
SR-IOV PF (child resource provider)
:
/ \
/ \
VF1 VF2 (inventory of child provider)
The resource provider model gets two new fields:
- root_provider_uuid: The "top" or "root" of the tree of nested
providers
- parent_provider_uuid: The immediate parent of the provider, or None
if the provider is a root provider.
The database schema adds two new columns to the resource_providers
table that contain the internal integer IDs that correspond to the
user-facing UUID values:
- root_provider_id
- parent_provider_id
The root_provider_uuid field is non-nullable in the ResourceProvider
object definition, and this code includes an online data migration to
automatically populate the root_provider_id field with the value of the
resource_providers.id field for any resource providers already in the
DB.
The root_provider_id field value is populated automatically when a
provider is created. If the parent provider UUID is set, then the
root_provider_id is set to the root_provider_id value of the parent. If
parent is unset, root_provider_id is set to the value of the id
attribute of the provider being created. The corresponding UUID values
for root and parent provider are fetched in the queries that retrieves
resource provider data using two self-referential joins.
The root_provider_id column allows us to do extremely quick lookups of
an entire tree of providers without needing to perform any recursive
database queries.
Logic in this patch ensures that no resource provider can be deleted if
any of its children have any allocations active on them. We also check
to ensure that when created or updated, a resource provider's parent
provider UUID actually points to an existing provider.
It's important to point out that qualitative trait information is only
associated with a resource provider entity, not the resources that
resource provider has in its inventory. This is the reason why nested
resource providers are necessary. In the case of things like NUMA nodes
or SRIOV physical functions, if a compute host had multiple SRIOV
physical functions, each associated with a different network trait,
there would be no way to differentiate between the SRIOV_NET_VF
resources that those multiple SRIOV physical functions provided if the
containing compute host had a single inventory item containing the
total number of VFs exposed by both PFs.
Change-Id: I2d8df57f77a03cde898d9ec792c5d59b75f61204
blueprint: nested-resource-providers
Co-Authored-By: Moshe Levi <moshele@mellanox.com>
This adds the description attribute to the Flavor object
and the accompanying database schema upgrade.
This is the first field on a flavor that we've allowed
to be updated, so it requires some new plumbing for doing
an update on the flavor record itself during save(). The
updateable fields are whitelisted so we don't leak other
updates in here, like name, flavorid, vcpus, etc.
As part of this change, we also have to be sure to not
persist the description in the embedded instance.flavor
since (1) it's a large field and (2) we are not going
to expose the instance.flavor.description out of the
servers API.
Versioned notifications will be handled in a later change.
Part of blueprint flavor-description
Change-Id: I6db4eb46df0d7ec025b969a46621823957503958
We were a bit naughty in Iecbe0eb5717afb0b13ca90d4868a3ca5f9e8902b in
that we didn't normalize the storage of Keystone project and user
identifiers in the API DB's new consumers table. This means that we
will use a whole lot more storage for what ends up being very
repetitive data.
This patch changes the consumers DB table schema's project_id and
user_id columns from VARCHAR(255) to INT data type. This should result
in significantly faster queries for usage information since 9X the
index records can fit into a single index block in memory (36-byte
UUIDs stored in VARCHAR(255) versus 4-byte integers. The more index
records we can fit into a single page of memory, the faster both scans
and seeks will be.
Let's address this now before anything uses the consumers table.
Change-Id: I1b7357739f2a7e55c55d3acb9bd604731c4a2b32
blueprint: placement-project-user
Originally it was felt that we would need this column to distinguish
between compute nodes and non-compute providers. With the advent of
traits, though, this column is no longer used or needed.
Closes-Bug: #1648197
Change-Id: I614db98727f4737deb6728ee874ab0f68024ebe5
This is the first patch of the series,
this patch adds tags to build_request
table and corresponding db update
script.
Change-Id: I01e1973449a572ecd647ede0a70d274eac1583bd
Part of blueprint support-tag-instance-when-boot
This patch adds DB table `traits` and `resource_provider_traits` into
the api database.
The traits table is used to represent the trait. The resource_provider_traits
table is used to represent the relationship between trait and resource provider.
Part of implementation blueprint shared-resources-pike
Change-Id: I5c34bdd1423beab53cc4af45e016d9a9bba5ffda
Normally we reserve five slots for the API database instead of the default
ten. However, given all the stuff that merged in Ocata related to API-level
services, I'm going to reserve ten here just so we have space if we need
it.
Change-Id: I57c2edcf1fb80e24017cb1b4be00065aa20b342c
Get nova.boot_server, nova.attach_volume, nova.detach_volume and
nova.delete_server working. Please see the following cinder review
for the experiment (see gate-rally-dsvm-py35-cinder-nv):
Id78b136ad15ac77717711ebcbb671c2f1dd3a10c
nova/api/openstack/placement/handlers:
* make sure we convert to bytes before we set the response.body
nova/tests/functional/db/api/test_migrations.py:
* range is an iterator and must be converted to a list
nova/tests/unit/virt/libvirt/test_driver.py:
nova/tests/unit/virt/libvirt/test_fakelibvirt.py:
nova/virt/libvirt/config.py:
nova/virt/libvirt/guest.py:
nova/virt/libvirt/host.py:
* libvirt API expects strings, some of the code paths ended up
being bytes, so convert them to strings before calling libvirt
Finally, eliminated the tests that now started to pass from
tests-functional-py3.txt
Change-Id: Ib721442e9d83a3b9a7fe597f3886430449a9e684
We have code going into Ocata that needs to be sure that cell and
host mappings are in place. Since this was required homework in
Newton, we can land a migration to intentionally fail if this was
not completed.
This is, however, a little difficult to require because a first-time
deployment will be initialized schema-wise with none of these records,
which is also sane. So, we look to see if any flavors are defined as
a sentinel to indicate that this is an upgrade of an existing
deployment instead of a first-time event. Not perfect, but since this
is really just a helper for the user, it seems like a reasonable
risk.
Depends-On: If1af9c478e8ea2420f2523a9bb8b70fafddc86b7
Change-Id: I72fb724dc13e1a5f4e97c58915b538ba761c582d
In order to facilitate a future extraction of the placement service
we want to record the association between a resource provider and an
arbitrary aggregate uuid in its own table.
A PlacementAggregate model is joined from ResourceProvider via
ResourceProviderAggregate. Note that this structure is used so we can
join on ids instead of strings (the uuids). A direct mapping between
ResourceProvider uuid and Aggregate uuid was mooted earlier in the year
but was determined to be suboptimal.
The name 'placement_aggregates' is used as the least problematic of
several choices after discussion amongst several parties.
The data will be used by the forthcoming get_ and set_aggregates
methods on the ResourceProvider object.
Change-Id: Id0355cb022f68e962af306ff04cf724d22b68d19
Partially-Implements: blueprint generic-resource-pools-ocata
The build_requests.instance column is a serialized
instance object, and the instances.user_data column
is MediumText, so the build_requests.instance column
itself needs to be at least MediumText in size for MySQL.
Change-Id: I7d65df37c02750593037744543ad15e5bc64e913
Closes-Bug: #1635446
Quotas are required to exist in the API database as we need to enforce
quotas across cells.
blueprint cells-quota-api-db
Change-Id: I52fd680eaa4880b06f7f8d4bd1bb74920e73195d
We will store custom resource classes in the new resource_classes table.
These custom resource classes represent non-standardized resource
classes. Followup patches add the plumbing to handle existing
standardized classes currently using the fields.ResourceClass field type
and to perform CRUD operations against a /resource-classes REST API
endpoint.
Change-Id: I60ea0dcb392c1b82fead4b859fc7ed6b32d4bda0
blueprint: custom-resource-classes
Much like in the unit tests, because these inherit from oslo_db they
don't correctly setup logging capture consistent with the rest of the
nova test suite. This adds the right fixture initialization before we
do setup to prevent corruption of the output stream.
Change-Id: I1243db75dd837b92c3b88df2064d4d9f74ed6058
When deploying with a very large number of volumes the
block_device_mappings column is not sufficient. The column
needs to be increased to MEDIUMTEXT size to support this use case.
Change-Id: Ia34d06429c1f8f0a8259616bcba0c349c4c9aa33
Closes-Bug: #1621138
Add the ability to sync the cell0 database using nova-manage.
The `db sync` command will be used to sync the cell0 database.
This ensures that operators will only have two db sync commands to
perform in the single cell case.
blueprint cells-cell0
Change-Id: I21ae13a6c029e8ac89484faa212434911160fd51
Stephen Finucane