This adds the compute_id field to the Instance object and adds
checks in save() and create() to make sure we no longer update node
without also updating compute_id.
Related to blueprint compute-object-ids
Change-Id: I0740a2e4e09a526da8565a18e6761b4dbdc4ec0b
Introduce a new 'nova.db.api.api' module to hold API database-specific
helpers, plus a generic 'nova.db.utils' module to hold code suitable for
both main and API databases. This highlights a level of complexity
around connection management that is present for the main database but
not for the API database. This is because we need to handle the
complexity of cells for the former but not the latter.
Change-Id: Ia5304c552ce552ae3c5223a2bfb3a9cd543ec57c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The two remaining modules, 'api_models' and 'api_migrations', are
moved to the new 'nova.db.api' module.
Change-Id: I138670fe36b07546db5518f78c657197780c5040
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Merge these, removing an unnecessary layer of abstraction, and place
them in the new 'nova.db.main' directory. The resulting change is huge,
but it's mainly the result of 's/sqlalchemy import api/main import api/'
and 's/nova.db.api/nova.db.main.api/' with some necessary cleanup. We
also need to rework how we do the blocking of API calls since we no
longer have a 'DBAPI' object that we can monkey patch as we were doing
before. This is now done via a global variable that is set by the 'main'
function of 'nova.cmd.compute'.
The main impact of this change is that it's no longer possible to set
'[database] use_db_reconnect' and have all APIs automatically wrapped in
a DB retry. Seeing as this behavior is experimental, isn't applied to
any of the API DB methods (which don't use oslo.db's 'DBAPI' helper),
and is used explicitly in what would appear to be the critical cases
(via the explicit 'oslo_db.api.wrap_db_retry' decorator), this doesn't
seem like a huge loss.
Change-Id: Iad2e4da4546b80a016e477577d23accb2606a6e4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab in Queens,
which was backported through to Newton, introduced a regression
when listing deleted servers with a marker because it assumes
that if BuildRequestList.get_by_filters does not raise
MarkerNotFound that the marker was found among the build requests
and does not account for that get_by_filters method short-circuiting
if filtering servers with deleted/cleaned/limit=0. The API code
then nulls out the marker which means you'll continue to get the
marker instance back in the results even though you shouldn't,
and that can cause an infinite loop in some client-side tooling like
nova's CLI:
nova list --deleted --limit -1
This fixes the bug by raising MarkerNotFound from
BuildRequestList.get_by_filters if we have a marker but are
short-circuiting and returning early from the method based on
limit or filters.
Change-Id: Ic2b19c2aa06b3059ab0344b6ac56ffd62b3f755d
Closes-Bug: #1849409
During a cross-cell move operation, an instance can
exist in multiple cell databases but can be mapped to
only one cell in the instances_mappings table in the API DB.
When listing servers out of the API, we need a way to
filter out hidden instances to make sure we return the
appropriate record. In other words, the cell with the
hidden instance should not be the one in the instance
mapping and should not be returned when listing instances.
Note that the API already filters out duplicates (see
_get_unique_filter_method in API.get_all) but this gives
us a way to control which is shown rather than get lucky.
The DB API method to list instances will filter out hidden
instances by default.
Quotas
======
Counting quota for instances, cores and ram has to be updated
to filter out hidden instances as well so we don't double
count usage for a hidden instance since we should count it
for the non-hidden copy of the instance in another cell DB.
Similarly, we filter hidden instances when counting server
group members (this required the change in the DB API
instance_get_all_by_filters_sort method to make that filter
work).
Functional tests are added for both quota scenarios.
REST API Filters and Sorting
============================
Note that this change does not add the ability to filter or
sort instances by the new hidden field in the REST API since
it's only used internally. There is no explicit blacklist
for this, but trying to sort on hidden will result in a 400
error because the field is not in VALID_SORT_KEYS. Filtering
on hidden will not result in an error because of the
additionalProperties=True in query_params_v21 but it will not
be passed through to the DB API either.
Part of blueprint cross-cell-resize
Change-Id: Iaffb27bd8c562ba120047c04bb62619c0864f594
This migration was added in Ocata and backported to Newton in change
I8a05ee01ec7f6a6f88b896f78414fb5487e0071e to deal with Mitaka-era
build_requests records that would not have an instance_uuid value
and thus raise a ValueError in BuildRequest._from_db_object (because
BuildRequest.instance_uuid is not nullable).
This is essentially a revert of that change now since operators
have had long enough to run the migration. If anyone were to skip
level upgrade from Mitaka to Train (which we don't support, we require
you to roll through), and hit an issue with this they could simply
execute this on their nova_api DB:
DELETE FROM build_requests WHERE instance_uuid IS NULL;
Change-Id: Ie9593657544b7aef1fd7a5c8f01e30e09e3fcce6
The BuildRequest object code assumed that IncompatibleObjectVersion
exception has a objver field that contains the object version. This
assumption is not true. The unit test made another mistake serializing
the function object obj_to_primitive instead of serializing the result
of the call of obj_to_primitive. This caused a false positive test
covering the error in the implementation as well.
Closes-Bug: #1812177
Change-Id: I1ef4a23aa2bf5cb46b481045f3d968f62f74606d
When listing "real" (already in cell) instances e.g. from
nova_cell1.instances table, a marker option means "start after the
instance with marker".
For VMs:
| uuid | name |
| 1 | vm1 |
| 2 | vm2 |
"openstack server list --marker 1" returns vm2 only.
But for VMs from nova_api.build_requests table it's different.
For VMs:
| uuid | name |
| 1 | vm1 |
| 2 | vm2 |
"openstack server list --marker 1" returns both vm1 and vm2.
This patch excludes instance with marker from listing for
instances from build_requests table.
Closes-Bug: #1808286
Change-Id: I5165b69f956fbf1904112a742698b2739f747e72
For some reason, probably because build requests are meant to be short lived
and we don't get a lot of bugs about paging misbehavior, when paging instances
with a marker, we didn't raise MarkerNotFound if we didn't find the marker in
the list of build requests. Doing so would match what we do when paging over
cells and listing instances using a marker. Once we find the marker, be that
in build_requests, or one of the cells, we need to set the marker to None to
stop looking for it elsewhere if we have more space to fill our limit.
For example, see change I8a957bebfcecd6ac712103c346e028d80f1ecd7c.
This patch fixes the issue by raising MarkerNotFound from BuildRequestList
get_by_filters if there is a marker and we didn't find a build request for
it. The compute API get_all() method handles that as normal and continues
looking for the marker in one of the cells.
Change-Id: I1aa3ca6cc70cef65d24dec1e7db9491c9b73f7ab
Closes-Bug: #1737856
This is the 4th patch of the series,
this patch adds a new microversion
in API to support adding tags when
booting instances.
Implemetes: blueprint support-tag-instance-when-boot
Change-Id: Ifcaaf285c8f98a1d0e8bbbc87b2f57fbce057346
The i18n team has decided not to translate the logs because it
seems like it not very useful; operators prefer to have them in
English so that they can search for those strings on the internet.
Partially fix on nova/keymgr, nova/network, nova/notifications,
nova/objects and nova/pci other paths will be fixed on next commits
Change-Id: Ie3a83fef0dc689b9d37ac43e047ce5d48f567adc
This is the second patch of the series,
this patch adds tags to build_request
object.
Change-Id: I1121eaf24856dcb74f07c224531a821b3cdc46fc
Part of blueprint support-tag-instance-when-boot
1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
The patch list:
1. cells.
2. compute api.
3. image.
4. network.
5. objects.
6. scheduler.
7. virt.
8. other resources.
Partial-Implements: blueprint replace-iteritems-with-items
Change-Id: Ic6e469eb80ee1774de1374bb36f38b5134b6b311
This exhibits the failure reported in bug 1633734 when upgrading
from mitaka to newton with some bad build request records that
weren't cleaned up, and were created before API DB migration
013_build_request_extended_attrs when we didn't have the
instance_uuid or instance records in the database.
After 013_build_request_extended_attrs and the object change to
BuildRequest in a5d3b57c3d if we try
loading up a 'dirty' build request DB record without the
instance_uuid it fails with a ValueError, as shown in the functional
test in this change.
This also provides an online data migration (which will be backported
to Newton for upgrades from Mitaka) that will query the API DB for
build requests where instance_uuid=NULL and delete them.
Change-Id: I8a05ee01ec7f6a6f88b896f78414fb5487e0071e
Related-Bug: #1633734
We should already have the rehydrated instance having the deleted field being
set. That field is also a BooleanField type, so we should never try to cast it
from an integer (even if we support it). Instead, we need to make sure that if
the field is not set for any reason (mostly because an Instance wasn't persisted
before we persisted the BuildRequest object that was nesting it), then we would
default the deleted field to False.
Change-Id: I828191f750a5978fed65a86e797388d11042b4a0
Related-Bug: #1644513
This patch is preparation step to move the instance creation to
the conductor. The goal is to move out the instance.create call from
the _provision_instances method.
The _provision_instances method return value is changed
from list of instances to the tuple containing a list of build_requests
and RequestSpec object. We also return the instance_mapping list so
that we can properly clean up residue if we fail to create in the caller.
Co-Authored-By: Dan Smith <dansmith@redhat.com>
Implements: bp cells-scheduling-interaction
Change-Id: I60abcd4f27dc877c4e420071be77c9fdb697ad99
String interpolation should be delayed to be handled
by the logging code, rather than being done
at the point of the logging call.
So add the following hacking rule for it.
- [N354] String interpolation should be delayed at logging calls.
See the oslo i18n guideline.
* http://docs.openstack.org/developer/oslo.i18n/guidelines.html
Change-Id: Ief6d3ee3539c0857098fffdb7acfeec3e0fed6eb
Closes-Bug: #1596829
There are some cases where the instance is still waiting to be scheduled and
consequently there is still a BuildRequest object where we would also like to
update the instance attributes, like the display name.
In that case, we need to update the nested instance field from the BuildRequest
instead of trying to update the instance object itself, which could be located
in a child cell.
Change-Id: I97cd6dbae10ca1a29f646b0b25bc08edd699edca
Closes-Bug: #1620248
The _from_db_object method in the BuildRequest object was written when
there were no objects in any of its fields. Now that there are Instance
and BlockDeviceMapping fields the recursive=True argument is needed when
calling obj_reset_changes.
Change-Id: I5ff919d93c987786532b57e102bb6e5e54eb0919
The BuildRequestList object can be used to fetch a list of pre-scheduled
instances which have not been written to an instance table yet. The
get_by_filters method emulates the behavior of Instance.get_by_filters
and returns a BuildRequestList object with the contained BuildRequest
objects sorted and filtered on the Instance object contained within.
Change-Id: If3638caba2c12d4c087a854bd8cbf4f79ef9652d
Partially-implements: bp add-buildrequest-obj
Partially-implements: bp cells-scheduling-interaction
This is preliminary work towards moving instance creation until after
scheduling. In order to do that we still need the ability to honor an
instance show request. This does a lookup to determine if an instance is
in a cell and if not it synthesizes an instance from the BuildRequest
object that has been stored.
If the instance is not mapped then the deployment has not migrated to
their first cell yet and the instance lookup proceeds as usual.
Until instance creation is moved to after scheduling this is mostly a
noop.
Honoring the instance list command will come later.
Change-Id: Ib984c30543acb3ca9cb95fb53d44d9ded0f5a5c8
Partially-implements: bp add-buildrequest-obj
Partially-implements: bp cells-scheduling-interaction
Due to the os-extended-volumes API it is necessary to be able to
retrieve block device mapping info for an instance at any time. In order
to do so it needs to be stored with the build request.
This also makes it available during instance deletion where it may be
useful to look up whether delete_on_termination is set on a bdm so that
it can be cleaned up.
Change-Id: Ib774a43e49b7153b3f7b099a59483c62003ee7a8
Partially-Implements: bp add-buildrequest-obj
After an instance has been scheduled to a cell and mapped to the
instance table there the BuildRequest for it will be destroyed. An
instance delete that occurs before that remapping has happened will also
destroy the BuildRequest.
In order to know that a delete has happened, and stop the build, the
destroy method should raise an exception if it is already deleted.
Change-Id: Ib86ac14e2bd933bb9e76ea86b2a1ec4f40d204a3
Now that there's a serialized instance object on the BuildRequest object
and in the build_requests table a lot of fields/columns are no longer
necessary. To prepare for removal of the columns from the db at a later
time references to them have been removed. This starts a deprecation
timer so that they can be removed in Ocata at the earliest.
Note that the BuildRequest object was never passed over the wire, and is
just created and immediately destroyed at this point so there is no need
to handle backwards compatibility for these changes.
Change-Id: Ie86ba66d5f73c153b68bef83661d4c8bdb0ae9dc
It turns out that BuildRequest was going to need to copy quite a bit of
the Instance fields and was starting to look very similar. Instead of
continuing to add fields to BuildRequest let's just store a serialized
Instance in there so everything is available.
Note that the BuildRequest object version is not being bumped because
this object is not yet passed over the wire anywhere or looked up
anywhere, so there are no backwards compatibility concerns.
Partially-implements: bp add-buildrequest-obj
Change-Id: I63023915b5b7907949e55d6fd2693fcbc7777511
Some api database models did not have backrefs setup for their
relationships which affected how queries were done and the result sets
were handled. This diverged from how it's done elsewhere in Nova which
led to many questions on reviews. So backrefs have been added to bring
this in line with the rest of the models.
back_populates is used in one spot which is functionally equivalent to
backref except that it requires the relationship to be defined on both
sides. Since one side was already defined I used back_populate here to
minimize changes.
Change-Id: If653143738c91771e845ceb127e60cead8d40fae
The BuildRequest object represents a boot request before an instance has
been created and written to the database. It will be required to respond
to API requests to list/show an instance in cellsv2 where the instance
is not written to the database immediately. An upcoming change will
modify the boot process to defer instance creation until after
scheduling has picked a cell/host and having this BuildRequest object is
a prerequisite for that work.
Change-Id: Ie9307388c2e068229177bcc4690cb834028c1481
Partially-implements: bp cells-scheduling-interaction
Dan Smith