When getting an instance using the compute.API we call
scatter_gather_single_cell() to be able to capture details when we fail
to retrieve a result from a cell such as timeouts and exceptions.
Currently however, we aren't logging the content of an exception if
scatter_gather_single_cell() returns an exception as the result. The
scatter gather method itself logs exceptions that are not of type
NovaException as these represent definite unexpected errors such as
database errors but NovaException handling are left for the caller to
decide whether they want to log it or re-raise it and so on.
It can be difficult to debug a situation where a cell is returning a
NovaException result so this adds logging of the exception content in
the compute API when we encounter an unexpected NovaException.
The existing log message has been updated to more accurately reflect
what has happened (did not respond vs exception). The assignment of the
exception object in scatter gather has also been updated to not
unnecessarily construct a new exception object because it (a) wasn't
necessary and (b) made asserting the LOG.exception() call argument in
the unit test difficult.
Related-Bug: #1970087
Change-Id: Iae1c61c72be5b6017b934293e3dc079a24eeb0e7
Merge these, removing an unnecessary layer of abstraction, and place
them in the new 'nova.db.main' directory. The resulting change is huge,
but it's mainly the result of 's/sqlalchemy import api/main import api/'
and 's/nova.db.api/nova.db.main.api/' with some necessary cleanup. We
also need to rework how we do the blocking of API calls since we no
longer have a 'DBAPI' object that we can monkey patch as we were doing
before. This is now done via a global variable that is set by the 'main'
function of 'nova.cmd.compute'.
The main impact of this change is that it's no longer possible to set
'[database] use_db_reconnect' and have all APIs automatically wrapped in
a DB retry. Seeing as this behavior is experimental, isn't applied to
any of the API DB methods (which don't use oslo.db's 'DBAPI' helper),
and is used explicitly in what would appear to be the critical cases
(via the explicit 'oslo_db.api.wrap_db_retry' decorator), this doesn't
seem like a huge loss.
Change-Id: Iad2e4da4546b80a016e477577d23accb2606a6e4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We have had a gate bug for a long time where occasionally the scheduler
service gets into a state where many requests fail in it with
CellTimeout errors. Example:
Timed out waiting for response from cell <cell uuid>
Through the use of much DNM patch debug logging in oslo.db, it was
revealed that service child processes (workers) were sometimes starting
off with already locked internal oslo.db locks. This is a known issue
in python [1] where if a parent process forks a child process while a
lock is held, the child will inherit the held lock which can never be
acquired.
The python issue is not considered a bug and the recommended way to
handle it is by making use of the os.register_at_fork() in the oslo.db
to reinitialize its lock. The method is new in python 3.7, so as long
as we still support python 3.6, we must handle the situation outside of
oslo.db.
We can do this by clearing the cell cache that holds oslo.db database
transaction context manager objects during service start(). This way,
we get fresh oslo.db locks that are in an unlocked state when a child
process begins.
We can also take this opportunity to resolve part of a TODO to clear
the same cell cache during service reset() (SIGHUP) since it is another
case where we intended to clear it. The rest of the TODO related to
periodic clearing of the cache is removed after discussion on the
review, as such clearing would be unsynchronized among multiple
services and for periods of time each service might have a different
view of cached cells than another.
Closes-Bug: #1844929
[1] https://bugs.python.org/issue6721
Change-Id: Id233f673a57461cc312e304873a41442d732c051
Framework for communication with the Cyborg API.
- Standard keystoneauth1 config options for setting up authentication in
the [cyborg] section of nova*.conf.
- A new nova.accelerator.cyborg module containing a get_client method to
return a client containing a keystoneauth1 adapter pointing
to the Cyborg service with user- and service- based authentication.
- Requirements updates to pull in the os-service-types release
containing the 'accelerator' service type.
Change-Id: Iee0766269d61948ad701911e8b0e5e24d3d6eb04
Blueprint: nova-cyborg-interaction
This reverts commit 0436a95f37.
This was meant to get us more debug details when hitting the
failure but the results are not helpful [1] so revert this
and the fix for the resulting regression [2].
[1] http://paste.openstack.org/show/782116/
[2] I7f9edc9a4b4930f4dce98df271888fa8082a1701
Change-Id: Iab8029f081a654278ea7dbbec79a766aea6764ae
Related-Bug: #1844929
Found this error in heat gate during running grenade test job.
Should not asking to provide cell_uuid if queue.get is timeout.
Closes-Bug: #1847131
Change-Id: I7f9edc9a4b4930f4dce98df271888fa8082a1701
When a call to a cell in scatter_gather_cells times out
we log a warning and set the did_not_respond_sentinel for
that cell but it would be useful if we logged the traceback
with the warning for debugging where the call is happening.
Change-Id: I8f4069474a3955eea6c967d3090f2960e739224c
Related-Bug: #1844929
The instance_lock_checked kwarg was added to RequestContext in [1], no
longer used in Nova since [2], and deprecated with a warning in [3].
This commit removes all handling of this kwarg, which will cause using
it to blow up with a TypeError in the super().__init__.
[1] I1127e31d86a061a93a64ee1eb4a4d900d8bf49b5
[2] Ic18017a16c5bffee85a43db65ff17283599a27ba
[3] Ib0b23a47c2e833073108af6700b16f8026631a83
Change-Id: Ie5f5a0ae018ce601c588a466399515159d8f58c0
Move default target from context.can() into policy.authorize() so
it is easier to unit test the context behaviour.
This was not done originally due to this meaning placement avoided the
strange default target, but that is no longer required.
Change-Id: I23c433dcd459e7f930ac2eb8a3583c857836cae2
This reverts commit 3cc3cc453d.
We had determined that this commit had not actually fixed any bug or
issue with:
TypeError: Can't upgrade a READER transaction to a WRITER
mid-transaction
and the real fix was change Id24dea7465bafc1f6f58c4a121c4ffb35b7a634a.
This reverts the unnecessary commit in order to simplify the code and
it's possible that targeting the context before spawning a thread will
result in decreased cell cache lock contention once the threads are
spawned.
Related-Bug: #1722404
Conflicts:
nova/context.py
NOTE(melwitt): The conflict is due to change
I861b223ee46b0f0a31f646a4b45f8a02410253cf which is new since the
original commit had landed.
Change-Id: I3890d14d8679b96cfebcd735f3b03ec808730b5c
This brings in a couple of new checks which must be addressed, many of
which involve a rather large amount of changes, so these are ignored for
now. A series of follow-up changes will resolved these.
'pycodestyle' is added as a dependency rather than it being pulled in
transitively. This is necessary since we're using it in tests.
Change-Id: I35c654bd39f343417e0a1124263ff31dcd0b05c9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
The target passed to Enforcer.authorize should be a dict,
similar to the target dict to the RequestContext.can method.
However, we were passing an instance of _DeprecatedPolicyValues
because that is ultimately what comes out of
RequestContext.to_policy_values(). As of change
I4642c57990b145c0e691140970574412682e66a5 in oslo.policy, that
incorrect type for the target parameter results in an error in
the debug logs for the policy check:
cannot format data, exception: Expected a dictionary, got
<class 'oslo_context.context._DeprecatedPolicyValues'> instead.
This resolves the issue by using the same default target dict
that RequestContext.can uses if a target is not supplied.
Note that we get here from NovaKeystoneContext via API middleware
before any request handler is invoked in the wsgi stack, so there
is no context from the request as to what to pass for the target
besides the user_id/project_id.
Change-Id: I4442a7b95d15233f76f7795d45b18ac440ddb831
Closes-Bug: #1807747
With change I861b223ee46b0f0a31f646a4b45f8a02410253cf
the caller of the various scatter_gatter functions can
get any exception result and decide what kind of logging
is necessary, so this change removes the logging from
the scatter_gather_cells method **for NovaExceptions**.
This is needed because of change
Iaea1cb4ed93bb98f451de4f993106d7891ca3682 where
showing a server which is deleted, which results in an
InstanceNotFound exception, gets logged as an exception
in the nova-api logs and shouldn't be.
Non-NovaExceptions will still get logged as before since
those are likely due to unexpected errors, like DBErrors.
Change-Id: I930ee897c46385c3c9719b393753c4484caf909f
Closes-Bug: #1804325
Scatter-gather utility returns a raised_exception_sentinel for
all kinds of exceptions that are caught and often times there
maybe situations where we may have to handle the different types
of exceptions differently. To facilitate that, it might be more
useful to return the Exception object itself instead of the dummy
raised_exception_sentinel so that based on the result's exception
type we can handle them differently.
Related to blueprint handling-down-cell
Change-Id: I861b223ee46b0f0a31f646a4b45f8a02410253cf
This patch adds the method to query a single cell using
the scatter-gather utility. Although we do have the
target_cell function to target a specific cell, in situations
where we need to handle a down cell (like for nova show), it
is advantageous to have this wrapper which in turn only calls
the scatter-gather-cells utility.
Change-Id: I31f2ab499aed795812a99b116cec3663760297e3
This has been something we've needed for a while in order to do
some optimization in places like multi_cell_list where we need to
know which cell we are looking at during a scatter_gather operation.
Change-Id: Ib47d4ed1072c9ee6b2d9d084e678c80a05604963
This unifies the various places that hard-code a 60-second timeout
for cells scatter/gather operations use a constant instead.
Change-Id: I77d719ca4588bf273271e4f138f53d09c7d3e330
nova/db/__init__.py was importing * from nova.db.api. This meant that
any time any code anywhere within the nova.db package was imported
then nova.db.api was too, leading to a cascade of imports that may
not have been desired. Also, in general, code in __init__.py is a pain.
Therefore, this change adjusts code that so that either:
* nova.db.api is used directly
* nova.db.api is imported as 'db'
In either case, the functionality remains the same.
The primary goal of this change was to make it possible to import the
model files without having to import the db api. Moving the model files
to a different place in the directory hierarchy was considered, but
given that "code in __init__.py is a pain" this mode was chosen.
This looks like a very large change, but it is essentially adjusting
package names, many in mocks.
Change-Id: Ic1fd7c87ceda05eeb96735da2a415ef37060bb1a
This was initially added in change I1127e31d86a061a93a64ee1eb4a4d900d8bf49b5
but is no longer used anywhere in nova since change
Ic18017a16c5bffee85a43db65ff17283599a27ba.
Anyway, this change still allows it to be passed as a kwarg for now but will
emit a UserWarning if it is, and then we can make it a hard failure starting
in the S release.
Change-Id: Ib0b23a47c2e833073108af6700b16f8026631a83
When booting instances, nova might create neutron resources. For that,
the network service endpoint needs to be available. Otherwise we run
into:
EndpointNotFound: ['internal', 'public'] endpoint for network service \
not found
Change-Id: Iaed84826b76ab976ffdd1c93106b7bae700a64a9
Closes-Bug: #1752289
Resolve the following deprecation warnings:
DeprecationWarning: Using the 'user' argument is deprecated in
version '2.18' and will be removed in version '3.0', please use the
'user_id' argument instead
DeprecationWarning: Using the 'tenant' argument is deprecated in
version '2.18' and will be removed in version '3.0', please use the
'project_id' argument instead
These were polluting the 'py35' logs in particular.
Change-Id: Id02851dcd86b333b85039e76f1e45e3e49eae908
Change the Adapter loading for glance to use the auth from the user
context instead of exposing and requiring it in the conf. With this
change, it is possible to leave the [glance] conf section empty and
still be able to discover the image API endpoint from the service
catalog.
Note that, when we do this, we often end up with the user auth being a
_ContextAuthPlugin, which doesn't conform to the characteristics of
keystoneauth1.identity.base.BaseIdentityPlugin as augmented in
keystoneauth1 3.1.0. This requires a series of workarounds until bug
1709118 is fixed. These, along with workarounds for bugs 1707993 and
1707995, are subsumed with this change set in a (hopefully temporary)
helper method nova.utils.get_endpoint.
This lays the foundation for other services that should use user
context for authentication - those via which Nova is acting on behalf
of the user, i.e. cinder, keystone, and (sometimes) neutron[1].
(Services such as placement and ironic (and sometimes neutron) should
continue to use admin auth context loaded from the conf.)
[1] bb4faf40df/nova/network/neutronv2/api.py (L149-L160)
Co-Authored-By: Eric Fried <efried@us.ibm.com>
Partial-Implements: bp use-ksa-adapter-for-endpoints
Change-Id: I4e755b9c66ec8bc3af0393e81cffd91c56064717
We are only doing a copy.copy() on context when we do target_cell(),
which may mean we're sharing more across threads than we intend to.
This makes that a full regeneration of the context (like we would
do over RPC). This will necessarily dump the targeting, if already
set, and any per-thread database context.
Closes-Bug: #1722404
Change-Id: Id24dea7465bafc1f6f58c4a121c4ffb35b7a634a
In the set_target_cell method, we synchronize access to the cached cell
database transaction context manager objects to prevent more than one
thread from using a cell's transaction context manager at the same
time.
In scatter_gather_cells, we're calling target_cell in such a way that
the lock is acquired and released BEFORE the green thread actually
accesses the database in the spawned function. So multiple threads can
access a cell's database transaction context manager and it's possible
for a database transaction to fail with the error:
TypeError: Can't upgrade a READER transaction to a WRITER
mid-transaction
because the in-scope transaction context might be in the middle of a
read when a concurrent green thread tries to do a write.
Closes-Bug: #1722404
Change-Id: I07dd4d5aebdc82e343ec2035dc94c744e4754c96
We've removed support for cinder v1 [1] and v2 [2]. The service catalog
entries for 'volume' and 'volumev2' are therefore no longer needed in
RequestContext.
Also, 'block-storage' is the official service type for cinder [3], so we
want to start supporting service catalogs with an entry for it.
[1] https://review.openstack.org/409904
[2] https://review.openstack.org/501874
[3] 671b42dda7/service-types.yaml (L68)
Change-Id: I8360b2a05e982285ad852aa475f1e1cd953c2c8b
In the last few cycles oslo.context has picked up a standard way of
representing most of the information available from a token context.
There's no more need for nova to manage these properties. Remove
properties that shadow the base oslo.context properties and helpers that
we don't need.
Change-Id: I1b11e405232b1acee053cb3bd30c18202d3b7c8f
This adds utilities: scatter_gather_all_cells(),
scatter_gather_skip_cell0(), and scatter_gather_cells() for querying
cells in parallel using eventlet green threads.
Change-Id: I289932176e8029b0f9a76dbfb963f8ac218fdc06
The nova.context should be a super set of oslo.context in all cases,
especially after we've deserialized on the other side of the RPC
bus. Our current code actually has a completely decoupled from_dict,
which means that as fields are added to oslo.context we don't pick up
any of those, and end up with a potentially broken and fragmented
context on the workers.
This fixes that by using the parent constructor for all the fields we
can, and only explicitly load in a few fields that we also need.
It also simplifies the testing so that we're just testing that our
extra fields end up in the context, and not exact matching everything
in the context, as oslo.context may add important things over time.
Change-Id: Ie683adb36d5e2a736ddbf714524c9c18f3c0d69c
This makes target_cell() do the thing it was originally intended to do,
which is yield a new context that is targeted at the proper cell. With this,
any objects you create inside the target context manager (assuming you use
the yielded context) will remain targeted after the thing returns.
Related to blueprint cells-aware-api
Change-Id: Iccdf6b80f5fc8adcc8a89ce6ece3f37b6cbcaee2
Recently in the gate we have seen a trace on some work-in-progress
patches:
OperationalError: (pymysql.err.OperationalError)
(1040, u'Too many connections')
and at least one operator has reported that the number of database
connections increased significantly going from Mitaka to Newton.
It was suspected that the increase was caused by creating new oslo.db
transaction context managers on-the-fly when switching database
connections for cells. Comparing the dstat --tcp output of runs of the
gate-tempest-dsvm-neutron-full-ubuntu-xenial job with and without
caching of the database connections showed a difference of 445 active
TCP connections and 1495 active TCP connections, respectively [1].
This adds caching of the oslo.db transaction context managers and the
oslo.messaging transports to avoid creating a large number of objects
that are not being garbage-collected as expected.
Closes-Bug: #1691545
[1] https://docs.google.com/spreadsheets/d/1DIfFfX3kaA_SRoCM-aO7BN4IBEShChXLztOBFeKryt4/edit?usp=sharing
Change-Id: I17e0eb836dd87aac5859f506e7d771d42753d31a
When deleting an instance we look it up in the _get_instance
method and if it's in cell0 then the context is permanently
targeted to that cell via the set_target_cell method.
When we delete the instance in _delete we need to temporarily
untarget the context when we decrement the quota usage otherwise
the quota usage gets decremented in the cell0 database rather than
the cell database. Once the instance is deleted then we
re-apply the target cell on the context.
Change-Id: I7de87dce216835729283bca69f0eff59a679b624
Closes-Bug: #1670627
Previously we had aimed to make things like compute RPC automatically
look up the InstanceMapping or HostMapping for the call being performed
to target appropriately. However, we cannot do that within the cell,
and even trying incurs some overhead. For now, just deprecate the
by_instance() and by_host() methods and honor what is in the context
(if set) and otherwise fall back to the default client. Make the context
target routines create and store the RPC transport and remove the caching
logic from the ClientRouter since we're removing its ability to do that.
Related to blueprint cells-aware-api
Change-Id: I10f374adca672576058c4dbab708c040d166df47
This makes the servers API look up the CellMapping for the instance
in question and target the context to the appropriate cell for the
subsequent call into compute/api.
Note that just as with get() and get_all(), we need to avoid going
direct to the cell database and thus short-circuiting the replication
in the case of cellsv1.
Related to blueprint cells-aware-api
Change-Id: If63a4fa7349890a6e7ac0fa3fe12917dc53334d0
Nova creates a limited service catalog using just the endpoints
required for Nova. This needs to include the new Cinder v3.
Change-Id: I06f9f2885bb1748897332cb15d4aafe4a99aa9dc
Closes-Bug: #1657496
this function was used to put into db layer to ensure
the request has admin context or a unauthorized exception
will be raised, this is not needed as we already move
all this kind of check into api layer and policy functions.
Change-Id: Ia411e6715e4c584952a275a001f1791ca1352d97
The context.get_admin_context is used in places that it's not necessary,
and that's likely because there's no helper method to retrieve a non
admin context. This adds such a helper method and adds a note to
get_admin_context warning that it's not usually the right choice.
Change-Id: I2e6a2efa4bcdf3f8688897972a6cf8a5af3f90d6
The base oslo.context defines to_policy_values with all the information
that it expects a service to require to enforce policy. Use that instead
of throwing everything in to_dict at policy enforcement.
Change-Id: I0a42b4425e9dd1bd062c48792c4d116dd370afe3
Closes-Bug: #1602081
context.can is used to verify that the provided action is valid.
This patch changes the argument name from 'rule' to 'action' to
avoid misunderstanding.
Change-Id: I315df6d1d29966d6554b48335a99bd9f826a8c6b
This moves the cell message queue switching code from the
RequestContext to the RPC layer where it's ultimately used.
Originally, it was thought that a separate DB query for
a CellMapping would occur per compute RPC API call in the
API cell and the context manager would be invoked to inject
the cell message queue transport to use for the RPC call.
Since compute RPC calls are based on the CellMapping of an
instance or a host, we could instead have generic functions
that take an instance or host and look up InstanceMapping or
HostMapping to get the CellMapping and return the
corresponding RPC client.
The RPC client objects are cached by CellMapping uuid and
expired clients are removed using a periodic task.
Co-Authored-By: Brian Elliott <bdelliott@gmail.com>
Depends-On: I6f211e9102f79418f9f94a15784f91c4150ab8a7
Change-Id: I96849888087f4b09433cb683a9eb4719d1c35c4c
This patch adds functionality to the scheduler "report client" to ensure
that the client calls the placement API to create a resource provider
record for the local compute host managed by the Nova resource tracker.
The report client keeps a cache of resource provider objects, keyed by
resource provider UUID and constructed from the results of placement
REST API calls to get information about a resource provider. If a
resource provider matching a UUID was not found in the placement REST
API, the report client automatically creates the resource provider
record via the placement REST API. These resource provider objects will
be used in followup patches that add creation of inventory and
allocation records to the scheduler report client.
Included in this patch is a new [placement] nova.conf configuration
section with a single os_region_name configuration option that allows
Nova to grab the placement API endpoint URL for the particular OpenStack
region that it is in. We do not support endpoint URL overrides for the
placement API service. We only use the Keystone service catalog for
finding the endpoint for the placement service. We intentionally modeled
the determination of the placement endpoint URL after similar code that
determines the volume endpoint URL in /nova/volume/cinder.py.
This redoes the placement API using keystone session, and stubs out
where we can do more reasonable handling of errors. This works if we
fill out the right credentials in the placement section of the config
file.
Co-Authored-By: Sean Dague <sean@dague.net>
Change-Id: I9d28b51da25c523d22c373039e6d8b36fd96eba6
blueprint: generic-resource-pools
The user_id and project_id are being provided to the base oslo.context
class which sets them on the user and tenant attributes. In future
oslo.context will provide the user_id and project_id parameters itself.
By force setting user_id and project_id on the context we override the
user and project params that were set in the base class with None
values.
For now make the user_id and project_id parameters renames of the base
class properties and then when oslo.context catches up we can simply
remove these properties. Having these as properties is consistent with
most other OpenStack projects.
Change-Id: I9cd3c1b1121011156e78388b6beaac0253324e22
The 2.6.0 release of oslo.context adds a new attribute
'is_admin_project'. Always include that attribute when serializing the
object, and expect it in our tests.
Change-Id: I563cf810385e84ab30d49ef079b75df279006f0d
Related-Bug: 1602081
The from_environ method is designed to pick up all the parameters set
from auth_token middleware and other oslo middlewares and create a
context with them.
By doing this there will be information available to libraries like
oslo.policy and oslo.logging without nova having to track each change to
the base library.
There is ongoing work here to move more values to the base class that
will be cleaned up in future.
Change-Id: I6b61028fcecb86cc6c25fb69977774e266a8ea5b
Related-Bug: #1602081
melanie witt