Commit Graph

104 Commits

Author SHA1 Message Date
jichen 5be3a8a415 Move placement test cases from db to placement
A set of placement functional test is in db layer and we plan to move
placement out of nova, better to move this file to
functional/api/openstack/placement

partial implement blueprint placement-extract

Change-Id: Ie25ed974a19d8fe7bc367e2e0593646a620a49e9
2018-03-22 13:27:44 +08:00
Chris Dent 1a072eab33 Move placement exceptions into the placement package
This change moves the exceptions raised within the placement service
into the nova.api.openstack.placement package. This is part of the
process of unifying all placement code into one subdirectory so that
a future lift and shift is easier and cleaner.

This is mostly a straightforward change of imports and a moving of
existing exceptions in the placements handlers and objects with a few
caveats:

* Dealing with nova/db/sqlalchemy/resource_class_cache.py has not
  yet been accomplished, so it remains where it is and imports
  exceptions from the placement hierarchy. A TODO indicating some of the
  options is left for future work.
* Exceptions with the name ResourceProviderInUse and InventoryInUse are
  used on both sides of the nova/placement interaction. This is
  noted with comments in both nova/exception.py and
  placement/exception.py.
* test_report_client.py has had a TODO added to make it clear that the
  exceptions it uses are nova-side, and perhaps the test file should be
  moved.
* The base class of the original exceptions, NoveException, does a bit
  more than is required on the placement side so a new private class
  _BaseException is created which removes support for handling 'code'.
  This is not required because all exceptions in placement are supposed
  to be manually caught in handlers and explicitly transformed into http
  exceptions.

blueprint placement-extract

Change-Id: I2b94945a0963d6a61af931505b69afe2d4733759
2018-03-19 15:28:02 +00:00
Chris Dent 8b19d5b735 Move resource provider objects into placement hierarchy
This is so we don't have all the other objects imported into the
the placement process, which happens by virtue of being in the
nova.objects package. It also eases the eventual extraction of
placement.

That extraction will involve lifting and shifting the
nova/api/openstack/placement directory (plus any tests) to a new
repository. .../placement will become the roo placement directory
in the repo, since a) placement is (for now) only an API, b) it
eases extraction. Subsequent patches will do things like move
exceptions and the wsgi application.

Change-Id: I3e5144217c194f2571526311ddae6661cfed2ea9
2018-03-13 09:01:31 +00:00
Chris Dent f346913594 Move resource class fields
Move the ResourceClass field to its own package, and move that package
to the top of the nova hierarchy since it is used by both nova tooling
and placement tooling but we don't want the placement version to have to
incorporate the nova code. Eventually we'd like to see an
os-resource-classes library, similar to os-traits, which will serve this
functionality. This is a step in that direction.

Changes in this patch are quite widespread, but are mostly only changes
of imports.

Change-Id: Iea182341f9419cb514a044f76864d6bec60a3683
2018-03-13 09:01:30 +00:00
Jay Pipes b3bc0a5a9b add _has_provider_trees() utility function
Adds a simple _has_provider_trees() function to
nova/objects/resource_provider.py that returns True if any provider has
a non-NULL parent_provider_id. This will allow future code to take
simpler, optimized code paths when retrieving allocation candidates and
the deployment is not using nested resource providers.

Change-Id: I7305c8c00dd5595dcbfc9ea551da33677dbf38dd
blueprint: nested-resource-providers
2018-01-16 12:55:07 -05:00
Chris Dent 3491f3d6f2 Do not set allocation.id in AllocationList.create_all()
The _set_allocations method used by AllocationList.create_all is
side-effecty: it sets the 'id' attribute on the list of Allocation
objects that is passed to it.

At the start of the method the incoming Allocation objects are
checked to see if they have already been created, by checking
for an 'id' field.

Meanwhile, _set_allocations is also configured to retry on db
deadlock. The deadlock can happen for a variety of reasons within
the transaction. The original theory, discussed in the original
fix, I2c276dc0125b5b9f7a54a1cd431b1b2f5239e93a, is that it is
during resource provider generation checks. In the associated bug it
looks like it may happen sometimes while inserting allocations.

In either case, if we have gone through the 'for alloc in allocs'
loop at least once, the contents of the 'allocs' list has been
modified to have at least one of the alloc.id fields set. Upon
retry, the 'id' field check at the start of the method will fail,
leading to an ObjectActionError and an eventual 500 at the placement
API level.

This change takes the simplest approach and simply removes the setting
of the 'id' attribute on the allocations in the 'allocs' list. There are
other ways to deal with this, this is the least intrusive. It works
because:

* create_all is only called from the allocation handler in placement,
  and the objects are not used (the response bodies are empty)
* other than the 'id' change, the alloc members in the allocs list
  are otherwise unchanged
* this kind of side-effecty business is dangerous, so let's not
  rely on it

Tests which were relying on the side-effecty business have been adjusted
accordingly.

Change-Id: I3c7aea7d8959a20c3c404bc6616b47336ff40b67
Closes-Bug: #1739453
2017-12-20 22:13:19 +00:00
Jay Pipes 6242a15e0c placement: allow filter providers in tree
Adds a new 'in_tree' filter parameter to the
ResourceProviderList.get_all_by_filters() method. This parameter allows
the caller to get providers in the "provider tree" of a supplied
provider UUID.

Change-Id: If35ee6a90ca0a999f6c4815e6f5e99d9962dcc46
blueprint: nested-resource-providers
2017-12-06 10:48:04 -06:00
Jay Pipes b10f11d7e8 placement: add nested resource providers
Adds initial support for storing the relationship between parent and
child resource providers. Nested resource providers are essential for
expressing certain types of resources -- in particular SR-IOV physical
functions and certain SR-IOV fully-programmable gate arrays. The
resources that these providers expose are of resource class
SRIOV_NET_VF and we will need a way of indicating that the physical
function providing these virtual function resources is tagged with
certain traits (representing vendor_id, product_id or the physical
network the PF is attached to).

The compute host is a resource provider which has an SR-IOV-enabled
physical function (NIC) as a child resource provider. The physical
function has an inventory containing some total amount of SRIOV_NET_VF
resources. These SRIOV_NET_VF resources are allocated to zero or more
consumers (instances) on the compute host.

                    compute host (parent resource provider)
                         |
                         |
                      SR-IOV PF  (child resource provider)
                         :
                        / \
                       /   \
                    VF1    VF2   (inventory of child provider)

The resource provider model gets two new fields:

 - root_provider_uuid: The "top" or "root" of the tree of nested
   providers
 - parent_provider_uuid: The immediate parent of the provider, or None
   if the provider is a root provider.

The database schema adds two new columns to the resource_providers
table that contain the internal integer IDs that correspond to the
user-facing UUID values:

 - root_provider_id
 - parent_provider_id

The root_provider_uuid field is non-nullable in the ResourceProvider
object definition, and this code includes an online data migration to
automatically populate the root_provider_id field with the value of the
resource_providers.id field for any resource providers already in the
DB.

The root_provider_id field value is populated automatically when a
provider is created. If the parent provider UUID is set, then the
root_provider_id is set to the root_provider_id value of the parent. If
parent is unset, root_provider_id is set to the value of the id
attribute of the provider being created. The corresponding UUID values
for root and parent provider are fetched in the queries that retrieves
resource provider data using two self-referential joins.

The root_provider_id column allows us to do extremely quick lookups of
an entire tree of providers without needing to perform any recursive
database queries.

Logic in this patch ensures that no resource provider can be deleted if
any of its children have any allocations active on them. We also check
to ensure that when created or updated, a resource provider's parent
provider UUID actually points to an existing provider.

It's important to point out that qualitative trait information is only
associated with a resource provider entity, not the resources that
resource provider has in its inventory. This is the reason why nested
resource providers are necessary. In the case of things like NUMA nodes
or SRIOV physical functions, if a compute host had multiple SRIOV
physical functions, each associated with a different network trait,
there would be no way to differentiate between the SRIOV_NET_VF
resources that those multiple SRIOV physical functions provided if the
containing compute host had a single inventory item containing the
total number of VFs exposed by both PFs.

Change-Id: I2d8df57f77a03cde898d9ec792c5d59b75f61204
blueprint: nested-resource-providers
Co-Authored-By: Moshe Levi <moshele@mellanox.com>
2017-11-28 15:29:28 -05:00
Zuul 43d28c2aab Merge "required traits for no sharing providers" 2017-11-27 19:23:03 +00:00
Jay Pipes 828d9eb516 required traits for no sharing providers
Adds filtering on a set of required traits to the
AllocationCandidates._get_by_filters() method, for deployments with no
shared providers that are involved in the request.

This patch should demonstrate where I'm heading with the breaking out of
the various SQL-related pieces into their own functions. The next patch
will add required traits support for scenarios with sharing providers.

Change-Id: Iac246245ef7aedfa2d23e623f83fdf384e252159
2017-11-27 15:13:07 +00:00
Chris Dent 3f33e89d79 Include project_id and user_id in AllocationList.get_all_by_consumer_id
To support including project_id and user_id in the output of GET
/allocations/{consumer_uuid} adjust the
_get_allocations_by_consumer_uuid query to join the necessary tables to
get project_id and user_id information.

Note that this is done as outer joins because in order to support older
(but not much older) microversions we need to allow those fields to be
null.

_get_allocations_by_provider_id is not extended in the same way as we
do not yet have any immediate need for the information in GET
/resource_providers/{uuid}/allocations.

Change-Id: I59175fa51e9553f41c73a6bcd1a77a134e0b19e3
Partially-Implements: bp symmetric-allocations
2017-11-20 08:17:22 +00:00
Jay Pipes 039c94a6b9 begin refactor AllocCandidates._get_by_filters()
The AllocationCandidates._get_by_filters() method is a long, complex
mega-method. We want to detangle it and make the code in it
understandable and readable. The first step in this refactoring involves
renaming a few poorly-named variables, adding more code comments, and
breaking out the SQL for the most common code path (where no providers
share resources with other providers) into a separate module-level
function.

Change-Id: I716b279f719eb1fe26401a776344b976eca6587a
2017-11-15 08:04:41 -06:00
Eric Fried b974e20862 Extract allocation candidates functional tests
This change set pulls AllocationCandidatesTestCase out of
test_resource_provider and into its own test_allocation_candidates
module.

There is no change to the code.  This is just a refactor.  We're going
to add a bunch more test cases for allocation candidates, and the
test_resource_provider module was already getting out of hand.

Change-Id: Iedfb712d4668a2d34112449aa6ef0263d02e24a4
2017-11-06 14:50:00 -06:00
Zuul 25ecb01963 Merge "[placement] Allow _set_allocations to delete allocations" 2017-10-27 20:53:00 +00:00
Jay Pipes 1df5aad910 rp: Remove RP.get_traits() method
When listing data, we use the
[Object]List.get_all_by_resource_provider_uuid() pattern. Except for
resource provider traits. For resource provider traits, we had a
ResourceProvider.get_traits() instance method.

This patch makes the getting of resource provider traits follow a
consistent pattern and makes a TraitList.get_all_by_provider()
@classmethod that accepts a ResourceProvider object and returns a
TraitList containing traits associated with the resource provider.

In following patches, I change similar method on List objects like
UsageList.get_all_by_resource_provider_uuid() and
AllocationList.get_all_by_resource_provider_uuid() to match this new
pattern of get_all_by_resource_provider() and passing in a
ResourceProvider object.

Note that instead of get_all_by_resource_provider_uuid(), I went with
get_all_by_resource_provider(). This is intentional. Asking for traits
(or usages or allocations for that matter) by resource provider UUID
when we already know the resource provider's internal ID (because we
always look up the resource provider object first by UUID) is wasteful
and causes us to do an unnecessary JOIN to the resource_providers
table.

blueprint: de-orm-resource-providers

Change-Id: If756b06a10aba8481de7e6cbcd940900557e78bc
2017-10-26 21:10:46 +00:00
Chris Dent b118a9ab84 [placement] Allow _set_allocations to delete allocations
If the used field of an allocation has the value of 0 then use that to
signal that the allocation identified for that consumer and resource
provider is to be deleted within the transaction of _set_allocations.

No explicit deletion is required because _set_allocations deletes all
allocations for a consumer before creating new ones. The trick here is
not creating another one if used is set to 0.

This is going to be used by a future commit that adds support POST
/allocations where it will be possible to add, modify, and delete
allocation records for >1 consumer in one request. At the HTTP API
layer it is currently not possible to provide empty allocations
(to PUT /allocations/{consumer_uuid}) by virtua of JSONSchema. This
will remain true. Only POST will use the functionality added in this
commit.

Change-Id: I585486a240d5c3b8e9ffec7223ca643b331463e6
2017-10-26 19:29:09 +01:00
Chris Dent e000a8f290 Move project_id and user_id to Allocation object
It had been on the AllocationList object but this makes it impossible to
have an AllocationList from multiple project ids. When it was decided to
allow setting allocations from multiple different consumers, it was
decided that since we were already in the code changing things, we
should also make adjustments to allow each allocation to have its own
project and user id.

In the process, ensure that if a persisted Allocation has a project_id
and user_id associated with it, load those values when the Allocation is
loaded (either list allocations by consumer or by resource provider).

We cannot enforce (at the object level) that Allocations must have a
project_id and user_id because there is a microversion where they are
not required that we continue to support. In newer microversions the
JSON schema enforces the their requirement.

Inspecting the placement fixtures related to this change revealed a bug
with a missing user_id in one Allocation being created.

Change-Id: I3cf887bd541c187ec3baca2ae3f8c16f1754e96e
2017-10-25 13:47:26 +01:00
Jay Pipes 399925fb72 rp: fix up AllocList.get_by_resource_provider_uuid
Changes the AllocationList.get_by_resource_provider_uuid() to
AllocationList.get_by_resource_provider(), accepting a ResourceProvider
object parameter instead of a UUID. This is exactly how the
InventoryList object was changed in an earlier patch. A new
module-scoped _get_allocations_by_provider_id() function is added that
bypasses the ORM and uses the SQLAlchemy core expression API directly
to produce a smaller query with no joins.

blueprint: de-orm-resource-providers

Change-Id: Id866ab8353077cfeeae693c2958f412930f0f24d
2017-10-11 12:03:35 -04:00
Jay Pipes db4d6799f1 rp: remove ability to delete 1 allocation record
The Allocation.destroy() method has been removed, making the Allocation object
a plain-old data (POD) object. Allocations may only be deleted as an atomic
unit -- all allocation records for a given consumer_id are considered a single
thing. We already had a function that batch-deleted allocation records by
consumer ID so this patch does little more than remove the singular
Allocation.destroy() method and plumb AllocationList.delete_all() to use the
existing batch-delete function.

blueprint: de-orm-resource-providers

Change-Id: I456ac2ffe506eba849a29b36343875c4e1423aed
2017-10-11 12:03:35 -04:00
Jay Pipes de5b8a4f2e rp: streamline InventoryList.get_all_by_rp_uuid()
Changes the implementation of the
InventoryList.get_all_by_resource_provider_uuid() method to streamline
the generated SQL for the query against the database and move the
@staticmethod out into module scope to be consistent with other DB
retrieval methods.

The get_all_by_resource_provider_uuid() interface is converted to a
get_all_by_resource_provider() interface, accepting a ResourceProvider
object instead of a UUID string. The reason for this is because all
calling locations of InventoryList.get_all_by_resource_provider_uuid()
were already calling ResourceProvider.get_by_uuid() and therefore
already had a ResourceProvider object that could be used to construct a
list of Inventory objects. Passing in this resource provider object
means we can eliminate the joins from inventories to resource_providers
table and the associated ORM costs.

blueprint: de-orm-resource-providers

Change-Id: I895114872a515a269487a683124b63303818e19c
2017-10-11 12:03:35 -04:00
Jay Pipes cb2e9dbb30 rp: remove CRUD operations on Inventory class
The nova.objects.resource_provider.Inventory class was only intended to
be a plain-old data (POD) class and never intended to be able to
create, update or delete as a single item. Rather, the ResourceProvider
class has a set_inventory() method that accepts an InventoryList object
and operates on a set of inventory records as an atomic unit.

This patch removes the old, unused code paths that added CRUD
operations for the Inventory class, turning Inventory into a POD class.

blueprint: de-orm-resource-providers

Change-Id: Id8b182b3074122fc0bdbe201960e174b8f8d52f0
2017-10-11 12:03:35 -04:00
Jay Pipes 9d400c3a0a rp: Move RP._get|set_aggregates() to module scope
Instead of ResourceProvider._get_aggregates() and
ResourceProvider._set_aggregates() being @classmethods, move them to
being module-level functions to be consistent with the similar
functions for inventory and allocation information.

blueprint: de-orm-resource-providers

Change-Id: I52db9b4ca89aeb2a4ce9d10820bdac6fabf43ea4
2017-10-09 09:10:08 -07:00
Chris Dent efafd919ef [placement] Unregister the ResourceProvider object
Unregister the ResourceProvider object because we do not need
RPC and versioning for the objects in nova.objects.resource_provider.

There are two primary changes here:

* unregistering the ResourceProvider class
* where objects.ResourceProvider is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: I564a4a10c9f07883025df70fecb1a59d547ef7ed
2017-09-15 16:49:45 +01:00
Chris Dent fda760e94f [placement] Unregister the ResourceProviderList object
Unregister the ResourceProviderList object because we do not need
RPC and versioning for the objects in nova.objects.resource_provider.

There are two primary changes here:

* unregistering the ResourceProviderList class
* where objects.ResourceProviderList is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: Icf1122ba856664fac4de7efa304ce6908e46f6d4
2017-09-15 16:49:45 +01:00
Chris Dent 8f39507996 [placement] Unregister the Inventory object
Unregister the Inventory object because we do not need RPC and
versioning for the objects in nova.objects.resource_provider.

There are two primary changes here:

* unregistering the Inventory class
* where objects.Inventory is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: Ideca0f09f5dc4ba6bb194a2413ed93cf592dd963
2017-09-15 16:49:45 +01:00
Chris Dent 52243451ec [placement] Unregister the InventoryList object
Unregister the InventoryList object because we do not need RPC and
versioning for the objects in nova.objects.resource_provider.

There are two primary changes here:

* unregistering the InventoryList class
* where objects.InventoryList is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: Ic11bbef4ccbf1f3df1d50837c49e8a4cbd4ba572
2017-09-15 16:49:45 +01:00
Chris Dent 85f4d28a49 [placement] Unregister the Allocation object
Unregister the Allocation object because we do not need RPC and
versioning for the objects in nova.objects.resource_provider.

There are two primary changes here:

* unregistering the Allocation class
* where objects.Allocation is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: Ib026438b5f843fce848277347a95f3c4b6f94fb1
2017-09-15 16:49:45 +01:00
Chris Dent 5a1ef8fa86 [placement] Unregister the AllocationList object
Unregister the AllocationList object because we do not need RPC and
versioning for the objects in nova.objects.resource_provider.

There are two primary changes here:

* unregistering the AllocationList class
* where objects.AllocationList is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: I554380e9b427a6c12261929989536d95086b3e0b
2017-09-15 16:49:45 +01:00
Chris Dent 4f00bab7ff [placement] Unregister the UsageList object
Unregister the UsageList object because we do not need RPC
and versioning for the objects in nova.objects.resource_provider.

There are two primary changes here:

* unregistering the UsageList class
* where objects.UsageList is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: Id1d3aeaf5831e0804016e98511f719705e9da706
2017-09-15 16:49:45 +01:00
Chris Dent 99f0387b94 [placement] Unregister the ResourceClass object
Unregister the ResourceClass object because we do not need RPC
and versioning for the objects in nova.objects.resource_provider.

There are three primary changes here:

* unregistering the ResourceClass class
* where objects.ResourceClass is used, point directly to the
  nova.objects.resource_provider package instead
* move the CUSTOM_NAMESPACE for resource class names to the the fields
  object. This avoids an import loop and also makes more sense as
  the field represents the name.

Partially-Implements: bp placement-deregister-objects
Change-Id: If537c786ea1a61383d82bad9a1de86c25c6cdbd2
2017-09-15 16:49:45 +01:00
Chris Dent 5cabc535af [placement] Unregister the ResourceClassList object
Unregister the ResourceClassList object because we do not need RPC
and versioning for the objects in nova.objects.resource_provider.

There are two primary changes here:

* unregistering the ResourceClassList class
* where objects.ResourceClassList is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: I109cceb08050b2714c2a2409bf4674f718ab29d6
2017-09-15 16:49:45 +01:00
Chris Dent 60d14c5915 [placement] Unregister the Trait object
Unregister the Trait object because we do not need RPC and
versioning for the objects in nova.objects.resource_provider.

There are two primary changes here:

* unregistering the Trait class
* where objects.Trait is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: I73e08aba095d843b59f73cf2ce8d8d4801740ed2
2017-09-15 16:49:45 +01:00
Chris Dent 171c0cab7d [placement] Unregister the TraitList object
Since we don't use RPC in placement and intend to never use RPC in
placement it makes little sense to register the resource provider
objects and keep track of their versions. They still need to be
activated as structured objects to get their fields set and properly
inherited.

Tests of versioned objects inspect the dependency trees of objects that
have other objects as fields so this process starts with the TraitList
object. Nothing depends on it. In subsequence patches the other objects
will be unregistered.

There are two primary changes here:

* unregistering the TraitList class
* where objects.TraitList is used, point directly to the
  nova.objects.resource_provider package instead

Partially-Implements: bp placement-deregister-objects
Change-Id: If284b7d595a7dd214a5cd3a322419074dcf0373e
2017-09-15 16:49:45 +01:00
He Jie Xu 2e5f293f13 placement: avoid returning duplicated alloc_reqs when no sharing rp
This patch returns from the loop early where there are no sharing
rps, avoiding generating duplicated requests.

Change-Id: I7c73b720f09bb2371770b1c6de537bb5527785a9
Closes-Bug: #1709788
2017-08-10 20:18:20 +08:00
Jenkins 4dc7682e79 Merge "placement: filtering the resource provider id when delete trait association" 2017-08-08 14:43:02 +00:00
He Jie Xu e66c1accbd placement: filtering the resource provider id when delete trait association
When delete a trait association for specific resource provider, the current
code will delete that trait associations for all the resource providers.

The expected behaviour is only the specific trait association for the
specific resource provider will be removed. This patch corrects the filter
for the deletion of trait association.

Change-Id: I4b8f1b8f0a8d16f1c415edbfb2a7d342d176f70a
Closes-Bug: #1708978
2017-08-07 10:08:05 +08:00
Jay Pipes afab0ca0c8 placement: remove existing allocs when set allocs
Bug #1707669 highlighted a situation that arose when attempting to
remove part of an allocation for a source host during a resize
operation where the exiting allocation was not being properly deleted.

In this patch, we remove the part of the WHERE condition that limited
deleted allocation records to only those referring to a particular
resource provider. In doing so, we make the creation of an allocation
for a consumer a proper overwrite operation.

Change-Id: I0835e5b4f22277465012aab9a5bf474608cb533b
Fixes-bug: #1707669
2017-08-01 12:58:02 -04:00
Jay Pipes f86b5610d4 use os_traits.MISC_SHARES_VIA_AGGREGATE
There was a TODO left in the test_resource_provider functional test
case to actually use the os_traits.MISC_SHARES_VIA_AGGREGATE trait
instead of creating a trait once os-traits library was released that
contained MISC_SHARES_VIA_AGGREGATE. Well, os-traits version 0.3.0
introduces this traits in commit
I4e9d8389c9b93ffee20adb53d864d55d335a27fd and Nova has had os-traits
0.3.0 in requirements for a while now, so this patch takes care of that
TODO.

Change-Id: I9f5290c00cdc7b740755fe2e988e2a9ebf2848f5
2017-07-26 12:47:50 +00:00
Jay Pipes 86fe9a70c2 style-only: s/context/ctx/
Changes the self.context attribute to self.ctx in the
test_resource_provider functional test. Later patches clean up and
condense this test file and having a shorter context attribute allows
us to reduce a number of lines of code in the test file and get rid of
some of the pesky parens on a line by themselves that so many people
hate.

Change-Id: I29c37e260465a574d3134a005dbc14e2a3c6ea00
2017-07-26 13:46:54 +01:00
Balazs Gibizer 5858fb7995 placement: proper JOIN order for shared resources
There is an existing functional test that proves that shared non custom
resource (DISK_GB) mixed with non shared compute resources works. This
patch adds a test case where the shared resource is also a custom
resource.

An issue in the _get_all_with_shared() function where the dict of
resource to sharing resource providers that is used to construct the
JOIN chain was not being sorted so that non-shared resources were added
to the JOIN chain ahead of shared resources.

The SQL JOINs that are generated by the _get_all_with_shared() function
depend on a specific order. For non-shared resources, an INNER JOIN is
done to the preceding derived query whereas for shared resources, a
LEFT JOIN is done.

If we do the LEFT JOIN followed by INNER JOINs, the SQL expression will
produce an incorrect projection.

Fixes-bug: #1705231
Co-Authored-By: Chris Dent <cdent@anticdent.org>
Co-Authored-By: Jay Pipes <jaypipes@gmai.com>
Change-Id: If47c970eb57dcb81f2ce40104ceec42a92c59066
2017-07-20 12:33:41 +01:00
Jay Pipes d67d3459af placement: alloc candidates only shared resources
When attempting to perform a GET /allocation_candidates request for
only resources that are shared, a KeyError was being produced:

http://paste.openstack.org/show/615753/

The problem is that the _get_usages_by_provider_and_rc() method returns
a dict with only the sharing resource provider usage information but
the _get_all_with_shared() returns a list of resource provider IDs
including both the *sharing* resource provider *and* the shared-with
providers.

When the code attempts to cross-reference provider summaries (which are
constructed by looping over the usage dicts) with each provider ID in
the result from _get_all_with_shared(), we hit a KeyError on the
shared-with provider IDs because there was no usage record (because the
usage query filters on resource class ID and we requested only a
resource class ID that was shared)

This patch fixes the problem with a KeyError being produced for those
providers that do not provide any resources (i.e. they are only
included in the returned results because they have requested resource
shared *with* them) by returning both the internal integer ID and the
UUID of providers from the _get_all_shared_with() function and then in
the loop to create allocation requests, simply ignoring any resource provider
that doesn't exist in the provider_summaries dict.

Closes-Bug: #1705071

Change-Id: I742fd093a8b33ff88244b2990021784e4b65f51f
2017-07-20 12:33:41 +01:00
He Jie Xu 0e6cac5fde placement: filter usage records by resource provider id
The usage records are used to build the provider summary in the
AllocationCandidates.get_by_filters. Due to miss a filter on the
resource provider id, the extra resource provider summary returned.

Change-Id: Idc27e3ab3d4e1d2c6ade2f719572cbc2f5eee133
Closes-Bug: #1702275
2017-07-04 20:39:28 +08:00
Jenkins 3863eca0ac Merge "placement: Add GET /usages to placement API" 2017-06-24 01:30:23 +00:00
Jenkins 2b887d0c10 Merge "placement project_id, user_id in PUT /allocations" 2017-06-23 23:25:18 +00:00
Jay Pipes 604f17d60c placement: produce set of allocation candidates
We want to allow the scheduler to claim resources against multiple
providers of resources involved in a single allocation request,
including providers of shared resources and eventually child providers
in a hierarchy. To do so, we need to return back to the scheduler a set
of allocation request structures and some details about the providers
involved in those alternative claim possibilities.

This patch adds some new objects to the resource providers object
modeling that are used to describe these allocation choices and the
underlying providers of resources. A top-level AllocationCandidates
object contains two fields: allocation_requests and provider_summaries.
These fields store lists of objects of type AllocationRequest and
ProviderSummary respectively. The AllocationCandidates.get_by_filters()
method accepts a dict of resources in a similar fashion to
ResourceProviderList.get_all_by_filters() and will be used in follow-up
patches to expose the proposed GET /allocation_requests REST endpoint
being discussed in the placement-allocation-requests spec
(https://review.openstack.org/#/c/471927/).

Following patches will also add support for traits and filtering by
UUIDs and member_of aggregate lists.

Change-Id: Ibc049496fb21fb34e7b91aa869fdff42127fb384
blueprint: placement-allocation-requests
2017-06-19 15:01:38 -04:00
melanie witt 77224c1feb placement: Add GET /usages to placement API
This adds GET /usages as part of a new microversion 1.9 of the
placement API. Usages can be queried by project or project/user:

  GET /usages?project_id=<project id>
  GET /usages?project_id=<project id>&user_id=<user id>

and will be returned as a sum of usages, for example:

  200 OK
  Content-Type: application/json

  {
    "usages": {
      "VCPU": 2,
      "MEMORY_MB": 1024,
      "DISK_GB": 50,
      ...
    }
  }

A new method UsageList.get_all_by_project_user() has been added
for usage queries.

Part of blueprint placement-project-user

Change-Id: I8b948a4dfe6a50bea053b5dcae8f039229e2e364
2017-06-15 18:23:37 +00:00
melanie witt a909673682 placement project_id, user_id in PUT /allocations
This adds project_id and user_id required request parameters as part of
a new microversion 1.8 of the placement API.

Two new fields, for project and user ID, have been added to the
AllocationList object, and the method AllocationList.create_all() has
been changed to ensure that records are written to the consumers,
projects, and users tables when project_id and user_id are not None.

After an upgrade, new allocations will write consumer records and
existing allocations will have corresponding consumer records written
when they are updated as part of the resource tracker periodic task for
updating available resources.

Part of blueprint placement-project-user

Co-Authored-By: Jay Pipes <jaypipes@gmail.com>
Change-Id: I3c3b0cfdd33da87160255ead51a0d9ff73667655
2017-06-15 18:23:37 +00:00
Chris Dent 11f4b62745 Reset the _TRAITS_SYNCED global in Traits tests
The global is reset before and after tests because we don't know
what other tests might have been messing with it (without even
knowing) in this process. We want to start from a clean slate and
leave the slate clean.

Change-Id: I42b9a7973ec5c4f42578779dc6ad59274212113f
Closes-Bug: #1696267
2017-06-06 22:56:11 +00:00
Chris Dent e013d1cabe Sync os-traits to Traits database table
When a new version of the os-traits library is released, the Traits
table in the api database needs to be updated to reflect those new
traits. This change does that, once, the first time either
Trait.get_by_name or TraitList.get_all is called in any process that is
using those objects.

This is an alternative to I729e84ea0ff8f333a156b24b15fe4d368209d015.
The major difference here is that there is no trait cache so the
surface area of the change is much smaller.

The list traits test in gabbit/traits.yaml has been changed to not
assert the length of traits returned: this can now change with each
release of the os-traits library.

Depends-on: I7a3f4bb8501fc3edad43e1aae5cb6b9ef1c0b00d
Co-Authored-By: Jay Pipes <jaypipes@gmail.com>
Change-Id: Ia92a4fd20b8991c6f4b34e3546cfb22aa5ed78aa
2017-06-06 10:31:05 +00:00
Jay Pipes a944ae1795 placement: test for agg association not sharing
Adds yet another functional test scenario that demonstrates an
environment with a set of compute nodes with no local disk connected to
a shared storage provider via an aggregate and a set of compute nodes
with no local disk associated to an aggregate with a provider having
DISK_GB inventory but not decorated with the MISC_SHARES_VIA_AGGREGATE
trait.

This emulates a deployment where you might have a mix of shared and
non-shared storage compute nodes grouped into different host aggregates
and you want fine-grained control over which providers are sharing some
resource with providers in the host aggregate.

Change-Id: Ie71e23b7f51756df4dab2ffb3b7fa938e1a4b475
blueprint: shared-resources-pike
2017-05-31 20:32:15 -04:00