[placement] Enable limiting GET /allocation_candidates

This adds a limit query parameter to GET
/allocation_candidates?limit=5&resource=VCPU:1

A 'limit' filter is added to the AllocationCandidates. If set, after
the database query has been run to create the allocation requests and
provider summaries, a slice or sample of the allocation requests is
taken to limit the results. The summaries are then filtered to only
include those in the allocation requests.

This method avoids needing to make changes to the generated SQL, the
creation of which is fairly complex, or the database tables. The amount
of data queried is still high in the extreme case, but the amount of
data sent over the wire (as JSON) is shrunk. This is a trade-off that
was discussed in the spec and the discussion surrounding its review.
If it turns out that memory use server-side is an issue we can
investigate changing the SQL.

A configuration setting, [placement]/randomize_allocation_candidates,
is added to allow deployers to declare whether they want the results
to be returned in whatever order the database chooses or a random
order. The default is "False" which is expected to preserve existing
behavior and impose a packing placement strategy.

When the config setting is combined with the limit parameter, if
"True" the limited results are a random sampling from the full
results. If "False", it is a slice from the front.

This is done as a new microversion, 1.16, with updates to docs, a reno
and adjustments to the api history doc.

Change-Id: I5f3d4f49c34fd3cd6b9d2e12b3c3c4cdcb409bec
Implements: bp allocation-candidates-limit

nova/api/openstack/placement/handlers/allocation_candidate.py

@@ -194,12 +194,21 @@ def list_allocation_candidates(req):
     """
     context = req.environ['placement.context']
     want_version = req.environ[microversion.MICROVERSION_ENVIRON]
-    util.validate_query_params(req, schema.GET_SCHEMA_1_10)
+    get_schema = schema.GET_SCHEMA_1_10
+    if want_version.matches((1, 16)):
+        get_schema = schema.GET_SCHEMA_1_16
+    util.validate_query_params(req, get_schema)
 
     requests = util.parse_qs_request_groups(req.GET)
+    limit = req.GET.getall('limit')
+    # JSONschema has already confirmed that limit has the form
+    # of an integer.
+    if limit:
+        limit = int(limit[0])
 
     try:
-        cands = rp_obj.AllocationCandidates.get_by_requests(context, requests)
+        cands = rp_obj.AllocationCandidates.get_by_requests(context, requests,
+                                                            limit)
     except exception.ResourceClassNotFound as exc:
         raise webob.exc.HTTPBadRequest(
             _('Invalid resource class in resources parameter: %(error)s') %

nova/api/openstack/placement/microversion.py

@@ -57,6 +57,7 @@ VERSIONS = [
     '1.14',  # Adds parent and root provider UUID on resource provider
              # representation and 'in_tree' filter on GET /resource_providers
     '1.15',  # Include last-modified and cache-control headers
+    '1.16',  # Add 'limit' query parameter to GET /allocation_candidates
 ]
 
 

nova/api/openstack/placement/rest_api_version_history.rst

@@ -207,3 +207,10 @@ actual last modified time of the most recently modified associated database
 entity or the current time if there is no direct mapping to the database. In
 addition, 'cache-control: no-cache' headers are added where the 'last-modified'
 header has been added to prevent inadvertent caching of resources.
+
+1.16 Limit allocation candidates
+--------------------------------
+
+Add support for a ``limit`` query parameter when making a
+``GET /allocation_candidates`` request. The parameter accepts an integer
+value, `N`, which limits the maximum number of candidates returned.

nova/api/openstack/placement/schemas/allocation_candidate.py

@@ -11,6 +11,9 @@
 #    under the License.
 """Placement API schemas for getting allocation candidates."""
 
+import copy
+
+
 # Represents the allowed query string parameters to the GET
 # /allocation_candidates API call
 GET_SCHEMA_1_10 = {
@@ -25,3 +28,15 @@ GET_SCHEMA_1_10 = {
     ],
     "additionalProperties": False,
 }
+
+
+# Add limit query parameter.
+GET_SCHEMA_1_16 = copy.deepcopy(GET_SCHEMA_1_10)
+GET_SCHEMA_1_16['properties']['limit'] = {
+    # A query parameter is always a string in webOb, but
+    # we'll handle integer here as well.
+    "type": ["integer", "string"],
+    "pattern": "^[1-9][0-9]*$",
+    "minimum": 1,
+    "minLength": 1
+}

nova/conf/placement.py

@@ -49,7 +49,17 @@ Possible values:
         help="""
 Endpoint interface for this node. This is used when picking the URL in the
 service catalog.
-""")
+"""),
+    cfg.BoolOpt('randomize_allocation_candidates',
+                default=False,
+                help="""
+If True, when limiting allocation candidate results, the results will be
+a random sampling of the full result set. If False, allocation candidates
+are returned in a deterministic but undefined order. That is, all things
+being equal, two requests for allocation candidates will return the same
+results in the same order; but no guarantees are made as to how that order
+is determined.
+"""),
 ]
 
 deprecated_opts = {

nova/tests/functional/api/openstack/placement/gabbits/allocation-candidates.yaml

@@ -43,6 +43,42 @@ tests:
   response_strings:
       - Invalid resource class in resources parameter
 
+- name: get bad limit microversion
+  GET: /allocation_candidates?resources=VCPU:1&limit=5
+  request_headers:
+      openstack-api-version: placement 1.15
+  status: 400
+  response_strings:
+      - Invalid query string parameters
+      - "'limit' was unexpected"
+
+- name: get bad limit type
+  GET: /allocation_candidates?resources=VCPU:1&limit=cow
+  request_headers:
+      openstack-api-version: placement 1.16
+  status: 400
+  response_strings:
+      - Invalid query string parameters
+      - "Failed validating 'pattern'"
+
+- name: get bad limit value negative
+  GET: /allocation_candidates?resources=VCPU:1&limit=-99
+  request_headers:
+      openstack-api-version: placement 1.16
+  status: 400
+  response_strings:
+      - Invalid query string parameters
+      - "Failed validating 'pattern'"
+
+- name: get bad limit value zero
+  GET: /allocation_candidates?resources=VCPU:1&limit=0
+  request_headers:
+      openstack-api-version: placement 1.16
+  status: 400
+  response_strings:
+      - Invalid query string parameters
+      - "Failed validating 'pattern'"
+
 - name: get allocation candidates no allocations yet
   GET: /allocation_candidates?resources=VCPU:1,MEMORY_MB:1024,DISK_GB:100
   status: 200
@@ -126,3 +162,11 @@ tests:
       cache-control: no-cache
       # Does last-modified look like a legit timestamp?
       last-modified:  /^\w+, \d+ \w+ \d{4} [\d:]+ GMT$/
+
+- name: get allocation candidates with limit
+  GET: /allocation_candidates?resources=VCPU:1,MEMORY_MB:1024,DISK_GB:100&limit=1
+  status: 200
+  request_headers:
+      openstack-api-version: placement 1.16
+  response_json_paths:
+      $.allocation_requests.`len`: 1

nova/tests/functional/api/openstack/placement/gabbits/microversion.yaml

@@ -39,13 +39,13 @@ tests:
   response_json_paths:
       $.errors[0].title: Not Acceptable
 
-- name: latest microversion is 1.15
+- name: latest microversion is 1.16
   GET: /
   request_headers:
       openstack-api-version: placement latest
   response_headers:
       vary: /OpenStack-API-Version/
-      openstack-api-version: placement 1.15
+      openstack-api-version: placement 1.16
 
 - name: other accept header bad version
   GET: /

placement-api-ref/source/allocation_candidates.inc

@@ -30,6 +30,7 @@ Request
 .. rest_parameters:: parameters.yaml
 
   - resources: resources_query_required
+  - limit: allocation_candidates_limit
 
 Response (microversions 1.12 - )
 --------------------------------

placement-api-ref/source/parameters.yaml

@@ -42,6 +42,14 @@ trait_name:
     The name of a trait.
 
 # variables in query
+allocation_candidates_limit:
+  type: integer
+  in: query
+  required: false
+  min_version: 1.16
+  description: >
+    A positive integer used to limit the maximum number of allocation
+    candidates returned in the response.
 member_of:
   type: string
   in: query

releasenotes/notes/allocation-candidates-limit-37fe5c2ce57daf7f.yaml

@@ -0,0 +1,11 @@
+---
+features:
+  - |
+    Add support, in new placement microversion 1.16, for a ``limit`` query
+    parameter when making a ``GET /allocation_candidates`` request. The
+    parameter accepts an integer value, `N`, which limits the number of
+    candidates returned. A new configuration item
+    ``[placement]/randomize_allocation_candidates``, defaulting to `False`,
+    controls how the limited results are chosen. If `True`, a random sampling
+    of the entire result set is taken, otherwise the first N results are
+    returned.