openstack
/
nova
Code Issues Proposed changes

Document differences and similaries between extra specs and hints 

Scheduler hints are not really documented very well at all except
for being mentioned per scheduler filter in the admin configuration
guide, nor are they documented within relation to flavor extra
specs which are both used for impacting scheduling decisions and
are choices that a deployer has to make based on how they configure
their cloud.

This change adds a document about scheduler hints and how they are
similar to and different from flavor extra specs, including end
user discoverability and interoperability, and thoughts on which
should be used if writing a custom scheduler filter.

The TODO in the API guide is also resolved by linking to this
document.

Change-Id: Ib1f35baacf59efafb9e4bccfcc4f0025d99ad5b2
This commit is contained in:
Matt Riedemann 2018-07-10 11:20:10 -04:00
parent 6522ea3ecf
commit 579e0cfd57
5 changed files with 169 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
api-guide/source/server_concepts.rst
View File

@ -608,7 +608,9 @@ TODO: Add some description about BDM.
Scheduler Hints
~~~~~~~~~~~~~~~
TODO: Add description about how to custom scheduling policy for server booting.
Refer to `this document`_ for information on scheduler hints.
.. _this document: https://docs.openstack.org/nova/latest/reference/scheduler-hints-vs-flavor-extra-specs.html#scheduler-hints
Server Consoles
~~~~~~~~~~~~~~~

1
doc/source/index.rst
View File

@ -239,6 +239,7 @@ looking parts of our architecture. These are collected below.
reference/threading
reference/update-provider-tree
reference/vm-states
reference/scheduler-hints-vs-flavor-extra-specs
user/index
user/aggregates
user/architecture

2
doc/source/reference/index.rst
View File

@ -13,6 +13,8 @@ The following is a dive into some of the internals in nova.
* :doc:`/reference/rpc`: How nova uses AMQP as an RPC transport
* :doc:`/reference/scheduling`: The workflow through the scheduling process
* :doc:`/reference/scheduler-hints-vs-flavor-extra-specs`: The similarities
and differences between flavor extra specs and scheduler hints.
* :doc:`/reference/live-migration`: The live migration flow
* :doc:`/reference/services`: Module descriptions for some of the key modules
used in starting / running services

161
doc/source/reference/scheduler-hints-vs-flavor-extra-specs.rst Normal file
View File

@ -0,0 +1,161 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
=========================================
Scheduler hints versus flavor extra specs
=========================================
People deploying and working on Nova often have questions about flavor extra
specs and scheduler hints and what role they play in scheduling decisions, and
which is a better choice for exposing capability to an end user of the cloud.
There are several things to consider and it can get complicated. This document
attempts to explain at a high level some of the major differences and
drawbacks with both flavor extra specs and scheduler hints.
Extra Specs
-----------
In general flavor extra specs are specific to the cloud and how it is
organized for capabilities, and should be abstracted from the end user.
Extra specs are tied to :doc:`host aggregates </user/aggregates>` and a lot
of them also define how a guest is created in the hypervisor, for example
what the watchdog action is for a VM. Extra specs are also generally
interchangeable with `image properties`_ when it comes to VM behavior, like
the watchdog example. How that is presented to the user is via the name of
the flavor, or documentation specifically for that deployment,
e.g. instructions telling a user how to setup a baremetal instance.
.. _image properties: https://docs.openstack.org/glance/latest/admin/useful-image-properties.html
Scheduler Hints
---------------
Scheduler hints, also known simply as "hints", can be specified during server
creation to influence the placement of the server by the scheduler depending
on which scheduler filters are enabled. Hints are mapped to specific filters.
For example, the ``ServerGroupAntiAffinityFilter`` scheduler filter is used
with the ``group`` scheduler hint to indicate that the server being created
should be a member of the specified anti-affinity group and the filter should
place that server on a compute host which is different from all other current
members of the group.
Hints are not more "dynamic" than flavor extra specs. The end user
specifies a flavor and optionally a hint when creating a server, but
ultimately what they can specify is static and defined by the deployment.
Similarities
------------
* Both scheduler hints and flavor extra specs can be used by
:doc:`scheduler filters </admin/configuration/schedulers>`.
* Both are totally customizable, meaning there is no whitelist within Nova of
acceptable hints or extra specs, unlike image properties [1]_.
* An end user cannot achieve a new behavior without deployer consent, i.e.
even if the end user specifies the ``group`` hint, if the deployer did not
configure the ``ServerGroupAntiAffinityFilter`` the end user cannot have the
``anti-affinity`` behavior.
Differences
-----------
* A server's host location and/or behavior can change when resized with a
flavor that has different extra specs from those used to create the server.
Scheduler hints can only be specified during server creation, not during
resize or any other "move" operation, but the original hints are still
applied during the move operation.
* The flavor extra specs used to create (or resize) a server can be retrieved
from the compute API using the `2.47 microversion`_. As of the 19.0.0 Stein
release, there is currently no way from the compute API to retrieve the
scheduler hints used to create a server.
.. note:: Exposing the hints used to create a server has been proposed [2]_.
Without this, it is possible to workaround the limitation by doing
things such as including the scheduler hint in the server metadata
so it can be retrieved via server metadata later.
* In the case of hints the end user can decide not to include a hint. On the
other hand the end user cannot create a new flavor (by default policy) to
avoid passing a flavor with an extra spec - the deployer controls the
flavors.
.. _2.47 microversion: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id42
Discoverability
---------------
When it comes to discoverability, by the default
``os_compute_api:os-flavor-extra-specs:index`` policy rule, flavor extra
specs are more "discoverable" by the end user since they can list them for a
flavor. However, one should not expect an average end user to understand what
different extra specs mean as they are just a key/value pair. There is some
documentation for some "standard" extra specs though [3]_. However, that is
not an exhaustive list and it does not include anything that different
deployments would define for things like linking a flavor to a set of
host aggregates, for example, when creating flavors
for baremetal instances, or what the chosen
:doc:`hypervisor driver </admin/configuration/hypervisors>` might support for
flavor extra specs.
Scheduler hints are less discoverable from an end user perspective than
extra specs. There are some standard hints defined in the API request
schema [4]_. However:
1. Those hints are tied to scheduler filters and the scheduler filters are
configurable per deployment, so for example the ``JsonFilter`` might not be
enabled (it is not enabled by default), so the ``query`` hint would not do
anything.
2. Scheduler hints are not restricted to just what is in that schema in the
upstream nova code because of the ``additionalProperties: True`` entry in
the schema. This allows deployments to define their own hints outside of
that API request schema for their own
:ref:`custom scheduler filters <custom-scheduler-filters>` which are not
part of the upstream nova code.
Interoperability
----------------
The only way an end user can really use scheduler hints is based
on documentation (or GUIs/SDKs) that a specific cloud deployment provides for
their setup. So if **CloudA** defines a custom scheduler filter X and a hint
for that filter in their documentation, an end user application can only run
with that hint on that cloud and expect it to work as documented. If the user
moves their application to **CloudB** which does not have that scheduler
filter or hint, they will get different behavior.
So obviously both flavor extra specs and scheduler hints are not interoperable.
Which to use?
-------------
When it comes to defining a custom scheduler filter, you could use a hint or
an extra spec. If you need a flavor extra spec anyway for some behavior in the
hypervisor when creating the guest, or to be able to retrieve the original
flavor extra specs used to create a guest later, then you might as well just
use the extra spec. If you do not need that, then a scheduler hint may be an
obvious choice, from an end user perspective, for exposing a certain scheduling
behavior but it must be well documented and the end user should realize that
hint might not be available in other clouds, and they do not have a good way
of finding that out either. Long-term, flavor extra specs are likely to be
more standardized than hints so ultimately extra specs are the recommended
choice.
Footnotes
---------
.. [1] https://github.com/openstack/nova/blob/fbe6f77bc/nova/objects/image_meta.py#L225
.. [2] https://review.openstack.org/#/c/440580/
.. [3] https://docs.openstack.org/nova/latest/user/flavors.html#extra-specs
.. [4] https://github.com/openstack/nova/blob/fbe6f77bc/nova/api/openstack/compute/schemas/scheduler_hints.py

2
doc/source/user/filter-scheduler.rst
View File

@ -335,6 +335,8 @@ resource calculating filters like RamFilter, CoreFilter.
In medium/large environments having AvailabilityZoneFilter before any
capability or resource calculating filters can be useful.
.. _custom-scheduler-filters:
Writing Your Own Filter
-----------------------