Merge "Support initial allocation ratios"
This commit is contained in:
Zuul
Gerrit Code Review
commit
4d2fbee4f9
|
|
@ -0,0 +1,313 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
======================================
|
||||
Default allocation ratio configuration
|
||||
======================================
|
||||
|
||||
https://blueprints.launchpad.net/nova/+spec/initial-allocation-ratios
|
||||
|
||||
Provide separate CONF options for specifying the initial allocation
|
||||
ratio for compute nodes. Change the default values for
|
||||
CONF.xxx_allocation_ratio options to None and change the behaviour of
|
||||
the resource tracker to only override allocation ratios for *existing*
|
||||
compute nodes if the CONF.xxx_allocation_ratio value is not None.
|
||||
|
||||
The primary goal of this feature is to support both the API and config way to
|
||||
pass allocation ratios.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Manually set placement allocation ratios are overwritten
|
||||
--------------------------------------------------------------------
|
||||
|
||||
There is currently no way for an admin to set the allocation ratio on an
|
||||
individual compute node resource provider's inventory record in the placement
|
||||
API without the resource tracker eventually overwriting that value the next
|
||||
time it runs the ``update_available_resources`` periodic task on the
|
||||
``nova-compute`` service.
|
||||
|
||||
The saga of the allocation ratio values on the compute host
|
||||
-----------------------------------------------------------
|
||||
|
||||
The process by which nova determines the allocation ratio for CPU, RAM and disk
|
||||
resources on a hypervisor is confusing and `error`_ `prone`_. The
|
||||
``compute_nodes`` table in the nova cell DB contains three fields representing
|
||||
the allocation ratio for CPU, RAM and disk resources on that hypervisor. These
|
||||
fields are populated using different default values depending on the version of
|
||||
nova running on the ``nova-compute`` service.
|
||||
|
||||
.. _error: https://bugs.launchpad.net/nova/+bug/1742747
|
||||
.. _prone: https://bugs.launchpad.net/nova/+bug/1789654
|
||||
|
||||
Upon starting up, the resource tracker in the ``nova-compute`` service worker
|
||||
`checks`_ to see if a record exists in the ``compute_nodes`` table of the nova
|
||||
cell DB for itself. If it does not find one, the resource tracker `creates`_ a
|
||||
record in the table, `setting`_ the associated allocation ratio values in the
|
||||
``compute_nodes`` table to the value it finds in the ``cpu_allocation_ratio``,
|
||||
``ram_allocation_ratio`` and ``disk_allocation_ratio`` nova.conf configuration
|
||||
options but only if the config option value is not equal to 0.0.
|
||||
|
||||
.. _checks: https://github.com/openstack/nova/blob/852de1e/nova/compute/resource_tracker.py#L566
|
||||
.. _creates: https://github.com/openstack/nova/blob/852de1e/nova/compute/resource_tracker.py#L577-L590
|
||||
.. _setting: https://github.com/openstack/nova/blob/6a68f9140/nova/compute/resource_tracker.py#L621-L645
|
||||
|
||||
The default values of the ``cpu_allocation_ratio``, ``ram_allocation_ratio``
|
||||
and ``disk_allocation_ratio`` CONF options is `currently set`_ to ``0.0``.
|
||||
|
||||
.. _currently set: https://github.com/openstack/nova/blob/852de1e/nova/conf/compute.py#L400
|
||||
|
||||
The resource tracker saves these default ``0.0`` values to the
|
||||
``compute_nodes`` table when the resource tracker calls ``save()`` on the
|
||||
compute node object. However, there is `code`_ in the
|
||||
``ComputeNode._from_db_obj`` that, upon **reading** the record back from the
|
||||
database on first save, changes the values from ``0.0`` to ``16.0``, ``1.5`` or
|
||||
``1.0``.
|
||||
|
||||
.. _code: https://github.com/openstack/nova/blob/852de1e/nova/objects/compute_node.py#L177-L207
|
||||
|
||||
The ``ComputeNode`` object that was ``save()``'d by the resource tracker has
|
||||
these new values for some period of time while the record in the
|
||||
``compute_nodes`` table continues to have the wrong ``0.0`` values. When the
|
||||
resource tracker runs its ``update_available_resource()`` next perioidic task,
|
||||
the new ``16.0``/``1.5``/``1.0`` values are then saved to the compute nodes
|
||||
table.
|
||||
|
||||
There is a `fix`_ for `bug/1789654`_, which is to not persist
|
||||
zero allocation ratios in ResourceTracker to avoid initializing placement
|
||||
allocation_ratio with 0.0 (due to the allocation ratio of 0.0 being multiplied
|
||||
by the total amount in inventory, leading to 0 resources shown on the system).
|
||||
|
||||
.. _fix: https://review.openstack.org/#/c/598365/
|
||||
.. _bug/1789654: https://bugs.launchpad.net/nova/+bug/1789654
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
An administrator would like to set allocation ratios for individual resources
|
||||
on a compute node via the placement API *without that value being overwritten*
|
||||
by the compute node's resource tracker.
|
||||
|
||||
An administrator chooses to only use the configuration file to set allocation
|
||||
ratio overrides on their compute nodes and does not want to use the placement
|
||||
API to set these ratios.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
First, we propose to change the default option values of existing
|
||||
``CONF.cpu_allocation_ratio``, ``CONF.ram_allocation_ratio`` and
|
||||
``CONF.disk_allocation_ratio`` options relating to allocation ratios to
|
||||
``None`` from the existing default values of ``0.0``. The reason we change
|
||||
it is that this value will be change from ``0.0`` to ``16.0``, ``1.5`` or
|
||||
``1.0`` later, which is weird and confusing.
|
||||
|
||||
We will also change the resource tracker to **only** overwrite the compute
|
||||
node's allocation ratios to the value of the ``cpu_allocation_ratio``,
|
||||
``ram_allocation_ratio`` and ``disk_allocation_ratio`` CONF options **if the
|
||||
value of these options is NOT ``None``**.
|
||||
|
||||
In other words, if any of these CONF options is set to something *other than*
|
||||
``None``, then the CONF option should be considered the complete override value
|
||||
for that resource class' allocation ratio. Even if an admin manually adjusts
|
||||
the allocation ratio of the resource class in the placement API, the next time
|
||||
the ``update_available_resource()`` periodic task runs, it will be overwritten
|
||||
to the value of the CONF option.
|
||||
|
||||
Second, we propose to add 3 new nova.conf configuration options:
|
||||
|
||||
* ``initial_cpu_allocation_ratio``
|
||||
* ``initial_ram_allocation_ratio``
|
||||
* ``initial_disk_allocation_ratio``
|
||||
|
||||
That will used to determine how to set the *initial* allocation ratio of
|
||||
``VCPU``, ``MEMORY_MB`` and ``DISK_GB`` resource classes when a compute worker
|
||||
first starts up and creates its compute node record in the nova cell DB and
|
||||
corresponding inventory records in the placement service. The value of these
|
||||
new configuration options will only be used if the compute service's resource
|
||||
tracker is not able to find a record in the placement service for the compute
|
||||
node the resource tracker is managing.
|
||||
|
||||
The default value of each of these CONF options shall be ``16.0``, ``1.5``, and
|
||||
``1.0`` respectively. This is to match the default values for the original
|
||||
allocation ratio CONF options before they were set to ``0.0``.
|
||||
|
||||
These new ``initial_xxx_allocation_ratio`` CONF options shall **ONLY** be used
|
||||
if the resource tracker detects no existing record in the ``compute_nodes``
|
||||
nova cell DB for that hypervisor.
|
||||
|
||||
Finally, we will need also add an online data migration and continue to read
|
||||
the ``xxx_allocation_ratio`` or ``initial_xxx_allocation_ratio`` config on
|
||||
read from the DB if the values are ``0.0`` or ``None``. If it's an existing
|
||||
record with 0.0 values, we'd want to do what the compute does, which is use
|
||||
the configure ``xxx_allocation_ratio`` config if it's not None, and fallback
|
||||
to using the ``initial_xxx_allocation_ratio`` otherwise.
|
||||
|
||||
And add an online data migration that updates all compute_nodes
|
||||
table records that have ``0.0`` or ``None`` allocation ratios. Then we drop
|
||||
that at some point with a blocker migration and remove the code in the
|
||||
``nova.objects.ComputeNode._from_db_obj`` that adjusts allocation ratios.
|
||||
|
||||
We propose to add a nova-status upgrade check to iterate the cells looking
|
||||
for compute_nodes records with ``0.0`` or ``None`` allocation ratios and signal
|
||||
that as a warning that you haven't done the online data migration. We could
|
||||
also check the conf options to see if they are explicitly set to 0.0 and if
|
||||
so, we should fail the status check.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
None
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Upgrade impact
|
||||
--------------
|
||||
|
||||
We need an online data migrations for any compute_nodes with existing ``0.0``
|
||||
and ``None`` allocation ratio. If it's an existing record with 0.0 values, we
|
||||
will replace it with the configure ``xxx_allocation_ratio`` config if it's not
|
||||
None, and fallback to using the ``initial_xxx_allocation_ratio`` otherwise.
|
||||
|
||||
.. note:: Migrating 0.0 allocation ratios from existing ``compute_nodes`` table
|
||||
records is necessary because the ComputeNode object based on those table
|
||||
records is what gets used in the scheduler [1]_, specifically the
|
||||
``NUMATopologyFilter`` and ``CPUWeigher`` (the ``CoreFilter``,
|
||||
``DiskFilter`` and ``RamFilter`` also use them but those filters are
|
||||
deprecated for removal so they are not a concern here).
|
||||
|
||||
And clearly in order to take advantage of the ability to manually set
|
||||
allocation ratios on a compute node, that hypervisor would need to be upgraded.
|
||||
No impact to old compute hosts.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
yikun
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Change the default values for ``CONF.xxx_allocation_ratio`` options to
|
||||
``None``.
|
||||
* Modify resource tracker to only set allocation ratios on the compute node
|
||||
object when the CONF options are non- ``None``
|
||||
* Add new ``initial_xxx_allocation_ratio`` CONF options and modify resource
|
||||
tracker's initial compute node creation to use these values
|
||||
* Remove code in the ``ComputeNode._from_db_obj()`` that changes allocation
|
||||
ratio values
|
||||
* Add a db online migration to process all compute_nodes with existing ``0.0``
|
||||
and ``None`` allocation ratio.
|
||||
* Add a nova-status upgrade check for ``0.0`` or ``None`` allocation ratio.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
No extraordinary testing outside normal unit and functional testing
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
A release note explaining the use of the new ``initial_xxx_allocation_ratio``
|
||||
CONF options should be created along with a more detailed doc in the admin
|
||||
guide explaining the following primary scenarios:
|
||||
|
||||
* When the deployer wants to **ALWAYS** set an override value for a resource on
|
||||
a compute node. This is where the deployer would ensure that the
|
||||
``cpu_allocation_ratio``, ``ram_allocation_ratio`` and
|
||||
``disk_allocation_ratio`` CONF options were set to a non- ``None`` value.
|
||||
* When the deployer wants to set an **INITIAL** value for a compute node's
|
||||
allocation ratio but wants to allow an admin to adjust this afterwards
|
||||
without making any CONF file changes. This scenario uses the new
|
||||
``initial_xxx_allocation_ratios`` for the initial ratio values and then shows
|
||||
the deployer using the osc placement commands to manually set an allocation
|
||||
ratio for a resource class on a resource provider.
|
||||
* When the deployer wants to **ALWAYS** use the placement API to set allocation
|
||||
ratios, then the deployer should ensure that ``CONF.xxx_allocation_ratio``
|
||||
options are all set to ``None`` and the deployer should issue Placement
|
||||
REST API calls to
|
||||
``PUT /resource_providers/{uuid}/inventories/{resource_class}`` [2]_ or
|
||||
``PUT /resource_providers/{uuid}/inventories`` [3]_ to set the allocation
|
||||
ratios of their resources as needed (or use the related ``osc-placement``
|
||||
plugin commands [4]_).
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] https://github.com/openstack/nova/blob/a534ccc5a7/nova/scheduler/host_manager.py#L255
|
||||
.. [2] https://developer.openstack.org/api-ref/placement/#update-resource-provider-inventory
|
||||
.. [3] https://developer.openstack.org/api-ref/placement/#update-resource-provider-inventories
|
||||
.. [4] https://docs.openstack.org/osc-placement/latest/
|
||||
|
||||
Nova Stein PTG discussion:
|
||||
|
||||
* https://etherpad.openstack.org/p/nova-ptg-stein
|
||||
|
||||
Bugs:
|
||||
|
||||
* https://bugs.launchpad.net/nova/+bug/1742747
|
||||
* https://bugs.launchpad.net/nova/+bug/1729621
|
||||
* https://bugs.launchpad.net/nova/+bug/1739349
|
||||
* https://bugs.launchpad.net/nova/+bug/1789654
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - Stein
|
||||
- Proposed
|
||||
Loading…
Reference in New Issue