10 KiB
Complex Anti-Affinity Policies
https://blueprints.launchpad.net/nova/+spec/complex-anti-affinity-policies
This blueprint proposes to enable users to define the rules on policy
to meet more advanced policy requirement, also proposes to implement an
example that adding the max_server_per_host
rule for
anti-affinity
policy.
Problem description
Nova supports filtering and weighting to make informed decisions on where a new instance should be created, and the ServerGroupAntiAffinityFilter implements anti-affinity for a server group based on this scheduler mechanism.
Users set the policy for the specific server group to enable the anti-affinity for the server group. This meets the most basic requirement of server group affinity, but it isn't enough for the more complex anti-affinity requirement.
For example, users want to enable anti-affinity policy with a limit other than 1, which is the static limit today. By doing this, the number of VMs in the same anti-affinity group per host can be limited by users. To achieve both the anti-affinity and resource utilization requirement, it's a very useful ability to users, especially, when the users don't have enough hosts in their cloud but still want some level of high reliability of their applications. In addition, we also can't use the soft-anti-affinity policy since that is implemented based on weights rather than host filtering.
So, the scheduler needs to provide some mechanism to enable users to define the rules on policy to meet more advanced policy requirement.
Use Cases
As a NFV user, in consideration of reliability and resource utilization, I want a mechanism to add the limit on the instances max number per host in the same anti-affinity group.
Proposed change
This spec proposes to:
Add a generic "rules" field which is a dict, can be applied to the policies.
Now, only
max_server_per_host
foranti-affinity
policy would be supported, the example usage as below:"max_server_per_host" rule for
anti-affinity
policy means that add the max limit on the number of VMs in a group on a given host. For example, if the user have a group of 6 instances and 2 hosts with aanti-affinity
policy, it will be rejected in current anti-affinity policy, then the user can specify{'max_server_per_host': 3}
rule for this group, and it means 6 instances get spread across 2 hosts and each host has 3 servers.Create a new API microversion to support these changes:
- Support passing a optional argument
rules
to the create instance group API. Now, we only support the "max_server_per_host" rule foranti-affinity
policy as a optional entry with an int value >= 1 which are validated with the json schema in API. - The responses of
POST /os-server-groups
,GET /os-server-groups
andGET /os-server-groups/{server_group_id}
also need to be changed to the new policy format. - Remove the empty and unused
metadata
field in the response ofPOST /os-server-groups
,GET /os-server-groups
andGET /os-server-groups/{server_group_id}
.
- Support passing a optional argument
Change the
ServerGroupAntiAffinityFilter
to adapt to complex policy model.The filter will get the max_server_per_host limit from policy rules, and compare it against the number of servers within the same group on a given host. If the filter finds the number is not satisfied with the limit, it will filter out this host.
The default
max_server_per_host
for the anti-affinity filter is 1 for backward compatibility.
Alternatives
An alternative would be to have a setting for the max spread before the scheduler would consider doubling up. The user can define how much redundancy they want at a minimum regardless of how many instances are in the group.
Data model impact
The
Text
columnrules
will be added to theinstance_group_policy
database table. A database schema migration will also be added in order to add the column.The format of the
rules
is a dict containing multiple key/value pairs like:{'max_server_per_host': 3, `other_key`: `other_value`}
Add a new
InstanceGroupPolicy
versioned object including thepolicy
,rules
andgroup_id
fields.Add a new
policy
field to theInstanceGroup
object.The InstanceGroup.policy would be an instance of
InstanceGroupPolicy
, and the originalpolicies
field would be deprecated.
REST API impact
Following changes will be introduced in a new API microversion.
POST /os-server-groups
Support passing
rules
to the create instance group API and change the schema of creating server group to avoid creating a server with no policies.Example Create Server Group JSON request:
{ "server_group": { "name": "test", "policy": { "name": "anti-affinity", "rules": { "max_server_per_host": 3 } } } }
The new JSON schema for the
POST /os-server-groups
as below:create = { 'type': 'object', 'properties': { 'server_group': { 'type': 'object', 'properties': { 'name': parameter_types.name, 'policy': { 'oneOf': [ { 'type': 'object', 'properties': { 'name': { 'type': 'string', 'enum': ['anti-affinity'], }, 'rules': { 'type': 'object', 'properties': { 'max_server_per_host': parameter_types.positive_integer, }, 'additionalProperties': False } }, 'required': ['name'], 'additionalProperties': False }, { 'type': 'object', 'properties': { 'name': { 'type': 'string', 'enum': ['affinity', 'soft-anti-affinity', 'soft-affinity'], }, }, 'required': ['name'], 'additionalProperties': False }] } }, 'required': ['name', 'policy'], 'additionalProperties': False, }}, 'required': ['server_group'], 'additionalProperties': False, }
Change the response to the new policy format and remove the empty and unused
metadata
field:{ "server_group": { "id": "5bbcc3c4-1da2-4437-a48a-66f15b1b13f9", "name": "test", "policy": { "name": "anti-affinity", "rules": { "max_server_per_host": 3 } } "members": [] } }
Note that: if the user creates a group without specifying the policy rules, the value of
rules
key is{}
.GET /os-server-groups
Change the response to the new policy format and remove the empty and unused
metadata
field:{ "server_groups": [ { "id": "616fb98f-46ca-475e-917e-2563e5a8cd19", "name": "test", "policy": { "name": "anti-affinity", "rules": { "max_server_per_host": 3 } }, "members": [], "project_id": "6f70656e737461636b20342065766572", "user_id": "fake" } ] }
GET /os-server-groups/{server_group_id}
Change the response to the new policy format and remove the empty and unused
metadata
field:{ "server_group": { "id": "5bbcc3c4-1da2-4437-a48a-66f15b1b13f9", "name": "test", "policy": { "name": "anti-affinity", "rules": { "max_server_per_host": 3 } }, "members": [] } }
Security impact
None
Notifications impact
The server_group.create
,
server_group.delete
and
server_group.add_member
versioned notifications will be
updated to include the new policy
field instead of the old
policies
field.
Other end user impact
- python-novaclient will be modified to add this new
rule
param to the nova server-group-create shell command. - python-openstackclient will be modified to add this new
rule
param to the openstack server group create shell command.
Performance Impact
None
Other deployer impact
None
Developer impact
None
Upgrade impact
None
Implementation
Assignee(s)
- Primary assignee:
-
Yikun Jiang
Work Items
- Add the
rules
attribute to theInstanceGroupPolicy
data model. - Create a new API microversion to support passing
rules
to the create instance group API. - Modify the Nova client to handle the new microversion.
- Change the
ServerGroupAntiAffinityFilter
to adapt to new policy model. - Change the
_validate_instance_group_policy
in the after resource tracker claim to adapt to new policy model
Dependencies
None
Testing
Would need new in-tree functional and unit tests.
Documentation Impact
Docs needed for new API microversion and usage.
References
History
Release Name | Description |
---|---|
Rocky | Proposed |