12 KiB
Count quota usage from placement
https://blueprints.launchpad.net/nova/+spec/count-quota-usage-from-placement
In Pike, we re-architected the quota system to count actual resource usage instead of using reservations and tracking quota usages in a separate database table. We're counting resources like instances, CPU, and RAM by querying each cell database and aggregating the results per project and per user. This approach is problematic in the context of "handling of a down cell. If a cell becomes unavailable, resources in its database cannot be counted and will not be included in resource usage until the cell returns. Cells could become unavailable if an operator is performing maintenance on a cell or if a cell database is experiencing problems and we cannot connect to it.
We can make resource usage counting for quotas resilient to temporary cell outages by querying placement and the API database for resource usage instead of reading separate cell databases.
Problem description
When we count quota resource usage for CPU and RAM, we do so by reading separate cell databases and aggregating the results. CPU and RAM amounts per instance are derived from the flavor and are stored in the database as columns in the instances table. So, each time we check quota usage against limits, we query a count of instance ID and a sum of CPU and RAM per cell database and aggregate them to calculate the resource usage.
This approach is sensitive to temporary cell outages which may occur during operator maintenance or if a cell database is experiencing issues and we cannot connect to it. While a cell is unavailable, we cannot count resource usage residing in that cell database and things would behave as though more quota is available than should be. That is, if someone has used all of their quota and part of it is in cell A and cell A goes offline temporarily, that person will suddenly be able to allocate more resources than their limit (assuming cell A returns, the person will have more resources allocated than their allowed quota).
We could take a different approach by querying the placement API and the API database to get resource usage counts. Since placement is managing resource allocations, it has the information we need to count resource usage for CPU and RAM quotas. By querying placement and the API database, we can avoid reading separate cell databases for resource usage.
Use Cases
Counting quota resource usage from placement would make quota behavior consistent in the event of temporary cell database disruptions. It would be easier for Operators to take cells offline if needed for maintenance without concern about the possibility of quota limits being exceeded during the maintenance. It could spare Operators the trouble of potentially having to fix cases where quota has been exceeded during maintenance or if a cell database connection could not be established.
Proposed change
We will add a new method for counting instances that queries the
instance_mappings
table in the API database and make a
separate limit check for number of instances.
The new method will contain:
- One query to the API database to get resource usage for instances.
We can get the number of instances for a project and user if we add a
new column
user_id
to thenova_api.instance_mappings
table. We already have aproject_id
column on the table. This will allow us to count instance mappings for a project and a user to represent the instance count.
We will rename the _instances_cores_ram_count
method to
_cores_ram_count
that counts cores and ram from the cell
databases and is only used if
[workarounds]disable_quota_usage_from_placement
is
True.
Because there is not yet an ability to partition allocations (or
perhaps, resource providers from which allocations could derive a
partition) in placement, in order to support deployments where multiple
Nova deployments share the same placement service, like possibly in an
Edge scenario, we can add a
[workarounds]disable_quota_usage_from_placement
which
defaults to False. If True, we use the legacy quota counting method for
instances, cores, and ram. If False, we use a quota counting method that
calls placement. This is a minimal way to keep "legacy" quota counting
available for the scenario of multiple Nova deployments sharing one
placement service. The config option will simply control which counting
method will be called by the pluggable quota system. For example
(pseudo-code):
- ::
-
- if CONF.workarounds.disable_quota_usage_from_placement:
-
CountableResource('cores', _cores_ram_count, 'cores') CountableResource('ram', _cores_ram_count, 'ram')
- else:
-
CountableResource('cores', _cores_ram_count_placement, 'cores') CountableResource('ram', _cores_ram_count_placement, 'ram')
We will add a new method for counting cores and ram from placement
that is used when
[workarounds]disable_quota_usage_from_placement
is False.
This method could be called _cores_ram_count_placement
.
The new method will contain:
One call to placement to get resource usage for CPU and RAM. We can get CPU and RAM usage for a project and user by querying the
/usages
resource:GET /usages?project_id=<project id>&user_id=<user id>
Alternatives
One alternative is to hold off on counting any quota usage from placement until placement has allocation partitioning support. The problem with that is in the meantime, the only solution we have for handling of down cells is to implement the policy-driven behavior where an operator has to choose between failing server create requests when a project has instances in a down cell or allowing server create requests to potentionally exceed quota limits.
Another alternative which has been discussed is, to use placement
aggregates to surround each entire Nova deployment and use that as a
means to partition placement usages. We would need to add a
aggregate=
query parameter to the placement /usages API in
this case. This approach would also require some work by either Nova or
the operator to keep the placement aggregate updated.
Data model impact
A nova_api database schema change will be required for adding the
user_id
column of type String to the
nova_api.instance_mappings
table.
REST API impact
None.
Security impact
None.
Notifications impact
None.
Other end user impact
End users will see consistent quota behavior even when cell databases are unavailable.
Performance Impact
The change involves making external REST API calls to placement instead of doing a parallel scatter-gather to all cells. It might be slower to make the external REST API calls if all cells are fast responding. It might be faster to make external REST API calls if any cells are slower responding.
Other deployer impact
None.
Developer impact
None.
Upgrade impact
The addition of the user_id
column to the
nova_api.instance_mappings
table will require a data
migration of all existing instance mappings to populate the
user_id
field. The migration routine would look for
mappings where user_id
is None and query cells by
corresponding project_id
in the mapping. The query could
filter on instance UUIDs, finding the user_id
values to
populate in the mappings. This would implement the batched
nova-manage db online_data_migration
way of doing the
migration.
We will also heal/populate an instance mapping on-the-fly when it is
accessed during a server GET request. This would provide some data
migration in the situation where an upgrade has not run
nova-manage db online_data_migration
yet.
In order to handle a live in-progress upgrade, we will need to be
able to fall back on the legacy counting method for instances, cores,
and ram if nova_api.instance_mappings
don't yet have
user_id
populated (if the operator has not yet run the data
migration). We will need a way to detect that the migration has not yet
been run in order to fall back on the legacy counting method. We could
have a check such as
if count(InstanceMapping.id) where project_id=<project id> and user_id=None > 0
,
then fall back on the legacy counting method to query cell databases. We
should cache the results of the each migration done success check by
project_id
so we avoid needlessly checking a
project_id
that has already been migrated every time quota
is checked.
We will populate the user_id
field even for instance
mappings that are queued_for_delete=True
because we will be
filtering on queued_for_delete=False
during the instance
count based on instance mappings.
The data migrations and fallback to the legacy counting method will
be temporary for Stein, to be dropped in T with a blocker migration.
That is, you cannot pass nova-manage api_db sync
if there
are any instance mappings with user_id=None
to force the
batched migration using nova-manage
.
Implementation
Assignee(s)
- Primary assignee:
-
melwitt
- Other contributors:
-
None
Work Items
- Add a new column
user_id
to thenova_api.instance_mappings
table. - Implement an online data migration to populate the
user_id
field. - Update the
_server_group_count_members_by_user
quota counting method to use only thenova_api.instance_mappings
table instead of querying cell databases. - Add a config option
[workarounds]disable_quota_usage_from_placement
that defaults to False. This will be able to be deprecated when partitioning of resource providers or allocations is available in placement. - Add a new method to count instances with a count of
nova_api.instance_mappings
filtering byproject_id=<project_id>
anduser_id=<user_id>
andqueued_for_delete=False
. - Add a new count method that queries the placement API for CPU and RAM usage. In the new count method, add a check for whether the online data migration has been run yet and if not, fall back on the legacy count method.
- Rename the
_instances_cores_ram_count
method to_cores_ram_count
and let it count only cores and ram in the legacy way, for use if[workarounds]disable_quota_usage_from_placement
is set to True. - Adjust the nova-next or nova-live-migration CI job to run with
[workarounds]disable_quota_usage_from_placement=True
.
Dependencies
None.
Testing
Unit tests and functional tests will be included to test the new
functionality. We will also adjust one CI job (nova-next or
nova-live-migration) to run with
[workarounds]disable_quota_usage_from_placement=True
to
make sure we have integration test coverage of that path.
Documentation Impact
The documentation of Cells v2 caveats will be updated to update the paragraph about the inability to correctly calculate quota usage when one or more cells are unreachable. We will document that beginning in Stein, there are new deployment options.
References
This builds upon the work done in Pike to re-architect quotas to count resources.
History
Release Name | Description |
---|---|
Stein | Introduced |