From 9187328f1d419680565141ecf795530afddebf2c Mon Sep 17 00:00:00 2001 From: melanie witt Date: Tue, 3 Oct 2017 01:50:22 +0000 Subject: [PATCH] Propose counting quota usage from placement and API database In Pike, we re-architected the quota system to count actual resource usage instead of using reservations and tracking quota usages in a separate database table. We're counting resources like instances, CPU, and RAM by querying each cell database and aggregating the results per project and per user. This approach is susceptible to undesirable behavior if a cell becomes unavailable. If a cell becomes unavailable, resources in its database cannot be counted and will not be included in resource usage until the cell returns. Cells could become unavailable if an operator is performing maintenance on a cell or if a cell database is experiencing problems and we cannot connect to it. We can make resource usage counting for quotas resilient to temporary cell outages by querying placement and the API database for resource usage instead of reading separate cell databases. Spec for blueprint count-quota-usage-from-placement Change-Id: Ib86a7994d7a577b8e6a2dd32f4fcf12b7f788af8 --- .../count-quota-usage-from-placement.rst | 291 ++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100644 specs/stein/approved/count-quota-usage-from-placement.rst diff --git a/specs/stein/approved/count-quota-usage-from-placement.rst b/specs/stein/approved/count-quota-usage-from-placement.rst new file mode 100644 index 000000000..e06e94229 --- /dev/null +++ b/specs/stein/approved/count-quota-usage-from-placement.rst @@ -0,0 +1,291 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +================================ +Count quota usage from placement +================================ + +https://blueprints.launchpad.net/nova/+spec/count-quota-usage-from-placement + +In Pike, we re-architected the quota system to count actual resource usage +instead of using reservations and tracking quota usages in a separate database +table. We're counting resources like instances, CPU, and RAM by querying each +cell database and aggregating the results per project and per user. This +approach is problematic in the context of "handling of a down cell. If a cell +becomes unavailable, resources in its database cannot be counted and will not +be included in resource usage until the cell returns. Cells could become +unavailable if an operator is performing maintenance on a cell or if a cell +database is experiencing problems and we cannot connect to it. + +We can make resource usage counting for quotas resilient to temporary cell +outages by querying placement and the API database for resource usage instead +of reading separate cell databases. + + +Problem description +=================== + +When we count quota resource usage for CPU and RAM, we do so by reading +separate cell databases and aggregating the results. CPU and RAM amounts per +instance are derived from the flavor and are stored in the database as columns +in the instances table. So, each time we check quota usage against limits, we +query a count of instance ID and a sum of CPU and RAM per cell database and +aggregate them to calculate the resource usage. + +This approach is sensitive to temporary cell outages which may occur during +operator maintenance or if a cell database is experiencing issues and we cannot +connect to it. While a cell is unavailable, we cannot count resource usage +residing in that cell database and things would behave as though more quota is +available than should be. That is, if someone has used all of their quota and +part of it is in cell A and cell A goes offline temporarily, that person will +suddenly be able to allocate more resources than their limit (assuming cell A +returns, the person will have more resources allocated than their allowed +quota). + +We could take a different approach by querying the placement API and the API +database to get resource usage counts. Since placement is managing resource +allocations, it has the information we need to count resource usage for CPU and +RAM quotas. By querying placement and the API database, we can avoid reading +separate cell databases for resource usage. + +Use Cases +--------- + +Counting quota resource usage from placement would make quota behavior +consistent in the event of temporary cell database disruptions. It would be +easier for Operators to take cells offline if needed for maintenance without +concern about the possibility of quota limits being exceeded during the +maintenance. It could spare Operators the trouble of potentially having to fix +cases where quota has been exceeded during maintenance or if a cell database +connection could not be established. + +Proposed change +=============== + +We will add a new method for counting instances that queries the +``instance_mappings`` table in the API database and make a separate limit check +for number of instances. + +The new method will contain: + +* One query to the API database to get resource usage for instances. We can get + the number of instances for a project and user if we add a new column + ``user_id`` to the ``nova_api.instance_mappings`` table. We already have a + ``project_id`` column on the table. This will allow us to count instance + mappings for a project and a user to represent the instance count. + +We will rename the ``_instances_cores_ram_count`` method to +``_cores_ram_count`` that counts cores and ram from the cell databases and +is only used if ``[workarounds]disable_quota_usage_from_placement`` is True. + +Because there is not yet an ability to partition allocations (or perhaps, +resource providers from which allocations could derive a partition) in +placement, in order to support deployments where multiple Nova deployments +share the same placement service, like possibly in an Edge scenario, we can add +a ``[workarounds]disable_quota_usage_from_placement`` which defaults to False. +If True, we use the legacy quota counting method for instances, cores, and +ram. If False, we use a quota counting method that calls placement. This is a +minimal way to keep "legacy" quota counting available for the scenario of +multiple Nova deployments sharing one placement service. The config option will +simply control which counting method will be called by the pluggable quota +system. For example (pseudo-code): + +:: + if CONF.workarounds.disable_quota_usage_from_placement: + CountableResource('cores', _cores_ram_count, 'cores') + CountableResource('ram', _cores_ram_count, 'ram') + else: + CountableResource('cores', _cores_ram_count_placement, 'cores') + CountableResource('ram', _cores_ram_count_placement, 'ram') + +We will add a new method for counting cores and ram from placement that is used +when ``[workarounds]disable_quota_usage_from_placement`` is False. This +method could be called ``_cores_ram_count_placement``. + +The new method will contain: + +* One call to placement to get resource usage for CPU and RAM. We can get CPU + and RAM usage for a project and user by querying the ``/usages`` resource:: + + GET /usages?project_id=&user_id= + +Alternatives +------------ + +One alternative is to hold off on counting any quota usage from placement +until placement has allocation partitioning support. The problem with that is +in the meantime, the only solution we have for handling of down cells is to +implement the `policy-driven behavior`_ where an operator has to choose between +failing server create requests when a project has instances in a down cell or +allowing server create requests to potentionally exceed quota limits. + +Another alternative which has been discussed is, to use placement aggregates to +surround each entire Nova deployment and use that as a means to partition +placement usages. We would need to add a ``aggregate=`` query parameter to the +placement /usages API in this case. This approach would also require some work +by either Nova or the operator to keep the placement aggregate updated. + +.. _policy-driven behavior: https://review.openstack.org/614783 + +Data model impact +----------------- + +A nova_api database schema change will be required for adding the ``user_id`` +column of type String to the ``nova_api.instance_mappings`` table. + +REST API impact +--------------- + +None. + +Security impact +--------------- + +None. + +Notifications impact +-------------------- + +None. + +Other end user impact +--------------------- + +End users will see consistent quota behavior even when cell databases are +unavailable. + +Performance Impact +------------------ + +The change involves making external REST API calls to placement instead of +doing a parallel scatter-gather to all cells. It might be slower to make the +external REST API calls if all cells are fast responding. It might be faster to +make external REST API calls if any cells are slower responding. + +Other deployer impact +--------------------- + +None. + +Developer impact +---------------- + +None. + +Upgrade impact +-------------- + +The addition of the ``user_id`` column to the ``nova_api.instance_mappings`` +table will require a data migration of all existing instance mappings to +populate the ``user_id`` field. The migration routine would look for mappings +where ``user_id`` is None and query cells by corresponding ``project_id`` in +the mapping. The query could filter on instance UUIDs, finding the ``user_id`` +values to populate in the mappings. This would implement the batched +``nova-manage db online_data_migration`` way of doing the migration. + +We will also heal/populate an instance mapping on-the-fly when it is accessed +during a server GET request. This would provide some data migration in the +situation where an upgrade has not run ``nova-manage db online_data_migration`` +yet. + +In order to handle a live in-progress upgrade, we will need to be able to fall +back on the legacy counting method for instances, cores, and ram if +``nova_api.instance_mappings`` don't yet have ``user_id`` populated (if the +operator has not yet run the data migration). We will need a way to detect that +the migration has not yet been run in order to fall back on the legacy counting +method. We could have a check such as ``if count(InstanceMapping.id) where +project_id= and user_id=None > 0``, then fall back on the legacy +counting method to query cell databases. We should cache the results of the +each migration done success check by ``project_id`` so we avoid needlessly +checking a ``project_id`` that has already been migrated every time quota is +checked. + +We will populate the ``user_id`` field even for instance mappings that are +``queued_for_delete=True`` because we will be filtering on +``queued_for_delete=False`` during the instance count based on instance +mappings. + +The data migrations and fallback to the legacy counting method will be +temporary for Stein, to be dropped in T with a blocker migration. That is, you +cannot pass ``nova-manage api_db sync`` if there are any instance mappings with +``user_id=None`` to force the batched migration using ``nova-manage``. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + melwitt + +Other contributors: + None + +Work Items +---------- + +* Add a new column ``user_id`` to the ``nova_api.instance_mappings`` table. +* Implement an online data migration to populate the ``user_id`` field. +* Update the ``_server_group_count_members_by_user`` quota counting method to + use only the ``nova_api.instance_mappings`` table instead of querying cell + databases. +* Add a config option ``[workarounds]disable_quota_usage_from_placement`` that + defaults to False. This will be able to be deprecated when partitioning of + resource providers or allocations is available in placement. +* Add a new method to count instances with a count of + ``nova_api.instance_mappings`` filtering by ``project_id=`` and + ``user_id=`` and ``queued_for_delete=False``. +* Add a new count method that queries the placement API for CPU and RAM usage. + In the new count method, add a check for whether the online data migration + has been run yet and if not, fall back on the legacy count method. +* Rename the ``_instances_cores_ram_count`` method to ``_cores_ram_count`` and + let it count only cores and ram in the legacy way, for use if + ``[workarounds]disable_quota_usage_from_placement`` is set to True. +* Adjust the nova-next or nova-live-migration CI job to run with + ``[workarounds]disable_quota_usage_from_placement=True``. + +Dependencies +============ + +None. + +Testing +======= + +Unit tests and functional tests will be included to test the new functionality. +We will also adjust one CI job (nova-next or nova-live-migration) to run with +``[workarounds]disable_quota_usage_from_placement=True`` to make sure we have +integration test coverage of that path. + +Documentation Impact +==================== + +The documentation_ of Cells v2 caveats will be updated to update the paragraph +about the inability to correctly calculate quota usage when one or more cells +are unreachable. We will document that beginning in Stein, there are new +deployment options. + +.. _documentation: https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#quota-related-quirks + +References +========== + +This builds upon the work done in Pike to re-architect quotas to count +resources. + +* http://specs.openstack.org/openstack/nova-specs/specs/pike/approved/cells-count-resources-to-check-quota-in-api.html + +History +======= + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - Stein + - Introduced