From 661963a667fb6f10c415f653110437131d78981d Mon Sep 17 00:00:00 2001 From: Kiall Mac Innes Date: Wed, 3 Jun 2015 13:37:59 +0100 Subject: [PATCH] Add Zone Exists Event Spec Change-Id: Ie474d02c3cb3b610d1fad64e88f863f37d9c01fe --- specs/liberty/zone-exists-event.rst | 153 ++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 specs/liberty/zone-exists-event.rst diff --git a/specs/liberty/zone-exists-event.rst b/specs/liberty/zone-exists-event.rst new file mode 100644 index 0000000..3f89148 --- /dev/null +++ b/specs/liberty/zone-exists-event.rst @@ -0,0 +1,153 @@ +.. + +This work is licensed under a Creative Commons Attribution 3.0 Unported License. +http://creativecommons.org/licenses/by/3.0/legalcode + +=================== + Zone Exists Event +=================== + +https://blueprints.launchpad.net/designate/+spec/zone-exists-event + +Problem description +=================== + +Designate emits several events for use by Ceilometer for Metering & Billing +purposes. Currently, these events are sent for zone/rrset/etc create, update +and delete actions. As these events are emitted over RabbitMQ, we cannot +guarantee delivery. In order to ensure accurate billing for customers, we need +to provide a mechanism for identifying "lost" events. This is particularity +important for Zone create/delete events, as per-Zone/per-Timeperiod is the +most common billing unit. + +The accepted standard solution to the loss of create/delete events is for each +service to emit a periodic "exists" event, this allows Ceilometer to, for +example, identify that N exists events in a row were not received, surmising +that a "delete" event was missed. + +In other services like Nova there is a clear "owner" for each core resource +(e.g. an instance is owned by the compute node it resides on), this allows +each nova-compute instance to periodically emit a `compute.instance.exists` +event for a small number of instances. Designate has no such clear owner, +and many thousands of zones may belong to a single pool (the smallest +grouping we current;y have available). This poses a problem, in that emitting +100's of thousands of events, or more, each hour may be too demaining for the +active pool manager. + +Proposed change +=============== + +We will introduce a new service `designate-zone-manager` which will handle all +periodic tasks relating to the zones it is responsible for. Initially, this +will only be the periodic `dns.domain.exists` event, but over time we will add +additional tasks, for example, polling of Secondary zones at their refresh +intervals. + +A concept of "zone shards" will be introduced, where every zone will be +allocated to a shard based on the first three characters of the zones UUID. +This will provide for 4,096 distinct shards to be distributed over the set of +available `designate-zone-manager` processes, ensuring that no single +`designate-zone-manager` is responsible for more zones than it can reasonably +handle. With 5 million zones, each shard should contain approx 1.2k zones. + +Finally, distribution of shards to available `designate-zone-manager` +processes will be handled by the `OpenStack Tooz +`_ library. Tooz provides the building +blocks required to implement a "Partitioner", using it's Group Membership +APIs to divvy up the available shards between active workers, including +dynamically re-allocating based on membership changes. + +API Changes +----------- + +None + +Central Changes +--------------- + +None + +Storage Changes +--------------- + +A new column will be added to the domains, recordsets and records tables, this +column will be populated by the storage driver with the integer representation +of the first 3 characters of the UUID, giving a whole number with a value +between 0 and 4095. We add the shard to the RecordSets/Records tables now, +as we know it will be used in a future blueprint (ALIAS records). + +Domains Table Additions +^^^^^^^^^^^^^^^^^^^^^^^ + ++-------+--------+-----------+---------+ +| Row | Type | Nullable? | Unique? | ++=======+========+===========+=========+ +| shard | int(2) | No | No | ++-------+--------+-----------+---------+ + +RecordSets/Records Table Additions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++--------------+--------+-----------+---------+ +| Row | Type | Nullable? | Unique? | ++==============+========+===========+=========+ +| domain_shard | int(2) | No | No | ++--------------+--------+-----------+---------+ + +Other Changes +------------- + +A new service, `designate-zone-manager` will be introduced. This service will +extend the Designate base `Service` class, but will not include the typical +`RPCService` mixin. For this initial use case, there is no requirement for +an RPC API. + +This service will use the existing oslo-incubator ThreadGroup.add_timer() +methods for scheduling tasks, and the tooz library for group membership. + +The timer interval will be exposed as a configuration value, defaulting to +3600 seconds. The interval will be independant of any other timers introduced +into this service in the future. + +Group membership will be implemented as a `Service` mixin, similar to +`RPCService`, allowing for it's inclusion in other services such as +`designate-pool-manager`. + +The `dns.domain.exists` event format will be identical to the domain +create/update/delete event format, with the only difference being the event +name. + +Finally, the list of Zones each `designate-zone-manager` is responsible for +will be gathered at the start of each timer interval, in batches of a +configurable size, based on the range of shard's allocated. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Endre Karlson + +Milestones +---------- + +Target Milestone for completion: + Liberty-2 + +Work Items +---------- + +* Implement "Partitioner" based on Tooz +* Implement `GroupMembership` mixin (Name TBD) +* Create new `designate-zone-manager` service +* Implement periodic exists event + + +Dependencies +============ + +- Requires the OpenStack Tooz library be added to our requirements +- Requires infrastructure for the the OpenStack Tooz library (memcache, redis, + or zookeeper)