From 37254966ac013fd0054fa81be65aafe651e49aae Mon Sep 17 00:00:00 2001 From: Tim Simmons Date: Tue, 5 May 2015 15:38:10 +0000 Subject: [PATCH] Asynchronous Zone Import Spec for making zone imports asynchronous. Change-Id: I71676a12be92fca481ee0fc79c705a3f17bb15da --- specs/liberty/async-import-export.rst | 307 ++++++++++++++++++++++++++ 1 file changed, 307 insertions(+) create mode 100644 specs/liberty/async-import-export.rst diff --git a/specs/liberty/async-import-export.rst b/specs/liberty/async-import-export.rst new file mode 100644 index 0000000..b147228 --- /dev/null +++ b/specs/liberty/async-import-export.rst @@ -0,0 +1,307 @@ +.. + +This work is licensed under a Creative Commons Attribution 3.0 Unported License. +http://creativecommons.org/licenses/by/3.0/legalcode + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/designate/+spec/awesome-thing should be named + awesome-thing.rst . Please do not delete any of the sections in this + template. If you have nothing to say for a whole section, just write: None + For help with syntax, see http://sphinx-doc.org/rest.html + To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html + +=============================== +Asynchronous Zone Import/Export +=============================== + +https://blueprints.launchpad.net/designate/+spec/async-import-export + +Problem description +=================== + +Large zone imports have the potential to take a lot longer than the +typically allowed time for API responses. Parsing large zone files, and inserting +the necessary records scales linearly with the amount of records that need to be +imported. + +Proposed change +=============== + +The API for Zone imports needs to be changed to be asynchronous. This would involve +a new database table for zone import "statuses", and a more robust system for managing +the parsing and insertion of the zone data. Exporting zones will also return to the v2 API. + +API Changes +----------- + +GET /v2/zones/tasks/imports +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This allows the user to view the statuses of their zone import requests. If the import has +completed successfully, they can get the zone id and a link to the zone. + +.. code-block:: http + + GET /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1 + Accept: application/json + + HTTP/1.1 200 OK + Content-Type: application/json; charset=UTF-8 + Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 + + { + "imports": [ + { + "id": "cddda8f0-f558-11e3-a3ac-0800200c9a66", + "zone_id": "agqm44f0-s638-15e3-f4d3-1893572c9a67", + "status": "SUCCESS", + "links":{ + "self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66", + "href": "http://127.0.0.1:9001/v2/zones/agqm44f0-s638-15e3-f4d3-1893572c9a67" + }, + { + "id": "addda8f0-f558-11e3-a3ac-0800200c9a66", + "zone_id": "qgqm44f0-s638-15e3-f4d3-1893572c9a67", + "status": "SUCCESS", + "links":{ + "self": "http://127.0.0.1:9001/v2/zones/tasks/import/addda8f0-f558-11e3-a3ac-0800200c9a66", + "href": "http://127.0.0.1:9001/v2/zones/qgqm44f0-s638-15e3-f4d3-1893572c9a67" + } + ] + } + +POST /v2/zones/tasks/imports +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This creates a request to import a zone. The zone data is passed via the request body, +and the Content-Type header must be set to 'text/dns'. + +This returns an id for the import request, for the user to query. + +.. code-block:: http + + POST /v2/zones/tasks/import HTTP/1.1 + Accept: application/json + Content-Type: text/dns + + $ORIGIN example.com. + example.com. 42 IN SOA ns.example.com. nsadmin.example.com. 42 42 42 42 42 + example.com. 42 IN NS ns.example.com. + example.com. 42 IN MX 10 mail.example.com. + ns.example.com. 42 IN A 10.0.0.1 + mail.example.com. 42 IN A 10.0.0.2 + + HTTP/1.1 201 Accepted + Content-Type: application/json; charset=UTF-8 + Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 + + { + "id": "cddda8f0-f558-11e3-a3ac-0800200c9a66", + "zone_id": null, + "status": "PENDING", + "links":{ + "self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66" + } + } + +GET /v2/zones/tasks/imports/ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This allows the user to view the status of their zone import request. If the import has +completed successfully, they can get the zone id and a link to the zone. + +.. code-block:: http + + GET /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1 + Accept: application/json + + HTTP/1.1 200 OK + Content-Type: application/json; charset=UTF-8 + Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 + + { + "id": "cddda8f0-f558-11e3-a3ac-0800200c9a66", + "zone_id": "agqm44f0-s638-15e3-f4d3-1893572c9a67", + "status": "SUCCESS", + "links":{ + "self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66", + "href": "http://127.0.0.1:9001/v2/zones/agqm44f0-s638-15e3-f4d3-1893572c9a67" + } + } + +GET /v2/zones/ +^^^^^^^^^^^^^^^^^^ + +This request, with the header "Accept:text/dns" exports the zone in DNS zonefile format to the user. + +.. code-block:: http + + GET /v2/zones/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1 + Accept: text/dns + + HTTP/1.1 200 OK + Content-Type: text/dns; charset=UTF-8 + Location: /v2/zones/cddda8f0-f558-11e3-a3ac-0800200c9a66 + + HTTP/1.1 200 OK + Content-Type: text/dns + + $ORIGIN example.com. + $TTL 42 + example.com. IN SOA ns.designate.com. nsadmin.example.com. ( + 1394213803 ; serial + 3600 ; refresh + 600 ; retry + 86400 ; expire + 3600 ; minimum + ) + example.com. IN NS ns.designate.com. + example.com. IN MX 10 mail.example.com. + ns.example.com. IN A 10.0.0.1 + mail.example.com. IN A 10.0.0.2 + +Central Changes +--------------- + +create_import_domain(body) +^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++---------------+-----------------------------------------------+--------------+ +| **Parameter** | **Description** | **Required** | ++===============+===============================================+==============+ +| *body* | Unserialized request data from the API request| Yes | ++---------------+-----------------------------------------------+--------------+ + +1. Create an entry in the zone_imports table to track the request with status +PENDING. +2. Kicks off a thread to _import_domain with the request body. +3. Returns a zone_import object. + +get_import_domain(id) +^^^^^^^^^^^^^^^^^^^^^ + ++---------------+-----------------------------------------------+--------------+ +| **Parameter** | **Description** | **Required** | ++===============+===============================================+==============+ +| *import_id* | The id of the zone import | Yes | ++---------------+-----------------------------------------------+--------------+ + +1. Calls storage.find_import to get a specific zone_import record from the +zone_imports table. +2. Returns the zone_import object + +find_import_domains(context) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++---------------+-----------------------------------------------+--------------+ +| **Parameter** | **Description** | **Required** | ++===============+===============================================+==============+ +| *context* | Context to be passed to storage.find_imports | Yes | ++---------------+-----------------------------------------------+--------------+ + +1. Calls off to the storage.find_imports to find all the zone imports for a +tenant. +2. Returns the zone_import_list object + +delete_import_domain(id) +^^^^^^^^^^^^^^^^^^^^^^^^ + ++---------------+-----------------------------------------------+--------------+ +| **Parameter** | **Description** | **Required** | ++===============+===============================================+==============+ +| *import_id* | The id of the zone import | Yes | ++---------------+-----------------------------------------------+--------------+ + +1. Calls off to storage.delete_zone_import to delete the zone import record from +the zone_imports table. + +_import_domain(body) +"""""""""""""""""""" + ++---------------+-----------------------------------------------+--------------+ +| **Parameter** | **Description** | **Required** | ++===============+===============================================+==============+ +| *body* | Unserialized request data from the API request| Yes | ++---------------+-----------------------------------------------+--------------+ +| *zone_import* | zone_import object from the original request | Yes | ++---------------+-----------------------------------------------+--------------+ + + +1. Tries to use dnspython's from_text method to convert the zone into a dnspython +object. If that doesn't work, updates the zone_import object in storage to ERROR +to indicate that the zone could not be imported. +2. Converts the dnspython zone to a Designate domain object. +3. Calls central.create_domain with the converted object. +4. Takes the return object's id and updates the zone_import object in storage +with the new zone_id and updates the status of the import to COMPLETE, or similar. + + +Storage Changes +--------------- + +A new table for zone import records will be added, along with the boilerplate +get, set, delete, update methods. + + +New Table - zone_imports +^^^^^^^^^^^^^^^^^^^^^^^^ + ++---------+---------+-----------+---------+-------------------------------------+ +| Row | Type | Nullable? | Unique? | Notes | ++=========+=========+===========+=========+=====================================+ +| id | uuid | No | Yes | Primary key | ++---------+---------+-----------+---------+-------------------------------------+ +| zone_id | uuid | Yes | Yes | Zone id when the import completes | ++---------+---------+-----------+---------+-------------------------------------+ +| status | ENUM | Yes | No | One of [COMPLETE, PENDING, ERROR] | ++---------+---------+-----------+---------+-------------------------------------+ +| message | VARCHAR | Yes | No | A message letting the user know why | +| | | | | their import failed. For example: | +| | | | | "Malformed zonefile", "Complete" | ++---------+---------+-----------+---------+-------------------------------------+ + +Other Changes +------------- + +New DesignateObjects for the Imports will have be created, ZoneImport, ZoneImportList. + +Exporting zones will also return to the v2 API, but should remain relatively unchanged. + +Alternatives +------------ + +Instead of creating a new zone_import table and object, it may be possible +to create an empty domain object and call central._create_domain_in_storage +and make an empty domain that is then updated with the result of the import. +Only then could the call to the pool manager be made. + +This might make the code simpler. But provides very little in the way of +letting the user know that their import failed. +You could add a status like MALFORMED_ZONEFILE or something, but that +would still require the user to delete the zone before they tried again. +Unless you soft-delete the zone when it fails and modify the default +find_domains criterion to find zones that have been deleted only if they +have that status. + +Assignee(s) +----------- + +Primary assignee: + tim-simmons-t + +Milestones +---------- + +Target Milestone for completion: + Liberty-1 + +Work Items +---------- + +- Implement the database table, migration. +- Implement the storage methods. +- Implement the central methods. +- Fix the API to use the new code. +- Move the APIs back into the v2 namespace