Merge "API Evolution - ETAG identifiers"
This commit is contained in:
commit
e466091ebb
|
@ -0,0 +1,406 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
======================
|
||||
API Evolution: etag ID
|
||||
======================
|
||||
|
||||
As time has gone on, the API surface has evolved, and the community has given
|
||||
rise to more use-cases and more users. This has resulted in the identification
|
||||
of several shortcomings in the current REST API: it is not as user-friendly
|
||||
as one might expect, and it does not service all of the use-cases we wish
|
||||
that it would.
|
||||
|
||||
This specification describes the implementation of etag identifiers on
|
||||
API responses, allowing for better conflict resolution between concurrent API
|
||||
clients.
|
||||
|
||||
https://bugs.launchpad.net/ironic/+bug/1605728
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Multiple clients attempting to update the same resource can overwrite each
|
||||
others' changes accidentally and without notice. This is referred to as the
|
||||
lost update problem, and it is a problem in the Bare Metal service. The lost
|
||||
update problem is commonly addressed through the use of "etag" identifiers,
|
||||
as described in this API Working Group `specification on etags`_.
|
||||
This specification proposes that the Bare Metal service provides etags support
|
||||
to address this problem.
|
||||
|
||||
.. _`specification on etags`: http://specs.openstack.org/openstack/api-wg/guidelines/etags.html
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
The Bare Metal service shall begin to store, internally, a unique etag
|
||||
identifier for every API modifiable resource (Node, Port, Portgroup, Chassis).
|
||||
This identifier shall be returned in the body and headers of successful
|
||||
responses to GET requests for requested resource beginning with the
|
||||
introduction of a new microversion. There are mainly two cases:
|
||||
|
||||
* GET one resource(subresource). The ETag will be returned in the response
|
||||
headers and body. Python clients may operate with resource using fresh etag
|
||||
(see client section). Ironic shell users see etag in the response.
|
||||
|
||||
* GET list of resources(subresources). Etag of each resource will be available
|
||||
in response body, as field of the resource:
|
||||
|
||||
{
|
||||
'ports': [
|
||||
{
|
||||
uuid: '11111111-2222-3333-4444-555555555555',
|
||||
|
||||
<all other fields>,
|
||||
|
||||
etag: 'W/eeeeeeeeeeeeee'
|
||||
|
||||
},
|
||||
|
||||
{
|
||||
uuid: '66666666-7777-8888-9999-222222222222',
|
||||
|
||||
<all other fields>,
|
||||
|
||||
etag: 'W/tttttttttttttt'
|
||||
|
||||
}
|
||||
|
||||
]
|
||||
|
||||
}
|
||||
|
||||
Note that, putting etags to headers does not make sense, because it is
|
||||
impossible to distinguish etags at client side in standard and simple way.
|
||||
|
||||
|
||||
All requests to modify a resource or subresource, whether through a PUT, PATCH,
|
||||
or DELETE request, SHOULD begin to require If-Match header with an appropriate
|
||||
etag identifier to be supplied in the request (for subresource it is etag of
|
||||
that subresource). Note that If-Match will not be required according to
|
||||
RFC standard `specification of If-Match Header`_.
|
||||
|
||||
Therefore SHOULD keyword used here which means that ``If-Match`` header will be
|
||||
optional parameter in client (see `keywords to indicate requirement levels`_).
|
||||
|
||||
This is important that originally according to rfc `entity-tags using`_
|
||||
``If-Match`` MUST be provided by clients to pass request successfully.
|
||||
When ETags are used for cache validation it is useful. Specifically to our use
|
||||
case making ``If-Match`` header necessary will decrease user-friendliness of
|
||||
ironic client. It is up to users to decide if they want to be aware of
|
||||
what they change.
|
||||
|
||||
``If-None-Match`` or any other 'Precondition Header Fields' will not be
|
||||
supported.
|
||||
|
||||
To save efficiency of the request the If-Match header SHALL be validated at
|
||||
API and conductor side by comparing the supplied If-Match header
|
||||
against the current etag string. If the supplied header does not match the
|
||||
actual value, a ``412 Precondition Failed`` SHALL be returned.
|
||||
|
||||
On the successful receipt of any request to modify a resource or subresource, a
|
||||
new etag identifier SHALL be generated by the server (it MUST NOT depend on
|
||||
which ``ironic-api-version`` came in the request). Where the modification is
|
||||
synchronous and the response already includes a representation of the resource,
|
||||
the new etag identifier SHALL be included in the response body and header.
|
||||
|
||||
Etag is a SHA-512 hash generated from JSON string encoded compactly (ordering
|
||||
of dict is not needed) from the dictionary of oslo versioned object fields.
|
||||
Etag will be generated for ALL create and modify requests taking into account
|
||||
the fields except of the ignored ones:
|
||||
|
||||
For node ignored fields are:
|
||||
|
||||
|
||||
``driver_internal_info, etag, updated_at``
|
||||
|
||||
For port, portgroup and chassis it is only ``etag`` and ``updated_at``.
|
||||
|
||||
|
||||
The generated etag will contain "W/" prefix to specify that weak validator is
|
||||
used. Strong validators are more useful for comparison, but in our use-case,
|
||||
taking into account, that etag is not changed on every update and not
|
||||
when metadata changes (e.g. Content-Type), weak validators are applied.
|
||||
See `weak vs strong validators`_.
|
||||
|
||||
|
||||
.. _`specification of If-Match Header`: https://tools.ietf.org/html/rfc7232#section-3.1
|
||||
.. _`entity-tags using`: https://tools.ietf.org/html/rfc7232.html#section-2.4
|
||||
.. _`keywords to indicate requirement levels`: https://www.ietf.org/rfc/rfc2119.txt
|
||||
.. _`weak vs strong validators`: https://tools.ietf.org/html/rfc7232#section-2.1
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
If we do not implement etag support, we will not have any means to prevent
|
||||
races between clients' PATCH requests, and we will not be able to implement
|
||||
support for PUT as a means to update resources or subresources.
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
As on every REST API request the objects are obtained from database and while
|
||||
it is not going to be changed, etag shall be retrieved from object returned
|
||||
from db. This is more efficiently than spending time to generate hash again and
|
||||
again.
|
||||
|
||||
For that purpose a new field ``etag`` SHALL be added to each resource
|
||||
table to store the etag identifier. Etag identifier will be String Field with
|
||||
data length limited to 130 characters (etag is a string containing prefix
|
||||
"W/" + SHA-512 hexadecimal number 128 digits long).
|
||||
|
||||
New field ``etag`` SHALL be also added to Object model in order to be
|
||||
consistent with db layer. Object layer will also be the place where etag will
|
||||
be regenerated based on current Object fields.
|
||||
|
||||
Etag will be also included to notifications payload to make them more flexible
|
||||
and usable.
|
||||
|
||||
|
||||
State Machine Impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
A new etag header MUST be sent in response to all GET and POST requests
|
||||
starting from specific API microversion, as well as synchronous PATCH and
|
||||
PUT requests.
|
||||
|
||||
The same If-Match etag header SHOULD be accepted in all PUT, PATCH and DELETE
|
||||
requests. That means that each endpoint offering any update capability should
|
||||
have logic to validate etag optioanlly.
|
||||
|
||||
A new error status ``412 PreconditionFailed`` SHALL be introduced and used to
|
||||
signal the clients that their version of a resource is stale, when the supplied
|
||||
etag header does not match the server's version.
|
||||
|
||||
Client (CLI) impact
|
||||
-------------------
|
||||
|
||||
Using new microversion clients get the ability to update resource being aware
|
||||
of what exactly they change. For that they SHOULD send an ``If-Match`` header
|
||||
with an etag identifier and supported Ironic API version in the header of
|
||||
requests to modify any resource. There are two options: doing this either
|
||||
through CLI or through Python Client API. The last option is available for any
|
||||
python developer script used in clouds (useful for production).
|
||||
|
||||
|
||||
The workflow of etags usage in ironicclient shell:
|
||||
|
||||
* Client does GET request.
|
||||
|
||||
* Starting from specific API microversion, response SHALL include an etag
|
||||
for requested resource(s) in the headers. ETag SHALL be also included within
|
||||
returned resources in the body, both for GET individual resource and resource
|
||||
collection.
|
||||
|
||||
* Etag may be specified by users if needed when doing any operation leading to
|
||||
resource changing by adding ``--etag`` flag to the command. Etag can be
|
||||
obtained from body or headers of response:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
ironic --ironic-api-version 1.40 node-update \
|
||||
--etag <etag_string> driver_info/foo=value
|
||||
|
||||
This etag string is put as ``If-Match`` header to the request.
|
||||
|
||||
* Ironic API pops ``If-Match`` header, checks it with rpc_node's etag and if
|
||||
they match, the entity tag is sent further to RPC where conductor validates
|
||||
it again. If etags are not matching at some point, the
|
||||
``412 PreconditionFailed`` error will be raised. If requested
|
||||
``X-Openstack-Ironic-Api-Version`` does not support etags yet,
|
||||
``NotAcceptable`` error is raised.
|
||||
|
||||
|
||||
To make Python Client API usable without shell, resources will be stored as
|
||||
full-featured objects (not just the bag of attributes), including the etag
|
||||
identifier. To do this the ironicclient API will be rewritten in the way
|
||||
that the ``Resource`` class is able to update itself and call manager to send
|
||||
requests available in NodeManager. The ``Resource`` will be stored in the
|
||||
memory like all Python objects are stored during process execution.
|
||||
|
||||
For the Python API all appropriate actions of ``Resource`` object will accept
|
||||
optional parameter etag. The workflow can be
|
||||
the following:
|
||||
|
||||
* In Python Shell or in some script clients do GET request to
|
||||
the resource. The etag returned in the response will be stored in the
|
||||
resource representation. E.g.
|
||||
|
||||
``node = node_manager.get(node_ident)``
|
||||
|
||||
|
||||
* Afterwards at any time user scripts can do the action on the resource itself:
|
||||
|
||||
``new_node = node.update(patch, etag=True)``
|
||||
|
||||
Afterwards they will have the up-to-date resource representation if the
|
||||
request will be validated on the server.
|
||||
|
||||
Note, that for 1 standard deprecation period ``If-Match`` will not be sent
|
||||
to server by default. Clients will be warned that in the next release
|
||||
``etag`` parameter will be ``True`` by default.
|
||||
|
||||
|
||||
If etag is requested it will be retrieved from current resource representation
|
||||
|
||||
(as ``node.etag`` or ``getattr(node, 'etag')``). Afterwards it will be sent as
|
||||
``If-Match`` header, it means that user cares about up-to-date information.
|
||||
If etag is not present at the resource and clients did not turn off ``etag``
|
||||
option, they will fail if using the API version greater or equal than etag API
|
||||
version.
|
||||
|
||||
Depending on the situation, the client may choose to transparently retry, or
|
||||
display a diff to the user of the stored vs. server-side resource.
|
||||
Clients should also begin alerting the user when an update request fails due to
|
||||
a resource conflict.
|
||||
|
||||
|
||||
"ironic" CLI
|
||||
~~~~~~~~~~~~
|
||||
|
||||
See above.
|
||||
|
||||
"openstack baremetal" CLI
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Same workflow described in the Client Impact.
|
||||
|
||||
RPC API impact
|
||||
--------------
|
||||
|
||||
RPC API version needs to be bumped to accept etag parameter for actions on the
|
||||
resource. The ``etag`` parameter, default as None, should be passed to the
|
||||
appropriate methods.
|
||||
|
||||
Driver API impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
Nova driver impact
|
||||
------------------
|
||||
|
||||
Nova ironic driver may use new Ironic API microversion, so ironic api version
|
||||
used by nova virt driver needs to be bumped. Until etag option in python
|
||||
ironicclient API is True, in the nova driver we should explicitly
|
||||
specify ``etag=False`` in appropriate methods through ``Node`` resource object.
|
||||
|
||||
Ramdisk impact
|
||||
--------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
Sending ``If-Match`` header may fail due to ``412 Precondition Failed`` error.
|
||||
A client may retry with new fresh etag or/and display a diff between two
|
||||
resource's representations.
|
||||
|
||||
Scalability impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
New etag's generation may increase the time to respond depending on the
|
||||
resource size.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Some services (e.g. Nova) change baremetal resources through API, so they
|
||||
may upgrade Ironic API to use etag. If services do not upgrade, warn the
|
||||
deployer about that in the case skipping these upgrades may violate some strong
|
||||
recommendations and information consistency is not guaranteed on ironic side.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Python developers can work with ``Resource`` object as with full-featured
|
||||
objects and perform modifying operations on them.
|
||||
Also they can implement scripts that are using etag option in parallel in
|
||||
efficient way.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
galyna
|
||||
|
||||
Other contributors:
|
||||
vdrok
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Implement database migration, adding an internal etag field to all top-level
|
||||
resources.
|
||||
|
||||
* Implement generation and validation utility functions in common code.
|
||||
|
||||
* Implement changes within the ironic.api.controllers.v1 modules to accept and
|
||||
return etag identifiers when fetching or changing resources as appropriate.
|
||||
|
||||
* Implement unit tests and tempest tests.
|
||||
|
||||
* Update api-ref documentation.
|
||||
|
||||
* Implement changes in the python client library and openstack CLI to begin
|
||||
caching etags on GET requests and sending etags on PUT/PATCH/DELETE
|
||||
requests.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Unit and tempest tests shall be added that ensure etag identifiers are
|
||||
returned, that they are validated by requests to modify resources, and that
|
||||
proper errors are returned when an invalid (or merely not current) etag is
|
||||
supplied.
|
||||
|
||||
Upgrades and Backwards Compatibility
|
||||
====================================
|
||||
|
||||
Backwards compatibility is retained because etags SHOULD only be returned and
|
||||
required in new microversions.
|
||||
|
||||
This change does not include substantive changes to any resource.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The proper use of etag identifiers shall be documented in our API reference.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* https://tools.ietf.org/html/rfc7232
|
||||
* http://specs.openstack.org/openstack/api-wg/guidelines/etags.html
|
||||
* https://etherpad.openstack.org/p/ironic-v2-api
|
|
@ -0,0 +1 @@
|
|||
../approved/evolve-etags.rst
|
Loading…
Reference in New Issue