Clean up contributor document
The contibutor document had subpages section on the top, but it was weird (especially in pdf) that subpages were before sections of "overview" and "big picture". This patch sorts it out creating a new section of "architecture". Change-Id: Iea2307774e7f6f778c72f5b07b526fad18dc4cc5
This commit is contained in:
Tetsuro Nakamura
parent
ac1e0f8afe
commit
04b6c2fb91
|
|
@ -0,0 +1,347 @@
|
|||
..
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
not use this file except in compliance with the License. You may obtain
|
||||
a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
License for the specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
==============
|
||||
Architecture
|
||||
==============
|
||||
|
||||
The placement service is straightforward: It is a `WSGI`_ application that
|
||||
sends and receives JSON, using an RDBMS (usually MySQL) for persistence.
|
||||
As state is managed solely in the DB, scaling the placement service is done by
|
||||
increasing the number of WSGI application instances and scaling the RDBMS using
|
||||
traditional database scaling techniques.
|
||||
|
||||
For sake of consistency and because there was initially intent to make the
|
||||
entities in the placement service available over RPC,
|
||||
:oslo.versionedobjects-doc:`versioned objects <>` were used to provide the
|
||||
interface between the HTTP application layer and the SQLAlchemy-driven
|
||||
persistence layer. In the Stein release, that interface was refactored to
|
||||
remove the use of versioned objects and split functionality into smaller
|
||||
modules.
|
||||
|
||||
Though the placement service does not aspire to be a `microservice` it does
|
||||
aspire to continue to be small and minimally complex. This means a relatively
|
||||
small amount of middleware that is not configurable, and a limited number of
|
||||
exposed resources where any given resource is represented by one (and only
|
||||
one) URL that expresses a noun that is a member of the system. Adding
|
||||
additional resources should be considered a significant change requiring robust
|
||||
review from many stakeholders.
|
||||
|
||||
The set of HTTP resources represents a concise and constrained grammar for
|
||||
expressing the management of resource providers, inventories, resource classes,
|
||||
traits, and allocations. If a solution is initially designed to need more
|
||||
resources or a more complex grammar that may be a sign that we need to give our
|
||||
goals greater scrutiny. Is there a way to do what we want with what we have
|
||||
already? Can some other service help? Is a new collaborating service required?
|
||||
|
||||
Minimal Framework
|
||||
=================
|
||||
|
||||
The API is set up to use a minimal framework that tries to keep the structure
|
||||
of the application as discoverable as possible and keeps the HTTP interaction
|
||||
near the surface. The goal of this is to make things easy to trace when
|
||||
debugging or adding functionality.
|
||||
|
||||
Functionality which is required for every request is handled in raw WSGI
|
||||
middleware that is composed in the `placement.deploy` module. Dispatch or
|
||||
routing is handled declaratively via the ``ROUTE_DECLARATIONS`` map defined in
|
||||
the `placement.handler` module.
|
||||
|
||||
Mapping is by URL plus request method. The destination is a complete WSGI
|
||||
application, using a subclass of the `wsgify`_ method from `WebOb`_ to provide
|
||||
a `Request`_ object that provides convenience methods for accessing request
|
||||
headers, bodies, and query parameters and for generating responses. In the
|
||||
placement API these mini-applications are called `handlers`. The `wsgify`
|
||||
subclass is provided in `placement.wsgi_wrapper` as `PlacementWsgify`. It is
|
||||
used to make sure that JSON formatted error responses are structured according
|
||||
to the API-SIG `errors`_ guideline.
|
||||
|
||||
This division between middleware, dispatch and handlers is supposed to
|
||||
provide clues on where a particular behavior or functionality should be
|
||||
implemented. Like most such systems, this does not always work but is a useful
|
||||
tool.
|
||||
|
||||
.. _microversion process:
|
||||
|
||||
Microversions
|
||||
=============
|
||||
|
||||
The placement API makes use of `microversions`_ to allow the release of new
|
||||
features on an opt in basis. See :doc:`/index` for an up to date
|
||||
history of the available microversions.
|
||||
|
||||
The rules around when a microversion is needed are modeled after those of the
|
||||
:nova-doc:`compute API <contributor/microversions>`. When adding a new
|
||||
microversion there are a few bits of required housekeeping that must be done in
|
||||
the code:
|
||||
|
||||
* Update the ``VERSIONS`` list in ``placement/microversion.py`` to indicate the
|
||||
new microversion and give a very brief summary of the added feature.
|
||||
* Update ``placement/rest_api_version_history.rst`` to add a more detailed
|
||||
section describing the new microversion.
|
||||
* Add a :reno-doc:`release note <>` with a ``features`` section announcing the
|
||||
new or changed feature and the microversion.
|
||||
* If the ``version_handler`` decorator (see below) has been used, increment
|
||||
``TOTAL_VERSIONED_METHODS`` in ``placement/tests/unit/test_microversion.py``.
|
||||
This provides a confirmatory check just to make sure you are paying attention
|
||||
and as a helpful reminder to do the other things in this list.
|
||||
* Include functional gabbi tests as appropriate (see :doc:`testing`). At the
|
||||
least, update the ``latest microversion`` test in
|
||||
``placement/tests/functional/gabbits/microversion.yaml``.
|
||||
* Update the `API Reference`_ documentation as appropriate. The source is
|
||||
located under ``api-ref/source/``.
|
||||
* If a new error code has been added in ``placement/errors.py``, it should
|
||||
be added to the `API Reference`_.
|
||||
|
||||
In the placement API, microversions only use the modern form of the
|
||||
version header::
|
||||
|
||||
OpenStack-API-Version: placement 1.2
|
||||
|
||||
If a valid microversion is present in a request it will be placed,
|
||||
as a ``Version`` object, into the WSGI environment with the
|
||||
``placement.microversion`` key. Often, accessing this in handler
|
||||
code directly (to control branching) is the most explicit and
|
||||
granular way to have different behavior per microversion. A
|
||||
``Version`` instance can be treated as a tuple of two ints and
|
||||
compared as such or there is a ``matches`` method.
|
||||
|
||||
A ``version_handler`` decorator is also available. It makes it possible to have
|
||||
multiple different handler methods of the same (fully-qualified by package)
|
||||
name, each available for a different microversion window. If a request wants a
|
||||
microversion that is not available, a defined status code is returned (usually
|
||||
``404`` or ``405``). There is a unit test in place which will fail if there are
|
||||
version intersections.
|
||||
|
||||
Adding a New Handler
|
||||
====================
|
||||
|
||||
Adding a new URL or a new method (e.g, ``PATCH``) to an existing URL
|
||||
requires adding a new handler function. In either case a new microversion and
|
||||
release note is required. When adding an entirely new route a request for a
|
||||
lower microversion should return a ``404``. When adding a new method to an
|
||||
existing URL a request for a lower microversion should return a ``405``.
|
||||
|
||||
In either case, the ``ROUTE_DECLARATIONS`` dictionary in the
|
||||
`placement.handler` module should be updated to point to a
|
||||
function within a module that contains handlers for the type of entity
|
||||
identified by the URL. Collection and individual entity handlers of the same
|
||||
type should be in the same module.
|
||||
|
||||
As mentioned above, the handler function should be decorated with
|
||||
``@wsgi_wrapper.PlacementWsgify``, take a single argument ``req`` which is a
|
||||
WebOb `Request`_ object, and return a WebOb `Response`_.
|
||||
|
||||
For ``PUT`` and ``POST`` methods, request bodies are expected to be JSON
|
||||
based on a content-type of ``application/json``. This may be enforced by using
|
||||
a decorator: ``@util.require_content('application/json')``. If the body is not
|
||||
`JSON`, a ``415`` response status is returned.
|
||||
|
||||
Response bodies are usually `JSON`. A handler can check the `Accept` header
|
||||
provided in a request using another decorator:
|
||||
``@util.check_accept('application/json')``. If the header does not allow
|
||||
`JSON`, a ``406`` response status is returned.
|
||||
|
||||
If a hander returns a response body, a ``Last-Modified`` header should be
|
||||
included with the response. If the entity or entities in the response body
|
||||
are directly associated with an object (or objects, in the case of a
|
||||
collection response) that has an ``updated_at`` (or ``created_at``)
|
||||
field, that field's value can be used as the value of the header (WebOb will
|
||||
take care of turning the datetime object into a string timestamp). A
|
||||
``util.pick_last_modified`` is available to help choose the most recent
|
||||
last-modified when traversing a collection of entities.
|
||||
|
||||
If there is no directly associated object (for example, the output is the
|
||||
composite of several objects) then the ``Last-Modified`` time should be
|
||||
``timeutils.utcnow(with_timezone=True)`` (the timezone must be set in order
|
||||
to be a valid HTTP timestamp). For example, the response__ to
|
||||
``GET /allocation_candidates`` should have a last-modified header of now
|
||||
because it is composed from queries against many different database entities,
|
||||
presents a mixture of result types (allocation requests and provider
|
||||
summaries), and has a view of the system that is only meaningful *now*.
|
||||
|
||||
__ https://docs.openstack.org/api-ref/placement/#list-allocation-candidates
|
||||
|
||||
If a ``Last-Modified`` header is set, then a ``Cache-Control`` header with a
|
||||
value of ``no-cache`` must be set as well. This is to avoid user-agents
|
||||
inadvertently caching the responses.
|
||||
|
||||
`JSON` sent in a request should be validated against a JSON Schema. A
|
||||
``util.extract_json`` method is available. This takes a request body and a
|
||||
schema. If multiple schema are used for different microversions of the same
|
||||
request, the caller is responsible for selecting the right one before calling
|
||||
``extract_json``.
|
||||
|
||||
When a handler needs to read or write the data store it should use methods on
|
||||
the objects found in the `placement.objects` package. Doing so requires a
|
||||
context which is provided to the handler method via the WSGI environment. It
|
||||
can be retrieved as follows::
|
||||
|
||||
context = req.environ['placement.context']
|
||||
|
||||
.. note:: If your change requires new methods or new objects in the
|
||||
`placement.objects` package, after you have made sure that you really
|
||||
do need those new methods or objects (you may not!) make those
|
||||
changes in a patch that is separate from and prior to the HTTP API
|
||||
change.
|
||||
|
||||
If a handler needs to return an error response, with the advent of `Placement
|
||||
API Error Handling`_, it is possible to include a code in the JSON error
|
||||
response. This can be used to distinguish different errors with the same HTTP
|
||||
response status code (a common case is a generation conflict versus an
|
||||
inventory in use conflict). Error codes are simple namespaced strings (e.g.,
|
||||
``placement.inventory.inuse``) for which symbols are maintained in
|
||||
``placement.errors``. Adding a symbol to a response is done
|
||||
by using the ``comment`` kwarg to a WebOb exception, like this::
|
||||
|
||||
except exception.InventoryInUse as exc:
|
||||
raise webob.exc.HTTPConflict(
|
||||
_('update conflict: %(error)s') % {'error': exc},
|
||||
comment=errors.INVENTORY_INUSE)
|
||||
|
||||
Code that adds newly raised exceptions should include an error code. Find
|
||||
additional guidelines on use in the docs for ``placement.errors``. When a
|
||||
new error code is added, also document it in the `API Reference`_.
|
||||
|
||||
Testing of handler code is described in :doc:`testing`.
|
||||
|
||||
Database Schema Changes
|
||||
=======================
|
||||
|
||||
At some point in every application's life it becomes necessary to change the
|
||||
structure of its database. Modifying the SQLAlchemy models (in
|
||||
placement/db/sqlachemy/models.py) is necessary for the application to
|
||||
understand the new structure, but that will not change the actual underlying
|
||||
database. To do that, Placement uses `alembic` to run database migrations.
|
||||
|
||||
Alembic calls each change a **revision**. To create a migration with alembic,
|
||||
run the `alembic revision` command. Alembic will then generate a new revision
|
||||
file with a unique file name, and place it in the `alembic/versions/`
|
||||
directory:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
ed@devenv:~/projects/placement$ alembic -c placement/db/sqlalchemy/alembic.ini revision -m "Add column foo to bar table"
|
||||
Generating /home/ed/projects/placement/placement/db/sqlalchemy/alembic/versions/dfb006498ad2_add_column_foo_to_bar_table.py ... done
|
||||
|
||||
Let us break down that command:
|
||||
|
||||
- The **-c** parameter tells alembic where to find its configuration file.
|
||||
- **revision** is the alembic subcommand for creating a new revision file.
|
||||
- The **-m** parameter specifies a brief comment explaining the change.
|
||||
- The generated file from alembic will have a name consisting of a random hash
|
||||
prefix, followed by an underscore, followed by your **-m** comment, and a
|
||||
**.py** extension. So be sure to keep your comment as brief as possible
|
||||
while still being descriptive.
|
||||
|
||||
The generated file will look something like this:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
"""Add column foo to bar table
|
||||
|
||||
Revision ID: dfb006498ad2
|
||||
Revises: 0378df171af3
|
||||
Create Date: 2018-10-29 20:02:58.290779
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'dfb006498ad2'
|
||||
down_revision = '0378df171af3'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
pass
|
||||
|
||||
The top of the file is the docstring that will show when you review your
|
||||
revision history. If we did not include the **-m** comment when we ran the
|
||||
`alembic revision` command, this would just contain "empty message". If you did
|
||||
not specify the comment when creating the file, be sure to replace "empty
|
||||
message" with a brief comment describing the reason for the database change.
|
||||
|
||||
You then need to define the changes in the `upgrade()` method. The code used in
|
||||
these methods is basic SQLAlchemy code for creating and modifying tables. You
|
||||
can examine existing migrations in the project to see examples of what this
|
||||
code looks like, as well as find more in-depth usage of Alembic in the `Alembic
|
||||
tutorial`_.
|
||||
|
||||
One other option when creating the revision is to add the ``--autogenerate``
|
||||
parameter to the revision command. This assumes that you have already updated
|
||||
the SQLAlchemy models, and have a connection to the placement database
|
||||
configured. When run with this option, the `upgrade()` method of the revision
|
||||
file is filled in for you by alembic as it compares the schema described in
|
||||
your models.py script and the actual state of the database. You should always
|
||||
verify the revision script to make sure it does just what you intended, both by
|
||||
reading the code as well as running the tests, as there are some things that
|
||||
autogenerate cannot deduce. See `autogenerate limitations`_ for more detailed
|
||||
information.
|
||||
|
||||
Gotchas
|
||||
=======
|
||||
|
||||
This section tries to shed some light on some of the differences between the
|
||||
placement API and some of the other OpenStack APIs or on situations which may
|
||||
be surprising or unexpected.
|
||||
|
||||
* The placement API is somewhat more strict about `Content-Type` and `Accept`
|
||||
headers in an effort to follow the HTTP RFCs.
|
||||
|
||||
If a user-agent sends some JSON in a `PUT` or `POST` request without a
|
||||
`Content-Type` of `application/json` the request will result in an error.
|
||||
|
||||
If a `GET` request is made without an `Accept` header, the response will
|
||||
default to being `application/json`.
|
||||
|
||||
If a request is made with an explicit `Accept` header that does not include
|
||||
`application/json` then there will be an error and the error will attempt to
|
||||
be in the requested format (for example, `text/plain`).
|
||||
|
||||
* If a URL exists, but a request is made using a method that that URL does not
|
||||
support, the API will respond with a `405` error. Sometimes in the nova APIs
|
||||
this can be a `404` (which is wrong, but understandable given the constraints
|
||||
of the code).
|
||||
|
||||
* Because each handler is individually wrapped by the `PlacementWsgify`
|
||||
decorator any exception that is a subclass of `webob.exc.WSGIHTTPException`
|
||||
that is raised from within the handler, such as `webob.exc.HTTPBadRequest`,
|
||||
will be caught by WebOb and turned into a valid `Response`_ containing
|
||||
headers and body set by WebOb based on the information given when the
|
||||
exception was raised. It will not be seen as an exception by any of the
|
||||
middleware in the placement stack.
|
||||
|
||||
In general this is a good thing, but it can lead to some confusion if, for
|
||||
example, you are trying to add some middleware that operates on exceptions.
|
||||
|
||||
Other exceptions that are not from `WebOb`_ will raise outside the handlers
|
||||
where they will either be caught in the `__call__` method of the
|
||||
`PlacementHandler` app that is responsible for dispatch, or by the
|
||||
`FaultWrap` middleware.
|
||||
|
||||
|
||||
.. _WSGI: https://www.python.org/dev/peps/pep-3333/
|
||||
.. _wsgify: http://docs.webob.org/en/latest/api/dec.html
|
||||
.. _WebOb: http://docs.webob.org/en/latest/
|
||||
.. _Request: http://docs.webob.org/en/latest/reference.html#request
|
||||
.. _Response: http://docs.webob.org/en/latest/#response
|
||||
.. _microversions: http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html
|
||||
.. _errors: http://specs.openstack.org/openstack/api-wg/guidelines/errors.html
|
||||
.. _API Reference: https://docs.openstack.org/api-ref/placement/
|
||||
.. _Placement API Error Handling: http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/placement-api-error-handling.html
|
||||
.. _`Alembic tutorial`: https://alembic.zzzcomputing.com/en/latest/tutorial.html
|
||||
.. _`autogenerate limitations`: https://alembic.zzzcomputing.com/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect
|
||||
|
|
@ -11,31 +11,9 @@
|
|||
License for the specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
===============================
|
||||
Placement API Developer Notes
|
||||
===============================
|
||||
|
||||
Subpages
|
||||
========
|
||||
|
||||
.. # TODO(cdent): What I really want here is to be able to have this list
|
||||
# in the info column that is to the left of the page on the docs theme.
|
||||
# Thus far there doesn't seem to be a way to do that, so this ugly way
|
||||
# provides a method for making sure the subpages are more discoverable.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
contributing
|
||||
api-ref-guideline
|
||||
goals
|
||||
quick-dev
|
||||
testing
|
||||
vision-reflection
|
||||
|
||||
|
||||
Overview
|
||||
========
|
||||
===========================
|
||||
Placement Developer Notes
|
||||
===========================
|
||||
|
||||
The Nova project introduced the placement service as part of the Newton
|
||||
release, and it was extracted to its own repository in the Stein release. The
|
||||
|
|
@ -51,335 +29,13 @@ the system was created and how it does its job see :doc:`/index`. For some
|
|||
insight into the longer term goals of the system see :doc:`goals` and
|
||||
:doc:`vision-reflection`.
|
||||
|
||||
Big Picture
|
||||
===========
|
||||
|
||||
The placement service is straightforward: It is a `WSGI`_ application that
|
||||
sends and receives JSON, using an RDBMS (usually MySQL) for persistence.
|
||||
As state is managed solely in the DB, scaling the placement service is done by
|
||||
increasing the number of WSGI application instances and scaling the RDBMS using
|
||||
traditional database scaling techniques.
|
||||
|
||||
For sake of consistency and because there was initially intent to make the
|
||||
entities in the placement service available over RPC,
|
||||
:oslo.versionedobjects-doc:`versioned objects <>` were used to provide the
|
||||
interface between the HTTP application layer and the SQLAlchemy-driven
|
||||
persistence layer. In the Stein release, that interface was refactored to
|
||||
remove the use of versioned objects and split functionality into smaller
|
||||
modules.
|
||||
|
||||
Though the placement service does not aspire to be a `microservice` it does
|
||||
aspire to continue to be small and minimally complex. This means a relatively
|
||||
small amount of middleware that is not configurable, and a limited number of
|
||||
exposed resources where any given resource is represented by one (and only
|
||||
one) URL that expresses a noun that is a member of the system. Adding
|
||||
additional resources should be considered a significant change requiring robust
|
||||
review from many stakeholders.
|
||||
|
||||
The set of HTTP resources represents a concise and constrained grammar for
|
||||
expressing the management of resource providers, inventories, resource classes,
|
||||
traits, and allocations. If a solution is initially designed to need more
|
||||
resources or a more complex grammar that may be a sign that we need to give our
|
||||
goals greater scrutiny. Is there a way to do what we want with what we have
|
||||
already? Can some other service help? Is a new collaborating service required?
|
||||
|
||||
Minimal Framework
|
||||
=================
|
||||
|
||||
The API is set up to use a minimal framework that tries to keep the structure
|
||||
of the application as discoverable as possible and keeps the HTTP interaction
|
||||
near the surface. The goal of this is to make things easy to trace when
|
||||
debugging or adding functionality.
|
||||
|
||||
Functionality which is required for every request is handled in raw WSGI
|
||||
middleware that is composed in the `placement.deploy` module. Dispatch or
|
||||
routing is handled declaratively via the ``ROUTE_DECLARATIONS`` map defined in
|
||||
the `placement.handler` module.
|
||||
|
||||
Mapping is by URL plus request method. The destination is a complete WSGI
|
||||
application, using a subclass of the `wsgify`_ method from `WebOb`_ to provide
|
||||
a `Request`_ object that provides convenience methods for accessing request
|
||||
headers, bodies, and query parameters and for generating responses. In the
|
||||
placement API these mini-applications are called `handlers`. The `wsgify`
|
||||
subclass is provided in `placement.wsgi_wrapper` as `PlacementWsgify`. It is
|
||||
used to make sure that JSON formatted error responses are structured according
|
||||
to the API-SIG `errors`_ guideline.
|
||||
|
||||
This division between middleware, dispatch and handlers is supposed to
|
||||
provide clues on where a particular behavior or functionality should be
|
||||
implemented. Like most such systems, this does not always work but is a useful
|
||||
tool.
|
||||
|
||||
Gotchas
|
||||
=======
|
||||
|
||||
This section tries to shed some light on some of the differences between the
|
||||
placement API and some of the other OpenStack APIs or on situations which may
|
||||
be surprising or unexpected.
|
||||
|
||||
* The placement API is somewhat more strict about `Content-Type` and `Accept`
|
||||
headers in an effort to follow the HTTP RFCs.
|
||||
|
||||
If a user-agent sends some JSON in a `PUT` or `POST` request without a
|
||||
`Content-Type` of `application/json` the request will result in an error.
|
||||
|
||||
If a `GET` request is made without an `Accept` header, the response will
|
||||
default to being `application/json`.
|
||||
|
||||
If a request is made with an explicit `Accept` header that does not include
|
||||
`application/json` then there will be an error and the error will attempt to
|
||||
be in the requested format (for example, `text/plain`).
|
||||
|
||||
* If a URL exists, but a request is made using a method that that URL does not
|
||||
support, the API will respond with a `405` error. Sometimes in the nova APIs
|
||||
this can be a `404` (which is wrong, but understandable given the constraints
|
||||
of the code).
|
||||
|
||||
* Because each handler is individually wrapped by the `PlacementWsgify`
|
||||
decorator any exception that is a subclass of `webob.exc.WSGIHTTPException`
|
||||
that is raised from within the handler, such as `webob.exc.HTTPBadRequest`,
|
||||
will be caught by WebOb and turned into a valid `Response`_ containing
|
||||
headers and body set by WebOb based on the information given when the
|
||||
exception was raised. It will not be seen as an exception by any of the
|
||||
middleware in the placement stack.
|
||||
|
||||
In general this is a good thing, but it can lead to some confusion if, for
|
||||
example, you are trying to add some middleware that operates on exceptions.
|
||||
|
||||
Other exceptions that are not from `WebOb`_ will raise outside the handlers
|
||||
where they will either be caught in the `__call__` method of the
|
||||
`PlacementHandler` app that is responsible for dispatch, or by the
|
||||
`FaultWrap` middleware.
|
||||
|
||||
.. _microversion process:
|
||||
|
||||
Microversions
|
||||
=============
|
||||
|
||||
The placement API makes use of `microversions`_ to allow the release of new
|
||||
features on an opt in basis. See :doc:`/index` for an up to date
|
||||
history of the available microversions.
|
||||
|
||||
The rules around when a microversion is needed are modeled after those of the
|
||||
:nova-doc:`compute API <contributor/microversions>`. When adding a new
|
||||
microversion there are a few bits of required housekeeping that must be done in
|
||||
the code:
|
||||
|
||||
* Update the ``VERSIONS`` list in ``placement/microversion.py`` to indicate the
|
||||
new microversion and give a very brief summary of the added feature.
|
||||
* Update ``placement/rest_api_version_history.rst`` to add a more detailed
|
||||
section describing the new microversion.
|
||||
* Add a :reno-doc:`release note <>` with a ``features`` section announcing the
|
||||
new or changed feature and the microversion.
|
||||
* If the ``version_handler`` decorator (see below) has been used, increment
|
||||
``TOTAL_VERSIONED_METHODS`` in ``placement/tests/unit/test_microversion.py``.
|
||||
This provides a confirmatory check just to make sure you are paying attention
|
||||
and as a helpful reminder to do the other things in this list.
|
||||
* Include functional gabbi tests as appropriate (see :doc:`testing`). At the
|
||||
least, update the ``latest microversion`` test in
|
||||
``placement/tests/functional/gabbits/microversion.yaml``.
|
||||
* Update the `API Reference`_ documentation as appropriate. The source is
|
||||
located under ``api-ref/source/``.
|
||||
* If a new error code has been added in ``placement/errors.py``, it should
|
||||
be added to the `API Reference`_.
|
||||
|
||||
In the placement API, microversions only use the modern form of the
|
||||
version header::
|
||||
|
||||
OpenStack-API-Version: placement 1.2
|
||||
|
||||
If a valid microversion is present in a request it will be placed,
|
||||
as a ``Version`` object, into the WSGI environment with the
|
||||
``placement.microversion`` key. Often, accessing this in handler
|
||||
code directly (to control branching) is the most explicit and
|
||||
granular way to have different behavior per microversion. A
|
||||
``Version`` instance can be treated as a tuple of two ints and
|
||||
compared as such or there is a ``matches`` method.
|
||||
|
||||
A ``version_handler`` decorator is also available. It makes it possible to have
|
||||
multiple different handler methods of the same (fully-qualified by package)
|
||||
name, each available for a different microversion window. If a request wants a
|
||||
microversion that is not available, a defined status code is returned (usually
|
||||
``404`` or ``405``). There is a unit test in place which will fail if there are
|
||||
version intersections.
|
||||
|
||||
Adding a New Handler
|
||||
====================
|
||||
|
||||
Adding a new URL or a new method (e.g, ``PATCH``) to an existing URL
|
||||
requires adding a new handler function. In either case a new microversion and
|
||||
release note is required. When adding an entirely new route a request for a
|
||||
lower microversion should return a ``404``. When adding a new method to an
|
||||
existing URL a request for a lower microversion should return a ``405``.
|
||||
|
||||
In either case, the ``ROUTE_DECLARATIONS`` dictionary in the
|
||||
`placement.handler` module should be updated to point to a
|
||||
function within a module that contains handlers for the type of entity
|
||||
identified by the URL. Collection and individual entity handlers of the same
|
||||
type should be in the same module.
|
||||
|
||||
As mentioned above, the handler function should be decorated with
|
||||
``@wsgi_wrapper.PlacementWsgify``, take a single argument ``req`` which is a
|
||||
WebOb `Request`_ object, and return a WebOb `Response`_.
|
||||
|
||||
For ``PUT`` and ``POST`` methods, request bodies are expected to be JSON
|
||||
based on a content-type of ``application/json``. This may be enforced by using
|
||||
a decorator: ``@util.require_content('application/json')``. If the body is not
|
||||
`JSON`, a ``415`` response status is returned.
|
||||
|
||||
Response bodies are usually `JSON`. A handler can check the `Accept` header
|
||||
provided in a request using another decorator:
|
||||
``@util.check_accept('application/json')``. If the header does not allow
|
||||
`JSON`, a ``406`` response status is returned.
|
||||
|
||||
If a hander returns a response body, a ``Last-Modified`` header should be
|
||||
included with the response. If the entity or entities in the response body
|
||||
are directly associated with an object (or objects, in the case of a
|
||||
collection response) that has an ``updated_at`` (or ``created_at``)
|
||||
field, that field's value can be used as the value of the header (WebOb will
|
||||
take care of turning the datetime object into a string timestamp). A
|
||||
``util.pick_last_modified`` is available to help choose the most recent
|
||||
last-modified when traversing a collection of entities.
|
||||
|
||||
If there is no directly associated object (for example, the output is the
|
||||
composite of several objects) then the ``Last-Modified`` time should be
|
||||
``timeutils.utcnow(with_timezone=True)`` (the timezone must be set in order
|
||||
to be a valid HTTP timestamp). For example, the response__ to
|
||||
``GET /allocation_candidates`` should have a last-modified header of now
|
||||
because it is composed from queries against many different database entities,
|
||||
presents a mixture of result types (allocation requests and provider
|
||||
summaries), and has a view of the system that is only meaningful *now*.
|
||||
|
||||
__ https://docs.openstack.org/api-ref/placement/#list-allocation-candidates
|
||||
|
||||
If a ``Last-Modified`` header is set, then a ``Cache-Control`` header with a
|
||||
value of ``no-cache`` must be set as well. This is to avoid user-agents
|
||||
inadvertently caching the responses.
|
||||
|
||||
`JSON` sent in a request should be validated against a JSON Schema. A
|
||||
``util.extract_json`` method is available. This takes a request body and a
|
||||
schema. If multiple schema are used for different microversions of the same
|
||||
request, the caller is responsible for selecting the right one before calling
|
||||
``extract_json``.
|
||||
|
||||
When a handler needs to read or write the data store it should use methods on
|
||||
the objects found in the `placement.objects` package. Doing so requires a
|
||||
context which is provided to the handler method via the WSGI environment. It
|
||||
can be retrieved as follows::
|
||||
|
||||
context = req.environ['placement.context']
|
||||
|
||||
.. note:: If your change requires new methods or new objects in the
|
||||
`placement.objects` package, after you have made sure that you really
|
||||
do need those new methods or objects (you may not!) make those
|
||||
changes in a patch that is separate from and prior to the HTTP API
|
||||
change.
|
||||
|
||||
If a handler needs to return an error response, with the advent of `Placement
|
||||
API Error Handling`_, it is possible to include a code in the JSON error
|
||||
response. This can be used to distinguish different errors with the same HTTP
|
||||
response status code (a common case is a generation conflict versus an
|
||||
inventory in use conflict). Error codes are simple namespaced strings (e.g.,
|
||||
``placement.inventory.inuse``) for which symbols are maintained in
|
||||
``placement.errors``. Adding a symbol to a response is done
|
||||
by using the ``comment`` kwarg to a WebOb exception, like this::
|
||||
|
||||
except exception.InventoryInUse as exc:
|
||||
raise webob.exc.HTTPConflict(
|
||||
_('update conflict: %(error)s') % {'error': exc},
|
||||
comment=errors.INVENTORY_INUSE)
|
||||
|
||||
Code that adds newly raised exceptions should include an error code. Find
|
||||
additional guidelines on use in the docs for ``placement.errors``. When a
|
||||
new error code is added, also document it in the `API Reference`_.
|
||||
|
||||
Testing of handler code is described in :doc:`testing`.
|
||||
|
||||
Database Schema Changes
|
||||
=======================
|
||||
|
||||
At some point in every application's life it becomes necessary to change the
|
||||
structure of its database. Modifying the SQLAlchemy models (in
|
||||
placement/db/sqlachemy/models.py) is necessary for the application to
|
||||
understand the new structure, but that will not change the actual underlying
|
||||
database. To do that, Placement uses `alembic` to run database migrations.
|
||||
|
||||
Alembic calls each change a **revision**. To create a migration with alembic,
|
||||
run the `alembic revision` command. Alembic will then generate a new revision
|
||||
file with a unique file name, and place it in the `alembic/versions/`
|
||||
directory:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
ed@devenv:~/projects/placement$ alembic -c placement/db/sqlalchemy/alembic.ini revision -m "Add column foo to bar table"
|
||||
Generating /home/ed/projects/placement/placement/db/sqlalchemy/alembic/versions/dfb006498ad2_add_column_foo_to_bar_table.py ... done
|
||||
|
||||
Let us break down that command:
|
||||
|
||||
- The **-c** parameter tells alembic where to find its configuration file.
|
||||
- **revision** is the alembic subcommand for creating a new revision file.
|
||||
- The **-m** parameter specifies a brief comment explaining the change.
|
||||
- The generated file from alembic will have a name consisting of a random hash
|
||||
prefix, followed by an underscore, followed by your **-m** comment, and a
|
||||
**.py** extension. So be sure to keep your comment as brief as possible
|
||||
while still being descriptive.
|
||||
|
||||
The generated file will look something like this:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
"""Add column foo to bar table
|
||||
|
||||
Revision ID: dfb006498ad2
|
||||
Revises: 0378df171af3
|
||||
Create Date: 2018-10-29 20:02:58.290779
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'dfb006498ad2'
|
||||
down_revision = '0378df171af3'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
pass
|
||||
|
||||
The top of the file is the docstring that will show when you review your
|
||||
revision history. If we did not include the **-m** comment when we ran the
|
||||
`alembic revision` command, this would just contain "empty message". If you did
|
||||
not specify the comment when creating the file, be sure to replace "empty
|
||||
message" with a brief comment describing the reason for the database change.
|
||||
|
||||
You then need to define the changes in the `upgrade()` method. The code used in
|
||||
these methods is basic SQLAlchemy code for creating and modifying tables. You
|
||||
can examine existing migrations in the project to see examples of what this
|
||||
code looks like, as well as find more in-depth usage of Alembic in the `Alembic
|
||||
tutorial`_.
|
||||
|
||||
One other option when creating the revision is to add the ``--autogenerate``
|
||||
parameter to the revision command. This assumes that you have already updated
|
||||
the SQLAlchemy models, and have a connection to the placement database
|
||||
configured. When run with this option, the `upgrade()` method of the revision
|
||||
file is filled in for you by alembic as it compares the schema described in
|
||||
your models.py script and the actual state of the database. You should always
|
||||
verify the revision script to make sure it does just what you intended, both by
|
||||
reading the code as well as running the tests, as there are some things that
|
||||
autogenerate cannot deduce. See `autogenerate limitations`_ for more detailed
|
||||
information.
|
||||
|
||||
.. _WSGI: https://www.python.org/dev/peps/pep-3333/
|
||||
.. _wsgify: http://docs.webob.org/en/latest/api/dec.html
|
||||
.. _WebOb: http://docs.webob.org/en/latest/
|
||||
.. _Request: http://docs.webob.org/en/latest/reference.html#request
|
||||
.. _Response: http://docs.webob.org/en/latest/#response
|
||||
.. _microversions: http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html
|
||||
.. _errors: http://specs.openstack.org/openstack/api-wg/guidelines/errors.html
|
||||
.. _API Reference: https://docs.openstack.org/api-ref/placement/
|
||||
.. _Placement API Error Handling: http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/placement-api-error-handling.html
|
||||
.. _`Alembic tutorial`: https://alembic.zzzcomputing.com/en/latest/tutorial.html
|
||||
.. _`autogenerate limitations`: https://alembic.zzzcomputing.com/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
contributing
|
||||
architecture
|
||||
api-ref-guideline
|
||||
goals
|
||||
quick-dev
|
||||
testing
|
||||
vision-reflection
|
||||
|
|
|
|||
Loading…
Reference in New Issue