Rolling upgrades: different object versions

This updates the rolling upgrades spec to change the design
of how ironic services will handle the different IronicObject
versions during a rolling upgrade.

It also adds some more alternatives.

Change-Id: I3c6caa639aa75ed87bddd59bf514e2d904db6e0a
Partial-Bug: #1526283
This commit is contained in:
Ruby Loo 2017-06-01 12:31:31 -04:00
parent e466091ebb
commit 5370fd3878
1 changed files with 280 additions and 157 deletions

View File

@ -57,9 +57,12 @@ individual ironic-api and ironic-conductor services to be upgraded one at a
time, with the rest of the services still available. This upgrade would have
minimal downtime.
The rolling upgrade solution presented here is modelled after nova's
upgrade process [2]_, with some differences. In addition, there have been
several discussions about rolling upgrades ([3]_, [9]_, [10]_).
Although we'd like the rolling upgrade solution presented here to be the same
as that used by nova's upgrade process [2]_, there are some differences between
nova and ironic that prevent us from using the same solution. The differences
are mentioned below, in other sections of this specification.
There have been several discussions about rolling upgrades ([3]_, [9]_, [10]_).
Proposed change
@ -119,22 +122,22 @@ Rolling upgrade process
The rolling upgrade process to upgrade ironic from version ``FromVer`` to the
next version ``ToVer`` is as follows:
#. Upgrade Ironic Python Agent image before upgrading ironic.
1. Upgrade Ironic Python Agent image before upgrading ironic.
#. Upgrade DB schema to ``ToVer`` via :command:`ironic-dbsync upgrade`.
2. Upgrade DB schema to ``ToVer`` via **ironic-dbsync upgrade**.
Ironic already has the code in place to do this. However, a new DB
migration policy (described below in `New DB model change policy`_) needs
to be documented.
#. Pin RPC and IronicObject versions to the same ``FromVer`` for both
3. Pin RPC and IronicObject versions to the same ``FromVer`` for both
ironic-api and ironic-conductor services, via the new configuration option
described below in `RPC and object version pinning`_.
#. Upgrade code and restart ironic-conductor services, one at a time.
4. Upgrade code and restart ironic-conductor services, one at a time.
#. Upgrade code and restart ironic-api services, one at a time.
5. Upgrade code and restart ironic-api services, one at a time.
#. Unpin RPC and object versions so that the services can now use the latest
6. Unpin RPC and object versions so that the services can now use the latest
versions in ``ToVer``. This is done via updating the new configuration
option described below in `RPC and object version pinning`_ and then
restarting the services. ironic-conductor services should be restarted
@ -142,24 +145,43 @@ next version ``ToVer`` is as follows:
functionality is exposed on the unpinned API service (via API micro
version), it is available on the backend.
#. Run a new command :command:`ironic-dbsync online_data_migration` to ensure
7. Run a new command **ironic-dbsync online_data_migration** to ensure
that all DB records are "upgraded" to the new data version.
This new command is discussed in a separate RFE [12]_ (and is a dependency
for this work).
#. Upgrade ironic client libraries (e.g. python-ironicclient) and other
8. Upgrade ironic client libraries (e.g. python-ironicclient) and other
services which use the newly introduced API features and depend on the
new version.
Changes needed to support rolling upgrade
-----------------------------------------
The above process will cause the ironic services to be running the ``FromVer``
and ``ToVer`` releases in this order (where 'step' refers to the steps above):
+------+---------------------------------+---------------------------------+
| step | ironic-api | ironic-conductor |
+======+=================================+=================================+
| 0 | all FromVer | all FromVer |
+------+---------------------------------+---------------------------------+
| 4.1 | all FromVer | some FromVer, some ToVer-pinned |
+------+---------------------------------+---------------------------------+
| 4.2 | all FromVer | all ToVer-pinned |
+------+---------------------------------+---------------------------------+
| 5.1 | some FromVer, some ToVer-pinned | all ToVer-pinned |
+------+---------------------------------+---------------------------------+
| 5.2 | all ToVer-pinned | all ToVer-pinned |
+------+---------------------------------+---------------------------------+
| 6.1 | all ToVer-pinned | some ToVer-pinned, some ToVer |
+------+---------------------------------+---------------------------------+
| 6.2 | all ToVer-pinned | all ToVer |
+------+---------------------------------+---------------------------------+
| 6.3 | some ToVer-pinned, some ToVer | all ToVer |
+------+---------------------------------+---------------------------------+
| 6.4 | all ToVer | all ToVer |
+------+---------------------------------+---------------------------------+
For the above to work, there are several changes that have to be done.
A framework patch [4]_ and an implementation reference patch [5]_ exist to
verify the concept.
New DB model change policy
``````````````````````````
--------------------------
This is not a code change but it impacts the SQLAlchemy DB model and needs to
be documented well for developers as well as reviewers.
@ -171,7 +193,7 @@ This new DB model change policy is as follows:
to ironic deprecation policy [8]_.
But its alembic script has to wait one more deprecation period, otherwise
an "unknown column" exception will be thrown when ``FromVer`` services
access the DB. This is because :command:`ironic-dbsync upgrade` upgrades the
access the DB. This is because **ironic-dbsync upgrade** upgrades the
DB schema but ``FromVer`` services still contain the dropped field in their
SQLAlchemy DB model.
@ -186,7 +208,39 @@ This new DB model change policy is as follows:
store a large dataset), these cases must be mentioned in the release notes.
RPC and object version pinning
``````````````````````````````
------------------------------
For the ironic (ironic-api and ironic-conductor) services to be running
old and new releases at the same time during a rolling upgrade, the services
need to be able to handle different RPC versions and object versions.
[4]_ has a good description of why we need RPC versioning, and describes how
nova deals with it. This proposes taking a similar approach in ironic.
For object versioning, ironic uses oslo.versionedobjects. [5]_ describes nova's
approach to the problem. Unfortunately, ironic's solution is different, since
ironic has a more complex situation. In nova, all database access (reads and
writes) is done via the nova-conductor service. This makes it possible for the
nova-conductor service to be the only service to handle conversions between
different object versions. (See [5]_ for more details.) Given an object that
it doesn't understand, a (non nova-conductor) service will issue an RPC request
to the nova-conductor service to get the object converted to its desired target
version. Furthermore, for a nova rolling upgrade, all the non-nova-compute
services are shut down, and then restarted with the new releases;
nova-conductor being the first service to be restarted ([2]_). Thus, the
nova-conductor services are always running the same release and don't have to
deal with differing object versions amongst themselves. Once they are running
the new release, they can handle requests from other services running old or
new releases.
Contrast that to ironic, where both the ironic-api and ironic-conductor
services access the database for reading and writing. Both these services need
to be aware of different object versions. For example, ironic-api can
create objects such as Chassis, Ports, and Portgroups, saving them directly to
the database without going through the conductor. We cannot take down the
ironic-conductor in a similar way as the nova-conductor service, because
ironic-conductor does a whole lot more than just interacting with the database,
and at least one ironic-conductor needs to be running during a rolling upgrade.
A new configuration option will be added. It will be used to pin the RPC and
IronicObject (e.g., Node, Conductor, Chassis, Port, and Portgroup) versions for
@ -196,7 +250,7 @@ to properly handle the communication between different versions of services.
The new configuration option is: ``[DEFAULT]/pin_release_version``.
The default value of empty indicates that ironic-api and ironic-conductor
will use the latest versions of RPC and IronicObjects. Its possible values are
releases, named (e.g. ``newton``) or sem-versioned (e.g. ``5.2``).
releases, named (e.g. ``ocata``) or sem-versioned (e.g. ``7.0``).
Internally, ironic will maintain a mapping that indicates the RPC and
IronicObject versions associated with each release. This mapping will be
@ -215,175 +269,199 @@ example:
{'mitaka': '1.33', '5.23': '1.33'}
Set version_cap to pinned version
:::::::::::::::::::::::::::::::::
``ConductorAPI.__init__()`` already sets a ``version_cap`` to the latest
RPC API version and passes it to the ``RPCClient`` as an initialization
During a rolling upgrade, the services using the new release should set this
value to be the name (or version) of the old release. This will indicate
to the services running the new release, which RPC and object versions that
they should be compatible with, in order to communicate with the services
using the old release.
Handling RPC versions
~~~~~~~~~~~~~~~~~~~~~
``ConductorAPI.__init__()`` already sets a ``version_cap`` variable to the
latest RPC API version and passes it to the ``RPCClient`` as an initialization
parameter. This ``version_cap`` is used to determine the maximum requested
message version that the ``RPCClient`` can send.
In order to make a compatible RPC call for a previous release, the code will
be changed so that the ``version_cap`` is set to a pinned version
(corresponding to the desired release) rather than the latest
(corresponding to the previous release) rather than the latest
``RPC_API_VERSION``. Then each RPC call will customize the request according
to this ``version_cap``.
Make use of target_version
::::::::::::::::::::::::::
Object instances will make use of a new value ``target_version`` when
the service interacts with another service.
If pinned version was given, the value of ``target_version`` will be set
to a corresponding value in ``objects_mapping``. Otherwise, the value is None.
Object instances are instantiated according to ``VERSION``, so ``FromVer``
and ``ToVer`` services are still running their own version object instances.
However at first, the database only contains old records in ``FromVer``.
When services interact, the serialized object instances can be converted to
``target_version`` by ``ToVer`` services. To assist in this, several methods
will be added or changed for ``ironic.objects.base.IronicObject``.
Handling IronicObject versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Putting it together
:::::::::::::::::::
Internally, ironic services (ironic-api and ironic-conductor) will deal with
IronicObjects in their latest versions. Only at these boundaries, when the
IronicObject enters or leaves the service, will we need to deal with object
versioning:
In the following descriptions:
* *getting objects from the database*: convert to latest version
* *saving objects to the database*: if pinned, save in pinned version; else
save in latest version
* *serializing objects (to send over RPC)*: if pinned, send pinned version;
else send latest version
* *deserializing objects (receiving objects from RPC)*: convert to latest
version
* ``FromVer`` uses version '1.14' of a Node object.
* ``ToVer`` uses version '1.15' of a Node object -- this has a deprecated
``extra`` field and a new ``fake`` field that replaces ``extra``.
* db_obj['fake'] and db_obj['extra'] are the database representations of those
The ironic-api service also has to handle API requests/responses
based on whether or how a feature is supported by the API version and object
versions. For example, when the ironic-api service is pinned, it can only
allow actions that are available to the object's pinned version, and cannot
allow actions that are only available for the latest version of that object.
To support this:
* add a new column named ``version`` to all the database tables (SQLAlchemy
models) of the IronicObjects. The value is the version of the object that
is saved in the database.
This version column will be null at first and will be filled with the
appropriate versions by a data migration script. If there is a change in
Ocata that requires migration of data, we will check for null in the new
version column.
No project uses the version column mechanism for this purpose, but it is more
complicated without it. For example, Cinder has a migration policy which
spans 4 releases in which data is duplicated for some time. Keystone uses
triggers to maintain duplicated data in one release cycle. In addition, the
version column may prove useful for zero-downtime upgrades (in the future).
* add a new method ``IronicObject.get_target_version(self)``. This will return
the target version. If pinned, the pinned version is returned. Otherwise,
the latest version is returned.
* add a new method ``IronicObject.convert_to_version(self, target_version)``.
This method will convert the object into the target version. The target
version may be a newer or older version that the existing version of the
object. The bulk of the work will be done in the new helper method
``IronicObject._convert_to_version(self, target_version)``. Subclasses that
have new versions should redefine this to perform the actual conversions.
* add a new method ``IronicObject.do_version_changes_for_db(self)``. This is
described below in `Saving objects to the database (API/conductor --> DB)`_.
* add a new method ``IronicObjectSerializer._process_object(self, context,
objprim)``. This is described below in
`Receiving objects via RPC (API/conductor <- RPC)`_.
In the following,
* The old release is ``FromVer``; it uses version '1.14' of a Node object.
* The new release is ``ToVer``; it uses version '1.15' of a Node object --
this has a deprecated ``extra`` field and a new ``meta`` field that replaces
``extra``.
* db_obj['meta'] and db_obj['extra'] are the database representations of those
node fields.
* ``ToVer`` is pinned to ``FromVer``.
Instantiate services' version objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``IronicObject._from_db_object(obj, db_object)`` is a method that already
exists; it converts a database entity to a formal object.
Each IronicObject class will need to be changed, to implement its own
_from_db_object() to instantiate its own version objects (regardless of
any pinning) from the database. In this case:
Getting objects from the database (API/conductor <-- DB)
::::::::::::::::::::::::::::::::::::::::::::::::::::::::
* ``FromVer`` will instantiate version '1.14' of Node object,
ignoring db_obj['fake'] but setting node.extra = db_obj['extra'].
* ``ToVer`` will instantiate version '1.15' of the Node object, setting
node.fake = db_obj['extra'] and setting node.extra = None. (With
pinning, db_obj['fake'] is still empty, but node.fake on the object should be
available for the new code release to use.)
Both ironic-api and ironic-conductor services read values from the database.
These values are converted to IronicObjects via the existing method
``IronicObject._from_db_object(context, obj, db_object)``. This method will be
changed so that the IronicObject will be in the latest version, even if it was
in an older version in the database. This is done regardless of the service
being pinned or not.
Get target version
~~~~~~~~~~~~~~~~~~
A new method ``IronicObjectSerializer._get_target_version(self, obj)`` will be
added. This will return a string, the IronicObject version corresponding to the
pinned release version. If there is no pinning, it will return the latest
version for that IronicObject.
Note that if an object is converted to a later version, that IronicObject will
retain any changes resulting from that conversion (in case the object later
gets saved in the latest version).
In this case, ``ToVer`` is pinned to ``FromVer``, so
serializer._get_target_version(Node) would return version '1.14'.
For example, if the node in the database is in version 1.14 and has
db_obj['extra'] set:
Convert passed instance values to the target version
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Based on the target version, an IronicObject instance will be serialized
to correspond to the target version. This will be done before sending the
object over-the-wire to other services with IronicObjectSerializer.
* a ``FromVer`` service will get a Node with node.extra = db_obj['extra']
(and no knowledge of node.meta since it doesn't exist).
In this case, since ``ToVer`` is pinned to ``FromVer``, a ``ToVer`` Node
object instance 'node' will be serialized to look like version '1.14',
with node.extra = node.fake, and no node.fake attribute.
* a ``ToVer`` service (pinned or unpinned), will get a Node with:
Save pinned-version values to DB (API/Conductor --> DB)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
During the rolling upgrade, some services will be running ``ToVer`` and pinned
to ``FromVer``, while the others are still running ``FromVer``.
Since these services access the same database and to avoid saving
into different columns, ironic won't save values in new columns until after
the unpinning.
* node.meta = db_obj['extra']
* node.extra = None
* node._changed_fields = ['meta', 'extra']
A new method ``IronicObject._obj_get_db_compatible_changes(self)``
will return a dictionary of DB-compatible changed fields and values.
These will be compatible with the pinned version.
Wherever an IronicObject writes to the database (typically in .create() and
.save()), it will call ._obj_get_db_compatible_changes() instead of
.obj_get_changes(), to get changes that are compatible with the database
(i.e., the database that corresponds to the pinned version).
Saving objects to the database (API/conductor --> DB)
:::::::::::::::::::::::::::::::::::::::::::::::::::::
Use actual versions when reading values from DB (API/Conductor <-- DB)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The unpinning operation is not atomic. There can also be a situation, where
all services are running in ``ToVer``, but some of the conductors are
unpinned. On the other extreme, all conductors can be unpinned and some of
api services may still be pinned. Because of this, a service may retrieve from
DB or receive via RPC a ``ToVer`` object instance, while it is still pinned to
``FromVer``.
The version used for saving IronicObjects to the database is determined as
follows:
For example, in case of retrieving an object from the DB, it could happen that
one conductor is pinned, the other is not. When data is
* for an unpinned service, the object will be saved in its latest version.
Since objects are always in their latest version, no conversions are needed.
* for a pinned service, the object will be saved in its pinned version. Since
objects are always in their latest version, the object will need to be
converted to the pinned version before being saved.
1. saved by the unpinned conductor
2. retrieved and saved by the pinned conductor
The new method ``IronicObject.do_version_changes_for_db()`` will handle this
logic, returning a dictionary of changed fields and their new values (similar
to the existing
``oslo.versionedobjects.VersionedObjectobj.obj_get_changes()``).
Since we do not keep track internally, of the database version of an object,
the object's ``version`` field will always be part of these changes.
and if the pinned conductor assumes the data is in a previous (pinned) version,
the resulting state is, that the data is inconsistent.
The `Rolling upgrade process`_ (at step 6.1) ensures that by the time an
object can be saved in its latest version, all services are running the newer
release (although some may still be pinned) and can handle the latest object
versions.
It may happen because there was a new column introduced (the ``fake`` column
from the above example), so:
An interesting situation can occur when the services are as described in step
6.1. It is possible for an IronicObject to be saved in a newer version and
subsequently get saved in an older version. For example, a ``ToVer`` unpinned
conductor might save a node in version 1.5. A subsequent request may cause a
``ToVer`` pinned conductor to replace and save the same node in version 1.4!
1. the unpinned conductor saves data into a new column (the ``fake`` column),
2. the pinned conductor reads and updates data in an old column (``extra``).
The end result is that we don't know in which column we have the up-to date
values. At the same time, if an outdated value was used to calculate a new one,
data is lost.
Sending objects via RPC (API/conductor -> RPC)
::::::::::::::::::::::::::::::::::::::::::::::
To maintain data consistency, when a ``ToVer`` object is retrieved from the DB
by a ``ToVer`` service, which is still pinned to ``FromVer``, it should ignore
the globally configured pin for this instance of the object and use its actual
version. The same follows for receiving a ``ToVer`` object through RPC.
When a service makes an RPC request, any IronicObjects that are sent as
part of that request are serialized into entities or primitives (via
``oslo.versionedobjects.VersionedObjectSerializer.serialize_entity()``). The
version used for objects being serialized is as follows:
In short, use target version: api/conductor --> db (when creating/saving a new
object) and use actual (unpinned) version: api/conductor <-- db.
* for an unpinned service, the object will be serialized in its latest version.
Since objects are always in their latest version, no conversions are needed.
* for a pinned service, the object will be serialized in its pinned version.
Since objects are always in their latest version, the object will need to be
converted to the pinned version before being serialized. The converted object
will include changes that resulted from the conversion; this is needed so
that the service at the other end of the RPC request has the necessary
information if that object will be saved to the database.
When we receive an unpinned object, we should save it in its actual version,
so that we don't lose data which may have been added to it in ``ToVer``.
The ``IronicObjectSerializer.serialize_entity()`` method will be modified to do
any IronicObject conversions.
To make sure reading the DB object versions happens transparently for the
developer, a version column will be introduced to SQLAlchemy models.
This version column will be null at first and will be filled with the
appropriate versions by a data migration script. If there is a change in
Ocata that requires migration of data, we will check for null in the new
version column.
Receiving objects via RPC (API/conductor <- RPC)
::::::::::::::::::::::::::::::::::::::::::::::::
No project uses the version column mechanism for this purpose, but it is more
complicated without it. For example, Cinder has a migration policy which spans
4 releases in which data is duplicated for some time. Keystone uses triggers to
maintain duplicated data in one release cycle. In addition, the version column
may prove useful for zero-downtime upgrades (in the future).
When a service receives an RPC request, any entities that are part of the
request need to be deserialized (via
``oslo.versionedobjects.VersionedObjectSerializer.deserialize_entity()``).
For entities that represent IronicObjects, we want the deserialization process
to result in IronicObjects that are in their latest version, regardless of the
version they were sent in and regardless of whether the receiving service is
pinned or not. Again, any objects that are converted will retain the changes
that resulted from the conversion, useful if that object is later saved to the
database.
Passing target version object instances via RPC (API <---> Conductor)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When ironic-api makes an RPC call to update an object instance (in
ironic.conductor.rpcapi.ConductorAPI: e.g., create_node(), update_node(),
update_port() and update_portgroup()), the object instance `with updates` is
passed as a parameter to the method. After updating the database, an updated
object instance is returned.
The deserialization method invokes
``VersionedObjectSerializer._process_object()`` to deserialize and get the
IronicObject. We will add ``IronicObjectSerializer._process_object()`` to
convert the IronicObject to its latest version.
It could happen that a ``FromVer`` API gets a '1.15' Node object from
a ``ToVer`` (unpinned) Conductor. In this case, IronicObjectSerializer will
convert the object based on the target version (``FromVer``), so that the API
is dealing with the correctly versioned node object.
The object_backport_versions RPC method is called to convert the object.
The sequence of events is the following:
#. API (FromVer) -> RPC -> conductor (unpinned ToVer)
#. conductor -> (1.15 obj) -> RPC -> API
#. API raises eyebrow, err exception -> RPC -> conductor
#. conductor -> (1.14 obj) -> RPC -> API
Likewise, a ``FromVer`` API could send an older version object to a
``ToVer`` conductor. So the conductor has to accept and handle older,
compatible objects.
destroy_port() and destroy_portgroup() RPC calls are also affected by this.
For example, a ``FromVer`` ironic-api could issue an update_node() RPC request
with a node in version 1.4, where node.extra was changed (so
node._changed_fields = ['extra']). This node will be serialized in version 1.4.
The receiving ``ToVer`` pinned ironic-conductor deserializes it and converts
it to version 1.5. The resulting node will have node.meta set (to the changed
value from node.extra in v1.4), node.extra = None, and node._changed_fields =
['meta', 'extra'].
Alternatives
------------
@ -391,12 +469,47 @@ Alternatives
A cold upgrade can be done, but it means the ironic services will not be
available during the upgrade, which may be time consuming.
Instead of having the services always treat objects in their latest versions,
a different design could be used, for example, where pinned services treat
their objects in their pinned versions. However, after some experimentation,
this proved to have more (corner) cases to consider and was more difficult
to understand. This approach would make it harder to maintain and trouble-shoot
in the future, assuming reviewers would be able to agree that it worked in the
first place!
What if we changed the ironic-api service, so that it had read-only access to
the DB and all writes would go via the ironic-conductor service. Would that
simplify the changes needed to support rolling upgrades? Perhaps; perhaps not.
(Although this author thinks it would be better, regardless, to have
all writes being done by the conductor.) With or without this change, we need
to ensure that objects are not saved in a newer version (i.e., that is newer
than the version in the older release) until all services are running with the
new release -- step 5.2 of the `Rolling upgrade process`_. The solution
described in this document has objects being saved in their newest versions
starting in step 6.1, because it seemed conceptually easy to understand if
we save objects in their latest versions only when a service is unpinned. We'd
need a similar mechanism regardless.
Of course, there are probably other ways to handle this, like having all
services "register" what versions they are running in the database and
leveraging that data somehow. Dmitry Tantsur mused about whether some remote
synchronization stuff (e.g. etcd) could be used for services to be aware of
the upgrade process.
Ideally, ironic would use some "OpenStack-preferred" way to implement
rolling upgrades but that doesn't seem to exist, so this tries to leverage the
work that nova did.
Data model impact
-----------------
A DB migration policy is adopted and introduced above in
`New DB model change policy`_.
A new ``version`` column will be added to all the database tables of the
IronicObject objects. Its value will be the version of the object that is
saved in the database.
State Machine Impact
--------------------
@ -424,6 +537,14 @@ Client (CLI) impact
-------------------
None
"ironic" CLI
~~~~~~~~~~~~
None
"openstack baremetal" CLI
~~~~~~~~~~~~~~~~~~~~~~~~~
None
RPC API impact
--------------
@ -504,11 +625,13 @@ Assignee(s)
-----------
Primary assignee:
xek
* xek
* rloo
Other contributors:
mario-villaplana-j (documentation)
* mario-villaplana-j (documentation)
Work Items
----------
@ -516,7 +639,7 @@ Work Items
#. Add new configuration option ``[DEFAULT]/pin_release_version``
and RPC/Object version mappings to releases.
#. Make Objects compatible with a previous version and handle the interaction
with DB and user.
with DB and services.
#. Make IronicObjectSerializer downgrade objects when passing via RPC.
#. Add tests.
#. Add documentation and pointers for RPC and oslo objects versioning.
@ -528,7 +651,7 @@ Work Items
Dependencies
============
* Needs the new command :command:`ironic-dbsync online_data_migration` [12]_.
* Needs the new command **ironic-dbsync online_data_migration** [12]_.
* Needs multi-node grenade CI working.
@ -581,8 +704,8 @@ References
.. [1] https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-rolling-upgrade.rst
.. [2] http://docs.openstack.org/developer/nova/upgrade.html
.. [3] https://etherpad.openstack.org/p/ironic-mitaka-midcycle
.. [4] Ironic rolling upgrade framework https://review.openstack.org/#/c/306357/
.. [5] Refactor configdrive into a new field https://review.openstack.org/#/c/306358/
.. [4] http://superuser.openstack.org/articles/upgrades-in-openstack-nova-remote-procedure-call-apis
.. [5] http://superuser.openstack.org/articles/upgrades-in-openstack-nova-objects/
.. [6] https://releases.openstack.org/reference/release_models.html
.. [7] http://semver.org/
.. [8] http://governance.openstack.org/reference/tags/assert_follows-standard-deprecation.html