This change modifies the nodes _get_nodes_collection method to
consider and pass in an explicit lisst of requested fields into
the node list method, while also including the required fields
for things like ownership/policy checking.
And slightly modifies node_convert_with_links method to simplify
it while enabling field validity to be checked, and specific
requisite field lists provided in based upon that value.
And also optionally builds the traits list as they are nolonger
*always* populated on all objects with fully populated objects
as only partially hydrated objects are provided back when specific
fields are requested.
Story: 2008885
Task: 42572
Change-Id: Ided419263d84184cab902944b6c518f98618c9d2
A node object has many fields, and a huge opportunity
for performance improvement is to reduce the amount of work
performed at the lower layers when it is not necessary.
In particular, the easiest case to identify and handle is
when a list of fields to be fulfilled is explicitly supplied.
This is particularlly noteworthy when we collecting a list of
nodes for synconization with Nova, where only 9 columns are
really needed to supply Nova with the information required,
and thus numerous fields are discarded.
This is all particularlly compounded when traits are used,
which presently uses a joined load pattern from SQL.
With this patch, we explicitly load and convert only the fields
requested at the lowest level, and then perform a different style
of loading node traits which requires less overhead by SQLAlchemy
to de-duplicate the returned result set to the application.
This turns out to be far more efficent as we're reducing the amount
of data/object conversion work by 360%, even before we consider
populating traits, which is performed as a separate query when
traits are requested.
Story: 2008885
Task: 42434
Change-Id: Iac703d2a9b7b240a47477be8a64c7c33e28f692f
In the code for handling rolling upgrades (and pinned ironic versions),
we were assuming that for an object class (eg 'Node') that had more than
one version in a release, the first version was the latest one. This was
error-prone as humans don't always know/remember to put the latest version
as the first one in the list.
This patch removes the assumption and gets the maximum version in the list.
Change-Id: Ie0a789f8bfa6f286b58cc7211be85ee18b2800f5
Also a few related errors based on some earlier investigation
may have been pulled in along the lines of E305.
Story: #2001985
Change-Id: Ifb2d3b481202fbd8cbb472e02de0f14f4d0809fd
The value returned by ironic.objects.IronicObject.as_dict() should be a
plain object, in order for it to be serialised to JSON. Currently,
nested object fields and object list fields are not converted to dict
format. This caused problems during cleaning, when the node object's
as_dict representation is JSON encoded and sent to IPA.
This change adds support for calling as_dict() on nested objects and
list objects, to ensure these are also returned in dict form.
We also change the method used in as_dict() for checking whether an
object has an attribute. The hasattr() function used previously has
problems when used with properties in python 2 [1], in that any
exceptions raised in the property getter result in hasattr() returning
False. Instead we use obj_attr_is_set() to determine whether the object
has a particular attribute.
[1] https://hynek.me/articles/hasattr/
Change-Id: Ib2166040508827db28d6f6e2d9a3e655c16f2993
Closes-Bug: #1750027
Adds a new traits object to expose traits DB operations to the API. It
also adds a new traits field into the node object, with the appropriate
version compatibility logic.
get_node_by_port_addresses is modified to ensure we correctly join in
the tags and traits in that DB call, this avoids a orphaned db object
lazy load style failure.
_set_from_db_object in the base object is modified such that the new
traits object doesn't have to include the dictionary style compatibility
mix-in.
Change-Id: I69403b9875a020fab7a7975810b57bf646417953
Partial-Bug: #1722194
Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Now that we've released Pike and master is open for Queens
development, we need to remove the release mapping information
for Ocata, to prevent unsupported rolling upgrades from
Ocata to Queens. We only support rolling upgrades from Pike
to Queens (master).
Change-Id: I440b9a4c8ef6024a47b66b0de4fd841b7f15df2e
This adds the new command 'ironic-dbsync online_data_migrations'.
To limit downtime during upgrades, data migrations will be done online
with migration scripts that could be run during normal operation of an
ironic cluster, after the upgrade but before the next one.
Each migration script should ensure that all related DB records are
migrated to the new format. Scripts can detect the format based on
the object version which is stored in the version column.
The online data migration has one script; a function that backfills
the new version column, using versions of objects from the release
prior to this.
This includes code to check the object versions for compatibility with
an ironic release. However, the check is turned off (and will be turned on
in Queens), since we need to boot-strap the new version column before
we can turn the check on. To do this check, we need to keep a list of all
supported versions for every object; release_mapping.RELEASE_MAPPING was
modified so that the object versions is now a list instead of one value.
Change-Id: I1a9fa829951ecf98cae6896d82ba20cf89062394
Closes-Bug: #1585141
Partial-bug: #1526283
Co-Authored-By: Ruby Loo <ruby.loo@intel.com>
In change Ib22753aa6ae0fedce7fb9ecf63f135fda0185c5b the port data model
was updated to include a physical_network field, but this was not
exposed to the user by the REST API. This change exposes the
physical_network field in the REST API.
The port CRUD notification object has been updated to include the
physical_network field.
The API reference and user guide have been updated to include
information about the ports' physical network field.
The API microversion has been bumped to 1.34. During a rolling upgrade
from Ocata when the API service is pinned, the port physical network
field is hidden from API responses, and API requests including the field
are rejected.
Change-Id: I7023a1d6618608c867c31396fa677d3016ca493e
Partial-Bug: #1666009
Renames the parameter 'remove_unavail_fields' to
'remove_unavailable_fields'.
This is a follow up to 8e7d795db5.
Change-Id: I57f7bde85e959f3edeb878089057bfcd26672570
Partial-Bug: #1526283
This contains some changes to conversions of objects during a rolling
upgrade.
This changes objects.base.IronicObject.convert_to_version() to have a new
parameter 'remove_unavail_fields': True (default) to remove fields that are
unavailable in the target version -- for serialization/deserialization
of objects. False to change unavailable fields as appropriate (for DB
interactions). The reason for doing this is to make sure that during
serialization (eg for RPC), that we don't include any object
fields that are not supported.
To make the code a bit more performant, we don't perform object
conversions when the API service is serializing the objects for RPC to
the conductor. This is because the conductor will always be running the
same or a newer release as the API service.
Change-Id: I6f77b24199412e3489dd6f3dcf0f51ed04c5c7c0
Partial-Bug: #1526283
Currently, the logic contained inside the _from_db_object base ironic
object method is simply assignment of values in the DB object to the
oslo versioned object. This might not be that simple in case of nested
objects, for example in case of tags contained inside the node. This
change adds a possibility to select which fields to set as usual by
passing them as fields argument to _from_db_object method. Other fields
are loaded in the new _set_from_db_object method.
Change-Id: Ib2ca1acc52b9ba05297976f2c7ad67a63e07dbde
This includes some changes to make the code easier to understand,
as well as updated docstrings and comments for more clarity.
This is a follow up to ebfc4fe4c4.
Change-Id: I3620b6ada9915eb68d6bba47279d61bf51a6e608
Partial-Bug: #1526283
This extends versioned objects and uses the object's
convert_to_version() to perform conversions between different
versions of the object. This conversion is only done at the
"boundaries" of the api and conductor services. That is, when
reading/writing to the database and when serializing/deserializing
(for RPC). Internally, the services deal with the latest versions
of the objects.
Version column is introduced to make sure reading the DB object
versions happens transparently for the developer. An exception
is raised in case of a version compatibility error.
The version column is null at first and will be filled with the
appropriate versions by a data migration script in a following patch.
Change-Id: I34629a5cbab7ff3f246ea19c0cb766badc7061db
Partial-Bug: #1526283
Co-Authored-By: Ruby Loo <ruby.loo@intel.com>
There is code in oslo.versionedobjects that expects an object
class to have this method:
_from_db_object(context, item_cls(), db_item, **extra_args)
To conform to this, this patch adds 'context' as the first
parameter of base.IronicObject._from_db_object().
As a bonus, some docstrings were cleaned up.
Change-Id: I09b02f47d47cec22d03460b693f9e5c488cf04c7
Closes-Bug: #1670778
We're missing unicode and long types in python 2 and bytes in python 3
here. These don't need to be serialized but will currently emit a
warning log when we hit this path. Add these types (via six so we're
compatible).
Change-Id: Ifb0fba3a82d7b24c25893ba258c74a66630617d9
To support rolling upgrades, capping of RPC communication and Ironic
objects is required. Old RPC services and objects may still be running
while an upgrade is in progress. This makes sure that these old
services are called and all objects are used in a supported RPC and
objects version.
This patch adds the configuration option "pin_release_version". Setting
this option caps (downgrades) the internal RPC communication to the
specified version to enable communication with older services. When
doing a rolling upgrade from version X to Y, set this to X. It defaults
to using the newest (current) possible RPC behavior and object versions.
Change-Id: Ie2342d4051f85392a8b10d39ebffc287da57bf2b
Partial-Bug: #1526283
Co-Authored-By: Szymon Borkowski <szymon.borkowski@intel.com>
Co-Authored-By: Ruby Loo <ruby.loo@intel.com>
This patch adds the VolumeTarget object. It handles the
volume target information which is stored in the database.
Co-Authored-By: Stephane Miller <stephane@alum.mit.edu>
Co-Authored-By: Ruby Loo <ruby.loo@intel.com>
Change-Id: I814454ec5a515080d21aad58f8d43abb49b77fb3
Partial-Bug: 1526231
There was a cyclic import between conductor.rpcapi and objects.base,
this was initally fixed by doing an import at runtime. This patch fixes
the import cycle for good.
Change-Id: I8ee8c47652de75430312f77ddee08adc7a237d2b
Closes-Bug: #1623074
Had five copies of the def _from_db_object() function in various
classes. Move this function into their common base class.
Change-Id: I543386a708d80f7c89455e556e2abc6d532661b0
This adds a register hook to the objects base class that allows us to
late-import objects classes into ironic.objects. This is useful to allow
object classes to depend on other objects.
For example, this will allow us to do the Port.node_uuid munging that we
currently do in the API layer, in the objects layer instead.
This is completely taken from Nova, much thanks to comstud and dansmith
and whoever else worked on that.
Change-Id: Ic655e7af533497bda5e8e72d3bd90c27394681a7
During a rolling upgrade, ironic conductor and api services are running
with different versions. When an object is received in an incompatible
version, IncompatibleObjectVersion is raised. Implementation of the
indirection API allows the object to be backported to a supported version
by the conductor (conductor has to be the first service to be upgraded).
This change enables backporting of objects from Mitaka. This lays the
foundation to be able to do rolling upgrades from Liberty (or from this
patch onwards) to Mitaka. There may still be other issues that will need
fixing in Mitaka before we will be able to do a full rolling upgrade.
Enabling the indirection_api causes all object methods decorated with
the remotable or remotable_classmethod to do RPC from ironic-api to
ironic-conductor service. To keep the current behavior, I'm removing all
remotable decorators on object methods and thus not enabling object RPC
calls in this patch. These calls caused random timeouts on the CI gates
(probably due to a race in which Nova calls the ironic-api service before
ironic-conductor is up). RPC calls made via the indirection_api should
be enabled one-by-one, when the implications are fully understood.
Change-Id: Ia381348da93f95d764c83f3ec2a2ed5a1db5ad6d
Closes-Bug: 1493816
Depends-On: I81400305f166d62aa4612aab54602abb8178b64c
1. After migrate to oslo.versionedobjects library, dt_serializer()
and dt_deserializer() are no longer needed.
2 check_object_version() and obj_to_primitive() were not used
at all, so remove them as well.
Partial-Bug: #1461239
Change-Id: I31af71fc3d6e88177d1720c782d67abcb017710f
Use remotable decorators from oslo.versionedobjects.
remotable() treats context as option parameter, so change some tests
accordingly.
Change-Id: I84aef23cfa13a6162120e048477228015defc7e6
Make IronicObject based on VersionedObject and introduce
VersionedObjectDictCompat class from oslo library. This is only a
temporary solution to support object['key']. Object should only be
called like object.key
Make all objects multiple inherit from these two classes can reduce the
future work to remove the support of IronicObjectDictCompat. This will be
addressed in next patches once finish the migration to oslo.versionedobject
library.
Partial-Bug: #1461239
Change-Id: Ieb2c406c10da75f9c42320085c563c166eda1703
Previously, objects are registered through metaclass, which is
complicate to understand. So oslo.versionedobjects move this to
ObjectRegistry class. Each object is registered with a decorator.
This also includes registering some test-only classes,
which use the register_if(False) scheme to do the object-building step,
but without actually inserting the class into the registry. This prevents
those test classes from ever being registered at runtime by accident
(and thus send/receive-able over RPC).
Co-Author: Dan Smith <dms@danplanet.com>
Partial-Bug: #1461239
Change-Id: Ice4d47185b888d1b33e761d7d1437bb20abb3d93
This removes the _Callable hack in ironic.objects.fields, and fixes all the
remaining cases where old-style fields were used. This change has been
complete in nova for probably two years now, and is a necessary thing
to cleanup before progressing further with oslo.versionedobjects
conversions.
Change-Id: I43c45ce4d3ce4de04e43b68f24435600aaefb74f
Related-Bug: #1461239
ObjectListBase is not used in Ironic at the moment, we should add it
when it is necessary.
Partial-Bug: #1488270
Change-Id: Ifc2aaa1fa09d95f1cfa6ed488f10871666b7fb3d
This replace Ironic's IronicObjectSerializer with
oslo.versionedobjects's VersionedObjectSerializer.
IPA's lookup should not return node as object anymore, because
Node object is not Json serializable. So return node as a pure dict.
Partial-Bug: #1461239
Change-Id: Ieec505613469536d941e54548cc228e5d05708b0
This change refactors the "refresh" method in ironic.objects.chassis,
ironic.objects.conductor, ironic.objects.node and ironic.objects.port to
reuse a more generic method added to ironic.objects.base.IronicObject
for identifying and applying changes of fields. It make the code in
refresh method more simpler to understand by removing details of
implementation, in addition decreases the code replication in four
classes.
Change-Id: I6171203690cfd282fa1250dbe0204151f5c665fb
The iteritems() of Python 2 dictionaries has been renamed to items()
to provide compatibility with Python 3.
Change-Id: Ia0e9f2fc5c2bbc5b1465bf38f2d05533526dc0e6
This patch also makes the Field object able to be called like a function,
which emulates the old interface that objects are expecting. This should
be removed once objects/base.py is updated to stop using the old interface
for the fields
Partial-Bug: #1461239
Change-Id: I34c0203e40d249348626a65584baf2e68cb1b642
Trivial steps to start using oslo.versionedobjects in Ironic, this patch
simply replace the dict compat class with the one in the library and
make the oslo.versionedobjects library a requirement.
Partial-Bug: #1461239
Change-Id: If4d2583f0d107d817539b60341861e3c2a29b6c1
versionutils from oslo.incubator was moved to oslo_utils (in v1.6.0).
This changes our code to use versionutils from oslo.incubator, and
deletes ironic's copy of oslo.incubator's versionutils.
Change-Id: I24c411ba14d99b02f8a22c6cf4f53b44508c9058
Closes-Bug: #1468956
This patch enforces the rules E123, E126, E127, and E128 in the ironic
code base:
E123 - closing bracket does not match indentation of opening
bracket’s line
E126 - continuation line over-indented for hanging indent
E127 - continuation line over-indented for visual indent
E128 - continuation line under-indented for visual indent
This fixes any parts of the current code which fails these rules and
removes these rules from the tox.ini flake8 ignore list.
Change-Id: Ia96582b5e9abc088d6c1694afc93c59be4a4065c
Closes-Bug: 1421522
Fix new warning that are introduced with hacking 0.10.x.
The failure messages fixed are:
H238 old style class declaration, use new style (inherit from `object`)
H105 Don't use author tags
H501 Do not use self.__dict__ for string formatting
Change-Id: Ia070d60754dfd85966178eadbf82a4d98b64407a
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: I695d2141c00a5ee36e042efbb9bac4e2803c1948
A lot of fixes to be compatible with python 3:
- fix encoding/decoding errors
- fix issues with comparison
- use `reload`, `reraise`, ext. modules from six
- use items() instead of iteritems()
- add a new file with py3 specific test requirements
- drop passing the arbitrary arguments to object.__new__ method.
See bug [1] for more details.
- add a workaround to bug in `mock` library
- add py33 and py34 test environment to tox.ini
[1] http://bugs.python.org/issue1683368
Change-Id: I90936cb6b6eaaf4b5e1ce67732caec3c8bdc1cc2
Use graduated library oslo_context and remove any reference
to the context module from oslo.incubator
Change-Id: I590f033c4b164cea9022cd951ded393b7406381e
Fixes files with the PEP8 error:
H405 multi line docstring summary not separated with an empty line
and enables the PEP8 check in tox.ini.
Change-Id: I632c89e2a1783be8305a8a8cea7ae9d3bb55b62c
RPC objects should have a context within it, this patch is making passing
the context mandatory when instantiating the object and is a plumbing
work to remove passing the context later for things like create(),
refresh(), destroy() and save().
Partial-Bug: #1314732
Change-Id: If9b175fa874bcb96c77cf22d176f1111f450f796
This part fixes ironic/conductor, ironic/dhcp and ironic/objects
folders.
Some modules use _ function for log translation, even when the
translated string is passed to e.g. LOG.warn function. This patch
fixes it, using translator functions appropriate for corresponding
log levels. Translator functions, imported from
ironic.openstack.common.gettextutils are now imported from
ironic.common.i18n. Translator functions are imported the same way
across all modules. All places that are changed conform to
http://docs.openstack.org/developer/oslo.i18n/guidelines.html
Partial-bug: #1364813
Change-Id: Ib4b93cc49f16836a95c7cfb4c5f1705f5cbbb31f
Julia Kreger