11 KiB
Metadata Assignment And Propagation
https://blueprints.launchpad.net/murano/+spec/metadata-assignment-and-propagation
Cloud users and operators may need to assign some arbitrary key-value pairs to environments and applications deployed by murano to associate some metadata information with them. If applicable this metadata attributes should be propagated to OpenStack resources provisioned by Murano.
Problem description
The meta information may describe various aspects of the deployed applications, for example:
- Deployed software: names, versions, descriptions and license terms of the software components deployed on the VMs;
- VM-related configuration features, like listening ports or interfaces;
- Hardware specs of provisioned virtual machines;
- Guest OS info, like the type of operating system, its family, version etc;
- Information about the deployments purpose: type of the deployed environment (development, staging, production), usage purpose, security consideration etc;
- Information about the user who deployed the app or environment: name, department, contact information, etc.
This information may be consumed directly by inspecting murano applications with dashboard or CLI, by indexing these metadata attributes into specialized search software, such as Searchlight [1] or by building custom reports with grouping or aggregation by these metadata.
For the latter case it is especially important to propagate the metadata from the initial objects it is assigned to (application or environment) to the resource objects spawned by them (VMs, Volumes, Networks etc).
Proposed change
UI change
It is proposed to utilize the functionality provided by Glance Meta
Definition Catalog [2] to define the schemas of the applicable metadata
attributes. It is proposed to add two types of resources for murano
applications and murano environments:
OS::Murano::Application
and
OS::Murano::Environment
accordingly (ability to add
metadata attributes for murano packages is out of scope for this spec,
however in future this may also become possible, so
OS::Murano::Package
is likely to be appropriate resource
type).
The standard Angular-based Metadata assignment dialog (the one being currently used to assign metadata to Instances, Images and Volumes) should be extended to support Murano Dashboard and these new resource types. This extended dialog will load the metadata definition schemas for attributes which are assignable to appropriate type of resources and will submit them to Murano API for persistence once the input is saved.
Murano Object Model change
For the applications it is proposed to persist the metadata attributes in the object model, in the new block called 'metadata', which should be placed under the '?' section of each object. For example, the '?' block of Apache application tagged with a couple of attributes may look like this:
"?": {
"_26411a1861294160833743e45d0eaad9": {
"name": "Apache HTTP Server"
},
"type": "com.example.apache.ApacheHttpServer",
"id": "19dfb844-6eee-48bd-826a-c6206d065e85",
"metadata": {
"sw_webserver_apache_version": "2.1",
"sw_webserver_apache_http_port": 80
}
}
Since the ?-headers of objects placed in the environment's object model may be modified with the existing Murano API no changes in the API is needed.
For the Environments a similar approach may be followed — to store the metadata attributes under the /?/metadata key of the Environment object in the object model. However an alternative may be considered: since each Environments have a dedicated entry in murano's database, the metadata attributes for environments may be stored in database in normalized form. See the "Alternatives" section below for more details.
Regardless of the chosen approach it is required to introduce a new API call to assign metadata attributes to the Environment, since the existing API does not have capabilities to edit the ?-headers of Environment object. The appropriate change into API is out of scope here and is covered with a separate spec [3]
MuranoPL Change
MuranoPL language should be extended with an ability to read the
metadata attributes assigned to any given object. It is proposed to
create a global yaql function named metadata()
for this
purpose (similar to id()
and name()
which read
appropriate information from the same ?-header).
Murano's serializer/deserializer should be properly updated to load the metadata attributes and persist them back.
Core Library Change
Core library should be extended with an ability to communicate with
Glance Metadata Definition API to query the namespaces applicable to a
given resource type and extract all the names of attributes from those
namespaces. This will allow the resource class to check if the given
metadata attribute attached to the application the class belongs to is
applicable for the resource itself. The binding to the Glance API should
be done via a python-backed class MetadefBrowser
, while the
rest of the logic may be implemented in MuranoPL.
A new MuranoPL class called MetadataAware
should be
implemented. This class will have two methods:
collectMetadata
and getResourceType
.
collectMetadata
method will return all the metadata
attributes assigned to the object itself and all the attributes assigned
to the owners of the object, traversing the ownership tree to the root
(i.e. to the Environment). All the attributes from the traversed objects
may be checked for applicability to the particular resource type. To do
this check, the code should call the getResourceType
method
which is supposed to return the resource type as defined in Glance
(OS::Nova::Server
for VMs, OS::Cinder::Volume
for volumes etc) and then use the MetadefBrowser
described
above to check if the attribute is applicable to the objects of this
type.
The resource-specific classes of the Core Library
(Instance
, CinderVolume
etc) have to extend
the MetadataAware
and override the
getResourceType
method.
Alternatives
There are a number of alternatives to consider.
First, instead of using Glance Metadef catalog to store the possible attribute names and constraints, we may assign attributes without any predefined schema. This will reduce our dependency on other OpenStack components (Glance in this case), but will introduce extra development burden (since we won't be able to reuse the existing metadata-assignment dialog) and will prevent us from using convenient schema-based validators of tags which are made for metadefs. So, it is proposed to stay with metadef-compliant implementation.
Second option to consider is the storage of metadata attributes on environment level. Most other OpenStack projects store metadata attributes in normalized form: they have the separate table mapped to the primary object table as one-to-many. The approach suggested above uses denormalized storage: the attributes are stored inside the json-document describing the complete environment, which is stored as a single value in the database. This is much easier to implement, also having metadata in object model of environment object is still needed, so duplicating data storage seems like an extra work.
Third option to consider: not to check for attribute applicability on
resource types. This simplifies the development (no need to create
MetadefBrowser
class and much simpler logic in
MetadataAware
), however this will degrade user experience,
as resources may get inappropriate attributes assigned, which will be
confusing.
Data model impact
This proposal adds new fields to the ?-headers of objects in object
model. If the normalized alternative is chosen, it will also introduce
extra table in database and a relationship with the
Environments
table. This will require to create a database
migration as well.
REST API impact
A modification of Murano API is required to allow assigning of metadata attributes to the Environment objects, however it is out of scope here. See [3] for details.
Versioning impact
Since the proposed change requires to add a new yaql function to the DSL and modifies the serializer/deserializer logic this requires to bump the MuranoPL format version.
Other end user impact
The users will see new action buttons "Update Metadata" in Murano Dashboard. This buttons will be available as row actions for Environment Components and Environments.
Deployer impact
The proposed change introduces a dependency on Glance API. To configure the connectivity to this API (endpoint details, encryption etc) a new optional configuration block should be added to Murano config.
Developer impact
Developers may implement the MetadataAware
class to add
metadata handling logic for their classes.
Murano-dashboard / Horizon impact
This change will add a custom version of metadata.service javascript module to Murano Dashboard. It will have the similar logic as the standard Horizon's module, but will add the references to murano api's, so the Metadata dialog can update the attributes assigned to Murano entities.
Implementation
Assignee(s)
- Primary assignee:
-
ativelkov
- Other contributors:
-
TBD
Work Items
- Add a rest api handlers for murano-dashboard to handle metadata updates and fetching;
- Add a javascript API to this handlers;
- Create a custom version of metadata.service, so the metadata modal dialog may utilize murano's javascript APIs to metadata;
- Modify Murano Dashboard to call metadata modal dialog as row handlers for Environments and Services;
- Modify Murano API to allow edits of ?-section of Environment objects;
- Add a
MetadefBrowser
class to Core Library; - Implement a
MetadataAware
class and extendInstance
andCinderVolume
classes with it.
Dependencies
- Glance Metadefinition Catalog [2]
- Environment modification API changes [3]
Testing
The integration tests should verify that the attributes are propagated properly
Documentation Impact
The new capabilities to add attributes should be documented in user
manual. The changes in core library and the usage of
MetadataAware
class should be reflected in developers'
manual.
References
[1] https://wiki.openstack.org/wiki/Searchlight [2] http://docs.openstack.org/developer/glance/glancemetadefcatalogapi.html [3] https://review.opendev.org/#/c/378602