432 lines
15 KiB
ReStructuredText
432 lines
15 KiB
ReStructuredText
..
|
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
|
License.
|
|
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
|
|
|
======================================
|
|
Metadata for all user facing resources
|
|
======================================
|
|
|
|
Launchpad blueprint:
|
|
|
|
https://blueprints.launchpad.net/manila/+spec/metadata-for-share-resources
|
|
|
|
Manila stores information regarding user-owned tangible technical assets in
|
|
its database as resources. These assets include file shares, snapshots,
|
|
access control rules, export locations, share networks and security services
|
|
among others. Resources are designed to accommodate distinguishable identifiers
|
|
geared at machines and humans such as ID, Name and Description. However
|
|
these identifiers may be insufficient or inflexible to end users' needs such
|
|
as being able to tag resources to a purpose, or attach labels that drive some
|
|
automation. This specification proposes a design allowing for user
|
|
modifiable metadata on all user facing resources.
|
|
|
|
|
|
Problem description
|
|
===================
|
|
|
|
End users interact with manila API to manage their shared file system storage
|
|
resources. These resources include:
|
|
|
|
- Shares
|
|
- Share Export Locations
|
|
- Share Access Rules
|
|
- Share Instances
|
|
- Share Instance Export Locations
|
|
- Share Replicas
|
|
- Share Replica Export Locations
|
|
- Share Snapshots
|
|
- Share Groups
|
|
- Share Group Snapshots
|
|
- Security Services
|
|
- Share Networks
|
|
- Share Network Subnets
|
|
|
|
All of these resources are identifiable by ID. Some of them even allow setting
|
|
free form text as Name and/or Description. Free form text is usually hard to
|
|
construct, breakdown or query, so these fields are mostly useful to humans
|
|
rather than machines. Many times, users want to be able to store much
|
|
more auxiliary contextual information.
|
|
|
|
One way they do that today with shares and access rules is with the help of
|
|
"metadata". Metadata is a term used to describe auxiliary information
|
|
associated with some data. For shares and access rules, users today use
|
|
metadata as flexible key=value pairs to attach useful optional/additional
|
|
information. This sort of interaction isn't possible with the other
|
|
resources enumerated above.
|
|
|
|
Metadata isn't just helpful for user interactions. It could also provide
|
|
additional representational space for manila itself. For example, the service
|
|
currently uses database objects to represent export location metadata to
|
|
relay helpful hints regarding the nature of specific exports to end users,
|
|
such as the preference and accessibility. This metadata is owned by the
|
|
service, and end users cannot modify this metadata.
|
|
|
|
Use Cases
|
|
=========
|
|
|
|
Users may want to use metadata to store contextual information such as:
|
|
|
|
- the purpose of the resource, e.g.: "usedfor=fileserver"
|
|
- details regarding the provisioner, e.g.: "createdby=manila-csi"
|
|
- grouping information, e.g.: "department=physics"
|
|
- additional chronological information, e.g.: "last_audited=2020-12-24T00:22:29"
|
|
- mount point tag for clients, e.g.: "fstag=website"
|
|
|
|
Metadata interactions are expected to be uniform and consistent across
|
|
resources, which makes it easier for building automation on top of these APIs.
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
A new metadata controller mixin will be created in manila's API modules that
|
|
can be inherited by all the user facing API controllers. This mixin will
|
|
implement API handler code to:
|
|
|
|
- get resource metadata (retrieve either all of it, or specific item)
|
|
- set/unset all resource metadata (create, update or delete metadata items as
|
|
key=value pairs)
|
|
- set a single resource metadata item (update a specific key=value pair)
|
|
- unset resource metadata item (delete a specific key=value pair)
|
|
|
|
To distinguish and protect service owned or admin only metadata,
|
|
there will be an ignore_keys parameter and ignored keys dictionary
|
|
to prevent these metadata items from being manipulated by the end user.
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
Alternatives to key-value metadata include Name and Description fields.
|
|
Resources that don't currently support name and/or description can be
|
|
updated to include these fields. This alternative suffers from the
|
|
inflexibility problems described above and doesn't cover all the use cases.
|
|
|
|
We could also just add metadata controllers one at a time, based on user
|
|
feedback instead of adding metadata controllers for each user facing
|
|
resource. This also allows us to test each metadata controller in isolation
|
|
and build incrementally. This is pretty non-controversial, however, each
|
|
resource API update will have to be made in a new microversion, and doing
|
|
things one at a time instead of using the inheritence pattern to subsume
|
|
common code will be costlier. Testing effort is significant, and unavoidable.
|
|
|
|
Instead of distinguishing "user" owned and "service" owned metadata, we
|
|
could allow all metadata to be modifiable by end users - even data that's
|
|
created by the service. Doing so would simplify user interactions, however,
|
|
it may harm the service and cause misbehavior.
|
|
|
|
Another option to including a ``user_modifiable`` boolean field to metadata
|
|
is to introduce granular RBAC. However, this may produce too many RBAC rules
|
|
that may be unnecessary. If this turns out to be a use case, it is possible
|
|
to easily build on top of the proposed design.
|
|
|
|
Data model impact
|
|
-----------------
|
|
|
|
To store resource metadata, new tables and corresponding ORM models are
|
|
necessary. These tables will be created during a database upgrade, and
|
|
nullified and destroyed during a database downgrade.
|
|
|
|
Existing metadata tables for Shares ("share_metadata"), Export Locations
|
|
("share_instance_export_locations_metadata") will be modified to include
|
|
a string attribute called ``deleted``. The default value for this attribute is
|
|
set to False.
|
|
|
|
Existing metadata tables will also be modified to replace the datatype of
|
|
the "id" field to a string of length 36 for UUIDs. This will be done to
|
|
avoid integer overflows and provide scalability.
|
|
|
|
A general ORM schema for a metadata table will be as follows
|
|
(where 'Resource' is a stand-in for the real resource name)::
|
|
|
|
class ResourceMetadata(BASE, ManilaBase):
|
|
"""Represents a metadata key/value pair for a Resource."""
|
|
__tablename__ = 'resource_metadata'
|
|
id = Column(String(36), primary_key=True)
|
|
key = Column(String(255), nullable=False)
|
|
value = Column(String(1023), nullable=False)
|
|
resource_id = Column(String(36), ForeignKey('resources.id'), nullable=False)
|
|
deleted = Column(String(36), default='False')
|
|
resource = orm.relationship(Resource, backref="resource_metadata",
|
|
foreign_keys=resource_id,
|
|
primaryjoin='and_('
|
|
'ResourceMetadata.resource_id == Resource.id,'
|
|
'ResourceMetadata.deleted == "False)')
|
|
|
|
|
|
Metadata items are not soft deleted when they are unset by the service or by
|
|
end users. The metadata table is not loaded alongside the resource unless
|
|
the resource has been queried with metadata, or a detailed view of the
|
|
resource has been requested.
|
|
|
|
REST API impact
|
|
---------------
|
|
|
|
New API endpoints will be created to index metadata, show metadata item,
|
|
create metadata, update metadata item, update_all metadata (delete
|
|
all existing metadata and update with requested metadata), and delete metadata
|
|
item for each resource. The general structure of these APIs is as follows:
|
|
|
|
Index Metadata
|
|
^^^^^^^^^^^^^^^
|
|
|
|
Retrieve all metadata key=value pairs as JSON::
|
|
|
|
GET /v2/{resource}/metadata
|
|
|
|
- *Sample request body*: null
|
|
- *Success Codes*: 200
|
|
- *Default API policy role*: Project Reader
|
|
- *Error Codes*: 401 (Unauthorized), 403 (Policy Not Authorized), 404
|
|
(Invalid resource)
|
|
- *Sample response body*::
|
|
|
|
{
|
|
"metadata": {
|
|
"project": "my_app",
|
|
"aim": "doc"
|
|
}
|
|
}
|
|
|
|
Show specific metadata item
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Retrieve a single metadata key=value pair::
|
|
|
|
GET /v2/{resource}/metadata/{key}
|
|
|
|
- *Sample request body*: null
|
|
- *Success Codes*: 200
|
|
- *Default API policy role*: Project Reader
|
|
- *Error Codes*: 401 (Unauthorized), 403 (Policy Not Authorized), 404
|
|
(Invalid resource)
|
|
- *Sample response body*::
|
|
|
|
{
|
|
"metadata": {
|
|
"project": "my_app",
|
|
}
|
|
}
|
|
|
|
Update all metadata
|
|
^^^^^^^^^^^^^^^^^^^
|
|
|
|
Replace all metadata with the updated set, can also be used to delete all
|
|
metadata::
|
|
|
|
PUT /v2/{resource}/metadata
|
|
|
|
- *Sample request body*::
|
|
|
|
{
|
|
"metadata": {
|
|
"aim": "changed_doc",
|
|
"project": "my_app",
|
|
"new_metadata_key": "new_information"
|
|
}
|
|
}
|
|
|
|
- *Success Codes*: 200
|
|
- *Default API policy role*: Project Member
|
|
- *Error Codes*: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource), 400 (Malformed request)
|
|
- *Sample response body*::
|
|
|
|
{
|
|
"metadata": {
|
|
"aim": "changed_doc",
|
|
"project": "my_app",
|
|
"new_metadata_key": "new_information"
|
|
}
|
|
}
|
|
|
|
*Note:* Metadata keys that are part of the admin-only dictionary will
|
|
not be deleted and updated if the user requesting this is not an
|
|
system or project admin.
|
|
|
|
Update specific metadata item
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Update a specific metadata item, leaving the rest unmodified::
|
|
|
|
POST /v2/{resource}/metadata/{key}
|
|
|
|
- *Sample request body*::
|
|
|
|
{
|
|
"metadata": {
|
|
"aim": "updated_doc",
|
|
}
|
|
}
|
|
|
|
- *Success Codes*: 200
|
|
- *Default API policy role*: Project Member
|
|
- *Error Codes*: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource), 400 (Malformed request)
|
|
- *Sample response body*::
|
|
|
|
{
|
|
"metadata": {
|
|
"aim": "updated_doc",
|
|
"project": "my_app",
|
|
"new_metadata_key": "new_information"
|
|
}
|
|
}
|
|
|
|
|
|
.. important::
|
|
|
|
Currently, the ``POST /v2/{share}/metadata`` API currently expects a
|
|
``meta`` object. However, the other metadata APIs expect a ``metadata``
|
|
object. For the sake of consistency, this error will be fixed in a new
|
|
API microversion.
|
|
|
|
Delete specific metadata item
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Hard delete a single metadata key=value pair::
|
|
|
|
DELETE /v2/{resource}/metadata/{key}
|
|
|
|
- *Sample request body*: null
|
|
- *Success Codes*: 200
|
|
- *Default API policy role*: Project Member
|
|
- *Error Codes*: 401 (Unauthorized), 403 (Policy Not Authorized), 404
|
|
(Invalid resource)
|
|
- *Sample response body*: null
|
|
|
|
Query resources by metadata items
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
URL encoding can be performed by the client::
|
|
|
|
GET /v2/{resource}?metadata=%7B%27foo%27%3A%27bar%27%2C%27clemson%27%3A%27tigers%27%7D
|
|
|
|
or the request can be made in a decoded format as well::
|
|
|
|
GET /v2/{resource}?metadata={'foo':'bar','clemson':'tigers'}
|
|
|
|
Driver impact
|
|
-------------
|
|
|
|
None. Metadata manipulation is directly performed on the manila database and
|
|
shared file system back end drivers are not invoked during the creation,
|
|
modification or deletion of resource metadata.
|
|
|
|
Security impact
|
|
---------------
|
|
|
|
It is advised that metadata operations are rate limited to prevent bad
|
|
actors or automation from adding a large number of metadata items to a
|
|
resource.
|
|
|
|
Notifications impact
|
|
--------------------
|
|
|
|
None
|
|
|
|
Other end user impact
|
|
---------------------
|
|
|
|
Python-manilaclient SDK will include support for the new APIs and we'll
|
|
ensure that there are corresponding CLI commands in the new OSC plugin
|
|
shell. Manila UI's support for share, export location and access rule
|
|
metadata is limited. This specification doesn't seek to address all the
|
|
UI gaps; but all effort will be made to close the feature parity between
|
|
the CLI utilities and the UI. Eventually users will be able to perform
|
|
all metadata interactions via the UI as well.
|
|
|
|
|
|
Performance Impact
|
|
------------------
|
|
|
|
API performance is bound to suffer when resource queries include metadata
|
|
items. Since we'll be providing metadata along with ``detail`` retrievals of
|
|
resources, the performance of those APIs will also be affected negatively
|
|
because of the new database joins that will be necessary. Over time, as the
|
|
number of shares and the metadata tables grow, performance degradation can
|
|
be severe. This impact will be documented; as best practice, it is
|
|
recommended that a resource have no more than a handful of metadata items.
|
|
|
|
Other deployer impact
|
|
---------------------
|
|
|
|
New APIs introduced may require tweaking of policy files if the default RBAC
|
|
policy is not acceptable.
|
|
|
|
Developer impact
|
|
----------------
|
|
|
|
When adding any new user facing metadata, the metadata mixin controller can
|
|
be inherited and extended by developers.
|
|
|
|
|
|
Implementation
|
|
==============
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignee:
|
|
ashrod98 <ashrod98@gmail.com>
|
|
|
|
Other contributors:
|
|
gouthamr
|
|
|
|
Work Items
|
|
----------
|
|
|
|
- Add database migration to convert the ``id`` field of share, export
|
|
location and access rule metadata to a string from integer and populate
|
|
the field with UUIDs
|
|
- Add database migrations to introduce the "deleted" field to share,
|
|
export location and access rule metadata tables.
|
|
- Add database migrations to create new metadata tables for all other resources
|
|
- Add ``MetadataControllerMixin``, inherit and extend in all resources and
|
|
bump up the API microversion.
|
|
- Add unit and integration tests
|
|
- Add support for metadata APIs in manilaclient SDK, and OSC CLI and SDK
|
|
- Add support for metadata interactions in the UI
|
|
- Add documentation
|
|
|
|
A further enhancement to this API would be to provide an interface
|
|
for administrators to create admin-only metadata values. Currently,
|
|
admin-only metadata values are delinated in a dictionary in
|
|
manila/manila/common/constants.py. Such a fixated list is sufficient for
|
|
the implementation of backend filtering for scheduling according to share
|
|
affinities. To implement this customization, we could revisit adjusting
|
|
the data-model to include an admin-only boolean value identifying
|
|
which keys are adjustable by admins or non-admins. Or we could provide
|
|
some configuration option in manila/manila/data/. Such an enhancement
|
|
requires further thought, and can be implemented at a later point.
|
|
|
|
Dependencies
|
|
============
|
|
|
|
Not a direct dependency, but this API change incorporates the metadata
|
|
changes necessary to implement this spec: `Affinity and anti-affinity
|
|
scheduler filter <https://specs.openstack.org/openstack/manila-specs/specs/xena/affinity-antiaffinity-filter.html>`_
|
|
|
|
|
|
Testing
|
|
=======
|
|
|
|
Extensive unit tests will be written to test the API and database methods
|
|
being added. A database "walk migrations" test will be added for all the
|
|
database changes. Tempest tests will be added to cover the new metadata
|
|
operations across resources.
|
|
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
- API reference
|
|
- End user guide
|
|
- Release notes
|
|
|
|
|
|
References
|
|
==========
|
|
|
|
* `Manila API Reference <https://docs.openstack.org/api-ref/shared-file-system>`_
|
|
* `API SIG Guideline on Metadata <https://specs.openstack.org/openstack/api-sig/guidelines/metadata.html>`_
|
|
* `Wallaby PTG Discussion Etherpad <https://etherpad.opendev.org/p/wallaby-ptg-manila-metadata-controller>`_
|