From b8f4ff7b1b600ed58b7223fbd2a0032d3cdda008 Mon Sep 17 00:00:00 2001 From: Goutham Pacha Ravi Date: Thu, 24 Dec 2020 03:12:56 -0800 Subject: [PATCH] Add metadata for share resources Proposal to accommodate metadata and associated operations on all user facing share resources (e.g.: snapshots, groups, share networks, security services, etc.) APIImpact Partially-Implements: bp metadata-for-share-resources Change-Id: I4d276311e3cd42ad9f0fab78a9b2c850c5727bb6 Signed-off-by: Goutham Pacha Ravi --- .../wallaby/metadata-for-share-resources.rst | 393 ++++++++++++++++++ 1 file changed, 393 insertions(+) create mode 100644 specs/wallaby/metadata-for-share-resources.rst diff --git a/specs/wallaby/metadata-for-share-resources.rst b/specs/wallaby/metadata-for-share-resources.rst new file mode 100644 index 0000000..eed4ebf --- /dev/null +++ b/specs/wallaby/metadata-for-share-resources.rst @@ -0,0 +1,393 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +====================================== +Metadata for all user facing resources +====================================== + +Launchpad blueprint: + +https://blueprints.launchpad.net/manila/+spec/metadata-for-share-resources + +Manila stores information regarding user-owned tangible technical assets in +its database as resources. These assets include file shares, snapshots, +access control rules, export locations, share networks and security services +among others. Resources are designed to accommodate distinguishable identifiers +geared at machines and humans such as ID, Name and Description. However +these identifiers may be insufficient or inflexible to end users' needs such +as being able to tag resources to a purpose, or attach labels that drive some +automation. This specification proposes a design allowing for user +modifiable metadata on all user facing resources. + + +Problem description +=================== + +End users interact with manila API to manage their shared file system storage +resources. These resources include: + +- Shares +- Snapshots +- Export Locations +- Share Replicas +- Share Groups +- Share Group Snapshots +- Security Services +- Share Networks +- Share Network Subnets + +All of these resources are identifiable by ID. Some of them even allow setting +free form text as Name and/or Description. Free form text is usually hard to +construct, breakdown or query, so these fields are mostly useful to humans +rather than machines. Many times, users want to be able to store much +more auxiliary contextual information. + +One way they do that today with shares and access rules is with the help of +"metadata". Metadata is a term used to describe auxiliary information +associated with some data. For shares and access rules, users today use +metadata as flexible key=value pairs to attach useful optional/additional +information. This sort of interaction isn't possible with the other +resources enumerated above. + +Metadata isn't just helpful for user interactions. It could also provide +additional representational space for manila itself. For example, the service +currently uses database objects to represent export location metadata to +relay helpful hints regarding the nature of specific exports to end users, +such as the preference and accessibility. This metadata is owned by the +service, and end users cannot modify this metadata. + +Use Cases +========= + +Users may want to use metadata to store contextual information such as: + +- the purpose of the resource, e.g.: "usedfor=fileserver" +- details regarding the provisioner, e.g.: "createdby=manila-csi" +- grouping information, e.g.: "department=physics" +- additional chronological information, e.g.: "last_audited=2020-12-24T00:22:29" +- mount point tag for clients, e.g.: "fstag=website" + +Metadata interactions are expected to be uniform and consistent across +resources, which makes it easier for building automation on top of these APIs. + +Proposed change +=============== + +A new metadata controller mixin will be created in manila's API modules that +can be inherited by all the user facing API controllers. This mixin will +implement API handler code to: + +- get resource metadata (retrieve either all of it, or specific item) +- set/unset all resource metadata (create, update or delete metadata items as + key=value pairs) +- set a single resource metadata item (update a specific key=value pair) +- unset resource metadata item (delete a specific key=value pair) + +To distinguish and protect service owned metadata, all metadata tables will +include an attribute named ``user_modifiable``. This attribute will +determine if users have the ability to manipulate (or delete) +the specific metadatum. + +Alternatives +------------ + +Alternatives to key-value metadata include Name and Description fields. +Resources that don't currently support name and/or description can be +updated to include these fields. This alternative suffers from the +inflexibility problems described above and doesn't cover all the use cases. + +We could also just add metadata controllers one at a time, based on user +feedback instead of adding metadata controllers for each user facing +resource. This also allows us to test each metadata controller in isolation +and build incrementally. This is pretty non-controversial, however, each +resource API update will have to be made in a new microversion, and doing +things one at a time instead of using the inheritence pattern to subsume +common code will be costlier. Testing effort is significant, and unavoidable. + +Instead of distinguishing "user" owned and "service" owned metadata, we +could allow all metadata to be modifiable by end users - even data that's +created by the service. Doing so would simplify user interactions, however, +it may harm the service and cause misbehavior. + +Another option to including a ``user_modifiable`` boolean field to metadata +is to introduce granular RBAC. However, this may produce too many RBAC rules +that may be unnecessary. If this turns out to be a use case, it is possible +to easily build on top of the proposed design. + +Data model impact +----------------- + +To store resource metadata, new tables and corresponding ORM models are +necessary. These tables will be created during a database upgrade, and +nullified and destroyed during a database downgrade. + +Existing metadata tables for Shares ("share_metadata"), Export Locations +("share_instance_export_locations_metadata") and Share Access +Rules ("share_access_rules_metadata") will be modified to include a boolean +attribute called ``user_modifiable``. The default value for this attribute is +set to True. All existing export location metadata will have this attribute +set to False during the database upgrade step. This is because we expect no +end user created metadata on export locations yet. + +Existing metadata tables will also be modified to replace the datatype of +the "id" field to uuids. This will be done to avoid integer overflows and +provide scalability. + +A general ORM schema for a metadata table will be as follows:: + + class ResourceMetadata(BASE, ManilaBase): + """Represents a metadata key/value pair for a resource.""" + __tablename__ = 'resource_metadata' + id = Column(String(36), primary_key=True) + key = Column(String(255), nullable=False) + value = Column(String(1023), nullable=False) + resource_id = Column(String(36), ForeignKey('resources.id'), nullable=False) + user_modifiable = Column(Boolean, default=True) + resource = orm.relationship(Resource, backref="resource_metadata", + foreign_keys=resource_id, + primaryjoin='and_(' + 'ResourceMetadata.resource_id == Resource.id,' + 'ResourceMetadata.deleted == 0)') + + +Metadata items are not soft deleted when they are unset by the service or by +end users. The metadata table is not loaded alongside the resource unless +the resource has been queried with metadata, or a detailed view of the +resource has been requested. + +REST API impact +--------------- + +New API endpoints will be created to get metadata, set metadata, unset a +metadata item, delete all metadata for each resource. The general structure of +these APIs is as follows: + +Get all Metadata +^^^^^^^^^^^^^^^^ + +Retrieve all metadata key=value pairs as JSON:: + + GET /v2/{resource}/metadata + +- *Sample request body*: null +- *Success Codes*: 200 +- *Default API policy role*: Project Reader +- *Error Codes*: 401 (Unauthorized), 403 (Policy Not Authorized), 404 + (Invalid resource) +- *Sample response body*:: + + { + "metadata": { + "project": "my_app", + "aim": "doc" + } + } + +Set/Unset all metadata +^^^^^^^^^^^^^^^^^^^^^^ + +Replace all metadata with the updated set, can also be used to delete all +metadata:: + + PUT /v2/{resource}/metadata + +- *Sample request body*:: + + { + "metadata": { + "aim": "changed_doc", + "project": "my_app", + "new_metadata_key": "new_information" + } + } + +- *Success Codes*: 200 +- *Default API policy role*: Project Member +- *Error Codes*: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource), 400 (Malformed request) +- *Sample response body*:: + + { + "metadata": { + "aim": "changed_doc", + "project": "my_app", + "new_metadata_key": "new_information" + } + } + +Set specific metadata item/s +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Update one or more specific metadata items, leaving the rest unmodified:: + + POST /v2/{resource}/metadata/{key} + +- *Sample request body*:: + + { + "metadata": { + "aim": "updated_doc", + } + } + +- *Success Codes*: 200 +- *Default API policy role*: Project Member +- *Error Codes*: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource), 400 (Malformed request) +- *Sample response body*:: + + { + "metadata": { + "aim": "updated_doc", + "project": "my_app", + "new_metadata_key": "new_information" + } + } + + +.. important:: + + Currently, the ``POST /v2/{share}/metadata`` API currently expects a + ``meta`` object. However, the other metadata APIs expect a ``metadata`` + object. For the sake of consistency, this error will be fixed in a new + API microversion. However, use of ``meta`` will be honored for this API even + after the new microversion. + +Delete specific metadata item +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Hard delete a single metadata key=value pair:: + + DELETE /v2/{resource}/metadata/{key} + +- *Sample request body*: null +- *Success Codes*: 200 +- *Default API policy role*: Project Member +- *Error Codes*: 401 (Unauthorized), 403 (Policy Not Authorized), 404 + (Invalid resource) +- *Sample response body*: null + +Query resources by metadata items +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +URL encoding can be performed by the client:: + + GET /v2/{resource}?metadata=%7B%27foo%27%3A%27bar%27%2C%27clemson%27%3A%27tigers%27%7D + +or the request can be made in a decoded format as well:: + + GET /v2/{resource}?metadata={'foo':'bar','clemson':'tigers'} + +Driver impact +------------- + +None. Metadata manipulation is directly performed on the manila database and +shared file system back end drivers are not invoked during the creation, +modification or deletion of resource metadata. + +Security impact +--------------- + +It is advised that metadata operations are rate limited to prevent bad +actors or automation from adding a large number of metadata items to a +resource. + +Notifications impact +-------------------- + +None + +Other end user impact +--------------------- + +Python-manilaclient SDK will include support for the new APIs and we'll +ensure that there are corresponding CLI commands in manilaclient shell as +well as the new OSC plugin shell. Manila UI's support for share, export +location and access rule metadata is limited. This specification doesn't +seek to address all the UI gaps; but all effort will be made to close the +feature parity between the CLI utilities and the UI. Eventually users will +be able to perform all metadata interactions via the UI as well. + + +Performance Impact +------------------ + +API performance is bound to suffer when resource queries include metadata +items. Since we'll be providing metadata along with ``detail`` retrievals of +resources, the performance of those APIs will also be affected negatively +because of the new database joins that will be necessary. Over time, as the +number of shares and the metadata tables grow, performance degradation can +be severe. This impact will be documented; as best practice, it is +recommended that a resource have no more than a handful of metadata items. + +Other deployer impact +--------------------- + +New APIs introduced may require tweaking of policy files if the default RBAC +policy is not acceptable. + +Developer impact +---------------- + +When adding any new user facing metadata, the metadata mixin controller can +be inherited and extended by developers. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + gouthamr + +Other contributors: + None + +Work Items +---------- + +- Add database migration to convert the ``id`` field of share, export + location and access rule metadata to a string from integer and populate + the field with UUIDs +- Add database migrations to introduce the "user_modifiable" field to share, + export location and access rule metadata tables. +- Add database migrations to create new metadata tables for all other resources +- Add ``MetadataControllerMixin``, inherit and extend in all resources and + bump up the API microversion. +- Add unit and integration tests +- Add support for metadata APIs in manilaclient SDK, manilaclient shell and OSC +- Add support for metadata interactions in the UI +- Add documentation + +Dependencies +============ + +None + + +Testing +======= + +Extensive unit tests will be written to test the API and database methods +being added. A database "walk migrations" test will be added for all the +database changes. Tempest tests will be added to cover the new metadata +operations across resources. + + +Documentation Impact +==================== + +- API reference +- End user guide +- Release notes + + +References +========== + +* `Manila API Reference `_ +* `API SIG Guideline on Metadata `_ +* `Wallaby PTG Discussion Etherpad `_