diff --git a/specs/rocky/approved/glance/multi-store.rst b/specs/rocky/approved/glance/multi-store.rst index 297ac689..e28318b3 100644 --- a/specs/rocky/approved/glance/multi-store.rst +++ b/specs/rocky/approved/glance/multi-store.rst @@ -21,17 +21,17 @@ at once. Consider the following use cases for providing multi-store backend support: - * Deployer might want to provide different level of costing for different - tiers of storage, i.e. One backend for SSDs and another for - spindles. Customer may choose one of those based on his need. - * Old storage is retired and deployer wants to have all new images being - added to new storage and old storage will be operational until data - is migrated. - * Operator wants to differentiate the images from images added by user. - * Different hypervisors provided from different backends (For - example, Ceph, Cinder, VMware etc.). - * Each site with their local backends which nova hosts are accessing - directly (Ceph) and users can select the site where image will be stored. +* Deployer might want to provide different level of costing for different + tiers of storage, i.e. One backend for SSDs and another for + spindles. Customer may choose one of those based on his need. +* Old storage is retired and deployer wants to have all new images being + added to new storage and old storage will be operational until data + is migrated. +* Operator wants to differentiate the images from images added by user. +* Different hypervisors provided from different backends (For + example, Ceph, Cinder, VMware etc.). +* Each site with their local backends which nova hosts are accessing + directly (Ceph) and users can select the site where image will be stored. Problem description =================== @@ -48,14 +48,14 @@ in order to replicate or target image bits on backend glance stores. For example, in order to replicate a existing image's bits to secondary storage of the same type / scheme as the primary: - * It's a manual out-of-band task to copy image bits to secondary storage. - * The operator must manage store locations manually; there is no way to - query the available stores to back an image's bits in glance. - * The operator must remember to register secondary location URL using - the glance API. - * Constructing the location URL by hand is error prone as some URLs are - lengthy and complex. Moreover they require knowledge of the backing store - in order to construct properly. +* It's a manual out-of-band task to copy image bits to secondary storage. +* The operator must manage store locations manually; there is no way to + query the available stores to back an image's bits in glance. +* The operator must remember to register secondary location URL using + the glance API. +* Constructing the location URL by hand is error prone as some URLs are + lengthy and complex. Moreover they require knowledge of the backing store + in order to construct properly. Also consider the case when a glance API consumer wants to download the image bits from a secondary backend location which was added out-of-band. Today @@ -63,11 +63,543 @@ the consumer must use the direct location URL which implies the consumer needs the logic necessary to translate that direct URL into a connection to the backend. +Proposed change +=============== -Current state -============= +This spec proposes the following high level features to support multiple +stores of the same scheme/type: -Glance community agrees to address the problem described above during -Rocky/S cycles. The actual detailed specification is still under discussion -and will amend this spec as https://review.openstack.org/#/c/562467 when -the implementation details are agreed on. +* Support in the glance-api.conf and supporting logic to configure, manage + and use multiple stores of the same or different type/scheme. +* Enhance the image upload API to (optionally) support targeting a backend + store for the image bits. +* Enhance image download code to download image from any of the enabled + backends. +* Enhance the image import API to (optionally) support targeting a backend + store to download the image bits from. +* Additional metadata attribute(s) added to an image locations' metadata + properties related to backing stores. +* A new REST API to list all configured stores known to the glance service. + +** Config/glance-store changes** + +In order to have multi-store support of same or different type/scheme +we propose to deprecate 'stores', 'default_store' config option and +add a new config option `enabled_backends` under DEFAULT section, +'default_backend' option under 'glance_store' section and +'description' option under each section of desired store of +glance-api.conf. Operator can use `enabled_backends` to specify comma +separated key:value of storage backends and its store type and then +each backends will have a different section to specify related configuration +options. If 'default_backend' is not explicitly set then appropriate +exception will be raised which will prevent the service from starting. + +The reason for deprecating 'default_store' config option is that glance +verifies the default store at the time of service start and if it is not +one of the listed store (file, http, rbd, swift, cinder, vmware or sheepdog) +then it raises the error and prevents the service from start. This validation +has been done using 'choices' attribute of ConfigOption object itself. After +enabling support of multi-store it's difficult to have this kind of +validation check using 'choices' attribute will be difficult as +default_store name will be store identifier and not actual store. + +Consider the following example snippet of glance-api.conf which configures +3 stores for use:: + + [DEFAULT] + # list of enabled stores identified by their property group name + enabled_backends = fast:rbd, cheap:rbd, reliable:file + + # the default store, if not set glance-api service will not start + default_backend = fast + + # conf props for file system store instance + [reliable] + filesystem_store_datadir = /var/lib/images/data/ + description = Reliable filesystem store + # etc.. + + # conf props for ceph store instance + [fast] + rbd_store_ceph_conf = /opt/ceph1/ceph.conf + description = Fast access to rbd store + # etc.. + + # conf props for ceph store instance + [cheap] + rbd_store_ceph_conf = /opt/ceph2/ceph.conf + description = Less expensive rbd store + # etc.. + +The example above is for Ceph, but it could apply to any storage backend. As +shown in the example above the 'enabled_backends' property is a comma delimited +list of store identifiers which are setup for glance. Each of the store +identifiers defines a property group name which itself contains the store +specific properties used to configure the store instance and a store type +which will be used to register configuration options related to that store +under the new configuration section defined using store identifiers. Under +the covers the store identifier (aka id) maps to the store class used for +the instance. + +While 'stores' and 'default_store' are available an operator can leave the +'enabled_backends' property unset in which case multi-store support is not +"enabled" and glance will work as-is. Once these config options are removed +operator at least need to set one store in the 'enabled_backends' or else +glance-api service will fail to start. + +In order to accommodate multiple stores of the same scheme, the scheme_map +used to index stores must also index by store identifier per scheme. The +store identifier is (implicitly) the name of the conf group given in +the glance-api.conf (e.g. reliable, fast and cheap from the example +conf given above). + +Consider the following indexing approach:: + + scheme_map[scheme][store_id] + +Where 'scheme' is the scheme(s) supported by the store implementation and +'store_id' is the identifier for the store. + +To maintain backward compatibility with old store API's while they are not +deprecated we also propose to add new store api's like add, get, delete +etc. to glance_store. The old store api's will be removed when the +deprecation period for 'stores' and 'default_store' config option is over. + +**API to list stores** + +To accommodate the management and discovery of known glance stores, glance +should also provide a REST API which permits users to list all stores +configured for glance. This is a read-only list of glance stores +which includes the store identifier, description for each store and a flag +which will tell whether a store is default or not. + +NOTE: The list of glance stores is based on the glance-api.conf with respect to +store configuration. Operators will not be able to configure new stores using +the API. So in order to show the description in the response as mentioned +earlier we propose to add new config option 'description' to each store +with some default description and operator can set it as per their need. + +As discussed herein, the store identifier can be used to address a particular +instance of a store for applicable operations such as image upload and +image import. + +We propose the new REST resource be located at the following URI:: + + /v2/info/stores + +More details on this API can be found in the REST API section of this spec. + +**Image upload API** +The existing image upload API permits an operator to upload image bits to +the glance service. However today this API does not provide a means for +the operator to target a specific store to back the image bits on +(technically the v1 API allows you to specify a store scheme to target). +This spec proposes the API be enhanced to permit an operator to specify +which store will back the image bits being uploaded. + +We propose a header field be used as a means to transport +the identifier of the store to back the image bits. Again the identifier +under the covers is the property group name of the respective glance +store driver. When the image upload API is used to upload image bits, the +glance logic will determine if the target store is specified, and if so +the image bits will be added to the target store. + +If no store is targeted in the upload request, the 'default_backend' +is used to back the image bits. + +Using this scheme operators will be able to specify which store they want +an image bits to be backed on during an upload operation. We propose the +'X-Image-Meta-Store' header be used as the means to transport a target +store identifier. + +More details on this API can be found in the REST API section of this spec. + +**Image download** +If cloud get upgraded to use multi-store support from the single store +then glance need to deal with the locations from old stores to new +stores. For example, if operator is using rbd (say ceph) backend +and now he has upgraded the environment and introduced two additional +rbd stores as ceph1 and ceph2 with default store as 'ceph1' then +somehow glance must be able to download the images from old stores +(ceph). + +With multiple backends available, Glance needs some enhancements +so it can fulfill the current image download API contract. The +download request will traverse through all the configured backends +to look for the image. As we are adding store information in location +metadata, at first it will look the image in the store which is set +in the location metadata. If location metadata is not set then +as per example from the config section above, if the location +is rbd://, then it will search the image in all available +ceph stores i.e. 'fast' and 'cheap' in this case. This way user will be +able to download his image from the old store even after cloud is +upgraded to use multiple stores. + +**Image import API** +The existing image import API allows end users to import image from the +staging area into glance backend. However today this API does not provide +a means for the end user to target a specific store to import the image. +This spec proposes the API be enhanced to permit an end user to specify +which store will back the image being imported. + +We propose a header field be used as a means to transport the identifier +of the store to import the image. Again the identifier under the covers +is the property group name of the respective glance store driver. When +the image import API is used to import the image, the glance logic will +determine if the target store header is specified, and if so the image +will be imported to the target store. + +If no store is targeted in the import request, the 'default_backend' is +used to import the image. + +Using this scheme end users will be able to specify which store they want +an image to be imported during an import operation. We propose the +'X-Image-Meta-Store' header be used as the means to transport a target +store identifier. + +More details on this API can be found in the REST API section of this spec. + +**Store locations metadata attributes** +In the current glance API, a consumer of the glance API has no way to correlate +an image location with its respective store other than by inspecting the +image's location URL. While this may work fine for many use cases, a more +user friendly relation is needed in a multi-store environment; user's need +an explicit relation between an image location and its respective store +(e.g. what store is this location backed on). + +This spec proposes that when image bits are added with the image upload API, +the core glance logic is responsible for adding a metadata attribute to the +image location URL to reflect the backing store's identifier. For example, +if a user uploads an image to the 'ceph1' store in our example above, +once the image bits are uploaded, the image location URL is added and in the +URL's metadata a property with the store's identifier is added to the location +metadata object. + +With the store identifier in the image location URL metadata, we can expose it +to the end user with a new attribute as 'store' in the image response so that +it can be used for subsequent operations or within the API consumers logic. + +The new image response with store attribute will be something like:: + + "size": 1234, + "store": ["reliable"], + "checksum": 1234567890, + "name": "Import image", + "status": active + +Alternatives +------------ + +Two major alternatives come to mind with respect to a multi-store approach in +glance; multi-store using a service per store and per driver multi-store +support. + +**Per store service** +In the service per store approach, each of the configured store instances would +run as a separate process/service. As a result each service would have its +own AMQP/RPC interface, own PID, etc.. To route requests to store services, +we'd use a scheduler which itself is another process / service. This is the +same approach used in cinder multi-backend support. + +Although the service per store approach may be a longer term goal for glance, +today we haven't seen enough justification from our consumer base to justify +the major refactoring/changes which are required to move glance to this +model. + +Therefore this spec proposes the multi-store approach outlined within this +spec - let's get an initial approximation multi-store working and gage +our next steps based on consumer feedback in the community. + +**Per driver support** +Another potential approach would be to push the multi-store logic down within +each glance store driver's implementation. For example the store driver +itself could contain logic to multiplex with multiple backends. + +While this approach would work, it would require each store driver's +implementation to change and would result in suboptimal reuse of code. + +This spec proposes a multi-store approach which has little or no impact +to each store driver; they can be used as-is in a multi-store implementation. +By pushing the logic to support multiple stores of the same scheme up into +the core of glance, we get maximal reuse and store driver implementations +needn't be concerned with such logic. + +Data model impact +----------------- + +This spec does not propose any changes to the data model. Rather the approach +herein can maintain all new stateful data either in memory or within the +existing schema used by glance. However store identifier will be stored as +a metadata in the location object. + +REST API impact +--------------- + +This spec proposes the following API changes: + +**New API** + +* List all stores known to the glance service. + +**Modified APIs** + +* Import an image. +* Upload an image file. +* Get image details. + +**Common Response Codes** + +* Create Success: `201 Created` +* Modify Success: `200 OK` +* Delete Success: `204 No Content` +* Failure: `400 Bad Request` with details. +* Forbidden: `403 Forbidden` + +**API Version** + +All URLS will be under the v2 Glance API. If it is not explicitly specified +assume /v2/ + +**[New API] List stores** + +List all stores known to the glance service:: + + GET /v2/info/stores + +This API takes no query parameters and when authorized returns a listing of all +stores known to the glance service. The stores known to glance are those which +have been configured in the glance-api.conf and have been loaded during glance +startup. The response body payload is JSON and contains a JSON object per +store. Each store JSON object contains the store's identifier (id), +description and if a particular store is a default store the it will +have a flag telling store is default. For example:: + + { + "stores":[ + { + "id":"reliable", + "description": "Reliable filesystem store" + }, + { + "id":"fast", + "description": "Fast access to rbd store", + "default": true + }, + { + "id":"cheap", + "description": "Less expensive rbd store" + } + ] + } + +Response codes: + +* 200 -- Upon authorization and successful request. The response body + contains the JSON payload with the known stores. + +**[Modified API] Create image** +We propose to add an 'OpenStack-image-store-ids' header to the image-create +response which would have the available stores. Using this user can decide +which store he needs to upload/import his image and wouldn't have to make +a separate get-stores call. + +New response headers +^^^^^^^^^^^^^^^^^^^^ + +``OpenStack-image-store-ids`` + + The value of this header will be a comma-separated list of stores + available. For example, + + OpenStack-image-store-ids: fast, cheap, reliable + +**[Modified API] Upload image binary data** + +Get image binary data:: + + PUT /v2/images/​{image_id}​/file + +This modifies the existing REST API to support a new header field which is +optional and if present specifies the store id to upload the image data to. +For backwards compatibility, if the header is not specified the store +specified as the default (e.g. default_store) is used to upload the image +to. + +New header fields: + +* X-Image-Meta-Store -- If present contains the store id to upload the image + binary data to. + +New / changed response codes: + +* 400 -- If the X-Image-Meta-Store header is present, but specifies a + store id for a store that doesn't exist by that id. + +Example curl usage:: + + curl -i -X PUT -H "X-Auth-Token: $token" -H "X-Image-Meta-Store: + ceph1" -H "Content-Type: application/octet-stream" + -d @/home/glance/ubuntu-12.10.qcow2 + $image_url/v2/images/{image_id}/file + + +**[Modified API] Get image details** + +Get the details for a specified image:: + + GET /v2/images/​{image_id}​ + +Although this spec does not impose any changes on the glance API layer, this +call will now shows the location's store id. In case if there are multiple +locations, then all locations will be displayed as comma separated list. The +new image response with store attribute will be something like:: + + "size": 1234, + "store": ["reliable"], + "checksum": 1234567890, + "name": "Import image", + "status": active + + +**[Modified API] Import image to the backend** + +Import image to the backend:: + + POST /v2/images/​{image_id}​/import + +This modifies the existing REST API to support a new header field which is +optional and if present specifies the store id to import the image data to. +For backwards compatibility, if the header is not specified the store +specified as the default (e.g. default_store) is used to import the image +to. + +New header fields: + +* X-Image-Meta-Store -- If present contains the store id to upload the image + binary data to. + +New / changed response codes: + +* 400 -- If the X-Image-Meta-Store header is present, but specifies a + store id for a store that doesn't exist by that id. + +Example curl usage:: + + curl -i -X PUT -H "X-Auth-Token: $token" -H "X-Image-Meta-Store: + ceph1" -H "Content-Type: application/json" + -d '{"method":{"name":"glance-direct"}}' + $image_url/v2/images/{image_id}/import + +Security impact +--------------- + +None + +Notifications impact +-------------------- + +Need to add 'stores' field in the notification response as one of the use +case of this proposal is offering different price tiers for storage which +will help systems which consumes notifications to perform the billing. + +Other end user impact +--------------------- + +This proposal introduces a few other user impacts worth noting. + +**Glance client** +Ideally the glance client (CLI + REST client) should be updated in accordance +with this spec. Notably: + +* CLI / API support for listing glance stores. +* CLI / API support for specifying a store id on upload/import. + +**Configuration** +Deployers will need to be aware of the configuration aspects for glance +multi-store. From a conf point of view, configuring multi-store for glance +will look very much (from a high level) like configuring cinder for +multi-backend. The conf file specifics will need to be documented. + +Performance Impact +------------------ + +A very little hit on the performance of downloading the image from the old +stores. For example, if existing user is using single rbd (say ceph) +backend and now he has upgraded the environment and introduced two +additional rbd stores as ceph1 and ceph2 with default store as 'ceph1' +then if image which needs to be download from old store (ceph) will +take some time as it needs to be looked in all the enabled backends. + +Other deployer impact +--------------------- + +Once merged, glance multi-store is not enabled unless the deployer +configures the enabled_backends property in glance-api.conf and thus is backwards +compatible out-of-the-box. When multi-store is disabled, v2 API users can +use the list stores API and will retrieve a list of the current +stores configured (of course only 1 store per scheme). + +Developer impact +---------------- + +None + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + abhishek-kekane + +Other contributors: + None + +Work Items +---------- + +Implementation tasks may consist of: + +* Conf support for multi-store. +* Multi-store loading and indexing. +* List stores API. +* Location URL metadata store id on upload. +* Add new 'store' attribute in image response. +* Image upload with store targeting. +* Image import with store targeting. +* Multi-store delete and other store access codepaths. +* Add python-glanceclient support + + +Dependencies +============ + +None + + +Testing +======= + +* Need to add new tempest tests to verify multi-store support + + +Documentation Impact +==================== + +As mentioned in the 'work items' section, we'll need to ensure the glance docs +are update for: + +* The new list stores REST API. +* New header field for image upload. +* New header field for image import. +* New store id in image response. +* Overall glance multi-store documentation to educate deployers on the + feature and how it's used. + + +References +========== + +* https://review.openstack.org/#/c/150967