summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorAbhishek Kekane <akekane@redhat.com>2018-04-19 05:24:51 +0000
committerAbhishek Kekane <akekane@redhat.com>2018-07-16 13:37:51 +0000
commit57da144125813dda87acb5966be1e6ed7f007e81 (patch)
treeaea1107500e1497f983b305e1f439bc39298135d
parent0cd9928970c888ac340fa289710f4a9c89d82161 (diff)
Provide multiple store backend support
Change-Id: I4b00cf8317adb4318ceb65f336c9fe0f17f4ca9c Implements: multi-store
Notes
Notes (review): Code-Review+2: Brian Rosmaita <rosmaita.fossdev@gmail.com> Code-Review+2: Sean McGinnis <sean.mcginnis@gmail.com> Code-Review+2: Erno Kuvaja <jokke@usr.fi> Workflow+1: Erno Kuvaja <jokke@usr.fi> Verified+2: Zuul Submitted-by: Zuul Submitted-at: Mon, 16 Jul 2018 19:40:55 +0000 Reviewed-on: https://review.openstack.org/562467 Project: openstack/glance-specs Branch: refs/heads/master
-rw-r--r--specs/rocky/approved/glance/multi-store.rst582
1 files changed, 557 insertions, 25 deletions
diff --git a/specs/rocky/approved/glance/multi-store.rst b/specs/rocky/approved/glance/multi-store.rst
index 297ac68..e28318b 100644
--- a/specs/rocky/approved/glance/multi-store.rst
+++ b/specs/rocky/approved/glance/multi-store.rst
@@ -21,17 +21,17 @@ at once.
21 21
22Consider the following use cases for providing multi-store backend support: 22Consider the following use cases for providing multi-store backend support:
23 23
24 * Deployer might want to provide different level of costing for different 24* Deployer might want to provide different level of costing for different
25 tiers of storage, i.e. One backend for SSDs and another for 25 tiers of storage, i.e. One backend for SSDs and another for
26 spindles. Customer may choose one of those based on his need. 26 spindles. Customer may choose one of those based on his need.
27 * Old storage is retired and deployer wants to have all new images being 27* Old storage is retired and deployer wants to have all new images being
28 added to new storage and old storage will be operational until data 28 added to new storage and old storage will be operational until data
29 is migrated. 29 is migrated.
30 * Operator wants to differentiate the images from images added by user. 30* Operator wants to differentiate the images from images added by user.
31 * Different hypervisors provided from different backends (For 31* Different hypervisors provided from different backends (For
32 example, Ceph, Cinder, VMware etc.). 32 example, Ceph, Cinder, VMware etc.).
33 * Each site with their local backends which nova hosts are accessing 33* Each site with their local backends which nova hosts are accessing
34 directly (Ceph) and users can select the site where image will be stored. 34 directly (Ceph) and users can select the site where image will be stored.
35 35
36Problem description 36Problem description
37=================== 37===================
@@ -48,14 +48,14 @@ in order to replicate or target image bits on backend glance stores. For
48example, in order to replicate a existing image's bits to secondary storage 48example, in order to replicate a existing image's bits to secondary storage
49of the same type / scheme as the primary: 49of the same type / scheme as the primary:
50 50
51 * It's a manual out-of-band task to copy image bits to secondary storage. 51* It's a manual out-of-band task to copy image bits to secondary storage.
52 * The operator must manage store locations manually; there is no way to 52* The operator must manage store locations manually; there is no way to
53 query the available stores to back an image's bits in glance. 53 query the available stores to back an image's bits in glance.
54 * The operator must remember to register secondary location URL using 54* The operator must remember to register secondary location URL using
55 the glance API. 55 the glance API.
56 * Constructing the location URL by hand is error prone as some URLs are 56* Constructing the location URL by hand is error prone as some URLs are
57 lengthy and complex. Moreover they require knowledge of the backing store 57 lengthy and complex. Moreover they require knowledge of the backing store
58 in order to construct properly. 58 in order to construct properly.
59 59
60Also consider the case when a glance API consumer wants to download the image 60Also consider the case when a glance API consumer wants to download the image
61bits from a secondary backend location which was added out-of-band. Today 61bits from a secondary backend location which was added out-of-band. Today
@@ -63,11 +63,543 @@ the consumer must use the direct location URL which implies the consumer
63needs the logic necessary to translate that direct URL into a connection 63needs the logic necessary to translate that direct URL into a connection
64to the backend. 64to the backend.
65 65
66Proposed change
67===============
66 68
67Current state 69This spec proposes the following high level features to support multiple
68============= 70stores of the same scheme/type:
69 71
70Glance community agrees to address the problem described above during 72* Support in the glance-api.conf and supporting logic to configure, manage
71Rocky/S cycles. The actual detailed specification is still under discussion 73 and use multiple stores of the same or different type/scheme.
72and will amend this spec as https://review.openstack.org/#/c/562467 when 74* Enhance the image upload API to (optionally) support targeting a backend
73the implementation details are agreed on. 75 store for the image bits.
76* Enhance image download code to download image from any of the enabled
77 backends.
78* Enhance the image import API to (optionally) support targeting a backend
79 store to download the image bits from.
80* Additional metadata attribute(s) added to an image locations' metadata
81 properties related to backing stores.
82* A new REST API to list all configured stores known to the glance service.
83
84** Config/glance-store changes**
85
86In order to have multi-store support of same or different type/scheme
87we propose to deprecate 'stores', 'default_store' config option and
88add a new config option `enabled_backends` under DEFAULT section,
89'default_backend' option under 'glance_store' section and
90'description' option under each section of desired store of
91glance-api.conf. Operator can use `enabled_backends` to specify comma
92separated key:value of storage backends and its store type and then
93each backends will have a different section to specify related configuration
94options. If 'default_backend' is not explicitly set then appropriate
95exception will be raised which will prevent the service from starting.
96
97The reason for deprecating 'default_store' config option is that glance
98verifies the default store at the time of service start and if it is not
99one of the listed store (file, http, rbd, swift, cinder, vmware or sheepdog)
100then it raises the error and prevents the service from start. This validation
101has been done using 'choices' attribute of ConfigOption object itself. After
102enabling support of multi-store it's difficult to have this kind of
103validation check using 'choices' attribute will be difficult as
104default_store name will be store identifier and not actual store.
105
106Consider the following example snippet of glance-api.conf which configures
1073 stores for use::
108
109 [DEFAULT]
110 # list of enabled stores identified by their property group name
111 enabled_backends = fast:rbd, cheap:rbd, reliable:file
112
113 # the default store, if not set glance-api service will not start
114 default_backend = fast
115
116 # conf props for file system store instance
117 [reliable]
118 filesystem_store_datadir = /var/lib/images/data/
119 description = Reliable filesystem store
120 # etc..
121
122 # conf props for ceph store instance
123 [fast]
124 rbd_store_ceph_conf = /opt/ceph1/ceph.conf
125 description = Fast access to rbd store
126 # etc..
127
128 # conf props for ceph store instance
129 [cheap]
130 rbd_store_ceph_conf = /opt/ceph2/ceph.conf
131 description = Less expensive rbd store
132 # etc..
133
134The example above is for Ceph, but it could apply to any storage backend. As
135shown in the example above the 'enabled_backends' property is a comma delimited
136list of store identifiers which are setup for glance. Each of the store
137identifiers defines a property group name which itself contains the store
138specific properties used to configure the store instance and a store type
139which will be used to register configuration options related to that store
140under the new configuration section defined using store identifiers. Under
141the covers the store identifier (aka id) maps to the store class used for
142the instance.
143
144While 'stores' and 'default_store' are available an operator can leave the
145'enabled_backends' property unset in which case multi-store support is not
146"enabled" and glance will work as-is. Once these config options are removed
147operator at least need to set one store in the 'enabled_backends' or else
148glance-api service will fail to start.
149
150In order to accommodate multiple stores of the same scheme, the scheme_map
151used to index stores must also index by store identifier per scheme. The
152store identifier is (implicitly) the name of the conf group given in
153the glance-api.conf (e.g. reliable, fast and cheap from the example
154conf given above).
155
156Consider the following indexing approach::
157
158 scheme_map[scheme][store_id]
159
160Where 'scheme' is the scheme(s) supported by the store implementation and
161'store_id' is the identifier for the store.
162
163To maintain backward compatibility with old store API's while they are not
164deprecated we also propose to add new store api's like add, get, delete
165etc. to glance_store. The old store api's will be removed when the
166deprecation period for 'stores' and 'default_store' config option is over.
167
168**API to list stores**
169
170To accommodate the management and discovery of known glance stores, glance
171should also provide a REST API which permits users to list all stores
172configured for glance. This is a read-only list of glance stores
173which includes the store identifier, description for each store and a flag
174which will tell whether a store is default or not.
175
176NOTE: The list of glance stores is based on the glance-api.conf with respect to
177store configuration. Operators will not be able to configure new stores using
178the API. So in order to show the description in the response as mentioned
179earlier we propose to add new config option 'description' to each store
180with some default description and operator can set it as per their need.
181
182As discussed herein, the store identifier can be used to address a particular
183instance of a store for applicable operations such as image upload and
184image import.
185
186We propose the new REST resource be located at the following URI::
187
188 /v2/info/stores
189
190More details on this API can be found in the REST API section of this spec.
191
192**Image upload API**
193The existing image upload API permits an operator to upload image bits to
194the glance service. However today this API does not provide a means for
195the operator to target a specific store to back the image bits on
196(technically the v1 API allows you to specify a store scheme to target).
197This spec proposes the API be enhanced to permit an operator to specify
198which store will back the image bits being uploaded.
199
200We propose a header field be used as a means to transport
201the identifier of the store to back the image bits. Again the identifier
202under the covers is the property group name of the respective glance
203store driver. When the image upload API is used to upload image bits, the
204glance logic will determine if the target store is specified, and if so
205the image bits will be added to the target store.
206
207If no store is targeted in the upload request, the 'default_backend'
208is used to back the image bits.
209
210Using this scheme operators will be able to specify which store they want
211an image bits to be backed on during an upload operation. We propose the
212'X-Image-Meta-Store' header be used as the means to transport a target
213store identifier.
214
215More details on this API can be found in the REST API section of this spec.
216
217**Image download**
218If cloud get upgraded to use multi-store support from the single store
219then glance need to deal with the locations from old stores to new
220stores. For example, if operator is using rbd (say ceph) backend
221and now he has upgraded the environment and introduced two additional
222rbd stores as ceph1 and ceph2 with default store as 'ceph1' then
223somehow glance must be able to download the images from old stores
224(ceph).
225
226With multiple backends available, Glance needs some enhancements
227so it can fulfill the current image download API contract. The
228download request will traverse through all the configured backends
229to look for the image. As we are adding store information in location
230metadata, at first it will look the image in the store which is set
231in the location metadata. If location metadata is not set then
232as per example from the config section above, if the location
233is rbd://<something>, then it will search the image in all available
234ceph stores i.e. 'fast' and 'cheap' in this case. This way user will be
235able to download his image from the old store even after cloud is
236upgraded to use multiple stores.
237
238**Image import API**
239The existing image import API allows end users to import image from the
240staging area into glance backend. However today this API does not provide
241a means for the end user to target a specific store to import the image.
242This spec proposes the API be enhanced to permit an end user to specify
243which store will back the image being imported.
244
245We propose a header field be used as a means to transport the identifier
246of the store to import the image. Again the identifier under the covers
247is the property group name of the respective glance store driver. When
248the image import API is used to import the image, the glance logic will
249determine if the target store header is specified, and if so the image
250will be imported to the target store.
251
252If no store is targeted in the import request, the 'default_backend' is
253used to import the image.
254
255Using this scheme end users will be able to specify which store they want
256an image to be imported during an import operation. We propose the
257'X-Image-Meta-Store' header be used as the means to transport a target
258store identifier.
259
260More details on this API can be found in the REST API section of this spec.
261
262**Store locations metadata attributes**
263In the current glance API, a consumer of the glance API has no way to correlate
264an image location with its respective store other than by inspecting the
265image's location URL. While this may work fine for many use cases, a more
266user friendly relation is needed in a multi-store environment; user's need
267an explicit relation between an image location and its respective store
268(e.g. what store is this location backed on).
269
270This spec proposes that when image bits are added with the image upload API,
271the core glance logic is responsible for adding a metadata attribute to the
272image location URL to reflect the backing store's identifier. For example,
273if a user uploads an image to the 'ceph1' store in our example above,
274once the image bits are uploaded, the image location URL is added and in the
275URL's metadata a property with the store's identifier is added to the location
276metadata object.
277
278With the store identifier in the image location URL metadata, we can expose it
279to the end user with a new attribute as 'store' in the image response so that
280it can be used for subsequent operations or within the API consumers logic.
281
282The new image response with store attribute will be something like::
283
284 "size": 1234,
285 "store": ["reliable"],
286 "checksum": 1234567890,
287 "name": "Import image",
288 "status": active
289
290Alternatives
291------------
292
293Two major alternatives come to mind with respect to a multi-store approach in
294glance; multi-store using a service per store and per driver multi-store
295support.
296
297**Per store service**
298In the service per store approach, each of the configured store instances would
299run as a separate process/service. As a result each service would have its
300own AMQP/RPC interface, own PID, etc.. To route requests to store services,
301we'd use a scheduler which itself is another process / service. This is the
302same approach used in cinder multi-backend support.
303
304Although the service per store approach may be a longer term goal for glance,
305today we haven't seen enough justification from our consumer base to justify
306the major refactoring/changes which are required to move glance to this
307model.
308
309Therefore this spec proposes the multi-store approach outlined within this
310spec - let's get an initial approximation multi-store working and gage
311our next steps based on consumer feedback in the community.
312
313**Per driver support**
314Another potential approach would be to push the multi-store logic down within
315each glance store driver's implementation. For example the store driver
316itself could contain logic to multiplex with multiple backends.
317
318While this approach would work, it would require each store driver's
319implementation to change and would result in suboptimal reuse of code.
320
321This spec proposes a multi-store approach which has little or no impact
322to each store driver; they can be used as-is in a multi-store implementation.
323By pushing the logic to support multiple stores of the same scheme up into
324the core of glance, we get maximal reuse and store driver implementations
325needn't be concerned with such logic.
326
327Data model impact
328-----------------
329
330This spec does not propose any changes to the data model. Rather the approach
331herein can maintain all new stateful data either in memory or within the
332existing schema used by glance. However store identifier will be stored as
333a metadata in the location object.
334
335REST API impact
336---------------
337
338This spec proposes the following API changes:
339
340**New API**
341
342* List all stores known to the glance service.
343
344**Modified APIs**
345
346* Import an image.
347* Upload an image file.
348* Get image details.
349
350**Common Response Codes**
351
352* Create Success: `201 Created`
353* Modify Success: `200 OK`
354* Delete Success: `204 No Content`
355* Failure: `400 Bad Request` with details.
356* Forbidden: `403 Forbidden`
357
358**API Version**
359
360All URLS will be under the v2 Glance API. If it is not explicitly specified
361assume /v2/<url>
362
363**[New API] List stores**
364
365List all stores known to the glance service::
366
367 GET /v2/info/stores
368
369This API takes no query parameters and when authorized returns a listing of all
370stores known to the glance service. The stores known to glance are those which
371have been configured in the glance-api.conf and have been loaded during glance
372startup. The response body payload is JSON and contains a JSON object per
373store. Each store JSON object contains the store's identifier (id),
374description and if a particular store is a default store the it will
375have a flag telling store is default. For example::
376
377 {
378 "stores":[
379 {
380 "id":"reliable",
381 "description": "Reliable filesystem store"
382 },
383 {
384 "id":"fast",
385 "description": "Fast access to rbd store",
386 "default": true
387 },
388 {
389 "id":"cheap",
390 "description": "Less expensive rbd store"
391 }
392 ]
393 }
394
395Response codes:
396
397* 200 -- Upon authorization and successful request. The response body
398 contains the JSON payload with the known stores.
399
400**[Modified API] Create image**
401We propose to add an 'OpenStack-image-store-ids' header to the image-create
402response which would have the available stores. Using this user can decide
403which store he needs to upload/import his image and wouldn't have to make
404a separate get-stores call.
405
406New response headers
407^^^^^^^^^^^^^^^^^^^^
408
409``OpenStack-image-store-ids``
410
411 The value of this header will be a comma-separated list of stores
412 available. For example,
413
414 OpenStack-image-store-ids: fast, cheap, reliable
415
416**[Modified API] Upload image binary data**
417
418Get image binary data::
419
420 PUT /v2/images/​{image_id}​/file
421
422This modifies the existing REST API to support a new header field which is
423optional and if present specifies the store id to upload the image data to.
424For backwards compatibility, if the header is not specified the store
425specified as the default (e.g. default_store) is used to upload the image
426to.
427
428New header fields:
429
430* X-Image-Meta-Store -- If present contains the store id to upload the image
431 binary data to.
432
433New / changed response codes:
434
435* 400 -- If the X-Image-Meta-Store header is present, but specifies a
436 store id for a store that doesn't exist by that id.
437
438Example curl usage::
439
440 curl -i -X PUT -H "X-Auth-Token: $token" -H "X-Image-Meta-Store:
441 ceph1" -H "Content-Type: application/octet-stream"
442 -d @/home/glance/ubuntu-12.10.qcow2
443 $image_url/v2/images/{image_id}/file
444
445
446**[Modified API] Get image details**
447
448Get the details for a specified image::
449
450 GET /v2/images/​{image_id}​
451
452Although this spec does not impose any changes on the glance API layer, this
453call will now shows the location's store id. In case if there are multiple
454locations, then all locations will be displayed as comma separated list. The
455new image response with store attribute will be something like::
456
457 "size": 1234,
458 "store": ["reliable"],
459 "checksum": 1234567890,
460 "name": "Import image",
461 "status": active
462
463
464**[Modified API] Import image to the backend**
465
466Import image to the backend::
467
468 POST /v2/images/​{image_id}​/import
469
470This modifies the existing REST API to support a new header field which is
471optional and if present specifies the store id to import the image data to.
472For backwards compatibility, if the header is not specified the store
473specified as the default (e.g. default_store) is used to import the image
474to.
475
476New header fields:
477
478* X-Image-Meta-Store -- If present contains the store id to upload the image
479 binary data to.
480
481New / changed response codes:
482
483* 400 -- If the X-Image-Meta-Store header is present, but specifies a
484 store id for a store that doesn't exist by that id.
485
486Example curl usage::
487
488 curl -i -X PUT -H "X-Auth-Token: $token" -H "X-Image-Meta-Store:
489 ceph1" -H "Content-Type: application/json"
490 -d '{"method":{"name":"glance-direct"}}'
491 $image_url/v2/images/{image_id}/import
492
493Security impact
494---------------
495
496None
497
498Notifications impact
499--------------------
500
501Need to add 'stores' field in the notification response as one of the use
502case of this proposal is offering different price tiers for storage which
503will help systems which consumes notifications to perform the billing.
504
505Other end user impact
506---------------------
507
508This proposal introduces a few other user impacts worth noting.
509
510**Glance client**
511Ideally the glance client (CLI + REST client) should be updated in accordance
512with this spec. Notably:
513
514* CLI / API support for listing glance stores.
515* CLI / API support for specifying a store id on upload/import.
516
517**Configuration**
518Deployers will need to be aware of the configuration aspects for glance
519multi-store. From a conf point of view, configuring multi-store for glance
520will look very much (from a high level) like configuring cinder for
521multi-backend. The conf file specifics will need to be documented.
522
523Performance Impact
524------------------
525
526A very little hit on the performance of downloading the image from the old
527stores. For example, if existing user is using single rbd (say ceph)
528backend and now he has upgraded the environment and introduced two
529additional rbd stores as ceph1 and ceph2 with default store as 'ceph1'
530then if image which needs to be download from old store (ceph) will
531take some time as it needs to be looked in all the enabled backends.
532
533Other deployer impact
534---------------------
535
536Once merged, glance multi-store is not enabled unless the deployer
537configures the enabled_backends property in glance-api.conf and thus is backwards
538compatible out-of-the-box. When multi-store is disabled, v2 API users can
539use the list stores API and will retrieve a list of the current
540stores configured (of course only 1 store per scheme).
541
542Developer impact
543----------------
544
545None
546
547
548Implementation
549==============
550
551Assignee(s)
552-----------
553
554Primary assignee:
555 abhishek-kekane
556
557Other contributors:
558 None
559
560Work Items
561----------
562
563Implementation tasks may consist of:
564
565* Conf support for multi-store.
566* Multi-store loading and indexing.
567* List stores API.
568* Location URL metadata store id on upload.
569* Add new 'store' attribute in image response.
570* Image upload with store targeting.
571* Image import with store targeting.
572* Multi-store delete and other store access codepaths.
573* Add python-glanceclient support
574
575
576Dependencies
577============
578
579None
580
581
582Testing
583=======
584
585* Need to add new tempest tests to verify multi-store support
586
587
588Documentation Impact
589====================
590
591As mentioned in the 'work items' section, we'll need to ensure the glance docs
592are update for:
593
594* The new list stores REST API.
595* New header field for image upload.
596* New header field for image import.
597* New store id in image response.
598* Overall glance multi-store documentation to educate deployers on the
599 feature and how it's used.
600
601
602References
603==========
604
605* https://review.openstack.org/#/c/150967