Move specs under the right project
Change-Id: Ieca0237226be9c6483644e7b81589f378139dc2a
This commit is contained in:
Flavio Percoco
parent
ec8ecc1ef9
commit
11712a947e
|
|
@ -1 +0,0 @@
|
|||
../../newton/approved/buffered-reader-for-swift-driver.rst
|
||||
|
|
@ -0,0 +1,208 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==========================================
|
||||
Buffered Reader for Swift Driver
|
||||
==========================================
|
||||
|
||||
Include the URL of your launchpad blueprint:
|
||||
https://blueprints.launchpad.net/glance/+spec/buffered-reader-for-swift-driver
|
||||
|
||||
This change proposal introduces a buffering wrapper around image data for use
|
||||
during image uploads. This wrapper helps improve the reliability of
|
||||
image uploads by providing the ability to recover from errors occurring during
|
||||
the upload process. The recovering ability comes from buffering the image
|
||||
segments before uploading to swift. When an error occurs, Swift client resumes
|
||||
the upload process from the last successful position determined using 'seek'
|
||||
and 'tell' operations on the buffered image segment.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
What is the problem?
|
||||
Nova uploads image data to glance via a gzip stream. Glance's swift backend
|
||||
driver reads segments of data from the gzip stream and uploads it to Swift
|
||||
cluster using the swift client. During this process, if an error occurs while
|
||||
uploading data to Swift cluster, swift client aborts the upload as it is
|
||||
incapable of recovering from errors. This eventually triggers the Swift driver
|
||||
to delete all the data uploaded thus far.
|
||||
|
||||
Why is the problem a problem?
|
||||
Being unable to recover from errors
|
||||
- Has a direct and negative impact on the reliability of snapshots.
|
||||
- Leads to wastage of network bandwidth utilized to upload data leading up to
|
||||
the error
|
||||
|
||||
Why is swift client incapable of recovering from errors? Why not improve it
|
||||
instead?
|
||||
Swift client is actually equipped to recover from socket errors, 401s, 408s,
|
||||
rate-limit errors, 5XXs and others. It handles a decent range of errors and
|
||||
indeed tries to recover from each of the errors in an appropriate way. When
|
||||
an error occurs, swift client tries to resume the upload by repositioning the
|
||||
input source to last successful position by using the reset function [0]. The
|
||||
reset function itself utilizes the 'seek' and 'tell' operations [1] on the
|
||||
input source to determine the appropriate position from which the upload shall
|
||||
be resumed. This approach works really well when swiftclient is working with
|
||||
a file on disk or any other input source that supports 'seek' and 'tell'
|
||||
operations. However, in the case of image uploads from nova to glance, the
|
||||
input source is a data stream, which unfortunately doesn't support the 'seek'
|
||||
and 'tell' operations. This leads to swift client aborting the upload. So,
|
||||
providing 'seek' and 'tell' operations on the image data source would help
|
||||
solve this issue automatically by virtue of swift client's retry mechanism.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Introduce a wrapper around image data, in the shape of a reader class called
|
||||
BufferedReader, that supports 'seek' and 'tell' operations.
|
||||
|
||||
Currently, glance uploads image data to Swift as Dynamic Large Object (DLO),
|
||||
which holds the data in several smaller objects called segments.
|
||||
Glance's swift driver uses the swift client to upload each segment
|
||||
individually to the backend storage. However, before passing on the image
|
||||
segments to swift client, the swift driver wraps [2] the segments with
|
||||
ChunkReader class [3], which enables the swift client to upload the image
|
||||
segment in smaller chunks, usally 64KB in size.
|
||||
|
||||
In the proposed approach, instead of wrapping every segment with ChunkReader,
|
||||
we wrap it with BufferedReader, which buffers the image segment by tee-ing
|
||||
image data to a temporary file as it's being uploaded to Swift. As the image
|
||||
segment is available on disk, BufferedReader proxies the 'read', 'seek' and
|
||||
'tell' operations to the underlying temporary file. The temporary file is
|
||||
deleted if the image segment is uploaded successfully or when the upload is
|
||||
aborted after exhausting retries. If the segment fails to upload successfully,
|
||||
swift client's retry mechanism resumes the upload by repositioning the input
|
||||
using 'seek' and 'tell' operations that are now available.
|
||||
|
||||
We propose to introduce the BufferedReader as a configurable option, while
|
||||
leaving the default behavior as is. Using this configuration option, the
|
||||
amount of disk space used for buffering can be regulated as well. If the
|
||||
allotted space for buffering is full, BufferedReader reverts to default
|
||||
behavior, i.e. data is uploaded to Swift without being buffered.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
- An alternative is to use memory buffering instead of disk buffering. With
|
||||
memory buffering, there'll be a substantial increase in the memory footprint,
|
||||
which may degrade the API performance. Also, disk is a cheaper resource than
|
||||
memory but not necessarily faster.
|
||||
|
||||
- A solution is proposed in [4] to solve this problem for 401s specifically.
|
||||
This proposal basically works by ensuring the token is not nearing
|
||||
expiration. If it is expiring soon, it renews the token. Though this works
|
||||
well for 401 in particular, a similar approach may not be possible for other
|
||||
errors.
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
Notifications won't be added, modified or deleted.
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
With buffering to disk, there is an additional cost involved with writing to
|
||||
and reading from disk. This may potentially add a performance penalty to image
|
||||
uploads. However, the bulk of time elapsed for image upload is spent in
|
||||
transferring data over the network. In comparison, read and write operations
|
||||
to disk are relatively much quicker. With some thorough testing, the
|
||||
performance impact of this approach can be studied in further detail.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
As the image data is buffered to disk, there will be more disk utilization on
|
||||
the API than before. The disk utilization will be equivalent to:
|
||||
(image segment size * number of simultaneous snapshots)
|
||||
|
||||
Deployers need to take the extra disk utilization into account while using
|
||||
this feature. Additional care should be taken so that this disk utilization
|
||||
doesn't impact the disk space available for glance cache in any way.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
hemanth-makkapati
|
||||
|
||||
Other contributors:
|
||||
belliott
|
||||
jesse-j-cook
|
||||
|
||||
Reviewers
|
||||
---------
|
||||
|
||||
Core reviewer(s):
|
||||
nikhil-komawar
|
||||
stuart-mclaren
|
||||
|
||||
Other reviewer(s):
|
||||
brian-rosmaita
|
||||
mfedosin
|
||||
|
||||
Work Items
|
||||
----------
|
||||
- Implement BufferedReader class
|
||||
- Make the reader class configurable and leave the default behavior as-is
|
||||
- Test on devstack
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
No dependencies as such but [4] is a related effort.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The new configuration option would require documentation.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
[0] https://github.com/openstack/python-swiftclient/blob/stable/kilo/swiftclient/client.py#L1296-L1297
|
||||
[1] https://github.com/openstack/python-swiftclient/blob/stable/kilo/swiftclient/client.py#L1376-L1381
|
||||
[2] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L546-L550
|
||||
[3] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L928-L942
|
||||
[4] https://review.openstack.org/#/c/199049/
|
||||
|
||||
|
|
@ -1 +0,0 @@
|
|||
../../newton/approved/swift-multi-tenant-store-service-token-support.rst
|
||||
|
|
@ -0,0 +1,330 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==============================================
|
||||
Swift Multi-tenant Store Service Token Support
|
||||
==============================================
|
||||
|
||||
This is a proposal for how Composite Tokens (aka Service Tokens) can be
|
||||
used by Glance to improve the Swift Multi-Tenant store.
|
||||
|
||||
Specifically, to:
|
||||
|
||||
* Guarantee consistency between the Glance database and the Swift store.
|
||||
(For example remove the ability for users to delete image objects
|
||||
without Glance knowing.)
|
||||
* Store images separately from users' regular Swift data, preventing
|
||||
noise in their account, and more clearly abstracting the Glance API.
|
||||
* Ensure policy enforcement
|
||||
* Remove the potential for namespace collisions
|
||||
|
||||
Blueprint:
|
||||
https://blueprints.launchpad.net/glance/+spec/swift-multi-tenant-store-service-token-support
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Glance uses Swift to store data on behalf of users.
|
||||
|
||||
There are two approaches to how the data is stored:
|
||||
|
||||
* *Single-tenant*. Objects are stored in a single dedicated Swift account
|
||||
(i.e., all data belonging to all users is stored in the same account).
|
||||
|
||||
* *Multi-tenant*. Objects are stored in the end-user's Swift account (project).
|
||||
Typically, dedicated container(s) are created to hold the objects.
|
||||
|
||||
There are advantages and limitations with both approaches as described in the
|
||||
following table:
|
||||
|
||||
==== ========================================== ========== ========
|
||||
Item Feature/Topic Single- Multi-
|
||||
Tenant Tenant
|
||||
---- ------------------------------------------ ---------- --------
|
||||
1 Fragile to password leak (CVE-2013-1840) Yes No
|
||||
2 Fragile to token leak (*) Yes No
|
||||
3 Fragile to container deletion Yes No
|
||||
4 Fragile to service user deletion Yes No
|
||||
5 "Noise" in Swift account No Yes
|
||||
6 Namespace collisions (user and service No Yes
|
||||
picking same name)
|
||||
7 Guarantee of consistency (Glance Yes No
|
||||
database vs swift account) ($), (+)
|
||||
8 Policy enforcement (e.g., Image Download) Yes No
|
||||
9 Fair Swift rate limiting (users unaffected
|
||||
by others' Swift use) No Yes
|
||||
==== ========================================== ========== ========
|
||||
|
||||
(*) "Fragile" here means that a single leaked token could be used to
|
||||
access all users' image data.
|
||||
($) Users can delete objects which underpin images in the multi-tenant
|
||||
case.
|
||||
(+) If delayed delete is enabled it's not clear that the multi-tenant
|
||||
case will 'scrub' properly. This is not being considered as part
|
||||
of this spec.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Optionally require a Service Token when accessing image data stored in a
|
||||
multi-tenant Swift store.
|
||||
|
||||
This proposal uses the "Service Token Composite Authorization" support in
|
||||
the Keystone middleware [1]_.
|
||||
|
||||
This proposal also uses the Swift support for multiple reseller prefixes
|
||||
which allow storing objects in project-specific accounts while retaining
|
||||
control over how those objects are accessed via Composite Tokens [2]_.
|
||||
|
||||
The changes will apply to all accesses to the Swift backend (upload,
|
||||
download etc).
|
||||
|
||||
Example: Existing Image Download::
|
||||
|
||||
+----------------+
|
||||
| User |
|
||||
+-------+--------+
|
||||
|
|
||||
| 1. Get Glance Image 123
|
||||
|
|
||||
| GET http://glance:9292/v2/images/123/file
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
|
|
||||
+-------v---------+
|
||||
| Glance |
|
||||
+-------+---------+
|
||||
|
|
||||
| 2. Get Swift Object Data
|
||||
|
|
||||
| GET http://swift:8080/v1/AUTH_abc/glance_123/123
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
|
|
||||
+-------v---------+
|
||||
| Swift |
|
||||
+-----------------+
|
||||
|
||||
|
||||
The proposed access model has two changes:
|
||||
|
||||
* Rather than using the standard ``AUTH_`` Swift reseller prefix a specialized
|
||||
prefix, eg ``IMAGE_`` will be used.
|
||||
* A service token will be generated and supplied with the Swift request
|
||||
|
||||
Example: Proposed Image Download::
|
||||
|
||||
+----------------+
|
||||
| User |
|
||||
+-------+--------+
|
||||
|
|
||||
| 1. Get Glance Image 456
|
||||
|
|
||||
| GET http://glance:9292/v2/images/456/file
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
|
|
||||
+-------v---------+
|
||||
| Glance |
|
||||
+-------+---------+
|
||||
|
|
||||
| 2. Get Swift Object Data
|
||||
|
|
||||
| GET http://swift:8080/v1/IMAGE_abc/glance_456/456
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
| X-SERVICE-TOKEN: <token for Glance service project>
|
||||
|
|
||||
+-------v---------+
|
||||
| Swift |
|
||||
+-----------------+
|
||||
|
||||
|
||||
The service token will be generated much as a token for the single-tenant
|
||||
Swift store is today, ie credentials will be stored as part of Glance's
|
||||
configuration. Unlike the single-tenant store credentials, if the
|
||||
multi-tenant service account credentials leak they will not give direct
|
||||
access to all images.
|
||||
|
||||
The combination of the user and service token will allow access for for
|
||||
project ``abc`` under the ``IMAGE_`` reseller prefix. Specifically, Swift
|
||||
will verify that the service token contains the particular role required
|
||||
to access the relevant reseller prefix. (For more detailed information
|
||||
see the relevant Swift spec [2]_).
|
||||
|
||||
The Swift reseller prefix can be operator defined, and will be part of
|
||||
both the Swift and Glance configuration.
|
||||
|
||||
Requests to Swift for the ``IMAGE_`` prefix which do not contain a suitably
|
||||
scoped service token will return HTTP Forbidden (403).
|
||||
|
||||
Existing non-service token behaviour will continue to be supported.
|
||||
|
||||
Service token generation will not be tied to a particular project. There
|
||||
is no reliance on a particular project. If the project is deleted a new
|
||||
project with the same role can be created and used to generate the service
|
||||
token.
|
||||
|
||||
A rolling password change of the service project can be performed by
|
||||
using either two separate projects or two users in the same project.
|
||||
|
||||
If an operator modifies their configuration to take advantage of the new
|
||||
behaviour pre-existing images — images stored under the old reseller prefix
|
||||
``AUTH_`` — will continue to be accessible. The service token will
|
||||
still be supplied to Swift, but it will be ignored.
|
||||
|
||||
Example: Image Download, backwards compatibility::
|
||||
|
||||
+----------------+
|
||||
| User |
|
||||
+-------+--------+
|
||||
|
|
||||
| 1. Get Glance Image 123
|
||||
|
|
||||
| GET http://glance:9292/v2/images/123/file
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
|
|
||||
+-------v---------+
|
||||
| Glance |
|
||||
+-------+---------+
|
||||
|
|
||||
| 2. Get Swift Object Data
|
||||
|
|
||||
| GET http://swift:8080/v1/AUTH_abc/glance_123/123
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
| X-SERVICE-TOKEN: <token for Glance service project>
|
||||
|
|
||||
+-------v---------+
|
||||
| Swift |
|
||||
+-----------------+
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Two Swift installations could be used to give similar behaviour by firewalling
|
||||
user access to one Swift. That would incur a lot of hardware and operator overhead.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
There is no impact on the data model per se.
|
||||
|
||||
(New image 'location' entries will be slightly different, as they will contain
|
||||
a different Swift path.)
|
||||
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None.
|
||||
|
||||
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
This change enhances security by preventing direct access to image
|
||||
data via Swift. This removes the ability to bypass, for example, the image
|
||||
download policy for public images, shared images, and user owned images.
|
||||
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
New image objects will not be listed in users' Swift accounts.
|
||||
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
A service token will need to be requested by the Glance API process when
|
||||
Swift data is accessed. This should have minimal impact. The token
|
||||
can be cached so will only impact a minority of requests which access
|
||||
Swift. Requests which do not access Swift (eg listing images) will not
|
||||
require a service token.
|
||||
|
||||
There may be more cases of tokens expiring (and hitting uploads/downloads)
|
||||
as both the user token and the service token can potentially expire.
|
||||
There are some current efforts around mitigating token expiration. It
|
||||
may be possible to re-use some of those efforts for the service token.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Operators will need to create a service project and modify their Swift
|
||||
and Glance configurations if they wish to take advantage of the new
|
||||
behaviour. (Unmodified configurations will work as before.)
|
||||
|
||||
Pre-existing images will continue to be accessible.
|
||||
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
We may propose some changes to python-swiftclient.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee: Stuart McLaren
|
||||
|
||||
|
||||
Reviewers
|
||||
---------
|
||||
|
||||
Core reviewer(s): Flavio Percoco, Nikhil Komawar
|
||||
|
||||
Other reviewer(s): TBD
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Handle new configuration (Service credentials, Swift reseller prefix)
|
||||
* Token generation/caching
|
||||
* Any swift client changes
|
||||
* Test rolling password change
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
Keystone changes to introduce the concept of Service Tokens have been implemented [1]_
|
||||
|
||||
Swift changes to introduce support for Service Tokens/multiple reseller prefixes have been implemented [2]_
|
||||
|
||||
Required Swift client changes have been implemented [3]_
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Ideally this would become the default configuration for Glance tests in
|
||||
Tempest, and also the default configuration for devstack.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
* Example policy files will need to be created that show how to use the new
|
||||
data provided from ``X-SERVICE-TOKEN`` when making policy enforcement
|
||||
decisions.
|
||||
|
||||
* Update the Glance configuration docs
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] http://git.openstack.org/cgit/openstack/keystone-specs/tree/specs/keystonemiddleware/implemented/service-tokens.rst
|
||||
.. [2] http://git.openstack.org/cgit/openstack/swift-specs/tree/specs/done/service_token.rst
|
||||
.. [3] https://review.openstack.org/#/c/182640
|
||||
|
|
@ -1,208 +0,0 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==========================================
|
||||
Buffered Reader for Swift Driver
|
||||
==========================================
|
||||
|
||||
Include the URL of your launchpad blueprint:
|
||||
https://blueprints.launchpad.net/glance/+spec/buffered-reader-for-swift-driver
|
||||
|
||||
This change proposal introduces a buffering wrapper around image data for use
|
||||
during image uploads. This wrapper helps improve the reliability of
|
||||
image uploads by providing the ability to recover from errors occurring during
|
||||
the upload process. The recovering ability comes from buffering the image
|
||||
segments before uploading to swift. When an error occurs, Swift client resumes
|
||||
the upload process from the last successful position determined using 'seek'
|
||||
and 'tell' operations on the buffered image segment.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
What is the problem?
|
||||
Nova uploads image data to glance via a gzip stream. Glance's swift backend
|
||||
driver reads segments of data from the gzip stream and uploads it to Swift
|
||||
cluster using the swift client. During this process, if an error occurs while
|
||||
uploading data to Swift cluster, swift client aborts the upload as it is
|
||||
incapable of recovering from errors. This eventually triggers the Swift driver
|
||||
to delete all the data uploaded thus far.
|
||||
|
||||
Why is the problem a problem?
|
||||
Being unable to recover from errors
|
||||
- Has a direct and negative impact on the reliability of snapshots.
|
||||
- Leads to wastage of network bandwidth utilized to upload data leading up to
|
||||
the error
|
||||
|
||||
Why is swift client incapable of recovering from errors? Why not improve it
|
||||
instead?
|
||||
Swift client is actually equipped to recover from socket errors, 401s, 408s,
|
||||
rate-limit errors, 5XXs and others. It handles a decent range of errors and
|
||||
indeed tries to recover from each of the errors in an appropriate way. When
|
||||
an error occurs, swift client tries to resume the upload by repositioning the
|
||||
input source to last successful position by using the reset function [0]. The
|
||||
reset function itself utilizes the 'seek' and 'tell' operations [1] on the
|
||||
input source to determine the appropriate position from which the upload shall
|
||||
be resumed. This approach works really well when swiftclient is working with
|
||||
a file on disk or any other input source that supports 'seek' and 'tell'
|
||||
operations. However, in the case of image uploads from nova to glance, the
|
||||
input source is a data stream, which unfortunately doesn't support the 'seek'
|
||||
and 'tell' operations. This leads to swift client aborting the upload. So,
|
||||
providing 'seek' and 'tell' operations on the image data source would help
|
||||
solve this issue automatically by virtue of swift client's retry mechanism.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Introduce a wrapper around image data, in the shape of a reader class called
|
||||
BufferedReader, that supports 'seek' and 'tell' operations.
|
||||
|
||||
Currently, glance uploads image data to Swift as Dynamic Large Object (DLO),
|
||||
which holds the data in several smaller objects called segments.
|
||||
Glance's swift driver uses the swift client to upload each segment
|
||||
individually to the backend storage. However, before passing on the image
|
||||
segments to swift client, the swift driver wraps [2] the segments with
|
||||
ChunkReader class [3], which enables the swift client to upload the image
|
||||
segment in smaller chunks, usally 64KB in size.
|
||||
|
||||
In the proposed approach, instead of wrapping every segment with ChunkReader,
|
||||
we wrap it with BufferedReader, which buffers the image segment by tee-ing
|
||||
image data to a temporary file as it's being uploaded to Swift. As the image
|
||||
segment is available on disk, BufferedReader proxies the 'read', 'seek' and
|
||||
'tell' operations to the underlying temporary file. The temporary file is
|
||||
deleted if the image segment is uploaded successfully or when the upload is
|
||||
aborted after exhausting retries. If the segment fails to upload successfully,
|
||||
swift client's retry mechanism resumes the upload by repositioning the input
|
||||
using 'seek' and 'tell' operations that are now available.
|
||||
|
||||
We propose to introduce the BufferedReader as a configurable option, while
|
||||
leaving the default behavior as is. Using this configuration option, the
|
||||
amount of disk space used for buffering can be regulated as well. If the
|
||||
allotted space for buffering is full, BufferedReader reverts to default
|
||||
behavior, i.e. data is uploaded to Swift without being buffered.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
- An alternative is to use memory buffering instead of disk buffering. With
|
||||
memory buffering, there'll be a substantial increase in the memory footprint,
|
||||
which may degrade the API performance. Also, disk is a cheaper resource than
|
||||
memory but not necessarily faster.
|
||||
|
||||
- A solution is proposed in [4] to solve this problem for 401s specifically.
|
||||
This proposal basically works by ensuring the token is not nearing
|
||||
expiration. If it is expiring soon, it renews the token. Though this works
|
||||
well for 401 in particular, a similar approach may not be possible for other
|
||||
errors.
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
Notifications won't be added, modified or deleted.
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
With buffering to disk, there is an additional cost involved with writing to
|
||||
and reading from disk. This may potentially add a performance penalty to image
|
||||
uploads. However, the bulk of time elapsed for image upload is spent in
|
||||
transferring data over the network. In comparison, read and write operations
|
||||
to disk are relatively much quicker. With some thorough testing, the
|
||||
performance impact of this approach can be studied in further detail.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
As the image data is buffered to disk, there will be more disk utilization on
|
||||
the API than before. The disk utilization will be equivalent to:
|
||||
(image segment size * number of simultaneous snapshots)
|
||||
|
||||
Deployers need to take the extra disk utilization into account while using
|
||||
this feature. Additional care should be taken so that this disk utilization
|
||||
doesn't impact the disk space available for glance cache in any way.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
hemanth-makkapati
|
||||
|
||||
Other contributors:
|
||||
belliott
|
||||
jesse-j-cook
|
||||
|
||||
Reviewers
|
||||
---------
|
||||
|
||||
Core reviewer(s):
|
||||
nikhil-komawar
|
||||
stuart-mclaren
|
||||
|
||||
Other reviewer(s):
|
||||
brian-rosmaita
|
||||
mfedosin
|
||||
|
||||
Work Items
|
||||
----------
|
||||
- Implement BufferedReader class
|
||||
- Make the reader class configurable and leave the default behavior as-is
|
||||
- Test on devstack
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
No dependencies as such but [4] is a related effort.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The new configuration option would require documentation.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
[0] https://github.com/openstack/python-swiftclient/blob/stable/kilo/swiftclient/client.py#L1296-L1297
|
||||
[1] https://github.com/openstack/python-swiftclient/blob/stable/kilo/swiftclient/client.py#L1376-L1381
|
||||
[2] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L546-L550
|
||||
[3] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L928-L942
|
||||
[4] https://review.openstack.org/#/c/199049/
|
||||
|
||||
|
|
@ -0,0 +1 @@
|
|||
../../../mitaka/approved/buffered-reader-for-swift-driver.rst
|
||||
|
|
@ -0,0 +1 @@
|
|||
../../../mitaka/approved/swift-multi-tenant-store-service-token-support.rst
|
||||
|
|
@ -1,330 +0,0 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==============================================
|
||||
Swift Multi-tenant Store Service Token Support
|
||||
==============================================
|
||||
|
||||
This is a proposal for how Composite Tokens (aka Service Tokens) can be
|
||||
used by Glance to improve the Swift Multi-Tenant store.
|
||||
|
||||
Specifically, to:
|
||||
|
||||
* Guarantee consistency between the Glance database and the Swift store.
|
||||
(For example remove the ability for users to delete image objects
|
||||
without Glance knowing.)
|
||||
* Store images separately from users' regular Swift data, preventing
|
||||
noise in their account, and more clearly abstracting the Glance API.
|
||||
* Ensure policy enforcement
|
||||
* Remove the potential for namespace collisions
|
||||
|
||||
Blueprint:
|
||||
https://blueprints.launchpad.net/glance/+spec/swift-multi-tenant-store-service-token-support
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Glance uses Swift to store data on behalf of users.
|
||||
|
||||
There are two approaches to how the data is stored:
|
||||
|
||||
* *Single-tenant*. Objects are stored in a single dedicated Swift account
|
||||
(i.e., all data belonging to all users is stored in the same account).
|
||||
|
||||
* *Multi-tenant*. Objects are stored in the end-user's Swift account (project).
|
||||
Typically, dedicated container(s) are created to hold the objects.
|
||||
|
||||
There are advantages and limitations with both approaches as described in the
|
||||
following table:
|
||||
|
||||
==== ========================================== ========== ========
|
||||
Item Feature/Topic Single- Multi-
|
||||
Tenant Tenant
|
||||
---- ------------------------------------------ ---------- --------
|
||||
1 Fragile to password leak (CVE-2013-1840) Yes No
|
||||
2 Fragile to token leak (*) Yes No
|
||||
3 Fragile to container deletion Yes No
|
||||
4 Fragile to service user deletion Yes No
|
||||
5 "Noise" in Swift account No Yes
|
||||
6 Namespace collisions (user and service No Yes
|
||||
picking same name)
|
||||
7 Guarantee of consistency (Glance Yes No
|
||||
database vs swift account) ($), (+)
|
||||
8 Policy enforcement (e.g., Image Download) Yes No
|
||||
9 Fair Swift rate limiting (users unaffected
|
||||
by others' Swift use) No Yes
|
||||
==== ========================================== ========== ========
|
||||
|
||||
(*) "Fragile" here means that a single leaked token could be used to
|
||||
access all users' image data.
|
||||
($) Users can delete objects which underpin images in the multi-tenant
|
||||
case.
|
||||
(+) If delayed delete is enabled it's not clear that the multi-tenant
|
||||
case will 'scrub' properly. This is not being considered as part
|
||||
of this spec.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Optionally require a Service Token when accessing image data stored in a
|
||||
multi-tenant Swift store.
|
||||
|
||||
This proposal uses the "Service Token Composite Authorization" support in
|
||||
the Keystone middleware [1]_.
|
||||
|
||||
This proposal also uses the Swift support for multiple reseller prefixes
|
||||
which allow storing objects in project-specific accounts while retaining
|
||||
control over how those objects are accessed via Composite Tokens [2]_.
|
||||
|
||||
The changes will apply to all accesses to the Swift backend (upload,
|
||||
download etc).
|
||||
|
||||
Example: Existing Image Download::
|
||||
|
||||
+----------------+
|
||||
| User |
|
||||
+-------+--------+
|
||||
|
|
||||
| 1. Get Glance Image 123
|
||||
|
|
||||
| GET http://glance:9292/v2/images/123/file
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
|
|
||||
+-------v---------+
|
||||
| Glance |
|
||||
+-------+---------+
|
||||
|
|
||||
| 2. Get Swift Object Data
|
||||
|
|
||||
| GET http://swift:8080/v1/AUTH_abc/glance_123/123
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
|
|
||||
+-------v---------+
|
||||
| Swift |
|
||||
+-----------------+
|
||||
|
||||
|
||||
The proposed access model has two changes:
|
||||
|
||||
* Rather than using the standard ``AUTH_`` Swift reseller prefix a specialized
|
||||
prefix, eg ``IMAGE_`` will be used.
|
||||
* A service token will be generated and supplied with the Swift request
|
||||
|
||||
Example: Proposed Image Download::
|
||||
|
||||
+----------------+
|
||||
| User |
|
||||
+-------+--------+
|
||||
|
|
||||
| 1. Get Glance Image 456
|
||||
|
|
||||
| GET http://glance:9292/v2/images/456/file
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
|
|
||||
+-------v---------+
|
||||
| Glance |
|
||||
+-------+---------+
|
||||
|
|
||||
| 2. Get Swift Object Data
|
||||
|
|
||||
| GET http://swift:8080/v1/IMAGE_abc/glance_456/456
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
| X-SERVICE-TOKEN: <token for Glance service project>
|
||||
|
|
||||
+-------v---------+
|
||||
| Swift |
|
||||
+-----------------+
|
||||
|
||||
|
||||
The service token will be generated much as a token for the single-tenant
|
||||
Swift store is today, ie credentials will be stored as part of Glance's
|
||||
configuration. Unlike the single-tenant store credentials, if the
|
||||
multi-tenant service account credentials leak they will not give direct
|
||||
access to all images.
|
||||
|
||||
The combination of the user and service token will allow access for for
|
||||
project ``abc`` under the ``IMAGE_`` reseller prefix. Specifically, Swift
|
||||
will verify that the service token contains the particular role required
|
||||
to access the relevant reseller prefix. (For more detailed information
|
||||
see the relevant Swift spec [2]_).
|
||||
|
||||
The Swift reseller prefix can be operator defined, and will be part of
|
||||
both the Swift and Glance configuration.
|
||||
|
||||
Requests to Swift for the ``IMAGE_`` prefix which do not contain a suitably
|
||||
scoped service token will return HTTP Forbidden (403).
|
||||
|
||||
Existing non-service token behaviour will continue to be supported.
|
||||
|
||||
Service token generation will not be tied to a particular project. There
|
||||
is no reliance on a particular project. If the project is deleted a new
|
||||
project with the same role can be created and used to generate the service
|
||||
token.
|
||||
|
||||
A rolling password change of the service project can be performed by
|
||||
using either two separate projects or two users in the same project.
|
||||
|
||||
If an operator modifies their configuration to take advantage of the new
|
||||
behaviour pre-existing images — images stored under the old reseller prefix
|
||||
``AUTH_`` — will continue to be accessible. The service token will
|
||||
still be supplied to Swift, but it will be ignored.
|
||||
|
||||
Example: Image Download, backwards compatibility::
|
||||
|
||||
+----------------+
|
||||
| User |
|
||||
+-------+--------+
|
||||
|
|
||||
| 1. Get Glance Image 123
|
||||
|
|
||||
| GET http://glance:9292/v2/images/123/file
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
|
|
||||
+-------v---------+
|
||||
| Glance |
|
||||
+-------+---------+
|
||||
|
|
||||
| 2. Get Swift Object Data
|
||||
|
|
||||
| GET http://swift:8080/v1/AUTH_abc/glance_123/123
|
||||
| X-AUTH-TOKEN: <token for user project 'abc'>
|
||||
| X-SERVICE-TOKEN: <token for Glance service project>
|
||||
|
|
||||
+-------v---------+
|
||||
| Swift |
|
||||
+-----------------+
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Two Swift installations could be used to give similar behaviour by firewalling
|
||||
user access to one Swift. That would incur a lot of hardware and operator overhead.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
There is no impact on the data model per se.
|
||||
|
||||
(New image 'location' entries will be slightly different, as they will contain
|
||||
a different Swift path.)
|
||||
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None.
|
||||
|
||||
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
This change enhances security by preventing direct access to image
|
||||
data via Swift. This removes the ability to bypass, for example, the image
|
||||
download policy for public images, shared images, and user owned images.
|
||||
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
New image objects will not be listed in users' Swift accounts.
|
||||
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
A service token will need to be requested by the Glance API process when
|
||||
Swift data is accessed. This should have minimal impact. The token
|
||||
can be cached so will only impact a minority of requests which access
|
||||
Swift. Requests which do not access Swift (eg listing images) will not
|
||||
require a service token.
|
||||
|
||||
There may be more cases of tokens expiring (and hitting uploads/downloads)
|
||||
as both the user token and the service token can potentially expire.
|
||||
There are some current efforts around mitigating token expiration. It
|
||||
may be possible to re-use some of those efforts for the service token.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Operators will need to create a service project and modify their Swift
|
||||
and Glance configurations if they wish to take advantage of the new
|
||||
behaviour. (Unmodified configurations will work as before.)
|
||||
|
||||
Pre-existing images will continue to be accessible.
|
||||
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
We may propose some changes to python-swiftclient.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee: Stuart McLaren
|
||||
|
||||
|
||||
Reviewers
|
||||
---------
|
||||
|
||||
Core reviewer(s): Flavio Percoco, Nikhil Komawar
|
||||
|
||||
Other reviewer(s): TBD
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Handle new configuration (Service credentials, Swift reseller prefix)
|
||||
* Token generation/caching
|
||||
* Any swift client changes
|
||||
* Test rolling password change
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
Keystone changes to introduce the concept of Service Tokens have been implemented [1]_
|
||||
|
||||
Swift changes to introduce support for Service Tokens/multiple reseller prefixes have been implemented [2]_
|
||||
|
||||
Required Swift client changes have been implemented [3]_
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Ideally this would become the default configuration for Glance tests in
|
||||
Tempest, and also the default configuration for devstack.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
* Example policy files will need to be created that show how to use the new
|
||||
data provided from ``X-SERVICE-TOKEN`` when making policy enforcement
|
||||
decisions.
|
||||
|
||||
* Update the Glance configuration docs
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] http://git.openstack.org/cgit/openstack/keystone-specs/tree/specs/keystonemiddleware/implemented/service-tokens.rst
|
||||
.. [2] http://git.openstack.org/cgit/openstack/swift-specs/tree/specs/done/service_token.rst
|
||||
.. [3] https://review.openstack.org/#/c/182640
|
||||
Loading…
Reference in New Issue