openstack
/
swift
Code Issues Proposed changes

Update S3api Docs 

Change-Id: I31f1dad95a2381cf355e5619810ee85bc54bf358
This commit is contained in:
Kota Tsuyuzaki 2018-03-13 14:17:46 +09:00
parent a46faca2c9
commit 112e087287
12 changed files with 312 additions and 139 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
doc/source/associated_projects.rst
View File

@ -65,7 +65,6 @@ Content Distribution Network Integration
Alternative API
---------------
* `Swift3 <https://github.com/openstack/swift3>`_ - Amazon S3 API emulation.
* `CDMI <https://github.com/osaddon/cdmi>`_ - CDMI support
* `SwiftHLM <https://github.com/ibm-research/SwiftHLM>`_ - a middleware for using OpenStack Swift with tape and other high latency media storage backends
@ -119,3 +118,4 @@ Other
* `Swift Durability Calculator <https://github.com/redhat-cip/swift-durability-calculator>`_ - Data Durability Calculation Tool for Swift
* `swiftbackmeup <https://github.com/redhat-cip/swiftbackmeup>`_ - Utility that allows one to create backups and upload them to OpenStack Swift
* `Multi Swift <https://github.com/ntata/multi-swift-POC>`_ - Bash scripts to spin up multiple Swift clusters sharing the same hardware
* `s3compat <https://github.com/swiftstack/s3compat>`_ - S3 API compatibility checker

92
doc/source/middleware.rst
View File

@ -11,6 +11,91 @@ Account Quotas
:members:
:show-inheritance:
.. _s3api:
Amazon S3 Api
=============
.. automodule:: swift.common.middleware.s3api.s3api
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.s3token
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.s3request
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.s3response
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.exception
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.etree
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.subresource
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.acl_handlers
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.acl_utils
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.base
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.service
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.bucket
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.obj
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.acl
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.s3_acl
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.multi_upload
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.multi_delete
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.versioning
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.location
:members:
:show-inheritance:
.. automodule:: swift.common.middleware.s3api.controllers.logging
:members:
:show-inheritance:
.. _bulk:
Bulk Operations (Delete and Archive Auto Extraction)
@ -277,10 +362,3 @@ XProfile
.. automodule:: swift.common.middleware.xprofile
:members:
:show-inheritance:
s3api
==============
.. automodule:: swift.common.middleware.s3api.s3api
:members:
:show-inheritance:

48
etc/proxy-server.conf-sample
View File

@ -445,19 +445,26 @@ user_test5_tester5 = testing5 service
[filter:s3api]
use = egg:swift#s3api
# s3api setup:
#
# With either tempauth or your custom auth:
# - Put s3api just before your auth filter(s) in the pipeline
# With keystone:
# - Put s3api and s3token before keystoneauth in the pipeline
#
# Swift has no concept of the S3's resource owner; the resources
# (i.e. containers and objects) created via the Swift API have no owner
# information. This option specifies how the s3api middleware handles them
# with the S3 API. If this option is 'false', such kinds of resources will be
# invisible and no users can access them with the S3 API. If set to 'true',
# the resource without owner is belong to everyone and everyone can access it
# a resource without an owner belongs to everyone and everyone can access it
# with the S3 API. If you care about S3 compatibility, set 'false' here. This
# option makes sense only when the s3_acl option is set to 'true' and your
# Swift cluster has the resources created via the Swift API.
# allow_no_owner = false
#
# Set a region name of your Swift cluster. Note that s3api doesn't choose a
# region of the newly created bucket actually. This value is used for the
# Set a region name of your Swift cluster. Note that the s3api doesn't choose
# a region of the newly created bucket. This value is used for the
# GET Bucket location API and v4 signatures calculation.
# location = US
#
@ -470,32 +477,31 @@ use = egg:swift#s3api
# max_bucket_listing = 1000
#
# Set the maximum number of parts returned in the List Parts operation.
# (default: 1000)
# When setting it to be larger than 10000, set to be larger
# container_listing_limit in swift.conf.(specification of S3: 1000)
# (default: 1000 as well as S3 specification)
# If setting it larger than 10000 (swift container_listing_limit default)
# make sure you also increase the container_listing_limit in swift.conf.
# max_parts_listing = 1000
#
# Set the maximum number of objects we can delete with the Multi-Object Delete
# operation.
# max_multi_delete_objects = 1000
#
# If set to 'true', s3api uses its own metadata for ACL
# If set to 'true', s3api uses its own metadata for ACLs
# (e.g. X-Container-Sysmeta-S3Api-Acl) to achieve the best S3 compatibility.
# If set to 'false', s3api tries to use Swift ACL (e.g. X-Container-Read)
# instead of S3 ACL as far as possible. If you want to keep backward
# compatibility with Swift3 1.7 or earlier, set false here
# If set to 'false' after set to 'true' and put some container/object,
# all users will be able to access container/object.
# Note that s3_acl doesn't keep the acl consistency between S3 API and Swift
# API. (e.g. when set s3acl to true and PUT acl, we won't get the acl
# information via Swift API at all and the acl won't be applied against to
# Swift API even if it is for a bucket currently supported.)
# If set to 'false', s3api tries to use Swift ACLs (e.g. X-Container-Read)
# instead of S3 ACLs as far as possible.
# There are some caveats that one should know about this setting. Firstly,
# if set to 'false' after being previously set to 'true' any new objects or
# containers stored while 'true' setting will be accessible to all users
# because the s3 ACLs will be ignored under s3_acl=False setting. Secondly,
# s3_acl True mode don't keep ACL consistency between both the S3 and Swift
# API. Meaning with s3_acl enabled S3 ACLs only effect objects and buckets
# via the S3 API. As this ACL information wont be available via the Swift API
# and so the ACL wont be applied.
# Note that s3_acl currently supports only keystone and tempauth.
# DON'T USE THIS for production before enough testing for your use cases.
# This stuff is still under development and it might cause something
# you don't expect.
# TODO: consider if we can decide s3_acl is true in default
# TODO: consider if we need to skip some acl tests if s3_acl is off
# s3_acl = false
#
# Specify a host name of your Swift cluster. This enables virtual-hosted style
@ -530,7 +536,7 @@ use = egg:swift#s3api
# If you set this to false, s3api returns all buckets.
# check_bucket_owner = false
#
# In default, Swift reports only S3 style access log.
# By default, Swift reports only S3 style access log.
# (e.g. PUT /bucket/object) If set force_swift_request_proxy_log
# to be 'true', Swift will become to output Swift style log
# (e.g. PUT /v1/account/container/object) in addition to S3 style log.
@ -548,7 +554,9 @@ use = egg:swift#s3api
# log_name = s3api
[filter:s3token]
# See swift manual for more details.
# s3token middleware authenticates with keystone using the s3 credentials
# provided in the request header. Please put s3token between s3api
# and keystoneauth if you're using keystoneauth.
use = egg:swift#s3token
# Prefix that will be prepended to the tenant to form the account

87
swift/common/middleware/s3api/acl_handlers.py
View File

@ -12,7 +12,43 @@
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
------------
Acl Handlers
------------
Why do we need this
^^^^^^^^^^^^^^^^^^^
To make controller classes clean, we need these handlers.
It is really useful for customizing acl checking algorithms for
each controller.
Basic Information
^^^^^^^^^^^^^^^^^
BaseAclHandler wraps basic Acl handling.
(i.e. it will check acl from ACL_MAP by using HEAD)
How to extend
^^^^^^^^^^^^^
Make a handler with the name of the controller.
(e.g. BucketAclHandler is for BucketController)
It consists of method(s) for actual S3 method on controllers as follows.
Example::
class BucketAclHandler(BaseAclHandler):
def PUT:
<< put acl handling algorithms here for PUT bucket >>
.. note::
If the method DON'T need to recall _get_response in outside of
acl checking, the method have to return the response it needs at
the end of method.
"""
import sys
from swift.common.middleware.s3api.subresource import ACL, Owner, encode_acl
@ -24,34 +60,6 @@ from swift.common.middleware.s3api.utils import MULTIUPLOAD_SUFFIX, \
sysmeta_header
from contextlib import contextmanager
"""
Acl Handlers:
Why do we need this:
To make controller classes clean, we need these handlers.
It is really useful for customizing acl checking algorithms for
each controller.
Basic Information:
BaseAclHandler wraps basic Acl handling.
(i.e. it will check acl from ACL_MAP by using HEAD)
How to extend:
Make a handler with the name of the controller.
(e.g. BucketAclHandler is for BucketController)
It consists of method(s) for actual S3 method on controllers as follows.
e.g.:
class BucketAclHandler(BaseAclHandler):
def PUT:
<< put acl handling algorithms here for PUT bucket >>
NOTE:
If the method DON'T need to recall _get_response in outside of
acl checking, the method have to return the response it needs at
the end of method.
"""
def get_acl_handler(controller_name):
for base_klass in [BaseAclHandler, MultiUploadAclHandler]:
@ -315,18 +323,19 @@ class MultiUploadAclHandler(BaseAclHandler):
request to backend Swift at incoming request.
Basic Rules:
- BASE container name is always w/o 'MULTIUPLOAD_SUFFIX'
- Any check timing is ok but we should check it as soon as possible.
- BASE container name is always w/o 'MULTIUPLOAD_SUFFIX'
- Any check timing is ok but we should check it as soon as possible.
Controller | Verb | CheckResource | Permission
--------------------------------------------------
Part | PUT | Container | WRITE
Uploads | GET | Container | READ
Uploads | POST | Container | WRITE
Upload | GET | Container | READ
Upload | DELETE | Container | WRITE
Upload | POST | Container | WRITE
-------------------------------------------------
========== ====== ============= ==========
Controller Verb CheckResource Permission
========== ====== ============= ==========
Part PUT Container WRITE
Uploads GET Container READ
Uploads POST Container WRITE
Upload GET Container READ
Upload DELETE Container WRITE
Upload POST Container WRITE
========== ====== ============= ==========
"""
def __init__(self, req, logger):

8
swift/common/middleware/s3api/controllers/acl.py
View File

@ -77,10 +77,10 @@ class AclController(Controller):
"""
Handles the following APIs:
- GET Bucket acl
- PUT Bucket acl
- GET Object acl
- PUT Object acl
* GET Bucket acl
* PUT Bucket acl
* GET Object acl
* PUT Object acl
Those APIs are logged as ACL operations in the S3 server log.
"""

4
swift/common/middleware/s3api/controllers/logging.py
View File

@ -26,8 +26,8 @@ class LoggingStatusController(Controller):
"""
Handles the following APIs:
- GET Bucket logging
- PUT Bucket logging
* GET Bucket logging
* PUT Bucket logging
Those APIs are logged as LOGGING_STATUS operations in the S3 server log.
"""

59
swift/common/middleware/s3api/controllers/multi_upload.py
View File

@ -12,34 +12,51 @@
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Implementation of S3 Multipart Upload.
This module implements S3 Multipart Upload APIs with the Swift SLO feature.
The following explains how s3api uses swift container and objects to store S3
The following explains how S3api uses swift container and objects to store S3
upload information:
- [bucket]+segments
-----------------
[bucket]+segments
-----------------
A container to store upload information. [bucket] is the original bucket
where multipart upload is initiated.
A container to store upload information. [bucket] is the original bucket
where multipart upload is initiated.
- [bucket]+segments/[upload_id]
-----------------------------
[bucket]+segments/[upload_id]
-----------------------------
A object of the ongoing upload id. The object is empty and used for
checking the target upload status. If the object exists, it means that the
upload is initiated but not either completed or aborted.
A object of the ongoing upload id. The object is empty and used for
checking the target upload status. If the object exists, it means that the
upload is initiated but not either completed or aborted.
-------------------------------------------
[bucket]+segments/[upload_id]/[part_number]
-------------------------------------------
- [bucket]+segments/[upload_id]/1
[bucket]+segments/[upload_id]/2
[bucket]+segments/[upload_id]/3
The last suffix is the part number under the upload id. When the client uploads
the parts, they will be stored in the namespace with
[bucket]+segments/[upload_id]/[part_number].
Example listing result in the [bucket]+segments container::
[bucket]+segments/[upload_id1] # upload id object for upload_id1
[bucket]+segments/[upload_id1]/1 # part object for upload_id1
[bucket]+segments/[upload_id1]/2 # part object for upload_id1
[bucket]+segments/[upload_id1]/3 # part object for upload_id1
[bucket]+segments/[upload_id2] # upload id object for upload_id2
[bucket]+segments/[upload_id2]/1 # part object for upload_id2
[bucket]+segments/[upload_id2]/2 # part object for upload_id2
.
.
Uploaded part objects. Those objects are directly used as segments of Swift
Static Large Object.
Those part objects are directly used as segments of a Swift
Static Large Object when the multipart upload is completed.
"""
import os
@ -91,8 +108,8 @@ class PartController(Controller):
"""
Handles the following APIs:
- Upload Part
- Upload Part - Copy
* Upload Part
* Upload Part - Copy
Those APIs are logged as PART operations in the S3 server log.
"""
@ -168,8 +185,8 @@ class UploadsController(Controller):
"""
Handles the following APIs:
- List Multipart Uploads
- Initiate Multipart Upload
* List Multipart Uploads
* Initiate Multipart Upload
Those APIs are logged as UPLOADS operations in the S3 server log.
"""
@ -369,9 +386,9 @@ class UploadController(Controller):
"""
Handles the following APIs:
- List Parts
- Abort Multipart Upload
- Complete Multipart Upload
* List Parts
* Abort Multipart Upload
* Complete Multipart Upload
Those APIs are logged as UPLOAD operations in the S3 server log.
"""

8
swift/common/middleware/s3api/controllers/s3_acl.py
View File

@ -25,10 +25,10 @@ class S3AclController(Controller):
"""
Handles the following APIs:
- GET Bucket acl
- PUT Bucket acl
- GET Object acl
- PUT Object acl
* GET Bucket acl
* PUT Bucket acl
* GET Object acl
* PUT Object acl
Those APIs are logged as ACL operations in the S3 server log.
"""

4
swift/common/middleware/s3api/controllers/versioning.py
View File

@ -25,8 +25,8 @@ class VersioningController(Controller):
"""
Handles the following APIs:
- GET Bucket versioning
- PUT Bucket versioning
* GET Bucket versioning
* PUT Bucket versioning
Those APIs are logged as VERSIONING operations in the S3 server log.
"""

53
swift/common/middleware/s3api/s3api.py
View File

@ -35,6 +35,59 @@ An example client using the python boto library is as follows::
is_secure=False,
calling_format=boto.s3.connection.OrdinaryCallingFormat())
----------
Deployment
----------
Proxy-Server Setting
^^^^^^^^^^^^^^^^^^^^
Set s3api before your auth in your pipeline in ``proxy-server.conf`` file.
To enable all compatiblity currently supported, you should make sure that
bulk, slo, and your auth middleware are also included in your proxy
pipeline setting.
Minimum example config is::
[pipeline:main]
pipeline = proxy-logging cache s3api tempauth bulk slo proxy-logging
proxy-server
When using keystone, the config will be::
[pipeline:main]
pipeline = proxy-logging cache s3api s3token keystoneauth bulk slo
proxy-logging proxy-server
.. note::
``keystonemiddleware.authtoken`` can be located before/after s3api but
we recommend to put it before s3api because when authtoken is after s3api,
both authtoken and s3token will issue the acceptable token to keystone
(i.e. authenticate twice).
Object-Server Setting
^^^^^^^^^^^^^^^^^^^^^
To get better compatibility, you may add S3 supported headers (
Cache-Control, Content-Language, Expires, and X-Robots-Tag), that are
not supporeted in Swift by default, into allowed_headers option in
``object-server.conf`` Please see ``object-server.conf`` for more detail.
-----------
Constraints
-----------
Currently, the s3api is being ported from https://github.com/openstack/swift3
so any existing issues in swift3 are still remaining. Please make sure
descriptions in the example ``proxy-server.conf`` and what happens with the
config, before enabling the options.
-------------
Supported API
-------------
The compatibility will continue to be improved upstream, you can keep and
eye on compatibility via a check tool build by SwiftStack. See
https://github.com/swiftstack/s3compat in detail.
"""
from paste.deploy import loadwsgi

8
swift/common/middleware/s3api/s3token.py
View File

@ -20,13 +20,15 @@
# See them for their copyright.
"""
-------------------
S3 Token Middleware
This WSGI component:
-------------------
s3token middleware is for authentication with s3api + keystone.
This middleware:
* Gets a request from the s3api middleware with an S3 Authorization
access key.
* Validates s3 token in Keystone.
* Validates s3 token with Keystone.
* Transforms the account name to AUTH_%(tenant_name).
"""

78
swift/common/middleware/s3api/subresource.py
View File

@ -12,7 +12,35 @@
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
---------------------------
s3api's ACLs implementation
---------------------------
s3api uses a different implementation approach to achieve S3 ACLs.
First, we should understand what we have to design to achieve real S3 ACLs.
Current s3api(real S3)'s ACLs Model is as follows::
AccessControlPolicy:
Owner:
AccessControlList:
Grant[n]:
(Grantee, Permission)
Each bucket or object has its own acl consisting of Owner and
AcessControlList. AccessControlList can contain some Grants.
By default, AccessControlList has only one Grant to allow FULL
CONTROLL to owner. Each Grant includes single pair with Grantee,
Permission. Grantee is the user (or user group) allowed the given permission.
This module defines the groups and the relation tree.
If you wanna get more information about S3's ACLs model in detail,
please see official documentation here,
http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html
"""
from functools import partial
from swift.common.utils import json
@ -28,30 +56,6 @@ XMLNS_XSI = 'http://www.w3.org/2001/XMLSchema-instance'
PERMISSIONS = ['FULL_CONTROL', 'READ', 'WRITE', 'READ_ACP', 'WRITE_ACP']
LOG_DELIVERY_USER = '.log_delivery'
"""
An entry point of this approach is here.
We should understand what we have to design to achieve real S3 ACL.
S3's ACL Model is as follows:
AccessControlPolicy:
Owner:
AccessControlList:
Grant[n]:
(Grantee, Permission)
Each bucket or object has its own acl consists of Owner and
AcessControlList. AccessControlList can contain some Grants.
By default, AccessControlList has only one Grant to allow FULL
CONTROLL to owner. Each Grant includes single pair with Grantee,
Permission. Grantee is the user (or user group) allowed the given permission.
If you wanna get more information about S3's ACL model in detail,
please see official documentation here,
http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html
"""
def encode_acl(resource, acl):
"""
@ -125,22 +129,24 @@ class Grantee(object):
"""
Base class for grantee.
:Definition (methods):
init -> create a Grantee instance
elem -> create an ElementTree from itself
Methods:
:Definition (static methods):
from_header -> convert a grantee string in the HTTP header
* init: create a Grantee instance
* elem: create an ElementTree from itself
Static Methods:
* from_header: convert a grantee string in the HTTP header
to an Grantee instance.
from_elem -> convert a ElementTree to an Grantee instance.
* from_elem: convert a ElementTree to an Grantee instance.
TODO (not yet):
NOTE: Needs confirmation whether we really need these methods or not.
encode (method) -> create a JSON which includes whole own elements
encode_from_elem (static method) -> convert from an ElementTree to a JSON
elem_from_json (static method) -> convert from a JSON to an ElementTree
from_json (static method) -> convert a Json string to an Grantee instance.
"""
# Needs confirmation whether we really need these methods or not.
# * encode (method): create a JSON which includes whole own elements
# * encode_from_elem (static method): convert from an ElementTree to a JSON
# * elem_from_json (static method): convert from a JSON to an ElementTree
# * from_json (static method): convert a Json string to an Grantee
# instance.
def __contains__(self, key):
"""