Merge remote-tracking branch 'remotes/origin/master' into feature/deep
Change-Id: I8cbc788476385645e0333ce481a76a666f02bb25
This commit is contained in:
Alistair Coles
commit
e2f7804924
|
|
@ -0,0 +1,137 @@
|
|||
- job:
|
||||
name: swift-tox-base
|
||||
parent: openstack-tox-py27
|
||||
description: |
|
||||
Base job for swift-tox jobs.
|
||||
|
||||
It sets TMPDIR to an XFS mount point created via
|
||||
tools/test-setup.sh.
|
||||
timeout: 2400
|
||||
vars:
|
||||
tox_environment:
|
||||
TMPDIR: "{{ ansible_env.HOME }}/xfstmp"
|
||||
|
||||
- job:
|
||||
name: swift-tox-py27
|
||||
parent: swift-tox-base
|
||||
description: |
|
||||
Run unit-tests for swift under cPython version 2.7.
|
||||
|
||||
Uses tox with the ``py27`` environment.
|
||||
It sets TMPDIR to an XFS mount point created via
|
||||
tools/test-setup.sh.
|
||||
vars:
|
||||
tox_envlist: py27
|
||||
|
||||
- job:
|
||||
name: swift-tox-py27-centos-7
|
||||
parent: swift-tox-py27
|
||||
nodeset: centos-7
|
||||
|
||||
- job:
|
||||
name: swift-tox-py35
|
||||
parent: swift-tox-base
|
||||
description: |
|
||||
Run unit-tests for swift under cPython version 3.5.
|
||||
|
||||
Uses tox with the ``py35`` environment.
|
||||
It sets TMPDIR to an XFS mount point created via
|
||||
tools/test-setup.sh.
|
||||
vars:
|
||||
tox_envlist: py35
|
||||
bindep_profile: test py35
|
||||
|
||||
- job:
|
||||
name: swift-tox-func
|
||||
parent: swift-tox-base
|
||||
description: |
|
||||
Run functional tests for swift under cPython version 2.7.
|
||||
|
||||
Uses tox with the ``func`` environment.
|
||||
It sets TMPDIR to an XFS mount point created via
|
||||
tools/test-setup.sh.
|
||||
vars:
|
||||
tox_envlist: func
|
||||
|
||||
- job:
|
||||
name: swift-tox-func-centos-7
|
||||
parent: swift-tox-func
|
||||
nodeset: centos-7
|
||||
|
||||
- job:
|
||||
name: swift-tox-func-post-as-copy
|
||||
parent: swift-tox-base
|
||||
description: |
|
||||
Run functional tests for swift under cPython version 2.7.
|
||||
|
||||
Uses tox with the ``func-post-as-copy`` environment.
|
||||
It sets TMPDIR to an XFS mount point created via
|
||||
tools/test-setup.sh.
|
||||
vars:
|
||||
tox_envlist: func-post-as-copy
|
||||
|
||||
- job:
|
||||
name: swift-tox-func-post-as-copy-centos-7
|
||||
parent: swift-tox-func-post-as-copy
|
||||
nodeset: centos-7
|
||||
|
||||
- job:
|
||||
name: swift-tox-func-encryption
|
||||
parent: swift-tox-base
|
||||
description: |
|
||||
Run functional tests for swift under cPython version 2.7.
|
||||
|
||||
Uses tox with the ``func-encryption`` environment.
|
||||
It sets TMPDIR to an XFS mount point created via
|
||||
tools/test-setup.sh.
|
||||
vars:
|
||||
tox_envlist: func-encryption
|
||||
|
||||
- job:
|
||||
name: swift-tox-func-encryption-centos-7
|
||||
parent: swift-tox-func-encryption
|
||||
nodeset: centos-7
|
||||
|
||||
- job:
|
||||
name: swift-tox-func-ec
|
||||
parent: swift-tox-base
|
||||
description: |
|
||||
Run functional tests for swift under cPython version 2.7.
|
||||
|
||||
Uses tox with the ``func-post-ec`` environment.
|
||||
It sets TMPDIR to an XFS mount point created via
|
||||
tools/test-setup.sh.
|
||||
branches: ^(?!stable/ocata).*$
|
||||
vars:
|
||||
tox_envlist: func-ec
|
||||
|
||||
- job:
|
||||
name: swift-tox-func-ec-centos-7
|
||||
parent: swift-tox-func-ec
|
||||
nodeset: centos-7
|
||||
|
||||
- project:
|
||||
name: openstack/swift
|
||||
check:
|
||||
jobs:
|
||||
- swift-tox-py27
|
||||
- swift-tox-py35
|
||||
- swift-tox-func
|
||||
- swift-tox-func-post-as-copy
|
||||
- swift-tox-func-encryption
|
||||
- swift-tox-func-ec
|
||||
gate:
|
||||
jobs:
|
||||
- swift-tox-py27
|
||||
- swift-tox-py35
|
||||
- swift-tox-func
|
||||
- swift-tox-func-post-as-copy
|
||||
- swift-tox-func-encryption
|
||||
- swift-tox-func-ec
|
||||
experimental:
|
||||
jobs:
|
||||
- swift-tox-py27-centos-7
|
||||
- swift-tox-func-centos-7
|
||||
- swift-tox-func-post-as-copy-centos-7
|
||||
- swift-tox-func-encryption-centos-7
|
||||
- swift-tox-func-ec-centos-7
|
||||
1
AUTHORS
1
AUTHORS
|
|
@ -30,6 +30,7 @@ David Goetz (david.goetz@rackspace.com)
|
|||
Greg Lange (greglange@gmail.com)
|
||||
Janie Richling (jrichli@us.ibm.com)
|
||||
Michael Barton (mike@weirdlooking.com)
|
||||
Mahati Chamarthy (mahati.chamarthy@gmail.com)
|
||||
|
||||
Contributors
|
||||
------------
|
||||
|
|
|
|||
|
|
@ -164,7 +164,7 @@ ETag_obj_req:
|
|||
manifest objects, this value is the MD5 checksum of the
|
||||
concatenated string of ETag values for each of the segments in
|
||||
the manifest. You are strongly recommended to compute
|
||||
the MD5 checksum value and include it in the request. This
|
||||
the MD5 checksum value and include it in the request. This
|
||||
enables the Object Storage API to check the integrity of the
|
||||
upload. The value is not quoted.
|
||||
in: header
|
||||
|
|
@ -850,6 +850,44 @@ X-Storage-Policy:
|
|||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Symlink-Target:
|
||||
description: |
|
||||
Set to specify that this is a symlink object.
|
||||
The value is the relative path of the target object in the
|
||||
format <container>/<object>. The target object does not need to
|
||||
exist at the time of symlink creation.
|
||||
You must UTF-8-encode and then URL-encode the names of the
|
||||
container and object before you include them in this header.
|
||||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Symlink-Target-Account:
|
||||
description: |
|
||||
Set to specify that this is a cross-account symlink to
|
||||
an object in the account specified in the value.
|
||||
The ``X-Symlink-Target`` must also be set for this to
|
||||
be effective.
|
||||
You must UTF-8-encode and then URL-encode the account name
|
||||
before you include it in this header.
|
||||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Symlink-Target-Account_resp:
|
||||
description: |
|
||||
If present, and ``X-Symlink-Target`` is present, then
|
||||
this is a cross-account symlink to
|
||||
an object in the account specified in the value.
|
||||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Symlink-Target_resp:
|
||||
description: |
|
||||
If present, this is a symlink object.
|
||||
The value is the relative path of the target object in the
|
||||
format <container>/<object>.
|
||||
in: header
|
||||
required: false
|
||||
type: string
|
||||
X-Timestamp:
|
||||
description: |
|
||||
The date and time in `UNIX Epoch time stamp
|
||||
|
|
@ -1092,6 +1130,23 @@ swiftinfo_sig:
|
|||
in: query
|
||||
required: false
|
||||
type: string
|
||||
symlink:
|
||||
description: |
|
||||
If you include the ``symlink=get`` query parameter
|
||||
and the object is a symlink, then the response will include
|
||||
data and metadata from the symlink itself rather than from the target.
|
||||
in: query
|
||||
required: false
|
||||
type: string
|
||||
symlink_copy:
|
||||
description: |
|
||||
If you include the ``symlink=get`` query parameter
|
||||
and the object is a symlink, the target object
|
||||
contents are not copied. Instead, the symlink is copied to
|
||||
create a new symlink to the same target.
|
||||
in: query
|
||||
required: false
|
||||
type: string
|
||||
temp_url_expires:
|
||||
description: |
|
||||
The date and time in `UNIX Epoch time stamp
|
||||
|
|
@ -1180,5 +1235,12 @@ name_in_container_get:
|
|||
in: body
|
||||
required: true
|
||||
type: string
|
||||
symlink_path:
|
||||
description: |
|
||||
This field exists only when the object is symlink.
|
||||
This is the target path of the symlink object.
|
||||
in: body
|
||||
required: true
|
||||
type: string
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -96,6 +96,7 @@ Response Parameters
|
|||
- content_type: content_type
|
||||
- bytes: bytes_in_container_get
|
||||
- name: name_in_container_get
|
||||
- symlink_path: symlink_path
|
||||
|
||||
|
||||
Response Example format=json
|
||||
|
|
|
|||
|
|
@ -111,6 +111,7 @@ Request
|
|||
- temp_url_expires: temp_url_expires
|
||||
- filename: filename
|
||||
- multipart-manifest: multipart-manifest_get
|
||||
- symlink: symlink
|
||||
- Range: Range
|
||||
- If-Match: If-Match
|
||||
- If-None-Match: If-None-Match-get-request
|
||||
|
|
@ -139,7 +140,8 @@ Response Parameters
|
|||
- X-Openstack-Request-Id: X-Openstack-Request-Id
|
||||
- Date: Date
|
||||
- X-Static-Large-Object: X-Static-Large-Object
|
||||
|
||||
- X-Symlink-Target: X-Symlink-Target_resp
|
||||
- X-Symlink-Target-Account: X-Symlink-Target-Account_resp
|
||||
|
||||
|
||||
Response Example
|
||||
|
|
@ -263,6 +265,8 @@ Request
|
|||
- X-Object-Meta-name: X-Object-Meta-name
|
||||
- If-None-Match: If-None-Match-put-request
|
||||
- X-Trans-Id-Extra: X-Trans-Id-Extra
|
||||
- X-Symlink-Target: X-Symlink-Target
|
||||
- X-Symlink-Target-Account: X-Symlink-Target-Account
|
||||
|
||||
|
||||
Response Parameters
|
||||
|
|
@ -321,6 +325,12 @@ The new object contains the same manifest as the original.
|
|||
The segment objects are not copied. Instead, both the original
|
||||
and new manifest objects share the same set of segment objects.
|
||||
|
||||
To copy a symlink either with a COPY or a PUT with the
|
||||
``X-Copy-From`` request, include the ``symlink=get`` query string.
|
||||
The new symlink will have the same target as the original.
|
||||
The target object is not copied. Instead, both the original
|
||||
and new symlinks point to the same target object.
|
||||
|
||||
All metadata is
|
||||
preserved during the object copy. If you specify metadata on the
|
||||
request to copy the object, either PUT or COPY , the metadata
|
||||
|
|
@ -396,6 +406,7 @@ Request
|
|||
- container: container
|
||||
- object: object
|
||||
- multipart-manifest: multipart-manifest_copy
|
||||
- symlink: symlink_copy
|
||||
- X-Auth-Token: X-Auth-Token
|
||||
- X-Service-Token: X-Service-Token
|
||||
- Destination: Destination
|
||||
|
|
@ -445,6 +456,9 @@ manifest=delete`` query parameter. This operation deletes the
|
|||
segment objects and, if all deletions succeed, this operation
|
||||
deletes the manifest object.
|
||||
|
||||
A DELETE request made to a symlink path will delete the symlink
|
||||
rather than the target object.
|
||||
|
||||
An alternative to using the DELETE operation is to use
|
||||
the POST operation with the ``bulk-delete`` query parameter.
|
||||
|
||||
|
|
@ -570,6 +584,7 @@ Request
|
|||
- temp_url_expires: temp_url_expires
|
||||
- filename: filename
|
||||
- multipart-manifest: multipart-manifest_head
|
||||
- symlink: symlink
|
||||
- X-Newest: X-Newest
|
||||
- If-Match: If-Match
|
||||
- If-None-Match: If-None-Match-get-request
|
||||
|
|
@ -597,7 +612,8 @@ Response Parameters
|
|||
- Date: Date
|
||||
- X-Static-Large-Object: X-Static-Large-Object
|
||||
- Content-Type: Content-Type_obj_resp
|
||||
|
||||
- X-Symlink-Target: X-Symlink-Target_resp
|
||||
- X-Symlink-Target-Account: X-Symlink-Target-Account_resp
|
||||
|
||||
|
||||
Response Example
|
||||
|
|
@ -659,6 +675,15 @@ body. There are alternate uses of the POST operation as follows:
|
|||
can be used to upload an archive (tar file). The archive is then extracted
|
||||
to create objects.
|
||||
|
||||
A POST request must not include X-Symlink-Target header. If it does then a
|
||||
400 status code is returned and the object metadata is not modified.
|
||||
|
||||
When a POST request is sent to a symlink, the metadata will be applied to the
|
||||
symlink, but the request will result in a ``307 Temporary Redirect`` response
|
||||
to the client. The POST is never redirected to the target object, thus a
|
||||
GET/HEAD request to the symlink without ``symlink=get`` will not return the
|
||||
metadata that was sent as part of the POST request.
|
||||
|
||||
Example requests and responses:
|
||||
|
||||
- Create object metadata:
|
||||
|
|
|
|||
13
bindep.txt
13
bindep.txt
|
|
@ -6,15 +6,22 @@ gcc [platform:rpm]
|
|||
gettext [!platform:suse]
|
||||
gettext-runtime [platform:suse]
|
||||
liberasurecode-dev [platform:dpkg]
|
||||
liberasurecode-devel [platform:rpm]
|
||||
# There's no library in CentOS 7 but Fedora and openSUSE have it.
|
||||
liberasurecode-devel [platform:rpm !platform:centos]
|
||||
libffi-dev [platform:dpkg]
|
||||
libffi-devel [platform:rpm]
|
||||
memcached
|
||||
python-dev [platform:dpkg]
|
||||
python-devel [platform:rpm]
|
||||
python3-dev [platform:dpkg]
|
||||
python34-devel [platform:redhat]
|
||||
python3-devel [platfrom:suse]
|
||||
python3-devel [platform:fedora platform:suse]
|
||||
# python3-devel does not pull in the python3 package on openSUSE so
|
||||
# we need to be explicit. The python3 package contains the XML module
|
||||
# which is required by a python3 virtualenv. Similarly, in python2,
|
||||
# the XML module is located in python-xml which is not pulled in
|
||||
# by python-devel as well. See https://bugzilla.suse.com/show_bug.cgi?id=1046990
|
||||
python3 [platform:suse]
|
||||
python-xml [platform:suse]
|
||||
rsync
|
||||
xfsprogs
|
||||
libssl-dev [platform:dpkg]
|
||||
|
|
|
|||
|
|
@ -353,7 +353,7 @@ Logging address. The default is /dev/log.
|
|||
.IP \fBinterval\fR
|
||||
Minimum time for a pass to take. The default is 300 seconds.
|
||||
.IP \fBconcurrency\fR
|
||||
Number of reaper workers to spawn. The default is 4.
|
||||
Number of updater workers to spawn. The default is 4.
|
||||
.IP \fBnode_timeout\fR
|
||||
Request timeout to external services. The default is 3 seconds.
|
||||
.IP \fBconn_timeout\fR
|
||||
|
|
|
|||
|
|
@ -211,7 +211,7 @@ The default is 'expiring_objects'.
|
|||
.IP \fBreport_interval\fR
|
||||
The default is 300 seconds.
|
||||
.IP \fBconcurrency\fR
|
||||
Number of replication workers to spawn. The default is 1.
|
||||
Number of expirer workers to spawn. The default is 1.
|
||||
.IP \fBprocesses\fR
|
||||
Processes is how many parts to divide the work into, one part per process that will be doing the work.
|
||||
Processes set 0 means that a single process will be doing all the work.
|
||||
|
|
|
|||
|
|
@ -495,7 +495,7 @@ Logging address. The default is /dev/log.
|
|||
.IP \fBinterval\fR
|
||||
Minimum time for a pass to take. The default is 300 seconds.
|
||||
.IP \fBconcurrency\fR
|
||||
Number of reaper workers to spawn. The default is 1.
|
||||
Number of updater workers to spawn. The default is 1.
|
||||
.IP \fBnode_timeout\fR
|
||||
Request timeout to external services. The default is 10 seconds.
|
||||
.IP \fBobjects_per_second\fR
|
||||
|
|
@ -546,7 +546,7 @@ system specs. 0 is unlimited. The default is 20.
|
|||
Maximum bytes audited per second. Should be tuned according to individual
|
||||
system specs. 0 is unlimited. The default is 10000000.
|
||||
.IP \fBconcurrency\fR
|
||||
Number of reaper workers to spawn. The default is 1.
|
||||
Number of auditor workers to spawn. The default is 1.
|
||||
.IP \fBlog_time\fR
|
||||
The default is 3600 seconds.
|
||||
.IP \fBzero_byte_files_per_second\fR
|
||||
|
|
|
|||
|
|
@ -82,10 +82,12 @@ Get cluster socket usage stats
|
|||
Get drive audit error stats
|
||||
.IP "\fB-T, --time\fR"
|
||||
Check time synchronization
|
||||
.IP "\fB--swift-versions\fR"
|
||||
Check swift version
|
||||
.IP "\fB--all\fR"
|
||||
Perform all checks. Equivalent to \-arudlqT
|
||||
\-\-md5 \-\-sockstat \-\-auditor \-\-updater \-\-expirer
|
||||
\-\-driveaudit \-\-validate\-servers
|
||||
\-\-driveaudit \-\-validate\-servers \-\-swift-versions
|
||||
.IP "\fB--region=REGION\fR"
|
||||
Only query servers in specified region
|
||||
.IP "\fB-z ZONE, --zone=ZONE\fR"
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ eventlet_debug = true
|
|||
[pipeline:main]
|
||||
# Yes, proxy-logging appears twice. This is so that
|
||||
# middleware-originated requests get logged too.
|
||||
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache listing_formats bulk tempurl ratelimit crossdomain container_sync tempauth staticweb copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
|
||||
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache listing_formats bulk tempurl ratelimit crossdomain container_sync tempauth staticweb copy container-quotas account-quotas slo dlo versioned_writes symlink proxy-logging proxy-server
|
||||
|
||||
[filter:catch_errors]
|
||||
use = egg:swift#catch_errors
|
||||
|
|
@ -74,6 +74,9 @@ use = egg:swift#copy
|
|||
[filter:listing_formats]
|
||||
use = egg:swift#listing_formats
|
||||
|
||||
[filter:symlink]
|
||||
use = egg:swift#symlink
|
||||
|
||||
[app:proxy-server]
|
||||
use = egg:swift#proxy
|
||||
allow_account_management = true
|
||||
|
|
|
|||
|
|
@ -87,7 +87,9 @@ The Object Storage system organizes data in a hierarchy, as follows:
|
|||
object.
|
||||
|
||||
- Upload objects directly to the Object Storage system from a
|
||||
browser by using form **POST** middleware
|
||||
browser by using form **POST** middleware.
|
||||
|
||||
- Create symbolic links to other objects.
|
||||
|
||||
The account, container, and object hierarchy affects the way you
|
||||
interact with the Object Storage API.
|
||||
|
|
|
|||
|
|
@ -49,6 +49,12 @@ returns the following values for this header,
|
|||
``X-Object-Meta-*`` for objects)
|
||||
* headers listed in ``X-Container-Meta-Access-Control-Expose-Headers``
|
||||
|
||||
.. note::
|
||||
An OPTIONS request to a symlink object will respond with the options for
|
||||
the symlink only, the request will not be redirected to the target object.
|
||||
Therefore, if the symlink's target object is in another container with
|
||||
CORS settings, the response will not reflect the settings.
|
||||
|
||||
|
||||
-----------------
|
||||
Sample Javascript
|
||||
|
|
|
|||
|
|
@ -1701,11 +1701,6 @@ etc/proxy-server.conf-sample in the source code repository.
|
|||
|
||||
The following configuration sections are available:
|
||||
|
||||
An example Account Server configuration can be found at
|
||||
etc/account-server.conf-sample in the source code repository.
|
||||
|
||||
The following configuration sections are available:
|
||||
|
||||
* :ref:`[DEFAULT] <proxy_server_default_options>`
|
||||
* `[proxy-server]`_
|
||||
* Individual sections for `Proxy middlewares`_
|
||||
|
|
|
|||
|
|
@ -104,6 +104,7 @@ KS :ref:`keystoneauth`
|
|||
RL :ref:`ratelimit`
|
||||
VW :ref:`versioned_writes`
|
||||
SSC :ref:`copy`
|
||||
SYM :ref:`symlink`
|
||||
======================= =============================
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -244,6 +244,15 @@ StaticWeb
|
|||
:members:
|
||||
:show-inheritance:
|
||||
|
||||
.. _symlink:
|
||||
|
||||
Symlink
|
||||
=======
|
||||
|
||||
.. automodule:: swift.common.middleware.symlink
|
||||
:members:
|
||||
:show-inheritance:
|
||||
|
||||
.. _common_tempauth:
|
||||
|
||||
TempAuth
|
||||
|
|
|
|||
|
|
@ -36,9 +36,15 @@ synchronization key.
|
|||
.. note::
|
||||
|
||||
If you are using encryption middleware in the cluster from which objects
|
||||
are being synced, then you should follow the instructions to configure
|
||||
are being synced, then you should follow the instructions for
|
||||
:ref:`container_sync_client_config` to be compatible with encryption.
|
||||
|
||||
.. note::
|
||||
|
||||
If you are using symlink middleware in the cluster from which objects
|
||||
are being synced, then you should follow the instructions for
|
||||
:ref:`symlink_container_sync_client_config` to be compatible with symlinks.
|
||||
|
||||
--------------------------
|
||||
Configuring Container Sync
|
||||
--------------------------
|
||||
|
|
@ -440,7 +446,7 @@ then a symlink to the container database is created in a sync-containers
|
|||
sub-directory on the same device.
|
||||
|
||||
Similarly, when the container sync metadata keys are deleted, the container
|
||||
server and container-replicator would take care of deleting the symlinks
|
||||
server and container-replicator would take care of deleting the symlinks
|
||||
from ``sync-containers``.
|
||||
|
||||
.. note::
|
||||
|
|
|
|||
|
|
@ -24,13 +24,17 @@
|
|||
# log_statsd_metric_prefix =
|
||||
|
||||
[pipeline:main]
|
||||
pipeline = catch_errors proxy-logging cache proxy-server
|
||||
pipeline = catch_errors proxy-logging cache symlink proxy-server
|
||||
|
||||
[app:proxy-server]
|
||||
use = egg:swift#proxy
|
||||
account_autocreate = true
|
||||
# See proxy-server.conf-sample for options
|
||||
|
||||
[filter:symlink]
|
||||
use = egg:swift#symlink
|
||||
# See proxy-server.conf-sample for options
|
||||
|
||||
[filter:cache]
|
||||
use = egg:swift#memcache
|
||||
# See proxy-server.conf-sample for options
|
||||
|
|
|
|||
|
|
@ -94,12 +94,12 @@ bind_port = 8080
|
|||
[pipeline:main]
|
||||
# This sample pipeline uses tempauth and is used for SAIO dev work and
|
||||
# testing. See below for a pipeline using keystone.
|
||||
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache listing_formats container_sync bulk tempurl ratelimit tempauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
|
||||
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache listing_formats container_sync bulk tempurl ratelimit tempauth copy container-quotas account-quotas slo dlo versioned_writes symlink proxy-logging proxy-server
|
||||
|
||||
# The following pipeline shows keystone integration. Comment out the one
|
||||
# above and uncomment this one. Additional steps for integrating keystone are
|
||||
# covered further below in the filter sections for authtoken and keystoneauth.
|
||||
#pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit authtoken keystoneauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
|
||||
#pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit authtoken keystoneauth copy container-quotas account-quotas slo dlo versioned_writes symlink proxy-logging proxy-server
|
||||
|
||||
[app:proxy-server]
|
||||
use = egg:swift#proxy
|
||||
|
|
@ -869,12 +869,6 @@ use = egg:swift#versioned_writes
|
|||
# If you don't put it in the pipeline, it will be inserted for you.
|
||||
[filter:copy]
|
||||
use = egg:swift#copy
|
||||
# By default object POST requests update metadata without modification of the
|
||||
# original data file
|
||||
# Set this to True to enable the old, slow behavior wherein object POST
|
||||
# requests are transformed into COPY requests where source and destination are
|
||||
# the same. All client-visible behavior (save response time) should be
|
||||
# identical.
|
||||
|
||||
# Note: To enable encryption, add the following 2 dependent pieces of crypto
|
||||
# middleware to the proxy-server pipeline. They should be to the right of all
|
||||
|
|
@ -934,3 +928,13 @@ use = egg:swift#encryption
|
|||
# be automatically inserted for you.
|
||||
[filter:listing_formats]
|
||||
use = egg:swift#listing_formats
|
||||
|
||||
# Note: Put after slo, dlo, versioned_writes, but before encryption in the
|
||||
# pipeline.
|
||||
[filter:symlink]
|
||||
use = egg:swift#symlink
|
||||
# Symlinks can point to other symlinks provided the number of symlinks in a
|
||||
# chain does not exceed the symloop_max value. If the number of chained
|
||||
# symlinks exceeds the limit symloop_max a 409 (HTTPConflict) error
|
||||
# response will be produced.
|
||||
# symloop_max = 2
|
||||
|
|
|
|||
|
|
@ -107,6 +107,7 @@ paste.filter_factory =
|
|||
encryption = swift.common.middleware.crypto:filter_factory
|
||||
kms_keymaster = swift.common.middleware.crypto.kms_keymaster:filter_factory
|
||||
listing_formats = swift.common.middleware.listing_formats:filter_factory
|
||||
symlink = swift.common.middleware.symlink:filter_factory
|
||||
|
||||
[build_sphinx]
|
||||
all_files = 1
|
||||
|
|
|
|||
|
|
@ -906,6 +906,41 @@ class SwiftRecon(object):
|
|||
matches, len(hosts), errors))
|
||||
print("=" * 79)
|
||||
|
||||
def version_check(self, hosts):
|
||||
"""
|
||||
Check OS Swift version of hosts. Inform if differs.
|
||||
|
||||
:param hosts: set of hosts to check. in the format of:
|
||||
set([('127.0.0.1', 6020), ('127.0.0.2', 6030)])
|
||||
"""
|
||||
versions = set()
|
||||
errors = 0
|
||||
print("[%s] Checking versions" % self._ptime())
|
||||
recon = Scout("version", self.verbose, self.suppress_errors,
|
||||
self.timeout)
|
||||
for url, response, status, ts_start, ts_end in self.pool.imap(
|
||||
recon.scout, hosts):
|
||||
if status != 200:
|
||||
errors = errors + 1
|
||||
continue
|
||||
versions.add(response['version'])
|
||||
if self.verbose:
|
||||
print("-> %s installed version %s" % (
|
||||
url, response['version']))
|
||||
|
||||
if not len(versions):
|
||||
print("No hosts returned valid data.")
|
||||
elif len(versions) == 1:
|
||||
print("Versions matched (%s), "
|
||||
"%s error[s] while checking hosts." % (
|
||||
versions.pop(), errors))
|
||||
else:
|
||||
print("Versions not matched (%s), "
|
||||
"%s error[s] while checking hosts." % (
|
||||
", ".join(sorted(versions)), errors))
|
||||
|
||||
print("=" * 79)
|
||||
|
||||
def _get_ring_names(self, policy=None):
|
||||
"""
|
||||
Retrieve name of ring files.
|
||||
|
|
@ -982,6 +1017,8 @@ class SwiftRecon(object):
|
|||
help="Check time synchronization")
|
||||
args.add_option('--jitter', type="float", default=0.0,
|
||||
help="Maximal allowed time jitter")
|
||||
args.add_option('--swift-versions', action="store_true",
|
||||
help="Check swift versions")
|
||||
args.add_option('--top', type='int', metavar='COUNT', default=0,
|
||||
help='Also show the top COUNT entries in rank order.')
|
||||
args.add_option('--lowest', type='int', metavar='COUNT', default=0,
|
||||
|
|
@ -990,7 +1027,7 @@ class SwiftRecon(object):
|
|||
args.add_option('--all', action="store_true",
|
||||
help="Perform all checks. Equal to \t\t\t-arudlqT "
|
||||
"--md5 --sockstat --auditor --updater --expirer "
|
||||
"--driveaudit --validate-servers")
|
||||
"--driveaudit --validate-servers --swift-versions")
|
||||
args.add_option('--region', type="int",
|
||||
help="Only query servers in specified region")
|
||||
args.add_option('--zone', '-z', type="int",
|
||||
|
|
@ -1063,6 +1100,7 @@ class SwiftRecon(object):
|
|||
self.server_type_check(hosts)
|
||||
self.driveaudit_check(hosts)
|
||||
self.time_check(hosts, options.jitter)
|
||||
self.version_check(hosts)
|
||||
else:
|
||||
if options.async:
|
||||
if self.server_type == 'object':
|
||||
|
|
@ -1112,6 +1150,8 @@ class SwiftRecon(object):
|
|||
self.driveaudit_check(hosts)
|
||||
if options.time:
|
||||
self.time_check(hosts, options.jitter)
|
||||
if options.swift_versions:
|
||||
self.version_check(hosts)
|
||||
|
||||
|
||||
def main():
|
||||
|
|
|
|||
|
|
@ -898,7 +898,9 @@ swift-ring-builder <builder_file> rebalance [options]
|
|||
min_part_seconds_left = builder.min_part_seconds_left
|
||||
try:
|
||||
last_balance = builder.get_balance()
|
||||
last_dispersion = builder.dispersion
|
||||
parts, balance, removed_devs = builder.rebalance(seed=get_seed(3))
|
||||
dispersion = builder.dispersion
|
||||
except exceptions.RingBuilderError as e:
|
||||
print('-' * 79)
|
||||
print("An error has occurred during ring validation. Common\n"
|
||||
|
|
@ -922,9 +924,25 @@ swift-ring-builder <builder_file> rebalance [options]
|
|||
# special value(MAX_BALANCE) until zero weighted device return all
|
||||
# its partitions. So we cannot check balance has changed.
|
||||
# Thus we need to check balance or last_balance is special value.
|
||||
if not options.force and \
|
||||
not devs_changed and abs(last_balance - balance) < 1 and \
|
||||
not (last_balance == MAX_BALANCE and balance == MAX_BALANCE):
|
||||
be_cowardly = True
|
||||
if options.force:
|
||||
# User said save it, so we save it.
|
||||
be_cowardly = False
|
||||
elif devs_changed:
|
||||
# We must save if a device changed; this could be something like
|
||||
# a changed IP address.
|
||||
be_cowardly = False
|
||||
else:
|
||||
# If balance or dispersion changed (presumably improved), then
|
||||
# we should save to get the improvement.
|
||||
balance_changed = (
|
||||
abs(last_balance - balance) >= 1 or
|
||||
(last_balance == MAX_BALANCE and balance == MAX_BALANCE))
|
||||
dispersion_changed = abs(last_dispersion - dispersion) >= 1
|
||||
if balance_changed or dispersion_changed:
|
||||
be_cowardly = False
|
||||
|
||||
if be_cowardly:
|
||||
print('Cowardly refusing to save rebalance as it did not change '
|
||||
'at least 1%.')
|
||||
exit(EXIT_WARNING)
|
||||
|
|
@ -1085,14 +1103,16 @@ swift-ring-builder <builder_file> write_ring
|
|||
'set_info' calls when no rebalance is needed but you want to send out the
|
||||
new device information.
|
||||
"""
|
||||
if not builder.devs:
|
||||
print('Unable to write empty ring.')
|
||||
exit(EXIT_ERROR)
|
||||
|
||||
ring_data = builder.get_ring()
|
||||
if not ring_data._replica2part2dev_id:
|
||||
if ring_data.devs:
|
||||
print('Warning: Writing a ring with no partition '
|
||||
'assignments but with devices; did you forget to run '
|
||||
'"rebalance"?')
|
||||
else:
|
||||
print('Warning: Writing an empty ring')
|
||||
ring_data.save(
|
||||
pathjoin(backup_dir, '%d.' % time() + basename(ring_file)))
|
||||
ring_data.save(ring_file)
|
||||
|
|
|
|||
|
|
@ -199,6 +199,10 @@ class SegmentError(SwiftException):
|
|||
pass
|
||||
|
||||
|
||||
class LinkIterError(SwiftException):
|
||||
pass
|
||||
|
||||
|
||||
class ReplicationException(Exception):
|
||||
pass
|
||||
|
||||
|
|
|
|||
|
|
@ -157,7 +157,8 @@ class InternalClient(object):
|
|||
'auto_create_account_prefix', default='.')
|
||||
|
||||
def make_request(
|
||||
self, method, path, headers, acceptable_statuses, body_file=None):
|
||||
self, method, path, headers, acceptable_statuses, body_file=None,
|
||||
params=None):
|
||||
"""Makes a request to Swift with retries.
|
||||
|
||||
:param method: HTTP method of request.
|
||||
|
|
@ -166,6 +167,8 @@ class InternalClient(object):
|
|||
:param acceptable_statuses: List of acceptable statuses for request.
|
||||
:param body_file: Body file to be passed along with request,
|
||||
defaults to None.
|
||||
:param params: A dict of params to be set in request query string,
|
||||
defaults to None.
|
||||
|
||||
:returns: Response object on success.
|
||||
|
||||
|
|
@ -185,6 +188,8 @@ class InternalClient(object):
|
|||
if hasattr(body_file, 'seek'):
|
||||
body_file.seek(0)
|
||||
req.body_file = body_file
|
||||
if params:
|
||||
req.params = params
|
||||
try:
|
||||
resp = req.get_response(self.app)
|
||||
if resp.status_int in acceptable_statuses or \
|
||||
|
|
@ -606,14 +611,30 @@ class InternalClient(object):
|
|||
headers=headers)
|
||||
|
||||
def get_object(self, account, container, obj, headers,
|
||||
acceptable_statuses=(2,)):
|
||||
acceptable_statuses=(2,), params=None):
|
||||
"""
|
||||
Returns a 3-tuple (status, headers, iterator of object body)
|
||||
Gets an object.
|
||||
|
||||
:param account: The object's account.
|
||||
:param container: The object's container.
|
||||
:param obj: The object name.
|
||||
:param headers: Headers to send with request, defaults to empty dict.
|
||||
:param acceptable_statuses: List of status for valid responses,
|
||||
defaults to (2,).
|
||||
:param params: A dict of params to be set in request query string,
|
||||
defaults to None.
|
||||
|
||||
:raises UnexpectedResponse: Exception raised when requests fail
|
||||
to get a response with an acceptable status
|
||||
:raises Exception: Exception is raised when code fails in an
|
||||
unexpected way.
|
||||
:returns: A 3-tuple (status, headers, iterator of object body)
|
||||
"""
|
||||
|
||||
headers = headers or {}
|
||||
path = self.make_path(account, container, obj)
|
||||
resp = self.make_request('GET', path, headers, acceptable_statuses)
|
||||
resp = self.make_request(
|
||||
'GET', path, headers, acceptable_statuses, params=params)
|
||||
return (resp.status_int, resp.headers, resp.app_iter)
|
||||
|
||||
def iter_object_lines(
|
||||
|
|
@ -697,7 +718,7 @@ class InternalClient(object):
|
|||
:param account: The object's account.
|
||||
:param container: The object's container.
|
||||
:param obj: The object.
|
||||
:param headers: Headers to send with request, defaults ot empty dict.
|
||||
:param headers: Headers to send with request, defaults to empty dict.
|
||||
|
||||
:raises UnexpectedResponse: Exception raised when requests fail
|
||||
to get a response with an acceptable status
|
||||
|
|
|
|||
|
|
@ -114,45 +114,18 @@ greater than 5GB.
|
|||
|
||||
"""
|
||||
|
||||
import os
|
||||
from six.moves.configparser import ConfigParser, NoSectionError, NoOptionError
|
||||
from six.moves.urllib.parse import quote, unquote
|
||||
from six.moves.urllib.parse import quote
|
||||
|
||||
from swift.common import utils
|
||||
from swift.common.utils import get_logger, \
|
||||
config_true_value, FileLikeIter, read_conf_dir, close_if_possible
|
||||
from swift.common.utils import get_logger, config_true_value, FileLikeIter, \
|
||||
close_if_possible
|
||||
from swift.common.swob import Request, HTTPPreconditionFailed, \
|
||||
HTTPRequestEntityTooLarge, HTTPBadRequest, HTTPException
|
||||
from swift.common.http import HTTP_MULTIPLE_CHOICES, is_success, HTTP_OK
|
||||
from swift.common.constraints import check_account_format, MAX_FILE_SIZE
|
||||
from swift.common.request_helpers import copy_header_subset, remove_items, \
|
||||
is_sys_meta, is_sys_or_user_meta, is_object_transient_sysmeta
|
||||
from swift.common.wsgi import WSGIContext, make_subrequest
|
||||
|
||||
|
||||
def _check_path_header(req, name, length, error_msg):
|
||||
"""
|
||||
Validate that the value of path-like header is
|
||||
well formatted. We assume the caller ensures that
|
||||
specific header is present in req.headers.
|
||||
|
||||
:param req: HTTP request object
|
||||
:param name: header name
|
||||
:param length: length of path segment check
|
||||
:param error_msg: error message for client
|
||||
:returns: A tuple with path parts according to length
|
||||
:raise HTTPPreconditionFailed: if header value
|
||||
is not well formatted.
|
||||
"""
|
||||
src_header = unquote(req.headers.get(name))
|
||||
if not src_header.startswith('/'):
|
||||
src_header = '/' + src_header
|
||||
try:
|
||||
return utils.split_path(src_header, length, length, True)
|
||||
except ValueError:
|
||||
raise HTTPPreconditionFailed(
|
||||
request=req,
|
||||
body=error_msg)
|
||||
is_sys_meta, is_sys_or_user_meta, is_object_transient_sysmeta, \
|
||||
check_path_header
|
||||
from swift.common.wsgi import WSGIContext, make_subrequest, load_app_config
|
||||
|
||||
|
||||
def _check_copy_from_header(req):
|
||||
|
|
@ -166,9 +139,9 @@ def _check_copy_from_header(req):
|
|||
:raise HTTPPreconditionFailed: if x-copy-from value
|
||||
is not well formatted.
|
||||
"""
|
||||
return _check_path_header(req, 'X-Copy-From', 2,
|
||||
'X-Copy-From header must be of the form '
|
||||
'<container name>/<object name>')
|
||||
return check_path_header(req, 'X-Copy-From', 2,
|
||||
'X-Copy-From header must be of the form '
|
||||
'<container name>/<object name>')
|
||||
|
||||
|
||||
def _check_destination_header(req):
|
||||
|
|
@ -182,9 +155,9 @@ def _check_destination_header(req):
|
|||
:raise HTTPPreconditionFailed: if destination value
|
||||
is not well formatted.
|
||||
"""
|
||||
return _check_path_header(req, 'Destination', 2,
|
||||
'Destination header must be of the form '
|
||||
'<container name>/<object name>')
|
||||
return check_path_header(req, 'Destination', 2,
|
||||
'Destination header must be of the form '
|
||||
'<container name>/<object name>')
|
||||
|
||||
|
||||
def _copy_headers(src, dest):
|
||||
|
|
@ -263,25 +236,9 @@ class ServerSideCopyMiddleware(object):
|
|||
# This takes preference over the one set in proxy app
|
||||
return
|
||||
|
||||
cp = ConfigParser()
|
||||
if os.path.isdir(conf['__file__']):
|
||||
read_conf_dir(cp, conf['__file__'])
|
||||
else:
|
||||
cp.read(conf['__file__'])
|
||||
|
||||
try:
|
||||
pipe = cp.get("pipeline:main", "pipeline")
|
||||
except (NoSectionError, NoOptionError):
|
||||
return
|
||||
|
||||
proxy_name = pipe.rsplit(None, 1)[-1]
|
||||
proxy_section = "app:" + proxy_name
|
||||
|
||||
try:
|
||||
conf['object_post_as_copy'] = cp.get(proxy_section,
|
||||
'object_post_as_copy')
|
||||
except (NoSectionError, NoOptionError):
|
||||
pass
|
||||
proxy_conf = load_app_config(conf['__file__'])
|
||||
if 'object_post_as_copy' in proxy_conf:
|
||||
conf['object_post_as_copy'] = proxy_conf['object_post_as_copy']
|
||||
|
||||
def __call__(self, env, start_response):
|
||||
req = Request(env)
|
||||
|
|
|
|||
|
|
@ -103,13 +103,14 @@ class BaseDecrypterContext(CryptoWSGIContext):
|
|||
to be decrypted but crypto meta was not
|
||||
found.
|
||||
"""
|
||||
value, crypto_meta = extract_crypto_meta(value)
|
||||
extracted_value, crypto_meta = extract_crypto_meta(value)
|
||||
if crypto_meta:
|
||||
self.crypto.check_crypto_meta(crypto_meta)
|
||||
value = self.decrypt_value(value, key, crypto_meta)
|
||||
value = self.decrypt_value(extracted_value, key, crypto_meta)
|
||||
elif required:
|
||||
raise EncryptionException(
|
||||
"Missing crypto meta in value %s" % value)
|
||||
|
||||
return value
|
||||
|
||||
def decrypt_value(self, value, key, crypto_meta):
|
||||
|
|
|
|||
|
|
@ -12,17 +12,13 @@
|
|||
# implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import os
|
||||
import string
|
||||
|
||||
import six
|
||||
|
||||
from swift.common.middleware.crypto.crypto_utils import CRYPTO_KEY_CALLBACK
|
||||
from swift.common.swob import Request, HTTPException
|
||||
from swift.common.utils import readconf
|
||||
from swift.common.utils import readconf, strict_b64decode
|
||||
from swift.common.wsgi import WSGIContext
|
||||
|
||||
|
||||
|
|
@ -141,17 +137,12 @@ class KeyMaster(object):
|
|||
conf = readconf(self.keymaster_config_path, 'keymaster')
|
||||
b64_root_secret = conf.get('encryption_root_secret')
|
||||
try:
|
||||
# b64decode will silently discard bad characters, but we should
|
||||
# treat them as an error
|
||||
if not isinstance(b64_root_secret, six.string_types) or any(
|
||||
c not in string.digits + string.ascii_letters + '/+\r\n'
|
||||
for c in b64_root_secret.strip('\r\n=')):
|
||||
raise ValueError
|
||||
binary_root_secret = base64.b64decode(b64_root_secret)
|
||||
binary_root_secret = strict_b64decode(b64_root_secret,
|
||||
allow_line_breaks=True)
|
||||
if len(binary_root_secret) < 32:
|
||||
raise ValueError
|
||||
return binary_root_secret
|
||||
except (TypeError, ValueError):
|
||||
except ValueError:
|
||||
raise ValueError(
|
||||
'encryption_root_secret option in %s must be a base64 '
|
||||
'encoding of at least 32 raw bytes' % (
|
||||
|
|
|
|||
|
|
@ -119,10 +119,8 @@ Here's an example using ``curl`` with tiny 1-byte segments::
|
|||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
import six
|
||||
from six.moves.configparser import ConfigParser, NoSectionError, NoOptionError
|
||||
from six.moves.urllib.parse import unquote
|
||||
|
||||
from hashlib import md5
|
||||
|
|
@ -132,10 +130,9 @@ from swift.common.http import is_success
|
|||
from swift.common.swob import Request, Response, \
|
||||
HTTPRequestedRangeNotSatisfiable, HTTPBadRequest, HTTPConflict
|
||||
from swift.common.utils import get_logger, \
|
||||
RateLimitedIterator, read_conf_dir, quote, close_if_possible, \
|
||||
closing_if_possible
|
||||
RateLimitedIterator, quote, close_if_possible, closing_if_possible
|
||||
from swift.common.request_helpers import SegmentedIterable
|
||||
from swift.common.wsgi import WSGIContext, make_subrequest
|
||||
from swift.common.wsgi import WSGIContext, make_subrequest, load_app_config
|
||||
|
||||
|
||||
class GetContext(WSGIContext):
|
||||
|
|
@ -381,26 +378,12 @@ class DynamicLargeObject(object):
|
|||
'__file__' not in conf):
|
||||
return
|
||||
|
||||
cp = ConfigParser()
|
||||
if os.path.isdir(conf['__file__']):
|
||||
read_conf_dir(cp, conf['__file__'])
|
||||
else:
|
||||
cp.read(conf['__file__'])
|
||||
|
||||
try:
|
||||
pipe = cp.get("pipeline:main", "pipeline")
|
||||
except (NoSectionError, NoOptionError):
|
||||
return
|
||||
|
||||
proxy_name = pipe.rsplit(None, 1)[-1]
|
||||
proxy_section = "app:" + proxy_name
|
||||
proxy_conf = load_app_config(conf['__file__'])
|
||||
for setting in ('rate_limit_after_segment',
|
||||
'rate_limit_segments_per_sec',
|
||||
'max_get_time'):
|
||||
try:
|
||||
conf[setting] = cp.get(proxy_section, setting)
|
||||
except (NoSectionError, NoOptionError):
|
||||
pass
|
||||
if setting in proxy_conf:
|
||||
conf[setting] = proxy_conf[setting]
|
||||
|
||||
def __call__(self, env, start_response):
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -92,6 +92,7 @@ def container_to_xml(listing, base_name):
|
|||
'last_modified'):
|
||||
SubElement(sub, field).text = six.text_type(
|
||||
record.pop(field))
|
||||
|
||||
return tostring(doc, encoding='UTF-8').replace(
|
||||
"<?xml version='1.0' encoding='UTF-8'?>",
|
||||
'<?xml version="1.0" encoding="UTF-8"?>', 1)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,579 @@
|
|||
# Copyright (c) 2010-2017 OpenStack Foundation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
# implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
"""
|
||||
Symlink Middleware
|
||||
|
||||
Symlinks are objects stored in Swift that contain a reference to another
|
||||
object (hereinafter, this is called "target object"). They are analogous to
|
||||
symbolic links in Unix-like operating systems. The existence of a symlink
|
||||
object does not affect the target object in any way. An important use case is
|
||||
to use a path in one container to access an object in a different container,
|
||||
with a different policy. This allows policy cost/performance trade-offs to be
|
||||
made on individual objects.
|
||||
|
||||
Clients create a Swift symlink by performing a zero-length PUT request
|
||||
with the header ``X-Symlink-Target: <container>/<object>``. For a cross-account
|
||||
symlink, the header ``X-Symlink-Target-Account: <account>`` must be included.
|
||||
If omitted, it is inserted automatically with the account of the symlink
|
||||
object in the PUT request process.
|
||||
|
||||
Symlinks must be zero-byte objects. Attempting to PUT a symlink
|
||||
with a non-empty request body will result in a 400-series error. Also, POST
|
||||
with X-Symlink-Target header always results in a 400-series error. The target
|
||||
object need not exist at symlink creation time. It is suggested to set the
|
||||
``Content-Type`` of symlink objects to a distinct value such as
|
||||
``application/symlink``.
|
||||
|
||||
A GET/HEAD request to a symlink will result in a request to the target
|
||||
object referenced by the symlink's ``X-Symlink-Target-Account`` and
|
||||
``X-Symlink-Target`` headers. The response of the GET/HEAD request will contain
|
||||
a ``Content-Location`` header with the path location of the target object. A
|
||||
GET/HEAD request to a symlink with the query parameter ``?symlink=get`` will
|
||||
result in the request targeting the symlink itself.
|
||||
|
||||
A symlink can point to another symlink. Chained symlinks will be traversed
|
||||
until target is not a symlink. If the number of chained symlinks exceeds the
|
||||
limit ``symloop_max`` an error response will be produced. The value of
|
||||
``symloop_max`` can be defined in the symlink config section of
|
||||
`proxy-server.conf`. If not specified, the default ``symloop_max`` value is 2.
|
||||
If a value less than 1 is specified, the default value will be used.
|
||||
|
||||
A HEAD/GET request to a symlink object behaves as a normal HEAD/GET request
|
||||
to the target object. Therefore issuing a HEAD request to the symlink will
|
||||
return the target metadata, and issuing a GET request to the symlink will
|
||||
return the data and metadata of the target object. To return the symlink
|
||||
metadata (with its empty body) a GET/HEAD request with the ``?symlink=get``
|
||||
query parameter must be sent to a symlink object.
|
||||
|
||||
A POST request to a symlink will result in a 307 TemporaryRedirect response.
|
||||
The response will contain a ``Location`` header with the path of the target
|
||||
object as the value. The request is never redirected to the target object by
|
||||
Swift. Nevertheless, the metadata in the POST request will be applied to the
|
||||
symlink because object servers cannot know for sure if the current object is a
|
||||
symlink or not in eventual consistency.
|
||||
|
||||
A DELETE request to a symlink will delete the symlink itself. The target
|
||||
object will not be deleted.
|
||||
|
||||
A COPY request, or a PUT request with a ``X-Copy-From`` header, to a symlink
|
||||
will copy the target object. The same request to a symlink with the query
|
||||
parameter ``?symlink=get`` will copy the symlink itself.
|
||||
|
||||
An OPTIONS request to a symlink will respond with the options for the symlink
|
||||
only, the request will not be redirected to the target object. Please note that
|
||||
if the symlink's target object is in another container with CORS settings, the
|
||||
response will not reflect the settings.
|
||||
|
||||
Tempurls can be used to GET/HEAD symlink objects, but PUT is not allowed and
|
||||
will result in a 400-series error. The GET/HEAD tempurls honor the scope of
|
||||
the tempurl key. Container tempurl will only work on symlinks where the target
|
||||
container is the same as the symlink. In case a symlink targets an object
|
||||
in a different container, a GET/HEAD request will result in a 401 Unauthorized
|
||||
error. The account level tempurl will allow cross container symlinks.
|
||||
|
||||
If a symlink object is overwritten while it is in a versioned container, the
|
||||
symlink object itself is versioned, not the referenced object.
|
||||
|
||||
A GET request with query parameter ``?format=json`` to a container which
|
||||
contains symlinks will respond with additional information ``symlink_path``
|
||||
for each symlink object in the container listing. The ``symlink_path`` value
|
||||
is the target path of the symlink. Clients can differentiate symlinks and
|
||||
other objects by this function. Note that responses of any other format
|
||||
(e.g.``?format=xml``) won't include ``symlink_path`` info.
|
||||
|
||||
Errors
|
||||
|
||||
* PUT with the header ``X-Symlink-Target`` with non-zero Content-Length
|
||||
will produce a 400 BadRequest error.
|
||||
|
||||
* POST with the header ``X-Symlink-Target`` will produce a
|
||||
400 BadRequest error.
|
||||
|
||||
* GET/HEAD traversing more than ``symloop_max`` chained symlinks will
|
||||
produce a 409 Conflict error.
|
||||
|
||||
* POSTs will produce a 307 TemporaryRedirect error.
|
||||
|
||||
----------
|
||||
Deployment
|
||||
----------
|
||||
|
||||
Symlinks are enabled by adding the `symlink` middleware to the proxy server
|
||||
WSGI pipeline and including a corresponding filter configuration section in the
|
||||
`proxy-server.conf` file. The `symlink` middleware should be placed after
|
||||
`slo`, `dlo` and `versioned_writes` middleware, but before `encryption`
|
||||
middleware in the pipeline. See the `proxy-server.conf-sample` file for further
|
||||
details. :ref:`Additional steps <symlink_container_sync_client_config>` are
|
||||
required if the container sync feature is being used.
|
||||
|
||||
.. note::
|
||||
|
||||
Once you have deployed `symlink` middleware in your pipeline, you should
|
||||
neither remove the `symlink` middleware nor downgrade swift to a version
|
||||
earlier than symlinks being supported. Doing so may result in unexpected
|
||||
container listing results in addition to symlink objects behaving like a
|
||||
normal object.
|
||||
|
||||
.. _symlink_container_sync_client_config:
|
||||
|
||||
Container sync configuration
|
||||
----------------------------
|
||||
|
||||
If container sync is being used then the `symlink` middleware
|
||||
must be added to the container sync internal client pipeline. The following
|
||||
configuration steps are required:
|
||||
|
||||
#. Create a custom internal client configuration file for container sync (if
|
||||
one is not already in use) based on the sample file
|
||||
`internal-client.conf-sample`. For example, copy
|
||||
`internal-client.conf-sample` to `/etc/swift/container-sync-client.conf`.
|
||||
#. Modify this file to include the `symlink` middleware in the pipeline in
|
||||
the same way as described above for the proxy server.
|
||||
#. Modify the container-sync section of all container server config files to
|
||||
point to this internal client config file using the
|
||||
``internal_client_conf_path`` option. For example::
|
||||
|
||||
internal_client_conf_path = /etc/swift/container-sync-client.conf
|
||||
|
||||
.. note::
|
||||
|
||||
These container sync configuration steps will be necessary for container
|
||||
sync probe tests to pass if the `symlink` middleware is included in the
|
||||
proxy pipeline of a test cluster.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
from cgi import parse_header
|
||||
from six.moves.urllib.parse import unquote
|
||||
|
||||
from swift.common.utils import get_logger, register_swift_info, split_path, \
|
||||
MD5_OF_EMPTY_STRING, closing_if_possible
|
||||
from swift.common.constraints import check_account_format
|
||||
from swift.common.wsgi import WSGIContext, make_subrequest
|
||||
from swift.common.request_helpers import get_sys_meta_prefix, \
|
||||
check_path_header
|
||||
from swift.common.swob import Request, HTTPBadRequest, HTTPTemporaryRedirect, \
|
||||
HTTPException, HTTPConflict, HTTPPreconditionFailed
|
||||
from swift.common.http import is_success
|
||||
from swift.common.exceptions import LinkIterError
|
||||
from swift.common.header_key_dict import HeaderKeyDict
|
||||
|
||||
DEFAULT_SYMLOOP_MAX = 2
|
||||
# Header values for symlink target path strings will be quoted values.
|
||||
TGT_OBJ_SYMLINK_HDR = 'x-symlink-target'
|
||||
TGT_ACCT_SYMLINK_HDR = 'x-symlink-target-account'
|
||||
TGT_OBJ_SYSMETA_SYMLINK_HDR = get_sys_meta_prefix('object') + 'symlink-target'
|
||||
TGT_ACCT_SYSMETA_SYMLINK_HDR = \
|
||||
get_sys_meta_prefix('object') + 'symlink-target-account'
|
||||
|
||||
|
||||
def _check_symlink_header(req):
|
||||
"""
|
||||
Validate that the value from x-symlink-target header is
|
||||
well formatted. We assume the caller ensures that
|
||||
x-symlink-target header is present in req.headers.
|
||||
|
||||
:param req: HTTP request object
|
||||
:raise: HTTPPreconditionFailed if x-symlink-target value
|
||||
is not well formatted.
|
||||
:raise: HTTPBadRequest if the x-symlink-target value points to the request
|
||||
path.
|
||||
"""
|
||||
# N.B. check_path_header doesn't assert the leading slash and
|
||||
# copy middleware may accept the format. In the symlink, API
|
||||
# says apparently to use "container/object" format so add the
|
||||
# validation first, here.
|
||||
if unquote(req.headers[TGT_OBJ_SYMLINK_HDR]).startswith('/'):
|
||||
raise HTTPPreconditionFailed(
|
||||
body='X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>',
|
||||
request=req, content_type='text/plain')
|
||||
|
||||
# check container and object format
|
||||
container, obj = check_path_header(
|
||||
req, TGT_OBJ_SYMLINK_HDR, 2,
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
|
||||
# Check account format if it exists
|
||||
account = check_account_format(
|
||||
req, unquote(req.headers[TGT_ACCT_SYMLINK_HDR])) \
|
||||
if TGT_ACCT_SYMLINK_HDR in req.headers else None
|
||||
|
||||
# Extract request path
|
||||
_junk, req_acc, req_cont, req_obj = req.split_path(4, 4, True)
|
||||
|
||||
if not account:
|
||||
account = req_acc
|
||||
|
||||
# Check if symlink targets the symlink itself or not
|
||||
if (account, container, obj) == (req_acc, req_cont, req_obj):
|
||||
raise HTTPBadRequest(
|
||||
body='Symlink cannot target itself',
|
||||
request=req, content_type='text/plain')
|
||||
|
||||
|
||||
def symlink_usermeta_to_sysmeta(headers):
|
||||
"""
|
||||
Helper function to translate from X-Symlink-Target and
|
||||
X-Symlink-Target-Account to X-Object-Sysmeta-Symlink-Target
|
||||
and X-Object-Sysmeta-Symlink-Target-Account.
|
||||
|
||||
:param headers: request headers dict. Note that the headers dict
|
||||
will be updated directly.
|
||||
"""
|
||||
# To preseve url-encoded value in the symlink header, use raw value
|
||||
if TGT_OBJ_SYMLINK_HDR in headers:
|
||||
headers[TGT_OBJ_SYSMETA_SYMLINK_HDR] = headers.pop(
|
||||
TGT_OBJ_SYMLINK_HDR)
|
||||
|
||||
if TGT_ACCT_SYMLINK_HDR in headers:
|
||||
headers[TGT_ACCT_SYSMETA_SYMLINK_HDR] = headers.pop(
|
||||
TGT_ACCT_SYMLINK_HDR)
|
||||
|
||||
|
||||
def symlink_sysmeta_to_usermeta(headers):
|
||||
"""
|
||||
Helper function to translate from X-Object-Sysmeta-Symlink-Target and
|
||||
X-Object-Sysmeta-Symlink-Target-Account to X-Symlink-Target and
|
||||
X-Sysmeta-Symlink-Target-Account
|
||||
|
||||
:param headers: request headers dict. Note that the headers dict
|
||||
will be updated directly.
|
||||
"""
|
||||
if TGT_OBJ_SYSMETA_SYMLINK_HDR in headers:
|
||||
headers[TGT_OBJ_SYMLINK_HDR] = headers.pop(
|
||||
TGT_OBJ_SYSMETA_SYMLINK_HDR)
|
||||
|
||||
if TGT_ACCT_SYSMETA_SYMLINK_HDR in headers:
|
||||
headers[TGT_ACCT_SYMLINK_HDR] = headers.pop(
|
||||
TGT_ACCT_SYSMETA_SYMLINK_HDR)
|
||||
|
||||
|
||||
class SymlinkContainerContext(WSGIContext):
|
||||
def __init__(self, wsgi_app, logger):
|
||||
super(SymlinkContainerContext, self).__init__(wsgi_app)
|
||||
self.logger = logger
|
||||
|
||||
def handle_container(self, req, start_response):
|
||||
"""
|
||||
Handle container requests.
|
||||
|
||||
:param req: a :class:`~swift.common.swob.Request`
|
||||
:param start_response: start_response function
|
||||
|
||||
:return: Response Iterator after start_response called.
|
||||
"""
|
||||
app_resp = self._app_call(req.environ)
|
||||
|
||||
if req.method == 'GET' and is_success(self._get_status_int()):
|
||||
app_resp = self._process_json_resp(app_resp, req)
|
||||
|
||||
start_response(self._response_status, self._response_headers,
|
||||
self._response_exc_info)
|
||||
|
||||
return app_resp
|
||||
|
||||
def _process_json_resp(self, resp_iter, req):
|
||||
"""
|
||||
Iterate through json body looking for symlinks and modify its content
|
||||
:return: modified json body
|
||||
"""
|
||||
with closing_if_possible(resp_iter):
|
||||
resp_body = ''.join(resp_iter)
|
||||
body_json = json.loads(resp_body)
|
||||
swift_version, account, _junk = split_path(req.path, 2, 3, True)
|
||||
new_body = json.dumps(
|
||||
[self._extract_symlink_path_json(obj_dict, swift_version, account)
|
||||
for obj_dict in body_json])
|
||||
self.update_content_length(len(new_body))
|
||||
return [new_body]
|
||||
|
||||
def _extract_symlink_path_json(self, obj_dict, swift_version, account):
|
||||
"""
|
||||
Extract the symlink path from the hash value
|
||||
:return: object dictionary with additional key:value pair if object
|
||||
is a symlink. The new key is symlink_path.
|
||||
"""
|
||||
if 'hash' in obj_dict:
|
||||
hash_value, meta = parse_header(obj_dict['hash'])
|
||||
obj_dict['hash'] = hash_value
|
||||
target = None
|
||||
for key in meta:
|
||||
if key == 'symlink_target':
|
||||
target = meta[key]
|
||||
elif key == 'symlink_target_account':
|
||||
account = meta[key]
|
||||
else:
|
||||
# make sure to add all other (key, values) back in place
|
||||
obj_dict['hash'] += '; %s=%s' % (key, meta[key])
|
||||
else:
|
||||
if target:
|
||||
obj_dict['symlink_path'] = os.path.join(
|
||||
'/', swift_version, account, target)
|
||||
|
||||
return obj_dict
|
||||
|
||||
|
||||
class SymlinkObjectContext(WSGIContext):
|
||||
|
||||
def __init__(self, wsgi_app, logger, symloop_max):
|
||||
super(SymlinkObjectContext, self).__init__(wsgi_app)
|
||||
self.symloop_max = symloop_max
|
||||
self.logger = logger
|
||||
# N.B. _loop_count and _last_target_path are used to keep
|
||||
# the statement in the _recursive_get. Hence they should not be touched
|
||||
# from other resources.
|
||||
self._loop_count = 0
|
||||
self._last_target_path = None
|
||||
|
||||
def handle_get_head_symlink(self, req):
|
||||
"""
|
||||
Handle get/head request when client sent parameter ?symlink=get
|
||||
|
||||
:param req: HTTP GET or HEAD object request with param ?symlink=get
|
||||
:returns: Response Iterator
|
||||
"""
|
||||
resp = self._app_call(req.environ)
|
||||
response_header_dict = HeaderKeyDict(self._response_headers)
|
||||
symlink_sysmeta_to_usermeta(response_header_dict)
|
||||
self._response_headers = response_header_dict.items()
|
||||
return resp
|
||||
|
||||
def handle_get_head(self, req):
|
||||
"""
|
||||
Handle get/head request and in case the response is a symlink,
|
||||
redirect request to target object.
|
||||
|
||||
:param req: HTTP GET or HEAD object request
|
||||
:returns: Response Iterator
|
||||
"""
|
||||
try:
|
||||
return self._recursive_get_head(req)
|
||||
except LinkIterError:
|
||||
errmsg = 'Too many levels of symbolic links, ' \
|
||||
'maximum allowed is %d' % self.symloop_max
|
||||
raise HTTPConflict(
|
||||
body=errmsg, request=req, content_type='text/plain')
|
||||
|
||||
def _recursive_get_head(self, req):
|
||||
resp = self._app_call(req.environ)
|
||||
|
||||
def build_traversal_req(symlink_target):
|
||||
"""
|
||||
:returns: new request for target path if it's symlink otherwise
|
||||
None
|
||||
"""
|
||||
version, account, _junk = split_path(req.path, 2, 3, True)
|
||||
account = self._response_header_value(
|
||||
TGT_ACCT_SYSMETA_SYMLINK_HDR) or account
|
||||
target_path = os.path.join(
|
||||
'/', version, account,
|
||||
symlink_target.lstrip('/'))
|
||||
self._last_target_path = target_path
|
||||
new_req = make_subrequest(
|
||||
req.environ, path=target_path, method=req.method,
|
||||
headers=req.headers, swift_source='SYM')
|
||||
new_req.headers.pop('X-Backend-Storage-Policy-Index', None)
|
||||
return new_req
|
||||
|
||||
symlink_target = self._response_header_value(
|
||||
TGT_OBJ_SYSMETA_SYMLINK_HDR)
|
||||
if symlink_target:
|
||||
if self._loop_count >= self.symloop_max:
|
||||
raise LinkIterError()
|
||||
# format: /<account name>/<container name>/<object name>
|
||||
new_req = build_traversal_req(symlink_target)
|
||||
self._loop_count += 1
|
||||
return self._recursive_get_head(new_req)
|
||||
else:
|
||||
if self._last_target_path:
|
||||
# Content-Location will be applied only when one or more
|
||||
# symlink recursion occurred.
|
||||
# In this case, Content-Location is applied to show which
|
||||
# object path caused the error response.
|
||||
# To preserve '%2F'(= quote('/')) in X-Symlink-Target
|
||||
# header value as it is, Content-Location value comes from
|
||||
# TGT_OBJ_SYMLINK_HDR, not req.path
|
||||
self._response_headers.extend(
|
||||
[('Content-Location', self._last_target_path)])
|
||||
|
||||
return resp
|
||||
|
||||
def handle_put(self, req):
|
||||
"""
|
||||
Handle put request when it contains X-Symlink-Target header.
|
||||
|
||||
Symlink headers are validated and moved to sysmeta namespace.
|
||||
:param req: HTTP PUT object request
|
||||
:returns: Response Iterator
|
||||
"""
|
||||
if req.content_length != 0:
|
||||
raise HTTPBadRequest(
|
||||
body='Symlink requests require a zero byte body',
|
||||
request=req,
|
||||
content_type='text/plain')
|
||||
|
||||
_check_symlink_header(req)
|
||||
symlink_usermeta_to_sysmeta(req.headers)
|
||||
# Store info in container update that this object is a symlink.
|
||||
# We have a design decision to use etag space to store symlink info for
|
||||
# object listing because it's immutable unless the object is
|
||||
# overwritten. This may impact the downgrade scenario that the symlink
|
||||
# info can be appreared as the suffix in the hash value of object
|
||||
# listing result for clients.
|
||||
# To create override etag easily, we have a contraint that the symlink
|
||||
# must be 0 byte so we can add etag of the empty string + symlink info
|
||||
# here, simply. Note that this override etag may be encrypted in the
|
||||
# container db by encrypion middleware.
|
||||
etag_override = [
|
||||
MD5_OF_EMPTY_STRING,
|
||||
'symlink_target=%s' % req.headers[TGT_OBJ_SYSMETA_SYMLINK_HDR]
|
||||
]
|
||||
if TGT_ACCT_SYSMETA_SYMLINK_HDR in req.headers:
|
||||
etag_override.append(
|
||||
'symlink_target_account=%s' %
|
||||
req.headers[TGT_OBJ_SYSMETA_SYMLINK_HDR])
|
||||
req.headers['X-Object-Sysmeta-Container-Update-Override-Etag'] = \
|
||||
'; '.join(etag_override)
|
||||
|
||||
return self._app_call(req.environ)
|
||||
|
||||
def handle_post(self, req):
|
||||
"""
|
||||
Handle post request. If POSTing to a symlink, a HTTPTemporaryRedirect
|
||||
error message is returned to client.
|
||||
|
||||
Clients that POST to symlinks should understand that the POST is not
|
||||
redirected to the target object like in a HEAD/GET request. POSTs to a
|
||||
symlink will be handled just like a normal object by the object server.
|
||||
It cannot reject it because it may not have symlink state when the POST
|
||||
lands. The object server has no knowledge of what is a symlink object
|
||||
is. On the other hand, on POST requests, the object server returns all
|
||||
sysmeta of the object. This method uses that sysmeta to determine if
|
||||
the stored object is a symlink or not.
|
||||
|
||||
:param req: HTTP POST object request
|
||||
:raises: HTTPTemporaryRedirect if POSTing to a symlink.
|
||||
:returns: Response Iterator
|
||||
"""
|
||||
if TGT_OBJ_SYMLINK_HDR in req.headers:
|
||||
raise HTTPBadRequest(
|
||||
body='A PUT request is required to set a symlink target',
|
||||
request=req,
|
||||
content_type='text/plain')
|
||||
|
||||
resp = self._app_call(req.environ)
|
||||
if not is_success(self._get_status_int()):
|
||||
return resp
|
||||
|
||||
tgt_co = self._response_header_value(TGT_OBJ_SYSMETA_SYMLINK_HDR)
|
||||
if tgt_co:
|
||||
version, account, _junk = req.split_path(2, 3, True)
|
||||
target_acc = self._response_header_value(
|
||||
TGT_ACCT_SYSMETA_SYMLINK_HDR) or account
|
||||
location_hdr = os.path.join(
|
||||
'/', version, target_acc, tgt_co)
|
||||
req.environ['swift.leave_relative_location'] = True
|
||||
errmsg = 'The requested POST was applied to a symlink. POST ' +\
|
||||
'directly to the target to apply requested metadata.'
|
||||
raise HTTPTemporaryRedirect(
|
||||
body=errmsg, headers={'location': location_hdr})
|
||||
else:
|
||||
return resp
|
||||
|
||||
def handle_object(self, req, start_response):
|
||||
"""
|
||||
Handle object requests.
|
||||
|
||||
:param req: a :class:`~swift.common.swob.Request`
|
||||
:param start_response: start_response function
|
||||
:returns: Response Iterator after start_response has been called
|
||||
"""
|
||||
if req.method in ('GET', 'HEAD'):
|
||||
# if GET request came from versioned writes, then it should get
|
||||
# the symlink only, not the referenced target
|
||||
if req.params.get('symlink') == 'get' or \
|
||||
req.environ.get('swift.source') == 'VW':
|
||||
resp = self.handle_get_head_symlink(req)
|
||||
else:
|
||||
resp = self.handle_get_head(req)
|
||||
elif req.method == 'PUT' and (TGT_OBJ_SYMLINK_HDR in req.headers):
|
||||
resp = self.handle_put(req)
|
||||
elif req.method == 'POST':
|
||||
resp = self.handle_post(req)
|
||||
else:
|
||||
# DELETE and OPTIONS reqs for a symlink and
|
||||
# PUT reqs without X-Symlink-Target behave like any other object
|
||||
resp = self._app_call(req.environ)
|
||||
|
||||
start_response(self._response_status, self._response_headers,
|
||||
self._response_exc_info)
|
||||
|
||||
return resp
|
||||
|
||||
|
||||
class SymlinkMiddleware(object):
|
||||
"""
|
||||
Middleware that implements symlinks.
|
||||
|
||||
Symlinks are objects stored in Swift that contain a reference to another
|
||||
object (i.e., the target object). An important use case is to use a path in
|
||||
one container to access an object in a different container, with a
|
||||
different policy. This allows policy cost/performance trade-offs to be made
|
||||
on individual objects.
|
||||
"""
|
||||
|
||||
def __init__(self, app, conf, symloop_max):
|
||||
self.app = app
|
||||
self.conf = conf
|
||||
self.logger = get_logger(self.conf, log_route='symlink')
|
||||
self.symloop_max = symloop_max
|
||||
|
||||
def __call__(self, env, start_response):
|
||||
req = Request(env)
|
||||
try:
|
||||
version, acc, cont, obj = req.split_path(3, 4, True)
|
||||
except ValueError:
|
||||
return self.app(env, start_response)
|
||||
|
||||
try:
|
||||
if obj:
|
||||
# object context
|
||||
context = SymlinkObjectContext(self.app, self.logger,
|
||||
self.symloop_max)
|
||||
return context.handle_object(req, start_response)
|
||||
else:
|
||||
# container context
|
||||
context = SymlinkContainerContext(self.app, self.logger)
|
||||
return context.handle_container(req, start_response)
|
||||
except HTTPException as err_resp:
|
||||
return err_resp(env, start_response)
|
||||
|
||||
|
||||
def filter_factory(global_conf, **local_conf):
|
||||
conf = global_conf.copy()
|
||||
conf.update(local_conf)
|
||||
|
||||
symloop_max = int(conf.get('symloop_max', DEFAULT_SYMLOOP_MAX))
|
||||
if symloop_max < 1:
|
||||
symloop_max = int(DEFAULT_SYMLOOP_MAX)
|
||||
register_swift_info('symlink', symloop_max=symloop_max)
|
||||
|
||||
def symlink_mw(app):
|
||||
return SymlinkMiddleware(app, conf, symloop_max)
|
||||
return symlink_mw
|
||||
|
|
@ -222,7 +222,7 @@ from swift.common.utils import split_path, get_valid_utf8_str, \
|
|||
register_swift_info, get_hmac, streq_const_time, quote
|
||||
|
||||
|
||||
DISALLOWED_INCOMING_HEADERS = 'x-object-manifest'
|
||||
DISALLOWED_INCOMING_HEADERS = 'x-object-manifest x-symlink-target'
|
||||
|
||||
#: Default headers to remove from incoming requests. Simply a whitespace
|
||||
#: delimited list of header names and names can optionally end with '*' to
|
||||
|
|
|
|||
|
|
@ -34,7 +34,8 @@ from swift.common.storage_policy import POLICIES
|
|||
from swift.common.exceptions import ListingIterError, SegmentError
|
||||
from swift.common.http import is_success
|
||||
from swift.common.swob import HTTPBadRequest, \
|
||||
HTTPServiceUnavailable, Range, is_chunked, multi_range_iterator
|
||||
HTTPServiceUnavailable, Range, is_chunked, multi_range_iterator, \
|
||||
HTTPPreconditionFailed
|
||||
from swift.common.utils import split_path, validate_device_partition, \
|
||||
close_if_possible, maybe_multipart_byteranges_to_document_iters, \
|
||||
multipart_byteranges_to_document_iters, parse_content_type, \
|
||||
|
|
@ -281,6 +282,31 @@ def copy_header_subset(from_r, to_r, condition):
|
|||
to_r.headers[k] = v
|
||||
|
||||
|
||||
def check_path_header(req, name, length, error_msg):
|
||||
"""
|
||||
Validate that the value of path-like header is
|
||||
well formatted. We assume the caller ensures that
|
||||
specific header is present in req.headers.
|
||||
|
||||
:param req: HTTP request object
|
||||
:param name: header name
|
||||
:param length: length of path segment check
|
||||
:param error_msg: error message for client
|
||||
:returns: A tuple with path parts according to length
|
||||
:raise: HTTPPreconditionFailed if header value
|
||||
is not well formatted.
|
||||
"""
|
||||
hdr = unquote(req.headers.get(name))
|
||||
if not hdr.startswith('/'):
|
||||
hdr = '/' + hdr
|
||||
try:
|
||||
return split_path(hdr, length, length, True)
|
||||
except ValueError:
|
||||
raise HTTPPreconditionFailed(
|
||||
request=req,
|
||||
body=error_msg)
|
||||
|
||||
|
||||
class SegmentedIterable(object):
|
||||
"""
|
||||
Iterable that returns the object contents for a large object.
|
||||
|
|
@ -645,7 +671,7 @@ def resolve_etag_is_at_header(req, metadata):
|
|||
middleware's alternate etag sysmeta (X-Object-Sysmeta-Crypto-Etag) but will
|
||||
then find the EC alternate etag (if EC policy). But if the object *is*
|
||||
encrypted then X-Object-Sysmeta-Crypto-Etag is found and used, which is
|
||||
correct because it should be preferred over X-Object-Sysmeta-Crypto-Etag.
|
||||
correct because it should be preferred over X-Object-Sysmeta-Ec-Etag.
|
||||
|
||||
:param req: a swob Request
|
||||
:param metadata: a dict containing object metadata
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@
|
|||
|
||||
from __future__ import print_function
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import bisect
|
||||
import errno
|
||||
|
|
@ -29,6 +30,7 @@ import operator
|
|||
import os
|
||||
import pwd
|
||||
import re
|
||||
import string
|
||||
import struct
|
||||
import sys
|
||||
import time
|
||||
|
|
@ -4591,6 +4593,36 @@ def safe_json_loads(value):
|
|||
return None
|
||||
|
||||
|
||||
def strict_b64decode(value, allow_line_breaks=False):
|
||||
'''
|
||||
Validate and decode Base64-encoded data.
|
||||
|
||||
The stdlib base64 module silently discards bad characters, but we often
|
||||
want to treat them as an error.
|
||||
|
||||
:param value: some base64-encoded data
|
||||
:param allow_line_breaks: if True, ignore carriage returns and newlines
|
||||
:returns: the decoded data
|
||||
:raises ValueError: if ``value`` is not a string, contains invalid
|
||||
characters, or has insufficient padding
|
||||
'''
|
||||
if not isinstance(value, six.string_types):
|
||||
raise ValueError
|
||||
# b64decode will silently discard bad characters, but we want to
|
||||
# treat them as an error
|
||||
valid_chars = string.digits + string.ascii_letters + '/+'
|
||||
strip_chars = '='
|
||||
if allow_line_breaks:
|
||||
valid_chars += '\r\n'
|
||||
strip_chars += '\r\n'
|
||||
if any(c not in valid_chars for c in value.strip(strip_chars)):
|
||||
raise ValueError
|
||||
try:
|
||||
return base64.b64decode(value)
|
||||
except (TypeError, binascii.Error): # (py2 error, py3 error)
|
||||
raise ValueError
|
||||
|
||||
|
||||
MD5_BLOCK_READ_BYTES = 4096
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -397,6 +397,24 @@ def loadapp(conf_file, global_conf=None, allow_modify_pipeline=True):
|
|||
return ctx.create()
|
||||
|
||||
|
||||
def load_app_config(conf_file):
|
||||
"""
|
||||
Read the app config section from a config file.
|
||||
|
||||
:param conf_file: path to a config file
|
||||
:return: a dict
|
||||
"""
|
||||
app_conf = {}
|
||||
try:
|
||||
ctx = loadcontext(loadwsgi.APP, conf_file)
|
||||
except LookupError:
|
||||
pass
|
||||
else:
|
||||
app_conf.update(ctx.app_context.global_conf)
|
||||
app_conf.update(ctx.app_context.local_conf)
|
||||
return app_conf
|
||||
|
||||
|
||||
def run_server(conf, logger, sock, global_conf=None):
|
||||
# Ensure TZ environment variable exists to avoid stat('/etc/localtime') on
|
||||
# some platforms. This locks in reported times to UTC.
|
||||
|
|
|
|||
|
|
@ -73,13 +73,17 @@ ic_conf_body = """
|
|||
# log_statsd_metric_prefix =
|
||||
|
||||
[pipeline:main]
|
||||
pipeline = catch_errors proxy-logging cache proxy-server
|
||||
pipeline = catch_errors proxy-logging cache symlink proxy-server
|
||||
|
||||
[app:proxy-server]
|
||||
use = egg:swift#proxy
|
||||
account_autocreate = true
|
||||
# See proxy-server.conf-sample for options
|
||||
|
||||
[filter:symlink]
|
||||
use = egg:swift#symlink
|
||||
# See proxy-server.conf-sample for options
|
||||
|
||||
[filter:cache]
|
||||
use = egg:swift#memcache
|
||||
# See proxy-server.conf-sample for options
|
||||
|
|
@ -568,7 +572,10 @@ class ContainerSync(Daemon):
|
|||
realm_key, ts_meta):
|
||||
return True
|
||||
exc = None
|
||||
# look up for the newest one
|
||||
# look up for the newest one; the symlink=get query-string has
|
||||
# no effect unless symlinks are enabled in the internal client
|
||||
# in which case it ensures that symlink objects retain their
|
||||
# symlink property when sync'd.
|
||||
headers_out = {'X-Newest': True,
|
||||
'X-Backend-Storage-Policy-Index':
|
||||
str(info['storage_policy_index'])}
|
||||
|
|
@ -577,7 +584,8 @@ class ContainerSync(Daemon):
|
|||
self.swift.get_object(info['account'],
|
||||
info['container'], row['name'],
|
||||
headers=headers_out,
|
||||
acceptable_statuses=(2, 4))
|
||||
acceptable_statuses=(2, 4),
|
||||
params={'symlink': 'get'})
|
||||
|
||||
except (Exception, UnexpectedResponse, Timeout) as err:
|
||||
headers = {}
|
||||
|
|
|
|||
|
|
@ -2468,7 +2468,12 @@ class BaseDiskFile(object):
|
|||
:raises DiskFileError: various exceptions from
|
||||
:func:`swift.obj.diskfile.DiskFile._verify_data_file`
|
||||
"""
|
||||
fp = open(data_file, 'rb')
|
||||
try:
|
||||
fp = open(data_file, 'rb')
|
||||
except IOError as e:
|
||||
if e.errno == errno.ENOENT:
|
||||
raise DiskFileNotExist()
|
||||
raise
|
||||
self._datafile_metadata = self._failsafe_read_metadata(
|
||||
fp, data_file,
|
||||
add_missing_checksum=modernize)
|
||||
|
|
|
|||
|
|
@ -794,7 +794,8 @@ class File(Base):
|
|||
['content_type', 'content-type'],
|
||||
['last_modified', 'last-modified'],
|
||||
['etag', 'etag']]
|
||||
optional_fields = [['x_object_manifest', 'x-object-manifest']]
|
||||
optional_fields = [['x_object_manifest', 'x-object-manifest'],
|
||||
['x_symlink_target', 'x-symlink-target']]
|
||||
|
||||
header_fields = self.header_fields(fields,
|
||||
optional_fields=optional_fields)
|
||||
|
|
|
|||
|
|
@ -18,14 +18,13 @@
|
|||
import unittest2
|
||||
import json
|
||||
from uuid import uuid4
|
||||
from unittest2 import SkipTest
|
||||
from string import ascii_letters
|
||||
|
||||
from six.moves import range
|
||||
from swift.common.middleware.acl import format_acl
|
||||
|
||||
from test.functional import check_response, retry, requires_acls, \
|
||||
load_constraint
|
||||
load_constraint, SkipTest
|
||||
import test.functional as tf
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -17,11 +17,10 @@
|
|||
|
||||
import json
|
||||
import unittest2
|
||||
from unittest2 import SkipTest
|
||||
from uuid import uuid4
|
||||
|
||||
from test.functional import check_response, cluster_info, retry, \
|
||||
requires_acls, load_constraint, requires_policies
|
||||
requires_acls, load_constraint, requires_policies, SkipTest
|
||||
import test.functional as tf
|
||||
|
||||
from six.moves import range
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@
|
|||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import unittest
|
||||
import test.functional as tf
|
||||
from test.functional.tests import Utils, Base, Base2, BaseEnv
|
||||
from test.functional.swift_test_client import Connection, ResponseError
|
||||
|
|
@ -31,11 +32,6 @@ class TestDloEnv(BaseEnv):
|
|||
@classmethod
|
||||
def setUp(cls):
|
||||
super(TestDloEnv, cls).setUp()
|
||||
config2 = tf.config.copy()
|
||||
config2['username'] = tf.config['username3']
|
||||
config2['password'] = tf.config['password3']
|
||||
cls.conn2 = Connection(config2)
|
||||
cls.conn2.authenticate()
|
||||
|
||||
cls.container = cls.account.container(Utils.create_name())
|
||||
cls.container2 = cls.account.container(Utils.create_name())
|
||||
|
|
@ -243,9 +239,15 @@ class TestDlo(Base):
|
|||
manifest.info(hdrs={'If-None-Match': "not-%s" % etag})
|
||||
self.assert_status(200)
|
||||
|
||||
@unittest.skipIf('username3' not in tf.config, "Requires user 3")
|
||||
def test_dlo_referer_on_segment_container(self):
|
||||
# First the account2 (test3) should fail
|
||||
headers = {'X-Auth-Token': self.env.conn2.storage_token,
|
||||
config2 = tf.config.copy()
|
||||
config2['username'] = tf.config['username3']
|
||||
config2['password'] = tf.config['password3']
|
||||
conn2 = Connection(config2)
|
||||
conn2.authenticate()
|
||||
headers = {'X-Auth-Token': conn2.storage_token,
|
||||
'Referer': 'http://blah.example.com'}
|
||||
dlo_file = self.env.container.file("mancont2")
|
||||
self.assertRaises(ResponseError, dlo_file.read,
|
||||
|
|
|
|||
|
|
@ -18,14 +18,13 @@
|
|||
import datetime
|
||||
import json
|
||||
import unittest2
|
||||
from unittest2 import SkipTest
|
||||
from uuid import uuid4
|
||||
import time
|
||||
|
||||
from six.moves import range
|
||||
|
||||
from test.functional import check_response, retry, requires_acls, \
|
||||
requires_policies
|
||||
requires_policies, SkipTest
|
||||
import test.functional as tf
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -19,10 +19,9 @@ import hashlib
|
|||
import itertools
|
||||
import json
|
||||
from copy import deepcopy
|
||||
from unittest2 import SkipTest
|
||||
|
||||
import test.functional as tf
|
||||
from test.functional import cluster_info
|
||||
from test.functional import cluster_info, SkipTest
|
||||
from test.functional.tests import Utils, Base, Base2, BaseEnv
|
||||
from test.functional.swift_test_client import Connection, ResponseError
|
||||
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -19,14 +19,13 @@ import hashlib
|
|||
import json
|
||||
from copy import deepcopy
|
||||
from six.moves import urllib
|
||||
from unittest2 import SkipTest
|
||||
from time import time, strftime, gmtime
|
||||
|
||||
import test.functional as tf
|
||||
from swift.common.middleware import tempurl
|
||||
from test.functional import cluster_info
|
||||
from test.functional.tests import Utils, Base, Base2, BaseEnv
|
||||
from test.functional import requires_acls
|
||||
from test.functional import requires_acls, SkipTest
|
||||
from test.functional.swift_test_client import Account, Connection, \
|
||||
ResponseError
|
||||
|
||||
|
|
|
|||
|
|
@ -17,12 +17,13 @@
|
|||
import json
|
||||
import time
|
||||
import unittest2
|
||||
from unittest2 import SkipTest
|
||||
|
||||
import test.functional as tf
|
||||
from copy import deepcopy
|
||||
|
||||
from swift.common.utils import MD5_OF_EMPTY_STRING
|
||||
from test.functional.tests import Base, Base2, BaseEnv, Utils
|
||||
from test.functional import cluster_info
|
||||
from test.functional import cluster_info, SkipTest
|
||||
from test.functional.swift_test_client import Account, Connection, \
|
||||
ResponseError
|
||||
|
||||
|
|
@ -207,8 +208,13 @@ class TestObjectVersioning(Base):
|
|||
try:
|
||||
# only delete files and not containers
|
||||
# as they were configured in self.env
|
||||
# get rid of any versions so they aren't restored
|
||||
self.env.versions_container.delete_files()
|
||||
# get rid of originals
|
||||
self.env.container.delete_files()
|
||||
# in history mode, deleted originals got copied to versions, so
|
||||
# clear that again
|
||||
self.env.versions_container.delete_files()
|
||||
except ResponseError:
|
||||
pass
|
||||
|
||||
|
|
@ -579,6 +585,125 @@ class TestObjectVersioning(Base):
|
|||
versioned_obj.delete()
|
||||
self.assertEqual("aaaaa", versioned_obj.read())
|
||||
|
||||
def _check_overwriting_symlink(self):
|
||||
# assertions common to x-versions-location and x-history-location modes
|
||||
container = self.env.container
|
||||
versions_container = self.env.versions_container
|
||||
|
||||
tgt_a_name = Utils.create_name()
|
||||
tgt_b_name = Utils.create_name()
|
||||
|
||||
tgt_a = container.file(tgt_a_name)
|
||||
tgt_a.write("aaaaa")
|
||||
|
||||
tgt_b = container.file(tgt_b_name)
|
||||
tgt_b.write("bbbbb")
|
||||
|
||||
symlink_name = Utils.create_name()
|
||||
sym_tgt_header = '%s/%s' % (container.name, tgt_a_name)
|
||||
sym_headers_a = {'X-Symlink-Target': sym_tgt_header}
|
||||
symlink = container.file(symlink_name)
|
||||
symlink.write("", hdrs=sym_headers_a)
|
||||
self.assertEqual("aaaaa", symlink.read())
|
||||
|
||||
sym_headers_b = {'X-Symlink-Target': '%s/%s' % (container.name,
|
||||
tgt_b_name)}
|
||||
symlink.write("", hdrs=sym_headers_b)
|
||||
self.assertEqual("bbbbb", symlink.read())
|
||||
|
||||
# the old version got saved off
|
||||
self.assertEqual(1, versions_container.info()['object_count'])
|
||||
versioned_obj_name = versions_container.files()[0]
|
||||
prev_version = versions_container.file(versioned_obj_name)
|
||||
prev_version_info = prev_version.info(parms={'symlink': 'get'})
|
||||
self.assertEqual("aaaaa", prev_version.read())
|
||||
self.assertEqual(MD5_OF_EMPTY_STRING, prev_version_info['etag'])
|
||||
self.assertEqual(sym_tgt_header,
|
||||
prev_version_info['x_symlink_target'])
|
||||
return symlink, tgt_a
|
||||
|
||||
def test_overwriting_symlink(self):
|
||||
if 'symlink' not in cluster_info:
|
||||
raise SkipTest("Symlinks not enabled")
|
||||
|
||||
symlink, target = self._check_overwriting_symlink()
|
||||
# test delete
|
||||
symlink.delete()
|
||||
sym_info = symlink.info(parms={'symlink': 'get'})
|
||||
self.assertEqual("aaaaa", symlink.read())
|
||||
self.assertEqual(MD5_OF_EMPTY_STRING, sym_info['etag'])
|
||||
self.assertEqual('%s/%s' % (self.env.container.name, target.name),
|
||||
sym_info['x_symlink_target'])
|
||||
|
||||
def _setup_symlink(self):
|
||||
target = self.env.container.file('target-object')
|
||||
target.write('target object data')
|
||||
symlink = self.env.container.file('symlink')
|
||||
symlink.write('', hdrs={
|
||||
'Content-Type': 'application/symlink',
|
||||
'X-Symlink-Target': '%s/%s' % (
|
||||
self.env.container.name, target.name)})
|
||||
return symlink, target
|
||||
|
||||
def _assert_symlink(self, symlink, target):
|
||||
self.assertEqual('target object data', symlink.read())
|
||||
self.assertEqual(target.info(), symlink.info())
|
||||
self.assertEqual('application/symlink',
|
||||
symlink.info(parms={
|
||||
'symlink': 'get'})['content_type'])
|
||||
|
||||
def _check_copy_destination_restore_symlink(self):
|
||||
# assertions common to x-versions-location and x-history-location modes
|
||||
symlink, target = self._setup_symlink()
|
||||
symlink.write('this is not a symlink')
|
||||
# the symlink is versioned
|
||||
version_container_files = self.env.versions_container.files(
|
||||
parms={'format': 'json'})
|
||||
self.assertEqual(1, len(version_container_files))
|
||||
versioned_obj_info = version_container_files[0]
|
||||
self.assertEqual('application/symlink',
|
||||
versioned_obj_info['content_type'])
|
||||
versioned_obj = self.env.versions_container.file(
|
||||
versioned_obj_info['name'])
|
||||
# the symlink is still a symlink
|
||||
self._assert_symlink(versioned_obj, target)
|
||||
# test manual restore (this creates a new backup of the overwrite)
|
||||
versioned_obj.copy(self.env.container.name, symlink.name,
|
||||
parms={'symlink': 'get'})
|
||||
self._assert_symlink(symlink, target)
|
||||
# symlink overwritten by write then copy -> 2 versions
|
||||
self.assertEqual(2, self.env.versions_container.info()['object_count'])
|
||||
return symlink, target
|
||||
|
||||
def test_copy_destination_restore_symlink(self):
|
||||
if 'symlink' not in cluster_info:
|
||||
raise SkipTest("Symlinks not enabled")
|
||||
|
||||
symlink, target = self._check_copy_destination_restore_symlink()
|
||||
# and versioned writes restore
|
||||
symlink.delete()
|
||||
self.assertEqual(1, self.env.versions_container.info()['object_count'])
|
||||
self.assertEqual('this is not a symlink', symlink.read())
|
||||
symlink.delete()
|
||||
self.assertEqual(0, self.env.versions_container.info()['object_count'])
|
||||
self._assert_symlink(symlink, target)
|
||||
|
||||
def test_put_x_copy_from_restore_symlink(self):
|
||||
if 'symlink' not in cluster_info:
|
||||
raise SkipTest("Symlinks not enabled")
|
||||
|
||||
symlink, target = self._setup_symlink()
|
||||
symlink.write('this is not a symlink')
|
||||
version_container_files = self.env.versions_container.files()
|
||||
self.assertEqual(1, len(version_container_files))
|
||||
versioned_obj = self.env.versions_container.file(
|
||||
version_container_files[0])
|
||||
symlink.write(parms={'symlink': 'get'}, cfg={
|
||||
'no_content_type': True}, hdrs={
|
||||
'X-Copy-From': '%s/%s' % (
|
||||
self.env.versions_container, versioned_obj.name)})
|
||||
self._assert_symlink(symlink, target)
|
||||
|
||||
|
||||
class TestObjectVersioningUTF8(Base2, TestObjectVersioning):
|
||||
|
||||
|
|
@ -692,6 +817,29 @@ class TestObjectVersioningHistoryMode(TestObjectVersioning):
|
|||
prev_version = self.env.versions_container.file(actual)
|
||||
self.assertEqual(expected, prev_version.read())
|
||||
|
||||
def test_overwriting_symlink(self):
|
||||
if 'symlink' not in cluster_info:
|
||||
raise SkipTest("Symlinks not enabled")
|
||||
|
||||
symlink, target = self._check_overwriting_symlink()
|
||||
# test delete
|
||||
symlink.delete()
|
||||
with self.assertRaises(ResponseError) as cm:
|
||||
symlink.read()
|
||||
self.assertEqual(404, cm.exception.status)
|
||||
|
||||
def test_copy_destination_restore_symlink(self):
|
||||
if 'symlink' not in cluster_info:
|
||||
raise SkipTest("Symlinks not enabled")
|
||||
|
||||
symlink, target = self._check_copy_destination_restore_symlink()
|
||||
symlink.delete()
|
||||
with self.assertRaises(ResponseError) as cm:
|
||||
symlink.read()
|
||||
self.assertEqual(404, cm.exception.status)
|
||||
# 2 versions plus delete marker and deleted version
|
||||
self.assertEqual(4, self.env.versions_container.info()['object_count'])
|
||||
|
||||
|
||||
class TestSloWithVersioning(unittest2.TestCase):
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,6 @@ import unittest2
|
|||
import uuid
|
||||
from copy import deepcopy
|
||||
import eventlet
|
||||
from unittest2 import SkipTest
|
||||
from swift.common.http import is_success, is_client_error
|
||||
from email.utils import parsedate
|
||||
|
||||
|
|
@ -36,7 +35,7 @@ from test.functional import normalized_urls, load_constraint, cluster_info
|
|||
from test.functional import check_response, retry
|
||||
import test.functional as tf
|
||||
from test.functional.swift_test_client import Account, Connection, File, \
|
||||
ResponseError
|
||||
ResponseError, SkipTest
|
||||
|
||||
|
||||
def setUpModule():
|
||||
|
|
@ -534,7 +533,7 @@ class TestContainer(Base):
|
|||
cont = self.env.account.container('a' * l)
|
||||
if l <= limit:
|
||||
self.assertTrue(cont.create())
|
||||
self.assert_status(201)
|
||||
self.assert_status((201, 202))
|
||||
else:
|
||||
self.assertFalse(cont.create())
|
||||
self.assert_status(400)
|
||||
|
|
|
|||
|
|
@ -27,11 +27,11 @@ from collections import defaultdict
|
|||
import unittest
|
||||
from hashlib import md5
|
||||
from uuid import uuid4
|
||||
|
||||
from six.moves.http_client import HTTPConnection
|
||||
import shutil
|
||||
from six.moves.http_client import HTTPConnection
|
||||
from six.moves.urllib.parse import urlparse
|
||||
|
||||
from swiftclient import get_auth, head_account
|
||||
from swiftclient import get_auth, head_account, client
|
||||
from swift.common import internal_client
|
||||
from swift.obj.diskfile import get_data_dir
|
||||
from swift.common.ring import Ring
|
||||
|
|
@ -387,6 +387,10 @@ class ProbeTest(unittest.TestCase):
|
|||
Manager(['all']).kill()
|
||||
except Exception:
|
||||
pass
|
||||
info_url = "%s://%s/info" % (urlparse(self.url).scheme,
|
||||
urlparse(self.url).netloc)
|
||||
proxy_conn = client.http_connection(info_url)
|
||||
self.cluster_info = client.get_capabilities(proxy_conn)
|
||||
|
||||
def tearDown(self):
|
||||
Manager(['all']).kill()
|
||||
|
|
|
|||
|
|
@ -18,7 +18,6 @@ import uuid
|
|||
import random
|
||||
import unittest
|
||||
|
||||
from six.moves.urllib.parse import urlparse
|
||||
from swift.common.manager import Manager
|
||||
from swift.common.internal_client import InternalClient
|
||||
from swift.common import utils, direct_client
|
||||
|
|
@ -240,12 +239,7 @@ class TestContainerMergePolicyIndex(ReplProbeTest):
|
|||
orig_policy_index, node))
|
||||
|
||||
def test_reconcile_manifest(self):
|
||||
info_url = "%s://%s/info" % (urlparse(self.url).scheme,
|
||||
urlparse(self.url).netloc)
|
||||
proxy_conn = client.http_connection(info_url)
|
||||
cluster_info = client.get_capabilities(proxy_conn)
|
||||
|
||||
if 'slo' not in cluster_info:
|
||||
if 'slo' not in self.cluster_info:
|
||||
raise unittest.SkipTest(
|
||||
"SLO not enabled in proxy; can't test manifest reconciliation")
|
||||
# this test is not only testing a split brain scenario on
|
||||
|
|
@ -367,6 +361,80 @@ class TestContainerMergePolicyIndex(ReplProbeTest):
|
|||
self.assertEqual(int(metadata['content-length']),
|
||||
sum(part['size_bytes'] for part in manifest_data))
|
||||
|
||||
def test_reconcile_symlink(self):
|
||||
if 'symlink' not in self.cluster_info:
|
||||
raise unittest.SkipTest(
|
||||
"Symlink not enabled in proxy; can't test "
|
||||
"symlink reconciliation")
|
||||
wrong_policy = random.choice(ENABLED_POLICIES)
|
||||
policy = random.choice([p for p in ENABLED_POLICIES
|
||||
if p is not wrong_policy])
|
||||
# get an old container stashed
|
||||
self.brain.stop_primary_half()
|
||||
self.brain.put_container(int(policy))
|
||||
self.brain.start_primary_half()
|
||||
# write some target data
|
||||
client.put_object(self.url, self.token, self.container_name, 'target',
|
||||
contents='this is the target data')
|
||||
|
||||
# write the symlink
|
||||
self.brain.stop_handoff_half()
|
||||
self.brain.put_container(int(wrong_policy))
|
||||
client.put_object(
|
||||
self.url, self.token, self.container_name, 'symlink',
|
||||
headers={
|
||||
'X-Symlink-Target': '%s/target' % self.container_name,
|
||||
'Content-Type': 'application/symlink',
|
||||
})
|
||||
|
||||
# at this point we have a broken symlink (the container_info has the
|
||||
# proxy looking for the target in the wrong policy)
|
||||
with self.assertRaises(ClientException) as ctx:
|
||||
client.get_object(self.url, self.token, self.container_name,
|
||||
'symlink')
|
||||
self.assertEqual(ctx.exception.http_status, 404)
|
||||
|
||||
# of course the symlink itself is fine
|
||||
metadata, body = client.get_object(self.url, self.token,
|
||||
self.container_name, 'symlink',
|
||||
query_string='symlink=get')
|
||||
self.assertEqual(metadata['x-symlink-target'],
|
||||
'%s/target' % self.container_name)
|
||||
self.assertEqual(metadata['content-type'], 'application/symlink')
|
||||
self.assertEqual(body, '')
|
||||
# ... although in the wrong policy
|
||||
object_ring = POLICIES.get_object_ring(int(wrong_policy), '/etc/swift')
|
||||
part, nodes = object_ring.get_nodes(
|
||||
self.account, self.container_name, 'symlink')
|
||||
for node in nodes:
|
||||
metadata = direct_client.direct_head_object(
|
||||
node, part, self.account, self.container_name, 'symlink',
|
||||
headers={'X-Backend-Storage-Policy-Index': int(wrong_policy)})
|
||||
self.assertEqual(metadata['X-Object-Sysmeta-Symlink-Target'],
|
||||
'%s/target' % self.container_name)
|
||||
|
||||
# let the reconciler run
|
||||
self.brain.start_handoff_half()
|
||||
self.get_to_final_state()
|
||||
Manager(['container-reconciler']).once()
|
||||
# clear proxy cache
|
||||
client.post_container(self.url, self.token, self.container_name, {})
|
||||
|
||||
# now the symlink works
|
||||
metadata, body = client.get_object(self.url, self.token,
|
||||
self.container_name, 'symlink')
|
||||
self.assertEqual(body, 'this is the target data')
|
||||
# and it's in the correct policy
|
||||
object_ring = POLICIES.get_object_ring(int(policy), '/etc/swift')
|
||||
part, nodes = object_ring.get_nodes(
|
||||
self.account, self.container_name, 'symlink')
|
||||
for node in nodes:
|
||||
metadata = direct_client.direct_head_object(
|
||||
node, part, self.account, self.container_name, 'symlink',
|
||||
headers={'X-Backend-Storage-Policy-Index': int(policy)})
|
||||
self.assertEqual(metadata['X-Object-Sysmeta-Symlink-Target'],
|
||||
'%s/target' % self.container_name)
|
||||
|
||||
def test_reconciler_move_object_twice(self):
|
||||
# select some policies
|
||||
old_policy = random.choice(ENABLED_POLICIES)
|
||||
|
|
|
|||
|
|
@ -25,14 +25,17 @@ from test.probe.brain import BrainSplitter
|
|||
from test.probe.common import ReplProbeTest, ENABLED_POLICIES
|
||||
|
||||
|
||||
def get_current_realm_cluster(url):
|
||||
def get_info(url):
|
||||
parts = urlparse(url)
|
||||
url = parts.scheme + '://' + parts.netloc + '/info'
|
||||
http_conn = client.http_connection(url)
|
||||
try:
|
||||
info = client.get_capabilities(http_conn)
|
||||
return client.get_capabilities(http_conn)
|
||||
except client.ClientException:
|
||||
raise unittest.SkipTest('Unable to retrieve cluster info')
|
||||
|
||||
|
||||
def get_current_realm_cluster(info):
|
||||
try:
|
||||
realms = info['container_sync']['realms']
|
||||
except KeyError:
|
||||
|
|
@ -44,11 +47,12 @@ def get_current_realm_cluster(url):
|
|||
raise unittest.SkipTest('Unable find current realm cluster')
|
||||
|
||||
|
||||
class TestContainerSync(ReplProbeTest):
|
||||
class BaseTestContainerSync(ReplProbeTest):
|
||||
|
||||
def setUp(self):
|
||||
super(TestContainerSync, self).setUp()
|
||||
self.realm, self.cluster = get_current_realm_cluster(self.url)
|
||||
super(BaseTestContainerSync, self).setUp()
|
||||
self.info = get_info(self.url)
|
||||
self.realm, self.cluster = get_current_realm_cluster(self.info)
|
||||
|
||||
def _setup_synced_containers(
|
||||
self, source_overrides=None, dest_overrides=None):
|
||||
|
|
@ -92,6 +96,9 @@ class TestContainerSync(ReplProbeTest):
|
|||
|
||||
return source['name'], dest['name']
|
||||
|
||||
|
||||
class TestContainerSync(BaseTestContainerSync):
|
||||
|
||||
def test_sync(self):
|
||||
source_container, dest_container = self._setup_synced_containers()
|
||||
|
||||
|
|
@ -377,5 +384,184 @@ class TestContainerSync(ReplProbeTest):
|
|||
self.assertEqual(body, 'new-test-body')
|
||||
|
||||
|
||||
class TestContainerSyncAndSymlink(BaseTestContainerSync):
|
||||
|
||||
def setUp(self):
|
||||
super(TestContainerSyncAndSymlink, self).setUp()
|
||||
symlinks_enabled = self.info.get('symlink') or False
|
||||
if not symlinks_enabled:
|
||||
raise unittest.SkipTest("Symlinks not enabled")
|
||||
|
||||
def test_sync_symlink(self):
|
||||
# Verify that symlinks are sync'd as symlinks.
|
||||
dest_account = self.account_2
|
||||
source_container, dest_container = self._setup_synced_containers(
|
||||
dest_overrides=dest_account
|
||||
)
|
||||
|
||||
# Create source and dest containers for target objects in separate
|
||||
# accounts.
|
||||
# These containers must have same name for the destination symlink
|
||||
# to use the same target object. Initially the destination has no sync
|
||||
# key so target will not sync.
|
||||
tgt_container = 'targets-%s' % uuid.uuid4()
|
||||
dest_tgt_info = dict(dest_account)
|
||||
dest_tgt_info.update({'name': tgt_container, 'sync_key': None})
|
||||
self._setup_synced_containers(
|
||||
source_overrides={'name': tgt_container, 'sync_key': 'tgt_key'},
|
||||
dest_overrides=dest_tgt_info)
|
||||
|
||||
# upload a target to source
|
||||
target_name = 'target-%s' % uuid.uuid4()
|
||||
target_body = 'target body'
|
||||
client.put_object(
|
||||
self.url, self.token, tgt_container, target_name,
|
||||
target_body)
|
||||
|
||||
# Note that this tests when the target object is in the same account
|
||||
target_path = '%s/%s' % (tgt_container, target_name)
|
||||
symlink_name = 'symlink-%s' % uuid.uuid4()
|
||||
put_headers = {'X-Symlink-Target': target_path}
|
||||
|
||||
# upload the symlink
|
||||
client.put_object(
|
||||
self.url, self.token, source_container, symlink_name,
|
||||
'', headers=put_headers)
|
||||
|
||||
# verify object is a symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
self.url, self.token, source_container, symlink_name,
|
||||
query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
|
||||
# verify symlink behavior
|
||||
resp_headers, actual_target_body = client.get_object(
|
||||
self.url, self.token, source_container, symlink_name)
|
||||
self.assertEqual(target_body, actual_target_body)
|
||||
|
||||
# cycle container-sync
|
||||
Manager(['container-sync']).once()
|
||||
|
||||
# verify symlink was sync'd
|
||||
resp_headers, dest_listing = client.get_container(
|
||||
dest_account['url'], dest_account['token'], dest_container)
|
||||
self.assertFalse(dest_listing[1:])
|
||||
self.assertEqual(symlink_name, dest_listing[0]['name'])
|
||||
|
||||
# verify symlink remained only a symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
dest_account['url'], dest_account['token'], dest_container,
|
||||
symlink_name, query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
|
||||
# attempt to GET the target object via symlink will fail because
|
||||
# the target wasn't sync'd
|
||||
with self.assertRaises(ClientException) as cm:
|
||||
client.get_object(dest_account['url'], dest_account['token'],
|
||||
dest_container, symlink_name)
|
||||
self.assertEqual(404, cm.exception.http_status)
|
||||
|
||||
# now set sync key on destination target container
|
||||
client.put_container(
|
||||
dest_account['url'], dest_account['token'], tgt_container,
|
||||
headers={'X-Container-Sync-Key': 'tgt_key'})
|
||||
|
||||
# cycle container-sync
|
||||
Manager(['container-sync']).once()
|
||||
|
||||
# sanity:
|
||||
resp_headers, body = client.get_object(
|
||||
dest_account['url'], dest_account['token'],
|
||||
tgt_container, target_name)
|
||||
|
||||
# sanity check - verify symlink remained only a symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
dest_account['url'], dest_account['token'], dest_container,
|
||||
symlink_name, query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
|
||||
# verify GET of target object via symlink now succeeds
|
||||
resp_headers, actual_target_body = client.get_object(
|
||||
dest_account['url'], dest_account['token'], dest_container,
|
||||
symlink_name)
|
||||
self.assertEqual(target_body, actual_target_body)
|
||||
|
||||
def test_sync_cross_acc_symlink(self):
|
||||
# Verify that cross-account symlinks are sync'd as cross-account
|
||||
# symlinks.
|
||||
source_container, dest_container = self._setup_synced_containers()
|
||||
|
||||
# Sync'd symlinks will have the same target path "/a/c/o".
|
||||
# So if we want to execute probe test with syncing targets,
|
||||
# two swift clusters will be required.
|
||||
# Therefore, for probe test in single cluster, target object is not
|
||||
# sync'd in this test.
|
||||
tgt_account = self.account_2
|
||||
tgt_container = 'targets-%s' % uuid.uuid4()
|
||||
|
||||
tgt_container_headers = {'X-Container-Read': 'test:tester'}
|
||||
if len(ENABLED_POLICIES) > 1:
|
||||
tgt_policy = random.choice(ENABLED_POLICIES)
|
||||
tgt_container_headers['X-Storage-Policy'] = tgt_policy.name
|
||||
client.put_container(tgt_account['url'], tgt_account['token'],
|
||||
tgt_container, headers=tgt_container_headers)
|
||||
|
||||
# upload a target to source
|
||||
target_name = 'target-%s' % uuid.uuid4()
|
||||
target_body = 'target body'
|
||||
client.put_object(tgt_account['url'], tgt_account['token'],
|
||||
tgt_container, target_name, target_body)
|
||||
|
||||
# Note that this tests when the target object is in a different account
|
||||
target_path = '%s/%s' % (tgt_container, target_name)
|
||||
symlink_name = 'symlink-%s' % uuid.uuid4()
|
||||
put_headers = {
|
||||
'X-Symlink-Target': target_path,
|
||||
'X-Symlink-Target-Account': tgt_account['account']}
|
||||
|
||||
# upload the symlink
|
||||
client.put_object(
|
||||
self.url, self.token, source_container, symlink_name,
|
||||
'', headers=put_headers)
|
||||
|
||||
# verify object is a cross-account symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
self.url, self.token, source_container, symlink_name,
|
||||
query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
self.assertIn('x-symlink-target-account', resp_headers)
|
||||
|
||||
# verify symlink behavior
|
||||
resp_headers, actual_target_body = client.get_object(
|
||||
self.url, self.token, source_container, symlink_name)
|
||||
self.assertEqual(target_body, actual_target_body)
|
||||
|
||||
# cycle container-sync
|
||||
Manager(['container-sync']).once()
|
||||
|
||||
# verify symlink was sync'd
|
||||
resp_headers, dest_listing = client.get_container(
|
||||
self.url, self.token, dest_container)
|
||||
self.assertFalse(dest_listing[1:])
|
||||
self.assertEqual(symlink_name, dest_listing[0]['name'])
|
||||
|
||||
# verify symlink remained only a symlink
|
||||
resp_headers, symlink_body = client.get_object(
|
||||
self.url, self.token, dest_container,
|
||||
symlink_name, query_string='symlink=get')
|
||||
self.assertEqual('', symlink_body)
|
||||
self.assertIn('x-symlink-target', resp_headers)
|
||||
self.assertIn('x-symlink-target-account', resp_headers)
|
||||
|
||||
# verify GET of target object via symlink now succeeds
|
||||
resp_headers, actual_target_body = client.get_object(
|
||||
self.url, self.token, dest_container, symlink_name)
|
||||
self.assertEqual(target_body, actual_target_body)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
|
|||
|
|
@ -1092,6 +1092,37 @@ class TestReconCommands(unittest.TestCase):
|
|||
# that is returned from the recon middleware, thus can't rely on it
|
||||
mock_print.assert_has_calls(default_calls, any_order=True)
|
||||
|
||||
@mock.patch('six.moves.builtins.print')
|
||||
def test_version_check(self, mock_print):
|
||||
version = "2.7.1.dev144"
|
||||
|
||||
def dummy_request(*args, **kwargs):
|
||||
return [
|
||||
('http://127.0.0.1:6010/recon/version',
|
||||
{'version': version},
|
||||
200,
|
||||
0,
|
||||
0),
|
||||
('http://127.0.0.1:6020/recon/version',
|
||||
{'version': version},
|
||||
200,
|
||||
0,
|
||||
0),
|
||||
]
|
||||
|
||||
cli = recon.SwiftRecon()
|
||||
cli.pool.imap = dummy_request
|
||||
|
||||
default_calls = [
|
||||
mock.call("Versions matched (%s), "
|
||||
"0 error[s] while checking hosts." % version)
|
||||
]
|
||||
|
||||
cli.version_check([('127.0.0.1', 6010), ('127.0.0.1', 6020)])
|
||||
# We need any_order=True because the order of calls depends on the dict
|
||||
# that is returned from the recon middleware, thus can't rely on it
|
||||
mock_print.assert_has_calls(default_calls, any_order=True)
|
||||
|
||||
@mock.patch('six.moves.builtins.print')
|
||||
@mock.patch('time.time')
|
||||
def test_time_check_jitter_mismatch(self, mock_now, mock_print):
|
||||
|
|
@ -1126,7 +1157,35 @@ class TestReconCommands(unittest.TestCase):
|
|||
]
|
||||
|
||||
cli.time_check([('127.0.0.1', 6010), ('127.0.0.1', 6020)], 3)
|
||||
# We need any_order=True because the order of calls depends on the dict
|
||||
# that is returned from the recon middleware, thus can't rely on it
|
||||
mock_print.assert_has_calls(default_calls, any_order=True)
|
||||
|
||||
@mock.patch('six.moves.builtins.print')
|
||||
def test_version_check_differs(self, mock_print):
|
||||
def dummy_request(*args, **kwargs):
|
||||
return [
|
||||
('http://127.0.0.1:6010/recon/version',
|
||||
{'version': "2.7.1.dev144"},
|
||||
200,
|
||||
0,
|
||||
0),
|
||||
('http://127.0.0.1:6020/recon/version',
|
||||
{'version': "2.7.1.dev145"},
|
||||
200,
|
||||
0,
|
||||
0),
|
||||
]
|
||||
|
||||
cli = recon.SwiftRecon()
|
||||
cli.pool.imap = dummy_request
|
||||
|
||||
default_calls = [
|
||||
mock.call("Versions not matched (2.7.1.dev144, 2.7.1.dev145), "
|
||||
"0 error[s] while checking hosts.")
|
||||
]
|
||||
|
||||
cli.version_check([('127.0.0.1', 6010), ('127.0.0.1', 6020)])
|
||||
# We need any_order=True because the order of calls depends on the dict
|
||||
# that is returned from the recon middleware, thus can't rely on it
|
||||
mock_print.assert_has_calls(default_calls, any_order=True)
|
||||
|
|
|
|||
|
|
@ -46,7 +46,6 @@ class RunSwiftRingBuilderMixin(object):
|
|||
|
||||
if 'exp_results' in kwargs:
|
||||
exp_results = kwargs['exp_results']
|
||||
argv = argv[:-1]
|
||||
else:
|
||||
exp_results = None
|
||||
|
||||
|
|
@ -1917,7 +1916,102 @@ class TestCommands(unittest.TestCase, RunSwiftRingBuilderMixin):
|
|||
ring.save(self.tmpfile)
|
||||
# Test No change to the device
|
||||
argv = ["", self.tmpfile, "rebalance", "3"]
|
||||
with mock.patch('swift.common.ring.RingBuilder.save') as mock_save:
|
||||
self.assertSystemExit(EXIT_WARNING, ringbuilder.main, argv)
|
||||
self.assertEqual(len(mock_save.calls), 0)
|
||||
|
||||
def test_rebalance_saves_dispersion_improvement(self):
|
||||
# We set up a situation where dispersion improves but balance
|
||||
# doesn't. We construct a ring with one zone, then add a second zone
|
||||
# concurrently with a new device in the first zone. That first
|
||||
# device won't acquire any partitions, so the ring's balance won't
|
||||
# change. However, dispersion will improve.
|
||||
|
||||
ring = RingBuilder(6, 5, 1)
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 1,
|
||||
'ip': '10.0.0.1', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sda'})
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 1,
|
||||
'ip': '10.0.0.1', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sdb'})
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 1,
|
||||
'ip': '10.0.0.1', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sdc'})
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 1,
|
||||
'ip': '10.0.0.1', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sdd'})
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 1,
|
||||
'ip': '10.0.0.1', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sde'})
|
||||
ring.rebalance()
|
||||
|
||||
# The last guy in zone 1
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 1,
|
||||
'ip': '10.0.0.1', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sdf'})
|
||||
|
||||
# Add zone 2 (same total weight as zone 1)
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 2,
|
||||
'ip': '10.0.0.2', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sda'})
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 2,
|
||||
'ip': '10.0.0.2', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sdb'})
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 2,
|
||||
'ip': '10.0.0.2', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sdc'})
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 2,
|
||||
'ip': '10.0.0.2', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sdd'})
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 2,
|
||||
'ip': '10.0.0.2', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sde'})
|
||||
ring.add_dev({
|
||||
'region': 1, 'zone': 2,
|
||||
'ip': '10.0.0.2', 'port': 20001, 'weight': 1000,
|
||||
'device': 'sdf'})
|
||||
ring.pretend_min_part_hours_passed()
|
||||
ring.save(self.tmpfile)
|
||||
del ring
|
||||
|
||||
# Rebalance once: this gets 1/5 replica into zone 2; the ring is
|
||||
# saved because devices changed.
|
||||
argv = ["", self.tmpfile, "rebalance", "5759339"]
|
||||
self.assertSystemExit(EXIT_WARNING, ringbuilder.main, argv)
|
||||
rb = RingBuilder.load(self.tmpfile)
|
||||
self.assertEqual(rb.dispersion, 100)
|
||||
self.assertEqual(rb.get_balance(), 100)
|
||||
self.run_srb('pretend_min_part_hours_passed')
|
||||
|
||||
# Rebalance again: this gets 2/5 replica into zone 2, but no devices
|
||||
# changed and the balance stays the same. The only improvement is
|
||||
# dispersion.
|
||||
|
||||
captured = {}
|
||||
|
||||
def capture_save(rb, path):
|
||||
captured['dispersion'] = rb.dispersion
|
||||
captured['balance'] = rb.get_balance()
|
||||
# The warning is benign; it's just telling the user to keep on
|
||||
# rebalancing. The important assertion is that the builder was
|
||||
# saved.
|
||||
with mock.patch('swift.common.ring.RingBuilder.save', capture_save):
|
||||
self.assertSystemExit(EXIT_WARNING, ringbuilder.main, argv)
|
||||
self.assertEqual(captured, {
|
||||
'dispersion': 0,
|
||||
'balance': 100,
|
||||
})
|
||||
|
||||
def test_rebalance_no_devices(self):
|
||||
# Test no devices
|
||||
|
|
@ -2078,6 +2172,13 @@ class TestCommands(unittest.TestCase, RunSwiftRingBuilderMixin):
|
|||
argv = ["", self.tmpfile, "write_ring"]
|
||||
self.assertSystemExit(EXIT_SUCCESS, ringbuilder.main, argv)
|
||||
|
||||
def test_write_empty_ring(self):
|
||||
ring = RingBuilder(6, 3, 1)
|
||||
ring.save(self.tmpfile)
|
||||
exp_results = {'valid_exit_codes': [2]}
|
||||
out, err = self.run_srb("write_ring", exp_results=exp_results)
|
||||
self.assertEqual('Unable to write empty ring.\n', out)
|
||||
|
||||
def test_write_builder(self):
|
||||
# Test builder file already exists
|
||||
self.create_sample_ring()
|
||||
|
|
|
|||
|
|
@ -19,6 +19,7 @@ import unittest
|
|||
|
||||
import mock
|
||||
|
||||
from swift.common.utils import MD5_OF_EMPTY_STRING
|
||||
from swift.common.header_key_dict import HeaderKeyDict
|
||||
from swift.common.middleware.crypto import decrypter
|
||||
from swift.common.middleware.crypto.crypto_utils import CRYPTO_KEY_CALLBACK, \
|
||||
|
|
@ -960,6 +961,27 @@ class TestDecrypterContainerRequests(unittest.TestCase):
|
|||
self.assertIn("Cipher must be AES_CTR_256",
|
||||
self.decrypter.logger.get_lines_for_level('error')[0])
|
||||
|
||||
def test_GET_container_json_not_encrypted_obj(self):
|
||||
pt_etag = '%s; symlink_path=/a/c/o' % MD5_OF_EMPTY_STRING
|
||||
|
||||
obj_dict = {"bytes": 0,
|
||||
"last_modified": "2015-04-14T23:33:06.439040",
|
||||
"hash": pt_etag,
|
||||
"name": "symlink",
|
||||
"content_type": 'application/symlink'}
|
||||
|
||||
listing = [obj_dict]
|
||||
fake_body = json.dumps(listing)
|
||||
|
||||
resp = self._make_cont_get_req(fake_body, 'json')
|
||||
|
||||
self.assertEqual('200 OK', resp.status)
|
||||
body = resp.body
|
||||
self.assertEqual(len(body), int(resp.headers['Content-Length']))
|
||||
body_json = json.loads(body)
|
||||
self.assertEqual(1, len(body_json))
|
||||
self.assertEqual(pt_etag, body_json[0]['hash'])
|
||||
|
||||
|
||||
class TestModuleMethods(unittest.TestCase):
|
||||
def test_purge_crypto_sysmeta_headers(self):
|
||||
|
|
|
|||
|
|
@ -1309,6 +1309,9 @@ class TestServerSideCopyConfiguration(unittest.TestCase):
|
|||
[pipeline:main]
|
||||
pipeline = catch_errors copy ye-olde-proxy-server
|
||||
|
||||
[filter:catch_errors]
|
||||
use = egg:swift#catch_errors
|
||||
|
||||
[filter:copy]
|
||||
use = egg:swift#copy
|
||||
|
||||
|
|
|
|||
|
|
@ -811,6 +811,9 @@ class TestDloConfiguration(unittest.TestCase):
|
|||
[pipeline:main]
|
||||
pipeline = catch_errors dlo ye-olde-proxy-server
|
||||
|
||||
[filter:catch_errors]
|
||||
use = egg:swift#catch_errors
|
||||
|
||||
[filter:dlo]
|
||||
use = egg:swift#dlo
|
||||
max_get_time = 3600
|
||||
|
|
@ -845,13 +848,16 @@ class TestDloConfiguration(unittest.TestCase):
|
|||
[pipeline:main]
|
||||
pipeline = catch_errors dlo ye-olde-proxy-server
|
||||
|
||||
[filter:catch_errors]
|
||||
use = egg:swift#catch_errors
|
||||
|
||||
[filter:dlo]
|
||||
use = egg:swift#dlo
|
||||
|
||||
[app:ye-olde-proxy-server]
|
||||
use = egg:swift#proxy
|
||||
rate_limit_after_segment = 13
|
||||
max_get_time = 2900
|
||||
set max_get_time = 2900
|
||||
""")
|
||||
|
||||
conffile = tempfile.NamedTemporaryFile()
|
||||
|
|
@ -878,6 +884,9 @@ class TestDloConfiguration(unittest.TestCase):
|
|||
""")
|
||||
|
||||
proxy_conf2 = dedent("""
|
||||
[filter:catch_errors]
|
||||
use = egg:swift#catch_errors
|
||||
|
||||
[filter:dlo]
|
||||
use = egg:swift#dlo
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,929 @@
|
|||
#!/usr/bin/env python
|
||||
# Copyright (c) 2016 OpenStack Foundation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
# implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import unittest
|
||||
import json
|
||||
import mock
|
||||
|
||||
from six.moves.urllib.parse import quote, parse_qs
|
||||
from swift.common import swob
|
||||
from swift.common.middleware import symlink, copy, versioned_writes, \
|
||||
listing_formats
|
||||
from swift.common.swob import Request
|
||||
from swift.common.utils import MD5_OF_EMPTY_STRING
|
||||
from test.unit.common.middleware.helpers import FakeSwift
|
||||
from test.unit.common.middleware.test_versioned_writes import FakeCache
|
||||
|
||||
|
||||
class TestSymlinkMiddlewareBase(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.app = FakeSwift()
|
||||
self.sym = symlink.filter_factory({
|
||||
'symloop_max': '2',
|
||||
})(self.app)
|
||||
self.sym.logger = self.app.logger
|
||||
|
||||
def call_app(self, req, app=None, expect_exception=False):
|
||||
if app is None:
|
||||
app = self.app
|
||||
|
||||
self.authorized = []
|
||||
|
||||
def authorize(req):
|
||||
self.authorized.append(req)
|
||||
|
||||
if 'swift.authorize' not in req.environ:
|
||||
req.environ['swift.authorize'] = authorize
|
||||
|
||||
status = [None]
|
||||
headers = [None]
|
||||
|
||||
def start_response(s, h, ei=None):
|
||||
status[0] = s
|
||||
headers[0] = h
|
||||
|
||||
body_iter = app(req.environ, start_response)
|
||||
body = ''
|
||||
caught_exc = None
|
||||
try:
|
||||
for chunk in body_iter:
|
||||
body += chunk
|
||||
except Exception as exc:
|
||||
if expect_exception:
|
||||
caught_exc = exc
|
||||
else:
|
||||
raise
|
||||
|
||||
if expect_exception:
|
||||
return status[0], headers[0], body, caught_exc
|
||||
else:
|
||||
return status[0], headers[0], body
|
||||
|
||||
def call_sym(self, req, **kwargs):
|
||||
return self.call_app(req, app=self.sym, **kwargs)
|
||||
|
||||
|
||||
class TestSymlinkMiddleware(TestSymlinkMiddlewareBase):
|
||||
def test_symlink_simple_put(self):
|
||||
self.app.register('PUT', '/v1/a/c/symlink', swob.HTTPCreated, {})
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c1/o'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
self.assertEqual(val, 'c1/o')
|
||||
self.assertNotIn('X-Object-Sysmeta-Symlink-Target-Account', hdrs)
|
||||
val = hdrs.get('X-Object-Sysmeta-Container-Update-Override-Etag')
|
||||
self.assertEqual(val, '%s; symlink_target=c1/o' % MD5_OF_EMPTY_STRING)
|
||||
|
||||
def test_symlink_put_different_account(self):
|
||||
self.app.register('PUT', '/v1/a/c/symlink', swob.HTTPCreated, {})
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c1/o',
|
||||
'X-Symlink-Target-Account': 'a1'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
self.assertEqual(val, 'c1/o')
|
||||
self.assertEqual(hdrs.get('X-Object-Sysmeta-Symlink-Target-Account'),
|
||||
'a1')
|
||||
|
||||
def test_symlink_put_leading_slash(self):
|
||||
self.app.register('PUT', '/v1/a/c/symlink', swob.HTTPCreated, {})
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': '/c1/o'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '412 Precondition Failed')
|
||||
self.assertEqual(body, "X-Symlink-Target header must be of "
|
||||
"the form <container name>/<object name>")
|
||||
|
||||
def test_symlink_put_non_zero_length(self):
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT', body='req_body',
|
||||
headers={'X-Symlink-Target': 'c1/o'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '400 Bad Request')
|
||||
self.assertEqual(body, 'Symlink requests require a zero byte body')
|
||||
|
||||
def test_symlink_put_bad_object_header(self):
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': 'o'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, "412 Precondition Failed")
|
||||
self.assertEqual(body, "X-Symlink-Target header must be of "
|
||||
"the form <container name>/<object name>")
|
||||
|
||||
def test_symlink_put_bad_account_header(self):
|
||||
req = Request.blank('/v1/a/c/symlink', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c1/o',
|
||||
'X-Symlink-Target-Account': 'a1/c1'},
|
||||
body='')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, "412 Precondition Failed")
|
||||
self.assertEqual(body, "Account name cannot contain slashes")
|
||||
|
||||
def test_get_symlink(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o'})
|
||||
req = Request.blank('/v1/a/c/symlink?symlink=get', method='GET')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
|
||||
def test_get_symlink_with_account(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
req = Request.blank('/v1/a/c/symlink?symlink=get', method='GET')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertIn(('X-Symlink-Target-Account', 'a2'), headers)
|
||||
|
||||
def test_get_symlink_not_found(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPNotFound, {})
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '404 Not Found')
|
||||
self.assertNotIn('Content-Location', dict(headers))
|
||||
|
||||
def test_get_target_object(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o', swob.HTTPOk, {}, 'resp_body')
|
||||
req_headers = {'X-Newest': 'True', 'X-Backend-Something': 'future'}
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET',
|
||||
headers=req_headers)
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(body, 'resp_body')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
calls = self.app.calls_with_headers
|
||||
req_headers['Host'] = 'localhost:80'
|
||||
self.assertEqual(req_headers, calls[0].headers)
|
||||
req_headers['User-Agent'] = 'Swift'
|
||||
self.assertEqual(req_headers, calls[1].headers)
|
||||
self.assertFalse(calls[2:])
|
||||
|
||||
def test_get_target_object_not_found(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o', swob.HTTPNotFound, {}, '')
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '404 Not Found')
|
||||
self.assertEqual(body, '')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
|
||||
def test_get_target_object_range_not_satisfiable(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o',
|
||||
swob.HTTPRequestedRangeNotSatisfiable, {}, '')
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET',
|
||||
headers={'Range': 'bytes=1-2'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '416 Requested Range Not Satisfiable')
|
||||
self.assertEqual(
|
||||
body, '<html><h1>Requested Range Not Satisfiable</h1>'
|
||||
'<p>The Range requested is not available.</p></html>')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
|
||||
def test_get_ec_symlink_range_unsatisfiable_can_redirect_to_target(self):
|
||||
self.app.register('GET', '/v1/a/c/symlink',
|
||||
swob.HTTPRequestedRangeNotSatisfiable,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o', swob.HTTPOk,
|
||||
{'Content-Range': 'bytes 1-2/10'}, 'es')
|
||||
req = Request.blank('/v1/a/c/symlink', method='GET',
|
||||
headers={'Range': 'bytes=1-2'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(body, 'es')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
self.assertIn(('Content-Range', 'bytes 1-2/10'), headers)
|
||||
|
||||
def test_get_non_symlink(self):
|
||||
# this is not symlink object
|
||||
self.app.register('GET', '/v1/a/c/obj', swob.HTTPOk, {}, 'resp_body')
|
||||
req = Request.blank('/v1/a/c/obj', method='GET')
|
||||
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(body, 'resp_body')
|
||||
|
||||
# Assert special headers for symlink are not in response
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertNotIn('Content-Location', dict(headers))
|
||||
|
||||
def test_head_symlink(self):
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
req = Request.blank('/v1/a/c/symlink?symlink=get', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
|
||||
def test_head_symlink_with_account(self):
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
req = Request.blank('/v1/a/c/symlink?symlink=get', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertIn(('X-Symlink-Target-Account', 'a2'), headers)
|
||||
self.assertIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
|
||||
def test_head_target_object(self):
|
||||
# this test is also validating that the symlink metadata is not
|
||||
# returned, but the target object metadata does return
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
self.app.register('HEAD', '/v1/a2/c1/o', swob.HTTPOk,
|
||||
{'X-Object-Meta-Color': 'Green'})
|
||||
req_headers = {'X-Newest': 'True', 'X-Backend-Something': 'future'}
|
||||
req = Request.blank('/v1/a/c/symlink', method='HEAD',
|
||||
headers=req_headers)
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertNotIn('X-Symlink-Target', dict(headers))
|
||||
self.assertNotIn('X-Symlink-Target-Account', dict(headers))
|
||||
self.assertNotIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
self.assertIn(('X-Object-Meta-Color', 'Green'), headers)
|
||||
self.assertIn(('Content-Location', '/v1/a2/c1/o'), headers)
|
||||
calls = self.app.calls_with_headers
|
||||
req_headers['Host'] = 'localhost:80'
|
||||
self.assertEqual(req_headers, calls[0].headers)
|
||||
req_headers['User-Agent'] = 'Swift'
|
||||
self.assertEqual(req_headers, calls[1].headers)
|
||||
self.assertFalse(calls[2:])
|
||||
|
||||
def test_symlink_too_deep(self):
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym1'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym1', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym2'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym2', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/o'})
|
||||
req = Request.blank('/v1/a/c/symlink', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '409 Conflict')
|
||||
|
||||
def test_symlink_change_symloopmax(self):
|
||||
# similar test to test_symlink_too_deep, but now changed the limit to 3
|
||||
self.sym = symlink.filter_factory({
|
||||
'symloop_max': '3',
|
||||
})(self.app)
|
||||
self.sym.logger = self.app.logger
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym1'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym1', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym2'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym2', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/o',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
self.app.register('HEAD', '/v1/a/c/o', swob.HTTPOk,
|
||||
{'X-Object-Meta-Color': 'Green'})
|
||||
req = Request.blank('/v1/a/c/symlink', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
|
||||
# assert that the correct metadata was returned
|
||||
self.assertNotIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
self.assertIn(('X-Object-Meta-Color', 'Green'), headers)
|
||||
|
||||
def test_sym_to_sym_to_target(self):
|
||||
# this test is also validating that the symlink metadata is not
|
||||
# returned, but the target object metadata does return
|
||||
self.app.register('HEAD', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c/sym1',
|
||||
'X-Object-Meta-Color': 'Red'})
|
||||
self.app.register('HEAD', '/v1/a/c/sym1', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Meta-Color': 'Yellow'})
|
||||
self.app.register('HEAD', '/v1/a/c1/o', swob.HTTPOk,
|
||||
{'X-Object-Meta-Color': 'Green'})
|
||||
req = Request.blank('/v1/a/c/symlink', method='HEAD')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertNotIn(('X-Symlink-Target', 'c1/o'), headers)
|
||||
self.assertNotIn(('X-Symlink-Target-Account', 'a2'), headers)
|
||||
self.assertNotIn(('X-Object-Meta-Color', 'Red'), headers)
|
||||
self.assertNotIn(('X-Object-Meta-Color', 'Yellow'), headers)
|
||||
self.assertIn(('X-Object-Meta-Color', 'Green'), headers)
|
||||
self.assertIn(('Content-Location', '/v1/a/c1/o'), headers)
|
||||
|
||||
def test_symlink_post(self):
|
||||
self.app.register('POST', '/v1/a/c/symlink', swob.HTTPAccepted,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o'})
|
||||
req = Request.blank('/v1/a/c/symlink', method='POST',
|
||||
headers={'X-Object-Meta-Color': 'Red'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '307 Temporary Redirect')
|
||||
self.assertEqual(body,
|
||||
'The requested POST was applied to a symlink. POST '
|
||||
'directly to the target to apply requested metadata.')
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
val = hdrs.get('X-Object-Meta-Color')
|
||||
self.assertEqual(val, 'Red')
|
||||
|
||||
def test_non_symlink_post(self):
|
||||
self.app.register('POST', '/v1/a/c/o', swob.HTTPAccepted, {})
|
||||
req = Request.blank('/v1/a/c/o', method='POST',
|
||||
headers={'X-Object-Meta-Color': 'Red'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '202 Accepted')
|
||||
|
||||
def test_set_symlink_POST_fail(self):
|
||||
# Setting a link with a POST request is not allowed
|
||||
req = Request.blank('/v1/a/c/o', method='POST',
|
||||
headers={'X-Symlink-Target': 'c1/regular_obj'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '400 Bad Request')
|
||||
self.assertEqual(body, "A PUT request is required to set a symlink "
|
||||
"target")
|
||||
|
||||
def test_symlink_post_but_fail_at_server(self):
|
||||
self.app.register('POST', '/v1/a/c/o', swob.HTTPNotFound, {})
|
||||
req = Request.blank('/v1/a/c/o', method='POST',
|
||||
headers={'X-Object-Meta-Color': 'Red'})
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '404 Not Found')
|
||||
|
||||
def test_check_symlink_header(self):
|
||||
def do_test(headers):
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers=headers)
|
||||
symlink._check_symlink_header(req)
|
||||
|
||||
# normal cases
|
||||
do_test({'X-Symlink-Target': 'c1/o1'})
|
||||
do_test({'X-Symlink-Target': 'c1/sub/o1'})
|
||||
do_test({'X-Symlink-Target': 'c1%2Fo1'})
|
||||
# specify account
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': 'another'})
|
||||
# URL encoded is safe
|
||||
do_test({'X-Symlink-Target': 'c1%2Fo1'})
|
||||
# URL encoded + multibytes is also safe
|
||||
do_test(
|
||||
{'X-Symlink-Target':
|
||||
u'\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'})
|
||||
target = u'\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'
|
||||
encoded_target = quote(target.encode('utf-8'), '')
|
||||
do_test({'X-Symlink-Target': encoded_target})
|
||||
|
||||
do_test(
|
||||
{'X-Symlink-Target': 'cont/obj',
|
||||
'X-Symlink-Target-Account': u'\u30b0\u30e9\u30d6\u30eb'})
|
||||
|
||||
def test_check_symlink_header_invalid_format(self):
|
||||
def do_test(headers, status, err_msg):
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers=headers)
|
||||
with self.assertRaises(swob.HTTPException) as cm:
|
||||
symlink._check_symlink_header(req)
|
||||
|
||||
self.assertEqual(cm.exception.status, status)
|
||||
self.assertEqual(cm.exception.body, err_msg)
|
||||
|
||||
do_test({'X-Symlink-Target': '/c1/o1'},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
do_test({'X-Symlink-Target': 'c1o1'},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': '/another'},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': 'an/other'},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
# url encoded case
|
||||
do_test({'X-Symlink-Target': '%2Fc1%2Fo1'},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': '%2Fanother'},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
do_test({'X-Symlink-Target': 'c1/o1',
|
||||
'X-Symlink-Target-Account': 'an%2Fother'},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
# with multi-bytes
|
||||
do_test(
|
||||
{'X-Symlink-Target':
|
||||
u'/\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
target = u'/\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'
|
||||
encoded_target = quote(target.encode('utf-8'), '')
|
||||
do_test(
|
||||
{'X-Symlink-Target': encoded_target},
|
||||
'412 Precondition Failed',
|
||||
'X-Symlink-Target header must be of the '
|
||||
'form <container name>/<object name>')
|
||||
account = u'\u30b0\u30e9\u30d6\u30eb/\u30a2\u30ba\u30ec\u30f3'
|
||||
encoded_account = quote(account.encode('utf-8'), '')
|
||||
do_test(
|
||||
{'X-Symlink-Target': 'c/o',
|
||||
'X-Symlink-Target-Account': encoded_account},
|
||||
'412 Precondition Failed',
|
||||
'Account name cannot contain slashes')
|
||||
|
||||
def test_check_symlink_header_points_to_itself(self):
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c/o'})
|
||||
with self.assertRaises(swob.HTTPException) as cm:
|
||||
symlink._check_symlink_header(req)
|
||||
self.assertEqual(cm.exception.status, '400 Bad Request')
|
||||
self.assertEqual(cm.exception.body, 'Symlink cannot target itself')
|
||||
|
||||
# Even if set account to itself, it will fail as well
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c/o',
|
||||
'X-Symlink-Target-Account': 'a'})
|
||||
with self.assertRaises(swob.HTTPException) as cm:
|
||||
symlink._check_symlink_header(req)
|
||||
self.assertEqual(cm.exception.status, '400 Bad Request')
|
||||
self.assertEqual(cm.exception.body, 'Symlink cannot target itself')
|
||||
|
||||
# sanity, the case to another account is safe
|
||||
req = Request.blank('/v1/a/c/o', method='PUT',
|
||||
headers={'X-Symlink-Target': 'c/o',
|
||||
'X-Symlink-Target-Account': 'a1'})
|
||||
symlink._check_symlink_header(req)
|
||||
|
||||
def test_symloop_max_config(self):
|
||||
self.app = FakeSwift()
|
||||
# sanity
|
||||
self.sym = symlink.filter_factory({
|
||||
'symloop_max': '1',
|
||||
})(self.app)
|
||||
self.assertEqual(self.sym.symloop_max, 1)
|
||||
# < 1 case will result in default
|
||||
self.sym = symlink.filter_factory({
|
||||
'symloop_max': '-1',
|
||||
})(self.app)
|
||||
self.assertEqual(self.sym.symloop_max, symlink.DEFAULT_SYMLOOP_MAX)
|
||||
|
||||
|
||||
class SymlinkCopyingTestCase(TestSymlinkMiddlewareBase):
|
||||
# verify interaction of copy and symlink middlewares
|
||||
|
||||
def setUp(self):
|
||||
self.app = FakeSwift()
|
||||
conf = {'symloop_max': '2'}
|
||||
self.sym = symlink.filter_factory(conf)(self.app)
|
||||
self.sym.logger = self.app.logger
|
||||
self.copy = copy.filter_factory({})(self.sym)
|
||||
|
||||
def call_copy(self, req, **kwargs):
|
||||
return self.call_app(req, app=self.copy, **kwargs)
|
||||
|
||||
def test_copy_symlink_target(self):
|
||||
req = Request.blank('/v1/a/src_cont/symlink', method='COPY',
|
||||
headers={'Destination': 'tgt_cont/tgt_obj'})
|
||||
self._test_copy_symlink_target(req)
|
||||
req = Request.blank('/v1/a/tgt_cont/tgt_obj', method='PUT',
|
||||
headers={'X-Copy-From': 'src_cont/symlink'})
|
||||
self._test_copy_symlink_target(req)
|
||||
|
||||
def _test_copy_symlink_target(self, req):
|
||||
self.app.register('GET', '/v1/a/src_cont/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('GET', '/v1/a2/c1/o', swob.HTTPOk, {}, 'resp_body')
|
||||
self.app.register('PUT', '/v1/a/tgt_cont/tgt_obj', swob.HTTPCreated,
|
||||
{}, 'resp_body')
|
||||
status, headers, body = self.call_copy(req)
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
self.assertEqual(method, 'GET')
|
||||
self.assertEqual(path, '/v1/a/src_cont/symlink')
|
||||
self.assertEqual('/src_cont/symlink', hdrs.get('X-Copy-From'))
|
||||
method, path, hdrs = self.app.calls_with_headers[1]
|
||||
self.assertEqual(method, 'GET')
|
||||
self.assertEqual(path, '/v1/a2/c1/o')
|
||||
self.assertEqual('/src_cont/symlink', hdrs.get('X-Copy-From'))
|
||||
method, path, hdrs = self.app.calls_with_headers[2]
|
||||
self.assertEqual(method, 'PUT')
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
# this is raw object copy
|
||||
self.assertEqual(val, None)
|
||||
self.assertEqual(status, '201 Created')
|
||||
|
||||
def test_copy_symlink(self):
|
||||
req = Request.blank(
|
||||
'/v1/a/src_cont/symlink?symlink=get', method='COPY',
|
||||
headers={'Destination': 'tgt_cont/tgt_obj'})
|
||||
self._test_copy_symlink(req)
|
||||
req = Request.blank(
|
||||
'/v1/a/tgt_cont/tgt_obj?symlink=get', method='PUT',
|
||||
headers={'X-Copy-From': 'src_cont/symlink'})
|
||||
self._test_copy_symlink(req)
|
||||
|
||||
def _test_copy_symlink(self, req):
|
||||
self.app.register('GET', '/v1/a/src_cont/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('PUT', '/v1/a/tgt_cont/tgt_obj', swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'c1/o',
|
||||
'X-Symlink-Target-Account': 'a2'})
|
||||
status, headers, body = self.call_copy(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
self.assertEqual(method, 'GET')
|
||||
self.assertEqual(path, '/v1/a/src_cont/symlink?symlink=get')
|
||||
self.assertEqual('/src_cont/symlink', hdrs.get('X-Copy-From'))
|
||||
method, path, hdrs = self.app.calls_with_headers[1]
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
self.assertEqual(val, 'c1/o')
|
||||
self.assertEqual(
|
||||
hdrs.get('X-Object-Sysmeta-Symlink-Target-Account'), 'a2')
|
||||
|
||||
def test_copy_symlink_new_target(self):
|
||||
req = Request.blank(
|
||||
'/v1/a/src_cont/symlink?symlink=get', method='COPY',
|
||||
headers={'Destination': 'tgt_cont/tgt_obj',
|
||||
'X-Symlink-Target': 'new_cont/new_obj',
|
||||
'X-Symlink-Target-Account': 'new_acct'})
|
||||
self._test_copy_symlink_new_target(req)
|
||||
req = Request.blank(
|
||||
'/v1/a/tgt_cont/tgt_obj?symlink=get', method='PUT',
|
||||
headers={'X-Copy-From': 'src_cont/symlink',
|
||||
'X-Symlink-Target': 'new_cont/new_obj',
|
||||
'X-Symlink-Target-Account': 'new_acct'})
|
||||
self._test_copy_symlink_new_target(req)
|
||||
|
||||
def _test_copy_symlink_new_target(self, req):
|
||||
self.app.register('GET', '/v1/a/src_cont/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('PUT', '/v1/a/tgt_cont/tgt_obj', swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'c1/o',
|
||||
'X-Symlink-Target-Account': 'a2'})
|
||||
status, headers, body = self.call_copy(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
self.assertEqual(method, 'GET')
|
||||
self.assertEqual(path, '/v1/a/src_cont/symlink?symlink=get')
|
||||
self.assertEqual('/src_cont/symlink', hdrs.get('X-Copy-From'))
|
||||
method, path, hdrs = self.app.calls_with_headers[1]
|
||||
self.assertEqual(method, 'PUT')
|
||||
self.assertEqual(path, '/v1/a/tgt_cont/tgt_obj?symlink=get')
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
self.assertEqual(val, 'new_cont/new_obj')
|
||||
self.assertEqual(hdrs.get('X-Object-Sysmeta-Symlink-Target-Account'),
|
||||
'new_acct')
|
||||
|
||||
def test_copy_symlink_with_slo_query(self):
|
||||
req = Request.blank(
|
||||
'/v1/a/src_cont/symlink?multipart-manifest=get&symlink=get',
|
||||
method='COPY', headers={'Destination': 'tgt_cont/tgt_obj'})
|
||||
self._test_copy_symlink_with_slo_query(req)
|
||||
req = Request.blank(
|
||||
'/v1/a/tgt_cont/tgt_obj?multipart-manifest=get&symlink=get',
|
||||
method='PUT', headers={'X-Copy-From': 'src_cont/symlink'})
|
||||
self._test_copy_symlink_with_slo_query(req)
|
||||
|
||||
def _test_copy_symlink_with_slo_query(self, req):
|
||||
self.app.register('GET', '/v1/a/src_cont/symlink', swob.HTTPOk,
|
||||
{'X-Object-Sysmeta-Symlink-Target': 'c1/o',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a2'})
|
||||
self.app.register('PUT', '/v1/a/tgt_cont/tgt_obj', swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'c1/o',
|
||||
'X-Symlink-Target-Account': 'a2'})
|
||||
status, headers, body = self.call_copy(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
method, path, hdrs = self.app.calls_with_headers[0]
|
||||
self.assertEqual(method, 'GET')
|
||||
path, query = path.split('?')
|
||||
query_dict = parse_qs(query)
|
||||
self.assertEqual(
|
||||
path, '/v1/a/src_cont/symlink')
|
||||
self.assertEqual(
|
||||
query_dict,
|
||||
{'multipart-manifest': ['get'], 'symlink': ['get'],
|
||||
'format': ['raw']})
|
||||
self.assertEqual('/src_cont/symlink', hdrs.get('X-Copy-From'))
|
||||
method, path, hdrs = self.app.calls_with_headers[1]
|
||||
val = hdrs.get('X-Object-Sysmeta-Symlink-Target')
|
||||
self.assertEqual(val, 'c1/o')
|
||||
self.assertEqual(
|
||||
hdrs.get('X-Object-Sysmeta-Symlink-Target-Account'), 'a2')
|
||||
|
||||
|
||||
class SymlinkVersioningTestCase(TestSymlinkMiddlewareBase):
|
||||
# verify interaction of versioned_writes and symlink middlewares
|
||||
|
||||
def setUp(self):
|
||||
self.app = FakeSwift()
|
||||
conf = {'symloop_max': '2'}
|
||||
self.sym = symlink.filter_factory(conf)(self.app)
|
||||
self.sym.logger = self.app.logger
|
||||
vw_conf = {'allow_versioned_writes': 'true'}
|
||||
self.vw = versioned_writes.filter_factory(vw_conf)(self.sym)
|
||||
|
||||
def call_vw(self, req, **kwargs):
|
||||
return self.call_app(req, app=self.vw, **kwargs)
|
||||
|
||||
def assertRequestEqual(self, req, other):
|
||||
self.assertEqual(req.method, other.method)
|
||||
self.assertEqual(req.path, other.path)
|
||||
|
||||
def test_new_symlink_version_success(self):
|
||||
self.app.register(
|
||||
'PUT', '/v1/a/c/symlink', swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'new_cont/new_tgt',
|
||||
'X-Symlink-Target-Account': 'a'}, None)
|
||||
self.app.register(
|
||||
'GET', '/v1/a/c/symlink', swob.HTTPOk,
|
||||
{'last-modified': 'Thu, 1 Jan 1970 00:00:01 GMT',
|
||||
'X-Object-Sysmeta-Symlink-Target': 'old_cont/old_tgt',
|
||||
'X-Object-Sysmeta-Symlink-Target-Account': 'a'},
|
||||
'')
|
||||
self.app.register(
|
||||
'PUT', '/v1/a/ver_cont/007symlink/0000000001.00000',
|
||||
swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'old_cont/old_tgt',
|
||||
'X-Symlink-Target-Account': 'a'}, None)
|
||||
cache = FakeCache({'sysmeta': {'versions-location': 'ver_cont'}})
|
||||
req = Request.blank(
|
||||
'/v1/a/c/symlink',
|
||||
headers={'X-Symlink-Target': 'new_cont/new_tgt'},
|
||||
environ={'REQUEST_METHOD': 'PUT', 'swift.cache': cache,
|
||||
'CONTENT_LENGTH': '0',
|
||||
'swift.trans_id': 'fake_trans_id'})
|
||||
status, headers, body = self.call_vw(req)
|
||||
self.assertEqual(status, '201 Created')
|
||||
# authorized twice now because versioned_writes now makes a check on
|
||||
# PUT
|
||||
self.assertEqual(len(self.authorized), 2)
|
||||
self.assertRequestEqual(req, self.authorized[0])
|
||||
self.assertEqual(['VW', 'VW', None], self.app.swift_sources)
|
||||
self.assertEqual({'fake_trans_id'}, set(self.app.txn_ids))
|
||||
calls = self.app.calls_with_headers
|
||||
method, path, req_headers = calls[2]
|
||||
self.assertEqual('PUT', method)
|
||||
self.assertEqual('/v1/a/c/symlink', path)
|
||||
self.assertEqual(
|
||||
'new_cont/new_tgt',
|
||||
req_headers['X-Object-Sysmeta-Symlink-Target'])
|
||||
|
||||
def test_delete_latest_version_no_marker_success(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/ver_cont?prefix=003sym/&marker=&reverse=on',
|
||||
swob.HTTPOk, {},
|
||||
'[{"hash": "y", '
|
||||
'"last_modified": "2014-11-21T14:23:02.206740", '
|
||||
'"bytes": 0, '
|
||||
'"name": "003sym/2", '
|
||||
'"content_type": "text/plain"}, '
|
||||
'{"hash": "x", '
|
||||
'"last_modified": "2014-11-21T14:14:27.409100", '
|
||||
'"bytes": 0, '
|
||||
'"name": "003sym/1", '
|
||||
'"content_type": "text/plain"}]')
|
||||
self.app.register(
|
||||
'GET', '/v1/a/ver_cont/003sym/2', swob.HTTPCreated,
|
||||
{'content-length': '0',
|
||||
'X-Object-Sysmeta-Symlink-Target': 'c/tgt'}, None)
|
||||
self.app.register(
|
||||
'PUT', '/v1/a/c/sym', swob.HTTPCreated,
|
||||
{'X-Symlink-Target': 'c/tgt', 'X-Symlink-Target-Account': 'a'},
|
||||
None)
|
||||
self.app.register(
|
||||
'DELETE', '/v1/a/ver_cont/003sym/2', swob.HTTPOk,
|
||||
{}, None)
|
||||
|
||||
cache = FakeCache({'sysmeta': {'versions-location': 'ver_cont'}})
|
||||
req = Request.blank(
|
||||
'/v1/a/c/sym',
|
||||
headers={'X-If-Delete-At': 1},
|
||||
environ={'REQUEST_METHOD': 'DELETE', 'swift.cache': cache,
|
||||
'CONTENT_LENGTH': '0', 'swift.trans_id': 'fake_trans_id'})
|
||||
status, headers, body = self.call_vw(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(len(self.authorized), 1)
|
||||
self.assertRequestEqual(req, self.authorized[0])
|
||||
self.assertEqual(4, self.app.call_count)
|
||||
self.assertEqual(['VW', 'VW', 'VW', 'VW'], self.app.swift_sources)
|
||||
self.assertEqual({'fake_trans_id'}, set(self.app.txn_ids))
|
||||
calls = self.app.calls_with_headers
|
||||
method, path, req_headers = calls[2]
|
||||
self.assertEqual('PUT', method)
|
||||
self.assertEqual('/v1/a/c/sym', path)
|
||||
self.assertEqual(
|
||||
'c/tgt',
|
||||
req_headers['X-Object-Sysmeta-Symlink-Target'])
|
||||
|
||||
|
||||
class TestSymlinkContainerContext(TestSymlinkMiddlewareBase):
|
||||
|
||||
def setUp(self):
|
||||
super(TestSymlinkContainerContext, self).setUp()
|
||||
self.context = symlink.SymlinkContainerContext(
|
||||
self.sym.app, self.sym.logger)
|
||||
|
||||
def test_extract_symlink_path_json_simple_etag(self):
|
||||
obj_dict = {"bytes": 6,
|
||||
"last_modified": "1",
|
||||
"hash": "etag",
|
||||
"name": "obj",
|
||||
"content_type": "application/octet-stream"}
|
||||
obj_dict = self.context._extract_symlink_path_json(
|
||||
obj_dict, 'v1', 'AUTH_a')
|
||||
self.assertEqual(obj_dict['hash'], 'etag')
|
||||
self.assertNotIn('symlink_path', obj_dict)
|
||||
|
||||
def test_extract_symlink_path_json_symlink_path(self):
|
||||
obj_dict = {"bytes": 6,
|
||||
"last_modified": "1",
|
||||
"hash": "etag; symlink_target=c/o",
|
||||
"name": "obj",
|
||||
"content_type": "application/octet-stream"}
|
||||
obj_dict = self.context._extract_symlink_path_json(
|
||||
obj_dict, 'v1', 'AUTH_a')
|
||||
self.assertEqual(obj_dict['hash'], 'etag')
|
||||
self.assertEqual(obj_dict['symlink_path'], '/v1/AUTH_a/c/o')
|
||||
|
||||
def test_extract_symlink_path_json_symlink_path_and_account(self):
|
||||
obj_dict = {
|
||||
"bytes": 6,
|
||||
"last_modified": "1",
|
||||
"hash": "etag; symlink_target=c/o; symlink_target_account=AUTH_a2",
|
||||
"name": "obj",
|
||||
"content_type": "application/octet-stream"}
|
||||
obj_dict = self.context._extract_symlink_path_json(
|
||||
obj_dict, 'v1', 'AUTH_a')
|
||||
self.assertEqual(obj_dict['hash'], 'etag')
|
||||
self.assertEqual(obj_dict['symlink_path'], '/v1/AUTH_a2/c/o')
|
||||
|
||||
def test_extract_symlink_path_json_extra_key(self):
|
||||
obj_dict = {"bytes": 6,
|
||||
"last_modified": "1",
|
||||
"hash": "etag; symlink_target=c/o; extra_key=value",
|
||||
"name": "obj",
|
||||
"content_type": "application/octet-stream"}
|
||||
obj_dict = self.context._extract_symlink_path_json(
|
||||
obj_dict, 'v1', 'AUTH_a')
|
||||
self.assertEqual(obj_dict['hash'], 'etag; extra_key=value')
|
||||
self.assertEqual(obj_dict['symlink_path'], '/v1/AUTH_a/c/o')
|
||||
|
||||
def test_get_container_simple(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/c',
|
||||
swob.HTTPOk, {},
|
||||
json.dumps(
|
||||
[{"hash": "etag; symlink_target=c/o;",
|
||||
"last_modified": "2014-11-21T14:23:02.206740",
|
||||
"bytes": 0,
|
||||
"name": "sym_obj",
|
||||
"content_type": "text/plain"},
|
||||
{"hash": "etag2",
|
||||
"last_modified": "2014-11-21T14:14:27.409100",
|
||||
"bytes": 32,
|
||||
"name": "normal_obj",
|
||||
"content_type": "text/plain"}]))
|
||||
req = Request.blank(path='/v1/a/c')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
obj_list = json.loads(body)
|
||||
self.assertIn('symlink_path', obj_list[0])
|
||||
self.assertIn(obj_list[0]['symlink_path'], '/v1/a/c/o')
|
||||
self.assertNotIn('symlink_path', obj_list[1])
|
||||
|
||||
def test_get_container_with_subdir(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/c?delimiter=/',
|
||||
swob.HTTPOk, {},
|
||||
json.dumps([{"subdir": "photos/"}]))
|
||||
req = Request.blank(path='/v1/a/c?delimiter=/')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, '200 OK')
|
||||
obj_list = json.loads(body)
|
||||
self.assertEqual(len(obj_list), 1)
|
||||
self.assertEqual(obj_list[0]['subdir'], 'photos/')
|
||||
|
||||
def test_get_container_error_cases(self):
|
||||
# No affect for error cases
|
||||
for error in (swob.HTTPNotFound, swob.HTTPUnauthorized,
|
||||
swob.HTTPServiceUnavailable,
|
||||
swob.HTTPInternalServerError):
|
||||
self.app.register('GET', '/v1/a/c', error, {}, '')
|
||||
req = Request.blank(path='/v1/a/c')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(status, error().status)
|
||||
|
||||
def test_no_affect_for_account_request(self):
|
||||
with mock.patch.object(self.sym, 'app') as mock_app:
|
||||
mock_app.return_value = 'ok'
|
||||
req = Request.blank(path='/v1/a')
|
||||
status, headers, body = self.call_sym(req)
|
||||
self.assertEqual(body, 'ok')
|
||||
|
||||
def test_get_container_simple_with_listing_format(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/c?format=json',
|
||||
swob.HTTPOk, {},
|
||||
json.dumps(
|
||||
[{"hash": "etag; symlink_target=c/o;",
|
||||
"last_modified": "2014-11-21T14:23:02.206740",
|
||||
"bytes": 0,
|
||||
"name": "sym_obj",
|
||||
"content_type": "text/plain"},
|
||||
{"hash": "etag2",
|
||||
"last_modified": "2014-11-21T14:14:27.409100",
|
||||
"bytes": 32,
|
||||
"name": "normal_obj",
|
||||
"content_type": "text/plain"}]))
|
||||
self.lf = listing_formats.filter_factory({})(self.sym)
|
||||
req = Request.blank(path='/v1/a/c?format=json')
|
||||
status, headers, body = self.call_app(req, app=self.lf)
|
||||
self.assertEqual(status, '200 OK')
|
||||
obj_list = json.loads(body)
|
||||
self.assertIn('symlink_path', obj_list[0])
|
||||
self.assertIn(obj_list[0]['symlink_path'], '/v1/a/c/o')
|
||||
self.assertNotIn('symlink_path', obj_list[1])
|
||||
|
||||
def test_get_container_simple_with_listing_format_xml(self):
|
||||
self.app.register(
|
||||
'GET',
|
||||
'/v1/a/c?format=json',
|
||||
swob.HTTPOk, {'Content-Type': 'application/json'},
|
||||
json.dumps(
|
||||
[{"hash": "etag; symlink_target=c/o;",
|
||||
"last_modified": "2014-11-21T14:23:02.206740",
|
||||
"bytes": 0,
|
||||
"name": "sym_obj",
|
||||
"content_type": "text/plain"},
|
||||
{"hash": "etag2",
|
||||
"last_modified": "2014-11-21T14:14:27.409100",
|
||||
"bytes": 32,
|
||||
"name": "normal_obj",
|
||||
"content_type": "text/plain"}]))
|
||||
self.lf = listing_formats.filter_factory({})(self.sym)
|
||||
req = Request.blank(path='/v1/a/c?format=xml')
|
||||
status, headers, body = self.call_app(req, app=self.lf)
|
||||
self.assertEqual(status, '200 OK')
|
||||
self.assertEqual(body.split('\n'), [
|
||||
'<?xml version="1.0" encoding="UTF-8"?>',
|
||||
'<container name="c"><object><name>sym_obj</name>'
|
||||
'<hash>etag</hash><bytes>0</bytes>'
|
||||
'<content_type>text/plain</content_type>'
|
||||
'<last_modified>2014-11-21T14:23:02.206740</last_modified>'
|
||||
'</object>'
|
||||
'<object><name>normal_obj</name><hash>etag2</hash>'
|
||||
'<bytes>32</bytes><content_type>text/plain</content_type>'
|
||||
'<last_modified>2014-11-21T14:14:27.409100</last_modified>'
|
||||
'</object></container>'])
|
||||
|
|
@ -857,18 +857,21 @@ class TestTempURL(unittest.TestCase):
|
|||
path = '/v1/a/c/o'
|
||||
key = 'abc'
|
||||
for method in ('PUT', 'POST'):
|
||||
hmac_body = '%s\n%s\n%s' % (method, expires, path)
|
||||
sig = hmac.new(key, hmac_body, sha1).hexdigest()
|
||||
req = self._make_request(
|
||||
path, method=method, keys=[key],
|
||||
headers={'x-object-manifest': 'private/secret'},
|
||||
environ={'QUERY_STRING': 'temp_url_sig=%s&temp_url_expires=%s'
|
||||
% (sig, expires)})
|
||||
resp = req.get_response(self.tempurl)
|
||||
self.assertEqual(resp.status_int, 400)
|
||||
self.assertTrue('header' in resp.body)
|
||||
self.assertTrue('not allowed' in resp.body)
|
||||
self.assertTrue('X-Object-Manifest' in resp.body)
|
||||
for hdr, value in [('X-Object-Manifest', 'private/secret'),
|
||||
('X-Symlink-Target', 'cont/symlink')]:
|
||||
hmac_body = '%s\n%s\n%s' % (method, expires, path)
|
||||
sig = hmac.new(key, hmac_body, sha1).hexdigest()
|
||||
req = self._make_request(
|
||||
path, method=method, keys=[key],
|
||||
headers={hdr: value},
|
||||
environ={'QUERY_STRING':
|
||||
'temp_url_sig=%s&temp_url_expires=%s'
|
||||
% (sig, expires)})
|
||||
resp = req.get_response(self.tempurl)
|
||||
self.assertEqual(resp.status_int, 400)
|
||||
self.assertTrue('header' in resp.body)
|
||||
self.assertTrue('not allowed' in resp.body)
|
||||
self.assertTrue(hdr in resp.body)
|
||||
|
||||
def test_removed_incoming_header(self):
|
||||
self.tempurl = tempurl.filter_factory({
|
||||
|
|
|
|||
|
|
@ -1072,6 +1072,11 @@ class TestRingBuilder(unittest.TestCase):
|
|||
new_dev_parts[dev_id] += 1
|
||||
for dev in rb._iter_devs():
|
||||
dev['parts'] = new_dev_parts[dev['id']]
|
||||
# reset the _last_part_gather_start otherwise
|
||||
# there is a chance it'll unluckly wrap and try and
|
||||
# move one of the device 1's from replica 2
|
||||
# causing the intermitant failure in bug 1724356
|
||||
rb._last_part_gather_start = 0
|
||||
|
||||
rb.pretend_min_part_hours_passed()
|
||||
rb.rebalance()
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ import os
|
|||
import six
|
||||
from six import StringIO
|
||||
from six.moves import range
|
||||
from six.moves.urllib.parse import quote
|
||||
from six.moves.urllib.parse import quote, parse_qsl
|
||||
from test.unit import FakeLogger
|
||||
from swift.common import exceptions, internal_client, swob
|
||||
from swift.common.header_key_dict import HeaderKeyDict
|
||||
|
|
@ -316,6 +316,29 @@ class TestInternalClient(unittest.TestCase):
|
|||
client = InternalClient(self)
|
||||
client.make_request('GET', '/', {}, (200,))
|
||||
|
||||
def test_make_request_sets_query_string(self):
|
||||
captured_envs = []
|
||||
|
||||
class InternalClient(internal_client.InternalClient):
|
||||
def __init__(self, test):
|
||||
self.test = test
|
||||
self.app = self.fake_app
|
||||
self.user_agent = 'some_agent'
|
||||
self.request_tries = 1
|
||||
|
||||
def fake_app(self, env, start_response):
|
||||
captured_envs.append(env)
|
||||
start_response('200 Ok', [('Content-Length', '0')])
|
||||
return []
|
||||
|
||||
client = InternalClient(self)
|
||||
params = {'param1': 'p1', 'tasty': 'soup'}
|
||||
client.make_request('GET', '/', {}, (200,), params=params)
|
||||
actual_params = dict(parse_qsl(captured_envs[0]['QUERY_STRING'],
|
||||
keep_blank_values=True,
|
||||
strict_parsing=True))
|
||||
self.assertEqual(params, actual_params)
|
||||
|
||||
def test_make_request_retries(self):
|
||||
class InternalClient(internal_client.InternalClient):
|
||||
def __init__(self, test):
|
||||
|
|
@ -1047,10 +1070,11 @@ class TestInternalClient(unittest.TestCase):
|
|||
client, app = get_client_app()
|
||||
headers = {'foo': 'bar'}
|
||||
body = 'some_object_body'
|
||||
params = {'symlink': 'get'}
|
||||
app.register('GET', path_info, swob.HTTPOk, headers, body)
|
||||
req_headers = {'x-important-header': 'some_important_value'}
|
||||
status_int, resp_headers, obj_iter = client.get_object(
|
||||
account, container, obj, req_headers)
|
||||
account, container, obj, req_headers, params=params)
|
||||
self.assertEqual(status_int // 100, 2)
|
||||
for k, v in headers.items():
|
||||
self.assertEqual(v, resp_headers[k])
|
||||
|
|
@ -1062,7 +1086,7 @@ class TestInternalClient(unittest.TestCase):
|
|||
'user-agent': 'test', # from InternalClient.make_request
|
||||
})
|
||||
self.assertEqual(app.calls_with_headers, [(
|
||||
'GET', path_info, HeaderKeyDict(req_headers))])
|
||||
'GET', path_info + '?symlink=get', HeaderKeyDict(req_headers))])
|
||||
|
||||
def test_iter_object_lines(self):
|
||||
class InternalClient(internal_client.InternalClient):
|
||||
|
|
|
|||
|
|
@ -41,6 +41,7 @@ import string
|
|||
import sys
|
||||
import json
|
||||
import math
|
||||
import inspect
|
||||
|
||||
import six
|
||||
from six import BytesIO, StringIO
|
||||
|
|
@ -3976,6 +3977,54 @@ cluster_dfw1 = http://dfw1.host/v1/
|
|||
self.fail('Invalid results from pure function:\n%s' %
|
||||
'\n'.join(failures))
|
||||
|
||||
def test_strict_b64decode(self):
|
||||
expectations = {
|
||||
None: ValueError,
|
||||
0: ValueError,
|
||||
b'': b'',
|
||||
u'': b'',
|
||||
b'A': ValueError,
|
||||
b'AA': ValueError,
|
||||
b'AAA': ValueError,
|
||||
b'AAAA': b'\x00\x00\x00',
|
||||
u'AAAA': b'\x00\x00\x00',
|
||||
b'////': b'\xff\xff\xff',
|
||||
u'////': b'\xff\xff\xff',
|
||||
b'A===': ValueError,
|
||||
b'AA==': b'\x00',
|
||||
b'AAA=': b'\x00\x00',
|
||||
b' AAAA': ValueError,
|
||||
b'AAAA ': ValueError,
|
||||
b'AAAA============': b'\x00\x00\x00',
|
||||
b'AA&AA==': ValueError,
|
||||
b'====': b'',
|
||||
}
|
||||
|
||||
failures = []
|
||||
for value, expected in expectations.items():
|
||||
try:
|
||||
result = utils.strict_b64decode(value)
|
||||
except Exception as e:
|
||||
if inspect.isclass(expected) and issubclass(
|
||||
expected, Exception):
|
||||
if not isinstance(e, expected):
|
||||
failures.append('%r raised %r (expected to raise %r)' %
|
||||
(value, e, expected))
|
||||
else:
|
||||
failures.append('%r raised %r (expected to return %r)' %
|
||||
(value, e, expected))
|
||||
else:
|
||||
if inspect.isclass(expected) and issubclass(
|
||||
expected, Exception):
|
||||
failures.append('%r => %r (expected to raise %r)' %
|
||||
(value, result, expected))
|
||||
elif result != expected:
|
||||
failures.append('%r => %r (expected %r)' % (
|
||||
value, result, expected))
|
||||
if failures:
|
||||
self.fail('Invalid results from pure function:\n%s' %
|
||||
'\n'.join(failures))
|
||||
|
||||
def test_replace_partition_in_path(self):
|
||||
# Check for new part = part * 2
|
||||
old = '/s/n/d/o/700/c77/af088baea4806dcaba30bf07d9e64c77/f'
|
||||
|
|
|
|||
|
|
@ -198,6 +198,97 @@ class TestWSGI(unittest.TestCase):
|
|||
app = wsgi.loadapp(wsgi.ConfigString(conf_body))
|
||||
self.assertTrue(isinstance(app, obj_server.ObjectController))
|
||||
|
||||
@with_tempdir
|
||||
def test_load_app_config(self, tempdir):
|
||||
conf_file = os.path.join(tempdir, 'file.conf')
|
||||
|
||||
def _write_and_load_conf_file(conf):
|
||||
with open(conf_file, 'wb') as fd:
|
||||
fd.write(dedent(conf))
|
||||
return wsgi.load_app_config(conf_file)
|
||||
|
||||
# typical case - DEFAULT options override same option in other sections
|
||||
conf_str = """
|
||||
[DEFAULT]
|
||||
dflt_option = dflt-value
|
||||
|
||||
[pipeline:main]
|
||||
pipeline = proxy-logging proxy-server
|
||||
|
||||
[filter:proxy-logging]
|
||||
use = egg:swift#proxy_logging
|
||||
|
||||
[app:proxy-server]
|
||||
use = egg:swift#proxy
|
||||
proxy_option = proxy-value
|
||||
dflt_option = proxy-dflt-value
|
||||
"""
|
||||
|
||||
proxy_conf = _write_and_load_conf_file(conf_str)
|
||||
self.assertEqual('proxy-value', proxy_conf['proxy_option'])
|
||||
self.assertEqual('dflt-value', proxy_conf['dflt_option'])
|
||||
|
||||
# 'set' overrides DEFAULT option
|
||||
conf_str = """
|
||||
[DEFAULT]
|
||||
dflt_option = dflt-value
|
||||
|
||||
[pipeline:main]
|
||||
pipeline = proxy-logging proxy-server
|
||||
|
||||
[filter:proxy-logging]
|
||||
use = egg:swift#proxy_logging
|
||||
|
||||
[app:proxy-server]
|
||||
use = egg:swift#proxy
|
||||
proxy_option = proxy-value
|
||||
set dflt_option = proxy-dflt-value
|
||||
"""
|
||||
|
||||
proxy_conf = _write_and_load_conf_file(conf_str)
|
||||
self.assertEqual('proxy-value', proxy_conf['proxy_option'])
|
||||
self.assertEqual('proxy-dflt-value', proxy_conf['dflt_option'])
|
||||
|
||||
# actual proxy server app name is dereferenced
|
||||
conf_str = """
|
||||
[pipeline:main]
|
||||
pipeline = proxy-logging proxyserverapp
|
||||
|
||||
[filter:proxy-logging]
|
||||
use = egg:swift#proxy_logging
|
||||
|
||||
[app:proxyserverapp]
|
||||
use = egg:swift#proxy
|
||||
proxy_option = proxy-value
|
||||
dflt_option = proxy-dflt-value
|
||||
"""
|
||||
proxy_conf = _write_and_load_conf_file(conf_str)
|
||||
self.assertEqual('proxy-value', proxy_conf['proxy_option'])
|
||||
self.assertEqual('proxy-dflt-value', proxy_conf['dflt_option'])
|
||||
|
||||
# no pipeline
|
||||
conf_str = """
|
||||
[filter:proxy-logging]
|
||||
use = egg:swift#proxy_logging
|
||||
|
||||
[app:proxy-server]
|
||||
use = egg:swift#proxy
|
||||
proxy_option = proxy-value
|
||||
"""
|
||||
proxy_conf = _write_and_load_conf_file(conf_str)
|
||||
self.assertEqual({}, proxy_conf)
|
||||
|
||||
# no matching section
|
||||
conf_str = """
|
||||
[pipeline:main]
|
||||
pipeline = proxy-logging proxy-server
|
||||
|
||||
[filter:proxy-logging]
|
||||
use = egg:swift#proxy_logging
|
||||
"""
|
||||
proxy_conf = _write_and_load_conf_file(conf_str)
|
||||
self.assertEqual({}, proxy_conf)
|
||||
|
||||
def test_init_request_processor_from_conf_dir(self):
|
||||
config_dir = {
|
||||
'proxy-server.conf.d/pipeline.conf': """
|
||||
|
|
|
|||
|
|
@ -968,7 +968,9 @@ class TestContainerSync(unittest.TestCase):
|
|||
logger=self.logger)
|
||||
cs.http_proxies = ['http://proxy']
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
return (200,
|
||||
|
|
@ -1004,7 +1006,9 @@ class TestContainerSync(unittest.TestCase):
|
|||
expected_put_count += 1
|
||||
self.assertEqual(cs.container_puts, expected_put_count)
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
|
|
@ -1055,7 +1059,9 @@ class TestContainerSync(unittest.TestCase):
|
|||
expected_put_count += 1
|
||||
self.assertEqual(cs.container_puts, expected_put_count)
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
|
|
@ -1090,7 +1096,9 @@ class TestContainerSync(unittest.TestCase):
|
|||
|
||||
exc = []
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
|
|
@ -1114,7 +1122,9 @@ class TestContainerSync(unittest.TestCase):
|
|||
|
||||
exc = []
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
|
|
@ -1137,7 +1147,9 @@ class TestContainerSync(unittest.TestCase):
|
|||
self.assertEqual(len(exc), 1)
|
||||
self.assertEqual(str(exc[-1]), 'test client exception')
|
||||
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses):
|
||||
def fake_get_object(acct, con, obj, headers, acceptable_statuses,
|
||||
params=None):
|
||||
self.assertEqual({'symlink': 'get'}, params)
|
||||
self.assertEqual(headers['X-Newest'], True)
|
||||
self.assertEqual(headers['X-Backend-Storage-Policy-Index'],
|
||||
'0')
|
||||
|
|
|
|||
|
|
@ -2235,11 +2235,11 @@ class TestECDiskFileManager(DiskFileManagerMixin, unittest.TestCase):
|
|||
rmtree(df._datadir, ignore_errors=True)
|
||||
|
||||
# sanity
|
||||
files = [
|
||||
good_files = [
|
||||
'0000000006.00000.meta',
|
||||
'0000000006.00000#1#d.data'
|
||||
]
|
||||
with create_files(class_under_test, files):
|
||||
with create_files(class_under_test, good_files):
|
||||
class_under_test.open()
|
||||
|
||||
scenarios = [['0000000007.00000.meta'],
|
||||
|
|
@ -2263,6 +2263,22 @@ class TestECDiskFileManager(DiskFileManagerMixin, unittest.TestCase):
|
|||
self.fail('expected DiskFileNotExist opening %s with %r' % (
|
||||
class_under_test.__class__.__name__, files))
|
||||
|
||||
# Simulate another process deleting the data after we list contents
|
||||
# but before we actually open them
|
||||
orig_listdir = os.listdir
|
||||
|
||||
def deleting_listdir(d):
|
||||
result = orig_listdir(d)
|
||||
for f in result:
|
||||
os.unlink(os.path.join(d, f))
|
||||
return result
|
||||
|
||||
with create_files(class_under_test, good_files), \
|
||||
mock.patch('swift.obj.diskfile.os.listdir',
|
||||
side_effect=deleting_listdir), \
|
||||
self.assertRaises(DiskFileNotExist):
|
||||
class_under_test.open()
|
||||
|
||||
def test_verify_ondisk_files(self):
|
||||
# _verify_ondisk_files should only return False if get_ondisk_files
|
||||
# has produced a bad set of files due to a bug, so to test it we need
|
||||
|
|
|
|||
|
|
@ -30,8 +30,7 @@ from eventlet.green import subprocess
|
|||
from eventlet import Timeout
|
||||
|
||||
from test.unit import (debug_logger, patch_policies, make_timestamp_iter,
|
||||
mocked_http_conn, FakeLogger, mock_check_drive,
|
||||
skip_if_no_xattrs)
|
||||
mocked_http_conn, mock_check_drive, skip_if_no_xattrs)
|
||||
from swift.common import utils
|
||||
from swift.common.utils import (hash_path, mkdirs, normalize_timestamp,
|
||||
storage_directory)
|
||||
|
|
@ -1558,6 +1557,7 @@ class TestObjectReplicator(unittest.TestCase):
|
|||
def raise_exception_rmdir(exception_class, error_no):
|
||||
instance = exception_class()
|
||||
instance.errno = error_no
|
||||
instance.strerror = os.strerror(error_no)
|
||||
|
||||
def func(directory):
|
||||
if directory == suffix_dir_path:
|
||||
|
|
@ -1596,7 +1596,10 @@ class TestObjectReplicator(unittest.TestCase):
|
|||
with mock.patch('os.rmdir',
|
||||
raise_exception_rmdir(OSError, ENOTDIR)):
|
||||
self.replicator.replicate()
|
||||
self.assertEqual(len(mock_logger.get_lines_for_level('error')), 1)
|
||||
self.assertEqual(mock_logger.get_lines_for_level('error'), [
|
||||
'Unexpected error trying to cleanup suffix dir:%r: ' %
|
||||
os.path.dirname(df._datadir),
|
||||
])
|
||||
self.assertFalse(os.access(whole_path_from, os.F_OK))
|
||||
self.assertTrue(os.access(suffix_dir_path, os.F_OK))
|
||||
self.assertTrue(os.access(part_path, os.F_OK))
|
||||
|
|
@ -1692,7 +1695,7 @@ class TestObjectReplicator(unittest.TestCase):
|
|||
mock_http_connect(200)):
|
||||
with mock.patch('swift.obj.replicator.dump_recon_cache'):
|
||||
replicator = object_replicator.ObjectReplicator(
|
||||
conf, logger=FakeLogger())
|
||||
conf, logger=self.logger)
|
||||
|
||||
self.get_hash_count = 0
|
||||
with mock.patch.object(replicator, 'sync', fake_sync):
|
||||
|
|
@ -1843,6 +1846,7 @@ class TestObjectReplicator(unittest.TestCase):
|
|||
# Check successful http_connection and exception with
|
||||
# incorrect pickle.loads(resp.read())
|
||||
resp.status = 200
|
||||
resp.read.return_value = 'garbage'
|
||||
expect = 'Error syncing with node: %r: '
|
||||
for job in jobs:
|
||||
set_default(self)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,30 @@
|
|||
#!/bin/bash -xe
|
||||
|
||||
# Set up a partition formatted with XFS to use as TMPDIR for our tests.
|
||||
# OpenStack CI will invoke this script as part of tox based tests.
|
||||
# The file .zuul.yaml set TMPDIR to $HOME/xfstmp.
|
||||
|
||||
# Create a large-ish file that we will mount as a loopback
|
||||
truncate -s 1GB $HOME/1G_xfs_file
|
||||
# Format the new file as XFS.
|
||||
/sbin/mkfs.xfs $HOME/1G_xfs_file
|
||||
# loopback mount the file
|
||||
mkdir -p $HOME/xfstmp
|
||||
sudo mount -o loop,noatime,nodiratime $HOME/1G_xfs_file $HOME/xfstmp
|
||||
sudo chmod 777 $HOME/xfstmp
|
||||
|
||||
# Install liberasurecode-devel for CentOS from RDO repository.
|
||||
|
||||
function is_rhel7 {
|
||||
[ -f /usr/bin/yum ] && \
|
||||
cat /etc/*release | grep -q -e "Red Hat" -e "CentOS" -e "CloudLinux" && \
|
||||
cat /etc/*release | grep -q 'release 7'
|
||||
}
|
||||
|
||||
|
||||
if is_rhel7; then
|
||||
# Install CentOS OpenStack repos so that we have access to some extra
|
||||
# packages.
|
||||
sudo yum install -y centos-release-openstack-pike
|
||||
sudo yum install -y liberasurecode-devel
|
||||
fi
|
||||
Loading…
Reference in New Issue