openstack
/
swift
Code Issues Proposed changes

Merge branch 'master' into feature/ec 

Change-Id: I0049baf1bb0dea2c338dd9f8c091c07c132046c9
This commit is contained in:
Samuel Merritt 2014-02-28 10:11:31 -08:00
parent 41794c0d2a 98ddc1e2f1
commit 2c22fb0a17
55 changed files with 1819 additions and 633 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
AUTHORS
View File

@ -19,6 +19,7 @@ Mehdi Abaakouk (mehdi.abaakouk@enovance.com)
Jesse Andrews (anotherjesse@gmail.com)
Joe Arnold (joe@swiftstack.com)
Ionuț Arțăriși (iartarisi@suse.cz)
Luis de Bethencourt (luis@debethencourt.com)
Darrell Bishop (darrell@swiftstack.com)
James E. Blair (jeblair@openstack.org)
Fabien Boucher (fabien.boucher@enovance.com)
@ -42,6 +43,7 @@ Morgan Fainberg (m@metacloud.com)
ZhiQiang Fan (aji.zqfan@gmail.com)
Flaper Fesp (flaper87@gmail.com)
Tom Fifield (tom@openstack.org)
Florent Flament (florent.flament-ext@cloudwatt.com)
Gaurav B. Gangalwar (gaurav@gluster.com)
Alex Gaynor (alex.gaynor@gmail.com)
Anne Gentle (anne@openstack.org)
@ -122,6 +124,7 @@ Christian Schwede (info@cschwede.de)
Andrew Clay Shafer (acs@parvuscaptus.com)
Chuck Short (chuck.short@canonical.com)
Michael Shuler (mshuler@gmail.com)
David Moreau Simard (dmsimard@iweb.com)
Scott Simpson (sasimpson@gmail.com)
Liu Siqi (meizu647@gmail.com)
Adrian Smith (adrian_f_smith@dell.com)
@ -142,6 +145,7 @@ Vincent Untz (vuntz@suse.com)
Daniele Valeriani (daniele@dvaleriani.net)
Koert van der Veer (koert@cloudvps.com)
Vladimir Vechkanov (vvechkanov@mirantis.com)
Shane Wang (shane.wang@intel.com)
Yaguang Wang (yaguang.wang@intel.com)
Chris Wedgwood (cw@f00f.org)
Conrad Weidenkeller (conrad.weidenkeller@rackspace.com)

52
CHANGELOG
View File

@ -1,3 +1,55 @@
swift (1.13.0)
* Account-level ACLs and ACL format v2
Accounts now have a new privileged header to represent ACLs or
any other form of account-level access control. The value of
the header is a JSON dictionary string to be interpreted by the
auth system. A reference implementation is given in TempAuth.
Please see the full docs at
http://swift.openstack.org/overview_auth.html
* Added a WSGI environment flag to stop swob from always using
absolute location. This is useful if middleware needs to use
out-of-spec Location headers in a response.
* Container sync proxies now support simple load balancing
* Config option to lower the timeout for recoverable object GETs
* Add a way to ratelimit all writes to an account
* Allow multiple storage_domain values in cname_lookup middleware
* Moved all DLO functionality into middleware
The proxy will automatically insert the dlo middleware at an
appropriate place in the pipeline the same way it does with the
gatekeeper middleware. Clusters will still support DLOs after upgrade
even with an old config file that doesn't mention dlo at all.
* Remove python-swiftclient dependency
* Add secondary groups to process user during privilege escalation
* When logging request headers, it is now possible to specify
specifically which headers should be logged
* Added log_requests config parameter to account and container servers
to match the parameter in the object server. This allows a deployer
to turn off log messages for these processes.
* Ensure swift.source is set for DLO/SLO requests
* Fixed an issue where overwriting segments in a dynamic manifest
could cause issues on pipelined requests.
* Properly handle COPY verb in container quota middleware
* Improved StaticWeb 404 error message on web-listings and index
* Various other minor bug fixes and improvements.
swift (1.12.0)
* Several important pieces of information have been added to /info:

2
bin/swift-drive-audit
View File

@ -56,7 +56,7 @@ def get_devices(device_dir, logger):
def get_errors(error_re, log_file_pattern, minutes):
# Assuming log rotation is being used, we need to examine
# recently rotated files in case the rotation occured
# recently rotated files in case the rotation occurred
# just before the script is being run - the data we are
# looking for may have rotated.
#

2
doc/manpages/object-server.conf.5
View File

@ -198,7 +198,7 @@ Max duration of a partition rsync. The default is 900 seconds.
.IP \fBrsync_io_timeout\fR
Passed to rsync for I/O OP timeout. The default is 30 seconds.
.IP \fBrsync_bwlimit\fR
Passed to rsync for bandwith limit in kB/s. The default is 0 (unlimited)
Passed to rsync for bandwidth limit in kB/s. The default is 0 (unlimited).
.IP \fBhttp_timeout\fR
Max duration of an HTTP request. The default is 60 seconds.
.IP \fBlockup_timeout\fR

28
doc/source/account.rst
View File

@ -4,16 +4,6 @@
Account
*******
.. _account-server:
Account Server
==============
.. automodule:: swift.account.server
:members:
:undoc-members:
:show-inheritance:
.. _account-auditor:
Account Auditor
@ -24,6 +14,16 @@ Account Auditor
:undoc-members:
:show-inheritance:
.. _account-backend:
Account Backend
===============
.. automodule:: swift.account.backend
:members:
:undoc-members:
:show-inheritance:
.. _account-reaper:
Account Reaper
@ -34,12 +34,12 @@ Account Reaper
:undoc-members:
:show-inheritance:
.. _account-backend:
.. _account-server:
Account Backend
===============
Account Server
==============
.. automodule:: swift.account.backend
.. automodule:: swift.account.server
:members:
:undoc-members:
:show-inheritance:

2
doc/source/admin_guide.rst
View File

@ -892,7 +892,7 @@ Metric Name Description
`object-server.PUT.timeouts` Count of object PUTs which exceeded max_upload_time.
`object-server.PUT.timing` Timing data for each PUT request not resulting in an
error.
`object-server.PUT.<device>.timing` Timing data per kB transfered (ms/kB) for each
`object-server.PUT.<device>.timing` Timing data per kB transferred (ms/kB) for each
non-zero-byte PUT request on each device.
Monitoring problematic devices, higher is bad.
`object-server.GET.errors.timing` Timing data for GET request errors: bad request,

56
doc/source/container.rst
View File

@ -4,26 +4,6 @@
Container
*********
.. _container-server:
Container Server
================
.. automodule:: swift.container.server
:members:
:undoc-members:
:show-inheritance:
.. _container-updater:
Container Updater
=================
.. automodule:: swift.container.updater
:members:
:undoc-members:
:show-inheritance:
.. _container-auditor:
Container Auditor
@ -34,14 +14,6 @@ Container Auditor
:undoc-members:
:show-inheritance:
Container Sync
==============
.. automodule:: swift.container.sync
:members:
:undoc-members:
:show-inheritance:
.. _container-backend:
Container Backend
@ -51,3 +23,31 @@ Container Backend
:members:
:undoc-members:
:show-inheritance:
.. _container-server:
Container Server
================
.. automodule:: swift.container.server
:members:
:undoc-members:
:show-inheritance:
Container Sync
==============
.. automodule:: swift.container.sync
:members:
:undoc-members:
:show-inheritance:
.. _container-updater:
Container Updater
=================
.. automodule:: swift.container.updater
:members:
:undoc-members:
:show-inheritance:

2
doc/source/crossdomain.rst
View File

@ -47,7 +47,7 @@ For continuation lines, put some whitespace before the continuation
text. Ensure you put a completely blank line to terminate the
cross_domain_policy value.
The cross_domain_policy name/value is optional. If omited, the policy
The cross_domain_policy name/value is optional. If omitted, the policy
defaults as if you had specified::
cross_domain_policy = <allow-access-from domain="*" secure="false" />

2
doc/source/deployment_guide.rst
View File

@ -478,7 +478,7 @@ handoffs_first false If set to True, partitions that are
extreme situations.
handoff_delete auto By default handoff partitions will be
removed when it has successfully
replicated to all the cannonical nodes.
replicated to all the canonical nodes.
If set to an integer n, it will remove
the partition if it is successfully
replicated to n nodes. The default

2
doc/source/development_saio.rst
View File

@ -28,7 +28,7 @@ administrator (``root``) privileges; however, we assume that administrator logs
in as an unprivileged user and can use ``sudo`` to run privileged commands.
Swift processes also run under a separate user and group, set by configuration
option, and refered as ``<your-user-name>:<your-group-name>``. The default user
option, and referenced as ``<your-user-name>:<your-group-name>``. The default user
is ``swift``, which may not exist on your system. These instructions are
intended to allow a developer to use his/her username for
``<your-user-name>:<your-group-name>``.

28
doc/source/logs.rst
View File

@ -64,13 +64,39 @@ will not change. Swift log processing utilities should look for the first N
fields they require (e.g. in Python using something like
``log_line.split()[:14]`` to get up through the transaction id).
Swift Source
============
The ``source`` value in the proxy logs is used to identify the originator of a
request in the system. For example, if the client initiates a bulk upload, the
proxy server may end up doing many requests. The initial bulk upload request
will be logged as normal, but all of the internal "child requests" will have a
source value indicating they came from the bulk functionality.
======================= =============================
**Logged Source Value** **Originator of the Request**
----------------------- -----------------------------
FP :ref:`formpost`
SLO :ref:`static-large-objects`
SW :ref:`staticweb`
TU :ref:`tempurl`
BD :ref:`bulk` (delete)
EA :ref:`bulk` (extract)
CQ :ref:`container-quotas`
CS :ref:`container-sync`
TA :ref:`common_tempauth`
DLO :ref:`dynamic-large-objects`
======================= =============================
-----------------
Storage Node Logs
-----------------
Swift's account, container, and object server processes each log requests
that they receive. The format for these log lines is::
that they receive, if they have been configured to do so with the
``log_requests`` config parameter (which defaults to true). The format for
these log lines is::
remote_addr - - [datetime] "request_method request_path" status_int
content_length "referer" "transaction_id" "user_agent" request_time

286
doc/source/middleware.rst
View File

@ -4,111 +4,19 @@
Middleware
**********
.. _common_tempauth:
Account Quotas
==============
TempAuth
========
.. automodule:: swift.common.middleware.tempauth
.. automodule:: swift.common.middleware.account_quotas
:members:
:show-inheritance:
KeystoneAuth
============
.. _bulk:
.. automodule:: swift.common.middleware.keystoneauth
:members:
:show-inheritance:
Bulk Operations (Delete and Archive Auto Extraction)
====================================================
.. _healthcheck:
Healthcheck
===========
.. automodule:: swift.common.middleware.healthcheck
:members:
:show-inheritance:
.. _recon:
Recon
===========
.. automodule:: swift.common.middleware.recon
:members:
:show-inheritance:
.. _memecached:
Ratelimit
=========
.. automodule:: swift.common.middleware.ratelimit
:members:
:show-inheritance:
StaticWeb
=========
.. automodule:: swift.common.middleware.staticweb
:members:
:show-inheritance:
.. _tempurl:
TempURL
=======
.. automodule:: swift.common.middleware.tempurl
:members:
:show-inheritance:
FormPost
========
.. automodule:: swift.common.middleware.formpost
:members:
:show-inheritance:
Domain Remap
============
.. automodule:: swift.common.middleware.domain_remap
:members:
:show-inheritance:
CNAME Lookup
============
.. automodule:: swift.common.middleware.cname_lookup
:members:
:show-inheritance:
Cross Domain Policies
=====================
.. automodule:: swift.common.middleware.crossdomain
:members:
:show-inheritance:
Name Check (Forbidden Character Filter)
=======================================
.. automodule:: swift.common.middleware.name_check
:members:
:show-inheritance:
Memcache
========
.. automodule:: swift.common.middleware.memcache
:members:
:show-inheritance:
Proxy Logging
=============
.. automodule:: swift.common.middleware.proxy_logging
.. automodule:: swift.common.middleware.bulk
:members:
:show-inheritance:
@ -121,21 +29,14 @@ CatchErrors
:members:
:show-inheritance:
.. _gatekeeper:
CNAME Lookup
============
GateKeeper
=============
.. automodule:: swift.common.middleware.gatekeeper
.. automodule:: swift.common.middleware.cname_lookup
:members:
:show-inheritance:
Bulk Operations (Delete and Archive Auto Extraction)
====================================================
.. automodule:: swift.common.middleware.bulk
:members:
:show-inheritance:
.. _container-quotas:
Container Quotas
================
@ -144,35 +45,7 @@ Container Quotas
:members:
:show-inheritance:
Account Quotas
==============
.. automodule:: swift.common.middleware.account_quotas
:members:
:show-inheritance:
.. _slo-doc:
Static Large Objects
====================
.. automodule:: swift.common.middleware.slo
:members:
:show-inheritance:
Dynamic Large Objects
=====================
.. automodule:: swift.common.middleware.dlo
:members:
:show-inheritance:
List Endpoints
==============
.. automodule:: swift.common.middleware.list_endpoints
:members:
:show-inheritance:
.. _container-sync:
Container Sync Middleware
=========================
@ -181,6 +54,13 @@ Container Sync Middleware
:members:
:show-inheritance:
Cross Domain Policies
=====================
.. automodule:: swift.common.middleware.crossdomain
:members:
:show-inheritance:
.. _discoverability:
Discoverability
@ -201,3 +81,131 @@ retrieve it, one must use an HMAC-signed request, similar to TempURL.
The signature may be produced like so::
swift-temp-url GET 3600 /info secret 2>/dev/null | sed s/temp_url/swiftinfo/g
Domain Remap
============
.. automodule:: swift.common.middleware.domain_remap
:members:
:show-inheritance:
Dynamic Large Objects
=====================
.. automodule:: swift.common.middleware.dlo
:members:
:show-inheritance:
.. _formpost:
FormPost
========
.. automodule:: swift.common.middleware.formpost
:members:
:show-inheritance:
.. _gatekeeper:
GateKeeper
=============
.. automodule:: swift.common.middleware.gatekeeper
:members:
:show-inheritance:
.. _healthcheck:
Healthcheck
===========
.. automodule:: swift.common.middleware.healthcheck
:members:
:show-inheritance:
KeystoneAuth
============
.. automodule:: swift.common.middleware.keystoneauth
:members:
:show-inheritance:
List Endpoints
==============
.. automodule:: swift.common.middleware.list_endpoints
:members:
:show-inheritance:
Memcache
========
.. automodule:: swift.common.middleware.memcache
:members:
:show-inheritance:
Name Check (Forbidden Character Filter)
=======================================
.. automodule:: swift.common.middleware.name_check
:members:
:show-inheritance:
Proxy Logging
=============
.. automodule:: swift.common.middleware.proxy_logging
:members:
:show-inheritance:
Ratelimit
=========
.. automodule:: swift.common.middleware.ratelimit
:members:
:show-inheritance:
.. _recon:
Recon
===========
.. automodule:: swift.common.middleware.recon
:members:
:show-inheritance:
.. _slo-doc:
Static Large Objects
====================
.. automodule:: swift.common.middleware.slo
:members:
:show-inheritance:
.. _staticweb:
StaticWeb
=========
.. automodule:: swift.common.middleware.staticweb
:members:
:show-inheritance:
.. _common_tempauth:
TempAuth
========
.. automodule:: swift.common.middleware.tempauth
:members:
:show-inheritance:
.. _tempurl:
TempURL
=======
.. automodule:: swift.common.middleware.tempurl
:members:
:show-inheritance:

118
doc/source/misc.rst
View File

@ -4,14 +4,22 @@
Misc
****
.. _exceptions:
.. _acls:
Exceptions
==========
ACLs
====
.. automodule:: swift.common.exceptions
.. automodule:: swift.common.middleware.acl
:members:
:show-inheritance:
.. _buffered_http:
Buffered HTTP
=============
.. automodule:: swift.common.bufferedhttp
:members:
:undoc-members:
:show-inheritance:
.. _constraints:
@ -24,53 +32,13 @@ Constraints
:undoc-members:
:show-inheritance:
.. _utils:
Container Sync Realms
=====================
Utils
=====
.. automodule:: swift.common.utils
.. automodule:: swift.common.container_sync_realms
:members:
:show-inheritance:
.. _swob:
Swob
====
.. automodule:: swift.common.swob
:members:
:show-inheritance:
:special-members: __call__
.. _acls:
ACLs
====
.. automodule:: swift.common.middleware.acl
:members:
:show-inheritance:
.. _wsgi:
WSGI
====
.. automodule:: swift.common.wsgi
:members:
:show-inheritance:
.. _request_helpers:
Request Helpers
===============
.. automodule:: swift.common.request_helpers
:members:
:undoc-members:
:show-inheritance:
.. _direct_client:
Direct Client
@ -81,6 +49,16 @@ Direct Client
:undoc-members:
:show-inheritance:
.. _exceptions:
Exceptions
==========
.. automodule:: swift.common.exceptions
:members:
:undoc-members:
:show-inheritance:
.. _internal_client:
Internal Client
@ -91,12 +69,10 @@ Internal Client
:undoc-members:
:show-inheritance:
.. _buffered_http:
Manager
=========
Buffered HTTP
=============
.. automodule:: swift.common.bufferedhttp
.. automodule:: swift.common.manager
:members:
:show-inheritance:
@ -107,16 +83,40 @@ MemCacheD
:members:
:show-inheritance:
Container Sync Realms
=====================
.. _request_helpers:
.. automodule:: swift.common.container_sync_realms
Request Helpers
===============
.. automodule:: swift.common.request_helpers
:members:
:undoc-members:
:show-inheritance:
.. _swob:
Swob
====
.. automodule:: swift.common.swob
:members:
:show-inheritance:
:special-members: __call__
.. _utils:
Utils
=====
.. automodule:: swift.common.utils
:members:
:show-inheritance:
Manager
=========
.. _wsgi:
.. automodule:: swift.common.manager
WSGI
====
.. automodule:: swift.common.wsgi
:members:
:show-inheritance:

48
doc/source/object.rst
View File

@ -4,12 +4,22 @@
Object
******
.. _object-server:
.. _object-auditor:
Object Server
=============
Object Auditor
==============
.. automodule:: swift.obj.server
.. automodule:: swift.obj.auditor
:members:
:undoc-members:
:show-inheritance:
.. _object-diskfile:
Object Backend
==============
.. automodule:: swift.obj.diskfile
:members:
:undoc-members:
:show-inheritance:
@ -34,6 +44,16 @@ Object Replicator
:undoc-members:
:show-inheritance:
.. _object-server:
Object Server
=============
.. automodule:: swift.obj.server
:members:
:undoc-members:
:show-inheritance:
.. _object-updater:
Object Updater
@ -43,23 +63,3 @@ Object Updater
:members:
:undoc-members:
:show-inheritance:
.. _object-auditor:
Object Auditor
==============
.. automodule:: swift.obj.auditor
:members:
:undoc-members:
:show-inheritance:
.. _object-diskfile:
Object Backend
==============
.. automodule:: swift.obj.diskfile
:members:
:undoc-members:
:show-inheritance:

22
doc/source/overview_auth.rst
View File

@ -74,21 +74,21 @@ Access Level Description
============ ==============================================================
read-only These identities can read *everything* (except privileged
headers) in the account. Specifically, a user with read-only
account access can get a list of containers in the account,
list the contents of any container, retrieve any object, and
see the (non-privileged) headers of the account, any
container, or any object.
account access can get a list of containers in the account,
list the contents of any container, retrieve any object, and
see the (non-privileged) headers of the account, any
container, or any object.
read-write These identities can read or write (or create) any container.
A user with read-write account access can create new
containers, set any unprivileged container headers, overwrite
objects, delete containers, etc. A read-write user can NOT
set account headers (or perform any PUT/POST/DELETE requests
on the account).
containers, set any unprivileged container headers, overwrite
objects, delete containers, etc. A read-write user can NOT
set account headers (or perform any PUT/POST/DELETE requests
on the account).
admin These identities have "swift_owner" privileges. A user with
admin account access can do anything the account owner can,
including setting account headers and any privileged headers
-- and thus granting read-only, read-write, or admin access
to other users.
including setting account headers and any privileged headers
-- and thus granting read-only, read-write, or admin access
to other users.
============ ==============================================================

2
doc/source/overview_container_sync.rst
View File

@ -432,7 +432,7 @@ this seemingly-redundant work, any container-sync failure results in
unsynchronized objects. Note that the container sync will persistently
retry to sync any faulty object until success, while logging each failure.
Once it's done with the fallback rows, and assuming no faults occured,
Once it's done with the fallback rows, and assuming no faults occurred,
SP2 is advanced to SP1. ::
SP2

4
doc/source/overview_large_objects.rst
View File

@ -15,6 +15,8 @@ special manifest file is created that, when downloaded, sends all the segments
concatenated as a single object. This also offers much greater upload speed
with the possibility of parallel uploads of the segments.
.. _dynamic-large-objects:
---------------------
Dynamic Large Objects
---------------------
@ -102,6 +104,8 @@ Here's an example using ``curl`` with tiny 1-byte segments::
curl -H 'X-Auth-Token: <token>' \
http://<storage_url>/container/myobject
.. _static-large-objects:
--------------------
Static Large Objects
--------------------

20
doc/source/proxy.rst
View File

@ -4,16 +4,6 @@
Proxy
*****
.. _proxy-server:
Proxy Server
============
.. automodule:: swift.proxy.server
:members:
:undoc-members:
:show-inheritance:
.. _proxy-controllers:
Proxy Controllers
@ -50,3 +40,13 @@ Object
:members:
:undoc-members:
:show-inheritance:
.. _proxy-server:
Proxy Server
============
.. automodule:: swift.proxy.server
:members:
:undoc-members:
:show-inheritance:

2
doc/source/ratelimit.rst
View File

@ -90,7 +90,7 @@ bottleneck from causing a problem. There could also be a problem where a single
account is just using too much of the cluster's resources. In this case, the
container ratelimits may not help because the customer could be doing thousands
of reqs/sec to distributed containers each getting a small fraction of the
total so those limits would never trigger. If a system adminstrator notices
total so those limits would never trigger. If a system administrator notices
this, he/she can set the X-Account-Sysmeta-Global-Write-Ratelimit on an account
and that will limit the total number of write requests (PUT, POST, DELETE,
COPY) that account can do for the whole account. This limit will be in addition

8
etc/object-server.conf-sample
View File

@ -150,15 +150,15 @@ use = egg:swift#recon
# stats_interval = 300
#
# The sync method to use; default is rsync but you can use ssync to try the
# EXPERIMENTAL all-swift-code-no-rsync-callouts method. Once verified as stable
# and nearly as efficient (or moreso) than rsync, we plan to deprecate rsync so
# we can move on with more features for replication.
# EXPERIMENTAL all-swift-code-no-rsync-callouts method. Once ssync is verified
# as having performance comparable to, or better than, rsync, we plan to
# deprecate rsync so we can move on with more features for replication.
# sync_method = rsync
#
# max duration of a partition rsync
# rsync_timeout = 900
#
# bandwith limit for rsync in kB/s. 0 means unlimited
# bandwidth limit for rsync in kB/s. 0 means unlimited
# rsync_bwlimit = 0
#
# passed to rsync for io op timeout

8
etc/proxy-server.conf-sample
View File

@ -396,7 +396,10 @@ use = egg:swift#cname_lookup
# set log_headers = false
# set log_address = /dev/log
#
# Specify the storage_domain that match your cloud, multiple domains
# can be specified separated by a comma
# storage_domain = example.com
#
# lookup_depth = 1
# Note: Put staticweb just after your auth filter(s) in the pipeline
@ -466,6 +469,11 @@ use = egg:swift#proxy_logging
# access_log_statsd_metric_prefix =
# access_log_headers = false
#
# If access_log_headers is True and access_log_headers_only is set only
# these headers are logged. Multiple headers can be defined as comma separated
# list like this: access_log_headers_only = Host, X-Object-Meta-Mtime
# access_log_headers_only =
#
# By default, the X-Auth-Token is logged. To obscure the value,
# set reveal_sensitive_prefix to the number of characters to log.
# For example, if set to 12, only the first 12 characters of the

28
swift/account/server.py
View File

@ -48,6 +48,7 @@ class AccountController(object):
def __init__(self, conf, logger=None):
self.logger = logger or get_logger(conf, log_route='account-server')
self.log_requests = config_true_value(conf.get('log_requests', 'true'))
self.root = conf.get('devices', '/srv/node')
self.mount_check = config_true_value(conf.get('mount_check', 'true'))
replication_server = conf.get('replication_server', None)
@ -296,19 +297,20 @@ class AccountController(object):
if res.headers.get('x-container-timestamp') is not None:
additional_info += 'x-container-timestamp: %s' % \
res.headers['x-container-timestamp']
log_message = '%s - - [%s] "%s %s" %s %s "%s" "%s" "%s" %s "%s"' % (
req.remote_addr,
time.strftime('%d/%b/%Y:%H:%M:%S +0000', time.gmtime()),
req.method, req.path,
res.status.split()[0], res.content_length or '-',
req.headers.get('x-trans-id', '-'),
req.referer or '-', req.user_agent or '-',
trans_time,
additional_info)
if req.method.upper() == 'REPLICATE':
self.logger.debug(log_message)
else:
self.logger.info(log_message)
if self.log_requests:
log_msg = '%s - - [%s] "%s %s" %s %s "%s" "%s" "%s" %s "%s"' % (
req.remote_addr,
time.strftime('%d/%b/%Y:%H:%M:%S +0000', time.gmtime()),
req.method, req.path,
res.status.split()[0], res.content_length or '-',
req.headers.get('x-trans-id', '-'),
req.referer or '-', req.user_agent or '-',
trans_time,
additional_info)
if req.method.upper() == 'REPLICATE':
self.logger.debug(log_msg)
else:
self.logger.info(log_msg)
return res(env, start_response)

2
swift/cli/recon.py
View File

@ -838,7 +838,7 @@ class SwiftRecon(object):
if self.server_type == 'object':
self.async_check(hosts)
else:
print "Error: Can't check async's on non object servers."
print "Error: Can't check asyncs on non object servers."
if options.unmounted:
self.umount_check(hosts)
if options.replication:

26
swift/cli/ringbuilder.py
View File

@ -620,11 +620,11 @@ swift-ring-builder <builder_file> rebalance <seed>
parts, balance = builder.rebalance(seed=get_seed(3))
except exceptions.RingBuilderError as e:
print '-' * 79
print ("An error has occurred during ring validation. Common\n"
"causes of failure are rings that are empty or do not\n"
"have enough devices to accommodate the replica count.\n"
"Original exception message:\n %s" % e.message
)
print("An error has occurred during ring validation. Common\n"
"causes of failure are rings that are empty or do not\n"
"have enough devices to accommodate the replica count.\n"
"Original exception message:\n %s" % e.message
)
print '-' * 79
exit(EXIT_ERROR)
if not parts:
@ -645,11 +645,11 @@ swift-ring-builder <builder_file> rebalance <seed>
builder.validate()
except exceptions.RingValidationError as e:
print '-' * 79
print ("An error has occurred during ring validation. Common\n"
"causes of failure are rings that are empty or do not\n"
"have enough devices to accommodate the replica count.\n"
"Original exception message:\n %s" % e.message
)
print("An error has occurred during ring validation. Common\n"
"causes of failure are rings that are empty or do not\n"
"have enough devices to accommodate the replica count.\n"
"Original exception message:\n %s" % e.message
)
print '-' * 79
exit(EXIT_ERROR)
print 'Reassigned %d (%.02f%%) partitions. Balance is now %.02f.' % \
@ -823,9 +823,9 @@ def main(arguments=None):
for line in wrap(' '.join(cmds), 79, initial_indent='Quick list: ',
subsequent_indent=' '):
print line
print ('Exit codes: 0 = operation successful\n'
' 1 = operation completed with warnings\n'
' 2 = error')
print('Exit codes: 0 = operation successful\n'
' 1 = operation completed with warnings\n'
' 2 = error')
exit(EXIT_SUCCESS)
builder_file, ring_file = parse_builder_ring_filename_args(argv)

16
swift/common/middleware/acl.py
View File

@ -207,17 +207,21 @@ def parse_acl_v2(data):
Parses a version-2 Swift ACL string and returns a dict of ACL info.
:param data: string containing the ACL data in JSON format
:returns: A dict containing ACL info, e.g.:
:returns: A dict (possibly empty) containing ACL info, e.g.:
{"groups": [...], "referrers": [...]}
:returns: None if data is None
:returns: empty dictionary if data does not parse as valid JSON
:returns: None if data is None, is not valid JSON or does not parse
as a dict
:returns: empty dictionary if data is an empty string
"""
if data is None:
return None
try:
return json.loads(data)
except ValueError:
if data is '':
return {}
try:
result = json.loads(data)
return (result if type(result) is dict else None)
except ValueError:
return None
def parse_acl(*args, **kwargs):

21
swift/common/middleware/cname_lookup.py
View File

@ -41,7 +41,7 @@ else: # executed if the try block finishes with no errors
MODULE_DEPENDENCY_MET = True
from swift.common.swob import Request, HTTPBadRequest
from swift.common.utils import cache_from_env, get_logger
from swift.common.utils import cache_from_env, get_logger, list_from_csv
def lookup_cname(domain): # pragma: no cover
@ -89,13 +89,22 @@ class CNAMELookupMiddleware(object):
# reraise the exception if the dependency wasn't met
raise ImportError('dnspython is required for this module')
self.app = app
self.storage_domain = conf.get('storage_domain', 'example.com')
if self.storage_domain and self.storage_domain[0] != '.':
self.storage_domain = '.' + self.storage_domain
storage_domain = conf.get('storage_domain', 'example.com')
self.storage_domain = ['.' + s for s in
list_from_csv(storage_domain)
if not s.startswith('.')]
self.storage_domain += [s for s in list_from_csv(storage_domain)
if s.startswith('.')]
self.lookup_depth = int(conf.get('lookup_depth', '1'))
self.memcache = None
self.logger = get_logger(conf, log_route='cname-lookup')
def _domain_endswith_in_storage_domain(self, a_domain):
for domain in self.storage_domain:
if a_domain.endswith(domain):
return True
return False
def __call__(self, env, start_response):
if not self.storage_domain:
return self.app(env, start_response)
@ -111,7 +120,7 @@ class CNAMELookupMiddleware(object):
if is_ip(given_domain):
return self.app(env, start_response)
a_domain = given_domain
if not a_domain.endswith(self.storage_domain):
if not self._domain_endswith_in_storage_domain(a_domain):
if self.memcache is None:
self.memcache = cache_from_env(env)
error = True
@ -131,7 +140,7 @@ class CNAMELookupMiddleware(object):
error = True
found_domain = None
break
elif found_domain.endswith(self.storage_domain):
elif self._domain_endswith_in_storage_domain(found_domain):
# Found it!
self.logger.info(
_('Mapped %(given_domain)s to %(found_domain)s') %

25
swift/common/middleware/container_quotas.py
View File

@ -79,9 +79,21 @@ class ContainerQuotaMiddleware(object):
return HTTPBadRequest(body='Invalid count quota.')
# check user uploads against quotas
elif obj and req.method == 'PUT':
container_info = get_container_info(
req.environ, self.app, swift_source='CQ')
elif obj and req.method in ('PUT', 'COPY'):
container_info = None
if req.method == 'PUT':
container_info = get_container_info(
req.environ, self.app, swift_source='CQ')
if req.method == 'COPY' and 'Destination' in req.headers:
dest = req.headers.get('Destination').lstrip('/')
path_info = req.environ['PATH_INFO']
req.environ['PATH_INFO'] = "/%s/%s/%s" % (
version, account, dest)
try:
container_info = get_container_info(
req.environ, self.app, swift_source='CQ')
finally:
req.environ['PATH_INFO'] = path_info
if not container_info or not is_success(container_info['status']):
# this will hopefully 404 later
return self.app
@ -90,10 +102,11 @@ class ContainerQuotaMiddleware(object):
'bytes' in container_info and \
container_info['meta']['quota-bytes'].isdigit():
content_length = (req.content_length or 0)
if 'x-copy-from' in req.headers:
src_cont, src_obj = check_copy_from_header(req)
if 'x-copy-from' in req.headers or req.method == 'COPY':
if 'x-copy-from' in req.headers:
container, obj = check_copy_from_header(req)
path = '/%s/%s/%s/%s' % (version, account,
src_cont, src_obj)
container, obj)
object_info = get_object_info(req.environ, self.app, path)
if not object_info or not object_info['length']:
content_length = 0

2
swift/common/middleware/crossdomain.py
View File

@ -47,7 +47,7 @@ class CrossDomainMiddleware(object):
text. Ensure you put a completely blank line to terminate the
cross_domain_policy value.
The cross_domain_policy name/value is optional. If omited, the policy
The cross_domain_policy name/value is optional. If omitted, the policy
defaults as if you had specified::
cross_domain_policy = <allow-access-from domain="*" secure="false" />

16
swift/common/middleware/dlo.py
View File

@ -21,9 +21,10 @@ from swift.common.exceptions import ListingIterError
from swift.common.http import is_success
from swift.common.swob import Request, Response, \
HTTPRequestedRangeNotSatisfiable, HTTPBadRequest
from swift.common.utils import get_logger, json, SegmentedIterable, \
from swift.common.utils import get_logger, json, \
RateLimitedIterator, read_conf_dir, quote
from swift.common.wsgi import WSGIContext
from swift.common.request_helpers import SegmentedIterable
from swift.common.wsgi import WSGIContext, make_request
from urllib import unquote
@ -35,13 +36,12 @@ class GetContext(WSGIContext):
def _get_container_listing(self, req, version, account, container,
prefix, marker=''):
con_req = req.copy_get()
con_req.script_name = ''
con_req.environ['swift.source'] = 'DLO'
con_req.range = None
con_req.path_info = '/'.join(['', version, account, container])
con_req = make_request(
req.environ, path='/'.join(['', version, account, container]),
method='GET',
headers={'x-auth-token': req.headers.get('x-auth-token')},
agent=('%(orig)s ' + 'DLO MultipartGET'), swift_source='DLO')
con_req.query_string = 'format=json&prefix=%s' % quote(prefix)
con_req.user_agent = '%s DLO MultipartGET' % con_req.user_agent
if marker:
con_req.query_string += '&marker=%s' % quote(marker)

16
swift/common/middleware/proxy_logging.py
View File

@ -77,7 +77,7 @@ from urllib import quote, unquote
from swift.common.swob import Request
from swift.common.utils import (get_logger, get_remote_client,
get_valid_utf8_str, config_true_value,
InputProxy)
InputProxy, list_from_csv)
from swift.common.constraints import MAX_HEADER_SIZE
QUOTE_SAFE = '/:'
@ -93,6 +93,10 @@ class ProxyLoggingMiddleware(object):
self.log_hdrs = config_true_value(conf.get(
'access_log_headers',
conf.get('log_headers', 'no')))
log_hdrs_only = list_from_csv(conf.get(
'access_log_headers_only', ''))
self.log_hdrs_only = [x.title() for x in log_hdrs_only]
# The leading access_* check is in case someone assumes that
# log_statsd_valid_http_methods behaves like the other log_statsd_*
# settings.
@ -151,8 +155,14 @@ class ProxyLoggingMiddleware(object):
the_request = the_request + '?' + req.query_string
logged_headers = None
if self.log_hdrs:
logged_headers = '\n'.join('%s: %s' % (k, v)
for k, v in req.headers.items())
if self.log_hdrs_only:
logged_headers = '\n'.join('%s: %s' % (k, v)
for k, v in req.headers.items()
if k in self.log_hdrs_only)
else:
logged_headers = '\n'.join('%s: %s' % (k, v)
for k, v in req.headers.items())
method = self.method_from_req(req)
end_gmtime_str = time.strftime('%d/%b/%Y/%H/%M/%S',
time.gmtime(end_time))

86
swift/common/middleware/slo.py
View File

@ -137,6 +137,7 @@ metadata which can be used for stats purposes.
from cStringIO import StringIO
from datetime import datetime
import mimetypes
import re
from hashlib import md5
from swift.common.exceptions import ListingIterError
from swift.common.swob import Request, HTTPBadRequest, HTTPServerError, \
@ -145,11 +146,12 @@ from swift.common.swob import Request, HTTPBadRequest, HTTPServerError, \
HTTPUnauthorized, HTTPRequestedRangeNotSatisfiable, Response
from swift.common.utils import json, get_logger, config_true_value, \
get_valid_utf8_str, override_bytes_from_content_type, split_path, \
register_swift_info, RateLimitedIterator, SegmentedIterable, \
closing_if_possible, close_if_possible, quote
register_swift_info, RateLimitedIterator, quote
from swift.common.request_helpers import SegmentedIterable, \
closing_if_possible, close_if_possible
from swift.common.constraints import check_utf8, MAX_BUFFERED_SLO_SEGMENTS
from swift.common.http import HTTP_NOT_FOUND, HTTP_UNAUTHORIZED, is_success
from swift.common.wsgi import WSGIContext
from swift.common.wsgi import WSGIContext, make_request
from swift.common.middleware.bulk import get_response_body, \
ACCEPTABLE_FORMATS, Bulk
@ -214,11 +216,11 @@ class SloGetContext(WSGIContext):
Fetch the submanifest, parse it, and return it.
Raise exception on failures.
"""
sub_req = req.copy_get()
sub_req.range = None
sub_req.environ['PATH_INFO'] = '/'.join(['', version, acc, con, obj])
sub_req.environ['swift.source'] = 'SLO'
sub_req.user_agent = "%s SLO MultipartGET" % sub_req.user_agent
sub_req = make_request(
req.environ, path='/'.join(['', version, acc, con, obj]),
method='GET',
headers={'x-auth-token': req.headers.get('x-auth-token')},
agent=('%(orig)s ' + 'SLO MultipartGET'), swift_source='SLO')
sub_resp = sub_req.get_response(self.slo.app)
if not is_success(sub_resp.status_int):
@ -297,6 +299,48 @@ class SloGetContext(WSGIContext):
first_byte = max(first_byte - seg_length, -1)
last_byte = max(last_byte - seg_length, -1)
def _need_to_refetch_manifest(self, req):
"""
Just because a response shows that an object is a SLO manifest does not
mean that response's body contains the entire SLO manifest. If it
doesn't, we need to make a second request to actually get the whole
thing.
Note: this assumes that X-Static-Large-Object has already been found.
"""
if req.method == 'HEAD':
return True
response_status = int(self._response_status[:3])
# These are based on etag, and the SLO's etag is almost certainly not
# the manifest object's etag. Still, it's highly likely that the
# submitted If-None-Match won't match the manifest object's etag, so
# we can avoid re-fetching the manifest if we got a successful
# response.
if ((req.if_match or req.if_none_match) and
not is_success(response_status)):
return True
if req.range and response_status in (206, 416):
content_range = ''
for header, value in self._response_headers:
if header.lower() == 'content-range':
content_range = value
break
# e.g. Content-Range: bytes 0-14289/14290
match = re.match('bytes (\d+)-(\d+)/(\d+)$', content_range)
if not match:
# Malformed or missing, so we don't know what we got.
return True
first_byte, last_byte, length = [int(x) for x in match.groups()]
# If and only if we actually got back the full manifest body, then
# we can avoid re-fetching the object.
got_everything = (first_byte == 0 and last_byte == length - 1)
return not got_everything
return False
def handle_slo_get_or_head(self, req, start_response):
"""
Takes a request and a start_response callable and does the normal WSGI
@ -336,23 +380,24 @@ class SloGetContext(WSGIContext):
self._response_exc_info)
return resp_iter
# Just because a response shows that an object is a SLO manifest does
# not mean that response's body contains the entire SLO manifest. If
# it doesn't, we need to make a second request to actually get the
# whole thing.
if req.method == 'HEAD' or req.range:
if self._need_to_refetch_manifest(req):
req.environ['swift.non_client_disconnect'] = True
close_if_possible(resp_iter)
del req.environ['swift.non_client_disconnect']
get_req = req.copy_get()
get_req.range = None
get_req.environ['swift.source'] = 'SLO'
get_req.user_agent = "%s SLO MultipartGET" % get_req.user_agent
get_req = make_request(
req.environ, method='GET',
headers={'x-auth-token': req.headers.get('x-auth-token')},
agent=('%(orig)s ' + 'SLO MultipartGET'), swift_source='SLO')
resp_iter = self._app_call(get_req.environ)
response = self.get_or_head_response(req, self._response_headers,
resp_iter)
# Any Content-Range from a manifest is almost certainly wrong for the
# full large object.
resp_headers = [(h, v) for h, v in self._response_headers
if not h.lower() == 'content-range']
response = self.get_or_head_response(
req, resp_headers, resp_iter)
return response(req.environ, start_response)
def get_or_head_response(self, req, resp_headers, resp_iter):
@ -384,7 +429,8 @@ class SloGetContext(WSGIContext):
req, content_length, response_headers, segments)
def _manifest_head_response(self, req, response_headers):
return HTTPOk(request=req, headers=response_headers, body='')
return HTTPOk(request=req, headers=response_headers, body='',
conditional_response=True)
def _manifest_get_response(self, req, content_length, response_headers,
segments):

26
swift/common/middleware/staticweb.py
View File

@ -209,7 +209,31 @@ class _StaticWebContext(WSGIContext):
:param prefix: Any prefix desired for the container listing.
"""
if not config_true_value(self._listings):
resp = HTTPNotFound()(env, self._start_response)
body = '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 ' \
'Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n' \
'<html>\n' \
'<head>\n' \
'<title>Listing of %s</title>\n' % cgi.escape(env['PATH_INFO'])
if self._listings_css:
body += ' <link rel="stylesheet" type="text/css" ' \
'href="%s" />\n' % self._build_css_path(prefix or '')
else:
body += ' <style type="text/css">\n' \
' h1 {font-size: 1em; font-weight: bold;}\n' \
' p {font-size: 2}\n' \
' </style>\n'
body += '</head>\n<body>' \
' <h1>Web Listing Disabled</h1>' \
' <p>The owner of this web site has disabled web listing.' \
' <p>If you are the owner of this web site, you can enable' \
' web listing by setting X-Container-Meta-Web-Listings.</p>'
if self._index:
body += '<h1>Index File Not Found</h1>' \
' <p>The owner of this web site has set ' \
' <b>X-Container-Meta-Web-Index: %s</b>. ' \
' However, this file is not found.</p>' % self._index
body += ' </body>\n</html>\n'
resp = HTTPNotFound(body=body)(env, self._start_response)
return self._error_response(resp, env, start_response)
tmp_env = make_pre_authed_env(
env, 'GET', '/%s/%s/%s' % (

2
swift/common/middleware/tempauth.py
View File

@ -324,7 +324,7 @@ class TempAuth(object):
acl_header = 'x-account-access-control'
acl_data = req.headers.get(acl_header)
result = parse_acl(version=2, data=acl_data)
if (not result and acl_data not in ('', '{}')):
if result is None:
return 'Syntax error in input (%r)' % acl_data
tempauth_acl_keys = 'admin read-write read-only'.split()

176
swift/common/request_helpers.py
View File

@ -20,11 +20,17 @@ Why not swift.common.utils, you ask? Because this way we can import things
from swob in here without creating circular imports.
"""
import sys
import time
from contextlib import contextmanager
from urllib import unquote
from swift.common.constraints import FORMAT2CONTENT_TYPE
from swift.common.exceptions import ListingIterError, SegmentError
from swift.common.http import is_success, HTTP_SERVICE_UNAVAILABLE
from swift.common.swob import HTTPBadRequest, HTTPNotAcceptable
from swift.common.utils import split_path, validate_device_partition
from urllib import unquote
from swift.common.storage_policy import POLICY_INDEX
from swift.common.wsgi import make_request
def get_param(req, name, default=None):
@ -208,3 +214,171 @@ def remove_items(headers, condition):
keys = filter(condition, headers)
removed.update((key, headers.pop(key)) for key in keys)
return removed
def close_if_possible(maybe_closable):
close_method = getattr(maybe_closable, 'close', None)
if callable(close_method):
return close_method()
@contextmanager
def closing_if_possible(maybe_closable):
"""
Like contextlib.closing(), but doesn't crash if the object lacks a close()
method.
PEP 333 (WSGI) says: "If the iterable returned by the application has a
close() method, the server or gateway must call that method upon
completion of the current request[.]" This function makes that easier.
"""
yield maybe_closable
close_if_possible(maybe_closable)
class SegmentedIterable(object):
"""
Iterable that returns the object contents for a large object.
:param req: original request object
:param app: WSGI application from which segments will come
:param listing_iter: iterable yielding the object segments to fetch,
along with the byte subranges to fetch, in the
form of a tuple (object-path, first-byte, last-byte)
or (object-path, None, None) to fetch the whole thing.
:param max_get_time: maximum permitted duration of a GET request (seconds)
:param logger: logger object
:param swift_source: value of swift.source in subrequest environ
(just for logging)
:param ua_suffix: string to append to user-agent.
:param name: name of manifest (used in logging only)
:param response: optional response object for the response being sent
to the client.
"""
def __init__(self, req, app, listing_iter, max_get_time,
logger, ua_suffix, swift_source,
name='<not specified>', response=None):
self.req = req
self.app = app
self.listing_iter = listing_iter
self.max_get_time = max_get_time
self.logger = logger
self.ua_suffix = " " + ua_suffix
self.swift_source = swift_source
self.name = name
self.response = response
def app_iter_range(self, *a, **kw):
"""
swob.Response will only respond with a 206 status in certain cases; one
of those is if the body iterator responds to .app_iter_range().
However, this object (or really, its listing iter) is smart enough to
handle the range stuff internally, so we just no-op this out for swob.
"""
return self
def __iter__(self):
start_time = time.time()
have_yielded_data = False
if self.response and self.response.content_length:
bytes_left = int(self.response.content_length)
else:
bytes_left = None
try:
for seg_path, seg_etag, seg_size, first_byte, last_byte \
in self.listing_iter:
if time.time() - start_time > self.max_get_time:
raise SegmentError(
'ERROR: While processing manifest %s, '
'max LO GET time of %ds exceeded' %
(self.name, self.max_get_time))
seg_req = make_request(
self.req.environ, path=seg_path, method='GET',
headers={'x-auth-token': self.req.headers.get(
'x-auth-token')},
agent=('%(orig)s ' + self.ua_suffix),
swift_source=self.swift_source)
if first_byte is not None or last_byte is not None:
seg_req.headers['Range'] = "bytes=%s-%s" % (
# The 0 is to avoid having a range like "bytes=-10",
# which actually means the *last* 10 bytes.
'0' if first_byte is None else first_byte,
'' if last_byte is None else last_byte)
seg_resp = seg_req.get_response(self.app)
if not is_success(seg_resp.status_int):
close_if_possible(seg_resp.app_iter)
raise SegmentError(
'ERROR: While processing manifest %s, '
'got %d while retrieving %s' %
(self.name, seg_resp.status_int, seg_path))
elif ((seg_etag and (seg_resp.etag != seg_etag)) or
(seg_size and (seg_resp.content_length != seg_size) and
not seg_req.range)):
# The content-length check is for security reasons. Seems
# possible that an attacker could upload a >1mb object and
# then replace it with a much smaller object with same
# etag. Then create a big nested SLO that calls that
# object many times which would hammer our obj servers. If
# this is a range request, don't check content-length
# because it won't match.
close_if_possible(seg_resp.app_iter)
raise SegmentError(
'Object segment no longer valid: '
'%(path)s etag: %(r_etag)s != %(s_etag)s or '
'%(r_size)s != %(s_size)s.' %
{'path': seg_req.path, 'r_etag': seg_resp.etag,
'r_size': seg_resp.content_length,
's_etag': seg_etag,
's_size': seg_size})
for chunk in seg_resp.app_iter:
have_yielded_data = True
if bytes_left is None:
yield chunk
elif bytes_left >= len(chunk):
yield chunk
bytes_left -= len(chunk)
else:
yield chunk[:bytes_left]
bytes_left -= len(chunk)
close_if_possible(seg_resp.app_iter)
raise SegmentError(
'Too many bytes for %(name)s; truncating in '
'%(seg)s with %(left)d bytes left' %
{'name': self.name, 'seg': seg_req.path,
'left': bytes_left})
close_if_possible(seg_resp.app_iter)
if bytes_left:
raise SegmentError(
'Not enough bytes for %s; closing connection' %
self.name)
except ListingIterError as err:
# I have to save this error because yielding the ' ' below clears
# the exception from the current stack frame.
excinfo = sys.exc_info()
self.logger.exception('ERROR: While processing manifest %s, %s',
self.name, err)
# Normally, exceptions before any data has been yielded will
# cause Eventlet to send a 5xx response. In this particular
# case of ListingIterError we don't want that and we'd rather
# just send the normal 2xx response and then hang up early
# since 5xx codes are often used to judge Service Level
# Agreements and this ListingIterError indicates the user has
# created an invalid condition.
if not have_yielded_data:
yield ' '
raise excinfo
except SegmentError as err:
self.logger.exception(err)
# This doesn't actually change the response status (we're too
# late for that), but this does make it to the logs.
if self.response:
self.response.status = HTTP_SERVICE_UNAVAILABLE
raise

28
swift/common/swob.py
View File

@ -591,7 +591,7 @@ class Range(object):
class Match(object):
"""
Wraps a Request's If-None-Match header as a friendly object.
Wraps a Request's If-[None-]Match header as a friendly object.
:param headerval: value of the header as a str
"""
@ -757,7 +757,7 @@ class Request(object):
remote_user = _req_environ_property('REMOTE_USER')
user_agent = _req_environ_property('HTTP_USER_AGENT')
query_string = _req_environ_property('QUERY_STRING')
if_match = _req_environ_property('HTTP_IF_MATCH')
if_match = _req_fancy_property(Match, 'if-match')
body_file = _req_environ_property('wsgi.input')
content_length = _header_int_property('content-length')
if_modified_since = _datetime_property('if-modified-since')
@ -1097,9 +1097,33 @@ class Response(object):
return content_size, content_type
def _response_iter(self, app_iter, body):
if self.conditional_response and self.request:
if self.etag and self.request.if_none_match and \
self.etag in self.request.if_none_match:
self.status = 304
self.content_length = 0
return ['']
if self.etag and self.request.if_match and \
self.etag not in self.request.if_match:
self.status = 412
self.content_length = 0
return ['']
if self.status_int == 404 and self.request.if_match \
and '*' in self.request.if_match:
# If none of the entity tags match, or if "*" is given and no
# current entity exists, the server MUST NOT perform the
# requested method, and MUST return a 412 (Precondition
# Failed) response. [RFC 2616 section 14.24]
self.status = 412
self.content_length = 0
return ['']
if self.request and self.request.method == 'HEAD':
# We explicitly do NOT want to set self.content_length to 0 here
return ['']
if self.conditional_response and self.request and \
self.request.range and self.request.range.ranges and \
not self.content_range:

192
swift/common/utils.py
View File

@ -28,6 +28,7 @@ import threading as stdlib_threading
import time
import uuid
import functools
import weakref
from hashlib import md5, sha1
from random import random, shuffle
from urllib import quote as _quote
@ -61,10 +62,8 @@ utf8_decoder = codecs.getdecoder('utf-8')
utf8_encoder = codecs.getencoder('utf-8')
from swift import gettext_ as _
from swift.common.exceptions import LockTimeout, MessageTimeout, \
ListingIterError, SegmentError
from swift.common.http import is_success, is_redirection, HTTP_NOT_FOUND, \
HTTP_SERVICE_UNAVAILABLE
from swift.common.exceptions import LockTimeout, MessageTimeout
from swift.common.http import is_success, is_redirection, HTTP_NOT_FOUND
# logging doesn't import patched as cleanly as one would like
from logging.handlers import SysLogHandler
@ -850,6 +849,22 @@ def timing_stats(**dec_kwargs):
return decorating_func
class LoggingHandlerWeakRef(weakref.ref):
"""
Like a weak reference, but passes through a couple methods that logging
handlers need.
"""
def close(self):
referent = self()
if referent:
referent.close()
def flush(self):
referent = self()
if referent:
referent.flush()
# double inheritance to support property with setter
class LogAdapter(logging.LoggerAdapter, object):
"""
@ -1133,6 +1148,32 @@ def get_logger(conf, name=None, log_to_console=False, log_route=None,
print >>sys.stderr, 'Error calling custom handler [%s]' % hook
except ValueError:
print >>sys.stderr, 'Invalid custom handler format [%s]' % hook
# Python 2.6 has the undesirable property of keeping references to all log
# handlers around forever in logging._handlers and logging._handlerList.
# Combine that with handlers that keep file descriptors, and you get an fd
# leak.
#
# And no, we can't share handlers; a SyslogHandler has a socket, and if
# two greenthreads end up logging at the same time, you could get message
# overlap that garbles the logs and makes eventlet complain.
#
# Python 2.7 uses weakrefs to avoid the leak, so let's do that too.
if sys.version_info[0] == 2 and sys.version_info[1] <= 6:
try:
logging._acquireLock() # some thread-safety thing
for handler in adapted_logger.logger.handlers:
if handler in logging._handlers:
wr = LoggingHandlerWeakRef(handler)
del logging._handlers[handler]
logging._handlers[wr] = 1
for i, handler_ref in enumerate(logging._handlerList):
if handler_ref is handler:
logging._handlerList[i] = LoggingHandlerWeakRef(
handler)
finally:
logging._releaseLock()
return adapted_logger
@ -2559,146 +2600,3 @@ def quote(value, safe='/'):
Patched version of urllib.quote that encodes utf-8 strings before quoting
"""
return _quote(get_valid_utf8_str(value), safe)
def close_if_possible(maybe_closable):
close_method = getattr(maybe_closable, 'close', None)
if callable(close_method):
return close_method()
@contextmanager
def closing_if_possible(maybe_closable):
"""
Like contextlib.closing(), but doesn't crash if the object lacks a close()
method.
PEP 333 (WSGI) says: "If the iterable returned by the application has a
close() method, the server or gateway must call that method upon
completion of the current request[.]" This function makes that easier.
"""
yield maybe_closable
close_if_possible(maybe_closable)
class SegmentedIterable(object):
"""
Iterable that returns the object contents for a large object.
:param req: original request object
:param app: WSGI application from which segments will come
:param listing_iter: iterable yielding the object segments to fetch,
along with the byte subranges to fetch, in the
form of a tuple (object-path, first-byte, last-byte)
or (object-path, None, None) to fetch the whole thing.
:param max_get_time: maximum permitted duration of a GET request (seconds)
:param logger: logger object
:param swift_source: value of swift.source in subrequest environ
(just for logging)
:param ua_suffix: string to append to user-agent.
:param name: name of manifest (used in logging only)
:param response: optional response object for the response being sent
to the client. Only affects logs.
"""
def __init__(self, req, app, listing_iter, max_get_time,
logger, ua_suffix, swift_source,
name='<not specified>', response=None):
self.req = req
self.app = app
self.listing_iter = listing_iter
self.max_get_time = max_get_time
self.logger = logger
self.ua_suffix = " " + ua_suffix
self.swift_source = swift_source
self.name = name
self.response = response
def app_iter_range(self, *a, **kw):
"""
swob.Response will only respond with a 206 status in certain cases; one
of those is if the body iterator responds to .app_iter_range().
However, this object (or really, its listing iter) is smart enough to
handle the range stuff internally, so we just no-op this out for swob.
"""
return self
def __iter__(self):
start_time = time.time()
have_yielded_data = False
try:
for seg_path, seg_etag, seg_size, first_byte, last_byte \
in self.listing_iter:
if time.time() - start_time > self.max_get_time:
raise SegmentError(
'ERROR: While processing manifest %s, '
'max LO GET time of %ds exceeded' %
(self.name, self.max_get_time))
seg_req = self.req.copy_get()
seg_req.range = None
seg_req.environ['PATH_INFO'] = seg_path
seg_req.environ['swift.source'] = self.swift_source
seg_req.user_agent = "%s %s" % (seg_req.user_agent,
self.ua_suffix)
if first_byte is not None or last_byte is not None:
seg_req.headers['Range'] = "bytes=%s-%s" % (
# The 0 is to avoid having a range like "bytes=-10",
# which actually means the *last* 10 bytes.
'0' if first_byte is None else first_byte,
'' if last_byte is None else last_byte)
seg_resp = seg_req.get_response(self.app)
if not is_success(seg_resp.status_int):
close_if_possible(seg_resp.app_iter)
raise SegmentError(
'ERROR: While processing manifest %s, '
'got %d while retrieving %s' %
(self.name, seg_resp.status_int, seg_path))
elif ((seg_etag and (seg_resp.etag != seg_etag)) or
(seg_size and (seg_resp.content_length != seg_size) and
not seg_req.range)):
# The content-length check is for security reasons. Seems
# possible that an attacker could upload a >1mb object and
# then replace it with a much smaller object with same
# etag. Then create a big nested SLO that calls that
# object many times which would hammer our obj servers. If
# this is a range request, don't check content-length
# because it won't match.
close_if_possible(seg_resp.app_iter)
raise SegmentError(
'Object segment no longer valid: '
'%(path)s etag: %(r_etag)s != %(s_etag)s or '
'%(r_size)s != %(s_size)s.' %
{'path': seg_req.path, 'r_etag': seg_resp.etag,
'r_size': seg_resp.content_length,
's_etag': seg_etag,
's_size': seg_size})
for chunk in seg_resp.app_iter:
yield chunk
have_yielded_data = True
close_if_possible(seg_resp.app_iter)
except ListingIterError as err:
# I have to save this error because yielding the ' ' below clears
# the exception from the current stack frame.
excinfo = sys.exc_info()
self.logger.exception('ERROR: While processing manifest %s, %s',
self.name, err)
# Normally, exceptions before any data has been yielded will
# cause Eventlet to send a 5xx response. In this particular
# case of ListingIterError we don't want that and we'd rather
# just send the normal 2xx response and then hang up early
# since 5xx codes are often used to judge Service Level
# Agreements and this ListingIterError indicates the user has
# created an invalid condition.
if not have_yielded_data:
yield ' '
raise excinfo
except SegmentError as err:
self.logger.exception(err)
# This doesn't actually change the response status (we're too
# late for that), but this does make it to the logs.
if self.response:
self.response.status = HTTP_SERVICE_UNAVAILABLE
raise

116
swift/common/wsgi.py
View File

@ -543,54 +543,10 @@ class WSGIContext(object):
return None
def make_pre_authed_request(env, method=None, path=None, body=None,
headers=None, agent='Swift', swift_source=None):
def make_env(env, method=None, path=None, agent='Swift', query_string=None,
swift_source=None):
"""
Makes a new swob.Request based on the current env but with the
parameters specified. Note that this request will be preauthorized.
:param env: The WSGI environment to base the new request on.
:param method: HTTP method of new request; default is from
the original env.
:param path: HTTP path of new request; default is from the
original env. path should be compatible with what you
would send to Request.blank. path should be quoted and it
can include a query string. for example:
'/a%20space?unicode_str%E8%AA%9E=y%20es'
:param body: HTTP body of new request; empty by default.
:param headers: Extra HTTP headers of new request; None by
default.
:param agent: The HTTP user agent to use; default 'Swift'. You
can put %(orig)s in the agent to have it replaced
with the original env's HTTP_USER_AGENT, such as
'%(orig)s StaticWeb'. You also set agent to None to
use the original env's HTTP_USER_AGENT or '' to
have no HTTP_USER_AGENT.
:param swift_source: Used to mark the request as originating out of
middleware. Will be logged in proxy logs.
:returns: Fresh swob.Request object.
"""
query_string = None
path = path or ''
if path and '?' in path:
path, query_string = path.split('?', 1)
newenv = make_pre_authed_env(env, method, path=unquote(path), agent=agent,
query_string=query_string,
swift_source=swift_source)
if not headers:
headers = {}
if body:
return Request.blank(path, environ=newenv, body=body, headers=headers)
else:
return Request.blank(path, environ=newenv, headers=headers)
def make_pre_authed_env(env, method=None, path=None, agent='Swift',
query_string=None, swift_source=None):
"""
Returns a new fresh WSGI environment with escalated privileges to
do backend checks, listings, etc. that the remote user wouldn't
be able to accomplish directly.
Returns a new fresh WSGI environment.
:param env: The WSGI environment to base the new environment on.
:param method: The new REQUEST_METHOD or None to use the
@ -635,10 +591,70 @@ def make_pre_authed_env(env, method=None, path=None, agent='Swift',
del newenv['HTTP_USER_AGENT']
if swift_source:
newenv['swift.source'] = swift_source
newenv['swift.authorize'] = lambda req: None
newenv['swift.authorize_override'] = True
newenv['REMOTE_USER'] = '.wsgi.pre_authed'
newenv['wsgi.input'] = StringIO('')
if 'SCRIPT_NAME' not in newenv:
newenv['SCRIPT_NAME'] = ''
return newenv
def make_request(env, method=None, path=None, body=None, headers=None,
agent='Swift', swift_source=None, make_env=make_env):
"""
Makes a new swob.Request based on the current env but with the
parameters specified.
:param env: The WSGI environment to base the new request on.
:param method: HTTP method of new request; default is from
the original env.
:param path: HTTP path of new request; default is from the
original env. path should be compatible with what you
would send to Request.blank. path should be quoted and it
can include a query string. for example:
'/a%20space?unicode_str%E8%AA%9E=y%20es'
:param body: HTTP body of new request; empty by default.
:param headers: Extra HTTP headers of new request; None by
default.
:param agent: The HTTP user agent to use; default 'Swift'. You
can put %(orig)s in the agent to have it replaced
with the original env's HTTP_USER_AGENT, such as
'%(orig)s StaticWeb'. You also set agent to None to
use the original env's HTTP_USER_AGENT or '' to
have no HTTP_USER_AGENT.
:param swift_source: Used to mark the request as originating out of
middleware. Will be logged in proxy logs.
:param make_env: make_request calls this make_env to help build the
swob.Request.
:returns: Fresh swob.Request object.
"""
query_string = None
path = path or ''
if path and '?' in path:
path, query_string = path.split('?', 1)
newenv = make_env(env, method, path=unquote(path), agent=agent,
query_string=query_string, swift_source=swift_source)
if not headers:
headers = {}
if body:
return Request.blank(path, environ=newenv, body=body, headers=headers)
else:
return Request.blank(path, environ=newenv, headers=headers)
def make_pre_authed_env(env, method=None, path=None, agent='Swift',
query_string=None, swift_source=None):
"""Same as :py:func:`make_env` but with preauthorization."""
newenv = make_env(
env, method=method, path=path, agent=agent, query_string=query_string,
swift_source=swift_source)
newenv['swift.authorize'] = lambda req: None
newenv['swift.authorize_override'] = True
newenv['REMOTE_USER'] = '.wsgi.pre_authed'
return newenv
def make_pre_authed_request(env, method=None, path=None, body=None,
headers=None, agent='Swift', swift_source=None):
"""Same as :py:func:`make_request` but with preauthorization."""
return make_request(
env, method=method, path=path, body=body, headers=headers, agent=agent,
swift_source=swift_source, make_env=make_pre_authed_env)

28
swift/container/server.py
View File

@ -57,6 +57,7 @@ class ContainerController(object):
def __init__(self, conf, logger=None):
self.logger = logger or get_logger(conf, log_route='container-server')
self.log_requests = config_true_value(conf.get('log_requests', 'true'))
self.root = conf.get('devices', '/srv/node')
self.mount_check = config_true_value(conf.get('mount_check', 'true'))
self.node_timeout = int(conf.get('node_timeout', 3))
@ -551,19 +552,20 @@ class ContainerController(object):
{'method': req.method, 'path': req.path})
res = HTTPInternalServerError(body=traceback.format_exc())
trans_time = '%.4f' % (time.time() - start_time)
log_message = '%s - - [%s] "%s %s" %s %s "%s" "%s" "%s" %s' % (
req.remote_addr,
time.strftime('%d/%b/%Y:%H:%M:%S +0000',
time.gmtime()),
req.method, req.path,
res.status.split()[0], res.content_length or '-',
req.headers.get('x-trans-id', '-'),
req.referer or '-', req.user_agent or '-',
trans_time)
if req.method.upper() == 'REPLICATE':
self.logger.debug(log_message)
else:
self.logger.info(log_message)
if self.log_requests:
log_message = '%s - - [%s] "%s %s" %s %s "%s" "%s" "%s" %s' % (
req.remote_addr,
time.strftime('%d/%b/%Y:%H:%M:%S +0000',
time.gmtime()),
req.method, req.path,
res.status.split()[0], res.content_length or '-',
req.headers.get('x-trans-id', '-'),
req.referer or '-', req.user_agent or '-',
trans_time)
if req.method.upper() == 'REPLICATE':
self.logger.debug(log_message)
else:
self.logger.info(log_message)
return res(env, start_response)

7
swift/container/updater.py
View File

@ -31,7 +31,7 @@ from swift.common.bufferedhttp import http_connect
from swift.common.exceptions import ConnectionTimeout
from swift.common.ring import Ring
from swift.common.utils import get_logger, config_true_value, ismount, \
dump_recon_cache
dump_recon_cache, quorum_size
from swift.common.daemon import Daemon
from swift.common.http import is_success, HTTP_INTERNAL_SERVER_ERROR
@ -225,13 +225,10 @@ class ContainerUpdater(Daemon):
info['object_count'], info['bytes_used'])
for node in nodes]
successes = 0
failures = 0
for event in events:
if is_success(event.wait()):
successes += 1
else:
failures += 1
if successes > failures:
if successes >= quorum_size(len(events)):
self.logger.increment('successes')
self.successes += 1
self.logger.debug(

19
swift/obj/server.py
View File

@ -474,14 +474,6 @@ class ObjectController(object):
with disk_file.open():
metadata = disk_file.get_metadata()
obj_size = int(metadata['Content-Length'])
if request.headers.get('if-match') not in (None, '*') and \
metadata['ETag'] not in request.if_match:
return HTTPPreconditionFailed(request=request)
if request.headers.get('if-none-match') is not None:
if metadata['ETag'] in request.if_none_match:
resp = HTTPNotModified(request=request)
resp.etag = metadata['ETag']
return resp
file_x_ts = metadata['X-Timestamp']
file_x_ts_flt = float(file_x_ts)
try:
@ -521,13 +513,8 @@ class ObjectController(object):
pass
response.headers['X-Timestamp'] = file_x_ts
resp = request.get_response(response)
except DiskFileNotExist:
if request.headers.get('if-match') == '*':
resp = HTTPPreconditionFailed(request=request)
else:
resp = HTTPNotFound(request=request)
except DiskFileQuarantined:
resp = HTTPNotFound(request=request)
except (DiskFileNotExist, DiskFileQuarantined):
resp = HTTPNotFound(request=request, conditional_response=True)
return resp
@public
@ -545,7 +532,7 @@ class ObjectController(object):
try:
metadata = disk_file.read_metadata()
except (DiskFileNotExist, DiskFileQuarantined):
return HTTPNotFound(request=request)
return HTTPNotFound(request=request, conditional_response=True)
response = Response(request=request, conditional_response=True)
response.headers['Content-Type'] = metadata.get(
'Content-Type', 'application/octet-stream')

88
test/functional/tests.py
View File

@ -1645,6 +1645,50 @@ class TestDlo(Base):
file_contents,
"aaaaaaaaaabbbbbbbbbbccccccccccddddddddddeeeeeeeeeeffffffffff")
def test_dlo_if_match_get(self):
manifest = self.env.container.file("man1")
etag = manifest.info()['etag']
self.assertRaises(ResponseError, manifest.read,
hdrs={'If-Match': 'not-%s' % etag})
self.assert_status(412)
manifest.read(hdrs={'If-Match': etag})
self.assert_status(200)
def test_dlo_if_none_match_get(self):
manifest = self.env.container.file("man1")
etag = manifest.info()['etag']
self.assertRaises(ResponseError, manifest.read,
hdrs={'If-None-Match': etag})
self.assert_status(304)
manifest.read(hdrs={'If-None-Match': "not-%s" % etag})
self.assert_status(200)
def test_dlo_if_match_head(self):
manifest = self.env.container.file("man1")
etag = manifest.info()['etag']
self.assertRaises(ResponseError, manifest.info,
hdrs={'If-Match': 'not-%s' % etag})
self.assert_status(412)
manifest.info(hdrs={'If-Match': etag})
self.assert_status(200)
def test_dlo_if_none_match_head(self):
manifest = self.env.container.file("man1")
etag = manifest.info()['etag']
self.assertRaises(ResponseError, manifest.info,
hdrs={'If-None-Match': etag})
self.assert_status(304)
manifest.info(hdrs={'If-None-Match': "not-%s" % etag})
self.assert_status(200)
class TestDloUTF8(Base2, TestDlo):
set_up = False
@ -1996,6 +2040,50 @@ class TestSlo(Base):
self.assertEqual('application/json; charset=utf-8',
got_info['content_type'])
def test_slo_if_match_get(self):
manifest = self.env.container.file("manifest-abcde")
etag = manifest.info()['etag']
self.assertRaises(ResponseError, manifest.read,
hdrs={'If-Match': 'not-%s' % etag})
self.assert_status(412)
manifest.read(hdrs={'If-Match': etag})
self.assert_status(200)
def test_slo_if_none_match_get(self):
manifest = self.env.container.file("manifest-abcde")
etag = manifest.info()['etag']
self.assertRaises(ResponseError, manifest.read,
hdrs={'If-None-Match': etag})
self.assert_status(304)
manifest.read(hdrs={'If-None-Match': "not-%s" % etag})
self.assert_status(200)
def test_slo_if_match_head(self):
manifest = self.env.container.file("manifest-abcde")
etag = manifest.info()['etag']
self.assertRaises(ResponseError, manifest.info,
hdrs={'If-Match': 'not-%s' % etag})
self.assert_status(412)
manifest.info(hdrs={'If-Match': etag})
self.assert_status(200)
def test_slo_if_none_match_head(self):
manifest = self.env.container.file("manifest-abcde")
etag = manifest.info()['etag']
self.assertRaises(ResponseError, manifest.info,
hdrs={'If-None-Match': etag})
self.assert_status(304)
manifest.info(hdrs={'If-None-Match': "not-%s" % etag})
self.assert_status(200)
class TestSloUTF8(Base2, TestSlo):
set_up = False

19
test/unit/account/test_server.py
View File

@ -20,6 +20,7 @@ import unittest
from tempfile import mkdtemp
from shutil import rmtree
from StringIO import StringIO
from test.unit import FakeLogger
import simplejson
import xml.dom.minidom
@ -1634,5 +1635,23 @@ class TestAccountController(unittest.TestCase):
response = self.controller.__call__(env, start_response)
self.assertEqual(response, answer)
def test_GET_log_requests_true(self):
self.controller.logger = FakeLogger()
self.controller.log_requests = True
req = Request.blank('/sda1/p/a', environ={'REQUEST_METHOD': 'GET'})
resp = req.get_response(self.controller)
self.assertEqual(resp.status_int, 404)
self.assertTrue(self.controller.logger.log_dict['info'])
def test_GET_log_requests_false(self):
self.controller.logger = FakeLogger()
self.controller.log_requests = False
req = Request.blank('/sda1/p/a', environ={'REQUEST_METHOD': 'GET'})
resp = req.get_response(self.controller)
self.assertEqual(resp.status_int, 404)
self.assertFalse(self.controller.logger.log_dict['info'])
if __name__ == '__main__':
unittest.main()

11
test/unit/common/middleware/test_acl.py
View File

@ -90,6 +90,17 @@ class TestACL(unittest.TestCase):
# No header "hdr" exists -- should return None
({}, None),
({'junk': 'junk'}, None),
# Empty ACLs should return empty dict
({'hdr': ''}, {}),
({'hdr': '{}'}, {}),
({'hdr': '{ }'}, {}),
# Bad input -- should return None
({'hdr': '["array"]'}, None),
({'hdr': 'null'}, None),
({'hdr': '"some_string"'}, None),
({'hdr': '123'}, None),
]
for hdrs_in, expected in tests:

44
test/unit/common/middleware/test_cname_lookup.py
View File

@ -14,6 +14,7 @@
# limitations under the License.
import unittest
import mock
from nose import SkipTest
try:
@ -193,3 +194,46 @@ class TestCNAMELookup(unittest.TestCase):
resp = app(req.environ, start_response)
self.assertEquals(resp, 'FAKE APP')
def test_storage_domains_conf_format(self):
conf = {'storage_domain': 'foo.com'}
app = cname_lookup.filter_factory(conf)(FakeApp())
self.assertEquals(app.storage_domain, ['.foo.com'])
conf = {'storage_domain': 'foo.com, '}
app = cname_lookup.filter_factory(conf)(FakeApp())
self.assertEquals(app.storage_domain, ['.foo.com'])
conf = {'storage_domain': 'foo.com, bar.com'}
app = cname_lookup.filter_factory(conf)(FakeApp())
self.assertEquals(app.storage_domain, ['.foo.com', '.bar.com'])
conf = {'storage_domain': 'foo.com, .bar.com'}
app = cname_lookup.filter_factory(conf)(FakeApp())
self.assertEquals(app.storage_domain, ['.foo.com', '.bar.com'])
conf = {'storage_domain': '.foo.com, .bar.com'}
app = cname_lookup.filter_factory(conf)(FakeApp())
self.assertEquals(app.storage_domain, ['.foo.com', '.bar.com'])
def test_multiple_storage_domains(self):
conf = {'storage_domain': 'storage1.com, storage2.com',
'lookup_depth': 2}
app = cname_lookup.CNAMELookupMiddleware(FakeApp(), conf)
def do_test(lookup_back):
req = Request.blank('/', environ={'REQUEST_METHOD': 'GET'},
headers={'Host': 'c.a.example.com'})
module = 'swift.common.middleware.cname_lookup.lookup_cname'
with mock.patch(module, lambda x: (0, lookup_back)):
return app(req.environ, start_response)
resp = do_test('c.storage1.com')
self.assertEquals(resp, 'FAKE APP')
resp = do_test('c.storage2.com')
self.assertEquals(resp, 'FAKE APP')
bad_domain = ['CNAME lookup failed to resolve to a valid domain']
resp = do_test('c.badtest.com')
self.assertEquals(resp, bad_domain)

151
test/unit/common/middleware/test_dlo.py
View File

@ -479,6 +479,75 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(headers.get("Content-Range"), None)
self.assertEqual(body, "aaaaabbbbbcccccdddddeeeee")
def test_if_match_matches(self):
manifest_etag = '"%s"' % hashlib.md5(
"seg01-etag" + "seg02-etag" + "seg03-etag" +
"seg04-etag" + "seg05-etag").hexdigest()
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-Match': manifest_etag})
status, headers, body = self.call_dlo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '200 OK')
self.assertEqual(headers['Content-Length'], '25')
self.assertEqual(body, 'aaaaabbbbbcccccdddddeeeee')
def test_if_match_does_not_match(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-Match': 'not it'})
status, headers, body = self.call_dlo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '412 Precondition Failed')
self.assertEqual(headers['Content-Length'], '0')
self.assertEqual(body, '')
def test_if_none_match_matches(self):
manifest_etag = '"%s"' % hashlib.md5(
"seg01-etag" + "seg02-etag" + "seg03-etag" +
"seg04-etag" + "seg05-etag").hexdigest()
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-None-Match': manifest_etag})
status, headers, body = self.call_dlo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '304 Not Modified')
self.assertEqual(headers['Content-Length'], '0')
self.assertEqual(body, '')
def test_if_none_match_does_not_match(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-None-Match': 'not it'})
status, headers, body = self.call_dlo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '200 OK')
self.assertEqual(headers['Content-Length'], '25')
self.assertEqual(body, 'aaaaabbbbbcccccdddddeeeee')
def test_get_with_if_modified_since(self):
# It's important not to pass the If-[Un]Modified-Since header to the
# proxy for segment GET requests, as it may result in 304 Not Modified
# responses, and those don't contain segment data.
req = swob.Request.blank(
'/v1/AUTH_test/mancon/manifest',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-Modified-Since': 'Wed, 12 Feb 2014 22:24:52 GMT',
'If-Unmodified-Since': 'Thu, 13 Feb 2014 23:25:53 GMT'})
status, headers, body, exc = self.call_dlo(req, expect_exception=True)
for _, _, hdrs in self.app.calls_with_headers[1:]:
self.assertFalse('If-Modified-Since' in hdrs)
self.assertFalse('If-Unmodified-Since' in hdrs)
def test_error_fetching_first_segment(self):
self.app.register(
'GET', '/v1/AUTH_test/c/seg_01',
@ -601,8 +670,10 @@ class TestDloGetManifest(DloTestCase):
environ={'REQUEST_METHOD': 'GET'})
with contextlib.nested(
mock.patch('swift.common.utils.time.time', mock_time),
mock.patch('swift.common.utils.is_success', mock_is_success),
mock.patch('swift.common.request_helpers.time.time',
mock_time),
mock.patch('swift.common.request_helpers.is_success',
mock_is_success),
mock.patch.object(dlo, 'is_success', mock_is_success)):
status, headers, body, exc = self.call_dlo(
req, expect_exception=True)
@ -611,6 +682,82 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(body, 'aaaaabbbbbccccc')
self.assertTrue(isinstance(exc, exceptions.SegmentError))
def test_get_oversize_segment(self):
# If we send a Content-Length header to the client, it's based on the
# container listing. If a segment gets bigger by the time we get to it
# (like if a client uploads a bigger segment w/the same name), we need
# to not send anything beyond the length we promised. Also, we should
# probably raise an exception.
# This is now longer than the original seg_03+seg_04+seg_05 combined
self.app.register(
'GET', '/v1/AUTH_test/c/seg_03',
swob.HTTPOk, {'Content-Length': '20', 'Etag': 'seg03-etag'},
'cccccccccccccccccccc')
req = swob.Request.blank(
'/v1/AUTH_test/mancon/manifest',
environ={'REQUEST_METHOD': 'GET'})
status, headers, body, exc = self.call_dlo(req, expect_exception=True)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '200 OK') # sanity check
self.assertEqual(headers.get('Content-Length'), '25') # sanity check
self.assertEqual(body, 'aaaaabbbbbccccccccccccccc')
self.assertTrue(isinstance(exc, exceptions.SegmentError))
self.assertEqual(
self.app.calls,
[('GET', '/v1/AUTH_test/mancon/manifest'),
('GET', '/v1/AUTH_test/c?format=json&prefix=seg'),
('GET', '/v1/AUTH_test/c/seg_01'),
('GET', '/v1/AUTH_test/c/seg_02'),
('GET', '/v1/AUTH_test/c/seg_03')])
def test_get_undersize_segment(self):
# If we send a Content-Length header to the client, it's based on the
# container listing. If a segment gets smaller by the time we get to
# it (like if a client uploads a smaller segment w/the same name), we
# need to raise an exception so that the connection will be closed by
# the WSGI server. Otherwise, the WSGI server will be waiting for the
# next request, the client will still be waiting for the rest of the
# response, and nobody will be happy.
# Shrink it by a single byte
self.app.register(
'GET', '/v1/AUTH_test/c/seg_03',
swob.HTTPOk, {'Content-Length': '4', 'Etag': 'seg03-etag'},
'cccc')
req = swob.Request.blank(
'/v1/AUTH_test/mancon/manifest',
environ={'REQUEST_METHOD': 'GET'})
status, headers, body, exc = self.call_dlo(req, expect_exception=True)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '200 OK') # sanity check
self.assertEqual(headers.get('Content-Length'), '25') # sanity check
self.assertEqual(body, 'aaaaabbbbbccccdddddeeeee')
self.assertTrue(isinstance(exc, exceptions.SegmentError))
def test_get_undersize_segment_range(self):
# Shrink it by a single byte
self.app.register(
'GET', '/v1/AUTH_test/c/seg_03',
swob.HTTPOk, {'Content-Length': '4', 'Etag': 'seg03-etag'},
'cccc')
req = swob.Request.blank(
'/v1/AUTH_test/mancon/manifest',
environ={'REQUEST_METHOD': 'GET'},
headers={'Range': 'bytes=0-14'})
status, headers, body, exc = self.call_dlo(req, expect_exception=True)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '206 Partial Content') # sanity check
self.assertEqual(headers.get('Content-Length'), '15') # sanity check
self.assertEqual(body, 'aaaaabbbbbcccc')
self.assertTrue(isinstance(exc, exceptions.SegmentError))
def fake_start_response(*args, **kwargs):
pass

20
test/unit/common/middleware/test_proxy_logging.py
View File

@ -343,6 +343,26 @@ class TestProxyLogging(unittest.TestCase):
headers = unquote(log_parts[14]).split('\n')
self.assert_('Host: localhost:80' in headers)
def test_access_log_headers_only(self):
app = proxy_logging.ProxyLoggingMiddleware(
FakeApp(), {'log_headers': 'yes',
'access_log_headers_only': 'FIRST, seCond'})
app.access_logger = FakeLogger()
req = Request.blank('/',
environ={'REQUEST_METHOD': 'GET'},
headers={'First': '1',
'Second': '2',
'Third': '3'})
resp = app(req.environ, start_response)
# exhaust generator
[x for x in resp]
log_parts = self._log_parts(app)
headers = unquote(log_parts[14]).split('\n')
self.assert_('First: 1' in headers)
self.assert_('Second: 2' in headers)
self.assert_('Third: 3' not in headers)
self.assert_('Host: localhost:80' not in headers)
def test_upload_size(self):
app = proxy_logging.ProxyLoggingMiddleware(FakeApp(),
{'log_headers': 'yes'})

54
test/unit/common/middleware/test_quotas.py
View File

@ -106,6 +106,18 @@ class TestContainerQuotas(unittest.TestCase):
res = req.get_response(app)
self.assertEquals(res.status_int, 413)
def test_exceed_bytes_quota_copy_verb(self):
app = container_quotas.ContainerQuotaMiddleware(FakeApp(), {})
cache = FakeCache({'bytes': 0, 'meta': {'quota-bytes': '2'}})
req = Request.blank('/v1/a/c2/o2',
environ={'REQUEST_METHOD': 'COPY',
'swift.object/a/c2/o2': {'length': 10},
'swift.cache': cache},
headers={'Destination': '/c/o'})
res = req.get_response(app)
self.assertEquals(res.status_int, 413)
def test_not_exceed_bytes_quota(self):
app = container_quotas.ContainerQuotaMiddleware(FakeApp(), {})
cache = FakeCache({'bytes': 0, 'meta': {'quota-bytes': '100'}})
@ -127,6 +139,17 @@ class TestContainerQuotas(unittest.TestCase):
res = req.get_response(app)
self.assertEquals(res.status_int, 200)
def test_not_exceed_bytes_quota_copy_verb(self):
app = container_quotas.ContainerQuotaMiddleware(FakeApp(), {})
cache = FakeCache({'bytes': 0, 'meta': {'quota-bytes': '100'}})
req = Request.blank('/v1/a/c2/o2',
environ={'REQUEST_METHOD': 'COPY',
'swift.object/a/c2/o2': {'length': 10},
'swift.cache': cache},
headers={'Destination': '/c/o'})
res = req.get_response(app)
self.assertEquals(res.status_int, 200)
def test_bytes_quota_copy_from_no_src(self):
app = container_quotas.ContainerQuotaMiddleware(FakeApp(), {})
cache = FakeCache({'bytes': 0, 'meta': {'quota-bytes': '100'}})
@ -148,6 +171,17 @@ class TestContainerQuotas(unittest.TestCase):
res = req.get_response(app)
self.assertEquals(res.status_int, 412)
def test_bytes_quota_copy_verb_no_src(self):
app = container_quotas.ContainerQuotaMiddleware(FakeApp(), {})
cache = FakeCache({'bytes': 0, 'meta': {'quota-bytes': '100'}})
req = Request.blank('/v1/a/c2/o3',
environ={'REQUEST_METHOD': 'COPY',
'swift.object/a/c2/o2': {'length': 10},
'swift.cache': cache},
headers={'Destination': '/c/o'})
res = req.get_response(app)
self.assertEquals(res.status_int, 200)
def test_exceed_counts_quota(self):
app = container_quotas.ContainerQuotaMiddleware(FakeApp(), {})
cache = FakeCache({'object_count': 1, 'meta': {'quota-count': '1'}})
@ -169,6 +203,16 @@ class TestContainerQuotas(unittest.TestCase):
res = req.get_response(app)
self.assertEquals(res.status_int, 413)
def test_exceed_counts_quota_copy_verb(self):
app = container_quotas.ContainerQuotaMiddleware(FakeApp(), {})
cache = FakeCache({'object_count': 1, 'meta': {'quota-count': '1'}})
req = Request.blank('/v1/a/c2/o2',
environ={'REQUEST_METHOD': 'COPY',
'swift.cache': cache},
headers={'Destination': '/c/o'})
res = req.get_response(app)
self.assertEquals(res.status_int, 413)
def test_not_exceed_counts_quota(self):
app = container_quotas.ContainerQuotaMiddleware(FakeApp(), {})
cache = FakeCache({'object_count': 1, 'meta': {'quota-count': '2'}})
@ -189,6 +233,16 @@ class TestContainerQuotas(unittest.TestCase):
res = req.get_response(app)
self.assertEquals(res.status_int, 200)
def test_not_exceed_counts_quota_copy_verb(self):
app = container_quotas.ContainerQuotaMiddleware(FakeApp(), {})
cache = FakeCache({'object_count': 1, 'meta': {'quota-count': '2'}})
req = Request.blank('/v1/a/c2/o2',
environ={'REQUEST_METHOD': 'COPY',
'swift.cache': cache},
headers={'Destination': '/c/o'})
res = req.get_response(app)
self.assertEquals(res.status_int, 200)
def test_invalid_quotas(self):
req = Request.blank(
'/v1/a/c',

226
test/unit/common/middleware/test_slo.py
View File

@ -726,6 +726,15 @@ class TestSloHeadManifest(SloTestCase):
md5("seg01-hashseg02-hash").hexdigest())
self.assertEqual(body, '') # it's a HEAD request, after all
def test_etag_matching(self):
etag = md5("seg01-hashseg02-hash").hexdigest()
req = Request.blank(
'/v1/AUTH_test/headtest/man',
environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-None-Match': etag})
status, headers, body = self.call_slo(req)
self.assertEqual(status, '304 Not Modified')
class TestSloGetManifest(SloTestCase):
def setUp(self):
@ -763,21 +772,25 @@ class TestSloGetManifest(SloTestCase):
'GET', '/v1/AUTH_test/gettest/manifest-bc',
swob.HTTPOk, {'Content-Type': 'application/json;swift_bytes=25',
'X-Static-Large-Object': 'true',
'X-Object-Meta-Plant': 'Ficus'},
'X-Object-Meta-Plant': 'Ficus',
'Etag': md5(_bc_manifest_json).hexdigest()},
_bc_manifest_json)
_abcd_manifest_json = json.dumps(
[{'name': '/gettest/a_5', 'hash': 'a',
'content_type': 'text/plain', 'bytes': '5'},
{'name': '/gettest/manifest-bc', 'sub_slo': True,
'content_type': 'application/json;swift_bytes=25',
'hash': md5("bc").hexdigest(),
'bytes': len(_bc_manifest_json)},
{'name': '/gettest/d_20', 'hash': 'd',
'content_type': 'text/plain', 'bytes': '20'}])
self.app.register(
'GET', '/v1/AUTH_test/gettest/manifest-abcd',
swob.HTTPOk, {'Content-Type': 'application/json',
'X-Static-Large-Object': 'true'},
json.dumps([{'name': '/gettest/a_5', 'hash': 'a',
'content_type': 'text/plain', 'bytes': '5'},
{'name': '/gettest/manifest-bc', 'sub_slo': True,
'content_type': 'application/json;swift_bytes=25',
'hash': 'manifest-bc',
'bytes': len(_bc_manifest_json)},
{'name': '/gettest/d_20', 'hash': 'd',
'content_type': 'text/plain', 'bytes': '20'}]))
'X-Static-Large-Object': 'true',
'Etag': md5(_abcd_manifest_json).hexdigest()},
_abcd_manifest_json)
self.app.register(
'GET', '/v1/AUTH_test/gettest/manifest-badjson',
@ -842,6 +855,77 @@ class TestSloGetManifest(SloTestCase):
self.assertFalse(
"SLO MultipartGET" in first_ua)
def test_if_none_match_matches(self):
manifest_etag = md5("a" + md5("bc").hexdigest() + "d").hexdigest()
req = Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-None-Match': manifest_etag})
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '304 Not Modified')
self.assertEqual(headers['Content-Length'], '0')
self.assertEqual(body, '')
def test_if_none_match_does_not_match(self):
manifest_etag = md5("a" + md5("bc").hexdigest() + "d").hexdigest()
req = Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-None-Match': "not-%s" % manifest_etag})
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '200 OK')
self.assertEqual(
body, 'aaaaabbbbbbbbbbcccccccccccccccdddddddddddddddddddd')
def test_if_match_matches(self):
manifest_etag = md5("a" + md5("bc").hexdigest() + "d").hexdigest()
req = Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-Match': manifest_etag})
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '200 OK')
self.assertEqual(
body, 'aaaaabbbbbbbbbbcccccccccccccccdddddddddddddddddddd')
def test_if_match_does_not_match(self):
manifest_etag = md5("a" + md5("bc").hexdigest() + "d").hexdigest()
req = Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-Match': "not-%s" % manifest_etag})
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '412 Precondition Failed')
self.assertEqual(headers['Content-Length'], '0')
self.assertEqual(body, '')
def test_if_match_matches_and_range(self):
manifest_etag = md5("a" + md5("bc").hexdigest() + "d").hexdigest()
req = Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-Match': manifest_etag,
'Range': 'bytes=3-6'})
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '206 Partial Content')
self.assertEqual(headers['Content-Length'], '4')
self.assertEqual(body, 'aabb')
def test_get_manifest_with_submanifest(self):
req = Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
@ -849,7 +933,7 @@ class TestSloGetManifest(SloTestCase):
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
manifest_etag = md5("a" + "manifest-bc" + "d").hexdigest()
manifest_etag = md5("a" + md5("bc").hexdigest() + "d").hexdigest()
self.assertEqual(status, '200 OK')
self.assertEqual(headers['Content-Length'], '50')
self.assertEqual(headers['Etag'], '"%s"' % manifest_etag)
@ -889,6 +973,103 @@ class TestSloGetManifest(SloTestCase):
self.assertEqual(self.app.swift_sources,
[None, 'SLO', 'SLO', 'SLO', 'SLO', 'SLO'])
def test_range_get_includes_whole_manifest(self):
# If the first range GET results in retrieval of the entire manifest
# body (which we can detect by looking at Content-Range), then we
# should not go make a second, non-ranged request just to retrieve the
# same bytes again.
req = Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
environ={'REQUEST_METHOD': 'GET'},
headers={'Range': 'bytes=0-999999999'})
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '206 Partial Content')
self.assertEqual(
body, 'aaaaabbbbbbbbbbcccccccccccccccdddddddddddddddddddd')
self.assertEqual(
self.app.calls,
[('GET', '/v1/AUTH_test/gettest/manifest-abcd'),
('GET', '/v1/AUTH_test/gettest/a_5'),
('GET', '/v1/AUTH_test/gettest/manifest-bc'),
('GET', '/v1/AUTH_test/gettest/b_10'),
('GET', '/v1/AUTH_test/gettest/c_15'),
('GET', '/v1/AUTH_test/gettest/d_20')])
def test_range_get_beyond_manifest(self):
big = 'e' * 1024 * 1024
big_etag = md5(big).hexdigest()
self.app.register(
'GET', '/v1/AUTH_test/gettest/big_seg',
swob.HTTPOk, {'Content-Type': 'application/foo',
'Etag': big_etag}, big)
big_manifest = json.dumps(
[{'name': '/gettest/big_seg', 'hash': big_etag,
'bytes': 1024 * 1024, 'content_type': 'application/foo'}])
self.app.register(
'GET', '/v1/AUTH_test/gettest/big_manifest',
swob.HTTPOk, {'Content-Type': 'application/octet-stream',
'X-Static-Large-Object': 'true',
'Etag': md5(big_manifest).hexdigest()},
big_manifest)
req = Request.blank(
'/v1/AUTH_test/gettest/big_manifest',
environ={'REQUEST_METHOD': 'GET'},
headers={'Range': 'bytes=100000-199999'})
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '206 Partial Content')
self.assertEqual(body, 'e' * 100000)
self.assertEqual(
self.app.calls, [
# has Range header, gets 416
('GET', '/v1/AUTH_test/gettest/big_manifest'),
# retry the first one
('GET', '/v1/AUTH_test/gettest/big_manifest'),
('GET', '/v1/AUTH_test/gettest/big_seg')])
def test_range_get_bogus_content_range(self):
# Just a little paranoia; Swift currently sends back valid
# Content-Range headers, but if somehow someone sneaks an invalid one
# in there, we'll ignore it.
def content_range_breaker_factory(app):
def content_range_breaker(env, start_response):
req = swob.Request(env)
resp = req.get_response(app)
resp.headers['Content-Range'] = 'triscuits'
return resp(env, start_response)
return content_range_breaker
self.slo = slo.filter_factory({})(
content_range_breaker_factory(self.app))
req = Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
environ={'REQUEST_METHOD': 'GET'},
headers={'Range': 'bytes=0-999999999'})
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
self.assertEqual(status, '206 Partial Content')
self.assertEqual(
body, 'aaaaabbbbbbbbbbcccccccccccccccdddddddddddddddddddd')
self.assertEqual(
self.app.calls,
[('GET', '/v1/AUTH_test/gettest/manifest-abcd'),
('GET', '/v1/AUTH_test/gettest/manifest-abcd'),
('GET', '/v1/AUTH_test/gettest/a_5'),
('GET', '/v1/AUTH_test/gettest/manifest-bc'),
('GET', '/v1/AUTH_test/gettest/b_10'),
('GET', '/v1/AUTH_test/gettest/c_15'),
('GET', '/v1/AUTH_test/gettest/d_20')])
def test_range_get_manifest_on_segment_boundaries(self):
req = Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
@ -1018,7 +1199,7 @@ class TestSloGetManifest(SloTestCase):
status, headers, body = self.call_slo(req)
headers = swob.HeaderKeyDict(headers)
manifest_etag = md5("a" + "manifest-bc" + "d").hexdigest()
manifest_etag = md5("a" + md5("bc").hexdigest() + "d").hexdigest()
self.assertEqual(status, '200 OK')
self.assertEqual(headers['Content-Length'], '50')
self.assertEqual(headers['Etag'], '"%s"' % manifest_etag)
@ -1086,6 +1267,21 @@ class TestSloGetManifest(SloTestCase):
# make sure we didn't keep asking for segments
self.assertEqual(self.app.call_count, 20)
def test_get_with_if_modified_since(self):
# It's important not to pass the If-[Un]Modified-Since header to the
# proxy for segment or submanifest GET requests, as it may result in
# 304 Not Modified responses, and those don't contain any useful data.
req = swob.Request.blank(
'/v1/AUTH_test/gettest/manifest-abcd',
environ={'REQUEST_METHOD': 'GET'},
headers={'If-Modified-Since': 'Wed, 12 Feb 2014 22:24:52 GMT',
'If-Unmodified-Since': 'Thu, 13 Feb 2014 23:25:53 GMT'})
status, headers, body, exc = self.call_slo(req, expect_exception=True)
for _, _, hdrs in self.app.calls_with_headers[1:]:
self.assertFalse('If-Modified-Since' in hdrs)
self.assertFalse('If-Unmodified-Since' in hdrs)
def test_error_fetching_segment(self):
self.app.register('GET', '/v1/AUTH_test/gettest/c_15',
swob.HTTPUnauthorized, {}, None)
@ -1223,8 +1419,10 @@ class TestSloGetManifest(SloTestCase):
environ={'REQUEST_METHOD': 'GET'})
with nested(patch.object(slo, 'is_success', mock_is_success),
patch('swift.common.utils.time.time', mock_time),
patch('swift.common.utils.is_success', mock_is_success)):
patch('swift.common.request_helpers.time.time',
mock_time),
patch('swift.common.request_helpers.is_success',
mock_is_success)):
status, headers, body, exc = self.call_slo(
req, expect_exception=True)

17
test/unit/common/middleware/test_staticweb.py
View File

@ -58,6 +58,8 @@ meta_map = {
'web-directory-type': 'text/directory'}},
'c12': {'meta': {'web-index': 'index.html',
'web-error': 'error.html'}},
'c13': {'meta': {'web-listings': 'f',
'web-listings-css': 'listing.css'}},
}
@ -604,6 +606,7 @@ class TestStaticWeb(unittest.TestCase):
def test_container7listing(self):
resp = Request.blank('/v1/a/c7/').get_response(self.test_staticweb)
self.assertEquals(resp.status_int, 404)
self.assert_('Web Listing Disabled' in resp.body)
def test_container8listingcss(self):
resp = Request.blank(
@ -665,6 +668,7 @@ class TestStaticWeb(unittest.TestCase):
resp = Request.blank('/v1/a/c11a/subdir/').get_response(
self.test_staticweb)
self.assertEquals(resp.status_int, 404)
self.assert_('Index File Not Found' in resp.body)
def test_container11subdirmarkeraltdirtype(self):
resp = Request.blank('/v1/a/c11a/subdir2/').get_response(
@ -682,6 +686,19 @@ class TestStaticWeb(unittest.TestCase):
self.assertEquals(resp.status_int, 200)
self.assert_('index file' in resp.body)
def test_container_404_has_css(self):
resp = Request.blank('/v1/a/c13/').get_response(
self.test_staticweb)
self.assertEquals(resp.status_int, 404)
self.assert_('listing.css' in resp.body)
def test_container_404_has_no_css(self):
resp = Request.blank('/v1/a/c7/').get_response(
self.test_staticweb)
self.assertEquals(resp.status_int, 404)
self.assert_('listing.css' not in resp.body)
self.assert_('<style' in resp.body)
def test_container_unicode_stdlib_json(self):
with mock.patch('swift.common.middleware.staticweb.json',
new=stdlib_json):

27
test/unit/common/middleware/test_tempauth.py
View File

@ -1016,13 +1016,16 @@ class TestAccountAcls(unittest.TestCase):
def test_acl_syntax_verification(self):
test_auth = auth.filter_factory({'user_admin_user': 'testing'})(
FakeApp(iter(NO_CONTENT_RESP * 3)))
FakeApp(iter(NO_CONTENT_RESP * 5)))
good_headers = {'X-Auth-Token': 'AUTH_t'}
good_acl = '{"read-only":["a","b"]}'
bad_acl = 'syntactically invalid acl -- this does not parse as JSON'
wrong_acl = '{"other-auth-system":["valid","json","but","wrong"]}'
bad_value_acl = '{"read-write":["fine"],"admin":"should be a list"}'
not_dict_acl = '["read-only"]'
not_dict_acl2 = 1
empty_acls = ['{}', '', '{ }']
target = '/v1/AUTH_firstacct'
# no acls -- no problem!
@ -1036,6 +1039,14 @@ class TestAccountAcls(unittest.TestCase):
resp = req.get_response(test_auth)
self.assertEquals(resp.status_int, 204)
# syntactically valid empty acls should go through
for acl in empty_acls:
update = {'x-account-access-control': acl}
req = self._make_request(target,
headers=dict(good_headers, **update))
resp = req.get_response(test_auth)
self.assertEquals(resp.status_int, 204)
errmsg = 'X-Account-Access-Control invalid: %s'
# syntactically invalid acls get a 400
update = {'x-account-access-control': bad_acl}
@ -1058,6 +1069,20 @@ class TestAccountAcls(unittest.TestCase):
self.assertEquals(resp.status_int, 400)
self.assertEquals(errmsg % "Value", resp.body[:39])
# acls with wrong json structure also get a 400
update = {'x-account-access-control': not_dict_acl}
req = self._make_request(target, headers=dict(good_headers, **update))
resp = req.get_response(test_auth)
self.assertEquals(resp.status_int, 400)
self.assertEquals(errmsg % "Syntax error", resp.body[:46])
# acls with wrong json structure also get a 400
update = {'x-account-access-control': not_dict_acl2}
req = self._make_request(target, headers=dict(good_headers, **update))
resp = req.get_response(test_auth)
self.assertEquals(resp.status_int, 400)
self.assertEquals(errmsg % "Syntax error", resp.body[:46])
def test_acls_propagate_to_sysmeta(self):
test_auth = auth.filter_factory({'user_admin_user': 'testing'})(
FakeApp(iter(NO_CONTENT_RESP * 3)))

6
test/unit/common/test_db.py
View File

@ -86,7 +86,7 @@ class TestChexor(unittest.TestCase):
class TestGreenDBConnection(unittest.TestCase):
def test_execute_when_locked(self):
# This test is dependant on the code under test calling execute and
# This test is dependent on the code under test calling execute and
# commit as sqlite3.Cursor.execute in a subclass.
class InterceptCursor(sqlite3.Cursor):
pass
@ -102,7 +102,7 @@ class TestGreenDBConnection(unittest.TestCase):
InterceptCursor.execute.call_count))
def text_commit_when_locked(self):
# This test is dependant on the code under test calling commit and
# This test is dependent on the code under test calling commit and
# commit as sqlite3.Connection.commit in a subclass.
class InterceptConnection(sqlite3.Connection):
pass
@ -129,7 +129,7 @@ class TestGetDBConnection(unittest.TestCase):
'invalid database path / name')
def test_locked_db(self):
# This test is dependant on the code under test calling execute and
# This test is dependent on the code under test calling execute and
# commit as sqlite3.Cursor.execute in a subclass.
class InterceptCursor(sqlite3.Cursor):
pass

124
test/unit/common/test_swob.py
View File

@ -347,7 +347,6 @@ class TestRequest(unittest.TestCase):
self.assertEquals(req.query_string, 'a=b&c=d')
self.assertEquals(req.environ['QUERY_STRING'], 'a=b&c=d')
req = blank('/', if_match='*')
self.assertEquals(req.if_match, '*')
self.assertEquals(req.environ['HTTP_IF_MATCH'], '*')
self.assertEquals(req.headers['If-Match'], '*')
@ -366,7 +365,6 @@ class TestRequest(unittest.TestCase):
self.assertEquals(req.user_agent, 'curl/7.22.0 (x86_64-pc-linux-gnu)')
self.assertEquals(req.query_string, 'a=b&c=d')
self.assertEquals(req.environ['QUERY_STRING'], 'a=b&c=d')
self.assertEquals(req.if_match, '*')
def test_invalid_req_environ_property_args(self):
# getter only property
@ -1325,5 +1323,127 @@ class TestUTC(unittest.TestCase):
self.assertEquals(swift.common.swob.UTC.tzname(None), 'UTC')
class TestConditionalIfNoneMatch(unittest.TestCase):
def fake_app(self, environ, start_response):
start_response('200 OK', [('Etag', 'the-etag')])
return ['hi']
def fake_start_response(*a, **kw):
pass
def test_simple_match(self):
# etag matches --> 304
req = swift.common.swob.Request.blank(
'/', headers={'If-None-Match': 'the-etag'})
resp = req.get_response(self.fake_app)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 304)
self.assertEquals(body, '')
def test_quoted_simple_match(self):
# double quotes don't matter
req = swift.common.swob.Request.blank(
'/', headers={'If-None-Match': '"the-etag"'})
resp = req.get_response(self.fake_app)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 304)
self.assertEquals(body, '')
def test_list_match(self):
# it works with lists of etags to match
req = swift.common.swob.Request.blank(
'/', headers={'If-None-Match': '"bert", "the-etag", "ernie"'})
resp = req.get_response(self.fake_app)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 304)
self.assertEquals(body, '')
def test_list_no_match(self):
# no matches --> whatever the original status was
req = swift.common.swob.Request.blank(
'/', headers={'If-None-Match': '"bert", "ernie"'})
resp = req.get_response(self.fake_app)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 200)
self.assertEquals(body, 'hi')
def test_match_star(self):
# "*" means match anything; see RFC 2616 section 14.24
req = swift.common.swob.Request.blank(
'/', headers={'If-None-Match': '*'})
resp = req.get_response(self.fake_app)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 304)
self.assertEquals(body, '')
class TestConditionalIfMatch(unittest.TestCase):
def fake_app(self, environ, start_response):
start_response('200 OK', [('Etag', 'the-etag')])
return ['hi']
def fake_start_response(*a, **kw):
pass
def test_simple_match(self):
# if etag matches, proceed as normal
req = swift.common.swob.Request.blank(
'/', headers={'If-Match': 'the-etag'})
resp = req.get_response(self.fake_app)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 200)
self.assertEquals(body, 'hi')
def test_quoted_simple_match(self):
# double quotes or not, doesn't matter
req = swift.common.swob.Request.blank(
'/', headers={'If-Match': '"the-etag"'})
resp = req.get_response(self.fake_app)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 200)
self.assertEquals(body, 'hi')
def test_no_match(self):
# no match --> 412
req = swift.common.swob.Request.blank(
'/', headers={'If-Match': 'not-the-etag'})
resp = req.get_response(self.fake_app)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 412)
self.assertEquals(body, '')
def test_match_star(self):
# "*" means match anything; see RFC 2616 section 14.24
req = swift.common.swob.Request.blank(
'/', headers={'If-Match': '*'})
resp = req.get_response(self.fake_app)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 200)
self.assertEquals(body, 'hi')
def test_match_star_on_404(self):
def fake_app_404(environ, start_response):
start_response('404 Not Found', [])
return ['hi']
req = swift.common.swob.Request.blank(
'/', headers={'If-Match': '*'})
resp = req.get_response(fake_app_404)
resp.conditional_response = True
body = ''.join(resp(req.environ, self.fake_start_response))
self.assertEquals(resp.status_int, 412)
self.assertEquals(body, '')
if __name__ == '__main__':
unittest.main()

4
test/unit/common/test_utils.py
View File

@ -120,6 +120,10 @@ class MockSys():
self.stdio_fds = [self.stdin.fileno(), self.stdout.fileno(),
self.stderr.fileno()]
@property
def version_info(self):
return sys.version_info
def reset_loggers():
if hasattr(utils.get_logger, 'handler4logger'):

18
test/unit/container/test_server.py
View File

@ -21,6 +21,7 @@ from contextlib import contextmanager
from shutil import rmtree
from StringIO import StringIO
from tempfile import mkdtemp
from test.unit import FakeLogger
from xml.dom import minidom
from eventlet import spawn, Timeout, listen
@ -2089,6 +2090,23 @@ class TestContainerController(unittest.TestCase):
response = self.controller.__call__(env, start_response)
self.assertEqual(response, answer)
def test_GET_log_requests_true(self):
self.controller.logger = FakeLogger()
self.controller.log_requests = True
req = Request.blank('/sda1/p/a/c', environ={'REQUEST_METHOD': 'GET'})
resp = req.get_response(self.controller)
self.assertEqual(resp.status_int, 404)
self.assertTrue(self.controller.logger.log_dict['info'])
def test_GET_log_requests_false(self):
self.controller.logger = FakeLogger()
self.controller.log_requests = False
req = Request.blank('/sda1/p/a/c', environ={'REQUEST_METHOD': 'GET'})
resp = req.get_response(self.controller)
self.assertEqual(resp.status_int, 404)
self.assertFalse(self.controller.logger.log_dict['info'])
if __name__ == '__main__':
unittest.main()

113
test/unit/obj/test_server.py
View File

@ -961,6 +961,65 @@ class TestObjectController(unittest.TestCase):
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 412)
def test_HEAD_if_match(self):
req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'PUT'},
headers={
'X-Timestamp': normalize_timestamp(time()),
'Content-Type': 'application/octet-stream',
'Content-Length': '4'})
req.body = 'test'
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 201)
etag = resp.etag
req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 200)
self.assertEquals(resp.etag, etag)
req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-Match': '*'})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 200)
self.assertEquals(resp.etag, etag)
req = Request.blank('/sda1/p/a/c/o2',
environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-Match': '*'})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 412)
req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-Match': '"%s"' % etag})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 200)
self.assertEquals(resp.etag, etag)
req = Request.blank(
'/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-Match': '"11111111111111111111111111111111"'})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 412)
req = Request.blank(
'/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'HEAD'},
headers={
'If-Match': '"11111111111111111111111111111111", "%s"' % etag})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 200)
req = Request.blank(
'/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'HEAD'},
headers={
'If-Match':
'"11111111111111111111111111111111", '
'"22222222222222222222222222222222"'})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 412)
def test_GET_if_none_match(self):
req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'PUT'},
headers={
@ -1011,6 +1070,60 @@ class TestObjectController(unittest.TestCase):
self.assertEquals(resp.status_int, 304)
self.assertEquals(resp.etag, etag)
def test_HEAD_if_none_match(self):
req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'PUT'},
headers={
'X-Timestamp': normalize_timestamp(time()),
'Content-Type': 'application/octet-stream',
'Content-Length': '4'})
req.body = 'test'
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 201)
etag = resp.etag
req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 200)
self.assertEquals(resp.etag, etag)
req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-None-Match': '*'})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 304)
self.assertEquals(resp.etag, etag)
req = Request.blank('/sda1/p/a/c/o2',
environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-None-Match': '*'})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 404)
req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-None-Match': '"%s"' % etag})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 304)
self.assertEquals(resp.etag, etag)
req = Request.blank(
'/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-None-Match': '"11111111111111111111111111111111"'})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 200)
self.assertEquals(resp.etag, etag)
req = Request.blank(
'/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'HEAD'},
headers={'If-None-Match':
'"11111111111111111111111111111111", '
'"%s"' % etag})
resp = req.get_response(self.object_controller)
self.assertEquals(resp.status_int, 304)
self.assertEquals(resp.etag, etag)
def test_GET_if_modified_since(self):
timestamp = normalize_timestamp(time())
req = Request.blank('/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'PUT'},