openstack
/
swift
Code Issues Proposed changes

Merge remote-tracking branch 'remotes/origin/master' into merge-master 

Conflicts resolved:
	.zuul.yaml
	requirements.txt
	swift/obj/replicator.py

Change-Id: I8c988890173f8e038f3285ac63537f5bf8698937
This commit is contained in:
Kota Tsuyuzaki 2019-04-25 17:18:43 +09:00
parent 648cf7a9aa 32bf43990c
commit d391957fd7
43 changed files with 407 additions and 325 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.zuul.yaml
View File

@ -331,6 +331,7 @@
irrelevant-files:
- ^(api-ref|doc|releasenotes)/.*$
- ^test/(functional|probe)/.*$
voting: false
- swift-tox-py36:
irrelevant-files:
- ^(api-ref|doc|releasenotes)/.*$
@ -340,7 +341,6 @@
irrelevant-files:
- ^(api-ref|doc|releasenotes)/.*$
- ^test/(functional|probe)/.*$
voting: false
- swift-tox-func:
irrelevant-files:
- ^(api-ref|doc|releasenotes)/.*$
@ -412,7 +412,7 @@
# long-running jobs, like probetests or (once they move to
# in-tree definitions) dsvm jobs.
- swift-tox-py27
- swift-tox-py35
- swift-tox-py37
- swift-tox-func
- swift-tox-func-encryption
- swift-tox-func-domain-remap-staticweb

2
CHANGELOG
View File

@ -1888,7 +1888,7 @@ swift (1.13.1, OpenStack Icehouse)
A new proxy config variable (strict_cors_mode, default to True)
has been added. Setting it to False keeps the old behavior. For
an overview of old versus new behavior, please see
https://review.openstack.org/#/c/69419/
https://review.opendev.org/#/c/69419/
* Invert the responsibility of the two instances of proxy-logging in
the proxy pipeline

2
CONTRIBUTING.rst
View File

@ -52,7 +52,7 @@ Reviewing Someone Else's Code
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All code reviews in OpenStack projects are done on
https://review.openstack.org/. Reviewing patches is one of the most effective
https://review.opendev.org/. Reviewing patches is one of the most effective
ways you can contribute to the community.
We've written REVIEW_GUIDELINES.rst (found in this source tree) to help you

5
api-ref/source/conf.py
View File

@ -162,6 +162,11 @@ try:
except OSError:
warnings.warn('Cannot get last updated time from git repository. '
'Not setting "html_last_updated_fmt".')
else:
if not isinstance(html_last_updated_fmt, str):
# for py3
html_last_updated_fmt = html_last_updated_fmt.decode('ascii')
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.

2
api-ref/source/parameters.yaml
View File

@ -1109,6 +1109,8 @@ multipart-manifest_put:
description: |
If you include the ``multipart-manifest=put`` query parameter, the object
is a static large object manifest and the body contains the manifest.
See `Static large objects <https://docs.openstack.org/swift/latest
/api/large_objects.html#static-large-objects>`_ for more information.
in: query
required: false
type: string

12
doc/source/deployment_guide.rst
View File

@ -2252,7 +2252,7 @@ For distros with more recent kernels (for example Ubuntu 12.04 Precise),
we recommend using the default settings (including the default inode size
of 256 bytes) when creating the file system::
mkfs.xfs /dev/sda1
mkfs.xfs -L D1 /dev/sda1
In the last couple of years, XFS has made great improvements in how inodes
are allocated and used. Using the default inode size no longer has an
@ -2262,7 +2262,7 @@ For distros with older kernels (for example Ubuntu 10.04 Lucid),
some settings can dramatically impact performance. We recommend the
following when creating the file system::
mkfs.xfs -i size=1024 /dev/sda1
mkfs.xfs -i size=1024 -L D1 /dev/sda1
Setting the inode size is important, as XFS stores xattr data in the inode.
If the metadata is too large to fit in the inode, a new extent is created,
@ -2272,15 +2272,15 @@ headroom.
The following example mount options are recommended when using XFS::
mount -t xfs -o noatime,nodiratime,nobarrier,logbufs=8 /dev/sda1 /srv/node/sda
mount -t xfs -o noatime,nodiratime,nobarrier,logbufs=8 -L D1 /srv/node/d1
We do not recommend running Swift on RAID, but if you are using
RAID it is also important to make sure that the proper sunit and swidth
settings get set so that XFS can make most efficient use of the RAID array.
For a standard Swift install, all data drives are mounted directly under
``/srv/node`` (as can be seen in the above example of mounting ``/dev/sda1`` as
``/srv/node/sda``). If you choose to mount the drives in another directory,
``/srv/node`` (as can be seen in the above example of mounting label ``D1``
as ``/srv/node/d1``). If you choose to mount the drives in another directory,
be sure to set the `devices` config option in all of the server configs to
point to the correct directory.
@ -2322,7 +2322,7 @@ The following settings should be in `/etc/sysctl.conf`::
# double amount of allowed conntrack
net.ipv4.netfilter.ip_conntrack_max = 262144
To load the updated sysctl settings, run ``sudo sysctl -p``
To load the updated sysctl settings, run ``sudo sysctl -p``.
A note about changing the TIME_WAIT values. By default the OS will hold
a port open for 60 seconds to ensure that any remaining packets can be

2
doc/source/first_contribution_swift.rst
View File

@ -117,7 +117,7 @@ Tracking your changes
---------------------
After proposing changes to Swift, you can track them at
https://review.openstack.org. After logging in, you will see a dashboard of
https://review.opendev.org. After logging in, you will see a dashboard of
"Outgoing reviews" for changes you have proposed, "Incoming reviews" for
changes you are reviewing, and "Recently closed" changes for which you were
either a reviewer or owner.

2
doc/source/install/controller-install-debian.rst
View File

@ -47,4 +47,4 @@ Install and configure components
.. code-block:: console
# curl -o /etc/swift/proxy-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample?h=stable/queens
# curl -o /etc/swift/proxy-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/proxy-server.conf-sample

2
doc/source/install/controller-install-rdo.rst
View File

@ -45,6 +45,6 @@ Install and configure components
.. code-block:: console
# curl -o /etc/swift/proxy-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample?h=stable/queens
# curl -o /etc/swift/proxy-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/proxy-server.conf-sample
3. .. include:: controller-include.txt

2
doc/source/install/controller-install-ubuntu.rst
View File

@ -47,6 +47,6 @@ Install and configure components
.. code-block:: console
# curl -o /etc/swift/proxy-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample?h=stable/queens
# curl -o /etc/swift/proxy-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/proxy-server.conf-sample
4. .. include:: controller-include.txt

2
doc/source/install/finalize-installation-rdo.rst
View File

@ -19,7 +19,7 @@ This section applies to Red Hat Enterprise Linux 7 and CentOS 7.
.. code-block:: console
# curl -o /etc/swift/swift.conf \
https://git.openstack.org/cgit/openstack/swift/plain/etc/swift.conf-sample?h=stable/queens
https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/swift.conf-sample
#. Edit the ``/etc/swift/swift.conf`` file and complete the following
actions:

2
doc/source/install/finalize-installation-ubuntu-debian.rst
View File

@ -19,7 +19,7 @@ This section applies to Ubuntu 14.04 (LTS) and Debian.
.. code-block:: console
# curl -o /etc/swift/swift.conf \
https://git.openstack.org/cgit/openstack/swift/plain/etc/swift.conf-sample?h=stable/queens
https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/swift.conf-sample
#. Edit the ``/etc/swift/swift.conf`` file and complete the following
actions:

6
doc/source/install/storage-install-rdo.rst
View File

@ -133,9 +133,9 @@ Install and configure components
.. code-block:: console
# curl -o /etc/swift/account-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/account-server.conf-sample?h=stable/queens
# curl -o /etc/swift/container-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/container-server.conf-sample?h=stable/queens
# curl -o /etc/swift/object-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/object-server.conf-sample?h=stable/queens
# curl -o /etc/swift/account-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/account-server.conf-sample
# curl -o /etc/swift/container-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/container-server.conf-sample
# curl -o /etc/swift/object-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/object-server.conf-sample
3. .. include:: storage-include1.txt
4. .. include:: storage-include2.txt

6
doc/source/install/storage-install-ubuntu-debian.rst
View File

@ -137,9 +137,9 @@ Install and configure components
.. code-block:: console
# curl -o /etc/swift/account-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/account-server.conf-sample?h=stable/queens
# curl -o /etc/swift/container-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/container-server.conf-sample?h=stable/queens
# curl -o /etc/swift/object-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/object-server.conf-sample?h=stable/queens
# curl -o /etc/swift/account-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/account-server.conf-sample
# curl -o /etc/swift/container-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/container-server.conf-sample
# curl -o /etc/swift/object-server.conf https://opendev.org/openstack/swift/raw/branch/stable/queens/etc/object-server.conf-sample
3. .. include:: storage-include1.txt
4. .. include:: storage-include2.txt

5
lower-constraints.txt
View File

@ -10,7 +10,7 @@ chardet==3.0.4
cliff==2.11.0
cmd2==0.8.1
coverage==3.6
cryptography==1.6
cryptography==1.8.2
debtcollector==1.19.0
dnspython==1.14.0
docutils==0.11
@ -40,7 +40,7 @@ mock==2.0
monotonic==1.4
msgpack==0.5.6
netaddr==0.7.19
netifaces==0.5
netifaces==0.8
nose==1.3.7
nosehtmloutput==0.0.3
nosexcover==1.0.10
@ -49,6 +49,7 @@ os-api-ref==1.0.0
os-testr==0.8.0
oslo.config==4.0.0
oslo.i18n==3.20.0
oslo.log==3.22.0
oslo.serialization==2.25.0
oslo.utils==3.36.0
PasteDeploy==1.3.3

14
requirements.txt
View File

@ -2,18 +2,18 @@
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
dnspython>=1.14.0;python_version=='2.7' # http://www.dnspython.org/LICENSE
eventlet>=0.17.4,!=0.23.0 # MIT
dnspython>=1.14.0;python_version=='2.7' # http://www.dnspython.org/LICENSE
eventlet>=0.17.4,!=0.23.0 # MIT
greenlet>=0.3.1
netifaces>=0.5,!=0.10.0,!=0.10.1
netifaces>=0.8,!=0.10.0,!=0.10.1
PasteDeploy>=1.3.3
lxml>=3.4.1
requests>=2.14.2 # Apache-2.0
requests>=2.14.2 # Apache-2.0
six>=1.9.0
xattr>=0.4
xattr>=0.4;sys_platform!='win32' # MIT
PyECLib>=1.3.1 # BSD
cryptography!=2.0,>=1.6 # BSD/Apache-2.0
ipaddress>=1.0.16;python_version<'3.3' # PSF
cryptography!=2.0,>=1.8.2 # BSD/Apache-2.0
ipaddress>=1.0.16;python_version<'3.3' # PSF
# grpcio will fail to work with eventlet starting with 1.3.5.
# see this for a similar issue with gevent: https://github.com/grpc/grpc/issues/4629 and https://github.com/gevent/gevent/issues/786
# don't use eventlet for the object-server ?

4
swift/common/middleware/acl.py
View File

@ -15,9 +15,7 @@
import json
import six
from six.moves.urllib.parse import unquote
from swift.common.utils import urlparse
from six.moves.urllib.parse import unquote, urlparse
def clean_acl(name, value):

51
swift/common/middleware/dlo.py
View File

@ -121,14 +121,14 @@ Here's an example using ``curl`` with tiny 1-byte segments::
import json
import six
from six.moves.urllib.parse import unquote
from hashlib import md5
from swift.common import constraints
from swift.common.exceptions import ListingIterError, SegmentError
from swift.common.http import is_success
from swift.common.swob import Request, Response, \
HTTPRequestedRangeNotSatisfiable, HTTPBadRequest, HTTPConflict
HTTPRequestedRangeNotSatisfiable, HTTPBadRequest, HTTPConflict, \
str_to_wsgi, wsgi_to_str, wsgi_quote, wsgi_unquote
from swift.common.utils import get_logger, \
RateLimitedIterator, quote, close_if_possible, closing_if_possible
from swift.common.request_helpers import SegmentedIterable
@ -143,9 +143,18 @@ class GetContext(WSGIContext):
def _get_container_listing(self, req, version, account, container,
prefix, marker=''):
'''
:param version: whatever
:param account: native
:param container: native
:param prefix: native
:param marker: native
'''
con_req = make_subrequest(
req.environ,
path=quote('/'.join(['', version, account, container])),
path=wsgi_quote('/'.join([
'', str_to_wsgi(version),
str_to_wsgi(account), str_to_wsgi(container)])),
method='GET',
headers={'x-auth-token': req.headers.get('x-auth-token')},
agent=('%(orig)s ' + 'DLO MultipartGET'), swift_source='DLO')
@ -156,14 +165,24 @@ class GetContext(WSGIContext):
con_resp = con_req.get_response(self.dlo.app)
if not is_success(con_resp.status_int):
if req.method == 'HEAD':
con_resp.body = ''
con_resp.body = b''
return con_resp, None
with closing_if_possible(con_resp.app_iter):
return None, json.loads(''.join(con_resp.app_iter))
return None, json.loads(b''.join(con_resp.app_iter))
def _segment_listing_iterator(self, req, version, account, container,
prefix, segments, first_byte=None,
last_byte=None):
'''
:param req: upstream request
:param version: native
:param account: native
:param container: native
:param prefix: native
:param segments: array of dicts, with native strings
:param first_byte: number
:param last_byte: number
'''
# It's sort of hokey that this thing takes in the first page of
# segments as an argument, but we need to compute the etag and content
# length from the first page, and it's better to have a hokey
@ -173,7 +192,6 @@ class GetContext(WSGIContext):
if last_byte is None:
last_byte = float("inf")
marker = ''
while True:
for segment in segments:
seg_length = int(segment['bytes'])
@ -188,7 +206,7 @@ class GetContext(WSGIContext):
break
seg_name = segment['name']
if isinstance(seg_name, six.text_type):
if six.PY2:
seg_name = seg_name.encode("utf-8")
# We deliberately omit the etag and size here;
@ -227,16 +245,18 @@ class GetContext(WSGIContext):
"Got status %d listing container /%s/%s" %
(error_response.status_int, account, container))
def get_or_head_response(self, req, x_object_manifest,
response_headers=None):
if response_headers is None:
response_headers = self._response_headers
def get_or_head_response(self, req, x_object_manifest):
'''
:param req: user's request
:param x_object_manifest: as unquoted, native string
'''
response_headers = self._response_headers
container, obj_prefix = x_object_manifest.split('/', 1)
container = unquote(container)
obj_prefix = unquote(obj_prefix)
version, account, _junk = req.split_path(2, 3, True)
version = wsgi_to_str(version)
account = wsgi_to_str(account)
error_response, segments = self._get_container_listing(
req, version, account, container, obj_prefix)
if error_response:
@ -311,7 +331,7 @@ class GetContext(WSGIContext):
if h.lower() != "etag"]
etag = md5()
for seg_dict in segments:
etag.update(seg_dict['hash'].strip('"'))
etag.update(seg_dict['hash'].strip('"').encode('utf8'))
response_headers.append(('Etag', '"%s"' % etag.hexdigest()))
app_iter = None
@ -353,7 +373,8 @@ class GetContext(WSGIContext):
for header, value in self._response_headers:
if (header.lower() == 'x-object-manifest'):
close_if_possible(resp_iter)
response = self.get_or_head_response(req, value)
response = self.get_or_head_response(
req, wsgi_to_str(wsgi_unquote(value)))
return response(req.environ, start_response)
# Not a dynamic large object manifest; just pass it through.
start_response(self._response_status,

6
swift/common/middleware/s3api/s3request.py
View File

@ -90,6 +90,8 @@ def _header_strip(value):
# behave as though it wasn't provided
return None
return stripped
_header_strip.re = re.compile('^[\x00-\x20]*|[\x00-\x20]*$')
@ -1427,8 +1429,10 @@ class S3Request(swob.Request):
else:
# otherwise we do naive HEAD request with the authentication
resp = self.get_response(app, 'HEAD', self.container_name, '')
headers = resp.sw_headers.copy()
headers.update(resp.sysmeta_headers)
return headers_to_container_info(
resp.sw_headers, resp.status_int) # pylint: disable-msg=E1101
headers, resp.status_int) # pylint: disable-msg=E1101
def gen_multipart_manifest_delete_query(self, app, obj=None):
if not self.allow_multipart_uploads:

14
swift/common/middleware/s3api/s3response.py
View File

@ -80,7 +80,7 @@ class S3Response(S3ResponseBase, swob.Response):
def __init__(self, *args, **kwargs):
swob.Response.__init__(self, *args, **kwargs)
sw_sysmeta_headers = swob.HeaderKeyDict()
s3_sysmeta_headers = swob.HeaderKeyDict()
sw_headers = swob.HeaderKeyDict()
headers = HeaderKeyDict()
self.is_slo = False
@ -103,12 +103,14 @@ class S3Response(S3ResponseBase, swob.Response):
key = sysmeta_prefix(_server_type) + \
key[len('x-%s-sysmeta-swift3-' % _server_type):]
if key not in sw_sysmeta_headers:
if key not in s3_sysmeta_headers:
# To avoid overwrite s3api sysmeta by older swift3
# sysmeta set the key only when the key does not exist
sw_sysmeta_headers[key] = val
s3_sysmeta_headers[key] = val
elif is_s3api_sysmeta(key, _server_type):
sw_sysmeta_headers[key] = val
s3_sysmeta_headers[key] = val
else:
sw_headers[key] = val
else:
sw_headers[key] = val
@ -132,7 +134,7 @@ class S3Response(S3ResponseBase, swob.Response):
self.is_slo = config_true_value(val)
# Check whether we stored the AWS-style etag on upload
override_etag = sw_sysmeta_headers.get(
override_etag = s3_sysmeta_headers.get(
sysmeta_header('object', 'etag'))
if override_etag is not None:
# Multipart uploads in AWS have ETags like
@ -153,7 +155,7 @@ class S3Response(S3ResponseBase, swob.Response):
# Used for pure swift header handling at the request layer
self.sw_headers = sw_headers
self.sysmeta_headers = sw_sysmeta_headers
self.sysmeta_headers = s3_sysmeta_headers
@classmethod
def from_swift_resp(cls, sw_resp):

9
swift/common/middleware/s3api/s3token.py
View File

@ -67,7 +67,7 @@ from six.moves import urllib
from swift.common.swob import Request, HTTPBadRequest, HTTPUnauthorized, \
HTTPException
from swift.common.utils import config_true_value, split_path, get_logger, \
cache_from_env
cache_from_env, append_underscore
from swift.common.wsgi import ConfigFileError
@ -149,7 +149,8 @@ class S3Token(object):
self._timeout = float(conf.get('http_timeout', '10.0'))
if not (0 < self._timeout <= 60):
raise ValueError('http_timeout must be between 0 and 60 seconds')
self._reseller_prefix = conf.get('reseller_prefix', 'AUTH_')
self._reseller_prefix = append_underscore(
conf.get('reseller_prefix', 'AUTH'))
self._delay_auth_decision = config_true_value(
conf.get('delay_auth_decision'))
@ -274,7 +275,9 @@ class S3Token(object):
string_to_sign = s3_auth_details['string_to_sign']
if isinstance(string_to_sign, six.text_type):
string_to_sign = string_to_sign.encode('utf-8')
token = base64.urlsafe_b64encode(string_to_sign).encode('ascii')
token = base64.urlsafe_b64encode(string_to_sign)
if isinstance(token, six.binary_type):
token = token.decode('ascii')
# NOTE(chmou): This is to handle the special case with nova
# when we have the option s3_affix_tenant. We will force it to

43
swift/common/middleware/staticweb.py
View File

@ -125,14 +125,17 @@ Example usage of this middleware via ``swift``:
import cgi
import json
import six
import time
from six.moves.urllib.parse import urlparse
from swift.common.utils import human_readable, split_path, config_true_value, \
quote, register_swift_info, get_logger, urlparse
quote, register_swift_info, get_logger
from swift.common.wsgi import make_env, WSGIContext
from swift.common.http import is_success, is_redirection, HTTP_NOT_FOUND
from swift.common.swob import Response, HTTPMovedPermanently, HTTPNotFound, \
Request
Request, wsgi_quote, wsgi_to_str
from swift.proxy.controllers.base import get_container_info
@ -145,6 +148,12 @@ class _StaticWebContext(WSGIContext):
that might need to be handled to make keeping contextual
information about the request a bit simpler than storing it in
the WSGI env.
:param staticweb: The staticweb middleware object in use.
:param version: A WSGI string representation of the swift api version.
:param account: A WSGI string representation of the account name.
:param container: A WSGI string representation of the container name.
:param obj: A WSGI string representation of the object name.
"""
def __init__(self, staticweb, version, account, container, obj):
@ -223,9 +232,9 @@ class _StaticWebContext(WSGIContext):
:param start_response: The original WSGI start_response hook.
:param prefix: Any prefix desired for the container listing.
"""
label = env['PATH_INFO']
label = wsgi_to_str(env['PATH_INFO'])
if self._listings_label:
groups = env['PATH_INFO'].split('/')
groups = wsgi_to_str(env['PATH_INFO']).split('/')
label = '{0}/{1}'.format(self._listings_label,
'/'.join(groups[4:]))
@ -262,14 +271,14 @@ class _StaticWebContext(WSGIContext):
self.agent, swift_source='SW')
tmp_env['QUERY_STRING'] = 'delimiter=/'
if prefix:
tmp_env['QUERY_STRING'] += '&prefix=%s' % quote(prefix)
tmp_env['QUERY_STRING'] += '&prefix=%s' % wsgi_quote(prefix)
else:
prefix = ''
resp = self._app_call(tmp_env)
if not is_success(self._get_status_int()):
return self._error_response(resp, env, start_response)
listing = None
body = ''.join(resp)
body = b''.join(resp)
if body:
listing = json.loads(body)
if not listing:
@ -280,7 +289,8 @@ class _StaticWebContext(WSGIContext):
'Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n' \
'<html>\n' \
' <head>\n' \
' <title>Listing of %s</title>\n' % cgi.escape(label)
' <title>Listing of %s</title>\n' % \
cgi.escape(label)
if self._listings_css:
body += ' <link rel="stylesheet" type="text/css" ' \
'href="%s" />\n' % (self._build_css_path(prefix))
@ -308,7 +318,8 @@ class _StaticWebContext(WSGIContext):
' </tr>\n'
for item in listing:
if 'subdir' in item:
subdir = item['subdir'].encode("utf-8")
subdir = item['subdir'] if six.PY3 else \
item['subdir'].encode('utf-8')
if prefix:
subdir = subdir[len(prefix):]
body += ' <tr class="item subdir">\n' \
@ -319,13 +330,16 @@ class _StaticWebContext(WSGIContext):
(quote(subdir), cgi.escape(subdir))
for item in listing:
if 'name' in item:
name = item['name'].encode("utf-8")
name = item['name'] if six.PY3 else \
item['name'].encode('utf-8')
if prefix:
name = name[len(prefix):]
content_type = item['content_type'].encode("utf-8")
content_type = item['content_type'] if six.PY3 else \
item['content_type'].encode('utf-8')
bytes = human_readable(item['bytes'])
last_modified = (
cgi.escape(item['last_modified'].encode("utf-8")).
cgi.escape(item['last_modified'] if six.PY3 else
item['last_modified'].encode('utf-8')).
split('.')[0].replace('T', ' '))
body += ' <tr class="item %s">\n' \
' <td class="colname"><a href="%s">%s</a></td>\n' \
@ -362,7 +376,8 @@ class _StaticWebContext(WSGIContext):
env['wsgi.url_scheme'] = self.url_scheme
if self.url_host:
env['HTTP_HOST'] = self.url_host
resp = HTTPMovedPermanently(location=(env['PATH_INFO'] + '/'))
resp = HTTPMovedPermanently(
location=wsgi_quote(env['PATH_INFO'] + '/'))
return resp(env, start_response)
def handle_container(self, env, start_response):
@ -466,9 +481,9 @@ class _StaticWebContext(WSGIContext):
self.version, self.account, self.container),
self.agent, swift_source='SW')
tmp_env['QUERY_STRING'] = 'limit=1&delimiter=/&prefix=%s' % (
quote(self.obj + '/'), )
quote(wsgi_to_str(self.obj) + '/'), )
resp = self._app_call(tmp_env)
body = ''.join(resp)
body = b''.join(resp)
if not is_success(self._get_status_int()) or not body or \
not json.loads(body):
resp = HTTPNotFound()(env, self._start_response)

13
swift/common/middleware/symlink.py
View File

@ -161,7 +161,7 @@ from cgi import parse_header
from six.moves.urllib.parse import unquote
from swift.common.utils import get_logger, register_swift_info, split_path, \
MD5_OF_EMPTY_STRING, closing_if_possible
MD5_OF_EMPTY_STRING, closing_if_possible, quote
from swift.common.constraints import check_account_format
from swift.common.wsgi import WSGIContext, make_subrequest
from swift.common.request_helpers import get_sys_meta_prefix, \
@ -208,6 +208,7 @@ def _check_symlink_header(req):
req, TGT_OBJ_SYMLINK_HDR, 2,
'X-Symlink-Target header must be of the '
'form <container name>/<object name>')
req.headers[TGT_OBJ_SYMLINK_HDR] = quote('%s/%s' % (container, obj))
# Check account format if it exists
account = check_account_format(
@ -217,7 +218,9 @@ def _check_symlink_header(req):
# Extract request path
_junk, req_acc, req_cont, req_obj = req.split_path(4, 4, True)
if not account:
if account:
req.headers[TGT_ACCT_SYMLINK_HDR] = quote(account)
else:
account = req_acc
# Check if symlink targets the symlink itself or not
@ -378,9 +381,9 @@ class SymlinkObjectContext(WSGIContext):
:returns: new request for target path if it's symlink otherwise
None
"""
version, account, _junk = split_path(req.path, 2, 3, True)
version, account, _junk = req.split_path(2, 3, True)
account = self._response_header_value(
TGT_ACCT_SYSMETA_SYMLINK_HDR) or account
TGT_ACCT_SYSMETA_SYMLINK_HDR) or quote(account)
target_path = os.path.join(
'/', version, account,
symlink_target.lstrip('/'))
@ -485,7 +488,7 @@ class SymlinkObjectContext(WSGIContext):
if tgt_co:
version, account, _junk = req.split_path(2, 3, True)
target_acc = self._response_header_value(
TGT_ACCT_SYSMETA_SYMLINK_HDR) or account
TGT_ACCT_SYSMETA_SYMLINK_HDR) or quote(account)
location_hdr = os.path.join(
'/', version, target_acc, tgt_co)
req.environ['swift.leave_relative_location'] = True

37
swift/common/utils.py
View File

@ -75,9 +75,8 @@ from six.moves import cPickle as pickle
from six.moves.configparser import (ConfigParser, NoSectionError,
NoOptionError, RawConfigParser)
from six.moves import range, http_client
from six.moves.urllib.parse import ParseResult
from six.moves.urllib.parse import quote as _quote
from six.moves.urllib.parse import urlparse as stdlib_urlparse
from six.moves.urllib.parse import urlparse
from swift import gettext_ as _
import swift.common.exceptions
@ -2265,7 +2264,7 @@ def get_hub():
Note about epoll:
Review: https://review.openstack.org/#/c/18806/
Review: https://review.opendev.org/#/c/18806/
There was a problem where once out of every 30 quadrillion
connections, a coroutine wouldn't wake up when the client
@ -3244,38 +3243,6 @@ class StreamingPile(GreenAsyncPile):
self.pool.__exit__(type, value, traceback)
class ModifiedParseResult(ParseResult):
"""Parse results class for urlparse."""
@property
def hostname(self):
netloc = self.netloc.split('@', 1)[-1]
if netloc.startswith('['):
return netloc[1:].split(']')[0]
elif ':' in netloc:
return netloc.rsplit(':')[0]
return netloc
@property
def port(self):
netloc = self.netloc.split('@', 1)[-1]
if netloc.startswith('['):
netloc = netloc.rsplit(']')[1]
if ':' in netloc:
return int(netloc.rsplit(':')[1])
return None
def urlparse(url):
"""
urlparse augmentation.
This is necessary because urlparse can't handle RFC 2732 URLs.
:param url: URL to parse.
"""
return ModifiedParseResult(*stdlib_urlparse(url))
def validate_sync_to(value, allowed_sync_hosts, realms_conf):
"""
Validates an X-Container-Sync-To header value, returning the

1
swift/common/wsgi.py
View File

@ -430,6 +430,7 @@ class SwiftHttpProtocol(wsgi.HttpProtocol):
# for py3:
def get_default_type(self):
'''If the client didn't provide a content type, leave it blank.'''
return ''

5
swift/container/sharder.py
View File

@ -365,8 +365,9 @@ class ContainerSharder(ContainerReplicator):
'Swift Container Sharder',
request_tries,
allow_modify_pipeline=False)
except IOError as err:
if err.errno != errno.ENOENT:
except (OSError, IOError) as err:
if err.errno != errno.ENOENT and \
not str(err).endswith(' not found'):
raise
raise SystemExit(
'Unable to load internal client from config: %r (%s)' %

8
swift/container/sync.py
View File

@ -23,6 +23,7 @@ from random import choice, random
from struct import unpack_from
from eventlet import sleep, Timeout
from six.moves.urllib.parse import urlparse
import swift.common.db
from swift.common.db import DatabaseConnectionError
@ -37,7 +38,7 @@ from swift.common.ring import Ring
from swift.common.ring.utils import is_local_device
from swift.common.utils import (
clean_content_type, config_true_value,
FileLikeIter, get_logger, hash_path, quote, urlparse, validate_sync_to,
FileLikeIter, get_logger, hash_path, quote, validate_sync_to,
whataremyips, Timestamp, decode_timestamps)
from swift.common.daemon import Daemon
from swift.common.http import HTTP_UNAUTHORIZED, HTTP_NOT_FOUND
@ -238,8 +239,9 @@ class ContainerSync(Daemon):
try:
self.swift = InternalClient(
internal_client_conf, 'Swift Container Sync', request_tries)
except IOError as err:
if err.errno != errno.ENOENT:
except (OSError, IOError) as err:
if err.errno != errno.ENOENT and \
not str(err).endswith(' not found'):
raise
raise SystemExit(
_('Unable to load internal client from config: '

5
swift/obj/replicator.py
View File

@ -818,9 +818,10 @@ class ObjectReplicator(Daemon):
except Exception:
self.logger.exception('ERROR creating %s' % obj_path)
continue
for partition in df_mgr.listdir(obj_path):
if (override_partitions is not None
and partition not in override_partitions):
if (override_partitions is not None and partition.isdigit()
and int(partition) not in override_partitions):
continue
if (partition.startswith('auditor_status_') and

13
test/functional/s3api/s3_test_client.py
View File

@ -16,8 +16,10 @@
import os
import test.functional as tf
from boto.s3.connection import S3Connection, OrdinaryCallingFormat, \
BotoClientError, S3ResponseError
S3ResponseError
import six
import sys
import traceback
RETRY_COUNT = 3
@ -92,11 +94,12 @@ class Connection(object):
# 404 means NoSuchBucket, NoSuchKey, or NoSuchUpload
if e.status != 404:
raise
except (BotoClientError, S3ResponseError) as e:
exceptions.append(e)
except Exception as e:
exceptions.append(''.join(
traceback.format_exception(*sys.exc_info())))
if exceptions:
# raise the first exception
raise exceptions.pop(0)
exceptions.insert(0, 'Too many errors to continue:')
raise Exception('\n========\n'.join(exceptions))
def make_request(self, method, bucket='', obj='', headers=None, body='',
query=None):

73
test/functional/test_staticweb.py
View File

@ -16,6 +16,8 @@
import functools
from unittest2 import SkipTest
from six.moves.urllib.parse import unquote
from swift.common.utils import quote
import test.functional as tf
from test.functional import cluster_info
from test.functional.tests import Utils, Base, BaseEnv
@ -74,8 +76,8 @@ class TestStaticWebEnv(BaseEnv):
'listings_css',
'dir/',
'dir/obj',
'dir/subdir/',
'dir/subdir/obj']
'dir/some sub%dir/',
'dir/some sub%dir/obj']
cls.objects = {}
for item in sorted(objects):
@ -168,12 +170,12 @@ class TestStaticWeb(Base):
def _test_redirect_slash_direct(self, anonymous):
host = self.env.account.conn.storage_netloc
path = '%s/%s' % (self.env.account.conn.storage_path,
self.env.container.name)
quote(self.env.container.name))
self._test_redirect_with_slash(host, path, anonymous=anonymous)
path = '%s/%s/%s' % (self.env.account.conn.storage_path,
self.env.container.name,
self.env.objects['dir/'].name)
quote(self.env.container.name),
quote(self.env.objects['dir/'].name))
self._test_redirect_with_slash(host, path, anonymous=anonymous)
def test_redirect_slash_auth_direct(self):
@ -185,10 +187,10 @@ class TestStaticWeb(Base):
@requires_domain_remap
def _test_redirect_slash_remap_acct(self, anonymous):
host = self.domain_remap_acct
path = '/%s' % self.env.container.name
path = '/%s' % quote(self.env.container.name)
self._test_redirect_with_slash(host, path, anonymous=anonymous)
path = '/%s/%s' % (self.env.container.name,
path = '/%s/%s' % (quote(self.env.container.name),
self.env.objects['dir/'].name)
self._test_redirect_with_slash(host, path, anonymous=anonymous)
@ -229,13 +231,14 @@ class TestStaticWeb(Base):
self._set_staticweb_headers(listings=True,
listings_css=(css is not None))
if title is None:
title = path
title = unquote(path)
expected_in = ['Listing of %s' % title] + [
'<a href="{0}">{0}</a>'.format(link) for link in links]
'<a href="{0}">{1}</a>'.format(quote(link), link)
for link in links]
expected_not_in = notins
if css:
expected_in.append('<link rel="stylesheet" type="text/css" '
'href="%s" />' % css)
'href="%s" />' % quote(css))
self._test_get_path(host, path, anonymous=anonymous,
expected_in=expected_in,
expected_not_in=expected_not_in)
@ -244,7 +247,7 @@ class TestStaticWeb(Base):
objects = self.env.objects
host = self.env.account.conn.storage_netloc
path = '%s/%s/' % (self.env.account.conn.storage_path,
self.env.container.name)
quote(self.env.container.name))
css = objects['listings_css'].name if listings_css else None
self._test_listing(host, path, anonymous=True, css=css,
links=[objects['index'].name,
@ -252,15 +255,15 @@ class TestStaticWeb(Base):
notins=[objects['dir/obj'].name])
path = '%s/%s/%s/' % (self.env.account.conn.storage_path,
self.env.container.name,
objects['dir/'].name)
quote(self.env.container.name),
quote(objects['dir/'].name))
css = '../%s' % objects['listings_css'].name if listings_css else None
self._test_listing(host, path, anonymous=anonymous, css=css,
links=[objects['dir/obj'].name.split('/')[-1],
objects['dir/subdir/'].name.split('/')[-1]
+ '/'],
notins=[objects['index'].name,
objects['dir/subdir/obj'].name])
self._test_listing(
host, path, anonymous=anonymous, css=css,
links=[objects['dir/obj'].name.split('/')[-1],
objects['dir/some sub%dir/'].name.split('/')[-1] + '/'],
notins=[objects['index'].name,
objects['dir/some sub%dir/obj'].name])
def test_listing_auth_direct_without_css(self):
self._test_listing_direct(False, False)
@ -293,13 +296,12 @@ class TestStaticWeb(Base):
title = '%s/%s/%s/' % (self.env.account.conn.storage_path,
self.env.container.name,
objects['dir/'])
self._test_listing(host, path, title=title, anonymous=anonymous,
css=css,
links=[objects['dir/obj'].name.split('/')[-1],
objects['dir/subdir/'].name.split('/')[-1]
+ '/'],
notins=[objects['index'].name,
objects['dir/subdir/obj'].name])
self._test_listing(
host, path, title=title, anonymous=anonymous, css=css,
links=[objects['dir/obj'].name.split('/')[-1],
objects['dir/some sub%dir/'].name.split('/')[-1] + '/'],
notins=[objects['index'].name,
objects['dir/some sub%dir/obj'].name])
def test_listing_auth_remap_acct_without_css(self):
self._test_listing_remap_acct(False, False)
@ -332,13 +334,12 @@ class TestStaticWeb(Base):
title = '%s/%s/%s/' % (self.env.account.conn.storage_path,
self.env.container.name,
objects['dir/'])
self._test_listing(host, path, title=title, anonymous=anonymous,
css=css,
links=[objects['dir/obj'].name.split('/')[-1],
objects['dir/subdir/'].name.split('/')[-1]
+ '/'],
notins=[objects['index'].name,
objects['dir/subdir/obj'].name])
self._test_listing(
host, path, title=title, anonymous=anonymous, css=css,
links=[objects['dir/obj'].name.split('/')[-1],
objects['dir/some sub%dir/'].name.split('/')[-1] + '/'],
notins=[objects['index'].name,
objects['dir/some sub%dir/obj'].name])
def test_listing_auth_remap_cont_without_css(self):
self._test_listing_remap_cont(False, False)
@ -369,12 +370,12 @@ class TestStaticWeb(Base):
objects = self.env.objects
host = self.env.account.conn.storage_netloc
path = '%s/%s/' % (self.env.account.conn.storage_path,
self.env.container.name)
quote(self.env.container.name))
self._test_index(host, path, anonymous=anonymous)
path = '%s/%s/%s/' % (self.env.account.conn.storage_path,
self.env.container.name,
objects['dir/'].name)
quote(self.env.container.name),
quote(objects['dir/'].name))
self._test_index(host, path, anonymous=anonymous, expected_status=404)
def test_index_auth_direct(self):

36
test/functional/test_symlink.py
View File

@ -270,23 +270,45 @@ class TestSymlink(Base):
target_obj = 'dealde%2Fl04 011e%204c8df/flash.png'
link_obj = uuid4().hex
# Now let's write a new target object and symlink will be able to
# return object
# create target using unnormalized path
resp = retry(
self._make_request, method='PUT', container=self.env.tgt_cont,
obj=target_obj, body=TARGET_BODY)
self.assertEqual(resp.status, 201)
# you can get it using either name
resp = retry(
self._make_request, method='GET', container=self.env.tgt_cont,
obj=target_obj)
self.assertEqual(resp.status, 200)
self.assertEqual(resp.content, TARGET_BODY)
normalized_quoted_obj = 'dealde/l04%20011e%204c8df/flash.png'
self.assertEqual(normalized_quoted_obj, urllib.parse.quote(
urllib.parse.unquote(target_obj)))
resp = retry(
self._make_request, method='GET', container=self.env.tgt_cont,
obj=normalized_quoted_obj)
self.assertEqual(resp.status, 200)
self.assertEqual(resp.content, TARGET_BODY)
# PUT symlink
# create a symlink using the un-normalized target path
self._test_put_symlink(link_cont=self.env.link_cont, link_obj=link_obj,
tgt_cont=self.env.tgt_cont,
tgt_obj=target_obj)
# and it's normalized
self._assertSymlink(
self.env.link_cont, link_obj,
expected_content_location="%s/%s" % (self.env.tgt_cont,
target_obj))
expected_content_location='%s/%s' % (
self.env.tgt_cont, normalized_quoted_obj))
# create a symlink using the normalized target path
self._test_put_symlink(link_cont=self.env.link_cont, link_obj=link_obj,
tgt_cont=self.env.tgt_cont,
tgt_obj=normalized_quoted_obj)
# and it's ALSO normalized
self._assertSymlink(
self.env.link_cont, link_obj,
expected_content_location='%s/%s' % (
self.env.tgt_cont, normalized_quoted_obj))
def test_symlink_put_head_get(self):
link_obj = uuid4().hex

9
test/functional/test_versioned_writes.py
View File

@ -18,7 +18,7 @@ from copy import deepcopy
import json
import time
import unittest2
from six.moves.urllib.parse import quote
from six.moves.urllib.parse import quote, unquote
import test.functional as tf
@ -652,7 +652,7 @@ class TestObjectVersioning(Base):
tgt_b.write("bbbbb")
symlink_name = Utils.create_name()
sym_tgt_header = '%s/%s' % (container.name, tgt_a_name)
sym_tgt_header = quote(unquote('%s/%s' % (container.name, tgt_a_name)))
sym_headers_a = {'X-Symlink-Target': sym_tgt_header}
symlink = container.file(symlink_name)
symlink.write("", hdrs=sym_headers_a)
@ -684,8 +684,9 @@ class TestObjectVersioning(Base):
sym_info = symlink.info(parms={'symlink': 'get'})
self.assertEqual("aaaaa", symlink.read())
self.assertEqual(MD5_OF_EMPTY_STRING, sym_info['etag'])
self.assertEqual('%s/%s' % (self.env.container.name, target.name),
sym_info['x_symlink_target'])
self.assertEqual(
quote(unquote('%s/%s' % (self.env.container.name, target.name))),
sym_info['x_symlink_target'])
def _setup_symlink(self):
target = self.env.container.file('target-object')

8
test/unit/common/middleware/s3api/test_s3request.py
View File

@ -302,9 +302,14 @@ class TestRequest(S3ApiTestCase):
self.assertTrue(sw_req.environ['swift.proxy_access_log_made'])
def test_get_container_info(self):
s3api_acl = '{"Owner":"owner","Grant":'\
'[{"Grantee":"owner","Permission":"FULL_CONTROL"}]}'
self.swift.register('HEAD', '/v1/AUTH_test/bucket', HTTPNoContent,
{'x-container-read': 'foo',
'X-container-object-count': 5,
'x-container-sysmeta-versions-location':
'bucket2',
'x-container-sysmeta-s3api-acl': s3api_acl,
'X-container-meta-foo': 'bar'}, None)
req = Request.blank('/bucket', environ={'REQUEST_METHOD': 'GET'},
headers={'Authorization': 'AWS test:tester:hmac',
@ -316,6 +321,9 @@ class TestRequest(S3ApiTestCase):
self.assertEqual(204, info['status']) # sanity
self.assertEqual('foo', info['read_acl']) # sanity
self.assertEqual('5', info['object_count']) # sanity
self.assertEqual(
'bucket2', info['sysmeta']['versions-location']) # sanity
self.assertEqual(s3api_acl, info['sysmeta']['s3api-acl']) # sanity
self.assertEqual({'foo': 'bar'}, info['meta']) # sanity
with patch(
'swift.common.middleware.s3api.s3request.get_container_info',

2
test/unit/common/middleware/s3api/test_s3response.py
View File

@ -54,6 +54,8 @@ class TestResponse(unittest.TestCase):
expected_headers = HeaderKeyDict(
{sysmeta_prefix(_server_type) + 'test': 'ok'})
self.assertEqual(expected_headers, s3resp.sysmeta_headers)
self.assertIn('x-%s-sysmeta-test-s3api' % _server_type,
s3resp.sw_headers)
def test_response_s3api_sysmeta_from_swift3_sysmeta(self):
for _server_type in ('object', 'container'):

17
test/unit/common/middleware/s3api/test_s3token.py
View File

@ -372,6 +372,15 @@ class S3TokenMiddlewareTestGood(S3TokenMiddlewareTestBase):
middleware = s3token.filter_factory(config)(self.app)
self.assertIs('false_ind', middleware._verify)
def test_reseller_prefix(self):
def do_test(conf, expected):
conf.update(self.conf)
middleware = s3token.filter_factory(conf)(self.app)
self.assertEqual(expected, middleware._reseller_prefix)
do_test({}, 'AUTH_')
do_test({'reseller_prefix': 'KEY_'}, 'KEY_')
do_test({'reseller_prefix': 'KEY'}, 'KEY_')
def test_auth_uris(self):
for conf, expected in [
({'auth_uri': 'https://example.com/v2.0'},
@ -456,12 +465,12 @@ class S3TokenMiddlewareTestGood(S3TokenMiddlewareTestBase):
with self.assertRaises(ConfigFileError) as cm:
s3token.filter_factory({'auth_uri': auth_uri})(self.app)
self.assertEqual('Invalid auth_uri; must include scheme and host',
cm.exception.message)
cm.exception.args[0])
with self.assertRaises(ConfigFileError) as cm:
s3token.filter_factory({
'auth_uri': 'nonhttp://example.com'})(self.app)
self.assertEqual('Invalid auth_uri; scheme must be http or https',
cm.exception.message)
cm.exception.args[0])
for auth_uri in [
'http://user@example.com/',
'http://example.com/?with=query',
@ -469,7 +478,7 @@ class S3TokenMiddlewareTestGood(S3TokenMiddlewareTestBase):
with self.assertRaises(ConfigFileError) as cm:
s3token.filter_factory({'auth_uri': auth_uri})(self.app)
self.assertEqual('Invalid auth_uri; must not include username, '
'query, or fragment', cm.exception.message)
'query, or fragment', cm.exception.args[0])
def test_unicode_path(self):
url = u'/v1/AUTH_cfa/c/euro\u20ac'.encode('utf8')
@ -568,7 +577,7 @@ class S3TokenMiddlewareTestGood(S3TokenMiddlewareTestBase):
MOCK_REQUEST.return_value = TestResponse({
'status_code': 201,
'text': json.dumps(GOOD_RESPONSE_V2)})
'text': json.dumps(GOOD_RESPONSE_V2).encode('ascii')})
req = Request.blank('/v1/AUTH_cfa/c/o')
req.environ['s3api.auth_details'] = {

117
test/unit/common/middleware/test_dlo.py
View File

@ -34,6 +34,8 @@ LIMIT = 'swift.common.constraints.CONTAINER_LISTING_LIMIT'
def md5hex(s):
if not isinstance(s, bytes):
s = s.encode('utf-8')
return hashlib.md5(s).hexdigest()
@ -52,7 +54,7 @@ class DloTestCase(unittest.TestCase):
headers[0] = h
body_iter = app(req.environ, start_response)
body = ''
body = b''
# appease the close-checker
with closing_if_possible(body_iter):
for chunk in body_iter:
@ -69,36 +71,36 @@ class DloTestCase(unittest.TestCase):
self.app.register(
'GET', '/v1/AUTH_test/c/seg_01',
swob.HTTPOk, {'Content-Length': '5', 'Etag': md5hex("aaaaa")},
'aaaaa')
b'aaaaa')
self.app.register(
'GET', '/v1/AUTH_test/c/seg_02',
swob.HTTPOk, {'Content-Length': '5', 'Etag': md5hex("bbbbb")},
'bbbbb')
b'bbbbb')
self.app.register(
'GET', '/v1/AUTH_test/c/seg_03',
swob.HTTPOk, {'Content-Length': '5', 'Etag': md5hex("ccccc")},
'ccccc')
b'ccccc')
self.app.register(
'GET', '/v1/AUTH_test/c/seg_04',
swob.HTTPOk, {'Content-Length': '5', 'Etag': md5hex("ddddd")},
'ddddd')
b'ddddd')
self.app.register(
'GET', '/v1/AUTH_test/c/seg_05',
swob.HTTPOk, {'Content-Length': '5', 'Etag': md5hex("eeeee")},
'eeeee')
b'eeeee')
# an unrelated object (not seg*) to test the prefix matching
self.app.register(
'GET', '/v1/AUTH_test/c/catpicture.jpg',
swob.HTTPOk, {'Content-Length': '9',
'Etag': md5hex("meow meow meow meow")},
'meow meow meow meow')
b'meow meow meow meow')
self.app.register(
'GET', '/v1/AUTH_test/mancon/manifest',
swob.HTTPOk, {'Content-Length': '17', 'Etag': 'manifest-etag',
'X-Object-Manifest': 'c/seg'},
'manifest-contents')
b'manifest-contents')
lm = '2013-11-22T02:42:13.781760'
ct = 'application/octet-stream'
@ -120,11 +122,11 @@ class DloTestCase(unittest.TestCase):
self.app.register(
'GET', '/v1/AUTH_test/c',
swob.HTTPOk, {'Content-Type': 'application/json; charset=utf-8'},
json.dumps(full_container_listing))
json.dumps(full_container_listing).encode('ascii'))
self.app.register(
'GET', '/v1/AUTH_test/c?prefix=seg',
swob.HTTPOk, {'Content-Type': 'application/json; charset=utf-8'},
json.dumps(segs))
json.dumps(segs).encode('ascii'))
# This is to let us test multi-page container listings; we use the
# trailing underscore to send small (pagesize=3) listings.
@ -135,26 +137,26 @@ class DloTestCase(unittest.TestCase):
'GET', '/v1/AUTH_test/mancon/manifest-many-segments',
swob.HTTPOk, {'Content-Length': '7', 'Etag': 'etag-manyseg',
'X-Object-Manifest': 'c/seg_'},
'manyseg')
b'manyseg')
self.app.register(
'GET', '/v1/AUTH_test/c?prefix=seg_',
swob.HTTPOk, {'Content-Type': 'application/json; charset=utf-8'},
json.dumps(segs[:3]))
json.dumps(segs[:3]).encode('ascii'))
self.app.register(
'GET', '/v1/AUTH_test/c?prefix=seg_&marker=seg_03',
swob.HTTPOk, {'Content-Type': 'application/json; charset=utf-8'},
json.dumps(segs[3:]))
json.dumps(segs[3:]).encode('ascii'))
# Here's a manifest with 0 segments
self.app.register(
'GET', '/v1/AUTH_test/mancon/manifest-no-segments',
swob.HTTPOk, {'Content-Length': '7', 'Etag': 'noseg',
'X-Object-Manifest': 'c/noseg_'},
'noseg')
b'noseg')
self.app.register(
'GET', '/v1/AUTH_test/c?prefix=noseg_',
swob.HTTPOk, {'Content-Type': 'application/json; charset=utf-8'},
json.dumps([]))
json.dumps([]).encode('ascii'))
class TestDloPutManifest(DloTestCase):
@ -284,7 +286,7 @@ class TestDloGetManifest(DloTestCase):
headers = HeaderKeyDict(headers)
self.assertEqual(headers["Etag"], expected_etag)
self.assertEqual(headers["Content-Length"], "25")
self.assertEqual(body, 'aaaaabbbbbcccccdddddeeeee')
self.assertEqual(body, b'aaaaabbbbbcccccdddddeeeee')
for _, _, hdrs in self.app.calls_with_headers[1:]:
ua = hdrs.get("User-Agent", "")
@ -302,7 +304,7 @@ class TestDloGetManifest(DloTestCase):
req = swob.Request.blank('/v1/AUTH_test/c/catpicture.jpg',
environ={'REQUEST_METHOD': 'GET'})
status, headers, body = self.call_dlo(req)
self.assertEqual(body, "meow meow meow meow")
self.assertEqual(body, b"meow meow meow meow")
def test_get_non_object_passthrough(self):
self.app.register('GET', '/info', swob.HTTPOk,
@ -311,7 +313,7 @@ class TestDloGetManifest(DloTestCase):
environ={'REQUEST_METHOD': 'GET'})
status, headers, body = self.call_dlo(req)
self.assertEqual(status, '200 OK')
self.assertEqual(body, 'useful stuff here')
self.assertEqual(body, b'useful stuff here')
self.assertEqual(self.app.call_count, 1)
def test_get_manifest_passthrough(self):
@ -328,7 +330,7 @@ class TestDloGetManifest(DloTestCase):
status, headers, body = self.call_dlo(req)
headers = HeaderKeyDict(headers)
self.assertEqual(headers["Etag"], "manifest-etag")
self.assertEqual(body, "manifest-contents")
self.assertEqual(body, b'manifest-contents')
def test_error_passthrough(self):
self.app.register(
@ -347,7 +349,7 @@ class TestDloGetManifest(DloTestCase):
headers = HeaderKeyDict(headers)
self.assertEqual(status, "206 Partial Content")
self.assertEqual(headers["Content-Length"], "10")
self.assertEqual(body, "bbcccccddd")
self.assertEqual(body, b'bbcccccddd')
expected_etag = '"%s"' % md5hex(
md5hex("aaaaa") + md5hex("bbbbb") + md5hex("ccccc") +
md5hex("ddddd") + md5hex("eeeee"))
@ -361,7 +363,7 @@ class TestDloGetManifest(DloTestCase):
headers = HeaderKeyDict(headers)
self.assertEqual(status, "206 Partial Content")
self.assertEqual(headers["Content-Length"], "10")
self.assertEqual(body, "cccccddddd")
self.assertEqual(body, b'cccccddddd')
def test_get_range_first_byte(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
@ -371,7 +373,7 @@ class TestDloGetManifest(DloTestCase):
headers = HeaderKeyDict(headers)
self.assertEqual(status, "206 Partial Content")
self.assertEqual(headers["Content-Length"], "1")
self.assertEqual(body, "a")
self.assertEqual(body, b'a')
def test_get_range_last_byte(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
@ -381,7 +383,7 @@ class TestDloGetManifest(DloTestCase):
headers = HeaderKeyDict(headers)
self.assertEqual(status, "206 Partial Content")
self.assertEqual(headers["Content-Length"], "1")
self.assertEqual(body, "e")
self.assertEqual(body, b'e')
def test_get_range_overlapping_end(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
@ -392,7 +394,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, "206 Partial Content")
self.assertEqual(headers["Content-Length"], "7")
self.assertEqual(headers["Content-Range"], "bytes 18-24/25")
self.assertEqual(body, "ddeeeee")
self.assertEqual(body, b'ddeeeee')
def test_get_range_unsatisfiable(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
@ -428,7 +430,7 @@ class TestDloGetManifest(DloTestCase):
#
# Since the truth is forbidden, we lie.
self.assertEqual(headers["Content-Range"], "bytes 3-12/15")
self.assertEqual(body, "aabbbbbccc")
self.assertEqual(body, b"aabbbbbccc")
self.assertEqual(
self.app.calls,
@ -449,7 +451,7 @@ class TestDloGetManifest(DloTestCase):
# this requires multiple pages of container listing, so we can't send
# a Content-Length header
self.assertIsNone(headers.get("Content-Length"))
self.assertEqual(body, "aaaaabbbbbcccccdddddeeeee")
self.assertEqual(body, b"aaaaabbbbbcccccdddddeeeee")
def test_get_suffix_range(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
@ -459,7 +461,7 @@ class TestDloGetManifest(DloTestCase):
headers = HeaderKeyDict(headers)
self.assertEqual(status, "206 Partial Content")
self.assertEqual(headers["Content-Length"], "25")
self.assertEqual(body, "aaaaabbbbbcccccdddddeeeee")
self.assertEqual(body, b"aaaaabbbbbcccccdddddeeeee")
def test_get_suffix_range_many_segments(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest-many-segments',
@ -471,7 +473,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, "200 OK")
self.assertIsNone(headers.get("Content-Length"))
self.assertIsNone(headers.get("Content-Range"))
self.assertEqual(body, "aaaaabbbbbcccccdddddeeeee")
self.assertEqual(body, b"aaaaabbbbbcccccdddddeeeee")
def test_get_multi_range(self):
# DLO doesn't support multi-range GETs. The way that you express that
@ -485,7 +487,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, "200 OK")
self.assertIsNone(headers.get("Content-Length"))
self.assertIsNone(headers.get("Content-Range"))
self.assertEqual(body, "aaaaabbbbbcccccdddddeeeee")
self.assertEqual(body, b'aaaaabbbbbcccccdddddeeeee')
def test_if_match_matches(self):
manifest_etag = '"%s"' % md5hex(
@ -500,7 +502,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, '200 OK')
self.assertEqual(headers['Content-Length'], '25')
self.assertEqual(body, 'aaaaabbbbbcccccdddddeeeee')
self.assertEqual(body, b'aaaaabbbbbcccccdddddeeeee')
def test_if_match_does_not_match(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
@ -512,7 +514,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, '412 Precondition Failed')
self.assertEqual(headers['Content-Length'], '0')
self.assertEqual(body, '')
self.assertEqual(body, b'')
def test_if_none_match_matches(self):
manifest_etag = '"%s"' % md5hex(
@ -527,7 +529,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, '304 Not Modified')
self.assertEqual(headers['Content-Length'], '0')
self.assertEqual(body, '')
self.assertEqual(body, b'')
def test_if_none_match_does_not_match(self):
req = swob.Request.blank('/v1/AUTH_test/mancon/manifest',
@ -539,7 +541,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, '200 OK')
self.assertEqual(headers['Content-Length'], '25')
self.assertEqual(body, 'aaaaabbbbbcccccdddddeeeee')
self.assertEqual(body, b'aaaaabbbbbcccccdddddeeeee')
def test_get_with_if_modified_since(self):
# It's important not to pass the If-[Un]Modified-Since header to the
@ -581,7 +583,8 @@ class TestDloGetManifest(DloTestCase):
headers = HeaderKeyDict(headers)
self.assertEqual(status, "200 OK")
self.assertEqual(''.join(body), "aaaaa") # first segment made it out
# first segment made it out
self.assertEqual(body, b'aaaaa')
self.assertEqual(self.dlo.logger.get_lines_for_level('error'), [
'While processing manifest /v1/AUTH_test/mancon/manifest, '
'got 403 while retrieving /v1/AUTH_test/c/seg_02',
@ -610,7 +613,7 @@ class TestDloGetManifest(DloTestCase):
with mock.patch(LIMIT, 3):
status, headers, body = self.call_dlo(req)
self.assertEqual(status, "200 OK")
self.assertEqual(body, "aaaaabbbbbccccc")
self.assertEqual(body, b'aaaaabbbbbccccc')
def test_error_listing_container_HEAD(self):
self.app.register(
@ -639,7 +642,8 @@ class TestDloGetManifest(DloTestCase):
headers = HeaderKeyDict(headers)
self.assertEqual(status, "200 OK")
self.assertEqual(''.join(body), "aaaaabbWRONGbb") # stop after error
# stop after error
self.assertEqual(body, b"aaaaabbWRONGbb")
def test_etag_comparison_ignores_quotes(self):
# a little future-proofing here in case we ever fix this in swob
@ -662,7 +666,7 @@ class TestDloGetManifest(DloTestCase):
status, headers, body = self.call_dlo(req)
headers = HeaderKeyDict(headers)
self.assertEqual(headers["Etag"],
'"' + hashlib.md5("abcdef").hexdigest() + '"')
'"' + hashlib.md5(b"abcdef").hexdigest() + '"')
def test_object_prefix_quoting(self):
self.app.register(
@ -675,22 +679,24 @@ class TestDloGetManifest(DloTestCase):
self.app.register(
'GET', '/v1/AUTH_test/c?prefix=%C3%A9',
swob.HTTPOk, {'Content-Type': 'application/json'},
json.dumps(segs))
json.dumps(segs).encode('ascii'))
# NB: wsgi string
path = '/v1/AUTH_test/c/\xC3\xa9'
self.app.register(
'GET', '/v1/AUTH_test/c/\xC3\xa91',
'GET', path + '1',
swob.HTTPOk, {'Content-Length': '5', 'Etag': md5hex("AAAAA")},
"AAAAA")
b"AAAAA")
self.app.register(
'GET', '/v1/AUTH_test/c/\xC3\xA92',
'GET', path + '2',
swob.HTTPOk, {'Content-Length': '5', 'Etag': md5hex("BBBBB")},
"BBBBB")
b"BBBBB")
req = swob.Request.blank('/v1/AUTH_test/man/accent',
environ={'REQUEST_METHOD': 'GET'})
status, headers, body = self.call_dlo(req)
self.assertEqual(status, "200 OK")
self.assertEqual(body, "AAAAABBBBB")
self.assertEqual(body, b'AAAAABBBBB')
def test_get_taking_too_long(self):
the_time = [time.time()]
@ -715,7 +721,7 @@ class TestDloGetManifest(DloTestCase):
status, headers, body = self.call_dlo(req)
self.assertEqual(status, '200 OK')
self.assertEqual(body, 'aaaaabbbbbccccc')
self.assertEqual(body, b'aaaaabbbbbccccc')
def test_get_oversize_segment(self):
# If we send a Content-Length header to the client, it's based on the
@ -738,7 +744,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, '200 OK') # sanity check
self.assertEqual(headers.get('Content-Length'), '25') # sanity check
self.assertEqual(body, 'aaaaabbbbbccccccccccccccc')
self.assertEqual(body, b'aaaaabbbbbccccccccccccccc')
self.assertEqual(
self.app.calls,
[('GET', '/v1/AUTH_test/mancon/manifest'),
@ -770,7 +776,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, '200 OK') # sanity check
self.assertEqual(headers.get('Content-Length'), '25') # sanity check
self.assertEqual(body, 'aaaaabbbbbccccdddddeeeee')
self.assertEqual(body, b'aaaaabbbbbccccdddddeeeee')
def test_get_undersize_segment_range(self):
# Shrink it by a single byte
@ -788,7 +794,7 @@ class TestDloGetManifest(DloTestCase):
self.assertEqual(status, '206 Partial Content') # sanity check
self.assertEqual(headers.get('Content-Length'), '15') # sanity check
self.assertEqual(body, 'aaaaabbbbbcccc')
self.assertEqual(body, b'aaaaabbbbbcccc')
def test_get_with_auth_overridden(self):
auth_got_called = [0]
@ -840,7 +846,7 @@ class TestDloConfiguration(unittest.TestCase):
max_get_time = 2900
""")
conffile = tempfile.NamedTemporaryFile()
conffile = tempfile.NamedTemporaryFile(mode='w')
conffile.write(proxy_conf)
conffile.flush()
@ -853,6 +859,8 @@ class TestDloConfiguration(unittest.TestCase):
self.assertEqual(10, mware.rate_limit_after_segment)
self.assertEqual(3600, mware.max_get_time)
conffile.close()
def test_finding_defaults_from_file(self):
# If DLO has no config vars, go pull them from the proxy server's
# config section
@ -875,7 +883,7 @@ class TestDloConfiguration(unittest.TestCase):
set max_get_time = 2900
""")
conffile = tempfile.NamedTemporaryFile()
conffile = tempfile.NamedTemporaryFile(mode='w')
conffile.write(proxy_conf)
conffile.flush()
@ -887,6 +895,8 @@ class TestDloConfiguration(unittest.TestCase):
self.assertEqual(13, mware.rate_limit_after_segment)
self.assertEqual(2900, mware.max_get_time)
conffile.close()
def test_finding_defaults_from_dir(self):
# If DLO has no config vars, go pull them from the proxy server's
# config section
@ -913,11 +923,13 @@ class TestDloConfiguration(unittest.TestCase):
conf_dir = self.tmpdir
conffile1 = tempfile.NamedTemporaryFile(dir=conf_dir, suffix='.conf')
conffile1 = tempfile.NamedTemporaryFile(mode='w',
dir=conf_dir, suffix='.conf')
conffile1.write(proxy_conf1)
conffile1.flush()
conffile2 = tempfile.NamedTemporaryFile(dir=conf_dir, suffix='.conf')
conffile2 = tempfile.NamedTemporaryFile(mode='w',
dir=conf_dir, suffix='.conf')
conffile2.write(proxy_conf2)
conffile2.flush()
@ -929,6 +941,9 @@ class TestDloConfiguration(unittest.TestCase):
self.assertEqual(13, mware.rate_limit_after_segment)
self.assertEqual(2900, mware.max_get_time)
conffile1.close()
conffile2.close()
if __name__ == '__main__':
unittest.main()

96
test/unit/common/middleware/test_staticweb.py
View File

@ -528,7 +528,7 @@ class TestStaticWeb(unittest.TestCase):
def test_container3indexhtml(self):
resp = Request.blank('/v1/a/c3/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertTrue('Test main index.html file.' in resp.body)
self.assertIn(b'Test main index.html file.', resp.body)
def test_container3subsubdir(self):
resp = Request.blank(
@ -539,16 +539,16 @@ class TestStaticWeb(unittest.TestCase):
resp = Request.blank(
'/v1/a/c3/subdir3/subsubdir/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertEqual(resp.body, 'index file')
self.assertEqual(resp.body, b'index file')
def test_container3subdir(self):
resp = Request.blank(
'/v1/a/c3/subdir/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertIn('Listing of /v1/a/c3/subdir/', resp.body)
self.assertIn('</style>', resp.body)
self.assertNotIn('<link', resp.body)
self.assertNotIn('listing.css', resp.body)
self.assertIn(b'Listing of /v1/a/c3/subdir/', resp.body)
self.assertIn(b'</style>', resp.body)
self.assertNotIn(b'<link', resp.body)
self.assertNotIn(b'listing.css', resp.body)
def test_container3subdirx(self):
resp = Request.blank(
@ -569,18 +569,18 @@ class TestStaticWeb(unittest.TestCase):
resp = Request.blank(
'/v1/a/c3/unknown').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 404)
self.assertNotIn("Chrome's 404 fancy-page sucks.", resp.body)
self.assertNotIn(b"Chrome's 404 fancy-page sucks.", resp.body)
def test_container3bindexhtml(self):
resp = Request.blank('/v1/a/c3b/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 204)
self.assertEqual(resp.body, '')
self.assertEqual(resp.body, b'')
def test_container4indexhtml(self):
resp = Request.blank('/v1/a/c4/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertIn('Listing of /v1/a/c4/', resp.body)
self.assertIn('href="listing.css"', resp.body)
self.assertIn(b'Listing of /v1/a/c4/', resp.body)
self.assertIn(b'href="listing.css"', resp.body)
def test_container4indexhtmlauthed(self):
resp = Request.blank('/v1/a/c4').get_response(self.test_staticweb)
@ -600,16 +600,16 @@ class TestStaticWeb(unittest.TestCase):
resp = Request.blank(
'/v1/a/c4/unknown').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 404)
self.assertIn("Chrome's 404 fancy-page sucks.", resp.body)
self.assertIn(b"Chrome's 404 fancy-page sucks.", resp.body)
def test_container4subdir(self):
resp = Request.blank(
'/v1/a/c4/subdir/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertIn('Listing of /v1/a/c4/subdir/', resp.body)
self.assertNotIn('</style>', resp.body)
self.assertIn('<link', resp.body)
self.assertIn('href="../listing.css"', resp.body)
self.assertIn(b'Listing of /v1/a/c4/subdir/', resp.body)
self.assertNotIn(b'</style>', resp.body)
self.assertIn(b'<link', resp.body)
self.assertIn(b'href="../listing.css"', resp.body)
self.assertEqual(resp.headers['content-type'],
'text/html; charset=UTF-8')
@ -631,7 +631,7 @@ class TestStaticWeb(unittest.TestCase):
resp = Request.blank(
'/v1/a/c5/unknown').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 404)
self.assertNotIn("Chrome's 404 fancy-page sucks.", resp.body)
self.assertNotIn(b"Chrome's 404 fancy-page sucks.", resp.body)
def test_container6subdir(self):
resp = Request.blank(
@ -649,7 +649,7 @@ class TestStaticWeb(unittest.TestCase):
staticweb.filter_factory({})(self.app), deny_listing=True)
resp = Request.blank('/v1/a/c6/').get_response(test_staticweb)
self.assertEqual(resp.status_int, 401)
self.assertIn("Hey, you're not authorized to see this!", resp.body)
self.assertIn(b"Hey, you're not authorized to see this!", resp.body)
# expect default 401 if request is not auth'd for listing or object GET
test_staticweb = FakeAuthFilter(
@ -657,20 +657,20 @@ class TestStaticWeb(unittest.TestCase):
deny_objects=True)
resp = Request.blank('/v1/a/c6/').get_response(test_staticweb)
self.assertEqual(resp.status_int, 401)
self.assertNotIn("Hey, you're not authorized to see this!", resp.body)
self.assertNotIn(b"Hey, you're not authorized to see this!", resp.body)
def test_container6blisting(self):
label = 'Listing of {0}/'.format(
meta_map['c6b']['meta']['web-listings-label'])
resp = Request.blank('/v1/a/c6b/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertIn(label, resp.body)
self.assertIn(label.encode('utf-8'), resp.body)
def test_container7listing(self):
# container7 has web-listings = f, web-error=error.html
resp = Request.blank('/v1/a/c7/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 404)
self.assertIn("Web Listing Disabled", resp.body)
self.assertIn(b"Web Listing Disabled", resp.body)
# expect 301 if auth'd but no trailing '/'
resp = Request.blank('/v1/a/c7').get_response(self.test_staticweb)
@ -682,14 +682,14 @@ class TestStaticWeb(unittest.TestCase):
deny_objects=True)
resp = Request.blank('/v1/a/c7').get_response(test_staticweb)
self.assertEqual(resp.status_int, 401)
self.assertNotIn("Hey, you're not authorized to see this!", resp.body)
self.assertNotIn(b"Hey, you're not authorized to see this!", resp.body)
# expect custom 401 if request is not auth'd for listing
test_staticweb = FakeAuthFilter(
staticweb.filter_factory({})(self.app), deny_listing=True)
resp = Request.blank('/v1/a/c7/').get_response(test_staticweb)
self.assertEqual(resp.status_int, 401)
self.assertIn("Hey, you're not authorized to see this!", resp.body)
self.assertIn(b"Hey, you're not authorized to see this!", resp.body)
# expect default 401 if request is not auth'd for listing or object GET
test_staticweb = FakeAuthFilter(
@ -697,69 +697,69 @@ class TestStaticWeb(unittest.TestCase):
deny_objects=True)
resp = Request.blank('/v1/a/c7/').get_response(test_staticweb)
self.assertEqual(resp.status_int, 401)
self.assertNotIn("Hey, you're not authorized to see this!", resp.body)
self.assertNotIn(b"Hey, you're not authorized to see this!", resp.body)
def test_container8listingcss(self):
resp = Request.blank(
'/v1/a/c8/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertTrue('Listing of /v1/a/c8/' in resp.body)
self.assertTrue('<link' in resp.body)
self.assertTrue(
'href="http://localhost/stylesheets/listing.css"' in resp.body)
self.assertIn(b'Listing of /v1/a/c8/', resp.body)
self.assertIn(b'<link', resp.body)
self.assertIn(b'href="http://localhost/stylesheets/listing.css"',
resp.body)
def test_container8subdirlistingcss(self):
resp = Request.blank(
'/v1/a/c8/subdir/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertTrue('Listing of /v1/a/c8/subdir/' in resp.body)
self.assertTrue('<link' in resp.body)
self.assertTrue(
'href="http://localhost/stylesheets/listing.css"' in resp.body)
self.assertIn(b'Listing of /v1/a/c8/subdir/', resp.body)
self.assertIn(b'<link', resp.body)
self.assertIn(b'href="http://localhost/stylesheets/listing.css"',
resp.body)
def test_container9listingcss(self):
resp = Request.blank(
'/v1/a/c9/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertTrue('Listing of /v1/a/c9/' in resp.body)
self.assertTrue('<link' in resp.body)
self.assertTrue('href="/absolute/listing.css"' in resp.body)
self.assertIn(b'Listing of /v1/a/c9/', resp.body)
self.assertIn(b'<link', resp.body)
self.assertIn(b'href="/absolute/listing.css"', resp.body)
def test_container9subdirlistingcss(self):
resp = Request.blank(
'/v1/a/c9/subdir/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertTrue('Listing of /v1/a/c9/subdir/' in resp.body)
self.assertTrue('<link' in resp.body)
self.assertTrue('href="/absolute/listing.css"' in resp.body)
self.assertIn(b'Listing of /v1/a/c9/subdir/', resp.body)
self.assertIn(b'<link', resp.body)
self.assertIn(b'href="/absolute/listing.css"', resp.body)
def test_container10unicodesubdirlisting(self):
resp = Request.blank(
'/v1/a/c10/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertTrue('Listing of /v1/a/c10/' in resp.body)
self.assertIn(b'Listing of /v1/a/c10/', resp.body)
resp = Request.blank(
'/v1/a/c10/\xe2\x98\x83/').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertTrue('Listing of /v1/a/c10/\xe2\x98\x83/' in resp.body)
self.assertIn(b'Listing of /v1/a/c10/\xe2\x98\x83/', resp.body)
resp = Request.blank(
'/v1/a/c10/\xe2\x98\x83/\xe2\x98\x83/'
).get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertTrue(
'Listing of /v1/a/c10/\xe2\x98\x83/\xe2\x98\x83/' in resp.body)
self.assertIn(
b'Listing of /v1/a/c10/\xe2\x98\x83/\xe2\x98\x83/', resp.body)
def test_container11subdirmarkerobjectindex(self):
resp = Request.blank('/v1/a/c11/subdir/').get_response(
self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertTrue('<h2>c11 subdir index</h2>' in resp.body)
self.assertIn(b'<h2>c11 subdir index</h2>', resp.body)
def test_container11subdirmarkermatchdirtype(self):
resp = Request.blank('/v1/a/c11a/subdir/').get_response(
self.test_staticweb)
self.assertEqual(resp.status_int, 404)
self.assertIn('Index File Not Found', resp.body)
self.assertIn(b'Index File Not Found', resp.body)
def test_container11subdirmarkeraltdirtype(self):
resp = Request.blank('/v1/a/c11a/subdir2/').get_response(
@ -775,27 +775,27 @@ class TestStaticWeb(unittest.TestCase):
resp = Request.blank('/v1/a/c12/').get_response(
self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertIn('index file', resp.body)
self.assertIn(b'index file', resp.body)
def test_container_404_has_css(self):
resp = Request.blank('/v1/a/c13/').get_response(
self.test_staticweb)
self.assertEqual(resp.status_int, 404)
self.assertIn('listing.css', resp.body)
self.assertIn(b'listing.css', resp.body)
def test_container_404_has_no_css(self):
resp = Request.blank('/v1/a/c7/').get_response(
self.test_staticweb)
self.assertEqual(resp.status_int, 404)
self.assertNotIn('listing.css', resp.body)
self.assertIn('<style', resp.body)
self.assertNotIn(b'listing.css', resp.body)
self.assertIn(b'<style', resp.body)
def test_subrequest_once_if_possible(self):
resp = Request.blank(
'/v1/a/c4/one.txt').get_response(self.test_staticweb)
self.assertEqual(resp.status_int, 200)
self.assertEqual(resp.headers['x-object-meta-test'], 'value')
self.assertEqual(resp.body, '1')
self.assertEqual(resp.body, b'1')
self.assertEqual(self.app.calls, 1)
def test_no_auth_middleware(self):

22
test/unit/common/test_utils.py
View File

@ -2463,28 +2463,6 @@ log_name = %(yarr)s'''
self.verify_under_pseudo_time(testfunc, target_runtime_ms=900)
def test_urlparse(self):
parsed = utils.urlparse('http://127.0.0.1/')
self.assertEqual(parsed.scheme, 'http')
self.assertEqual(parsed.hostname, '127.0.0.1')
self.assertEqual(parsed.path, '/')
parsed = utils.urlparse('http://127.0.0.1:8080/')
self.assertEqual(parsed.port, 8080)
parsed = utils.urlparse('https://127.0.0.1/')
self.assertEqual(parsed.scheme, 'https')
parsed = utils.urlparse('http://[::1]/')
self.assertEqual(parsed.hostname, '::1')
parsed = utils.urlparse('http://[::1]:8080/')
self.assertEqual(parsed.hostname, '::1')
self.assertEqual(parsed.port, 8080)
parsed = utils.urlparse('www.example.com')
self.assertEqual(parsed.hostname, '')
def test_search_tree(self):
# file match & ext miss
with temptree(['asdf.conf', 'blarg.conf', 'asdf.cfg']) as t:

6
test/unit/container/test_reconciler.py
View File

@ -1171,7 +1171,7 @@ class TestReconciler(unittest.TestCase):
q_path = '.misplaced_objects/%s' % container
self._mock_listing({
(None, "/%s/1:/AUTH_bob/c/o1" % q_path): q_ts,
(1, '/AUTH_bob/c/o1'): q_ts - 0.00001, # slightly older
(1, '/AUTH_bob/c/o1'): q_ts - 1, # slightly older
})
self._mock_oldest_spi({'c': 0})
deleted_container_entries = self._run_once()
@ -1411,7 +1411,7 @@ class TestReconciler(unittest.TestCase):
self._mock_listing({
(None, "/.misplaced_objects/3600/1:/AUTH_bob/c/o1"): 3679.2019,
(1, "/AUTH_bob/c/o1"): 3679.2019,
(0, "/AUTH_bob/c/o1"): 3679.2019 + 0.00001, # slightly newer
(0, "/AUTH_bob/c/o1"): 3679.2019 + 1, # slightly newer
})
self._mock_oldest_spi({'c': 0})
deleted_container_entries = self._run_once()
@ -1452,7 +1452,7 @@ class TestReconciler(unittest.TestCase):
self._mock_listing({
(None, "/.misplaced_objects/36000/1:/AUTH_bob/c/o1"): 36123.38393,
(1, "/AUTH_bob/c/o1"): 36123.38393,
(0, "/AUTH_bob/c/o1"): 36123.38393 - 0.00001, # slightly older
(0, "/AUTH_bob/c/o1"): 36123.38393 - 1, # slightly older
})
self._mock_oldest_spi({'c': 0})
deleted_container_entries = self._run_once()

4
test/unit/obj/test_auditor.py
View File

@ -678,7 +678,7 @@ class TestAuditor(unittest.TestCase):
auditor_worker.audit_all_objects(device_dirs=['sda'])
log_lines = self.logger.get_lines_for_level('info')
self.assertGreater(len(log_lines), 0)
self.assertTrue(log_lines[0].index('ALL - parallel, sda'))
self.assertIn('ALL - parallel, sda', log_lines[0])
self.logger.clear()
auditor_worker = auditor.AuditorWorker(self.conf, self.logger,
@ -687,7 +687,7 @@ class TestAuditor(unittest.TestCase):
auditor_worker.audit_all_objects(device_dirs=['sda'])
log_lines = self.logger.get_lines_for_level('info')
self.assertGreater(len(log_lines), 0)
self.assertTrue(log_lines[0].index('ZBF - sda'))
self.assertIn('ZBF - sda', log_lines[0])
def test_object_run_recon_cache(self):
ts = Timestamp(time.time())

24
test/unit/obj/test_replicator.py
View File

@ -536,9 +536,27 @@ class TestObjectReplicator(unittest.TestCase):
self._write_disk_data('sdd', with_json=True)
_create_test_rings(self.testdir, devs)
self.replicator.collect_jobs()
self.replicator.collect_jobs(override_partitions=[1])
self.assertEqual(self.replicator.total_stats.failure, 0)
def test_collect_jobs_with_override_parts_and_unexpected_part_dir(self):
self.replicator.collect_jobs(override_partitions=[0, 2])
self.assertEqual(self.replicator.total_stats.failure, 0)
os.mkdir(os.path.join(self.objects_1, 'foo'))
jobs = self.replicator.collect_jobs(override_partitions=[0, 2])
found_jobs = set()
for j in jobs:
found_jobs.add((int(j['policy']), int(j['partition'])))
self.assertEqual(found_jobs, {
(0, 0),
(0, 2),
(1, 0),
(1, 2),
})
num_disks = len(POLICIES[1].object_ring.devs)
# N.B. it's not clear why the UUT increments failure per device
self.assertEqual(self.replicator.total_stats.failure, num_disks)
@mock.patch('swift.obj.replicator.random.shuffle', side_effect=lambda l: l)
def test_collect_jobs_multi_disk(self, mock_shuffle):
devs = [
@ -1401,10 +1419,10 @@ class TestObjectReplicator(unittest.TestCase):
self.assertTrue(os.access(part_path, os.F_OK))
self.replicator.replicate(override_devices=['sdb'])
self.assertTrue(os.access(part_path, os.F_OK))
self.replicator.replicate(override_partitions=['9'])
self.replicator.replicate(override_partitions=[9])
self.assertTrue(os.access(part_path, os.F_OK))
self.replicator.replicate(override_devices=['sda'],
override_partitions=['1'])
override_partitions=[1])
self.assertFalse(os.access(part_path, os.F_OK))
def test_delete_policy_override_params(self):

2
tools/test-setup.sh
View File

@ -25,6 +25,6 @@ function is_rhel7 {
if is_rhel7; then
# Install CentOS OpenStack repos so that we have access to some extra
# packages.
sudo yum install -y centos-release-openstack-queens
sudo yum install -y centos-release-openstack-rocky
sudo yum install -y liberasurecode-devel
fi

37
tox.ini
View File

@ -1,18 +1,19 @@
[tox]
envlist = py35,py27,pep8
envlist = py37,py27,pep8
minversion = 2.3.2
skipsdist = True
[testenv]
usedevelop = True
install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} -U {opts} {packages}
install_command = pip install -U {opts} {packages}
setenv = VIRTUAL_ENV={envdir}
NOSE_WITH_COVERAGE=1
NOSE_COVER_BRANCHES=1
deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt}
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = find . -type f -name "*.py[c|o]" -delete
commands = find . ( -type f -o -type l ) -name "*.py[c|o]" -delete
find . -type d -name "__pycache__" -delete
nosetests {posargs:test/unit}
whitelist_externals = find
@ -31,12 +32,15 @@ setenv = VIRTUAL_ENV={envdir}
# tests file by file, add them into the list below, and never go back.
# This list also serves as a shared task board for those helping with py3.
# But mind that reviews expanding this list may be outstanding in Gerrit.
[testenv:py35]
[testenv:py37]
commands =
find . ( -type f -o -type l ) -name "*.py[c|o]" -delete
find . -type d -name "__pycache__" -delete
nosetests {posargs:\
test/unit/account \
test/unit/cli \
test/unit/common/middleware/crypto \
test/unit/common/middleware/s3api/test_s3token.py \
test/unit/common/middleware/test_account_quotas.py \
test/unit/common/middleware/test_acl.py \
test/unit/common/middleware/test_catch_errors.py \
@ -44,6 +48,7 @@ commands =
test/unit/common/middleware/test_container_sync.py \
test/unit/common/middleware/test_copy.py \
test/unit/common/middleware/test_crossdomain.py \
test/unit/common/middleware/test_dlo.py \
test/unit/common/middleware/test_domain_remap.py \
test/unit/common/middleware/test_formpost.py \
test/unit/common/middleware/test_gatekeeper.py \
@ -59,6 +64,7 @@ commands =
test/unit/common/middleware/test_read_only.py \
test/unit/common/middleware/test_recon.py \
test/unit/common/middleware/test_subrequest_logging.py \
test/unit/common/middleware/test_staticweb.py \
test/unit/common/middleware/test_tempauth.py \
test/unit/common/middleware/test_versioned_writes.py \
test/unit/common/middleware/test_xprofile.py \
@ -95,31 +101,23 @@ commands =
test/unit/proxy/controllers/test_obj.py}
[testenv:py36]
commands = {[testenv:py35]commands}
commands = {[testenv:py37]commands}
[testenv:py37]
commands = {[testenv:py35]commands}
[testenv:py35]
commands = {[testenv:py37]commands}
[testenv:pep8]
basepython = python2.7
commands =
flake8 {posargs:swift test doc setup.py}
flake8 --filename=swift* bin
python ./setup.py check --restructuredtext --strict
bandit -c bandit.yaml -r swift -n 5
./.manpages {posargs}
[testenv:py3pep8]
basepython = python3
install_command = echo {packages}
whitelist_externals = echo
commands =
# Gross hack. There's no other way to get it to /not/ install swift itself
# (which triggers installing eventlet) but also get flake8 installed.
pip install flake8
flake8 swift test doc setup.py
flake8 --filename=swift* bin
python ./setup.py check --restructuredtext --strict
./.manpages {posargs}
commands = {[testenv:pep8]commands}
[testenv:func]
basepython = python2.7
@ -149,14 +147,14 @@ setenv = SWIFT_TEST_IN_PROCESS=1
commands = {posargs}
[testenv:docs]
basepython = python2.7
basepython = python3
deps = -r{toxinidir}/doc/requirements.txt
commands = sphinx-build -W -b html doc/source doc/build/html
[testenv:api-ref]
# This environment is called from CI scripts to test and publish
# the API Ref to developer.openstack.org.
basepython = python2.7
basepython = python3
deps = -r{toxinidir}/doc/requirements.txt
commands =
rm -rf api-ref/build
@ -191,6 +189,7 @@ deps = bindep
commands = bindep test
[testenv:releasenotes]
basepython = python3
deps = -r{toxinidir}/doc/requirements.txt
commands = sphinx-build -a -W -E -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html