openstack
/
swift
Code Issues Proposed changes

Enable middleware to set metadata on object POST 

Sysmeta cannot be updated by an object POST because
that would cause all existing sysmeta to be deleted.
Crypto middleware will want to add 'system' metadata
to object metadata on PUTs and POSTs, but it is ok
for this metadata to be replaced en-masse on every
POST.

This patch introduces x-object-transient-sysmeta-*
that is persisted by object servers and returned
in GET and HEAD responses, just like user metadata,
without polluting the x-object-meta-* namespace.
All headers in this namespace will be filtered
inbound and outbound by the gatekeeper, so cannot
be set or read by clients.

Change-Id: I498928c2e38ea379b0bc80f82ddf5ca0fcb8b90f
This commit is contained in:
Alistair Coles 2015-02-23 16:14:43 +00:00 committed by Alistair Coles
parent 6731dd6da8
commit 7d391626d0
13 changed files with 337 additions and 34 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
doc/source/development_middleware.rst
View File

@ -209,7 +209,7 @@ User metadata takes the form of ``X-<type>-Meta-<key>: <value>``, where
and ``<key>`` and ``<value>`` are set by the client.
User metadata should generally be reserved for use by the client or
client applications. An perfect example use-case for user metadata is
client applications. A perfect example use-case for user metadata is
`python-swiftclient`_'s ``X-Object-Meta-Mtime`` which it stores on
object it uploads to implement its ``--changed`` option which will only
upload files that have changed since the last upload.
@ -249,3 +249,47 @@ modified directly by client requests, and the outgoing filter ensures
that removing middleware that uses a specific system metadata key
renders it benign. New middleware should take advantage of system
metadata.
System metadata may be set on accounts and containers by including headers with
a PUT or POST request. Where a header name matches the name of an existing item
of system metadata, the value of the existing item will be updated. Otherwise
existing items are preserved. A system metadata header with an empty value will
cause any existing item with the same name to be deleted.
System metadata may be set on objects using only PUT requests. All items of
existing system metadata will be deleted and replaced en-masse by any system
metadata headers included with the PUT request. System metadata is neither
updated nor deleted by a POST request: updating individual items of system
metadata with a POST request is not yet supported in the same way that updating
individual items of user metadata is not supported. In cases where middleware
needs to store its own metadata with a POST request, it may use Object Transient
Sysmeta.
^^^^^^^^^^^^^^^^^^^^^^^^
Object Transient-Sysmeta
^^^^^^^^^^^^^^^^^^^^^^^^
If middleware needs to store object metadata with a POST request it may do so
using headers of the form ``X-Object-Transient-Sysmeta-<key>: <value>``.
All headers on client requests in the form of ``X-Object-Transient-Sysmeta-<key>``
will be dropped from the request before being processed by any
middleware. All headers on responses from back-end systems in the form
of ``X-Object-Transient-Sysmeta-<key>`` will be removed after all middleware has
processed the response but before the response is sent to the client.
See :ref:`gatekeeper` middleware for more information.
Transient-sysmeta updates have the same semantic as user metadata updates on an
object: whenever any POST (or PUT) request is made to an object, all existing
items of transient-sysmeta are deleted "en-masse" and replaced with any
transient-sysmeta included with the POST request. Transient-sysmeta set by a
middleware is therefore prone to deletion by a subsequent client-generated POST
request unless the middleware is careful to include its transient-sysmeta with
every POST.
Transient-sysmeta deliberately uses a different header prefix to user metadata
so that middlewares can avoid potential conflict with user metadata keys.
Transient-sysmeta deliberately uses a different header prefix to system
metadata to emphasize the fact that the data is only persisted until a
subsequent POST.

9
swift/common/middleware/gatekeeper.py
View File

@ -33,22 +33,25 @@ automatically inserted close to the start of the pipeline by the proxy server.
from swift.common.swob import Request
from swift.common.utils import get_logger
from swift.common.request_helpers import remove_items, get_sys_meta_prefix
from swift.common.request_helpers import (
remove_items, get_sys_meta_prefix, get_object_transient_sysmeta_prefix
)
import re
#: A list of python regular expressions that will be used to
#: match against inbound request headers. Matching headers will
#: be removed from the request.
# Exclude headers starting with a sysmeta prefix.
# Exclude headers starting with object transient system metadata prefix.
# Exclude headers starting with an internal backend header prefix.
# If adding to this list, note that these are regex patterns,
# so use a trailing $ to constrain to an exact header match
# rather than prefix match.
inbound_exclusions = [get_sys_meta_prefix('account'),
get_sys_meta_prefix('container'),
get_sys_meta_prefix('object'),
get_object_transient_sysmeta_prefix(),
'x-backend']
# 'x-object-sysmeta' is reserved in anticipation of future support
# for system metadata being applied to objects
#: A list of python regular expressions that will be used to

28
swift/common/request_helpers.py
View File

@ -251,6 +251,34 @@ def get_obj_persisted_sysmeta_prefix():
return get_sys_meta_prefix('object')
def is_object_transient_sysmeta(key):
"""
Tests if a header key starts with and is longer than the prefix for object
transient syetem metadata.
:param key: header key
:returns: True if the key satisfies the test, False otherwise
"""
prefix = get_object_transient_sysmeta_prefix()
if len(key) <= len(prefix):
return False
return key.lower().startswith(prefix)
def get_object_transient_sysmeta_prefix():
"""
Returns the prefix for object transient system metadata.
This prefix defines the namespace for headers that will be persisted
by backend object servers. These headers are treated in the same way
as object user metadata i.e. all headers in this namespace will be
replaced on every POST request.
:returns: prefix string for object transient system metadata.
"""
return 'x-object-transient-sysmeta-'
def remove_items(headers, condition):
"""
Removes items from a dict whose keys satisfy

21
swift/obj/server.py
View File

@ -45,7 +45,7 @@ from swift.obj import ssync_receiver
from swift.common.http import is_success
from swift.common.base_storage_server import BaseStorageServer
from swift.common.request_helpers import get_name_and_placement, \
is_user_meta, is_sys_or_user_meta
is_user_meta, is_sys_or_user_meta, is_object_transient_sysmeta
from swift.common.swob import HTTPAccepted, HTTPBadRequest, HTTPCreated, \
HTTPInternalServerError, HTTPNoContent, HTTPNotFound, \
HTTPPreconditionFailed, HTTPRequestTimeout, HTTPUnprocessableEntity, \
@ -496,7 +496,8 @@ class ObjectController(BaseStorageServer):
metadata = {'X-Timestamp': req_timestamp.internal}
self._preserve_slo_manifest(metadata, orig_metadata)
metadata.update(val for val in request.headers.items()
if is_user_meta('object', val[0]))
if (is_user_meta('object', val[0]) or
is_object_transient_sysmeta(val[0])))
headers_to_copy = (
request.headers.get(
'X-Backend-Replication-Headers', '').split() +
@ -734,9 +735,11 @@ class ObjectController(BaseStorageServer):
'Content-Length': str(upload_size),
}
metadata.update(val for val in request.headers.items()
if is_sys_or_user_meta('object', val[0]))
if (is_sys_or_user_meta('object', val[0]) or
is_object_transient_sysmeta(val[0])))
metadata.update(val for val in footer_meta.items()
if is_sys_or_user_meta('object', val[0]))
if (is_sys_or_user_meta('object', val[0]) or
is_object_transient_sysmeta(val[0])))
headers_to_copy = (
request.headers.get(
'X-Backend-Replication-Headers', '').split() +
@ -831,8 +834,9 @@ class ObjectController(BaseStorageServer):
response.headers['Content-Type'] = metadata.get(
'Content-Type', 'application/octet-stream')
for key, value in metadata.items():
if is_sys_or_user_meta('object', key) or \
key.lower() in self.allowed_headers:
if (is_sys_or_user_meta('object', key) or
is_object_transient_sysmeta(key) or
key.lower() in self.allowed_headers):
response.headers[key] = value
response.etag = metadata['ETag']
response.last_modified = math.ceil(float(file_x_ts))
@ -886,8 +890,9 @@ class ObjectController(BaseStorageServer):
response.headers['Content-Type'] = metadata.get(
'Content-Type', 'application/octet-stream')
for key, value in metadata.items():
if is_sys_or_user_meta('object', key) or \
key.lower() in self.allowed_headers:
if (is_sys_or_user_meta('object', key) or
is_object_transient_sysmeta(key) or
key.lower() in self.allowed_headers):
response.headers[key] = value
response.etag = metadata['ETag']
ts = Timestamp(metadata['X-Timestamp'])

11
swift/proxy/controllers/base.py
View File

@ -56,7 +56,8 @@ from swift.common.swob import Request, Response, HeaderKeyDict, Range, \
status_map
from swift.common.request_helpers import strip_sys_meta_prefix, \
strip_user_meta_prefix, is_user_meta, is_sys_meta, is_sys_or_user_meta, \
http_response_to_document_iters
http_response_to_document_iters, get_object_transient_sysmeta_prefix, \
is_object_transient_sysmeta
from swift.common.storage_policy import POLICIES
@ -182,12 +183,18 @@ def headers_to_object_info(headers, status_int=HTTP_OK):
Construct a cacheable dict of object info based on response headers.
"""
headers, meta, sysmeta = _prep_headers_to_info(headers, 'object')
transient_sysmeta = {}
for key, val in headers.iteritems():
if is_object_transient_sysmeta(key):
key = key.lower()[len(get_object_transient_sysmeta_prefix()):]
transient_sysmeta[key] = val
info = {'status': status_int,
'length': headers.get('content-length'),
'type': headers.get('content-type'),
'etag': headers.get('etag'),
'meta': meta,
'sysmeta': sysmeta
'sysmeta': sysmeta,
'transient_sysmeta': transient_sysmeta
}
return info

8
swift/proxy/controllers/obj.py
View File

@ -72,7 +72,7 @@ from swift.common.swob import HTTPAccepted, HTTPBadRequest, HTTPNotFound, \
HTTPClientDisconnect, HTTPUnprocessableEntity, Response, HTTPException, \
HTTPRequestedRangeNotSatisfiable, Range, HTTPInternalServerError
from swift.common.request_helpers import is_sys_or_user_meta, is_sys_meta, \
copy_header_subset, update_content_type
copy_header_subset, update_content_type, is_object_transient_sysmeta
def copy_headers_into(from_r, to_r):
@ -83,7 +83,9 @@ def copy_headers_into(from_r, to_r):
"""
pass_headers = ['x-delete-at']
for k, v in from_r.headers.items():
if is_sys_or_user_meta('object', k) or k.lower() in pass_headers:
if (is_sys_or_user_meta('object', k) or
is_object_transient_sysmeta(k) or
k.lower() in pass_headers):
to_r.headers[k] = v
@ -511,7 +513,7 @@ class BaseObjectController(Controller):
# remove_items(sink_req.headers, condition)
copy_header_subset(source_resp, sink_req, condition)
else:
# copy/update existing sysmeta and user meta
# copy/update existing sysmeta, transient_sysmeta and user meta
copy_headers_into(source_resp, sink_req)
copy_headers_into(req, sink_req)

25
test/probe/test_object_metadata_replication.py
View File

@ -314,6 +314,8 @@ class Test(ReplProbeTest):
def test_sysmeta_after_replication_with_subsequent_post(self):
sysmeta = {'x-object-sysmeta-foo': 'sysmeta-foo'}
usermeta = {'x-object-meta-bar': 'meta-bar'}
transient_sysmeta = {
'x-object-transient-sysmeta-bar': 'transient-sysmeta-bar'}
self.brain.put_container(policy_index=int(self.policy))
# put object
self._put_object()
@ -328,11 +330,13 @@ class Test(ReplProbeTest):
# post some user meta to second server subset
self.brain.stop_handoff_half()
self._post_object(usermeta)
user_and_transient_sysmeta = dict(usermeta.items() +
transient_sysmeta.items())
self._post_object(user_and_transient_sysmeta)
metadata = self._get_object_metadata()
for key in usermeta:
for key in user_and_transient_sysmeta:
self.assertTrue(key in metadata)
self.assertEqual(metadata[key], usermeta[key])
self.assertEqual(metadata[key], user_and_transient_sysmeta[key])
for key in sysmeta:
self.assertFalse(key in metadata)
self.brain.start_handoff_half()
@ -346,6 +350,7 @@ class Test(ReplProbeTest):
metadata = self._get_object_metadata()
expected = dict(sysmeta)
expected.update(usermeta)
expected.update(transient_sysmeta)
for key in expected.keys():
self.assertTrue(key in metadata, key)
self.assertEqual(metadata[key], expected[key])
@ -366,17 +371,21 @@ class Test(ReplProbeTest):
def test_sysmeta_after_replication_with_prior_post(self):
sysmeta = {'x-object-sysmeta-foo': 'sysmeta-foo'}
usermeta = {'x-object-meta-bar': 'meta-bar'}
transient_sysmeta = {
'x-object-transient-sysmeta-bar': 'transient-sysmeta-bar'}
self.brain.put_container(policy_index=int(self.policy))
# put object
self._put_object()
# put user meta to first server subset
self.brain.stop_handoff_half()
self._post_object(headers=usermeta)
user_and_transient_sysmeta = dict(usermeta.items() +
transient_sysmeta.items())
self._post_object(user_and_transient_sysmeta)
metadata = self._get_object_metadata()
for key in usermeta:
for key in user_and_transient_sysmeta:
self.assertTrue(key in metadata)
self.assertEqual(metadata[key], usermeta[key])
self.assertEqual(metadata[key], user_and_transient_sysmeta[key])
self.brain.start_handoff_half()
# put newer object with sysmeta to second server subset
@ -398,7 +407,7 @@ class Test(ReplProbeTest):
for key in sysmeta:
self.assertTrue(key in metadata)
self.assertEqual(metadata[key], sysmeta[key])
for key in usermeta:
for key in user_and_transient_sysmeta:
self.assertFalse(key in metadata)
self.brain.start_primary_half()
@ -409,7 +418,7 @@ class Test(ReplProbeTest):
for key in sysmeta:
self.assertTrue(key in metadata)
self.assertEqual(metadata[key], sysmeta[key])
for key in usermeta:
for key in user_and_transient_sysmeta:
self.assertFalse(key in metadata)
self.brain.start_handoff_half()

10
test/unit/common/middleware/test_gatekeeper.py
View File

@ -74,10 +74,16 @@ class TestGatekeeper(unittest.TestCase):
x_backend_headers = {'X-Backend-Replication': 'true',
'X-Backend-Replication-Headers': 'stuff'}
object_transient_sysmeta_headers = {
'x-object-transient-sysmeta-': 'value',
'x-object-transient-sysmeta-foo': 'value'}
forbidden_headers_out = dict(sysmeta_headers.items() +
x_backend_headers.items())
x_backend_headers.items() +
object_transient_sysmeta_headers.items())
forbidden_headers_in = dict(sysmeta_headers.items() +
x_backend_headers.items())
x_backend_headers.items() +
object_transient_sysmeta_headers.items())
def _assertHeadersEqual(self, expected, actual):
for key in expected:

10
test/unit/common/test_request_helpers.py
View File

@ -21,7 +21,7 @@ from swift.common.storage_policy import POLICIES, EC_POLICY, REPL_POLICY
from swift.common.request_helpers import is_sys_meta, is_user_meta, \
is_sys_or_user_meta, strip_sys_meta_prefix, strip_user_meta_prefix, \
remove_items, copy_header_subset, get_name_and_placement, \
http_response_to_document_iters
http_response_to_document_iters, is_object_transient_sysmeta
from test.unit import patch_policies
from test.unit.common.test_utils import FakeResponse
@ -68,6 +68,14 @@ class TestRequestHelpers(unittest.TestCase):
self.assertEqual(strip_user_meta_prefix(st, 'x-%s-%s-a'
% (st, mt)), 'a')
def test_is_object_transient_sysmeta(self):
self.assertTrue(is_object_transient_sysmeta(
'x-object-transient-sysmeta-foo'))
self.assertFalse(is_object_transient_sysmeta(
'x-object-transient-sysmeta-'))
self.assertFalse(is_object_transient_sysmeta(
'x-object-meatmeta-foo'))
def test_remove_items(self):
src = {'a': 'b',
'c': 'd'}

5
test/unit/obj/test_diskfile.py
View File

@ -2273,6 +2273,7 @@ class DiskFileMixin(BaseDiskFileTestMixin):
def test_disk_file_default_disallowed_metadata(self):
# build an object with some meta (at t0+1s)
orig_metadata = {'X-Object-Meta-Key1': 'Value1',
'X-Object-Transient-Sysmeta-KeyA': 'ValueA',
'Content-Type': 'text/garbage'}
df = self._get_open_disk_file(ts=self.ts().internal,
extra_metadata=orig_metadata)
@ -2281,6 +2282,7 @@ class DiskFileMixin(BaseDiskFileTestMixin):
# write some new metadata (fast POST, don't send orig meta, at t0+1)
df = self._simple_get_diskfile()
df.write_metadata({'X-Timestamp': self.ts().internal,
'X-Object-Transient-Sysmeta-KeyB': 'ValueB',
'X-Object-Meta-Key2': 'Value2'})
df = self._simple_get_diskfile()
with df.open():
@ -2288,8 +2290,11 @@ class DiskFileMixin(BaseDiskFileTestMixin):
self.assertEqual('text/garbage', df._metadata['Content-Type'])
# original fast-post updateable keys are removed
self.assertTrue('X-Object-Meta-Key1' not in df._metadata)
self.assertNotIn('X-Object-Transient-Sysmeta-KeyA', df._metadata)
# new fast-post updateable keys are added
self.assertEqual('Value2', df._metadata['X-Object-Meta-Key2'])
self.assertEqual('ValueB',
df._metadata['X-Object-Transient-Sysmeta-KeyB'])
def test_disk_file_preserves_sysmeta(self):
# build an object with some meta (at t0)

75
test/unit/obj/test_server.py
View File

@ -1428,7 +1428,8 @@ class TestObjectController(unittest.TestCase):
'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One',
'X-Object-Sysmeta-Two': 'Two'})
'X-Object-Sysmeta-Two': 'Two',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
req.body = 'VERIFY SYSMETA'
resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 201)
@ -1447,7 +1448,8 @@ class TestObjectController(unittest.TestCase):
'name': '/a/c/o',
'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One',
'X-Object-Sysmeta-Two': 'Two'})
'X-Object-Sysmeta-Two': 'Two',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
def test_PUT_succeeds_with_later_POST(self):
ts_iter = make_timestamp_iter()
@ -1620,6 +1622,62 @@ class TestObjectController(unittest.TestCase):
resp = req.get_response(self.object_controller)
check_response(resp)
def test_POST_transient_sysmeta(self):
# check that diskfile transient system meta is changed by a POST
timestamp1 = normalize_timestamp(time())
req = Request.blank(
'/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'PUT'},
headers={'X-Timestamp': timestamp1,
'Content-Type': 'text/plain',
'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
req.body = 'VERIFY SYSMETA'
resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 201)
timestamp2 = normalize_timestamp(time())
req = Request.blank(
'/sda1/p/a/c/o', environ={'REQUEST_METHOD': 'POST'},
headers={'X-Timestamp': timestamp2,
'X-Object-Meta-1': 'Not One',
'X-Object-Sysmeta-1': 'Not One',
'X-Object-Transient-Sysmeta-Foo': 'Not Bar'})
resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 202)
# original .data file metadata should be unchanged
objfile = os.path.join(
self.testdir, 'sda1',
storage_directory(diskfile.get_data_dir(0), 'p',
hash_path('a', 'c', 'o')),
timestamp1 + '.data')
self.assertTrue(os.path.isfile(objfile))
self.assertEqual(open(objfile).read(), 'VERIFY SYSMETA')
self.assertDictEqual(diskfile.read_metadata(objfile),
{'X-Timestamp': timestamp1,
'Content-Length': '14',
'Content-Type': 'text/plain',
'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'name': '/a/c/o',
'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
# .meta file metadata should have only user meta items
metafile = os.path.join(
self.testdir, 'sda1',
storage_directory(diskfile.get_data_dir(0), 'p',
hash_path('a', 'c', 'o')),
timestamp2 + '.meta')
self.assertTrue(os.path.isfile(metafile))
self.assertDictEqual(diskfile.read_metadata(metafile),
{'X-Timestamp': timestamp2,
'name': '/a/c/o',
'X-Object-Meta-1': 'Not One',
'X-Object-Transient-Sysmeta-Foo': 'Not Bar'})
def test_PUT_then_fetch_system_metadata(self):
timestamp = normalize_timestamp(time())
req = Request.blank(
@ -1629,7 +1687,8 @@ class TestObjectController(unittest.TestCase):
'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One',
'X-Object-Sysmeta-Two': 'Two'})
'X-Object-Sysmeta-Two': 'Two',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
req.body = 'VERIFY SYSMETA'
resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 201)
@ -1648,6 +1707,8 @@ class TestObjectController(unittest.TestCase):
self.assertEqual(resp.headers['x-object-meta-1'], 'One')
self.assertEqual(resp.headers['x-object-sysmeta-1'], 'One')
self.assertEqual(resp.headers['x-object-sysmeta-two'], 'Two')
self.assertEqual(resp.headers['x-object-transient-sysmeta-foo'],
'Bar')
req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'})
@ -1668,7 +1729,8 @@ class TestObjectController(unittest.TestCase):
'ETag': '1000d172764c9dbc3a5798a67ec5bb76',
'X-Object-Meta-1': 'One',
'X-Object-Sysmeta-1': 'One',
'X-Object-Sysmeta-Two': 'Two'})
'X-Object-Sysmeta-Two': 'Two',
'X-Object-Transient-Sysmeta-Foo': 'Bar'})
req.body = 'VERIFY SYSMETA'
resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 201)
@ -1679,7 +1741,8 @@ class TestObjectController(unittest.TestCase):
headers={'X-Timestamp': timestamp2,
'X-Object-Meta-1': 'Not One',
'X-Object-Sysmeta-1': 'Not One',
'X-Object-Sysmeta-Two': 'Not Two'})
'X-Object-Sysmeta-Two': 'Not Two',
'X-Object-Transient-Sysmeta-Foo': 'Not Bar'})
resp = req.get_response(self.object_controller)
self.assertEqual(resp.status_int, 202)
@ -1698,6 +1761,8 @@ class TestObjectController(unittest.TestCase):
self.assertEqual(resp.headers['x-object-meta-1'], 'Not One')
self.assertEqual(resp.headers['x-object-sysmeta-1'], 'One')
self.assertEqual(resp.headers['x-object-sysmeta-two'], 'Two')
self.assertEqual(resp.headers['x-object-transient-sysmeta-foo'],
'Not Bar')
req = Request.blank('/sda1/p/a/c/o',
environ={'REQUEST_METHOD': 'HEAD'})

13
test/unit/proxy/controllers/test_base.py
View File

@ -31,7 +31,9 @@ from swift.common.http import is_success
from swift.common.storage_policy import StoragePolicy
from test.unit import fake_http_connect, FakeRing, FakeMemcache
from swift.proxy import server as proxy_server
from swift.common.request_helpers import get_sys_meta_prefix
from swift.common.request_helpers import (
get_sys_meta_prefix, get_object_transient_sysmeta_prefix
)
from test.unit import patch_policies
@ -598,6 +600,15 @@ class TestFuncs(unittest.TestCase):
self.assertEqual(resp['sysmeta']['whatevs'], 14)
self.assertEqual(resp['sysmeta']['somethingelse'], 0)
def test_headers_to_object_info_transient_sysmeta(self):
prefix = get_object_transient_sysmeta_prefix()
headers = {'%sWhatevs' % prefix: 14,
'%ssomethingelse' % prefix: 0}
resp = headers_to_object_info(headers.items(), 200)
self.assertEqual(len(resp['transient_sysmeta']), 2)
self.assertEqual(resp['transient_sysmeta']['whatevs'], 14)
self.assertEqual(resp['transient_sysmeta']['somethingelse'], 0)
def test_headers_to_object_info_values(self):
headers = {
'content-length': '1024',

110
test/unit/proxy/test_sysmeta.py
View File

@ -26,6 +26,7 @@ from swift.common.wsgi import monkey_patch_mimetools, WSGIContext
from swift.obj import server as object_server
from swift.proxy import server as proxy
import swift.proxy.controllers
from swift.proxy.controllers.base import get_object_info
from test.unit import FakeMemcache, debug_logger, FakeRing, \
fake_http_connect, patch_policies
@ -169,6 +170,15 @@ class TestObjectSysmeta(unittest.TestCase):
'x-object-meta-test1': 'meta1 changed'}
new_meta_headers = {'x-object-meta-test3': 'meta3'}
bad_headers = {'x-account-sysmeta-test1': 'bad1'}
# these transient_sysmeta headers get changed...
original_transient_sysmeta_headers_1 = \
{'x-object-transient-sysmeta-testA': 'A'}
# these transient_sysmeta headers get deleted...
original_transient_sysmeta_headers_2 = \
{'x-object-transient-sysmeta-testB': 'B'}
changed_transient_sysmeta_headers = \
{'x-object-transient-sysmeta-testA': 'changed_A'}
new_transient_sysmeta_headers = {'x-object-transient-sysmeta-testC': 'C'}
def test_PUT_sysmeta_then_GET(self):
path = '/v1/a/c/o'
@ -177,6 +187,7 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs = dict(self.original_sysmeta_headers_1)
hdrs.update(self.original_meta_headers_1)
hdrs.update(self.bad_headers)
hdrs.update(self.original_transient_sysmeta_headers_1)
req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(self.app)
self._assertStatus(resp, 201)
@ -186,6 +197,7 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertStatus(resp, 200)
self._assertInHeaders(resp, self.original_sysmeta_headers_1)
self._assertInHeaders(resp, self.original_meta_headers_1)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_1)
self._assertNotInHeaders(resp, self.bad_headers)
def test_PUT_sysmeta_then_HEAD(self):
@ -195,6 +207,7 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs = dict(self.original_sysmeta_headers_1)
hdrs.update(self.original_meta_headers_1)
hdrs.update(self.bad_headers)
hdrs.update(self.original_transient_sysmeta_headers_1)
req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(self.app)
self._assertStatus(resp, 201)
@ -205,6 +218,7 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertStatus(resp, 200)
self._assertInHeaders(resp, self.original_sysmeta_headers_1)
self._assertInHeaders(resp, self.original_meta_headers_1)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_1)
self._assertNotInHeaders(resp, self.bad_headers)
def test_sysmeta_replaced_by_PUT(self):
@ -304,6 +318,8 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs.update(self.original_sysmeta_headers_2)
hdrs.update(self.original_meta_headers_1)
hdrs.update(self.original_meta_headers_2)
hdrs.update(self.original_transient_sysmeta_headers_1)
hdrs.update(self.original_transient_sysmeta_headers_2)
req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(self.app)
self._assertStatus(resp, 201)
@ -313,6 +329,8 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs.update(self.new_sysmeta_headers)
hdrs.update(self.changed_meta_headers)
hdrs.update(self.new_meta_headers)
hdrs.update(self.changed_transient_sysmeta_headers)
hdrs.update(self.new_transient_sysmeta_headers)
hdrs.update(self.bad_headers)
hdrs.update({'Destination': dest})
req = Request.blank(path, environ=env, headers=hdrs)
@ -324,6 +342,9 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertInHeaders(resp, self.changed_meta_headers)
self._assertInHeaders(resp, self.new_meta_headers)
self._assertInHeaders(resp, self.original_meta_headers_2)
self._assertInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertInHeaders(resp, self.new_transient_sysmeta_headers)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_2)
self._assertNotInHeaders(resp, self.bad_headers)
req = Request.blank('/v1/a/c/o2', environ={})
@ -335,6 +356,9 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertInHeaders(resp, self.changed_meta_headers)
self._assertInHeaders(resp, self.new_meta_headers)
self._assertInHeaders(resp, self.original_meta_headers_2)
self._assertInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertInHeaders(resp, self.new_transient_sysmeta_headers)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_2)
self._assertNotInHeaders(resp, self.bad_headers)
def test_sysmeta_updated_by_COPY_from(self):
@ -345,6 +369,8 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs.update(self.original_sysmeta_headers_2)
hdrs.update(self.original_meta_headers_1)
hdrs.update(self.original_meta_headers_2)
hdrs.update(self.original_transient_sysmeta_headers_1)
hdrs.update(self.original_transient_sysmeta_headers_2)
req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(self.app)
self._assertStatus(resp, 201)
@ -354,6 +380,8 @@ class TestObjectSysmeta(unittest.TestCase):
hdrs.update(self.new_sysmeta_headers)
hdrs.update(self.changed_meta_headers)
hdrs.update(self.new_meta_headers)
hdrs.update(self.changed_transient_sysmeta_headers)
hdrs.update(self.new_transient_sysmeta_headers)
hdrs.update(self.bad_headers)
hdrs.update({'X-Copy-From': '/c/o'})
req = Request.blank('/v1/a/c/o2', environ=env, headers=hdrs, body='')
@ -365,6 +393,9 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertInHeaders(resp, self.changed_meta_headers)
self._assertInHeaders(resp, self.new_meta_headers)
self._assertInHeaders(resp, self.original_meta_headers_2)
self._assertInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertInHeaders(resp, self.new_transient_sysmeta_headers)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_2)
self._assertNotInHeaders(resp, self.bad_headers)
req = Request.blank('/v1/a/c/o2', environ={})
@ -376,4 +407,83 @@ class TestObjectSysmeta(unittest.TestCase):
self._assertInHeaders(resp, self.changed_meta_headers)
self._assertInHeaders(resp, self.new_meta_headers)
self._assertInHeaders(resp, self.original_meta_headers_2)
self._assertInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertInHeaders(resp, self.new_transient_sysmeta_headers)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_2)
self._assertNotInHeaders(resp, self.bad_headers)
def _test_transient_sysmeta_replaced_by_PUT_or_POST(self):
# check transient_sysmeta is replaced en-masse by a POST
path = '/v1/a/c/o'
env = {'REQUEST_METHOD': 'PUT'}
hdrs = dict(self.original_transient_sysmeta_headers_1)
hdrs.update(self.original_transient_sysmeta_headers_2)
hdrs.update(self.original_meta_headers_1)
req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(self.app)
self._assertStatus(resp, 201)
req = Request.blank(path, environ={})
resp = req.get_response(self.app)
self._assertStatus(resp, 200)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_1)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_2)
self._assertInHeaders(resp, self.original_meta_headers_1)
info = get_object_info(req.environ, self.app)
self.assertEqual(2, len(info.get('transient_sysmeta', ())))
self.assertEqual({'testa': 'A', 'testb': 'B'},
info['transient_sysmeta'])
# POST will replace all existing transient_sysmeta and usermeta values
env = {'REQUEST_METHOD': 'POST'}
hdrs = dict(self.changed_transient_sysmeta_headers)
hdrs.update(self.new_transient_sysmeta_headers)
req = Request.blank(path, environ=env, headers=hdrs)
resp = req.get_response(self.app)
self._assertStatus(resp, 202)
req = Request.blank(path, environ={})
resp = req.get_response(self.app)
self._assertStatus(resp, 200)
self._assertNotInHeaders(resp, self.original_meta_headers_2)
self._assertInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertInHeaders(resp, self.new_transient_sysmeta_headers)
self._assertNotInHeaders(resp,
self.original_transient_sysmeta_headers_2)
info = get_object_info(req.environ, self.app)
self.assertEqual(2, len(info.get('transient_sysmeta', ())))
self.assertEqual({'testa': 'changed_A', 'testc': 'C'},
info['transient_sysmeta'])
# subsequent PUT replaces all transient_sysmeta and usermeta values
env = {'REQUEST_METHOD': 'PUT'}
hdrs = dict(self.original_transient_sysmeta_headers_2)
hdrs.update(self.original_meta_headers_2)
req = Request.blank(path, environ=env, headers=hdrs, body='x')
resp = req.get_response(self.app)
self._assertStatus(resp, 201)
req = Request.blank(path, environ={})
resp = req.get_response(self.app)
self._assertStatus(resp, 200)
self._assertInHeaders(resp, self.original_meta_headers_2)
self._assertInHeaders(resp, self.original_transient_sysmeta_headers_2)
self._assertNotInHeaders(resp, self.changed_meta_headers)
self._assertNotInHeaders(resp, self.new_meta_headers)
self._assertNotInHeaders(resp, self.changed_transient_sysmeta_headers)
self._assertNotInHeaders(resp, self.new_transient_sysmeta_headers)
info = get_object_info(req.environ, self.app)
self.assertEqual(1, len(info.get('transient_sysmeta', ())))
self.assertEqual({'testb': 'B'}, info['transient_sysmeta'])
def test_transient_sysmeta_replaced_by_POST(self):
self.app.object_post_as_copy = False
self._test_transient_sysmeta_replaced_by_PUT_or_POST()
def test_transient_sysmeta_replaced_by_POST_as_copy(self):
self.app.object_post_as_copy = True
self._test_transient_sysmeta_replaced_by_PUT_or_POST()