Reserve the namespace starting with the NULL byte for internal
use-cases. Backend services will allow path names to include the NULL
byte in urls and validate names in the reserved namespace. Database
services will filter all names starting with the NULL byte from
responses unless the request includes the header:
X-Backend-Allow-Reserved-Names: true
The proxy server will not allow path names to include the NULL byte in
urls unless a middlware has set the X-Backend-Allow-Reserved-Names
header. Middlewares can use the reserved namespace to create objects
and containers that can not be directly manipulated by clients. Any
objects and bytes created in the reserved namespace will be aggregated
to the user's account totals.
When deploying internal proxys developers and operators may configure
the gatekeeper middleware to translate the X-Allow-Reserved-Names header
to the Backend header so they can manipulate the reserved namespace
directly through the normal API.
UpgradeImpact: it's not safe to rollback from this change
Change-Id: If912f71d8b0d03369680374e8233da85d8d38f85
Instead, require that callers provide an encoding.
Related-Change: I31408f525ba9836f634a35581d4aee6fa2c9428f
Change-Id: I3e5ed9e4401eea76c375bb43ad4afc58b1d8006a
There were a couple of cleanups in swob as part of this. First,
status lines are always native str objects (as PEP 3333 wants), rather
than being encoded to bytes under py3. Second, _resp_body_property
now works (only) with bytestrings from the app iter.
In gatekeeper, we now deal with dict.items() returning an object of
type "dict_items" in py3, not a list. Also fixed a NameError caused by
py2's list comprehensions leaking variables to function scope where
py3's don't.
Change-Id: I6da8eceb91edb2b47aa345d61b825c7199a5569b
Why we need this:
Some middlewares want to keep HTTP Location header as relative path
(e.g. using Load balancer in front of proxy).
What is the problem in current Swift:
Current Swift already has the flag to keep it as relative when returning
the reponse using swift.common.swob.Response. However, auth_token middleware,
that is from keystonemiddleware, unfortunately can change the relative path
to absolute because of using webob instead of swob.
What this patch is doing:
Make gate_keeper able to re-transform the location header from absolute path
to relative path if 'swift.leave_relative_location' is explicitely set because
gate_keeper should be the most left side middleware except catch_errors middleware
in the pipeline.
Change-Id: Ic634c3f1b1e26635206d5a54df8b15354e8df163
* Replace "c = dict(a.items() + b.items())" with
"c = dict(a); c.update(b)". It works on Python 2 and Python 3, and
it may be a little bit more efficient on Python 2 (no need to
create a temporary list of items).
* Replace "dict.values() + dict.values()" with
"list(dict.values()) + list(dict.values())": on Python 3,
dict.values() is a view which doesn't support a+b operator.
Change-Id: Id5a65628fe2fb7a02c713b758fcaa81154db28a0
Adds a new form of system metadata for objects.
Sysmeta cannot be updated by an object POST because
that would cause all existing sysmeta to be deleted.
Crypto middleware will want to add 'system' metadata
to object metadata on PUTs and POSTs, but it is ok
for this metadata to be replaced en-masse on every
POST.
This patch introduces x-object-transient-sysmeta-*
that is persisted by object servers and returned
in GET and HEAD responses, just like user metadata,
without polluting the x-object-meta-* namespace.
All headers in this namespace will be filtered
inbound and outbound by the gatekeeper, so cannot
be set or read by clients.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Change-Id: I5075493329935ba6790543fc82ea6e039704811d
With the X-Timestamp validation added in commit e619411, end users
could upload objects with
X-Timestamp: 9999999999.99999_ffffffffffffffff
(the maximum value) and Swift would be unable to delete them.
Now, inbound X-Timestamp headers will be moved to
X-Backend-Inbound-X-Timestamp, effectively rendering them harmless.
The primary reason to allow X-Timestamp before was to prevent
Last-Modified changes for objects coming from either:
* container_sync or
* a migration from another storage system.
To enable the former use-case, the container_sync middleware will now
translate X-Backend-Inbound-X-Timestamp headers back to X-Timestamp
after verifying the request.
Additionally, a new option is added to the gatekeeper filter config:
# shunt_inbound_x_timestamp = true
To enable the latter use-case (or any other use-case not mentioned), set
this to false.
Upgrade Consideration
=====================
If your cluster workload requires that clients be allowed to specify
objects' X-Timestamp values, disable the shunt_inbound_x_timestamp
option before upgrading.
UpgradeImpact
Change-Id: I8799d5eb2ae9d795ba358bb422f69c70ee8ebd2c
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.
There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.
Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
By moving the blocking to gatekeeper from the proxy server, we gain
the ability to pass X-Backend headers in via InternalClient while
still keeping real clients from using them.
I wanted this functionality while working on storage policies; I had
an InternalClient and wanted to tell it to use a specific policy index
instead of what the container said, and that seemed like a good time
for an X-Backend header.
Change-Id: I4089e980d3cfca660365c7df799723b1f16ba277
removes @wsgify decorator
changes code to a more traditional way of messing with response headers
Change-Id: I13ce7f98bae48eb0b990a64aa6f3d63ff8d13bf5
Closes-Bug: 1285392
Middleware or core features may need to store metadata
against accounts or containers. This patch adds a
generic mechanism for system metadata to be persisted
in backend databases, without polluting the user
metadata namespace, by using the reserved header
namespace x-<server_type>-sysmeta-*.
Modifications are firstly that backend servers persist
system metadata headers alongside user metadata and
other system state.
For accounts and containers, system metadata in PUT
and POST requests is treated in a similar way to user
metadata. System metadata is not yet supported for
object requests.
Secondly, changes in the proxy controllers ensure that
headers in the system metadata namespace will pass through
in requests to backend servers.
Thirdly, system metadata returned from backend servers
in GET or HEAD responses is added to the cached info
dict, which middleware can access.
Finally, a gatekeeper middleware module is provided
which filters all system metadata headers from requests
and responses by removing headers with names starting
x-account-sysmeta-, x-container-sysmeta-. The gatekeeper
also removes headers starting x-object-sysmeta- in
anticipation of future support for system metadata being
set for objects. This prevents clients from writing or
reading system metadata.
The required_filters list in swift/proxy/server.py is
modified to include the gatekeeper middleware so that
if the gatekeeper has not been configured in the
pipeline then it will be automatically inserted close
to the start of the pipeline.
blueprint cluster-federation
Change-Id: I80b8b14243cc59505f8c584920f8f527646b5f45
Clay Gerrard