We stuff the access key into the request path until we get back a
more-authoritative account name from auth. But it needs to be a WSGI
string when we do!
Closes-Bug: #2058748
Change-Id: I34adb8141cc9e62d17a27f01c63f40d1dd25991c
Previously, the set_statsd_prefix method was used to mutate a logger's
StatsdClient tail prefix after a logger was instantiated. This pattern
had led to unexpected mutations (see Related-Change). The tail_prefix
can now be passed as an argument to get_logger(), and is then
forwarded to the StatsdClient constructor, for a more explicit
assignment pattern.
The set_statsd_prefix method is left in place for backwards
compatibility. A DeprecationWarning will be raised if it is used
to mutate the StatsdClient tail prefix.
Change-Id: I7692860e3b741e1bc10626e26bb7b27399c325ab
Related-Change: I0522b1953722ca96021a0002cf93432b973ce626
The *_swift_info functions use in module global dicts to provide a
registry mechanism for registering and getting swift info.
This is an abnormal pattern and doesn't quite fit into utils. Further
we looking at following this pattern for sensitive info to trim in the
future.
So this patch does some house cleaning and moves this registry to a new
module swift.common.registry. And updates all the references to it.
For backwards compat we still import the *_swift_info methods into utils
for any 3rd party tools or middleware.
Change-Id: I71fd7f50d1aafc001d6905438f42de4e58af8421
- add a message saying the reason for the value error exception
- add a unit test to validate the expected message
Change-Id: I1d6cc0faa3a43852c46089e509d48cc3ee9f9cf8
Closes-Bug: #1911811
The repo is Python using both Python 2 and 3 now, so update hacking to
version 2.0 which supports Python 2 and 3. Note that latest hacking
release 3.0 only supports version 3.
Fix problems found.
Remove hacking and friends from lower-constraints, they are not needed
for installation.
Change-Id: I9bd913ee1b32ba1566c420973723296766d1812f
Note that the user DB is stored in-memory as native strings, so we do
some crazy-decoding to make comparisons for auth decisions. Seems to
keep the config handling mostly sane, though.
I maybe need to look harder at container ACLs?
Change-Id: Ia58698c9b30d2211eeee8ecb3bbdd1c26fa4034d
While s3api is the preferred method going forward, it would be nice to
allow for backwards compatibility with swift3 in the tempauth
middleware.
Change-Id: I4fd4772b1d7b173ee0faf72d1c1f1531646dde8b
Instead, require that callers provide an encoding.
Related-Change: I31408f525ba9836f634a35581d4aee6fa2c9428f
Change-Id: I3e5ed9e4401eea76c375bb43ad4afc58b1d8006a
Taking user_tést_tester and sending back a link to http://saio:8080/v1/AUTH_tést
may work, but it would be better as http://saio:8080/v1/AUTH_t%C3%A9st
This is particularly important if you define something like user_test%ff_tester.
Change-Id: I19d21af94c21fccb4fb835acae231dec424790bb
Related-Bug: 1774238
The first item in groups is the account, the second item is the account_user.
The second one is *way* more useful for debugging.
Change-Id: I12eb55a42e5a8dc2dff5a4687f7fe31452f9508e
This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.
1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)
2. Port unittests to test/unit/s3api dir to be able to run on the gate.
3. Port functests to test/functional/s3api and setup in-process testing
4. Port docs to doc dir, then address the namespace change.
5. Use get_logger() instead of global logger instance
6. Avoid global conf instance
Ex. fix various minor issue on those steps (e.g. packages, dependencies,
deprecated things)
The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)
Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.
Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Unicode characters in tempauth user names currently prevent these users
from authenticating. This manifests in two places:
1. the call to get_groups() fails to return a set that matches the user
preventing authentication
2. a new token is generated every time for such users, as the check
against existing memcache entries always fails to match
The patch fixes the existing behavior with respect to unicode users. As
part of it, the FakeMemcache class had to change to decode the strings
passed to it to mimic the actual memcache client behavior.
Darrell Bishop developed the patch and Timur Alperovich added the tests
for it.
Change-Id: I4ecfae2bca6ffa08ad15e584579ebce707f4628d
This adds support for v4 while getting us out of needing to know
how signatures work.
Related-Change: Iafb6114c12deb9a40d0f8324611de27b48ed95f6
Change-Id: I14be2845101f6af8f73bc46a416c09e4b9449515
Instead, link to the middleware list and auth overview, as well as
referring readers to proxy-server.conf-sample
TempAuth-related content that was previously in the deployment guide has
been moved to TempAuth's own docs, which have been cleaned up a bit.
Change-Id: I00070bb09294362c069f7ee9426ac570bc1b3ddb
Previously, Swift3 used client-facing HTTP headers to pass the S3 access
key, signature, and normalized request through the WSGI pipeline.
However, tempauth did not validate that Swift3 actually set the headers;
as a result, an attacker who has captured either a single valid S3-style
temporary URL or a single valid request through the S3 API may impersonate
the user that signed the URL or issued the request indefinitely through
the Swift API.
Now, the S3 authentication information will be taken from a separate
namespace in the WSGI environment, completely inaccessible to the
client. Specifically,
environ['swift3.auth_details'] = {
'access_key': <access key>,
'signature': <signature>,
'string_to_sign': <normalized request>,
}
Note that tempauth is not expected to be in production use, but may have
been used as a template by other authentication middlewares to add their
own Swift3 support.
Change-Id: Ib90adcc2f059adaf203fba1c95b2154561ea7487
Related-Change: Ia3fbb4938f0daa8845cba4137a01cc43bc1a713c
Previously, we gave no indication of when a token would expire. Users
would have to just use it until it stopped working, then re-auth.
Now, a successful auth response will include a new header,
X-Auth-Token-Expires, with the number of seconds remaining until the
token is invalid. This allows the client to attempt to re-auth before
sending a request that will definitely fail.
For comparison, swauth already uses the X-Auth-Token-Expires header with
identical semantics. Additionally, Keystone (v2 and v3) already exposes
expiration times in its JSON responses. The security impact should be
minimal.
Change-Id: I5a4a74276bc0df6dda94e4bc150065c0d77de0eb
Updated string prefix and suffix checker slicing to startswith()
and endswith() methods.
Using startswith() and endswith() improves readability, error-proneness
and enhances maintainability.
Change-Id: I1d5fbf116a61763346c6f92fd8023dbbe9bb37cf
The doc for these sections was missing because of an rst error - the
source is there in rst file but didn't make it into the html output.
Add doc for per_diff and max_diffs in account and container doc sections.
Also, fix a bunch of other sphinx build errors and most of the warnings.
Change-Id: If9ed2619b2f92c6c65a94f41d8819db8726d3893
The builtin basestring type was removed in Python 3. Replace it with
six.string_types which works on Python 2 and Python 3.
Change-Id: Ib92a729682322cc65b41050ae169167be2899e2c
The urllib, urllib2 and urlparse modules of Python 2 were reorganized
into a new urllib namespace on Python 3. Replace urllib, urllib2 and
urlparse imports with six.moves.urllib to make the modified code
compatible with Python 2 and Python 3.
The initial patch was generated by the urllib operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93
In case of a COPY request the swift_owner was already set to True, and the
following PUT request was granted access no matter if a service token was used
or not. This allowed to copy data to service accounts without any service
token.
Service token unit tests have been added to verify that when
swift_owner is set to True in a request environ, this setting is
ignored when authorizing another request based on the same
environ. Applying only this test change on master fails currently, and
only passes with the fix in this patch.
Tempauth seems to be not affected, however a small doc update has been added to
make it more clear that a service token is not needed to access a service account
when an ACL is used.
Further details with an example are available in the bug report
(https://bugs.launchpad.net/swift/+bug/1483007).
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com>
Co-Authored-By: Donagh McCabe <donagh.mccabe@hp.com>
Closes-Bug: 1483007
Change-Id: I1207b911f018b855362b1078f68c38615be74bbd
Fix pep8 warnings of the E category of hacking 0.10:
* E113: unexpected indentation
* E121: continuation line under-indented for hanging indent
* E122: continuation line missing indentation or outdented
* E123: closing bracket does not match indentation of opening bracket's
line
* E126: continuation line over-indented for hanging indent
* E251: unexpected spaces around keyword / parameter equals
Change-Id: I0b24eebdf1a37dc1b572b6c9a3d3d4832d050237
As documented in linked bug report, tempauth unit tests
were seen to fail on a system where simplejson was
installed but without the speedups extension. This
is because the tempauth account acl validation checks
that values are type str, but without the speedups
extension the json parser is returning unicode objects.
Fix is to have the acl validator tolerate those objects
being unicode or str.
Also change common/bufferedhttp.py to coerce ring device
to type str when constructing a path, in order to avoid
a UnicodeDecodeError when httplib sends a message that
has non-ascii header values.
Change-Id: I01524282cbaa25dc4b6dfa09f3f4723516cdba99
Closes-Bug: 1425776
This fixes a kind of trivial issue with TypeError caused by a lack
of method argument at calling.
When using 'Authorization' header with tempauth, tempauth calls
_get_user_groups with 3 arguments (includes self) on current code
but the method requires 4 arguments so it will go to fail with 500.
This patch fixes to take correct arguments and adds an unit test
for the case.
Closes-Bug:1434465
Change-Id: Ibf00c41fa8e5cfdaf49bc4d571e575d7f229ebfe
Our functional tests are skipping Account ACL tests when using tempauth
because it does not register itself with in swift.common.utils._swift_info
Add the register_swift_info call back to the filter_factory function in the
tempauth from which it was removed, and add a test to validate creating the
filter will add the middleware to global info dict.
Change-Id: I3761da3241601c93245796beb5d29bf3f36d69f6
Sending an account POST with an X-Account-Access-Control
header value that is valid json but not a valid ACL was
causing a 500 Internal Error if the value did not parse
to a dict due to an exception being raised in tempauth.py.
This patch modifies acl.py to check that the header value
is both json and parses to a dict. The existing
tests are extended to cover these invalid header values.
This patch also enables json encoded dicts with whitespace
(e.g. '{ }') to be accepted as a value for
X-Account-Access-Control in the same way that '{}' is.
These previously resulted in a 400 response.
Closes-bug: 1281626
Change-Id: Ia06ba9c9d16f749f801a8158e73d3898c4a42888
* Introduce a new privileged account header: X-Account-Access-Control
* Introduce JSON-based version 2 ACL syntax -- see below for discussion
* Implement account ACL authorization in TempAuth
X-Account-Access-Control Header
-------------------------------
Accounts now have a new privileged header to represent ACLs or any other
form of account-level access control. The value of the header is an opaque
string to be interpreted by the auth system, but it must be a JSON-encoded
dictionary. A reference implementation is given in TempAuth, with the
knowledge that historically other auth systems often use TempAuth as a
starting point.
The reference implementation describes three levels of account access:
"admin", "read-write", and "read-only". Adding new access control
features in a future patch (e.g. "write-only" account access) will
automatically be forward- and backward-compatible, due to the JSON
dictionary header format.
The privileged X-Account-Access-Control header may only be read or written
by a user with "swift_owner" status, traditionally the account owner but
now also any user on the "admin" ACL.
Access Levels:
Read-only access is intended to indicate to the auth system that this
list of identities can read everything (except privileged headers) in
the account. Specifically, a user with read-only account access can get
a list of containers in the account, list the contents of any container,
retrieve any object, and see the (non-privileged) headers of the
account, any container, or any object.
Read-write access is intended to indicate to the auth system that this
list of identities can read or write (or create) any container. A user
with read-write account access can create new containers, set any
unprivileged container headers, overwrite objects, delete containers,
etc. A read-write user can NOT set account headers (or perform any
PUT/POST/DELETE requests on the account).
Admin access is intended to indicate to the auth system that this list of
identities has "swift_owner" privileges. A user with admin account access
can do anything the account owner can, including setting account headers
and any privileged headers -- and thus changing the value of
X-Account-Access-Control and thereby granting read-only, read-write, or
admin access to other users.
The auth system is responsible for making decisions based on this header,
if it chooses to support its use. Therefore the above access level
descriptions are necessarily advisory only for other auth systems.
When setting the value of the header, callers are urged to use the new
format_acl() method, described below.
New ACL Format
--------------
The account ACLs introduce a new format for ACLs, rather than reusing the
existing format from X-Container-Read/X-Container-Write. There are several
reasons for this:
* Container ACL format does not support Unicode
* Container ACLs have a different structure than account ACLs
+ account ACLs have no concept of referrers or rlistings
+ accounts have additional "admin" access level
+ account access levels are structured as admin > rw > ro, which seems more
appropriate for how people access accounts, rather than reusing
container ACLs' orthogonal read and write access
In addition, the container ACL syntax is a bit arbitrary and highly custom,
so instead of parsing additional custom syntax, I'd rather propose a next
version and introduce a means for migration. The V2 ACL syntax has the
following benefits:
* JSON is a well-known standard syntax with parsers in all languages
* no artificial value restrictions (you can grant access to a user named
".rlistings" if you want)
* forward and backward compatibility: you may have extraneous keys, but
your attempt to parse the header won't raise an exception
I've introduced hooks in parse_acl and format_acl which currently default
to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
to default to V2. I'm not changing the default or adding code to rewrite
V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
but this seems like a sensible milestone in the migration.
TempAuth Account ACL Implementation
-----------------------------------
As stated above, core Swift is responsible for privileging the
X-Account-Access-Control header (making it only accessible to swift_owners),
for translating it to -sysmeta-* headers to trigger persistence by the
account server, and for including the header in the responses to requests
by privileged users. Core Swift puts no expectation on the *content* of
this header. Auth systems (including TempAuth) are responsible for
defining the content of the header and taking action based on it.
In addition to the changes described above, this patch defines a format
to be used by TempAuth for these headers in the common.middleware.acl
module, in the methods format_v2_acl() and parse_v2_acl(). This patch
also teaches TempAuth to take action based on the header contents. TempAuth
now sets swift_owner=True if the user is on the Admin ACL, authorizes
GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.
Note that the action of setting swift_owner=True triggers core Swift to
add or strip the privileged headers from the responses. Core Swift (not
the auth system) is responsible for that.
DocImpact: Documentation for the new ACL usage and format appears in
summary form in doc/source/overview_auth.rst, and in more detail in
swift/common/middleware/tempauth.py in the TempAuth class docstring.
I leave it to the Swift doc team to determine whether more is needed.
Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826
Swift can now optionally be configured to allow requests to '/info',
providing information about the swift cluster. Additionally a HMAC
signed requests to
'/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be
configured allowing privileged access to more sensitive information
not meant to be public.
DocImpact
Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495
Implements: blueprint capabilities
Closes-Bug: #1245694
When tempauth is used together with swift3 and keystone, groups are
empty and tempauth denies the request too early without a chance for
keystone to authenticate the request.
Change-Id: I21d9b22ecbd18a5f1fba901abd94221a332c45ea
Closes-Bug: 1244545
Per http://www.ietf.org/rfc/rfc2616.txt, when a 401 error is returned, the
Www-Authenticate response header MUST also be returned. The format is
described in http://www.ietf.org/rfc/rfc2617.txt.
Swift supports and/or implements a number of authentication schemes
including tempauth, Keystone, tempurl, formpost and container sync. In
this fix, we use a catch-all, "Swift". The realm is the account (where
known) or "unknown" (bad path or where the 401 is returned from code
that does not have the request). Examples:
Www-Authenticate: Swift realm="AUTH_1234567889"
Www-Authenticate: Swift realm="unknown"
Fixes bug #1215491
Change-Id: I03362789318dfa156d3733ef9348795062a9cfc4
Tim Burke