We've been working toward separating our logger from our statsd client.
This is generally a good idea; it's always been a little weird to have
our special-case loggers that would allow you to *also* increment some
counters.
The end goal is to take a bunch of places that look like
logger = utils.get_logger(conf)
...
logger.info(...)
logger.increment(...)
and turn them into something more like
logger = logs.get_adapted_logger(conf)
stats = statsd_client.get_statsd_client(conf, logger=logger)
...
logger.info(...)
stats.increment(...)
Take a lesson from logging: callers don't need to know whether the
log_level is high enough that their message will be logged, or even
whether logging is enabled at all. Code wanting to emit stats shouldn't
need to know whether statsd collection has been configured, either.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I6eb5b27a387cc2b7310ee11cc49d38fd2b6cbab8
- Move statsd client into it's own module
- Move all logging functions into their own module
- Move all config functions into their own module
- Move all helper functions into their own module
Partial-Bug: #2015274
Change-Id: Ic4b5005e3efffa8dba17d91a41e46d5c68533f9a
We stuff the access key into the request path until we get back a
more-authoritative account name from auth. But it needs to be a WSGI
string when we do!
Closes-Bug: #2058748
Change-Id: I34adb8141cc9e62d17a27f01c63f40d1dd25991c
Previously, the set_statsd_prefix method was used to mutate a logger's
StatsdClient tail prefix after a logger was instantiated. This pattern
had led to unexpected mutations (see Related-Change). The tail_prefix
can now be passed as an argument to get_logger(), and is then
forwarded to the StatsdClient constructor, for a more explicit
assignment pattern.
The set_statsd_prefix method is left in place for backwards
compatibility. A DeprecationWarning will be raised if it is used
to mutate the StatsdClient tail prefix.
Change-Id: I7692860e3b741e1bc10626e26bb7b27399c325ab
Related-Change: I0522b1953722ca96021a0002cf93432b973ce626
- add a message saying the reason for the value error exception
- add a unit test to validate the expected message
Change-Id: I1d6cc0faa3a43852c46089e509d48cc3ee9f9cf8
Closes-Bug: #1911811
Note that the user DB is stored in-memory as native strings, so we do
some crazy-decoding to make comparisons for auth decisions. Seems to
keep the config handling mostly sane, though.
I maybe need to look harder at container ACLs?
Change-Id: Ia58698c9b30d2211eeee8ecb3bbdd1c26fa4034d
While s3api is the preferred method going forward, it would be nice to
allow for backwards compatibility with swift3 in the tempauth
middleware.
Change-Id: I4fd4772b1d7b173ee0faf72d1c1f1531646dde8b
Instead, require that callers provide an encoding.
Related-Change: I31408f525ba9836f634a35581d4aee6fa2c9428f
Change-Id: I3e5ed9e4401eea76c375bb43ad4afc58b1d8006a
Taking user_tést_tester and sending back a link to http://saio:8080/v1/AUTH_tést
may work, but it would be better as http://saio:8080/v1/AUTH_t%C3%A9st
This is particularly important if you define something like user_test%ff_tester.
Change-Id: I19d21af94c21fccb4fb835acae231dec424790bb
Related-Bug: 1774238
This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.
1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)
2. Port unittests to test/unit/s3api dir to be able to run on the gate.
3. Port functests to test/functional/s3api and setup in-process testing
4. Port docs to doc dir, then address the namespace change.
5. Use get_logger() instead of global logger instance
6. Avoid global conf instance
Ex. fix various minor issue on those steps (e.g. packages, dependencies,
deprecated things)
The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)
Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.
Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Unicode characters in tempauth user names currently prevent these users
from authenticating. This manifests in two places:
1. the call to get_groups() fails to return a set that matches the user
preventing authentication
2. a new token is generated every time for such users, as the check
against existing memcache entries always fails to match
The patch fixes the existing behavior with respect to unicode users. As
part of it, the FakeMemcache class had to change to decode the strings
passed to it to mimic the actual memcache client behavior.
Darrell Bishop developed the patch and Timur Alperovich added the tests
for it.
Change-Id: I4ecfae2bca6ffa08ad15e584579ebce707f4628d
This adds support for v4 while getting us out of needing to know
how signatures work.
Related-Change: Iafb6114c12deb9a40d0f8324611de27b48ed95f6
Change-Id: I14be2845101f6af8f73bc46a416c09e4b9449515
Use assertIsNone() instead of assertEqual(), because assertEqual()
still fails on false values when compared to None
Change-Id: Ic52c319e3e55135df834fdf857982e1721bc44bb
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)
Change-Id: If4db8872c4f5705c1fff017c4891626e9ce4d1e4
Previously, Swift3 used client-facing HTTP headers to pass the S3 access
key, signature, and normalized request through the WSGI pipeline.
However, tempauth did not validate that Swift3 actually set the headers;
as a result, an attacker who has captured either a single valid S3-style
temporary URL or a single valid request through the S3 API may impersonate
the user that signed the URL or issued the request indefinitely through
the Swift API.
Now, the S3 authentication information will be taken from a separate
namespace in the WSGI environment, completely inaccessible to the
client. Specifically,
environ['swift3.auth_details'] = {
'access_key': <access key>,
'signature': <signature>,
'string_to_sign': <normalized request>,
}
Note that tempauth is not expected to be in production use, but may have
been used as a template by other authentication middlewares to add their
own Swift3 support.
Change-Id: Ib90adcc2f059adaf203fba1c95b2154561ea7487
Related-Change: Ia3fbb4938f0daa8845cba4137a01cc43bc1a713c
As swift no longer supports Python 2.6, replace assertEqual(None, *)
with assertIsNone in tests to have more clear messages in case of
failure.
Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56
Closes-Bug: #1280522
Previously, we gave no indication of when a token would expire. Users
would have to just use it until it stopped working, then re-auth.
Now, a successful auth response will include a new header,
X-Auth-Token-Expires, with the number of seconds remaining until the
token is invalid. This allows the client to attempt to re-auth before
sending a request that will definitely fail.
For comparison, swauth already uses the X-Auth-Token-Expires header with
identical semantics. Additionally, Keystone (v2 and v3) already exposes
expiration times in its JSON responses. The security impact should be
minimal.
Change-Id: I5a4a74276bc0df6dda94e4bc150065c0d77de0eb
contextlib.nested() is missing completely in Python 3.
Since 2.7, we can use multiple context managers in a 'with' statement,
like so:
with thing1() as t1, thing2() as t2:
do_stuff()
Now, if we had some code that needed to nest an arbitrary number of
context managers, there's stuff we could do with contextlib.ExitStack
and such... but we don't. We only use contextlib.nested() in tests to
set up bunches of mocks without crazy-deep indentation, and all that
stuff fits perfectly into multiple-context-manager 'with' statements.
Change-Id: Id472958b007948f05dbd4c7fb8cf3ffab58e2681
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
The Python 2 next() method of iterators was renamed to __next__() on
Python 3. Use the builtin next() function instead which works on Python
2 and Python 3.
Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
As documented in linked bug report, tempauth unit tests
were seen to fail on a system where simplejson was
installed but without the speedups extension. This
is because the tempauth account acl validation checks
that values are type str, but without the speedups
extension the json parser is returning unicode objects.
Fix is to have the acl validator tolerate those objects
being unicode or str.
Also change common/bufferedhttp.py to coerce ring device
to type str when constructing a path, in order to avoid
a UnicodeDecodeError when httplib sends a message that
has non-ascii header values.
Change-Id: I01524282cbaa25dc4b6dfa09f3f4723516cdba99
Closes-Bug: 1425776
This fixes a kind of trivial issue with TypeError caused by a lack
of method argument at calling.
When using 'Authorization' header with tempauth, tempauth calls
_get_user_groups with 3 arguments (includes self) on current code
but the method requires 4 arguments so it will go to fail with 500.
This patch fixes to take correct arguments and adds an unit test
for the case.
Closes-Bug:1434465
Change-Id: Ibf00c41fa8e5cfdaf49bc4d571e575d7f229ebfe
Our functional tests are skipping Account ACL tests when using tempauth
because it does not register itself with in swift.common.utils._swift_info
Add the register_swift_info call back to the filter_factory function in the
tempauth from which it was removed, and add a test to validate creating the
filter will add the middleware to global info dict.
Change-Id: I3761da3241601c93245796beb5d29bf3f36d69f6
Sending an account POST with an X-Account-Access-Control
header value that is valid json but not a valid ACL was
causing a 500 Internal Error if the value did not parse
to a dict due to an exception being raised in tempauth.py.
This patch modifies acl.py to check that the header value
is both json and parses to a dict. The existing
tests are extended to cover these invalid header values.
This patch also enables json encoded dicts with whitespace
(e.g. '{ }') to be accepted as a value for
X-Account-Access-Control in the same way that '{}' is.
These previously resulted in a 400 response.
Closes-bug: 1281626
Change-Id: Ia06ba9c9d16f749f801a8158e73d3898c4a42888
* Introduce a new privileged account header: X-Account-Access-Control
* Introduce JSON-based version 2 ACL syntax -- see below for discussion
* Implement account ACL authorization in TempAuth
X-Account-Access-Control Header
-------------------------------
Accounts now have a new privileged header to represent ACLs or any other
form of account-level access control. The value of the header is an opaque
string to be interpreted by the auth system, but it must be a JSON-encoded
dictionary. A reference implementation is given in TempAuth, with the
knowledge that historically other auth systems often use TempAuth as a
starting point.
The reference implementation describes three levels of account access:
"admin", "read-write", and "read-only". Adding new access control
features in a future patch (e.g. "write-only" account access) will
automatically be forward- and backward-compatible, due to the JSON
dictionary header format.
The privileged X-Account-Access-Control header may only be read or written
by a user with "swift_owner" status, traditionally the account owner but
now also any user on the "admin" ACL.
Access Levels:
Read-only access is intended to indicate to the auth system that this
list of identities can read everything (except privileged headers) in
the account. Specifically, a user with read-only account access can get
a list of containers in the account, list the contents of any container,
retrieve any object, and see the (non-privileged) headers of the
account, any container, or any object.
Read-write access is intended to indicate to the auth system that this
list of identities can read or write (or create) any container. A user
with read-write account access can create new containers, set any
unprivileged container headers, overwrite objects, delete containers,
etc. A read-write user can NOT set account headers (or perform any
PUT/POST/DELETE requests on the account).
Admin access is intended to indicate to the auth system that this list of
identities has "swift_owner" privileges. A user with admin account access
can do anything the account owner can, including setting account headers
and any privileged headers -- and thus changing the value of
X-Account-Access-Control and thereby granting read-only, read-write, or
admin access to other users.
The auth system is responsible for making decisions based on this header,
if it chooses to support its use. Therefore the above access level
descriptions are necessarily advisory only for other auth systems.
When setting the value of the header, callers are urged to use the new
format_acl() method, described below.
New ACL Format
--------------
The account ACLs introduce a new format for ACLs, rather than reusing the
existing format from X-Container-Read/X-Container-Write. There are several
reasons for this:
* Container ACL format does not support Unicode
* Container ACLs have a different structure than account ACLs
+ account ACLs have no concept of referrers or rlistings
+ accounts have additional "admin" access level
+ account access levels are structured as admin > rw > ro, which seems more
appropriate for how people access accounts, rather than reusing
container ACLs' orthogonal read and write access
In addition, the container ACL syntax is a bit arbitrary and highly custom,
so instead of parsing additional custom syntax, I'd rather propose a next
version and introduce a means for migration. The V2 ACL syntax has the
following benefits:
* JSON is a well-known standard syntax with parsers in all languages
* no artificial value restrictions (you can grant access to a user named
".rlistings" if you want)
* forward and backward compatibility: you may have extraneous keys, but
your attempt to parse the header won't raise an exception
I've introduced hooks in parse_acl and format_acl which currently default
to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
to default to V2. I'm not changing the default or adding code to rewrite
V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
but this seems like a sensible milestone in the migration.
TempAuth Account ACL Implementation
-----------------------------------
As stated above, core Swift is responsible for privileging the
X-Account-Access-Control header (making it only accessible to swift_owners),
for translating it to -sysmeta-* headers to trigger persistence by the
account server, and for including the header in the responses to requests
by privileged users. Core Swift puts no expectation on the *content* of
this header. Auth systems (including TempAuth) are responsible for
defining the content of the header and taking action based on it.
In addition to the changes described above, this patch defines a format
to be used by TempAuth for these headers in the common.middleware.acl
module, in the methods format_v2_acl() and parse_v2_acl(). This patch
also teaches TempAuth to take action based on the header contents. TempAuth
now sets swift_owner=True if the user is on the Admin ACL, authorizes
GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.
Note that the action of setting swift_owner=True triggers core Swift to
add or strip the privileged headers from the responses. Core Swift (not
the auth system) is responsible for that.
DocImpact: Documentation for the new ACL usage and format appears in
summary form in doc/source/overview_auth.rst, and in more detail in
swift/common/middleware/tempauth.py in the TempAuth class docstring.
I leave it to the Swift doc team to determine whether more is needed.
Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826
When tempauth is used together with swift3 and keystone, groups are
empty and tempauth denies the request too early without a chance for
keystone to authenticate the request.
Change-Id: I21d9b22ecbd18a5f1fba901abd94221a332c45ea
Closes-Bug: 1244545
Per http://www.ietf.org/rfc/rfc2616.txt, when a 401 error is returned, the
Www-Authenticate response header MUST also be returned. The format is
described in http://www.ietf.org/rfc/rfc2617.txt.
Swift supports and/or implements a number of authentication schemes
including tempauth, Keystone, tempurl, formpost and container sync. In
this fix, we use a catch-all, "Swift". The realm is the account (where
known) or "unknown" (bad path or where the 401 is returned from code
that does not have the request). Examples:
Www-Authenticate: Swift realm="AUTH_1234567889"
Www-Authenticate: Swift realm="unknown"
Fixes bug #1215491
Change-Id: I03362789318dfa156d3733ef9348795062a9cfc4
Address all the "hacking" lines that are flagged, and all the modules
that just have one item flagged.
Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b
Signed-off-by: Peter Portante <peter.portante@redhat.com>
In current codes, tempauth token will be updated only when it expires. In
token's lifetime, any changes in tempauth config file doesn't update token,
which means if you update your config file and restart proxy server, the
role in runtime might not change. So a good fix is updating token when
it changes or expires.
fixes bug #1201678
Change-Id: Ieb080e87268e16d0a2e5896431aa41fcedc7a900
When checking referer (.r, .rlistings), tempauth return 401/403 if '.r'
exists without '.rlistings' or obj. Actually it should skip this kind of
authorizing and check the following authorizing.
And remove some unnecessary lines
Change-Id: Ib60e59b7c80331a699cb5c3496f948cf32135d0b
The recent account_quotas (https://review.openstack.org/23434)
patch added a new setting request.environ[reseller_request].
This patch adds tests for tempauth and keystoneauth as well as
an updated overview_auth.rst.
Change-Id: Icdb7ec9948ae7424b0721fc51a143782b2fdc5a6
Tim Burke