Following PyCQA/astroid@206d8a2 we sarted getting a whole bunch of
errors like
E:266,44: Value 'headers' doesn't support membership test
and
E:267,25: Value 'headers' is unsubscriptable
Digging around a bit, apparently astroid thinks the headers returned
from call_app will always be None -- I guess it doesn't like our use of
a list to work around py2's lack of `nonlocal`.
By using a proper object to encapsulate state, we can shut up those
"error"s.
Also, pin upper-constraints to pike for keystone-related jobs.
Change-Id: I5ff21260872f4089b030cd94e494dc346ae74b8e
From RFC 7230 [1]:
> The presence of a message body in a request is signaled by a
> Content-Length or Transfer-Encoding header field.
Thus, if a client sent neither a Content-Length nor a Transfer-Encoding
header, we should assume there is no body and never read from
wsgi.input. In such a case, return an empty string unless the caller
has specified that a body is required.
This revealed a whole bunch of edge cases around our XML parsing:
* Attempting to PUT bucket ACLs while not actually supplying an ACL
should return a MissingSecurityHeader error.
* Attempting to perform a multi-delete without a body should return a
MissingRequestBody error.
* Attempting to complete a multi-part upload without a body should
return an InvalidRequest error.
[1] https://tools.ietf.org/html/rfc7230#section-3.3
Change-Id: I47ad946cff64a3da16a759a4364e6fe29b000e11
Closes-Bug: 1593870
ProxyFS is a soon-to-be-released project that provides a filesystem on
top of Swift, then also tries to provide the Swift API on top of that
filesystem. However, there are some places where
Swift-on-filesystem-on-Swift works differently from plain Swift, and
this is one of them.
When we make a multipart upload, it looks like this:
/v1/a/c+segments/obj/uploadId
/v1/a/c+segments/obj/uploadId/1
/v1/a/c+segments/obj/uploadId/2
...
The problem is that .../obj/uploadId is a file, and in
Swift-on-filesystem-on-Swift, that means you can't make another file
.../obj/uploadId/1, just like you can't on your local filesystem.
However, if you create .../obj/uploadId with a Content-Type of
application/directory and a Content-Length of 0, then it *is* a
directory, and the rest of the upload can proceed.
Swift3 is currently using Content-Type on the upload marker to store
the user-supplied Content-Type. This commit moves that data into
sysmeta in the handler for Initiate Multipart Upload, and then looks
in both new and old locations in the handler for Finalize Multipart
Upload.
Change-Id: If64c914b6d9ace7700ca77eead3ef66a771cd92e
The docs were recommending a bad config (see related change).
Related-Change: I21e38884a2aefbb94b76c76deccd815f01db7362
Change-Id: Idca96a39f552083b55dc5a86d14ee4357777d6fe
Even though we don't support versioning yet, we can at least
tolerate a client that explicitly requests a null versionId
The syntax is described at
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
Change-Id: Iedd8cc1b0c6f3a770f28be74eafc58bbef0259ad
Previously, we hardcoded a v2.0 path to use when validating requests
against Keystone. Now, the version to use may be specified in a new
auth_version config option.
In the future, we may want to implement some form of version
discovery, but that will be complicated by:
* trying to determine whether the S3 extension is actually enabled for a
given version (particularly since the extensions endpoint [1] seems to
have gone away in v3), and
* needing to be able to perform this detection as part of the
client-request cycle, in case Keystone is down when the proxy is
coming up.
[1] http://developer.openstack.org/api-ref/identity/v2/index.html?expanded=list-extensions-detail
Change-Id: I3a9c702123fd1b76d45214a89ec0583caf3719f0
The previous sha used by s3compat was rather out of date.
Most of the new failures have to do with server-side encryption,
lifecycle management, and tags.
Change-Id: Ib03ca6de0fd6e6973384a83ac4d5ba07110f26ac
And also swift3 middleare log name can be configured as well as
s3token middleware.
Change-Id: I882208579e8df89ebd0033033e1e035c370b80a6
Related-Change: be22c9d2fd
Before this commit, V2 listing parameters ('start-after',
'continuation-token' and 'fetch-owner') were just ignored, making some
S3 clients return errors, or handle paging badly.
V2 listing is selected by passing 'list-type=2' in query string.
'marker' is replaced by either 'start-after' or 'continuation-token'.
This commit wraps 'start-after' and 'continuation-token' in 'marker',
which is passed to swift. 'NextContinuationToken' is a base64 encoding
of the last returned object, so it is opaque to the client.
Change-Id: I23bf83cb8bbaf4c4935bf6b56791051c032c688c
Some S3 client libs send an Etag/Content-Md5 header during multi-part object
initialization. The S3 API reference does not mention how the MD5 checksum
header is treated at this stage, and the API itself appears to ignore the
headers.
Prior to this commit, swift3 passed the headers on, which were later compared
to the md5sum of the request's body, which is always empty. This results in the
upload failing when the client-supplied checksum (generally the checksum for the
entire object) does not match the checksum for a null object.
After this commit, the Etag and Content-Md5 headers are ignored during the
multi-part initialization phase. This mimics the behavior of AWS' S3 API.
Closes-Bug: 1697741
Change-Id: I2cb5376994bf270890bd9b06ec2bf521350c826d
Sphinx wants babel!=2.0,>=1.3 which drags in 2.4.0, but oslo.i18n
(required by keystone) wants Babel!=2.4.0,>=2.3.4 leading to an
error when starting keystone.
Also, bump up the sleep before verifying that keystone is running
-- at just one second, keystone hadn't had a chance to raise the
ContextualVersionConflict yet. Further, verify that curl can actually
reach keystone on the expected port.
Change-Id: I6cbbfd7260571f42ea65c6622aa6b410a0e43b28
Some S3 clients rely on AWS's ability to preserve underscores in
header names. This doesn't mesh well with WSGI, which treats dashes
and underscores equivalently. Starting in eventlet 0.19.0, however,
the raw headers from the client are available in a `headers_raw`
environment key. If available, use it.
Change-Id: If96fc2f0713e3ec18764766225446a16a7c07f94
This allows (some) auth middlewares to stop needing to know the details
of S3 request signing. The function takes the user's secret and returns
True if the signatures match, False otherwise.
This offers an easy way for auth middlewares that have user secrets
in-memory (such as swauth and tempauth) to add support for v4 signatures.
Change-Id: Iafb6114c12deb9a40d0f8324611de27b48ed95f6
TrivilalFix
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)
Change-Id: Ia4e28fbcb836df9f0bebe2734bceb0e2ca98a6b8
Keystone master currently responds with v3 tokens. Preserve v2 support
for stable/newton and earlier Keystones. Note that stable/ocata simply
doesn't work.
Related-Change: I5d0c18121ba4bf8e33209daa48b9d87864951362
Change-Id: I747de516ab69a47622eecbf8ab3faa34444b3ad5
Previously, we would use client-accessible headers to pass the S3 access
key, signature, and normalized request to authentication middleware.
Specifically, we would send the following headers:
Authorization: AWS <access key>:<signature>
X-Auth-Token: <base64-encoded normalized request>
However, few authentication middleware would validate that the
Authorization header actually started with "AWS ", the only prefix that
Swift3 would actually handle. As a result, the authentication
middlewares had no way to validate that the normalized request came from
swift3 rather than the client itself. This leads to a security hole
wherein an attacker who has captured a single valid request through the
S3 API or who has obtained a valid pre-signed URL may impersonate the
user that issued the request or pre-signed URL indefinitely through the
Swift API.
Now, the S3 authentication information will be placed in a separate
namespace in the WSGI environment, completely inaccessible to the
client. Specifically,
environ['swift3.auth_details'] = {
'access_key': <access key>,
'signature': <signature>,
'string_to_sign': <normalized request>,
}
(Note that the normalized request is no longer base64-encoded.)
UpgradeImpact
This is a breaking API change. No currently-deployed authentication
middlewares will work with this. This patch includes a fix for s3_token
(used to authenticate against Keystone); any deployers still using
keystonemiddleware to provide s3_token should switch to using swift3.
Similar changes are being proposed for Swauth and tempauth. Proprietary
authentication middlewares will need to be updated to use the new
environment keys as well. When upgrading Swift3, operators will need to
upgrade their Swift3-capable authentication middleware at the same time.
Closes-Bug: 1561199
Change-Id: Ia3fbb4938f0daa8845cba4137a01cc43bc1a713c
Depends-On: Ib90adcc2f059adaf203fba1c95b2154561ea7487
Use:
* v3 auth
* keystone-manage bootstrap
* uwsgi
... which all brings us loads closer to Keystone's current documented
developer setup.
Change-Id: Id7a31545e6bfb0da900b1456d7baa449636f12d7
Previously, s3token would make one request to Keystone to validate the
request signature provided by the user, then use the response to add an
X-Auth-Token header to the request environment. This would get picked up
by the authtoken middleware which would make *another* request to
validate the token *we just got*.
Now, we'll populate the request environment with the following headers:
* X-Identity-Status
* X-Roles
* X-User-Id
* X-User-Name
* X-Tenant-Id
* X-Tenant-Name
* X-Project-Id
* X-Project-Name
This allows Swift's keystoneauth middleware to function without needing
the authtoken middleware at all.
UpgradeImpact
-------------
The recommended pipeline ordering has changed. Whereas authoken previously
had to be between s3token and keystoneauth like
... swift3 s3token authtoken keystoneauth ...
it should now be placed before swift3, as in
... authtoken swift3 s3token keystoneauth ...
Alternatively, if Keystone users should only ever access Swift through
the S3 API, the authtoken middleware may be removed entirely. Note that
the old pipeline ordering will continue to work, but still requires two
Keystone requests per client request as before.
To upgrade an existing cluster to take advantage of this change
operators should, for each proxy server:
1. Upgrade swift3
2. Optionally, restart proxy-server
3. Update proxy-server.conf with the new pipeline
4. Restart proxy-server
Updating proxy-server.conf *before* upgrading swift3 will prevent the
proxy from starting if there is an unexpected reboot before the upgrade.
Closes-Bug: #1653017
Change-Id: I21e38884a2aefbb94b76c76deccd815f01db7362
Previously, if a user-provided timestamp was after the largest-possible
Swift timestamp, we would raise AccessDenied. However, AWS continues to
complain about time skew.
Note that we may regret this come 2286, but by then I'll be dead.
Change-Id: I88952a28a7e7c42540c61514f82582815fabf611
We are already passing arguments to nosetests for non ceph functional
tests, this change does the same for run_ceph_tests.py.
Change-Id: I7c54fddb98871ed3a6437a457dcf775d91f0ed45
Note that we already have tests to verify the correct behavior for
multipart uploads in test_abort_multi_upload_error.
Related-Change: I6e154594dfda6c3065774c23b24f728625a842bc
Change-Id: If23ea4be53875cbe1db34f7f9b2baa4d3187f3a6
Swift has removed the minimum segment size setting for multipart upload.
To make it compatible with S3, we are re-implementing it in swift3.
Each upload part except the last should be more than the minimum segment
size (default 5MB, same as the S3 multipart upload chunk size). When a
"complete multipart upload" request comes, check all the parts and
return a EntityTooSmall error if they are smaller than the minimum
segment size.
Change-Id: I883b25ab3d43d330ffc60fa2c3ade7a6b5802cee
Previously, when clients sent non-printable characters in headers, we
might reply with InvalidDigest or some other flavor of 400 while AWS
would have sent a 403 SignatureDoesNotMatch in a similar situation.
See, for example, the removed known failures for ceph/s3-tests.
Additionally, factor out canonical request from string-to-sign for
SigV4Mixin. This simplifies the diagnosing of certain unit test failures.
Change-Id: I703d3db134b8e2202d271eca018b920fbedd08e7
Tim Burke