Following PyCQA/astroid@206d8a2 we sarted getting a whole bunch of
errors like
E:266,44: Value 'headers' doesn't support membership test
and
E:267,25: Value 'headers' is unsubscriptable
Digging around a bit, apparently astroid thinks the headers returned
from call_app will always be None -- I guess it doesn't like our use of
a list to work around py2's lack of `nonlocal`.
By using a proper object to encapsulate state, we can shut up those
"error"s.
Also, pin upper-constraints to pike for keystone-related jobs.
Change-Id: I5ff21260872f4089b030cd94e494dc346ae74b8e
Zuul no longer requires the project-name for in-repo configuration.
Omitting it makes forking or renaming projects easier.
Change-Id: I810ca29398f3879521a42932f23b5dda79fb79df
Import the legacy Zuul tox jobs and convert them to native Zuul v3
native jobs.
Needed-By: I377e3c75ecbea3c17b5073865af254000b1800ff
Change-Id: I67ac8cb4ce20baa003dfb94dedf145b0779a6b99
From RFC 7230 [1]:
> The presence of a message body in a request is signaled by a
> Content-Length or Transfer-Encoding header field.
Thus, if a client sent neither a Content-Length nor a Transfer-Encoding
header, we should assume there is no body and never read from
wsgi.input. In such a case, return an empty string unless the caller
has specified that a body is required.
This revealed a whole bunch of edge cases around our XML parsing:
* Attempting to PUT bucket ACLs while not actually supplying an ACL
should return a MissingSecurityHeader error.
* Attempting to perform a multi-delete without a body should return a
MissingRequestBody error.
* Attempting to complete a multi-part upload without a body should
return an InvalidRequest error.
[1] https://tools.ietf.org/html/rfc7230#section-3.3
Change-Id: I47ad946cff64a3da16a759a4364e6fe29b000e11
Closes-Bug: 1593870
ProxyFS is a soon-to-be-released project that provides a filesystem on
top of Swift, then also tries to provide the Swift API on top of that
filesystem. However, there are some places where
Swift-on-filesystem-on-Swift works differently from plain Swift, and
this is one of them.
When we make a multipart upload, it looks like this:
/v1/a/c+segments/obj/uploadId
/v1/a/c+segments/obj/uploadId/1
/v1/a/c+segments/obj/uploadId/2
...
The problem is that .../obj/uploadId is a file, and in
Swift-on-filesystem-on-Swift, that means you can't make another file
.../obj/uploadId/1, just like you can't on your local filesystem.
However, if you create .../obj/uploadId with a Content-Type of
application/directory and a Content-Length of 0, then it *is* a
directory, and the rest of the upload can proceed.
Swift3 is currently using Content-Type on the upload marker to store
the user-supplied Content-Type. This commit moves that data into
sysmeta in the handler for Initiate Multipart Upload, and then looks
in both new and old locations in the handler for Finalize Multipart
Upload.
Change-Id: If64c914b6d9ace7700ca77eead3ef66a771cd92e
The docs were recommending a bad config (see related change).
Related-Change: I21e38884a2aefbb94b76c76deccd815f01db7362
Change-Id: Idca96a39f552083b55dc5a86d14ee4357777d6fe
Even though we don't support versioning yet, we can at least
tolerate a client that explicitly requests a null versionId
The syntax is described at
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
Change-Id: Iedd8cc1b0c6f3a770f28be74eafc58bbef0259ad
Previously, we hardcoded a v2.0 path to use when validating requests
against Keystone. Now, the version to use may be specified in a new
auth_version config option.
In the future, we may want to implement some form of version
discovery, but that will be complicated by:
* trying to determine whether the S3 extension is actually enabled for a
given version (particularly since the extensions endpoint [1] seems to
have gone away in v3), and
* needing to be able to perform this detection as part of the
client-request cycle, in case Keystone is down when the proxy is
coming up.
[1] http://developer.openstack.org/api-ref/identity/v2/index.html?expanded=list-extensions-detail
Change-Id: I3a9c702123fd1b76d45214a89ec0583caf3719f0
The previous sha used by s3compat was rather out of date.
Most of the new failures have to do with server-side encryption,
lifecycle management, and tags.
Change-Id: Ib03ca6de0fd6e6973384a83ac4d5ba07110f26ac
And also swift3 middleare log name can be configured as well as
s3token middleware.
Change-Id: I882208579e8df89ebd0033033e1e035c370b80a6
Related-Change: be22c9d2fd
Before this commit, V2 listing parameters ('start-after',
'continuation-token' and 'fetch-owner') were just ignored, making some
S3 clients return errors, or handle paging badly.
V2 listing is selected by passing 'list-type=2' in query string.
'marker' is replaced by either 'start-after' or 'continuation-token'.
This commit wraps 'start-after' and 'continuation-token' in 'marker',
which is passed to swift. 'NextContinuationToken' is a base64 encoding
of the last returned object, so it is opaque to the client.
Change-Id: I23bf83cb8bbaf4c4935bf6b56791051c032c688c
Some S3 client libs send an Etag/Content-Md5 header during multi-part object
initialization. The S3 API reference does not mention how the MD5 checksum
header is treated at this stage, and the API itself appears to ignore the
headers.
Prior to this commit, swift3 passed the headers on, which were later compared
to the md5sum of the request's body, which is always empty. This results in the
upload failing when the client-supplied checksum (generally the checksum for the
entire object) does not match the checksum for a null object.
After this commit, the Etag and Content-Md5 headers are ignored during the
multi-part initialization phase. This mimics the behavior of AWS' S3 API.
Closes-Bug: 1697741
Change-Id: I2cb5376994bf270890bd9b06ec2bf521350c826d
The recommended pipeline ordering changed beginning with 6ffcc294e3 .
After moving to use keystone v3 API, the new order becomes mandatory.
Change-Id: Ic9af387b9192f285f0f486e7171eefb23968007e
...and pin us to stable/newton. (I wanted stable/mitaka so it'd match
Keystone, but apparently then pbr is too old for Keystone to install
a keystone-wsgi-admin script.)
This prevents various version conflicts; the most recent was from
oslo.service blocking too-new eventlet.
Change-Id: I115dc231a9156a5bceacaa21d6242bb934fbbd24
Related-Change: I3b2196fdef9936f1c7d468f9c7c9b9246d3d26fd
Related-Change: I6cbbfd7260571f42ea65c6622aa6b410a0e43b28
Sphinx wants babel!=2.0,>=1.3 which drags in 2.4.0, but oslo.i18n
(required by keystone) wants Babel!=2.4.0,>=2.3.4 leading to an
error when starting keystone.
Also, bump up the sleep before verifying that keystone is running
-- at just one second, keystone hadn't had a chance to raise the
ContextualVersionConflict yet. Further, verify that curl can actually
reach keystone on the expected port.
Change-Id: I6cbbfd7260571f42ea65c6622aa6b410a0e43b28
Some S3 clients rely on AWS's ability to preserve underscores in
header names. This doesn't mesh well with WSGI, which treats dashes
and underscores equivalently. Starting in eventlet 0.19.0, however,
the raw headers from the client are available in a `headers_raw`
environment key. If available, use it.
Change-Id: If96fc2f0713e3ec18764766225446a16a7c07f94
This allows (some) auth middlewares to stop needing to know the details
of S3 request signing. The function takes the user's secret and returns
True if the signatures match, False otherwise.
This offers an easy way for auth middlewares that have user secrets
in-memory (such as swauth and tempauth) to add support for v4 signatures.
Change-Id: Iafb6114c12deb9a40d0f8324611de27b48ed95f6
TrivilalFix
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)
Change-Id: Ia4e28fbcb836df9f0bebe2734bceb0e2ca98a6b8
Keystone master currently responds with v3 tokens. Preserve v2 support
for stable/newton and earlier Keystones. Note that stable/ocata simply
doesn't work.
Related-Change: I5d0c18121ba4bf8e33209daa48b9d87864951362
Change-Id: I747de516ab69a47622eecbf8ab3faa34444b3ad5
This prevents version conflicts when Keystone drags in a too-new
oslo.policy, which blackballs latest requests (but Sphinx would try to
pull it in anyway).
Change-Id: I3b2196fdef9936f1c7d468f9c7c9b9246d3d26fd
Previously, we would use client-accessible headers to pass the S3 access
key, signature, and normalized request to authentication middleware.
Specifically, we would send the following headers:
Authorization: AWS <access key>:<signature>
X-Auth-Token: <base64-encoded normalized request>
However, few authentication middleware would validate that the
Authorization header actually started with "AWS ", the only prefix that
Swift3 would actually handle. As a result, the authentication
middlewares had no way to validate that the normalized request came from
swift3 rather than the client itself. This leads to a security hole
wherein an attacker who has captured a single valid request through the
S3 API or who has obtained a valid pre-signed URL may impersonate the
user that issued the request or pre-signed URL indefinitely through the
Swift API.
Now, the S3 authentication information will be placed in a separate
namespace in the WSGI environment, completely inaccessible to the
client. Specifically,
environ['swift3.auth_details'] = {
'access_key': <access key>,
'signature': <signature>,
'string_to_sign': <normalized request>,
}
(Note that the normalized request is no longer base64-encoded.)
UpgradeImpact
This is a breaking API change. No currently-deployed authentication
middlewares will work with this. This patch includes a fix for s3_token
(used to authenticate against Keystone); any deployers still using
keystonemiddleware to provide s3_token should switch to using swift3.
Similar changes are being proposed for Swauth and tempauth. Proprietary
authentication middlewares will need to be updated to use the new
environment keys as well. When upgrading Swift3, operators will need to
upgrade their Swift3-capable authentication middleware at the same time.
Closes-Bug: 1561199
Change-Id: Ia3fbb4938f0daa8845cba4137a01cc43bc1a713c
Depends-On: Ib90adcc2f059adaf203fba1c95b2154561ea7487
OpenDev Sysadmins