If the global configuration option 'enable_open_expired' is set
to true in the config, then the client will be able to make a
request with the header 'x-open-expired' set to true in order
to access an object that has expired, provided it is in its
grace period. If this config flag is set to false, the client
will not be able to access any expired objects, even with the
header, which is the default behavior unless the flag is set.
When a client sets a 'x-open-expired' header to a true value for a
GET/HEAD/POST request the proxy will forward x-backend-open-expired to
storage server. The storage server will allow clients that set
x-backend-open-expired to open and read an object that has not yet
been reaped by the object-expirer, even after the x-delete-at time
has passed.
The header is always ignored when used with temporary URLs.
Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Related-Change: I106103438c4162a561486ac73a09436e998ae1f0
Change-Id: Ibe7dde0e3bf587d77e14808b169c02f8fb3dddb3
Normally, the proxy object controller would be adding these, but when
encrypted, there won't be any headers in the x-object-meta-* namespace.
Closes-Bug: #1868045
Change-Id: I8e708a60ee63f679056300fc9d68227e46d605e8
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() is added to
swift/common/utils.py. This encapsulation is identical to the one being
added to oslo.utils, but is recreated here to avoid adding a dependency.
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
While this patch seems large, it is really just the same change over and
again. Reviewers need to pay particular attention as to whether the
keyword parameter (usedforsecurity) is set correctly. Right now, all
of them appear to be not used in a security context.
Now that all the instances have been converted, we can update the bandit
run to look for these instances and ensure that new invocations do not
creep in.
With this latest patch, the functional and unit tests all pass
on a FIPS enabled system.
Co-Authored-By: Pete Zaitcev
Change-Id: Ibb4917da4c083e1e094156d748708b87387f2d87
Users have complained for a while that Swift's ETags don't match the
expected RFC formats. We've resisted fixing this for just as long,
worrying that the fix would break innumerable clients that expect the
value to be a hex-encoded MD5 digest and *nothing else*.
But, users keep asking for it, and some consumers (including some CDNs)
break if we *don't* have quoted etags -- so, let's make it an option.
With this middleware, Swift users can set metadata per-account or even
per-container to explicitly request RFC compliant etags or not. Swift
operators also get an option to change the default behavior
cluster-wide; it defaults to the old, non-compliant format.
See also:
- https://tools.ietf.org/html/rfc2616#section-3.11
- https://tools.ietf.org/html/rfc7232#section-2.3
Closes-Bug: 1099087
Closes-Bug: 1424614
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I380c6e34949d857158e11eb428b3eda9975d855d
unittest2 was needed for Python version <= 2.6, so it hasn't been needed
for quite some time. See unittest2 note one:
https://docs.python.org/2.7/library/unittest.html
This drops unittest2 in favor of the standard unittest module.
Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
When we fixed bulk response heartbeating in https://review.openstack.org/#/c/510715/,
code review raised the issue of moving the xml header down to after the
early-exit clauses. At the time, it didn't seem to break anything, so
it was left in place. However, that insight was correct.
The purpose of the earlier patch was to force eventlet to use chunked
transfer encoding on the response in order to prevent eventlet from
buffering the whole response, thus defeating the purpose of the
heartbeat responses.
Moving the first line of the body lower (ie after the early exit
checks), allows other headers in a chunked transfer encoding response
to be appropriately processed before sending the headers. Sending the
xml declaration early causes it to get intermingled in the 100-continue
protocol, thus breaking the chunked transfer encoding semantics.
Closes-Bug: #1819252
Change-Id: I072f4dab21cd7cdb81b9e41072eb504131411dc8
Since JSON object listings deserialize as unicode, obj['name'] would
hoist *everything* to unicode. If the account or container name was a
byte string, though, it would trip a UnicodeDecodeError.
Change-Id: I2c1932143b78521c6bdcfa48182b475528fc1bb3
The existing test works fine if you're running the tests on an
all-in-one, but is pretty brittle if you aren't running them on the
one and only proxy-server they're hitting.
Add 0.1s sleep to allow *some* clock slippage between client and server.
Change-Id: Iacd08e9f703d08d0092b5e8eb53fe287ba1d1596
Currently, our integrity checking for objects is pretty weak when it
comes to object metadata. If the extended attributes on a .data or
.meta file get corrupted in such a way that we can still unpickle it,
we don't have anything that detects that.
This could be especially bad with encrypted etags; if the encrypted
etag (X-Object-Sysmeta-Crypto-Etag or whatever it is) gets some bits
flipped, then we'll cheerfully decrypt the cipherjunk into plainjunk,
then send it to the client. Net effect is that the client sees a GET
response with an ETag that doesn't match the MD5 of the object *and*
Swift has no way of detecting and quarantining this object.
Note that, with an unencrypted object, if the ETag metadatum gets
mangled, then the object will be quarantined by the object server or
auditor, whichever notices first.
As part of this commit, I also ripped out some mocking of
getxattr/setxattr in tests. It appears to be there to allow unit tests
to run on systems where /tmp doesn't support xattrs. However, since
the mock is keyed off of inode number and inode numbers get re-used,
there's lots of leakage between different test runs. On a real FS,
unlinking a file and then creating a new one of the same name will
also reset the xattrs; this isn't the case with the mock.
The mock was pretty old; Ubuntu 12.04 and up all support xattrs in
/tmp, and recent Red Hat / CentOS releases do too. The xattr mock was
added in 2011; maybe it was to support Ubuntu Lucid Lynx?
Bonus: now you can pause a test with the debugger, inspect its files
in /tmp, and actually see the xattrs along with the data.
Since this patch now uses a real filesystem for testing filesystem
operations, tests are skipped if the underlying filesystem does not
support setting xattrs (eg tmpfs or more than 4k of xattrs on ext4).
References to "/tmp" have been replaced with calls to
tempfile.gettempdir(). This will allow setting the TMPDIR envvar in
test setup and getting an XFS filesystem instead of ext4 or tmpfs.
THIS PATCH SIGNIFICANTLY CHANGES TESTING ENVIRONMENTS
With this patch, every test environment will require TMPDIR to be
using a filesystem that supports at least 4k of extended attributes.
Neither ext4 nor tempfs support this. XFS is recommended.
So why all the SkipTests? Why not simply raise an error? We still need
the tests to run on the base image for OpenStack's CI system. Since
we were previously mocking out xattr, there wasn't a problem, but we
also weren't actually testing anything. This patch adds functionality
to validate xattr data, so we need to drop the mock.
`test.unit.skip_if_no_xattrs()` is also imported into `test.functional`
so that functional tests can import it from the functional test
namespace.
The related OpenStack CI infrastructure changes are made in
https://review.openstack.org/#/c/394600/.
Co-Authored-By: John Dickinson <me@not.mn>
Change-Id: I98a37c0d451f4960b7a12f648e4405c6c6716808
In test.functional.test_object.TestObject.setUp, we create a container
in account 2. However, if we've only got one account, we don't skip
this class, resulting in a TypeError down in requests somewhere and a
stack trace. Since we're using account 2 in setup, we should skip the
tests if account 2 is not configured.
Change-Id: I569d98baf071d2dce7cf34a9538070f00afda388
This looks like a case of copy-paste-itis. The cross-account-copy
functest is skipped if we have no test accounts configured, but not if
we have only one.
Change-Id: Ifbefdd9aeb98e3d02c536e9d29759f86ec9af6a1
X-Delete-After: 1 is known to be flakey; use 2 instead.
When the proxy receives an X-Delete-After header, it automatically
converts it to an X-Delete-At header based on the current time. So far,
so good. But in normalize_delete_at_timestamp we convert our
time.time() + int(req.headers['X-Delete-After'])
to a string representation of an integer and in the process always round
*down*. As a result, we lose up to a second worth of object validity,
meaning the object server can (rarely) respond 400, complaining that the
X-Delete-At is in the past.
Change-Id: Ib5e5a48f5cbed0eade8ba3bca96b26c82a9f9d84
Related-Change: I643be9af8f054f33897dd74071027a739eaa2c5c
Related-Change: I10d3b9fcbefff3c415a92fa284a1ea1eda458581
Related-Change: Ifdb1920e5266aaa278baa0759fc0bfaa1aff2d0d
Related-Bug: #1597520
Closes-Bug: #1699114
When an object is deleted, we retrieve its metadata on the next PUT
requests, which make if-none-match requests fail while the object
shouldn't be here. It seems we're only interested in the timestamp of
the deleted object, so get that but forget the metadata.
Change-Id: I4a4a8be9b631598ca1cd52c53885c68c3fbdfc4a
Closes-Bug: #1640448
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I80ec96e0b729bef38213a6be4ff4b6eb65c7612d
With the X-Timestamp validation added in commit e619411, end users
could upload objects with
X-Timestamp: 9999999999.99999_ffffffffffffffff
(the maximum value) and Swift would be unable to delete them.
Now, inbound X-Timestamp headers will be moved to
X-Backend-Inbound-X-Timestamp, effectively rendering them harmless.
The primary reason to allow X-Timestamp before was to prevent
Last-Modified changes for objects coming from either:
* container_sync or
* a migration from another storage system.
To enable the former use-case, the container_sync middleware will now
translate X-Backend-Inbound-X-Timestamp headers back to X-Timestamp
after verifying the request.
Additionally, a new option is added to the gatekeeper filter config:
# shunt_inbound_x_timestamp = true
To enable the latter use-case (or any other use-case not mentioned), set
this to false.
Upgrade Consideration
=====================
If your cluster workload requires that clients be allowed to specify
objects' X-Timestamp values, disable the shunt_inbound_x_timestamp
option before upgrading.
UpgradeImpact
Change-Id: I8799d5eb2ae9d795ba358bb422f69c70ee8ebd2c
Previously, if a user could write to (but not read from) a container,
the behavior for object POST would vary depending on whether
object_post_as_copy was enabled (403 response) or disabled (202
response).
Now, POSTs will consistently be allowed, regardless of whether fast-POST
is enabled.
Change-Id: I1d6dcbc4f5034a322a1073850fc3b059ebb1c0fa
This patch adds test cases for PUT, DELETE, GET, HEAD, POST and OPTIONS
requests to accounts, containers and objects using various combinations
of users/projects, roles and/or service tokens.
Change-Id: Iea8141ac74ad949a3ae7fa47fda3135d0f2612f6
This patch changes functional test classes to subclass
unittest2.TestCase rather than unittest.TestCase.
This fixes errors when attempting to use
tox -e func -- -n <test_path_including_test_method>
and
tox -e func -- --until-failure
Also migrate from using nose.SkipTest to unittest2.SkipTest
Change-Id: I903033f5e01833550b2f2b945894edca4233c4a2
Closes-Bug: 1526725
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Defcore uses Tempest, which uses Test Repository.
This change makes it easier for Defcore to pull functional
tests from Swift and run them. Additionally, using testr
allows tests to be run in parallel.
Concurrency set to 1 for now, >1 causes failures for
reasons that are still TBD.
With switch to ostestr all the server logs are being sent to stdout
which makes it completely unreadable. Suppressing the logs by default
now with a flag to enable it if desired.
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Robert Collins <rbtcollins@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Change-Id: I53ef4a116996a772cf1f3abc2eb0ad60047322d5
Related-Bug: 1177924
We have a bunch of assertions like
self.assertTrue(resp.status in (200, 204))
Sometimes we get smart about failure messages and have something like
self.assertTrue(resp.status in (200, 204), resp.status)
so we can see what the status was when it failed.
Since we don't have to support Python 2.6 any more, we can use
assertIn/assertNotIn and get nice failure messages for free.
Change-Id: I2d46c9969d41207a89e01017b4c2bc533c3d744f
The assert_() method is deprecated and can be safely replaced by assertTrue().
This patch makes sure that running the tests does not generate warnings
all over the screen.
Change-Id: I74705c6498249337bfdf955d62e0ad972035bc1f
Currently, a COPY request for an EC object might go to fail as 499 Client
disconnected because of the difference between destination request content
length and actual transferred bytes.
That is because the conditional response status and content length for
an EC object range GET is handled at calling the response instance on
proxy server. Therefore the calling response instance (resp()) will change
the conditional status from 200 (HTTP_OK) to 206 (PartialContent) and will
change the content length for the range GET.
In EC case, sometimes Swift needs whole stored contents to decode a segment.
It will make 200 HTTP OK response from object-server and proxy-server
will unfortunately set whole content length to the destination content
length and it makes the bug 1467677.
This patch introduces a new method "fix_conditional_response" for
swift.common.swob.Response that calling _response_iter() and cached the
iter in the Response instance. By calling it, Swift can set correct condtional
response any time after setting whole content_length to the response
instance like EC case.
Change-Id: If85826243f955d2f03c6ad395215c73daab509b1
Closes-Bug: #1467677
Since we're dropping Python 2.6 support, we can rely on stdlib's json
and get rid of our dependency on simplejson.
This commit just takes simplejson out of the unit and functional
tests. They still pass.
Change-Id: I96f17df81fa5d265395a938b19213d2638682106
Adds ability to copy objects between different accounts (on server side)
Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.
Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.
If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`
These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.
DocImpact
Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
If a swift cluster configures the proxy server to not accept
/info requests, just do the best we can, skipping tests as
necessary.
To that end, if cors mode is not known we also skip the test.
Change-Id: I34f296c8717e0baf24ae552a7e38f9354ee91974
* additional container tests
* refactor test cross policy copy
* make functional tests cleanup better
In-process functional tests only define a single ring and will skip some of
the multi-storage policy tests, but have been updated to reload_policies with
the patched swift.conf.
DocImpact
Implements: blueprint storage-policies
Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002
Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.
Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a
Merge the swift_testing module into the functional test module itself,
so that we can read the configuration once for all unit tests, sharing
the same constraints.
Change-Id: I9fbbfdade9adca329cd79f7d4291ba009327c842
indianwhocodes