The *_swift_info functions use in module global dicts to provide a
registry mechanism for registering and getting swift info.
This is an abnormal pattern and doesn't quite fit into utils. Further
we looking at following this pattern for sensitive info to trim in the
future.
So this patch does some house cleaning and moves this registry to a new
module swift.common.registry. And updates all the references to it.
For backwards compat we still import the *_swift_info methods into utils
for any 3rd party tools or middleware.
Change-Id: I71fd7f50d1aafc001d6905438f42de4e58af8421
Previously, logs would often show 499s in places where some other status
would be more appropriate.
Change-Id: I68dbb8593101cd3b5b64a1a947c68e340e36ce02
This patch adds a new object versioning mode. This new mode provides
a new set of APIs for users to interact with older versions of an
object. It also changes the naming scheme of older versions and adds
a version-id to each object.
This new mode is not backwards compatible or interchangeable with the
other two modes (i.e., stack and history), especially due to the changes
in the namimg scheme of older versions. This new mode will also serve
as a foundation for adding S3 versioning compatibility in the s3api
middleware.
Note that this does not (yet) support using a versioned container as
a source in container-sync. Container sync should be enhanced to sync
previous versions of objects.
Change-Id: Ic7d39ba425ca324eeb4543a2ce8d03428e2225a1
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>
Apparently FakeSwift has always been persnickety about the order of
query params of registered responses and the app making the calls.
Since query params can often be converted to dictionaries the order of
the encoded params should be able to change without effecting the apps
dispatching of registered responses.
Change-Id: Ied68c9334201a7663e9c85f3bdaa5b0643d4b6db
... by embedding something like `If-Match: <etag>` semantics in the
symlink.
When creating a symlink, users may now specify an optional
X-Symlink-Target-Etag header. If present, the etag of the final object
returned to the client will be checked; if it does not match the
X-Symlink-Target-Etag header, a 409 Conflict error will be returned to
the client.
Note that, unlike "dynamic symlink" behavior, the target object must
exist with the matching Etag for the "static symlink" to be created.
Since we're validating the Etag anyway, we also set the content-type of
the symlink to match if the client didn't otherwise specifiy and send
the etag & content-length along to the container listing as well.
Bonus goodness:
- Tighten assertions on Content-Location
- Get rid of swift.source-sniffing by making versioned_writes
symlink-aware ('cause I'm going to want to make it symlink-aware
later anyway)
- Allow middlewares left of symlink to set their own
Container-Update-Override-Etag when creating a symlink
- Set dynamic symlink content type if client doesn't supply something
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I179ea6180d31146bb947061c69b1807c59529ac8
With url encoded object name like '%25ff' that can be url-encoded
value after decoded can cause 412 Precondition Failed. And more,
that can do nothing (no versioned object creation) even it returns
a successful response.
The root causes are in versioned_writes middleware as follows:
A. unnecessary unquote in object_request method
B. incorrect use of make_pre_authed_request that takes 'quoted'
path in the args. That is described at [1] explicitely.
This patch resolved those 2 bugs at once, and then, now we can create
%25ff versioned object reported in the launchpad with this patch.
Perhaps, more tests would be nice to have. This patch added a few
test cases on that.
1: https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L1174
Note that make_subrequest and its caller should have *quoted* path but
make_env should *NOT*. That might make us confused.
Closes-Bug: #1755554
Change-Id: Ibcd90cc633c68973929ee5249c6598c22b342e3e
It was deprecated and we discussed on this topic in Denver PTG
for Queen cycle. Main motivation for this work is that deprecated
post_as_copy option and its gate blocks future symlink work.
Change-Id: I411893db1565864ed5beb6ae75c38b982a574476
Make some json -> (text, xml) stuff in a common module, reference that in
account/container servers so we don't break existing clients (including
out-of-date proxies), but have the proxy controllers always force a json
listing.
This simplifies operations on listings (such as the ones already happening in
decrypter, or the ones planned for symlink and sharding) by only needing to
consider a single response type.
There is a downside of larger backend requests for text/plain listings, but
it seems like a net win?
Change-Id: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d
While we're at it, drop some unused expect_exception code from
test_versioned_writes.py
Change-Id: Id232880c7259839159c057471b61083d5862d189
Related-Change: I430c48c4a81e8392fa271160bcbc1817ef0a88f7
Previously, requests involving DLOs would bypass versioned_writes:
* Any existing DLOs wouldn't get copied to the archive container during
overwrites (or deletes, with history-mode), so there would be no
evidence they had ever existed.
* Any new DLOs wouldn't copy overwritten objects to the archive
container, potentially leading to data loss.
Now, DLOs will behave like every other type of object under
versioned_writes.
Change-Id: I488e13eead2f33dd272d03f6f898adc52fc7fdad
Related-Change: Ie899290b3312e201979eafefb253d1a60b65b837
Related-Change: Ib5b29a19e1d577026deb50fc9d26064a8da81cd7
Closes-Bug: #1626989
Now, instead of saying
X-Versions-Location: <container>
X-Versions-Mode: history
clients should just say
X-History-Location: <container>
Since we've never had a release featuring a user-settable
X-Versions-Mode header, support may be dropped and that is now ignored.
Change-Id: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9
Currently a versioned write PUT uses a pre-authed request to move
it into the versioned container before checking whether the
user is authorised. This can lead to some interesting behaviour
whereby a user can select a versioned object path that it does not
have access to, request a put on that versioned object, and this
request will execute the copy part of the request before it fails
due to lack of permissions.
This patch changes the behaviour to be the same as versioned DELETE
where the request is authorised before anything is moved.
Change-Id: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Closes-Bug: #1562175
This change introduces the concept of a "versioning mode" for
versioned_writes. The following modes are supported:
* stack
When deleting, check whether any previous versions exist in the
versions container. If none is found, the object is deleted. If the
most-recent version in the versions container is not a delete
marker, it is copied into the versioned container (overwriting the
current version if one exists) and then deleted from the versions
container. This preserves the previous behavior.
If the most-recent version in the versions container is a delete
marker and a current version exists in the versioned container, the
current version is deleted. If the most-recent version in the
versions container is a delete marker and no current version exists
in the versioned container, we copy the next-most-recent version
from the versions container into the versioned container (assuming
it exists and is not a delete marker) and delete both the
most-recent version (i.e., the delete marker) and the just-copied
next-most-recent version from the versions container.
With this mode, DELETEs to versioned containers "undo" operations
on containers. Previously this was limited to undoing PUTs, but now
it will also undo DELETEs performed while in "history" mode.
* history
When deleting, check whether a current version exists in the
versioned container. If one is found, it is copied to the versions
container. Then an empty "delete marker" object is also put into the
versions container; this records when the object was deleted.
Finally, the original current version is deleted from the versioned
container. As a result, subsequent GETs or HEADs will return a 404,
and container listings for the versioned container do not include
the object.
With this mode, DELETEs to versioned containers behave like DELETEs
to other containers, but with a history of what has happened.
Clients may specify (via a new X-Versions-Mode header) which mode a
container should use. By default, the existing "stack" mode is used.
Upgrade consideration:
======================
Clients should not use the "history" mode until all proxies in the
cluster have been upgraded. Attempting to use the "history" mode during
a rolling upgrade may result in some requests being served by proxies
running old code (which necessarily uses the "stack" mode), leading to
data loss.
Change-Id: I555dc17fefd0aa9ade681aa156da24e018ebe74b
Rewrite server side copy and 'object post as copy' feature as middleware to
simplify the PUT method in the object controller code. COPY is no longer
a verb implemented as public method in Proxy application.
The server side copy middleware is inserted to the left of dlo, slo and
versioned_writes middlewares in the proxy server pipeline. As a result,
dlo and slo copy_hooks are no longer required. SLO manifests are now
validated when copied so when copying a manifest to another account the
referenced segments must be readable in that account for the manifest
copy to succeed (previously this validation was not made, meaning the
manifest was copied but could be unusable if the segments were not
readable).
With this change, there should be no change in functionality or existing
behavior. This is asserted with (almost) no changes required to existing
functional tests.
Some notes (for operators):
* Middleware required to be auto-inserted before slo and dlo and
versioned_writes
* Turning off server side copy is not configurable.
* object_post_as_copy is no longer a configurable option of proxy server
but of this middleware. However, for smooth upgrade, config option set
in proxy server app is also read.
DocImpact: Introducing server side copy as middleware
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: Ic96a92e938589a2f6add35a40741fd062f1c29eb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
This change removes the use of the COPY request in the versioned
writes middleware. It changes the COPY verb for GETs and PUTs
requests. The main reasoning for this change is to remove any
dependency that versioning had on copy, which will allow for the COPY
functionality to be moved to middleware and to be to the left of the
versioned writes middleware in the proxy pipeline. In this way,
no COPY request will ever arrive at the versioned writes middleware.
A side benefit of this change is that it removes a HEAD request from
the PUT path. Instead of checking if a current version exists, a
GET request is sent, in case of success, a PUT is sent to the
versions container.
A unit test was removed that tested non-default storage policies.
This test is no longer necessary, since it was used to test
specific policy handling code in the COPY method in the proxy
object controller.
Closes-Bug: #1365862
Change-Id: Idf34fa8d04ff292df7134b6d4aa94ff40887b3a4
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Previously, versioned_writes assumed that all container servers would
always have the latest Swift code, allowing them to return reversed
listings. This could cause the wrong version of a file to be restored
during rolling upgrades.
Now, versioned_writes will check that the listing returned is actually
reversed. If it isn't, we will revert to getting the full (in-order)
listing of versions and reversing it on the proxy.
Change-Id: Ib53574ff71961592426cb386ef00a75eb5824def
Closes-Bug: 1562083
time.mktime produces a POSIX timestamp that takes into account the local
timezone, but we're parsing a Last-Modified header with a known (GMT)
timezone. As a result, if one proxy-server is configured with a timezone
other than GMT/UTC, we may have previous versions in the wrong order.
Change-Id: I320e2368b243f4245725e73bfabc7ad19bc5bacb
This will reduce the overhead required for versioned containers in terms
of memory, back-end requests, and wall time.
Change-Id: I156216a14141db547de3cae1dcd5315ae31eaac2
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
When this code lived in the proxy, it was protected by an "except
HTTPException" clause in proxy.Application.handle_request(). When it
moved to middleware, it lost that, and then things like
constraints.check_name_format that raised HTTPException would cause
500s. The HTTPException would make it all the way out to catch_errors
and get translated to a 500.
This commit just wraps a couple try/excepts around the bits in
versioned writes that can raise HTTPException. I tried to make it use
wsgify so I could get that for free, but that wound up being a real
pain because env/start_response are plumbed through pretty much the
whole versioned-writes middleware.
Closes-Bug: 1483705
Change-Id: Ife165bf709e64f313ed07c779b21914045e51f25
Rewrite object versioning as middleware to simplify the PUT method
in the object controller.
The functionality remains basically the
same with the only major difference being the ability to now
version slo manifest files. dlo manifests are still not
supported as part of this patch.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
DocImpact
Change-Id: Ie899290b3312e201979eafefb253d1a60b65b837
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Matthew Oliver