... by storing SLO Etag and size in sysmeta.
Previously, we had to GET the manifest for every HEAD or conditional
request to an SLO. Worse, since SLO PUTs require that we HEAD every
segment, we'd GET all included sub-SLO manifests. This was necessary so
we could recompute the large object's Etag and content-length.
Since we already know both of those during PUT, we'll now store it in
object sysmeta. This allows us to:
* satisfy HEAD requests based purely off of the manifest's HEAD
response, and
* perform the If-(None-)Match comparison on the object server, without
any additional subrequests.
Note that the large object content-length can't just be parsed from
content-type -- with fast-POST enabled, the content-type coming out of
the object-server won't necessarily include swift_bytes.
Also note that we must still fall back to GETting the manifest if the
sysmeta headers were not found. Otherwise, we'd break existing large
objects.
Change-Id: Ia6ad32354105515560b005cea750aa64a88c96f9
tests.py is currently at ~5500 lines of code, it's
time to break it down into smaller files.
I started with an easy middleware set of tests
(i.e., versioned writes, ~600 lines of code ) so I can get
some feedback. There are more complicated tests that cover
multiple middlewares for example, it is not so clear where
those should go.
Change-Id: I2aa6c18ee5b68d0aae73cc6add8cac6fbf7f33da
Signed-off-by: Thiago da Silva <thiago@redhat.com>
This is a follow up for https://review.openstack.org/#/c/399237
'id' is assigned as a builtin function so that we should not use 'id'
for the local variable name.
Change-Id: Ic27460d49e68f6cd50bda1d5b3810e01ccb07a37
I have reordered the Contributors section of the AUTHORS
file to simply be ordered by the first character in the
author name.
Change-Id: I824022e582cadc361484b20c411840f46c4c5b50
Previously, we could over-assign how many parts should be in a tier.
This would cause the local `parts` variable to go negative, which meant
that our `while parts` loop would never terminate.
Change-Id: Id7e7889742ca37cf1a9c0d55fba78d967e90e8d0
Closes-Bug: 1642538
Previously in copy middleware, if a user entered an invalid destination
path with an invalid `container/object` path the server would return
a 500 Internal Server Error. However, the correct response should be
a 412 Precondition Failed. This patch updates copy so that it catches
the 412 Precondition Failed exception and returns it to the client.
Closes-Bug: #1641980
Change-Id: Ic4677ae033d05b8730c6ad1041bd9c07268e11a9
Before creating a static large object, we must verify that all of the
referenced segments exist. Previously, this was done sequentially; due
to latency between proxy and object nodes, clients must be careful to
either keep their segment count low or use very long (minute+) timeouts.
We mitigate this somewhat by enforcing a hard limit on segment count,
but even then, HEADing a thousand segments (the default limit) with an
average latency of (say) 100ms will require more than a minute and a
half.
Further, the nested-SLO approach requires multiple requests from the
client -- as a result, Swift3 is in the position of enforcing a lower
limit than S3's 10,000 (which will break some clients) or requiring that
clients have timeouts on the order of 15-20 minutes (!).
Now, we'll perform the segment HEADs in parallel, with a concurrency
factor set by the operator. This is very similar to (and builds upon)
the parallel-bulk-delete work. By default, two HEAD requests will be
allowed at a time.
As a side-effect, we'll also only ever HEAD a path once per manifest.
Previously, if a manifest alternated between two paths repeatedly (for
instance, because the user wanted to splice together various ranges from
two sub-SLOs), then each entry in the manifest would trigger a fresh
HEAD request.
Upgrade Consideration
=====================
If operators would like to preserve the prior (single-threaded) SLO
creation behavior, they must add the following line to their
[filter:slo] proxy config section:
concurrency = 1
This may be done prior to upgrading Swift.
UpgradeImpact
Closes-Bug: #1637133
Related-Change: I128374d74a4cef7a479b221fd15eec785cc4694a
Change-Id: I567949567ecdbd94fa06d1dd5d3cdab0d97207b6
Swift returns `X-Openstack-Request-Id` for container and
object requests as well as account. A couple api-ref docs are
missing this value in the response examples.
Change-Id: Ifcd67a620e04be5e92b43c7749ee4acb50bb171d
Continuing the clean up in account and container
info, removed duplicated validation from account_info
and container_info methods, since the same validations
were recently added to get_account_info and get_container_info.
Change-Id: I1ad745fe809367d22649c83f38c4de7a74cac44e
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Changes the file parameters on command line to use the convention of
<file> for a mandatory option rather than [file] which suggests an
optional flag. Also drop to lower case as is convention in other man
pages.
Change-Id: Icfb4b4786343eb493a73b092cad1fdadc935d066
swift-account-info, swift-container-info, swift-object-info
and swift-get-node do not contain any information about the available
options. This patch adds OPTIONS in these manpages.
Change-Id: I832c621460ee6bf19adac8e0aadeab32be4b8da0
The warning message for rsync tempfiles was removed in the related
change. However, because our regex match might result in a false
positive maybe it's still useful to log a debug message. Instead of
silently ignoring rsync tempfiles, when running in debug we note the
file and how we classified it - but still no warning will occur.
I also consolidate our use of the regex for rsync tempfiles into the
diskfile module and move the negative test for the warning logger
message a little next to the positive test.
Change-Id: Idb2a1a76aa275c9c2e9bad8aceea913b8f5b1c71
Related-Change: I5a5d6e24710e4880776b32edcbc07021acf77676
This should give the necessary tooling to set up the right
links in the right places so Swift has release notes listed
alongside other projects.
Change-Id: I4e5f1ce1fcfbb2943036c821a24a0b4a3a2d9fc8
Decode header to latin1 on python3
encode header to utf-8 on python2.
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Change-Id: I10f205a05bb3a566e52a597d9315b3a8b8c14664
Despite a check to prevent zero values in the denominator python
integer division could result in ZeroDivisionError in the compute_eta
helper function. Make sure we always have a non-zero value even if it
is small.
NotImplemented:
* stats calculation is still not great, see lp bug #1488608
Closes-Bug: #1549110
Change-Id: I54f2081c92c2a0b8f02c31e82f44f4250043d837
The intentional use of "bare except" handling in catch_errors and some
daemons to prevent propagation on unexpected errors that do not
inherit from Exception (like eventlet.Timeout) or even BaseException
(like old-style classes) has the side effect of spuriously "handling"
*expected* errors like when a signal handler raises SystemExit.
The signal handler installed in our Daemon is intended to ensure first
that the entire process group and any forked processes (like rsync's)
receive the SIGTERM signal and also that the process itself
terminates.
The use of sys.exit was not a concious grandiose plans for graceful
shutdown (like the running[0] = False trick that wsgi server parent
process do) - the desired behavior for SIGTERM is to stop - hard.
This change ensures the original goals and intentions of our signal
handler are fulfilled without the undesirable side effect that can
cause our daemons to confusingly log an expected message to stop as an
unexpected error, and start ignoring additional SIGTERM messages;
forcing our kind operators to resort to brutal process murder.
Closes-Bug: #1489209
Change-Id: I9d2886611f6db2498cd6a8f81a58f2a611f40905
When an SSYNC connection fails after shipping a fragment, but before
cleaning itself up - it can lead to multiple replicas of the same
fragment index serviceable in the node_iter at the same time. Or
perhaps more likely if a partner nodes win a race to rebuild waiting
on a handoff. In this case, the proxy need not fail to service a GET
just because of extra information. A similar guard was added to the
reconstructor in a related change [1].
This underlying bug is acctually fixed by the related change [2], this
probetest is just encoding the failure to help with future maintenance.
1. Related-Change: I91f3d4af52cbc66c9f7ce00726f247b5462e66f9
2. Related-Change: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
Closes-Bug: 1624176
Change-Id: I06fc381a25613585dd18916d50e7ca2c68d406b6
Previously the reconstructor would only reconstruct a missing fragment
when a set of ec_ndata other fragments was available, *all* of which
were durable. Since change [1] it has been possible to retrieve
non-durable fragments from object servers. This patch changes the
reconstructor to take advantage of [1] and use non-durable fragments.
A new probe test is added to test scenarios with a mix of failed and
non-durable nodes. The existing probe tests in
test_reconstructor_rebuild.py and test_reconstructor_durable.py were
broken. These were intended to simulate cases where combinations of
nodes were either failed or had non-durable fragments, but the test
scenarios defined were not actually created - every test scenario
broke only one node instead of the intent of breaking multiple
nodes. The existing tests have been refactored to re-use most of their
setup and assertion code, and merged with the new test into a single
class in test_reconstructor_rebuild.py.
test_reconstructor_durable.py is removed.
[1] Related-Change: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
Change-Id: Ic0cdbc7cee657cea0330c2eb1edabe8eb52c0567
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Closes-Bug: #1624088
OpenDev Sysadmins