While the existing named kwargs (marker, limit, prefix and etc)
are common parameters among different use cases of
direct_get_container, a new 'extra_params' kwarg is added to
support additional private parameters for each individual use case.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: If1d355fbc5d292cee4f44c0e5dc52e5df9ae990b
There may be circumstances when an internal client wishes to modify
container sysmeta that is hidden from the user. It is desirable that
this happens without modifying the put-timestamp and therefore the
last-modified time that is reported in responses to client HEADs and
GETs.
This patch modifies the container server so that a POST will not
update the container put_timestamp if an X-Backend-No-Timestamp-Update
header is included with the request and has a truthy value.
Note: there are already circumstances in which container sysmeta is
modified without changing the put_timestamp:
- PUT requests with shard range content do not update put_timestamp.
- the sharder updates sysmeta directly via the ContainerBroker without
modifying put_timestamp.
Change-Id: I835b2dd58bc1d4fb911629e4da2ea4b9697dd21b
To make it easier to have access to the sharding stats add
/recon/sharding as a recon middleware endpoint.
This allows an easy way to ask a container server for it's sharding
stats using REST inside the cluster:
curl <container-server>/recon/sharding
Also add a get_recon method to the direct client so it can also be used
easily inside tooling and probe tests.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I2a6024277d1198d8c996682682bfe28797344951
A new header `X-Backend-Use-Replication-Network` is added; if true, use
the replication network instead of the client-data-path network.
Several background daemons are updated to use the replication network:
* account-reaper
* container-reconciler
* container-sharder
* container-sync
* object-expirer
Note that if container-sync is being used to sync data within the same
cluster, the replication network will only be used when communicating
with the "source" container; the "destination" traffic will continue to
use the configured realm endpoint.
The direct and internal client APIs still default to using the
client-data-path network; this maintains backwards compatibility for
external tools written against them.
UpgradeImpact
=============
Until recently, servers configured with
replication_server = true
would only handle REPLICATE (and, in the case of object servers, SSYNC)
requests, and would respond 405 Method Not Allowed to other requests.
When upgrading from Swift 2.25.0 or earlier, remove the config option
and restart services prior to upgrade to avoid a flood of background
daemon errors in logs.
Note that some background daemons find work by querying Swift rather
than walking local drives that should be available on the replication
network:
* container-reconciler
* object-expirer
Previosuly these may have been configured without access to the
replication network; ensure they have access before upgrading.
Closes-Bug: #1883302
Related-Bug: #1446873
Related-Change: Ica2b41a52d11cb10c94fa8ad780a201318c4fc87
Change-Id: Ieef534bf5d5fb53602e875b51c15ef565882fbff
Reserve the namespace starting with the NULL byte for internal
use-cases. Backend services will allow path names to include the NULL
byte in urls and validate names in the reserved namespace. Database
services will filter all names starting with the NULL byte from
responses unless the request includes the header:
X-Backend-Allow-Reserved-Names: true
The proxy server will not allow path names to include the NULL byte in
urls unless a middlware has set the X-Backend-Allow-Reserved-Names
header. Middlewares can use the reserved namespace to create objects
and containers that can not be directly manipulated by clients. Any
objects and bytes created in the reserved namespace will be aggregated
to the user's account totals.
When deploying internal proxys developers and operators may configure
the gatekeeper middleware to translate the X-Allow-Reserved-Names header
to the Backend header so they can manipulate the reserved namespace
directly through the normal API.
UpgradeImpact: it's not safe to rollback from this change
Change-Id: If912f71d8b0d03369680374e8233da85d8d38f85
I'm tired of creating code churn where I just slap
.decode("nearly arbitrary choice of encoding")
in a bunch of places.
Change-Id: I79b2bc59fed130ca537e96c1074212861d7db6b8
Changing gen_headers to set User-Agent if not already added
and set x-timestamp by default if it's not already set nor
it's requested not to be put.
Change-Id: Ied073778cf5d34ddcb2f5af59d5b762e51c9b11f
Partial-Bug: 1757250
I wanna see how far I can get *without* mucking around in swob and
request_helpers. Maybe eventually we can get some helpers out there to
make working with UTF-8-pretending-to-be-Latin-1 strings better, but for
the time being, I feel more at ease *embracing* the crazy.
Change-Id: I0b9983a182daedd9dbec483b805d263238fcfac7
Co-Authored-By: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Support PUTs to container server with json serialized ShardRanges in
body. Shard range PUTs may autocreate containers.
Support GET of shard ranges from container server.Shard range GETs
support X-Backend-Include-Deleted to include deleted shard ranges in
list and X-Backend-Override-Delete to get shard ranges when container
has been marked as deleted.
The X-Backend-Record-Type = ['object'|'shard'|'auto'] is introduced
to differentiate container server requests for object versus shard
ranges. When 'auto' is used with a GET request the container server
will return whichever record type is appropriate for fetchng object
listings, depending on whether the container is sharded or not.
Support container PUTs with body in direct_client .py
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I029782ae348f38c5fb76d2759609f67a06c883ef
Make some json -> (text, xml) stuff in a common module, reference that in
account/container servers so we don't break existing clients (including
out-of-date proxies), but have the proxy controllers always force a json
listing.
This simplifies operations on listings (such as the ones already happening in
decrypter, or the ones planned for symlink and sharding) by only needing to
consider a single response type.
There is a downside of larger backend requests for text/plain listings, but
it seems like a net win?
Change-Id: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d
Because the direct_client module uses the buffered_http module it's
requests are already robust to receiving either unicode or utf-8 paths.
The DirectClientException message however encodes the given path with
the device key from a ring node - which having come from a json
de-serialized ring will be a unicode type. Despite the device key
almost always being only ascii characters; python string interpolation
with any unicode type will always force all binary strings to be
converted to unicode - which will raise an error if any byte strings
includes non-ascii characters.
To maintain robustness in DirectClientException, when the provided path
is not already unicode we decode the bytes as utf-8 before mixing them
with the other unicode strings and then normalize everything back to a
quoted utf-8 byte string.
Change-Id: I162d2e093a3110856d6e1d513de3c7919079c9e7
Often, we want the current timestamp. May as well improve the ergonomics
a bit and provide a class method for it.
Change-Id: I3581c635c094a8c4339e9b770331a03eab704074
Currently the direct_get_container and direct_get_account methods
of the direct client don't support passing in the 'end_marker' and
'reverse' params.
This change adds support for these params in direct client.
Change-Id: I846fc70ff3abdb1674152a8d9e0521c709f254c4
When direct_client.direct_get_suffix_hashes raises a
DirectClientException the exception message and variables
should report the replication_ip and replication_port, as
opposed to the ip and port values reported for all other
case when the exception is raised.
Add option to override ip and port reported in
DirectClientException.
Also adds unit tests to verify both cases.
Related-Bug: 1566395
Change-Id: If3d952847c7199f4e9f6164858085367266386d2
direct_client.direct_get_suffix_hashes doesn't use replication ip and
port for REPLICATE request. Since we have an option of doing
replication in separate network, we can add replication_ip and port
while creating rings if not it will get filled in with the regular
node's ip.
Change-Id: I34067df27042fc3146b795191ab8043ee1aed3ce
Closes-Bug:1566395
There was a function in swift.common.utils that was importing
swob.HeaderKeyDict at call time. It couldn't import it at compilation
time since utils can't import from swob or else it blows up with a
circular import error.
This commit just moves HeaderKeyDict into swift.common.header_key_dict
so that we can remove the inline import.
Change-Id: I656fde8cc2e125327c26c589cf1045cb81ffc7e5
Consider two replicas of the same object whose ondisk files
have diverged due to failures:
A has t2.ts
B has t1.data, t4.meta
(The DELETE at t2 did not make it to B. The POST at t4 was
rejected by A.)
After ssync replication the two ondisk file sets will not be
consistent:
A has t2.ts (ssync cannot POST t4.meta to this node)
B has t2.ts, t4.meta (ssync should not delete t4.meta,
there may be a t3.data somewhere)
Consequenty the two nodes will report different hashes for the
object's suffix, and replication will repeat, always with the
inconsistent outcome. This scenario is reproduced by the probe
test added in this patch.
(Note that rsync replication does result in (t2.ts, t4.meta)
on both nodes.)
The solution is to change the way that suffix hashes are
calculated. Currently the names of *all* files found in each
object dir are added to the hash. With this patch the
timestamps of only those files that could be used to
construct a valid diskfile are added to the hash. File
extensions are appended to the timestamp so that in most
'normal' situations the result of the hashing is the same
as before this patch. That avoids a storm of hash mismatches
when this patch is deployed in an existing cluster.
In the problem case described above, t4.meta is no longer
added to the hash, since it is not useful for constructing
a diskfile. (Note that t4.meta is not deleted because it
may become useful should a t3.data be replicated in future).
Closes-Bug: 1534276
Change-Id: I99e88b8d5f5d9bc22b42112a99634ba942415e05
a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.
There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.
(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)
Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a
The builtin basestring type was removed in Python 3. Replace it with
six.string_types which works on Python 2 and Python 3.
Change-Id: Ib92a729682322cc65b41050ae169167be2899e2c
openstack/python-swiftclient@5ae4b423 changed python-swiftclient's
ClientException to have its http_status attribute default to
None (rather than 0) and to use super in its __init__ method. For
consistency's sake, it's nice for Swift's inlined copy of
ClientException to receive the same patch. Also, the retry function in
direct_client (a major user of ClientException) was using a somewhat
awkward conditional-assignment-and-delete construction where the .pop
method of dictionaries would be more idiomatic.
Change-Id: I70a12f934f84f57549617af28b86f7f5637bd8fa
Fix pep8 warnings of the E category of hacking 0.10:
* E113: unexpected indentation
* E121: continuation line under-indented for hanging indent
* E122: continuation line missing indentation or outdented
* E123: closing bracket does not match indentation of opening bracket's
line
* E126: continuation line over-indented for hanging indent
* E251: unexpected spaces around keyword / parameter equals
Change-Id: I0b24eebdf1a37dc1b572b6c9a3d3d4832d050237
Get configparser, queue, http_client modules from six.moves.
Patch generated by the six_moves operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Change-Id: I666241ab50101b8cc6f992dd80134ce27327bd7d
The iteritems() of Python 2 dictionaries has been renamed to items() on
Python 3. According to a discussion on the openstack-dev mailing list,
the overhead of creating a temporary list using dict.items() on Python 2
is very low because most dictionaries are small:
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Patch generated by the following command:
sed -i 's,iteritems,items,g' \
$(find swift -name "*.py") \
$(find test -name "*.py")
Change-Id: I6070bb6c684be76e8e77222a7d280ec6edd43496
Added a missing docstring to the direct_delete_container method.
Also checked other docstrings in the same file and fixed a wrong docstring
element in direct_delete_object. Added raises: docstring entry to all methods
that raise an Exception.
Change-Id: If463a0f9ddff3fe2d13f6d97fcfa955e91d0f01f
The account reaper is using a unique timestamp when deleting
replicas of the same resource. This will result in unnecessary
replication traffic after reaping.
This patch makes the reaper use a single timestamp per resource.
Probe test is modified to check that delete times are equal
across replicas before replicators run.
test_direct_client.py is modified to check that it uses explicit
timestamp when passed to direct_delete_[object|container]
methods.
Drive-by bug fixes in the probe test e.g. it was not sending
X-Backend-Storage-Policy-Index when doing a direct GET to check
object state, so the 404s being verified could in fact be due
to diskfile not existing rather than diskfile being deleted.
Closes-Bug: 1442879
Change-Id: I8bab22d66308bb9d3294e1e0def017c784228423
The call to _get_direct_account_container in direct_get_account
has several of its args =None instead of set to the value passed
to direct_get_account.
The same applies to _get_direct_account_container in
direct_get_container.
The direct_get_container is only called by the account-reaper
and this bug will have limited impact on it. The marker,
maintained in reap_container, is ignored by direct_get_container.
This is not as bad as it sounds, if the account-reaper successfully
deletes the first 10K objects, assuming the container has > 10K
objects, the next call to direct_get_container will in fact return
the next 10K objects even though it sets marker=None (assuming the
first 10K objects were successfully deleted).
This patch also updates test_direct_get_account and
test_direct_get_container to ensure the appropriate
args are included in the connection query_string.
Closes-Bug: #1369558
Change-Id: If1c8aa1240d38354ebc9b1ebca92dc1c8c36cb5f
The normalized form of the X-Timestamp header looks like a float with a fixed
width to ensure stable string sorting - normalized timestamps look like
"1402464677.04188"
To support overwrites of existing data without modifying the original
timestamp but still maintain consistency a second internal offset
vector is append to the normalized timestamp form which compares and
sorts greater than the fixed width float format but less than a newer
timestamp. The internalized format of timestamps looks like
"1402464677.04188_0000000000000000" - the portion after the underscore
is the offset and is a formatted hexadecimal integer.
The internalized form is not exposed to clients in responses from Swift.
Normal client operations will not create a timestamp with an offset.
The Timestamp class in common.utils supports internalized and normalized
formatting of timestamps and also comparison of timestamp values. When the
offset value of a Timestamp is 0 - it's considered insignificant and need not
be represented in the string format; to support backwards compatibility during
a Swift upgrade the internalized and normalized form of a Timestamp with an
insignificant offset are identical. When a timestamp includes an offset it
will always be represented in the internalized form, but is still excluded
from the normalized form. Timestamps with an equivalent timestamp portion
(the float part) will compare and order by their offset. Timestamps with a
greater timestamp portion will always compare and order greater than a
Timestamp with a lesser timestamp regardless of it's offset. String
comparison and ordering is guaranteed for the internalized string format, and
is backwards compatible for normalized timestamps which do not include an
offset.
The reconciler currently uses a offset bump to ensure that objects can move to
the wrong storage policy and be moved back. This use-case is valid because
the content represented by the user-facing timestamp is not modified in way.
Future consumers of the offset vector of timestamps should be mindful of HTTP
semantics of If-Modified and take care to avoid deviation in the response from
the object server without an accompanying change to the user facing timestamp.
DocImpact
Implements: blueprint storage-policies
Change-Id: Id85c960b126ec919a481dc62469bf172b7fb8549
Rework header handling and add some methods needed by the reconciler.
* response headers are case insensitive HeaderKeyDicts
* add direct client container obj put and delete
* add headers param to direct head object
* add headers to DirectClientException
DirectClientException is a subclass of ClientException with a convience
constructor. ClientException now supports an http_headers kwarg.
Exceptions raised from direct_client will include headers.
DocImpact
Implements: blueprint storage-policies
Change-Id: Ia484d569619df0bf85f973e4e916de2ac6401d5e
Place all the methods related to on-disk layout and / or configuration
into a new common module that can be shared by the various modules
using the same on-disk layout.
Change-Id: I27ffd4665d5115ffdde649c48a4d18e12017e6a9
Signed-off-by: Peter Portante <peter.portante@redhat.com>
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
- Add a base function _get_direct_account_container to do the direct
account stuff instead of duplicating both methods.
Change-Id: I12059a741dd22d0dde20012d1dbc4be9b8e90957
Enhance internally logged messages to report referer and user-agent.
Pass the referering URL and METHOD between internal servers (when
known), and set the user-agent to be the server type (obj-server,
container-server, proxy-server, obj-updater, obj-replicator,
container-updater, direct-client, etc.) with the process PID. In
conjunction with the transaction ID, it helps to track down which PID
from a given system was responsible for initiating the request and
what that server was working on to make this request.
This has been helpful in tracking down interactions between object,
container and account servers.
We also take things a bit further performaing a bit of refactoring to
consolidate calls to transfer_headers() now that we have a helper
method for constructing them.
Finally we performed further changes to avoid header key duplication
due to string literal header key values and the various objects
representing headers for requests and responses. See below for more
details.
====
Header Keys
There seems to be a bit of a problem with the case of the various
string literals used for header keys and the interchangable way
standard Python dictionaries, HeaderKeyDict() and HeaderEnvironProxy()
objects are used.
If one is not careful, a header object of some sort (one that does not
normalize its keys, and that is not necessarily a dictionary) can be
constructed containing header keys which differ only by the case of
their string literals. E.g.:
{ 'x-trans-id': '1234', 'X-Trans-Id': '5678' }
Such an object, when passed to http_connect() will result in an
on-the-wire header where the key values are merged together, comma
separated, that looks something like:
HTTP_X_TRANS_ID: 1234,5678
For some headers in some contexts, this is behavior is desirable. For
example, one can also use a list of tuples which enumerate the multiple
values a single header should have.
However, in almost all of the contexts used in the code base, this is
not desirable.
This behavior arises from a combination of factors:
1. Header strings are not constants and different lower-case and
title-case header strings values are used interchangably in the
code at times
It might be worth the effort to make a pass through the code to
stop using string literals and use constants instead, but there
are plusses and minuses to doing that, so this was not attempted
in this effort
2. HeaderEnvironProxy() objects report their keys in ".title()"
case, but normalize all other key references to the form
expected by the Request class's environ field
swob.Request.headers fields are HeaderEnvironProxy() objects.
3. HeaderKeyDict() objects report their keys in ".lower()" case,
and normalize all other key references to ".lower()" case
swob.Response.headers fields are HeaderKeyDict() objects.
Depending on which object is used and how it is used, one can end up
with such a mismatch.
This commit takes the following steps as a (PROPOSED) solution:
1. Change HeaderKeyDict() to normalize using ".title()" case to
match HeaderEnvironProxy()
2. Replace standard python dictionary objects with HeaderKeyDict()
objects where possible
This gives us an object that normalizes key references to avoid
fixing the code to normalize the string literals.
3. Fix up a few places to use title case string literals to match
the new defaults
Change-Id: Ied56a1df83ffac793ee85e796424d7d20f18f469
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Before, swift-bench would create a string of the right size and pass
that to [direct_]put_object. Uploading 5 GB objects w/concurrency of
4? Hope you've got a spare 20 GB of RAM in that machine.
Now it lazily generates chunks (of size 64 KiB) to send. In my limited
testing, this keeps RAM usage low while not impacting speed.
There's backwards-compatibility code in
direct_client.direct_put_object for any callers who are still passing
in strings. There aren't any such in Swift's codebase, though. All one
of those callers now pass an iterable.
Fixes bug 911925.
Change-Id: I40669fc00c7fb6699d8fb514e1be9b69436eca42
- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.
Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0
Jianjian Huo