Commit Graph

515 Commits

Author SHA1 Message Date
Zuul fa9718deee Merge "Add transaction id to errors" 2024-03-01 20:38:52 +00:00
Yan Xiao ac2a60dc0d Add transaction id to errors
Add transaction id to errors to help troubleshoot, including error
when downloading object with truncted/missing segment, stat'ing or
listing non-existent container.

Change response headers returned by client.py Connection APIs
such as get_object() from lowercase dict to CaseInsensitiveDict,
as some error paths rely on this to retrieve the transaction id,
while there are existing code paths using response.headers which
is CaseInsensitiveDict from requests lib.

Change-Id: Ice9cc9fe68684563f18ee527996e5a4292230a96
2024-03-01 09:59:33 -05:00
Zuul 9366d86220 Merge "shell: Print friendly account byte quotas" 2024-02-27 07:32:53 +00:00
Matthew Oliver 1e0ab242ca lint: Up-rev hacking
Without up-reving hacking python 3.12 can't be used for pep8/flake8
checks. Also address a few new pyflakes issues that seem reasonable
to enforce:

   E275 missing whitespace after keyword
   H214 Use assertIn/NotIn(A, B) rather than
        assertTrue/False(A in/not in B)
   H211 do not compare types, use assert{Is,IsNot}Instance

Add F811 (redefinition of unused 'function'), as this check makes
client.py fail.

And noqa some longer lines that we're happy to carry.

Change-Id: Ic97698a310ffe7114b668b8c4bc0195f6997bb45
2024-02-21 17:14:27 +00:00
Tim Burke 4698f1f642 shell: Print friendly account byte quotas
This gives us an opportunity to give it more human-friendly units
when doing a `stat --lh`.

Change-Id: I01c02ffcba94d89ae03f87c1c64afca4d491a895
2023-10-06 10:00:39 -07:00
Zuul 1ffe514621 Merge "Ensure v*password auth_type correctly implies auth version" 2023-03-24 18:02:58 +00:00
Zuul fe013f7417 Merge "trivial: Refactor os_options processing" 2023-03-24 18:02:57 +00:00
Tim Burke ee952c1bf2 Ensure v*password auth_type correctly implies auth version
Change-Id: I787b0988b7518850f1e1f93ca269f39f8cdec15c
2023-03-23 20:32:23 -07:00
Tim Burke 825145e9e6 trivial: Refactor os_options processing
Change-Id: I01c096084172ea8ebeda791016a603c59c7924cd
2023-03-23 20:32:23 -07:00
Tim Burke 9c45343845 shell: Allow uploads from stdin to specify --use-dlo
Change-Id: Id6818ab416380242cc215a980a541fc59109c69c
2023-03-23 09:05:45 -07:00
Zuul 31c279ff0e Merge "Use SLO by default for segmented uploads if the cluster supports it" 2023-03-22 16:00:44 +00:00
Tim Burke 6a8675e897 Use SLO by default for segmented uploads if the cluster supports it
We've had SLO for nearly ten years, and it is preferable for most
use-cases. It's time to default to SLO rather than DLO.

Add a new --use-dlo option to give users a way to return to old
behavior; ensure there is still a --use-slo option so we don't break
existing scripts that may use it.

UpgradeImpact:
==============
The default segmented-upload behavior has changed; Static Large Objects
are now used by default rather than Dynamic Large Objects. To revert to
the old behavior:

* CLI users may use the new `--use-dlo` option
* Service API users may explicitly set `use_slo` to False in their
  options dicts

Change-Id: Iebcd64ced8eab6efebb9f02a0c4be6bf6338cb3b
2023-03-21 12:46:44 -07:00
Zuul 62641a2f49 Merge "Retry on SSLError" 2023-03-21 19:37:06 +00:00
Tim Burke 9e703b9345 shell: Allow timeouts to have units
Related-Change: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab
Change-Id: Ifeaaea790d1dadc84b157a7cf2be7590949c70f0
2023-03-20 17:23:38 -07:00
Tim Burke 099dd66265 Retry on SSLError
...provided it isn't something to do with certification validation.

Closes-Bug: #1641363
Closes-Bug: #2011731
Change-Id: Ic3acbf431e444fcd8bc0fe79571fe2db4140cf22
2023-03-20 16:57:15 -07:00
Tim Burke e343298240 Include transaction ID on content-check failures
Change-Id: I6b667db26ffc5dccdcadfc8c73f7accb81f03dac
2023-03-15 12:37:02 -07:00
Zuul d572ccfae9 Merge "service: Check content-length before etag" 2023-03-15 11:30:08 +00:00
Tim Burke 6adb8bb17f service: Check content-length before etag
If the received content-length does not match expectations, of course
the etag won't match!

Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I1a0c066c11b94718fffbb11e13b82d0b16e01626
2023-03-14 11:56:03 -07:00
Tim Burke 16f0034e67 Switch from pkg_resources to importlib
Seen on Arch Linux:

    ModuleNotFoundError: No module named 'pkg_resources'

On new enough Python, we don't need it anyway.

Change-Id: I3d46319a36a1c2477e9f7dae0c7e40a362a17639
2023-03-14 10:41:52 -07:00
Tim Burke ff6b359d06 Retry with fresh socket on 499
Change-Id: I0c22eefb587375997672724c03744c9cda473708
2023-02-15 21:01:07 -08:00
Tim Burke f4e62191bc Allow tempurl to be used to sign /info requests
Change-Id: I1f9d26541e9c8f5aec7a6790c87df397d178efe6
2022-08-29 13:13:34 -07:00
Zuul 662e530d8d Merge "Allow tempurl times to have units" 2022-08-25 07:38:36 +00:00
Zuul 02509acc71 Merge "Add option to skip container PUT during upload" 2022-08-23 00:37:52 +00:00
Tim Burke defbb4a8f3 Allow tempurl times to have units
Specifically, let users add a suffix for seconds, minutes, hours, or
days.

Change-Id: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab
2022-08-19 17:40:10 -07:00
Zuul c1f5e8b7c1 Merge "Expand retry handling on ratelimit response" 2022-08-19 00:43:25 +00:00
Zuul 3b30f155a7 Merge "Enable retry_on_ratelimit by default" 2022-08-19 00:42:56 +00:00
Clay Gerrard 653cbcb686 Expand retry handling on ratelimit response
We have seen middlewares that return ratelimit responses as 498 or 429,
so tolerate either.

Closes-Bug: #1879572
Change-Id: I027222157f6c2ad7882a0508302c9de097baae4c
2022-08-18 12:14:04 -07:00
Tim Burke a1d2f31131 Enable retry_on_ratelimit by default
UpgradeImpact
=============
The Connection class now enables retry_on_ratelimit by default.
If you need to return to the old behavior, explicitly pass
retry_on_ratelimit=False as a keyword arg.

The SwiftService class will now enables the retry_on_ratelimit option by
default. If you need to return to the old behavior, explicitly set it to
false in your options dict.

Change-Id: I3221fda84f0b8031c50128aa600e2c19deb5b102
2022-08-18 12:12:35 -07:00
Zuul 7f2649bfb7 Merge "More cleanup following py2 removal" 2022-08-18 01:49:01 +00:00
Tim Burke 9eee29d2e4 tempurl: Support sha256 and sha512 signatures
Up the default to sha256 since

* the proxy has supported (and defaulted to allowing) it for four years
  now, and
* Rackspace has supported it for even longer.

Include a note in the --help about older clusters likely requiring sha1.

Change-Id: Ibac2bb7e2e4c9946c7384f0aab8e43d0d79ba645
Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Related-Bug: #1733634
Closes-Bug: #1977867
2022-06-08 09:30:21 -07:00
Tim Burke 5d451fb920 More cleanup following py2 removal
* Drop py2-only hacking pin from test-requirements.

* Remove quote() helper; urllib.parse.quote() works fine.

* Remove some useless code.

Change-Id: I9ffc923f58f1d11538f83ff26f7beb53cdf134c3
2022-05-24 08:33:02 -07:00
Pavel Abalikhin 95f68cd673 Add timeout for Swift service
Connection class has timeout parameter but SwiftService and shell don't use it.
That can lead to an endless wait when network is unreachable.

Change-Id: Iafa42fc2f8b56feefa2bc8ea6a1b8845717d3bab
2022-04-21 18:00:27 +03:00
Stephen Finucane 20d837a276 Remove unnecessary unicode prefixes
All strings are unicode by default in Python 3. No need to mark them as
such.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I68fb60ef271abfddebcc9d2137424f5db2a17e92
2022-03-21 18:32:30 +00:00
Stephen Finucane 61ce5ac824 Remove unnecessary object subclassing
All classes subclass from object by default in Python 3.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I5a1ad57bcc092861ce969759b06a07c880ad3d35
2022-03-21 18:32:00 +00:00
Stephen Finucane fa137a5bf1 Remove six
This mostly affects tests. Nothing too complicated

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Iabc78f651e1d48db35638280722f8019798eccd6
2022-03-21 18:31:10 +00:00
Stephen Finucane 4983b90983 Remove coding comments
Everything is unicode in Python 3.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I6a076dc67c461f265ed99878e3959e1992a88189
2022-03-21 18:29:25 +00:00
Stephen Finucane 3d10744c55 Remove __future__ imports
These aren't needed in modern Python 3 versions.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I5e81d6fb2e2cb8e4bfae4ed746da002f44e871c4
2022-03-21 18:29:25 +00:00
Aarni Koskela c09621eb42 Don't patch Requests globally on import
This also upgrades the Requests dependency to 2.4+ (released in 2014)
to avoid having to do version comparisons altogether.

Refs https://bugs.launchpad.net/python-swiftclient/+bug/1904551

Signed-off-by: Aarni Koskela <akx@iki.fi>
Change-Id: I58399f6c526b0b78462f31739c43076314ba9e76
2022-03-21 08:05:54 -07:00
Stephen Finucane 2636965f38 Drop support for Python 2
There's a lot of cleanup possible, but this is a start.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ia1176b7fd5434d52070d482a37abfbb98800cdb3
2022-03-16 15:06:19 -07:00
Tim Burke f1858d89e0 Add option to skip container PUT during upload
Currently, a user with read/write access to a container (but without
access to creat new containers) recieves a warning every time they
upload. Now, allow them to avoid the extra request and warning by
specifying --skip-container-put on the command line.

This is also useful when testing: developers can HEAD a container to
ensure it's in memcache, shut down all container servers, then upload
and creaate a bunch of async pendings. Previously, the 503 on container
PUT would prevent the object upload from even being attempted.

Closes-Bug: 1317956
Related-Bug: 1204558
Change-Id: I3d9129a0b6b65c6c6187ae6af003b221afceef47
Related-Change: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
2022-01-11 16:39:38 -08:00
Zuul 3f5d5b0252 Merge "Include storage policy when listing account with --long" 2021-10-13 01:11:03 +00:00
Tim Burke 373fa26ce1 Correctly aggregate totals for >10k items
Previously, we would write out totals for every page of listings, like

    $ swift list sync --prefix=09-21 --total -l
     80000000000
     80000000000
     80000000000
     58096000000

Now, roll those all into a single total:

    $ swift list sync --prefix=09-21 --total -l
    298096000000

Change-Id: Icc265636815220e33e8c9eec0a3ab80e9f899038
2021-09-21 16:06:56 -07:00
Tim Burke ad3e8e49d0 Include storage policy when listing account with --long
Change-Id: Ibc2f9445b5a8e80cfb73d0706e20a7e4c62eec4a
2021-09-21 08:40:45 -07:00
Tim Burke 5129b33505 Only log the traceback for non-404s
Change-Id: I08ba4a3120e99b444b13f1ca6f5493529868df26
2021-09-20 15:49:59 -07:00
Tim Burke 553e34ebfe Improve formatting for billions of objects
Change-Id: If8aa08c4c8c8ad6ca2c861602baf1eefa8642a8a
2021-09-20 15:49:59 -07:00
jonasdlindner 58f0700ad9 Fix Typo in shell.py
Change-Id: I2615e0d6b54d8cc020c24d1b5b4064e038f0934b
2021-04-29 17:39:28 +02:00
Tim Burke 74c50dee2d Have `delete --all` imply `--versions` for the CLI
Change-Id: Id5a6d4cef3d4ed76c897a099a62a4ba3ed8f8dab
2020-10-16 13:57:02 -07:00
Zuul bf862981a6 Merge "Close connections created when calling module-level functions" 2020-10-15 22:41:10 +00:00
Tim Burke 97aa3e6541 Close connections created when calling module-level functions
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Id62e63afc6f2ffa32eb8640787c78559481050f9
Related-Change: I200ad0cdc8b7999c3f5521b9a822122bd18714bf
Related-Bug: #1873435
Closes-Bug: #1838775
2020-10-13 10:45:07 -07:00
Zuul d63ec1b38d Merge "Include transaction ID in ClientException.__str__" 2020-09-22 01:56:50 +00:00