Commit Graph

10536 Commits

Author SHA1 Message Date
Zuul ec3224cc18 Merge "Clean up watchdog threads" 2024-05-02 18:17:12 +00:00
Zuul c12e4ac9db Merge "CI: Add rolling-upgrade job for Caracal" 2024-05-02 16:31:55 +00:00
Tim Burke 38d0b3fabc Clean up watchdog threads
This shouldn't impact real servers, as those processes were about to
wrap up anyway. It *can* cause some confusing behaviors in tests,
though.

Change-Id: Ifd8a64efcd3fc983596ba7cd9fe28eb9663c93d6
2024-05-02 08:30:54 -07:00
Tim Burke e3eab3facb CI: Add rolling-upgrade job for Caracal
Change-Id: Ief0771dcb17c4ed3d239d60872454e2d35936e39
2024-05-01 22:10:38 -07:00
Tim Burke 761d919677 tests: Use mock.patch more
Change-Id: I68974338f8e0284ed77960048a83f72855b93348
2024-05-01 17:33:11 -07:00
Tim Burke b4dddb7406 tests: Use @with_tempdir more
Change-Id: I33e71f6c201bb4f2cf3481afd40cf489eb1fcd1f
2024-05-01 17:30:52 -07:00
Zuul 0a3c46e6c4 Merge "Refactor utils" 2024-05-02 00:27:02 +00:00
Shreeya Deshpande bc3a59bdd3 Refactor utils
- Move statsd client into it's own module
- Move all logging functions into their own module
- Move all config functions into their own module
- Move all helper functions into their own module

Partial-Bug: #2015274
Change-Id: Ic4b5005e3efffa8dba17d91a41e46d5c68533f9a
2024-04-30 20:27:47 +00:00
OpenStack Release Bot 51e4b6e279 reno: Update master for unmaintained/zed
Update the zed release notes configuration to build from
unmaintained/zed.

Change-Id: I852953d8c06e44b4416818ae1ef1b1cf70864fb2
2024-04-30 11:06:13 +00:00
OpenStack Proposal Bot aa8604645c Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: Ifa3ea75a59bf946c800331c642574c87629ddc92
2024-04-27 02:55:15 +00:00
Anish Kachinthaya 1d30e8e696 Add Anish to authors.
Change-Id: I0441226e279b36a787ec85a459d6d1b72a2c8c84
2024-04-26 11:51:17 -07:00
Zuul 937af35e62 Merge "object-expirer: add example to delay_reaping sample config" 2024-04-26 15:14:34 +00:00
Zuul f7b9a72a6e Merge "object-expirer: test for mix of async and delayed deletes" 2024-04-26 15:04:15 +00:00
Zuul b6c377e7e5 Merge "support x-open-expired header for expired objects" 2024-04-26 13:07:57 +00:00
Zuul 2e487ba695 Merge "expirer: account and container level delay_reaping" 2024-04-26 10:28:47 +00:00
Clay Gerrard b490857b47 object-expirer: test for mix of async and delayed deletes
Related-Change: I106103438c4162a561486ac73a09436e998ae1f0
Change-Id: I95606aed5a5bd70424fbc51dd965f238fa7f064f
2024-04-26 10:53:19 +01:00
indianwhocodes 11eb17d3b2 support x-open-expired header for expired objects
If the global configuration option 'enable_open_expired' is set
to true in the config, then the client will be able to make a
request with the header 'x-open-expired' set to true in order
to access an object that has expired, provided it is in its
grace period. If this config flag is set to false, the client
will not be able to access any expired objects, even with the
header, which is the default behavior unless the flag is set.

When a client sets a 'x-open-expired' header to a true value for a
GET/HEAD/POST request the proxy will forward x-backend-open-expired to
storage server. The storage server will allow clients that set
x-backend-open-expired to open and read an object that has not yet
been reaped by the object-expirer, even after the x-delete-at time
has passed.

The header is always ignored when used with temporary URLs.

Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Related-Change: I106103438c4162a561486ac73a09436e998ae1f0
Change-Id: Ibe7dde0e3bf587d77e14808b169c02f8fb3dddb3
2024-04-26 10:13:40 +01:00
Alistair Coles ce619137db object-expirer: add example to delay_reaping sample config
Add an example of a delay_reaping config option with quoted key.

Change-Id: I0c7ead6795822ea0fb0e81abc1e4685d7946942c
Related-Change: I106103438c4162a561486ac73a09436e998ae1f0
2024-04-26 09:37:40 +01:00
Mandell Degerness 5961ba0ca7 expirer: account and container level delay_reaping
The object expirer can be configured to delay the reaping of
objects from disk after their expiration time using account
and container level delay_reaping values. The delay_reaping
value of accounts and containers in seconds is configured in
the object server config. The object expirer references these
configured values to only reap objects from specified accounts
and containers after their corresponding delays.

The goal of the delay_reaping feature is to prevent accidental or
premature data loss if an object marked for deletion with the
'x-delete-at' feature should not be reaped immediately, for
whatever reason.

Configuring the delay_reaping value at a granular account and
container level is beneficial for being able to keep storage
capacity consumption in control while maintaining a desired
data recovery window.

This patch also adds a sample configuration, documentation, and
tests for bad configurations and grace period functionality.

Co-Authored-By: Anish Kachinthaya <akachinthaya@nvidia.com>
Change-Id: I106103438c4162a561486ac73a09436e998ae1f0
2024-04-25 13:59:36 -07:00
Zuul 5ba1fbf86b Merge "proxy_logging: unit test first-byte.timing metrics" 2024-04-19 16:36:09 +00:00
Zuul 22c4c9ba1c Merge "proxy-logging: emit stats more consistently" 2024-04-19 13:40:49 +00:00
Zuul dcf9c7ef54 Merge "test: assert behavior of proxy_logging metrics" 2024-04-19 13:29:05 +00:00
Alistair Coles 3d5a97f76b proxy_logging: unit test first-byte.timing metrics
Add some test assertions to cover the first-byte timing metrics
introduced in the related change.

Add ttfb param to log_request docstring.

Change-Id: I530652dd672d7d4e5eac351ccbad318773414f7d
Related-Change: I1611e34846e586703e9d3709fa64e8df41f2d685
2024-04-19 12:33:48 +01:00
Clay Gerrard 6387949a54 lower-constraints: update to mock 3.0
The main motivation here is that mock.call becomes a namedtuple and you
can say `m.call_args_list[0].args` instead of `m.call_args_list[0][0]`

Change-Id: Ibb1a64ef0bfdebf06d26636cdb6ea191c10705f7
2024-04-04 16:54:40 -05:00
Clay Gerrard d4435e1229 proxy-logging: emit stats more consistently
Change-Id: I526bbcc59c9eb5923c3784d5d06bc38998cb48db
2024-04-03 18:50:04 -05:00
Clay Gerrard 3c449e78e4 test: assert behavior of proxy_logging metrics
Change-Id: I651ddd40e9115a56727096d4a3aa84589146308f
2024-04-03 18:49:13 -05:00
Zuul 0e5aeb5045 Merge "s3api: Fix handling of non-ascii access keys" 2024-03-25 21:40:58 +00:00
Zuul cdc4f264d2 Merge "recon-cron: Tolerate missing directories" 2024-03-25 06:45:56 +00:00
Tim Burke 8424b02290 s3api: Fix handling of non-ascii access keys
We stuff the access key into the request path until we get back a
more-authoritative account name from auth. But it needs to be a WSGI
string when we do!

Closes-Bug: #2058748
Change-Id: I34adb8141cc9e62d17a27f01c63f40d1dd25991c
2024-03-22 10:02:39 -07:00
Tim Burke f31b6f7353 recon-cron: Tolerate missing directories
Any of these directories may get unlinked between when we saw them in
their parent's directory listing and when we go to descend.

Change-Id: I1dfc0ee1d9e70cb0600557cde980bd5880bd40b3
2024-03-21 14:10:14 -07:00
Zuul 537b6885e3 Merge "Update master for stable/2024.1" 2024-03-21 16:21:07 +00:00
Zuul d3938c3547 Merge "CI: Move probe tests to centos 9 stream" 2024-03-21 01:46:32 +00:00
Zuul fd3997f027 Merge "tests: Update CORS geckodriver" 2024-03-20 03:48:32 +00:00
OpenStack Release Bot 5d25108648 Update master for stable/2024.1
Add file to the reno documentation build to show release notes for
stable/2024.1.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2024.1.

Sem-Ver: feature
Change-Id: Ic940ff424aef9cc402bf54ebe5e5fc16330fc25c
2024-03-15 08:27:17 -07:00
Tim Burke 1c03d58643 CI: Move probe tests to centos 9 stream
Pin selenium to 3.x for now, until we can run down the issues with 4.x

Change-Id: I596415d17f77f48a6e8a63a61b734a8ca0865847
2024-03-15 08:09:25 -07:00
Tim Burke af15ad53fb tests: Update CORS geckodriver
Change-Id: I5ab762dfe0f85e346c4868ec4540884ba5f0a7f4
2024-03-14 20:45:47 -07:00
Jianjian Huo 27ef11ea14 test: implement cache expiration time in MockMemcached
Change-Id: I16ec414f87ac1a5e1e87e7560290c5ef0ca4f7cf
2024-03-15 13:50:32 +11:00
Zuul b6dc24dbc0 Merge "s3api test for zero byte mpu" 2024-03-13 17:21:42 +00:00
Zuul 891d06345e Merge "s3api: Support GET/HEAD request with ?partNumber" 2024-03-13 08:28:03 +00:00
Zuul 60db1f847c Merge "slo: part-number=N query parameter support" 2024-03-13 00:13:45 +00:00
Tim Burke 5f95b6b26d AUTHORS/CHANGELOG for 2.33.0
Change-Id: I6b0a074fca4d6bd3b8b95888ae8721a62f3a6df0
2024-03-12 11:59:59 -07:00
Zuul 7941cf0efc Merge "CI: Bring unit test jobs in line with 2024.1 tested runtimes" 2024-03-12 18:51:46 +00:00
Zuul 2c4ff7bf2c Merge "CI: Update rolling-upgrade jobs to point to unmaintained branches" 2024-03-12 18:39:52 +00:00
Zuul b6f5971471 Merge "reno: Update master for unmaintained/wallaby" 2024-03-12 17:23:07 +00:00
Zuul e951a42788 Merge "reno: Update master for unmaintained/victoria" 2024-03-12 17:23:04 +00:00
Tim Burke 9279e3d2c0 CI: Bring unit test jobs in line with 2024.1 tested runtimes
The last time I really looked at this was probably Yoga, when we were
targetting 3.6 through 3.9 (and left 3.7 and 3.8 as experimental jobs).

Now, though, OpenStack is targetting 3.8 through 3.11; as before, we
can assume that if tests pass on those two versions, they should pass
on the versions in-between, too. (But still have them as experimental,
on-demand jobs).

See https://governance.openstack.org/tc/reference/runtimes/2024.1.html

Keep 2.7 and 3.6 testing as our own self-imposed minimums.

Change-Id: I7700aa3c93df311644655e7ebaf0b67aa692ee80
2024-03-12 09:19:53 -07:00
Tim Burke b06ffea941 CI: Update rolling-upgrade jobs to point to unmaintained branches
Change-Id: I936d9074ab60e34b379fb207d63f10bd5c3a4312
2024-03-12 09:06:23 -07:00
Clay Gerrard d10351db30 s3api test for zero byte mpu
Change-Id: I89050cead3ef2d5f8ebfc9cb58f736f33b1c44fe
2024-03-12 13:48:02 +00:00
indianwhocodes 46e7da97c6 s3api: Support GET/HEAD request with ?partNumber
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Closes-Bug: #1735284
Change-Id: Ib396309c706fbc6bc419377fe23fcf5603a89f45
2024-03-12 13:47:55 +00:00
indianwhocodes 6adbeb4036 slo: part-number=N query parameter support
This change allows individual SLO segments to be downloaded by adding
an extra 'part-number' query parameter to the GET request.  You can
also retrieve the Content-Length of an individual segment with a HEAD
request.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I7af0dc9898ca35f042b52dd5db000072f2c7512e
2024-03-12 06:47:02 -07:00