Commit Graph

2870 Commits

Author SHA1 Message Date
Zuul db020073ec Merge "reno: Update master for unmaintained/victoria" 2024-03-12 13:17:11 +00:00
Zuul 1d26f6a8f8 Merge "Update master for stable/2024.1" 2024-03-12 13:10:06 +00:00
Takashi Kajinami 4adf2790fe Revert "Use consistent credential for Redis and Redis Sentinel"
This reverts commit 3fbd05078f.

Reason for revert:
Some deployment tools such as kolla already rely on the previous
behavior which requires authentication for only redis.

Conflicts:
	taskflow/jobs/backends/impl_redis.py

Closes-Bug: #2056656
Change-Id: I24e0272c269c6fd287234fd2d3b2754983911a7f
2024-03-11 01:03:34 +00:00
OpenStack Release Bot f635f8cd45 Update master for stable/2024.1
Add file to the reno documentation build to show release notes for
stable/2024.1.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2024.1.

Sem-Ver: feature
Change-Id: Ia061784457bac37b5c09f3e9245417cd5c5acecb
2024-03-08 14:24:09 +00:00
OpenStack Release Bot 9a62640fd3 reno: Update master for unmaintained/victoria
Update the victoria release notes configuration to build from
unmaintained/victoria.

Change-Id: I6162f2d5a4d890ed59ecd3c45233f7609b100834
2024-03-07 14:19:35 +00:00
Takashi Kajinami b389cb5e93 Prevent potential ReDoS attack
Although the logic is used to parse a config value, it'd be better to
eliminate a risk.

Change-Id: Id30a69071ef9c3877f3153b95ee2d00d08c17921
2024-02-13 09:39:38 +09:00
Takashi Kajinami fe74dae2fe redis: Support fallback servers
Sentinel clients can use fallback sentinel services so that clients can
obtain available instances even if some of the sentinel services are
unavailable.

Change-Id: Ibdf8993f7ca912bd5acf75ffbc89cf928b607b99
2024-02-04 15:27:05 +00:00
Takashi Kajinami 3fbd05078f Use consistent credential for Redis and Redis Sentinel
We generally expects that users enabled authentication both in Redis
and Redis Sentinel at the same time, and it'd be more reasonable to
use the same credential by default. This behavior is more consistent
with the other libraries used in tooz or dogpile.cache, which are used
in multiple OpenStack services.

Change-Id: Ie8c96415f72bbd5f11f908bb9f96242d57b2bd4e
2024-02-05 00:26:18 +09:00
Takashi Kajinami 14444acab8 redis: Enable SSL for sentinel
Sentinel class supports ssl arguments since v4.0.0[1]. Setting these
are required to use SSL in Redis Sentinel.

[1] fea7b85dde

Related-Bug: #2052372
Change-Id: I2fdbb2cbd26ffd277b18adf45ebb7312ec6f0a24
2024-02-04 20:07:38 +09:00
Takashi Kajinami 78f3ef26a1 redis: Add username
Redis introduced ACL feature in 4.0.0, and this feature is supported by
redis-py since 3.4.0[1]. When ACL is enabled, authentication requires
username in addition to password.

[1] 8df8cd54d1

Closes-Bug: #2052371
Change-Id: Ie2de917603e752e45377edd81496cc46f5675c2c
2024-02-04 19:52:00 +09:00
Zuul 6aa9a431da Merge "Cleanup setup.py and requirements" 2024-01-29 12:46:04 +00:00
Zuul b46a6c8b6d Merge "Delete the job from backend if it cannot be consumed" 2024-01-29 12:46:02 +00:00
Zuul a0dc4553cb Merge "Bump hacking" 2024-01-29 10:50:08 +00:00
Takashi Kajinami 0000c79203 Bump hacking
hacking 3.0.x is too old.

Change-Id: Idbc349cbd348fb73a45c68c5f5b884c6a877ef4f
2024-01-28 16:04:41 +09:00
Zuul 05a14400c3 Merge "Replace deprecated perl-style regex" 2024-01-23 14:48:48 +00:00
Zuul 5ff3e51192 Merge "Avoid endless backtraces on StorageFailure" 2024-01-23 13:49:52 +00:00
Takashi Kajinami 60c01f2c9a Replace deprecated perl-style regex
Zuul deprecated usage of perl-style regex.

```
  All regular expressions must conform to RE2 syntax, but an
  expression using the deprecated Perl-style syntax has been detected.
  Adjust the configuration to conform to RE2 syntax.
```

Change-Id: I0dd4596cb0138e44122486e699b8f64714e8418a
2024-01-22 15:00:02 +09:00
Zuul c56b292c9e Merge "Fix incorrect handling of storage exceptions" 2024-01-12 10:19:19 +00:00
Takashi Kajinami 0d5c948a74 Cleanup setup.py and requirements
- Python 2 is no longer supported
- setup.py is no longer managed by the global tooling
- Recent pip does not require appropriate order in requirement files

Change-Id: I8853a4221c6da78f860e821be4a063176c5734f3
2024-01-12 17:55:09 +09:00
Ghanshyam Mann 1c4af17623 Update python classifier in setup.cfg
As per the current release tested runtime, we test
python version from 3.8 to 3.11 so updating the
same in python classifier in setup.cfg

Change-Id: Ib88b74861cbd1260183f668b6cf616751f6b03e7
2024-01-11 16:37:06 -08:00
Takashi Kajinami 76d7ce091b coveragerc: Remove non-existent path
The taskflow/openstack directory does not exist.

Change-Id: If5efa6e190745cbfbb627f812fefb60c570a9893
2023-12-19 20:56:21 +09:00
Takashi Kajinami 78d8f7f741 Fix python shebang
The current shebang requires /usr/bin/python which is not available in
Ubuntu Jammy by default.

Change-Id: I8658c1e5c6c2c05df69865663a0fca12c12d4568
2023-10-17 16:04:41 +00:00
Gregory Thiemonge 981052a6cb Avoid endless backtraces on StorageFailure
If the storage is down (for instance sqlalchemy), the logbook cannot be
loaded, then taskflow goes into a endless loop of exceptions. To
mitigate this issue, use the retry feature of tenacity, the backtrace
will be displayed only every 5 seconds

Partial-Bug: #2037050

Change-Id: I656abdf7325c46d3afb2cc7ca905f1a335fb0d2f
2023-09-22 08:26:09 -04:00
Gregory Thiemonge 07a1a3f417 Fix incorrect handling of storage exceptions
Since [0] _save_flow_detail and _save_atom_detail use tenacious.retry,
it means that they no longer raise StorageException but a
tenacious.RetryError which is not caught by the callers.

Some jobs may have been incorrectly consumed (instead of being
rescheduled) when StorageException was raised.
Now tenacious reraises only the last exception (StorageException), which
is correctly handled.

[0] Ic6b0a78d20124cc027468ecc6aeff189c25d1a8a

Closes-Bug: #2037050
Change-Id: I030dd8fc5b65833243cf0948af53dc1aeabf41d9
2023-09-22 03:25:00 -04:00
Zuul cd80fc96de Merge "bindep: Use new mysql-* package names" 2023-09-14 03:27:52 +00:00
Stephen Finucane db27bef4a4 bindep: Use new mysql-* package names
Debian Bookworm switched from MySQL to MariaDB.

Change-Id: Iffacf1bb001eca68937debe8e688002348f4a818
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-09-08 12:56:37 +01:00
OpenStack Release Bot b329ec7ead Update master for stable/2023.2
Add file to the reno documentation build to show release notes for
stable/2023.2.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.

Sem-Ver: feature
Change-Id: I4c568a542aab8b4e45899124f30fbf7e9041749f
2023-09-07 09:39:30 +00:00
Tobias Urdin 3b40c04594 Update redis dep to >=3.0.0
This was not done in [1] when we removed the
StrictRedis alias usage.

[1] https://review.opendev.org/c/openstack/taskflow/+/859541

Change-Id: I8b7f02c090862ea117ba37da90ae6942e57184af
2023-08-16 15:57:37 +02:00
Stephen Finucane 4a29c44699 Add job to test with SQLAlchemy master (2.x)
Change-Id: I3486492c00b8294bfe67fa81a09d6f241f1a6057
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-07-14 14:19:31 +01:00
Stephen Finucane 17a6c571dc db: Don't rely on autocommit behavior
Resolve the following RemovedIn20Warning warning:

  The current statement is being autocommitted using implicit
  autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
  method of Engine or Connection in order to use an explicit transaction
  for DML and DDL statements.

There's only a single case of this, though we take the opportunity to
remove some unnecessary contextlib.closing calls: the Connection object
already supports being called as a context manager.

Change-Id: Ic8594773acbc3ff52469d6585d348076d13da67e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-07-14 14:16:33 +01:00
Stephen Finucane 384e630e16 tests: Enable SQLAlchemy 2.0 deprecation warnings
Well, sort of. We enable them but immediately filter out the ones we're
actually seeing, the rationale being that we can address these in a
piecemeal fashion without the risk of introducing new issues.

There's a lot more to be done here. However, the work done in oslo.db
[1], nova [2], cinder [2] etc. should provide a guide for how to resolve
the outstanding issues.

[1] https://review.opendev.org/q/topic:sqlalchemy-20+project:openstack/oslo.db
[2] https://review.opendev.org/q/topic:sqlalchemy-20+project:openstack/nova
[3] https://review.opendev.org/q/topic:sqlalchemy-20+project:openstack/cinder

Change-Id: I5b11e63451b0e76969421fa8fe39ac2b920f90ab
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-07-14 14:12:05 +01:00
Stephen Finucane 171580c4d3 tests: Use valid IPv6 address
Python 3.11.4 [1] includes a fix for gh-103848 [2] and
urllib.parse.urlsplit will now validate that bracketed IP addresses are
valid IPv6 address. Fix this.

[1] https://docs.python.org/release/3.11.4/whatsnew/changelog.html#python-3-11-4
[2] https://github.com/python/cpython/issues/103848

Change-Id: Ibd3d24e07f0c5670224b3e186b329c207666a2ab
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-07-14 14:11:10 +01:00
Zuul b18848b394 Merge "Upgrade development status" 2023-05-10 21:30:49 +00:00
Zuul 8fc904f5a0 Merge "Revert "Moves supported python runtimes from version 3.8 to 3.10"" 2023-05-05 13:43:14 +00:00
Ghanshyam 352cdb36f3 Revert "Moves supported python runtimes from version 3.8 to 3.10"
This reverts commit 8535f84148.

Keeping Python 3.10 in setup.cfg classifier. 

Reason for revert:

Needed-By: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/882175

TC has been discussing about re-adding the python 3.8
testing in current master 2023.2 release testing.

- https://meetings.opendev.org/meetings/tc/2023/tc.2023-04-25-18.00.log.html#l-191
- https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033469.html

While governance changes are under review, TC agreed to add py3.8 testing
so that we do not see more project/lib dropping python 3.8 and make them
uninstalable on python 3.8

- https://meetings.opendev.org/meetings/tc/2023/tc.2023-05-02-18.00.log.html#l-17
- https://review.opendev.org/c/openstack/governance/+/882165

Also adding py3.8 testing back in job https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/882175

Change-Id: Ie1d0a1ce0439ca5ffe3d3d377f3ac14a4fa6702f
2023-05-04 20:20:24 +00:00
Zuul 5c1b7e58dd Merge "Fix parsing of zookeeper jobboard backend options" 2023-04-19 00:11:52 +00:00
Zuul 314118bd61 Merge "Prepare taskflow for sqlalchemy2" 2023-04-13 19:28:27 +00:00
Zuul d3dc83e523 Merge "Fix doc building with Sphinx 6.0" 2023-04-13 16:04:44 +00:00
Takashi Kajinami 8535f84148 Moves supported python runtimes from version 3.8 to 3.10
Within 2023.2 python version 3.9 and 3.10 are the
supported python runtimes [1].
[1] https: //review.opendev.org/c/openstack/governance/+/872232

Change-Id: I7f5323109ab03cdf3bd36fabee1bee37e7887679
2023-04-10 13:12:05 +09:00
Takashi Kajinami aa2980f9d3 Upgrade development status
This library has been used by several OpenStack projects and we can
safely assume this is not mature and can be used for production.

Change-Id: I26540b2d887d2a5e9208a08dc3d2b4ce149924ac
2023-04-10 11:00:20 +09:00
Eric Harney a2078f4e09 Fix doc building with Sphinx 6.0
Fix Sphinx 6.0 doc build error.

Change-Id: I9a1571cf416d32d040d8ff05d1c8b1769d1d0558
2023-03-21 11:28:33 -04:00
Gregory Thiemonge 77bc3cb7f8 Prepare taskflow for sqlalchemy2
* convert_unicode engine option is no longer supported
* use positional arguments for select, lists are no longer supported
* insert no longer accepts keyword constructor arguments
* explicitly convert Rows to dicts

Change-Id: I858d60b3d142e8ade8b2c0154872c8bfc19a42ba
2023-02-27 02:18:54 -05:00
OpenStack Release Bot dd73b125d1 Update master for stable/2023.1
Add file to the reno documentation build to show release notes for
stable/2023.1.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.1.

Sem-Ver: feature
Change-Id: I11c1dfdc64a0905df5e22c979579cdf5f55cd517
2023-02-24 15:26:38 +00:00
Zuul 325d45629d Merge "Replace abc.abstractproperty with property and abc.abstractmethod" 2023-02-17 15:08:24 +00:00
Zuul e5e572ab8e Merge "Change StrictRedis usage to Redis" 2023-02-17 13:27:20 +00:00
Gregory Thiemonge 086b3e4335 Fix parsing of zookeeper jobboard backend options
Fix the zookeeper backend options when values are passed as strings,
a "False" string is now treated as the False boolean.

Closes-Bug: #1999174
Change-Id: I048faf06d89ebf980efe0598e647f2ec89f65ada
2023-01-12 08:59:29 +00:00
Gregory Thiemonge 5bcac4c7d4 Fix test_while_is_not with python 3.11
The test_while_is_not function relied on strings and literals to test
the 'while_is_not' function. But the while_is_not function uses the 'is'
operator to test the equivalency of 2 objects.
This triggers a bug with python 3.11 where using 'is' with literals is
not advised (it is not recommended since python 3.8 [0]).
The test now uses objects from a specific class to evaluate the
while_is_not function.

[0] https://docs.python.org/3/whatsnew/3.8.html#changes-in-python-behavior

Change-Id: I38a0135aaf73e25aa20a11c0685d5c2a7b587a07
2022-12-05 18:21:48 +01:00
Zuul 012b7c5aa9 Merge "Adapt to new jsonschema versions" 2022-11-21 10:31:01 +00:00
Tobias Urdin 4ff2d00178 Change StrictRedis usage to Redis
The StrictRedis class is only an alias for
Redis in >= 3.0.0

Change-Id: Ief27531f120a50805053c214cb61bb4151678d70
2022-09-29 08:37:05 +00:00
niuke 299c2afe14 remove unicode prefix from code
Change-Id: I59e8ae94eb83ae3a1ddcc99091e53686d2d55186
2022-08-24 15:17:18 +08:00