Commit Graph

2891 Commits

Author SHA1 Message Date
Zuul 4446f9ced8 Merge "Python 3.12: cast random delays as int" 2024-03-20 09:29:16 +00:00
Thomas Goirand 2b2c8759d4 Python 3.12: cast random delays as int
This patch fixes 128 unit test failures under Python 3.12 when
building the Debian package under Unstable. Without this patch,
I'm getting:

Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/<<PKGBUILDDIR>>/mistral/services/legacy_scheduler.py", line 152, in _loop
    random.Random().randint(0, self._random_delay * 1000) * 0.001
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/random.py", line 336, in randint
    return self.randrange(a, b+1)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/random.py", line 312, in randrange
    istop = _index(stop)
            ^^^^^^^^^^^^
TypeError: 'float' object cannot be interpreted as an integer

Change-Id: Ia85ea13923216d090ef1e4deee7840e70f9185b4
2024-03-19 09:04:43 +01:00
Zuul 6e190e65ea Merge "Context versioning feature" 2024-03-14 13:03:26 +00:00
Oleg Ovcharuk f2cbe1c59d Context versioning feature
With complex parallel joins mistral had no mechanism to choose which
publish (left or right in terms of merge) should it use. It is a
common case when one branch updates existing value, but after merge
we see the old version.
This patch introduce context versioning feature, where every existing
key of mistral context has its version, and this version is used in
the context merge stage.

Change-Id: I604a9a8391150ac4801115b9892f781c33ecfdcb
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2024-03-14 09:25:17 +00:00
Takashi Kajinami 44cd95684b Bump hacking
hacking 3.0.x is too old.

Also remove the note about pip's behavior which was already fixed in
recent versions.

Change-Id: I65d350943649c3346ed5741631c01724ddd256ef
2024-02-19 02:23:53 +09:00
Zuul 494d210f58 Merge "Deprecate unused [coordination] heartbeat_interval" 2023-12-19 17:58:29 +00:00
Zuul 973e6ac996 Merge "Add optional healthcheck middleware" 2023-12-14 00:35:55 +00:00
Zuul 6a417dce53 Merge "Add checksum field to wf definition" 2023-12-13 23:31:55 +00:00
Zuul e38cfcbbdf Merge "Partial Workflow Failure Handling" 2023-12-13 23:23:35 +00:00
Takashi Kajinami f34e250fb5 Deprecate unused [coordination] heartbeat_interval
This option has never been used by actual logic. This deprecates
the ineffective option so that we can remove it in a future release.

Change-Id: I350e45fc9aef28db8790614ade7a5ad3071e574b
2023-12-13 22:47:37 +00:00
Vadim Zelenevsky 7cc007415b Partial Workflow Failure Handling
This feature introduces an enhanced error-handling
mechanism for workflows, allowing them to gracefully
handle issues within individual tasks without
causing a complete workflow failure. Previously,
when using subworkflow and passing an incomplete set
of parameters, the entire workflow would terminate.
With this feature, the workflow continues execution,
isolating errors at the task level. Consequently,
partial issues in one task no longer impact other
branches of workflow execution.

Implements blueprint partial-workflow-failure-handling

Change-Id: Id6a910c85c1d6953408682a2a724c4826333422f
2023-11-29 07:55:38 +00:00
Vadim Zelenevskii ac8b2d0488 Add checksum field to wf definition
We have no way to determine whether the workflow definition has
changed or not.
This complicates the work with Mistral for external services.
A 'checksum' field has been added to workflow_definitions_v2
to solve this issue.

Implements: blueprint mistral-add-checksum-field-to-wf-definition

Change-Id: I99d095c6c74c62f134aedfa3a63308a0712db38d
2023-11-29 07:55:28 +00:00
Takashi Kajinami 6410ca6518 Avoid RuntimeError caused by iteration over sys.modules
As is described in the doc[1], we should not iterate over sys.modules
directly because sys.modules can be changed during iterations.

[1] https://docs.python.org/3/library/sys.html#sys.modules

Change-Id: Ib6ab36f93b408fdae59c1c08146f2ca069ed4f96
2023-11-28 01:12:35 +00:00
Zuul 7baff017d0 Merge "Fix test not waiting for action to be created" 2023-07-05 15:39:49 +00:00
Oleg Ovcharuk 5cb9cb74e5 Fix test not waiting for action to be created
Change-Id: I415bb3fbefa91da0e7aaca9fe5d3714989ffda69
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2023-06-28 02:50:10 +03:00
Zuul bbe6b2ff66 Merge "Fix join task not refreshing inbound context" 2023-06-25 20:59:41 +00:00
Vasudeo Nimbekar 51574ef6ef Added maintenance mode
Change-Id: Ie42fcfa96044da1793f1afbff5fb2ae5534d5c2e
2023-05-31 17:31:42 +05:30
Oleg Ovcharuk 6d3018ea01 Fix join task not refreshing inbound context
In error cases join task could lose context of some branches

Change-Id: I58a94c4ebc5d860473c9b48df326f6ea29cba9fa
Closes-Bug: #2020370
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2023-05-24 18:16:59 +03:00
Zuul d6b7f7305d Merge "Starting tasks via RPC" 2023-05-17 09:20:40 +00:00
Zuul 431ece5581 Merge "Info endpoint" 2023-05-17 09:05:03 +00:00
Takashi Kajinami 1f9f1f3ffb [coordination] backend_url should be secret
The backend_url option can sometimes contain secrets.

For example when redis coordination backend is used and authentication
is enabled in redis, the plain redis password is put as an URL element.

[coordination]
backend_url=redis://:password@127.0.0.1:6379

Change-Id: Ie073d9ac8fd8580f7442370291814d99aad92066
2023-03-20 18:39:45 +09:00
Vasudeo Nimbekar 9f52e2b6f3 Starting tasks via RPC
After this patch mistral will run tasks using RPC which will distribute tasks amongst available engine threads. this will enhance performance in case of executing huge executions containing multiple tasks.

Implements: blueprint distribute-mistral-operations

Change-Id: I0b7202589eee68ba5560bf2aa60fbbd6118f3719
2023-02-16 13:19:42 +05:30
Zuul 4289317a91 Merge "Task skipping feature" 2023-02-14 13:22:17 +00:00
Zuul 5e581ee040 Merge "Adding root_execution_id parameter to mistral loggers" 2023-02-14 10:44:01 +00:00
Vasudeo Nimbekar 88e7e7ceee Adding root_execution_id parameter to mistral loggers
After this patch, user can update logging format to include root_execution_id in logs, which will be helpful to find and debug logs related to specific workflow execution.

  - Logs about creation and status changes of Mistral entities(execution,
    task, action execution, etc) are changed to INFO log level.
  - User can update logging_context_format_string to include root_execution_id in logs.

Implements: Implements: blueprint improve-mistral-loggers

Change-Id: I54fe058e5451abba6ea7f69d03d498d78a90993e
2023-02-13 05:01:39 +00:00
Zuul c9bd45d073 Merge "Add "fields" filter support on single objects" 2023-02-09 13:06:08 +00:00
Takashi Kajinami f17dd7a04b Add optional healthcheck middleware
This change introduces a new option, [healthcheck] enabled, which
enables the healthcheck middleware in mistral-api pipeline.

This middleware allows status check at /healthcheck path, which is
useful for load balancers or any monitoring services to validate health
of its backend services.

This change is created based on the same change proposed to ironic[1].

[1] 6f439414bdcef9fc02f844f475ec798d48d42558

Co-Authored-By: Jim Rollenhagen <jim@jimrollenhagen.com>
Change-Id: I9bf3de8a5ae6a8c9346285147732b773a3667f7e
2023-02-03 04:09:13 +00:00
Vasudeo Nimbekar 7c35734300 Merge mistral tasks data to execution context
after this patch user can choose option whether to replace or merge task data to the execution context.
ex: merge_strategy: replace/merge

Implements: blueprint merge-mistral-tasks-data
Change-Id: I3c96bab9953c4995f2b718ac48dff0f153872026
2023-01-31 17:27:17 +05:30
Zuul bfcc417044 Merge "Add an ability to hide sensitive data from http action logs" 2023-01-25 13:49:52 +00:00
Oleg Ovcharuk 3919e6a52b Add an ability to hide sensitive data from http action logs
Opportunity to hide sensitive data from http action logs, such as:
* Request headers
* Request body
* Response body

Change-Id: I6d1b1844898343b8fa30f704761096e3d2936c4d
Implements: blueprint mistral-hide-sensitive-data-from-http-actions-logs
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2023-01-24 21:54:00 +00:00
Tobias Urdin 920d6b0336 Use new get_rpc_client API from oslo.messaging
Use the new API that is consistent with
the existing API instead of instantiating the client
class directly.

This was introduced in release 14.1.0 here [1] and
added into oslo.messaging here [2]

[1] https://review.opendev.org/c/openstack/requirements/+/869340
[2] https://review.opendev.org/c/openstack/oslo.messaging/+/862419

Change-Id: I13c3cfa3e2207144b23230766f9cad97668d57e8
2023-01-19 20:41:03 +00:00
Arnaud Morin 70af40becc Remove tenant when building mistral context
When building mistral context (using oslo.context) from environment, the
tenant extra key was given, which is not accepted by oslo.context.

This issue was detected when debugging a mistral-client test:
mistralclient.tests.functional.cli.v2.test_cli_v2.NegativeCLITests.test_target_action_execution

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I72ce8851ea5b379a8af75b32fb600691638836af
2023-01-18 21:20:28 +01:00
Tobias Urdin b6a47992e7 Fix sorting error when type is different
There was a logical flaw where we compared types
that were different causing a TypeError.

There was also a flaw in that we default to the
name of the action when sorting if the key does
not exist, to compare the data both value should
come from the same key as well otherwise they
should be treated as not equal since the data
cannot be.

Change-Id: Idcb276912582bb097dc5c1c9692facde63d5886b
2023-01-18 21:20:28 +01:00
Xavier Hardy 9c5324ce9a Add "fields" filter support on single objects
Change-Id: Iba0401445921f9ea7932628c05590472307ba83a
Closes-Bug: #1718214
Signed-off-by: Xavier Hardy <xavier.hardy@corp.ovh.com>
Co-Authored-By: Oleg Ovcharuk <vgvoleg@gmail.com>
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2022-12-21 06:40:50 +00:00
Vasudeo Nimbekar 87c08ff421 Update get execution's tasks functionality
After this patch, it will validate execution-id exists in the DB while fetching tasks for the execution-id So, GET /v2/executions/execution_id/tasks API will return 404 for absent execution id.

Closes-Bug: #1998210
Change-Id: I6ac05f4b87c470a68cd67f19d1bd95a4d6cebe59
2022-12-01 07:39:48 +00:00
Oleg Ovcharuk e72a4e9a70 Task skipping feature
This patch adds an ability to rerun failed workflow by
skipping failed tasks. Workflow behavior in skip case could
be configured by new fields in task definition:
* on-skip
* publish-on-skip

Change-Id: Ib802a1b54e69c29b4d0361f048c2b9c076a4c176
Implements: blueprint mistral-task-skipping-feature
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2022-12-01 01:47:30 +03:00
Oleg Ovcharuk 517789943a Info endpoint
For many various support reasons, Mistral should have a special
endpoint to store all necessary info data. This endpoint will read
json from created by admin info file. To configure this you should
use mistral configuration:

[api]
enable_info_endpoint = True
info_json_file_path = info.json

Change-Id: I6f344dc15a4ca5c69a6b21841544a31f95eb393f
Implements: blueprint mistral-info-endpoint
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2022-11-17 22:06:37 +00:00
Zuul 55745a750c Merge "Remove extra description on cron trigger executions" 2022-10-04 10:49:17 +00:00
Arnaud Morin 5e069e1c98 Remove extra description on cron trigger executions
When creating a periodic workflow execution from a cron trigger, the
description field was filled with info like cron trigger ID, name, etc.

The cron trigger name can be set by user and is then injected in the
description field.
Beside the fact that this is not useful, it may also be dangerous.

The purpose of this patch is to remove the name from the description so
that the lenght of description field is not user dependant anymore.

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I326400a4a04fba71ccc90545909212a41d6b3b13
2022-09-29 10:42:03 +02:00
Zuul 95e30947d7 Merge "Do not guess content from extension on API" 2022-09-09 13:10:05 +00:00
Zuul 3554581cbe Merge "Fix race condition when comparing dates" 2022-09-09 12:49:25 +00:00
Zuul 540664346f Merge "Fix race condition when cancelling workflows" 2022-09-09 11:29:14 +00:00
Zuul 5782b3be2c Merge "Update type of 'description' field to 'text' for table workflow_executions_v2" 2022-09-09 10:32:18 +00:00
Arnaud Morin 2a312cc814 Do not guess content from extension on API
When getting a cron trigger (but also any other object) by name, pecan
core will try to guess the content_type from the extension.
While for most name, the guess will fail, it may succeed, e.g. when a
name is ending with '.pl'.
Pecan guess it as a perl file and then reject the call (with a 404
error).

The fix is to tell pecan to not guess any content_type from extension by
setting guess_content_type_from_ext to False on initialisation.

Closes-Bug: 1987418

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I1bc20b953787cae9b42d12bb5eb3c7a94ed4bbdd
2022-09-09 10:40:22 +02:00
Arnaud Morin b350f58f3e Fix race condition when comparing dates
A test is randomly failing due to a comparison between two dates:
created_at and started_at.
Most of the time these two dates are identical, but we can have one
second difference when the creation is done at the very end of a second
and the start is done at the beginning of the next one.

The fix is to make sure that we have less or equal than 1 sec between
started_at and created_at.

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I081bb659047f7639bed52dfac8b3858b10859404
2022-09-09 10:39:20 +02:00
Arnaud Morin 60c9dbf078 Fix race condition when cancelling workflows
Some tests which are cancelling workflows are also checking that only one task
was executed.
But the second task is delayed (wait-before: 3) so we can endup in a
situation when the second task appeared in the task_execs.

Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I425b5ca0c1bedc22db1aa464f5f14aae69f9370e
2022-08-30 10:44:49 +02:00
Zuul 5881bcc466 Merge "Added sorting and marker for pagination of actions" 2022-08-24 07:18:54 +00:00
Rajat Jain 6fb78f3561 Update type of 'description' field to 'text' for table workflow_executions_v2
Change-Id: I5ae7f0f39822c99d11bd7cd6191e036afd7e6721
2022-08-23 11:44:42 +00:00
Zuul c7d5bc8b08 Merge "Replace deprecated assertDictContainsSubset" 2022-08-16 16:36:10 +00:00
Tobias Urdin 1a6f5d0e5f Added sorting and marker for pagination of actions
This solves the sorting and marker for pagination
when listing actions.

Change-Id: I9bb15642585af22ad354dd4f8f2253a47129639e
2022-08-15 08:07:13 +00:00