With complex parallel joins mistral had no mechanism to choose which
publish (left or right in terms of merge) should it use. It is a
common case when one branch updates existing value, but after merge
we see the old version.
This patch introduce context versioning feature, where every existing
key of mistral context has its version, and this version is used in
the context merge stage.
Change-Id: I604a9a8391150ac4801115b9892f781c33ecfdcb
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
hacking 3.0.x is too old.
Also remove the note about pip's behavior which was already fixed in
recent versions.
Change-Id: I65d350943649c3346ed5741631c01724ddd256ef
This feature introduces an enhanced error-handling
mechanism for workflows, allowing them to gracefully
handle issues within individual tasks without
causing a complete workflow failure. Previously,
when using subworkflow and passing an incomplete set
of parameters, the entire workflow would terminate.
With this feature, the workflow continues execution,
isolating errors at the task level. Consequently,
partial issues in one task no longer impact other
branches of workflow execution.
Implements blueprint partial-workflow-failure-handling
Change-Id: Id6a910c85c1d6953408682a2a724c4826333422f
We have no way to determine whether the workflow definition has
changed or not.
This complicates the work with Mistral for external services.
A 'checksum' field has been added to workflow_definitions_v2
to solve this issue.
Implements: blueprint mistral-add-checksum-field-to-wf-definition
Change-Id: I99d095c6c74c62f134aedfa3a63308a0712db38d
As is described in the doc[1], we should not iterate over sys.modules
directly because sys.modules can be changed during iterations.
[1] https://docs.python.org/3/library/sys.html#sys.modules
Change-Id: Ib6ab36f93b408fdae59c1c08146f2ca069ed4f96
In error cases join task could lose context of some branches
Change-Id: I58a94c4ebc5d860473c9b48df326f6ea29cba9fa
Closes-Bug: #2020370
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
After this patch mistral will run tasks using RPC which will distribute tasks amongst available engine threads. this will enhance performance in case of executing huge executions containing multiple tasks.
Implements: blueprint distribute-mistral-operations
Change-Id: I0b7202589eee68ba5560bf2aa60fbbd6118f3719
After this patch, user can update logging format to include root_execution_id in logs, which will be helpful to find and debug logs related to specific workflow execution.
- Logs about creation and status changes of Mistral entities(execution,
task, action execution, etc) are changed to INFO log level.
- User can update logging_context_format_string to include root_execution_id in logs.
Implements: Implements: blueprint improve-mistral-loggers
Change-Id: I54fe058e5451abba6ea7f69d03d498d78a90993e
This change introduces a new option, [healthcheck] enabled, which
enables the healthcheck middleware in mistral-api pipeline.
This middleware allows status check at /healthcheck path, which is
useful for load balancers or any monitoring services to validate health
of its backend services.
This change is created based on the same change proposed to ironic[1].
[1] 6f439414bdcef9fc02f844f475ec798d48d42558
Co-Authored-By: Jim Rollenhagen <jim@jimrollenhagen.com>
Change-Id: I9bf3de8a5ae6a8c9346285147732b773a3667f7e
after this patch user can choose option whether to replace or merge task data to the execution context.
ex: merge_strategy: replace/merge
Implements: blueprint merge-mistral-tasks-data
Change-Id: I3c96bab9953c4995f2b718ac48dff0f153872026
Opportunity to hide sensitive data from http action logs, such as:
* Request headers
* Request body
* Response body
Change-Id: I6d1b1844898343b8fa30f704761096e3d2936c4d
Implements: blueprint mistral-hide-sensitive-data-from-http-actions-logs
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
There was a logical flaw where we compared types
that were different causing a TypeError.
There was also a flaw in that we default to the
name of the action when sorting if the key does
not exist, to compare the data both value should
come from the same key as well otherwise they
should be treated as not equal since the data
cannot be.
Change-Id: Idcb276912582bb097dc5c1c9692facde63d5886b
After this patch, it will validate execution-id exists in the DB while fetching tasks for the execution-id So, GET /v2/executions/execution_id/tasks API will return 404 for absent execution id.
Closes-Bug: #1998210
Change-Id: I6ac05f4b87c470a68cd67f19d1bd95a4d6cebe59
This patch adds an ability to rerun failed workflow by
skipping failed tasks. Workflow behavior in skip case could
be configured by new fields in task definition:
* on-skip
* publish-on-skip
Change-Id: Ib802a1b54e69c29b4d0361f048c2b9c076a4c176
Implements: blueprint mistral-task-skipping-feature
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
A test is randomly failing due to a comparison between two dates:
created_at and started_at.
Most of the time these two dates are identical, but we can have one
second difference when the creation is done at the very end of a second
and the start is done at the beginning of the next one.
The fix is to make sure that we have less or equal than 1 sec between
started_at and created_at.
Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I081bb659047f7639bed52dfac8b3858b10859404
Some tests which are cancelling workflows are also checking that only one task
was executed.
But the second task is delayed (wait-before: 3) so we can endup in a
situation when the second task appeared in the task_execs.
Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I425b5ca0c1bedc22db1aa464f5f14aae69f9370e
The method is deprecated since Python 3.2[1] and shows the following
DeprecationWarning.
/usr/lib/python3.9/unittest/case.py:1134: DeprecationWarning:
assertDictContainsSubset is deprecated
warnings.warn('assertDictContainsSubset is deprecated',
[1] https://docs.python.org/3/whatsnew/3.2.html#unittest
Closes-Bug: #1938103
Change-Id: I17f56606eb43fba46281e9bdc08ace8727455a40
The tenant argument of RequestContext is no longer available since
oslo.context >= 4.0.0 . This change fixes the compatibility issue
caused by that removal.
Note that this still keeps reference to 'tenant' argument to make
the code compatible with older oslo.context, but that can be removed
once oslo.context >= 4.0.0 becomes available in upper-constraints.
Change-Id: Ie671f50e5ff5a7c746f9e95691eaf4dd19937b52
Since PyYAML 5.1, yaml.load without specifying the Loader option is
deprecated and shows the following warning.
YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated,
as the default Loader is unsafe.
Please read https://msg.pyyaml.org/load for full details.
This change replaces yaml.load by Mistral's own safe_load method which
is widely used to load yaml files.
Change-Id: I6cca093050c2ef696cf17174efc7b46330c10143
Currently unit tests are failing because of the following type mismatch
detected.
testtools.matchers._impl.MismatchError: '{'k2', 'k1'}' is not an
instance of list
This change fixes that error.
Change-Id: I8ecedd341ba45d1919c9073dda8b2fc390306a52
This change fixes the following two patterns of errors detected in
gate.
1)
sqlalchemy.exc.ArgumentError: Textual column expression '...' should be
explicitly declared with text('...'), or use column('...') for more
specificity
2)
testtools.matchers._impl.MismatchError: '('...',)' is not an instance
of tuple
Change-Id: Ie080bdcacf9aa2dd2fe0454bcb814285c08bbfa8
* With the recent update of the cachetools library the internal
behavior of the cache became different and we made some assumptions
in the test about how it should work. We did mocking for this.
Now we have to modify it to reflect the latest behavior.
Change-Id: If76690481b5e0b88c034e35ef17657d59aaa4b4d
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also replace policy.json to policy.yaml ref from doc and tests.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Change-Id: I3b9aeb3379a76f7e40dab0c46e27f4447a0c3d03
* Reworked /code_sources and /dynamic_actions API endpoints to
simplify them. For now they don't work with multiple objects and
they are consistent with other endpoints. If needed, we'll add
support for multiple objects (i.e. adding multiple dynamic actions
with a single request) later in a backwards compatible manner.
* Simplified unit tests.
* Got rid of services/*.py modules since they didn't do anything
useful. They were just wrappers around DB API calls.
Change-Id: Ib5a53f1f1a185f0395ffae1ab0c401633fcdd0fc
* Style improvements to make sure the code is compliant with the
coding guidelines
* Fixing small but important things like the mismatch between the
sinatures of the methods find_all() in the DynamicActionProvider
class and the base ActionProvider interface.
* Improved the tests.
* Simplified the implementation of DynamicActionProvider.
Change-Id: Idbfb15b4c3bb415e7fa9c7ece27eabfe674b6059
* added dynamic actions:
these actions are created and modified in runtime,
each action needs a code source to be imported from and a
class name.
- there are 2 new endpoints:
- /v2/code_sources/:
used to add new code sources to mistral.
- /v2/dynamic_actions/:
used to add dynamic actions to mistral in runtime
- a new Action provider (DynamicActionProvider) was added:
it provides the actions created from the dynamic actions api.
Change-Id: I9fe8c28ffdef71016d9dc13aea60a288c8ebaa0a
Signed-off-by: ali <ali.abdelal@nokia.com>
* Added a test action generator to the test that checks the config
option "load_action_generators".
Change-Id: I3ec465d5d9505a93850494d767668c021ef63847
* This patch refactors Mistral with the action provider concept
that is responsible for delivering actions to the system. So
it takes all the burden of managing action definitions w/o
having to spread that across multiple subsystems like Engine
and API and w/o having to assume that action definitions are
always stored in DB.
* Added LegacyActionProvider that represents the old way of
delivering action definitions to the system. It pretty much just
analyses what entries are configured in the entry point
"mistral.actions" in setup.cfg and build a collection of
corresponding Python action classes in memory accessible by names.
* The module mistral/services/actions.py is now renamed to
adhoc_actions.py because it's effectively responsible only for
ad-hoc actions (those defined in YAML).
* Added the new entry point in setup.cfg "mistral.action.providers"
to register action provider classes
* Added the module mistral/services/actions.py that will be a facade
for action providers. Engine and other subsystems will need to
work with it.
* Other small code changes.
Depends-On: I13033253d5098655a001135c8702d1b1d13e76d4
Depends-On: Ic9108c9293731b3576081c75f2786e1156ba0ccd
Change-Id: I8e826657acb12bbd705668180f7a3305e1e597e2
Remove six.moves Replace the following items with Python 3 style code.
- six.moves.urllib
- six.moves.queue
- six.moves.range
- six.moves.http_client
Subsequent patches will replace other six usages.
Change-Id: I80c713546fcc97391c64e95ef708830632e1ef32
The assertItemsEqual method has been removed in Python 3.3 [1] but
it was kept alive by unittest2, imported by testtools. For better
compatibility, change to assertCountEqual.
[1] https://bugs.python.org/issue17866
Change-Id: Ib025191bdc2762a9f920dd6c83e427d75d9a8d01
* While fetching a list of object in the REST layer we now check
if the object has already been deleted in parallel (by catching
the corresponding exception caused by lazy loading) and don't
include it into the result set.
* Added the corresponding test that uses mocking and two synchronized
threads to reproduce this corner case.
* Minor style changes.
Change-Id: Ia92d799a421e07542f270223c1add2aae7b32aab
Closes-Bug: #1887357
* There were references to Mistral specific constants mistakenly
placed into mistral-lib. This patch adds these constants into
the module where task language specifications are declared and
changes the corresponding references.
Change-Id: I8c7c6896f01894ac76cf9365abfdce17e7df7662
* All calls to a notifier within the Task class have now been
moved into the method set_state() so that the relation between
a state change and a notification is now straightforward and the
notification calls don't have to be spread out across different
modules.
Change-Id: I9c0647235e1439049d3e7db13f19bef542f10508
Zuul