This patch adds an ability to rerun failed workflow by
skipping failed tasks. Workflow behavior in skip case could
be configured by new fields in task definition:
* on-skip
* publish-on-skip
Change-Id: Ib802a1b54e69c29b4d0361f048c2b9c076a4c176
Implements: blueprint mistral-task-skipping-feature
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
* There were references to Mistral specific constants mistakenly
placed into mistral-lib. This patch adds these constants into
the module where task language specifications are declared and
changes the corresponding references.
Change-Id: I8c7c6896f01894ac76cf9365abfdce17e7df7662
* Adding "convert_output_data" config property gives an opportunity
to increase overal performance. If YAQL always converts an expression
result, it often takes significant CPU time and overall workflow
execution time increases. It is especially important when a workflow
publishes lots of data into the context and uses big workflow
environments. It's been tested on a very big workflow (~7k tasks)
with a big workflow environment (~2.5mb) that often uses the YAQL
function "<% env() %>". This function basically just returns the
workflow environment.
* Created all necessary unit tests.
* Other style fixes.
Change-Id: Ie3169ec884ec9a0e7e50327dd03cd78dcda0a39b
* Initialization of profiler was also missing for a thread
spawned within post_tx_queue.py so we were loosing important
profiling info
* Changed the profiler test since its logic was already obsolete.
Now we initialize profiler in every thread so the only reason to
not get any profiler traces when a workflow completed is
"enabled = False" in the "profiler" group in the configuration.
* Added more profiler traces
* Small readability changes in the workflow language spec
Change-Id: I35e6711f8e10bb08d7e842f4bca8753b929328fd
Workflow and task executions will inherit tags from
definition. Executions filtering by tag is included.
Change-Id: Id5d615b829901258af2be7ca99178ad92b60d1fb
Closes-Bug: #1853457
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
bug-fix, on-success, on complete published variables are not merged
with regular published variables
Closes-Bug: #1791449
Change-Id: Iba05b9acbfea06214a3ab93911d2eaaa4729278f
* For the sake of the service performance, it may make sense to
disable validation of the workflow language syntax if it is
affordable for a particular use case. For example, if all
workflows are auto-generated by a 3rd party system and tested
thoroughly (either by running them with Mistral or at least
validating them via the special Mistral endpoint) then we can
safely disable validation of the language syntax when uploading
workflow definitions. For production systems it makes a big
difference if workflow texts are large (thousands of tasks).
This patch adds the boolean parameter "skip_validation" for API
requests like "POST /v2/workflows" to disable validation, if
needed, and the new configuration property "validation_mode"
to set a desired validation mode.
The option is an enumeration and has the following valid values:
1) "enabled" - enabled for all API requests unless it's
explicitly disabled in the request itself
2) "mandatory" - enabled for all API requests regardless
of the flag in the request
3) "disabled" - disabled for all API requrests regardless
of the flag in the request
"mandatory" is choosen as the default value for this new
property to keep compatibility with the previous versions.
* Minor style changes.
Closes-Bug: #1844242
Change-Id: Ib509653d38254954f8449be3031457e5f636ccf2
* Various simple optimizations for creation of specs. Mostly
additional caching of non-growing data related to the
structure of the language spec classes. For huge workflows
with thousands of tasks these changes reduce execution time
by dozens of seconds.
* Minor style changes.
Partial-bug: #1844242
Change-Id: Ia700e25752d9d35ece18609f2977e6568062e4bf
Fail-on policy allows to fail success tasks by condition. It is useful
in cases we have to fail task if its result is unacceptable and it makes
workflow definition more readable.
Change-Id: I57b4f3d1533982d3b9b7063925f8d70f044aefea
Implements: blueprint fail-on-policy
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
Context evaluation is not required for non-conditional transitions,
because the list of next tasks is known in advance.
For this reason we can skip this operation, that can be quite
time-consuming, and generate a list of task directly.
Also a little cleanup was done to remove unnecessary methods.
Change-Id: Ia419c47a7d71db46a5cae557fe8bc7512390715e
Current joining mechanism in some cases can be expensive because
it uses a multi-step recursive search, which leads to a huge amount
of db requests.
This work changes this behavior by precaching required task
executions to prevent hammering the database during the lookup.
Change-Id: I2d1b7e72c728a14c85b015dfdb0f8800b95f3749
Using default mutable parameter is bad.
Default parameters are evaluated only once
if you mutate it you will get unexpected results.
Since we don't mutate here the default paramter, make
sure it is unmutable.
Change-Id: Ib5c451a8c8cad7b6c9a009369c1c039563023368
The OnClauseSpec required Task names to be \w+ or [a-zA-Z0-9_]
which is not enforced by the DSL, so it was possible to have
valid task names that could not be referenced in an on-clause.
YAML enforces some restrictions on characters in task names (#, !, |)
but other than that any JSON-schema valid string should be a valid Task
name
Change-Id: I3f1056cad7c67e160a082c2a0de2e3bfd476bc63
Closes-Bug: 1797439
Currently when we get a specification using the instantiate_spec function,
we always validate their schema and semantics over and over again.
To prevent it we add new validate parameter to a Spec class.
The validate parameter must be True when we create a workflow, workbook
or action using a mistral-api. In all other cases, it must be False.
Change-Id: Ia450ea9635bc75c204fe031cfeeab154f1d03862
Closes-Bug: #1738769
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
A change for disabling some task names has introduced a massively
backward incompatible behavior. E.g. even though there is a 'noop' engine
command, the usual way of handling noop is in many cases is to create a
task called the same. The other commands are not used that often but
noop is massively present in currently deployed workflows and it is
not possible to mitigate the error if the workflows are coming from
3rd parties.
This change re-enables the usage of the engine commands as task names
and adds documentation on why this is a useful feature.
Change-Id: If90ee5f787e4587a25c156d12c7750407081bf0d
Related-Change: https://review.openstack.org/#/c/535297
This patch places restrictions on the content of the on-success,
on-complete, on-error, retry, concurrency, timeout, wait-before,
wait-after and pause-before statements.
test_direct_transition was refactored to exclude repeated test cases
for on clause keys.
Co-Authored-By: Vitalii Solodilov <mcdkr@yandex.ru>
Closes-Bug: #1714341
Change-Id: I8b314c8759a46111a81cf4a9400aa1cab2ea5201
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
* In case if the same workflow definition is used multiple times
for many executions (e.g. on-success clause has many tasks where
each one of them calls the same subworkflow) then the corresponding
cache entries for those executions should refer to the same
workflow specification. Mistral shouldn't create a new spec and
reuse the existing one that's already cached. Note this behavior
is expected only when there aren't any workflow definition updates
while creating executions and the engine hasn't restarted.
* Completing a docstring for one of the methods in parser.py
Change-Id: Ic420aa0a90337378206332604facdcb898bf9619
* Fixed the implementation of data_flow.evaluate_task_outbound_context
method so that it doesn't use copy.deepcopy() because it may be
very expensive (both CPU and RAM) on big dictionaries.
Change-Id: I6cc77c6ded1e8f00ff164156be9312e1ccb86efb
* verified the length of task name while creating a workflow
* fixed bug when join task has a long name
* added the length limitation to the spec docs
Change-Id: I233e6b0f30d42b939757e9c1caf4965ecc375aee
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
All of these operations (fetch from dict and modify dict) are
already atomic in python implementations with GIL. These locks are
unnecessary.
Change-Id: I70a8efab61f4570ee00e04021949517fd4e6a20d
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
If an invalid tag section was specified, Mistral would throw a 500,
with no useful error message. This makes the tags get checked and
throws a proper error if so.
Change-Id: If2ae69ef76a857bcafe1e4b3d72956c6deeeea09
Closes-Bug: #1747950
Evaluate action names dynamically, so yaql or jinja expression can be
used
Change-Id: I48761c215f0255976c330ffa34f27bb695c944a9
Implements: blueprint mistral-dynamic-actions
Evaluate workflow names dynamically, so yaql or jinja expression
is allowed as sub-workflow name. Tasks names are not yet
dynamically evaluated.
Partially implements: blueprint mistral-dynamic-actions
Change-Id: Icfe591e27a4f45c2e3dcfa83512217f3b2122189
* Before this patch some language specification schemas were
validated twice (or even more) because some parent specs had
references to specific schemas of child specs in their schemas.
And due to our recursive parsing algorithm the same schemas were
checked many times. It reduces performance and complicates
the entire lang specification framework because there's too many
relationships between different specs. The better approach is to
reduce a number of relationships. One way is to not check schemas
child specs when checking parent spec schema. This patch removes
such references replacing them just common schemas like
"non-empty" or "any" which don't lead to validating sub-schemas
when validating parent schemas.
* Minor style changes
Change-Id: I6b695c1870bf8b70112332d4052115543382cdc7
* 'on-success', 'on-error' and 'on-complete' can now look like
described in "Advanced Publishing" specification [1]
* Refactored all places related to the spec changes
* Added unit tests for advanced schema of 'on-xxx' clauses
[1] https://github.com/openstack/mistral-specs/blob/master/specs/pike/approved/advanced_publishing.rst
Change-Id: I190fcec0a40ca6f97d712168f4be7a418bd1f0e8
Partially implements: blueprint mistral-advanced-publishing-global-vars
* The previous versions of these methods were too specific for the
general utils module. Now they are more generic thereby less
confusing.
Change-Id: Ifa8bbf0cc8a63bf1de1142dc8c52b1f8ad3958c5
* The previous name of the package is not relevant anymore, it was
chosen for reasons that don't make sense anymore. A name of the
package should reflect that it's about the workflow language.
Change-Id: I7e05ba97abf0aa5db4e19d9d72b4ea0d52e328fb