Commit Graph

38 Commits

Author SHA1 Message Date
Oleg Ovcharuk e72a4e9a70 Task skipping feature
This patch adds an ability to rerun failed workflow by
skipping failed tasks. Workflow behavior in skip case could
be configured by new fields in task definition:
* on-skip
* publish-on-skip

Change-Id: Ib802a1b54e69c29b4d0361f048c2b9c076a4c176
Implements: blueprint mistral-task-skipping-feature
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2022-12-01 01:47:30 +03:00
Q.hongtao 4bc6162515 Remove six library
Remove six-library Replace the following items with Python 3 style code.
- six.interger_types
- six.itervalues
- six.text_type
- six.string_types
- six.StringIO
- six.next
- six.b
- six.PY3

Change-Id: I299c90d5cbeb41be0132691265b8dcbeae65520e
2020-09-23 10:27:12 +08:00
Renat Akhmerov 2ad7bb829b Moving task type constants into Mistral task specifications module
* There were references to Mistral specific constants mistakenly
  placed into mistral-lib. This patch adds these constants into
  the module where task language specifications are declared and
  changes the corresponding references.

Change-Id: I8c7c6896f01894ac76cf9365abfdce17e7df7662
2020-06-08 13:40:45 +07:00
Renat Akhmerov 8d75784356 Add "convert_output_data" config property for YAQL
* Adding "convert_output_data" config property gives an opportunity
  to increase overal performance. If YAQL always converts an expression
  result, it often takes significant CPU time and overall workflow
  execution time increases. It is especially important when a workflow
  publishes lots of data into the context and uses big workflow
  environments. It's been tested on a very big workflow (~7k tasks)
  with a big workflow environment (~2.5mb) that often uses the YAQL
  function "<% env() %>". This function basically just returns the
  workflow environment.
* Created all necessary unit tests.
* Other style fixes.

Change-Id: Ie3169ec884ec9a0e7e50327dd03cd78dcda0a39b
2020-02-13 17:31:41 +07:00
Renat Akhmerov 829e822581 Init profiler in for a new thread in post_tx_queue.py
* Initialization of profiler was also missing for a thread
  spawned within post_tx_queue.py so we were loosing important
  profiling info
* Changed the profiler test since its logic was already obsolete.
  Now we initialize profiler in every thread so the only reason to
  not get any profiler traces when a workflow completed is
  "enabled = False" in the "profiler" group in the configuration.
* Added more profiler traces
* Small readability changes in the workflow language spec

Change-Id: I35e6711f8e10bb08d7e842f4bca8753b929328fd
2020-02-07 13:42:55 +07:00
Eyal eac23d9e77 Disable the use of anchors when parsing yaml
This can be used as a DDoS attack

Closes-Bug: 1785657
Change-Id: Icf460fea113e9279715cae87df3ef88a77575e04
2019-12-17 12:08:10 +02:00
Oleg Ovcharuk e596ee2e63 Enlarge tags support
Workflow and task executions will inherit tags from
definition. Executions filtering by tag is included.

Change-Id: Id5d615b829901258af2be7ca99178ad92b60d1fb
Closes-Bug: #1853457
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2019-11-29 18:40:37 +03:00
ali c7a54d22df Task publish is ignored silently
bug-fix, on-success, on complete published variables are not merged
 with regular published variables

Closes-Bug: #1791449
Change-Id: Iba05b9acbfea06214a3ab93911d2eaaa4729278f
2019-11-11 09:46:17 +00:00
Renat Akhmerov ac41f94d11 Add an ability to disable workflow text validation
* For the sake of the service performance, it may make sense to
  disable validation of the workflow language syntax if it is
  affordable for a particular use case. For example, if all
  workflows are auto-generated by a 3rd party system and tested
  thoroughly (either by running them with Mistral or at least
  validating them via the special Mistral endpoint) then we can
  safely disable validation of the language syntax when uploading
  workflow definitions. For production systems it makes a big
  difference if workflow texts are large (thousands of tasks).
  This patch adds the boolean parameter "skip_validation" for API
  requests like "POST /v2/workflows" to disable validation, if
  needed, and the new configuration property "validation_mode"
  to set a desired validation mode.
  The option is an enumeration and has the following valid values:
    1) "enabled" - enabled for all API requests unless it's
       explicitly disabled in the request itself
    2) "mandatory" - enabled for all API requests regardless
       of the flag in the request
    3) "disabled" - disabled for all API requrests regardless
       of the flag in the request
  "mandatory" is choosen as the default value for this new
  property to keep compatibility with the previous versions.
* Minor style changes.

Closes-Bug: #1844242

Change-Id: Ib509653d38254954f8449be3031457e5f636ccf2
2019-09-27 04:33:04 +00:00
Renat Akhmerov 64c4038b2c Optimize creation of language specs
* Various simple optimizations for creation of specs. Mostly
  additional caching of non-growing data related to the
  structure of the language spec classes. For huge workflows
  with thousands of tasks these changes reduce execution time
  by dozens of seconds.
* Minor style changes.

Partial-bug: #1844242
Change-Id: Ia700e25752d9d35ece18609f2977e6568062e4bf
2019-09-19 05:03:13 +00:00
ali 7e7f1cb92b moved generic util functions from mistral to mistral-lib
Depends-On: I780c270e4b1a184d7d4dcc580d23697ba75edab1
Closes-bug: #1815183
Change-Id: I5a1d402baa3f69c37f9347c8b3d02a83b8f60423
2019-09-13 04:06:27 +00:00
Oleg Ovcharuk 4e926a1f13 Fail-on policy
Fail-on policy allows to fail success tasks by condition. It is useful
in cases we have to fail task if its result is unacceptable and it makes
workflow definition more readable.

Change-Id: I57b4f3d1533982d3b9b7063925f8d70f044aefea
Implements: blueprint fail-on-policy
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
2019-08-11 07:21:57 +00:00
Eyal 9b576aec7c Use raw strings in all pattern matching strings
This removes some of the deprecation warnings on
unsupported escape sequence for python3.6 and up

Change-Id: I2a6b99aeb3db3b45e0a4c76f1ea4af3b798de73d
2019-08-06 17:57:13 +03:00
Mike Fedosin f09c8ebec1 Skip context evaluation for non-conditional transitions
Context evaluation is not required for non-conditional transitions,
because the list of next tasks is known in advance.
For this reason we can skip this operation, that can be quite
time-consuming, and generate a list of task directly.

Also a little cleanup was done to remove unnecessary methods.
Change-Id: Ia419c47a7d71db46a5cae557fe8bc7512390715e
2019-06-05 18:31:56 +02:00
Mike Fedosin ff00c9c778 Rework joining mechanism
Current joining mechanism in some cases can be expensive because
it uses a multi-step recursive search, which leads to a huge amount
of db requests.

This work changes this behavior by precaching required task
executions to prevent hammering the database during the lookup.

Change-Id: I2d1b7e72c728a14c85b015dfdb0f8800b95f3749
2019-05-17 12:24:12 +02:00
Eyal 97b493ade6 Don't use default mutable parameter
Using default mutable parameter is bad.
Default parameters are evaluated only once
if you mutate it you will get unexpected results.
Since we don't mutate here the default paramter, make
sure it is unmutable.

Change-Id: Ib5c451a8c8cad7b6c9a009369c1c039563023368
2019-04-15 10:30:34 +03:00
Bob Haddleton e98614cd60 Update OnClauseSPec task name criteria
The OnClauseSpec required Task names to be \w+ or [a-zA-Z0-9_]
which is not enforced by the DSL, so it was possible to have
valid task names that could not be referenced in an on-clause.

YAML enforces some restrictions on characters in task names (#, !, |)
but other than that any JSON-schema valid string should be a valid Task
name

Change-Id: I3f1056cad7c67e160a082c2a0de2e3bfd476bc63
Closes-Bug: 1797439
2018-10-12 11:58:22 -05:00
Zuul fc11871b86 Merge "Remove extra a specification validation" 2018-07-31 08:27:18 +00:00
Zuul c54c35c82d Merge "Allow engine commands as task name" 2018-07-30 15:39:01 +00:00
Vitalii Solodilov 4bf03d8e7d Remove extra a specification validation
Currently when we get a specification using the instantiate_spec function,
we always validate their schema and semantics over and over again.
To prevent it we add new validate parameter to a Spec class.
The validate parameter must be True when we create a workflow, workbook
or action using a mistral-api. In all other cases, it must be False.

Change-Id: Ia450ea9635bc75c204fe031cfeeab154f1d03862
Closes-Bug: #1738769
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
2018-07-30 11:55:35 +04:00
Andras Kovi c08e44f17b Allow engine commands as task name
A change for disabling some task names has introduced a massively
backward incompatible behavior. E.g. even though there is a 'noop' engine
command, the usual way of handling noop is in many cases is to create a
task called the same. The other commands are not used that often but
noop is massively present in currently deployed workflows and it is
not possible to mitigate the error if the workflows are coming from
3rd parties.

This change re-enables the usage of the engine commands as task names
and adds documentation on why this is a useful feature.

Change-Id: If90ee5f787e4587a25c156d12c7750407081bf0d
Related-Change: https://review.openstack.org/#/c/535297
2018-07-19 14:23:18 +00:00
Vitalii Solodilov 160948e3c6 Use on-clause and retry_policy get_spec for validation
This patch places restrictions on the content of the on-success,
on-complete, on-error, retry, concurrency, timeout, wait-before,
wait-after and pause-before statements.
test_direct_transition was refactored to exclude repeated test cases
for on clause keys.

Co-Authored-By: Vitalii Solodilov <mcdkr@yandex.ru>
Closes-Bug: #1714341
Change-Id: I8b314c8759a46111a81cf4a9400aa1cab2ea5201
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
2018-06-24 05:34:19 +00:00
Zuul bfb6548027 Merge "Make sure there are no duplicates in the spec cache w/o restarts" 2018-04-26 08:45:14 +00:00
Renat Akhmerov 56adb2ed65 Make sure there are no duplicates in the spec cache w/o restarts
* In case if the same workflow definition is used multiple times
  for many executions (e.g. on-success clause has many tasks where
  each one of them calls the same subworkflow) then the corresponding
  cache entries for those executions should refer to the same
  workflow specification. Mistral shouldn't create a new spec and
  reuse the existing one that's already cached. Note this behavior
  is expected only when there aren't any workflow definition updates
  while creating executions and the engine hasn't restarted.
* Completing a docstring for one of the methods in parser.py

Change-Id: Ic420aa0a90337378206332604facdcb898bf9619
2018-04-26 11:30:59 +07:00
Vitalii Solodilov a29c7d9f7f Added 'safe-rerun' policy to task-defaults section
Change-Id: Ib6641bda008a813660af5584c9319fcaddcd416f
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
2018-04-23 20:46:43 +04:00
Renat Akhmerov 4936d778e5 Optimizing big 'on-XXX' clauses
* Fixed the implementation of data_flow.evaluate_task_outbound_context
  method so that it doesn't use copy.deepcopy() because it may be
  very expensive (both CPU and RAM) on big dictionaries.

Change-Id: I6cc77c6ded1e8f00ff164156be9312e1ccb86efb
2018-04-17 14:30:41 +07:00
Zuul 3426fb4080 Merge "Fixed the length of a task name" 2018-03-13 11:50:26 +00:00
Vitalii Solodilov 4a077b6f98 Fixed the length of a task name
* verified the length of task name while creating a workflow
* fixed bug when join task has a long name
* added the length limitation to the spec docs

Change-Id: I233e6b0f30d42b939757e9c1caf4965ecc375aee
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
2018-03-09 11:02:48 +04:00
Vitalii Solodilov 6c7d6414a2 Remove unnecessary locks
All of these operations (fetch from dict and modify dict) are
already atomic in python implementations with GIL. These locks are
unnecessary.

Change-Id: I70a8efab61f4570ee00e04021949517fd4e6a20d
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
2018-03-08 14:05:44 +04:00
Brad P. Crochet 7c807d3d82 Tags in workflows were not being properly checked
If an invalid tag section was specified, Mistral would throw a 500,
with no useful error message. This makes the tags get checked and
throws a proper error if so.

Change-Id: If2ae69ef76a857bcafe1e4b3d72956c6deeeea09
Closes-Bug: #1747950
2018-02-07 13:33:59 -05:00
hardikj b448f88c46 task name can not be reserved keyword
Task name can not be one of the engine commands: noop, fail, pause, succeed.

Change-Id: I4e92aba560daa3e045375a6728a7aee2b80473fc
Closes-Bug: #1705186
2018-01-23 05:42:48 +00:00
Adriano Petrich 3ce34674cf Dynamic action name evaluation.
Evaluate action names dynamically, so yaql or jinja expression can be
used

Change-Id: I48761c215f0255976c330ffa34f27bb695c944a9
Implements: blueprint mistral-dynamic-actions
2017-10-05 03:30:59 +00:00
Istvan Imre ff78d7f659 Dynamic workflow name evaluation.
Evaluate workflow names dynamically, so yaql or jinja expression
is allowed as sub-workflow name. Tasks names are not yet
dynamically evaluated.

Partially implements: blueprint mistral-dynamic-actions

Change-Id: Icfe591e27a4f45c2e3dcfa83512217f3b2122189
2017-08-16 13:17:10 +02:00
Renat Akhmerov 5c185c3481 Optimizing lang schema validation
* Before this patch some language specification schemas were
  validated twice (or even more) because some parent specs had
  references to specific schemas of child specs in their schemas.
  And due to our recursive parsing algorithm the same schemas were
  checked many times. It reduces performance and complicates
  the entire lang specification framework because there's too many
  relationships between different specs. The better approach is to
  reduce a number of relationships. One way is to not check schemas
  child specs when checking parent spec schema. This patch removes
  such references replacing them just common schemas like
  "non-empty" or "any" which don't lead to validating sub-schemas
  when validating parent schemas.
* Minor style changes

Change-Id: I6b695c1870bf8b70112332d4052115543382cdc7
2017-04-13 14:30:42 +07:00
Renat Akhmerov 6c6e212688 Advanced publishing: change workflow lang schema
* 'on-success', 'on-error' and 'on-complete' can now look like
  described in "Advanced Publishing" specification [1]
* Refactored all places related to the spec changes
* Added unit tests for advanced schema of 'on-xxx' clauses

[1] https://github.com/openstack/mistral-specs/blob/master/specs/pike/approved/advanced_publishing.rst

Change-Id: I190fcec0a40ca6f97d712168f4be7a418bd1f0e8
Partially implements: blueprint mistral-advanced-publishing-global-vars
2017-04-13 12:05:34 +07:00
Renat Akhmerov f7e04b3ef9 Add missing schema validation and unit tests for 'publish-on-error'
Change-Id: I87719d9ed9af7560fdff7af9215d99a686c4d8df
2017-03-24 16:34:14 +07:00
Renat Akhmerov 97311a6c04 Refactor methods in utils related to dicts
* The previous versions of these methods were too specific for the
  general utils module. Now they are more generic thereby less
  confusing.

Change-Id: Ifa8bbf0cc8a63bf1de1142dc8c52b1f8ad3958c5
2017-03-13 19:07:47 +07:00
Renat Akhmerov 5c140a2fce Rename package 'workbook' to 'lang'
* The previous name of the package is not relevant anymore, it was
  chosen for reasons that don't make sense anymore. A name of the
  package should reflect that it's about the workflow language.

Change-Id: I7e05ba97abf0aa5db4e19d9d72b4ea0d52e328fb
2017-03-06 15:41:37 +07:00