We have no way to determine whether the workflow definition has
changed or not.
This complicates the work with Mistral for external services.
A 'checksum' field has been added to workflow_definitions_v2
to solve this issue.
Implements: blueprint mistral-add-checksum-field-to-wf-definition
Change-Id: I99d095c6c74c62f134aedfa3a63308a0712db38d
* These workflows are openstack workflows and should be in mistral-extra
* Rename the function name to register_preinstalled_workflows
* Rename then function name to register_preinstalled_actions
Change-Id: I80fc1dfe4ec5e81b1b5163c9771eea5a49432f88
Depends-On: https://review.opendev.org/#/c/709970/
Depends-On: https://review.opendev.org/#/c/710024/
Update to PyYAML 5.1 to be able to turn off key sorting
in workflow definitions. We should not change user's data
until user asks to.
Change-Id: I888008ac8f9c12bd92a9bd72bd7c276a11827847
Closes-Bug: #1815515
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
* For the sake of the service performance, it may make sense to
disable validation of the workflow language syntax if it is
affordable for a particular use case. For example, if all
workflows are auto-generated by a 3rd party system and tested
thoroughly (either by running them with Mistral or at least
validating them via the special Mistral endpoint) then we can
safely disable validation of the language syntax when uploading
workflow definitions. For production systems it makes a big
difference if workflow texts are large (thousands of tasks).
This patch adds the boolean parameter "skip_validation" for API
requests like "POST /v2/workflows" to disable validation, if
needed, and the new configuration property "validation_mode"
to set a desired validation mode.
The option is an enumeration and has the following valid values:
1) "enabled" - enabled for all API requests unless it's
explicitly disabled in the request itself
2) "mandatory" - enabled for all API requests regardless
of the flag in the request
3) "disabled" - disabled for all API requrests regardless
of the flag in the request
"mandatory" is choosen as the default value for this new
property to keep compatibility with the previous versions.
* Minor style changes.
Closes-Bug: #1844242
Change-Id: Ib509653d38254954f8449be3031457e5f636ccf2
In case we load definition of one workflow, there is no need to convert
it to yaml and back.
Change-Id: Ia7fb914843e56650c8f6ff5e8c16d1b57001c0fc
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
In case of creating/updating multiple workflows from one yaml,
we should not save the whole input to each workflow.
Closes-Bug: #1792975
Change-Id: I724c041ab3441805fcfa2cfc4a50afd774998cc7
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
Currently when we get a specification using the instantiate_spec function,
we always validate their schema and semantics over and over again.
To prevent it we add new validate parameter to a Spec class.
The validate parameter must be True when we create a workflow, workbook
or action using a mistral-api. In all other cases, it must be False.
Change-Id: Ia450ea9635bc75c204fe031cfeeab154f1d03862
Closes-Bug: #1738769
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
This patch brings namespace support to workbooks.
Namespace of the workbook is inherited by workflows.
Implements: blueprint mistral-namespace-for-actions-workbooks
Change-Id: I2c66b3961915f0f35a9c468eb6dd0c0c70995234
* Previously, we had two copies of the workflow environment
passed by a user: one was in the 'params' fields under ke 'env'
key and another one was copied into the 'context' field under
the '__env' key so that we can evaluate expressions involving
the env() function (YAQL or Jinja).
This patch removes the copy from the 'context' field in favor
of using an ad-hoc ContextView structure where we now also
weave in the environment dictionary under the same key '__env'.
Related-Bug: #1757966
Change-Id: I1204b082794b376787d126136a79dd204ec3af07
* Fixed the implementation of data_flow.evaluate_task_outbound_context
method so that it doesn't use copy.deepcopy() because it may be
very expensive (both CPU and RAM) on big dictionaries.
Change-Id: I6cc77c6ded1e8f00ff164156be9312e1ccb86efb
Some log information starts with capital letter
Some log information starts with lowercase letter
Change them to starting with capital letter
Closes-Bug:#1732375
Change-Id: Ib40a327d299292ba3a417f3f0384f466fcefaa80
Allow adding many workflows with the same name.
The way this works is by adding the new workflows under another
namespace.
Notice that:
1. Namespaces are not part of the mistral DSL.
2. When executing a workflow, the namespace passes down to the
subworkflow.
3. When searching for the sub-workflow definition -
If no workflow was found in the given namespace, than the defualt
namespace will be used.
4. The default namespace is an empty string.
5. The namespace property or the workflow execution will be under
params
Partially-Implements: blueprint create-and-run-workflows-within-a-namespace
Change-Id: Id248ec5906a0899d188675d002b45f6249d36d90
if called by mistral-db-manage it will log info level
to /var/log/mistral/mistral-db-manage
Included some debug level logging also for main actions.
Change-Id: I0574b38efcc0dd7485953bf3089ff0a5a3ef6394
Closes-Bug: #1689821
* The previous name of the package is not relevant anymore, it was
chosen for reasons that don't make sense anymore. A name of the
package should reflect that it's about the workflow language.
Change-Id: I7e05ba97abf0aa5db4e19d9d72b4ea0d52e328fb
* It's redundant to keep environment data in task inbound context,
it is immutable and we can always take it from workflow execution
object which is more efficient from DB space consumption
standpoint. The only case when it's allowed to modify data
in a workflow environment is when we either resume or re-run
a workflow and in this case we can change it for the whole
workflow execution.
Change-Id: I244c1768aaa306f8ad41084325107a40005d874c
* The problem was that cache instance was instantiated on different
Mistral instances and invalidation didn't work properly (known
problem for local cache implementations). The solution, first of
all, is to cache specifications by workflow execution ids so that
we get a consistent spec value during workflow execution lifetime.
And secondly, if we need to build a specification based on
workflow definition id we also need to use 'updated_at' as part of
cache key so that if workflow definition has changed the cache is
updated properly. Old cache entities would be kicked out of cache
by LRU algorithm as the cache runs out of space.
Change-Id: If97b2e47d8adcbd7b5d5844b56e24eac1b1ae6c1
* Introduced new class Workflow that manages life-cycle of running
workflows and is responsible for managing workflow persistent state
* Moved all workflow level logic to workflow handler and Workflow class
* Changed semantics if how workflows start errors are handled.
Previously, in case of invalid user input Mistral engine would store
information about error in "state_info" field of workflow execution
and bubble up an exception to the user. This approach was incorrect
for a number of reasons including broken semantics: if an exception
was raised due to invalid input it's normal to expect that system
state has not changed. After this refactoring, engine only raises
an exception in case of bad user input. That way behavior is
consistent with the idea of exceptional situations.
* Fixed unit tests in according to the previous point
* Fixed a number of logical issues in tests. For example, in
test_default_engine.py we expected one type of errors (e.g. env not
found) but effectively received another one (invalid input).
Partially implements: blueprint mistral-engine-error-handling
Change-Id: I09070411fd833df8284cb80db69b8401a40eb6fe
Add optional env kwarg to resume_workflow and rerun_workflow in default
engine. Add database API to update env in the params and context of the
existing workflow execution. The updated env will be merged into existing
__env of workflow context. For existing task, the env will be merged into
__env of in_context.
Partially implements: blueprint mistral-rerun-update-env
Change-Id: Id378762d90ca5fc62d14e22134f18f9bce71a123
* When updating a workflow with UUID provided, only one workflow
definition could be contained in request body.
* Workflow name can also be updated when using UUID.
* Tenant can not update workflows of other tenants.
Partially implements: blueprint use-workflow-id-in-rest-api
Change-Id: I87f9122b8ad9727a3eeb429ce19835c57c76b32d
* Fixed work of mistral-db-manage populate
* Fixed modifying of standard workflows
Closes-Bug: #1478448
Closes-Bug: #1396121
Change-Id: Ia2825408ee43a6dfc80e33fa9614e11f3ff6d97b
Keep behavior of updating workflow consistent with that of workbook and
environment. The update command will fail if the workflow doesn't exist.
Change-Id: I7a871c2ac378ac0b7f6360f03f395816b9d0f4b0
Closes-Bug: #1418545
* Fixed standard actions and workflows paths.
If mistral is installed in system, 'resources'
are not copied to dist-packages/mistral, so
mistral-db-manage populate command won't work.
Now resources are included in mistral.
Closes-Bug: #1457842
Change-Id: I3149867e615acd188691b299a051e7458e2eb386
* Created a Execution hierarcy mapped on a single table:
- ActionExecution
- WorkflowExecution
- TaskExecution
* Created necessary associations within Execution hierarchy
* Created necessary methods for Execution objects on DB API
* Created Definition hierarchy mapped on separate tables:
- Workbook
- WorkflowDefinition
- ActionDefinition
* Renamed attributes
- 'wf_name' -> 'workflow_name'
- 'wf_spec' -> 'spec'
* Fixed all unit tests
TODO:
* Complete renaming throughout the code
* Further refactoring of workflow engine
Change-Id: I0032bea573d9200025f9f7dd951e93cb6f1661bb
* Refactored DB models with new mixin class MistralSecureModelMixin
* Refactored services working with workflows, worbooks, actions
* Other minor changes
Change-Id: I9f2cb5c02670aa8c4c3dd06c4301f2e613f5c013
* Standard workflows are creating during sync_db.sh
* Standard workflows include:
- std.create_instance
- std.delete_instance
(list will be extended in future)
* Standard actions:
- std.wait_ssh (needed for std.create_instance)
* Make it possible to see these workflows from any project (global scope)
* Small changes in sqlalchemy api and workflows_service
Partially implements blueprint mistral-multitenancy
Change-Id: I8a8ace40949b2b711a292aac94d7e6354d1dff9c
* Added WorkflowListSpec that represents a top level specification
for uploading workflows (shouldn't be confused with WorkflowSpecList)
* Added workflow service that takes care of parsing, validating and
creating multiple workflows from definition
* Added unit tests for workflow service
* Fixed workflow controller to work with multiple workflows on post/put
* Fixed workflow controller tests
Change-Id: I70a46a16e1970673fec0352091e4b610f320fbb8
Vadim Zelenevskii