Yaql Tasks Function
This new function will allow user to get a list of tasks matching certain filter. For example only task in state ERROR from the current execution. This will work in both YAQL an Jinja2 expressions. blueprint yaql-tasks-function Change-Id: I218ad84770e93a25708ff64ce17e10f08187c6d3
This commit is contained in:
parent
20d3fb5e0f
commit
a4bd1de3f8
3
AUTHORS
3
AUTHORS
|
@ -1,3 +0,0 @@
|
||||||
Lingxian Kong <anlin.kong@gmail.com>
|
|
||||||
Winson Chan <wcchan@stackstorm.com>
|
|
||||||
hparekh <hardik.parekh@nectechnologies.in>
|
|
|
@ -0,0 +1,340 @@
|
||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
===================
|
||||||
|
Yaql Tasks Function
|
||||||
|
===================
|
||||||
|
|
||||||
|
Launchpad blueprint:
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/mistral/+spec/yaql-tasks-function
|
||||||
|
|
||||||
|
This new function will allow user to get a list of tasks matching certain
|
||||||
|
filters. For example: only task in state ERROR from the current execution.
|
||||||
|
|
||||||
|
Work on this draft started before **Jinja2** support was added. That said, a
|
||||||
|
user will be able to use this function both as part of **YAQL** expression and
|
||||||
|
as part of **Jinja2** expression.
|
||||||
|
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
There's no easy way in the error handler to know which task failed and failure
|
||||||
|
information.
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
---------
|
||||||
|
|
||||||
|
* When an error happens and default on-error is triggered, decide how to handle
|
||||||
|
it according to the task that failed, even if it is in a nested workflow.
|
||||||
|
|
||||||
|
* Be able to get all tasks that failed in a workflow easily. Including nested
|
||||||
|
tasks.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Add a new **YAQL** function called **tasks**, which will be similar to the
|
||||||
|
**task** function we have today. The main difference is that **tasks** function
|
||||||
|
will return a list and will have more querying options.
|
||||||
|
|
||||||
|
Positional parameters for the new function:
|
||||||
|
|
||||||
|
#. ``wf-execution-id`` (optional) - will allow to get tasks information from a
|
||||||
|
specific workflow execution (either the current execution or a different
|
||||||
|
one), if not passed, it will list all tasks.
|
||||||
|
#. ``recursive`` (optional. Default: false) - if true treat all tasks in
|
||||||
|
nested workflows as if they where also a part of all higher level
|
||||||
|
wf-executions. Relevant mostly when filtering using ``wf-execution-id``.
|
||||||
|
#. ``state`` (optional) - get only tasks with the given state, for example
|
||||||
|
all ERROR tasks. If not passed, it will list all tasks.
|
||||||
|
#. ``flat`` (optional. Default: false) - if true, only list the tasks that
|
||||||
|
match at least one of the next conditions:
|
||||||
|
|
||||||
|
* tasks of type action
|
||||||
|
* tasks of type workflow, that also have a different state than the one
|
||||||
|
of the nested workflow execution that was executed because of the task.
|
||||||
|
|
||||||
|
|
||||||
|
workflow with **YAQL** function example:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
---
|
||||||
|
version: '2.0'
|
||||||
|
wf:
|
||||||
|
type: direct
|
||||||
|
input:
|
||||||
|
- tasks_in_error: []
|
||||||
|
tasks:
|
||||||
|
my_example:
|
||||||
|
action: std.noop
|
||||||
|
publish:
|
||||||
|
# publish all tasks in state ERROR of the current execution only
|
||||||
|
tasks_in_error: <% tasks(execution().id, ERROR) %>
|
||||||
|
|
||||||
|
When running 'mistral task-get-published $TASK_ID' where TASK_ID is the ID
|
||||||
|
of my_example task execution, mistral will return:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
{
|
||||||
|
"tasks_in_error": []
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
The items in the list (when the list isn't empty) are from the same type and
|
||||||
|
structure as returned today from the task function.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
The parameters order and function name can be different.
|
||||||
|
We might want to pass a list of states and not just one.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
This change doesn't have to influence the data model.
|
||||||
|
|
||||||
|
However, right now the behavior is that a task state is updated before
|
||||||
|
publishing values are evaluated, and that is what is visible from the context
|
||||||
|
of the function. We might want to change it in the future.
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
End user impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
The user will have a new function that can be used as part of **YAQL** or
|
||||||
|
**Jinja2** expressions.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
There is a possibility each call for this new function will trigger multiple
|
||||||
|
DB queries. The more nested the workflow is, the more queries. This is not
|
||||||
|
very efficient, but we can improve this later on if necessary.
|
||||||
|
|
||||||
|
Another thing that might happen is, if the user will not filter the tasks,
|
||||||
|
the amount of data might cause a timeout.
|
||||||
|
|
||||||
|
Example for clarity - a workflow and the published result
|
||||||
|
::
|
||||||
|
|
||||||
|
---
|
||||||
|
version: '2.0'
|
||||||
|
|
||||||
|
wf:
|
||||||
|
type: direct
|
||||||
|
input:
|
||||||
|
- tasks_in_error: []
|
||||||
|
tasks:
|
||||||
|
my_example:
|
||||||
|
action: std.noop
|
||||||
|
publish:
|
||||||
|
# publish all the tasks of the current execution only
|
||||||
|
all_tasks_in_execution: <% tasks(execution().id) %>
|
||||||
|
|
||||||
|
The result of publishing:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
{
|
||||||
|
"all_tasks_in_execution": [
|
||||||
|
{
|
||||||
|
"state_info": null,
|
||||||
|
"name": "my_example",
|
||||||
|
"spec": {
|
||||||
|
"action": "std.noop",
|
||||||
|
"version": "2.0",
|
||||||
|
"type": "direct",
|
||||||
|
"name": "my_example",
|
||||||
|
"publish": {
|
||||||
|
"all_tasks_in_execution": "<% tasks(execution().id) %>"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"state": "SUCCESS",
|
||||||
|
"result": null,
|
||||||
|
"published": {},
|
||||||
|
"id": "a8b4787c-5b10-488a-8539-8370488fed8c"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
To summarize the issue:
|
||||||
|
Right now the state of the task when publishing is SUCCESS and not RUNNING,
|
||||||
|
even though a user might expect it to be RUNNING. We don't have to do
|
||||||
|
anything about it, but we should document this really well and say this might
|
||||||
|
change in the future.
|
||||||
|
|
||||||
|
Deployer impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
michal-gershenzon
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
melisha
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* implement the new function and filters based on argument position in the
|
||||||
|
function.
|
||||||
|
* write tests.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Examples for the next scenario: a mistral setup with 3 workflow executions,
|
||||||
|
each execution started from a different workflow (just to make it more easy):
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
execution of workflow1 (workflow execution id = 1)
|
||||||
|
|-top_level_wf1_task_1 SUCCESS (workflow execution id = 1)
|
||||||
|
|---second_level_wf1_task_1 SUCCESS (workflow execution id = 2)
|
||||||
|
|-----third_level_wf1_task_1 SUCCESS (workflow execution id = 3)
|
||||||
|
|-----third_level_wf1_task_2 SUCCESS (workflow execution id = 3)
|
||||||
|
|-----third_level_wf1_task_3 ERROR (workflow execution id = 3)
|
||||||
|
|---second_level_wf1_task_2 SUCCESS (workflow execution id = 2)
|
||||||
|
|---second_level_wf1_task_3 SUCCESS (workflow execution id = 2)
|
||||||
|
|-top_level_wf1_task_2 SUCCESS (workflow execution id = 1)
|
||||||
|
|
||||||
|
execution of workflow2 (workflow execution id = 1001)
|
||||||
|
|-top_level_wf2_task_1 SUCCESS (workflow execution id = 1001)
|
||||||
|
|-top_level_wf2_task_2 SUCCESS (workflow execution id = 1001)
|
||||||
|
|
||||||
|
execution of workflow3 (workflow execution id = 300001)
|
||||||
|
|-top_level_wf3_task_1 ERROR (workflow execution id = 300001)
|
||||||
|
|---second_level_wf3_task_1 ERROR (workflow execution id = 300002)
|
||||||
|
|-----third_level_wf3_task_1 SUCCESS (workflow execution id = 300003)
|
||||||
|
|-----third_level_wf3_task_2 SUCCESS (workflow execution id = 300003)
|
||||||
|
|-----third_level_wf3_task_3 ERROR (workflow execution id = 300003)
|
||||||
|
|---second_level_wf3_task_2 SUCCESS (workflow execution id = 300002)
|
||||||
|
|---second_level_wf3_task_3 SUCCESS (workflow execution id = 300002)
|
||||||
|
|-top_level_wf3_task_2 ERROR (workflow execution id = 300001)
|
||||||
|
|
||||||
|
|
||||||
|
Here is a table representation with additional info:
|
||||||
|
|
||||||
|
::
|
||||||
|
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| top level execution id | execution id | task name | task state | task type | inner execution state if exist | inner execution id if exist |
|
||||||
|
+========================+==============+==============================+================+===============+=================================+==============================+
|
||||||
|
| 1 | 1 | top_level_wf1_task_1 | SUCCESS | WORKFLOW | SUCCESS | 2 |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 1 | 1 | top_level_wf1_task_2 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 1 | 2 | second_level_wf1_task_1 | SUCCESS | WORKFLOW | ERROR | 3 |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 1 | 2 | second_level_wf1_task_2 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 1 | 2 | second_level_wf1_task_3 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 1 | 3 | third_level_wf1_task_1 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 1 | 3 | third_level_wf1_task_2 | ERROR | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 1 | 3 | third_level_wf1_task_3 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 1001 | 1001 | top_level_wf2_task_1 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 1001 | 1001 | top_level_wf2_task_2 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 300001 | 300001 | top_level_wf3_task_1 | ERROR | WORKFLOW | ERROR | 300002 |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 300001 | 300001 | top_level_wf3_task_2 | ERROR | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 300001 | 300002 | second_level_wf3_task_1 | ERROR | WORKFLOW | ERROR | 300003 |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 300001 | 300002 | second_level_wf3_task_2 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 300001 | 300002 | second_level_wf3_task_3 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 300001 | 300003 | third_level_wf3_task_1 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 300001 | 300003 | third_level_wf3_task_2 | SUCCESS | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
| 300001 | 300003 | third_level_wf3_task_3 | ERROR | ACTION | - | - |
|
||||||
|
+------------------------+--------------+------------------------------+----------------+---------------+---------------------------------+------------------------------+
|
||||||
|
|
||||||
|
reminder: the order of the function arguments is:
|
||||||
|
|
||||||
|
1. wf-execution-id
|
||||||
|
2. recursive
|
||||||
|
3. state
|
||||||
|
4. flat
|
||||||
|
|
||||||
|
calling 'tasks()' will:
|
||||||
|
return all 18 tasks.
|
||||||
|
|
||||||
|
calling 'tasks(1)' or 'tasks(1, false)' will:
|
||||||
|
return 2 tasks of workflow1 execution (only tasks with execution id of 1):
|
||||||
|
* top_level_task_1
|
||||||
|
* top_level_task_2
|
||||||
|
|
||||||
|
calling 'tasks(1, true)' will:
|
||||||
|
return all 8 tasks of the workflow1 workflow execution.
|
||||||
|
|
||||||
|
calling 'tasks(1001)' or 'tasks(1001, true)' or 'tasks(1001, false)' will:
|
||||||
|
return the 2 tasks of workflow2 execution.
|
||||||
|
|
||||||
|
calling 'tasks(1, true, ERROR)' will:
|
||||||
|
return 1 task of workflow1 execution:
|
||||||
|
* third_level_wf1_task_3
|
||||||
|
|
||||||
|
calling 'tasks(1, false, ERROR)' will:
|
||||||
|
return an empty list.
|
||||||
|
|
||||||
|
calling 'tasks(300001, true, ERROR)' or 'tasks(300001, true, ERROR, false)'
|
||||||
|
will:
|
||||||
|
return 4 task of workflow3 execution:
|
||||||
|
* top_level_wf3_task_1
|
||||||
|
* second_level_wf3_task_1
|
||||||
|
* third_level_wf3_task_3
|
||||||
|
* top_level_wf3_task_2
|
||||||
|
|
||||||
|
calling 'tasks(300001, true, ERROR, true)' will:
|
||||||
|
return 2 task of workflow3 execution:
|
||||||
|
* third_level_wf3_task_3
|
||||||
|
* top_level_wf3_task_2
|
||||||
|
|
||||||
|
calling 'tasks(1, true, SUCCESS, true)' will:
|
||||||
|
return 6 tasks of workflow1 execution:
|
||||||
|
* top_level_wf1_task_2
|
||||||
|
* second_level_wf1_task_1
|
||||||
|
* second_level_wf1_task_2
|
||||||
|
* second_level_wf1_task_3
|
||||||
|
* third_level_wf1_task_1
|
||||||
|
* third_level_wf1_task_3
|
||||||
|
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
None.
|
Loading…
Reference in New Issue