openstack
/
mistral
Code Issues Proposed changes

Changes made to comply with OpenStack writing style 

The writing styles at few places were not in compliance with
the OpenStack writing style as mentioned in
http://docs.openstack.org/contributor-guide/writing-style
/general-writing-guidelines.html. This patch changes the heading
styles and adds a few other minor changes in the documents.

Change-Id: I0d90ba32ddcef0427b1dc4358206210f166e798a
Partial-Implements: blueprint mistral-doc
This commit is contained in:
Sharat Sharma 2016-10-04 19:17:19 +05:30
parent e6134eeec0
commit c07cf48a60
12 changed files with 254 additions and 170 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
doc/source/architecture.rst
View File

@ -3,8 +3,14 @@ Architecture
.. image:: img/mistral_architecture.png .. image:: img/mistral_architecture.png
* Engine - handles control and data flow of workflow executions. Computes which tasks are ready and places them in a queue. Passes the data from task to task, deals with condition transitions, etc. * Engine - handles control and data flow of workflow executions. Computes which
* Task Executors - execute task Actions. Pick up tasks from the queue, run actions, and send results back to the engine. tasks are ready and places them in a queue. Passes the data from task to
task, deals with condition transitions, etc.
* Task Executors - execute task Actions. Pick up tasks from the queue, run
actions, and send results back to the engine.
* API server - exposes REST API to operate and monitor workflow executions. * API server - exposes REST API to operate and monitor workflow executions.
* Scheduler - stores and executes delayed calls. It is the important Mistral component since it interacts with engine and executors. It also triggers workflows on events (e.g., periodic cron event) * Scheduler - stores and executes delayed calls. It is the important Mistral
* Persistence - stores workflow definitions, current execution states, and past execution results. component since it interacts with engine and executors. It also triggers
workflows on events (e.g., periodic cron event)
* Persistence - stores workflow definitions, current execution states, and
past execution results.

4
doc/source/developer/devstack.rst
View File

@ -3,8 +3,8 @@ Mistral Devstack Installation
1. Download DevStack:: 1. Download DevStack::
git clone https://github.com/openstack-dev/devstack.git $ git clone https://github.com/openstack-dev/devstack.git
cd devstack $ cd devstack
2. Add this repo as an external repository, edit ``localrc`` file:: 2. Add this repo as an external repository, edit ``localrc`` file::

28
doc/source/developer/extending_yaql.rst
View File

@ -10,9 +10,9 @@ Tutorial
.. code-block:: bash .. code-block:: bash
mkdir my_project $ mkdir my_project
cd my_project $ cd my_project
vim setup.py $ vim setup.py
.. code-block:: python .. code-block:: python
@ -68,9 +68,9 @@ the paths or file names used.
.. code-block:: bash .. code-block:: bash
mkdir -p my_package/sub_package $ mkdir -p my_package/sub_package
touch my_package/__init__.py $ touch my_package/__init__.py
touch my_package/sub_package/__init__.py $ touch my_package/sub_package/__init__.py
3. Write a function in ``yaql.py``. 3. Write a function in ``yaql.py``.
@ -79,8 +79,8 @@ YAQL context available inside the function.
.. code-block:: bash .. code-block:: bash
cd my_package/sub_package $ cd my_package/sub_package
vim yaql.py $ vim yaql.py
.. code-block:: python .. code-block:: python
@ -120,15 +120,15 @@ For more information about YAQL, read the `official YAQL documentation <http://y
.. code-block:: bash .. code-block:: bash
curl https://bootstrap.pypa.io/get-pip.py | python $ curl https://bootstrap.pypa.io/get-pip.py | python
pip install --upgrade setuptools $ pip install --upgrade setuptools
cd - $ cd -
5. Install the package (note that there is a dot ``.`` at the end of the line). 5. Install the package (note that there is a dot ``.`` at the end of the line).
.. code-block:: bash .. code-block:: bash
pip install . $ pip install .
6. The YAQL function can be called in Mistral using its name ``random_uuid``. 6. The YAQL function can be called in Mistral using its name ``random_uuid``.
@ -165,8 +165,8 @@ parent folder of ``my_package`` is ``/path/to/my_project``.
.. code-block:: bash .. code-block:: bash
export PYTHONPATH=$PYTHONPATH:/path/to/my_project $ export PYTHONPATH=$PYTHONPATH:/path/to/my_project
vim $(find / -name "mistral.*egg-info*")/entry_points.txt $ vim $(find / -name "mistral.*egg-info*")/entry_points.txt
.. code-block:: ini .. code-block:: ini

4
doc/source/developer/troubleshooting.rst
View File

@ -1,7 +1,7 @@
Troubleshooting And Debugging Troubleshooting And Debugging
============================= =============================
Mistral-Dashboard Debug Instructions Mistral-Dashboard debug instructions
------------------------------------ ------------------------------------
**Pycharm** **Pycharm**
@ -66,6 +66,6 @@ After a successful login bring the port back - it will continue your session.
- If you would like to manually restart the apache server, - If you would like to manually restart the apache server,
open a terminal and run:: open a terminal and run::
sudo service apache2 restart $ sudo service apache2 restart
*(if not under Ubuntu, replace "sudo" with an identical command)* *(if not under Ubuntu, replace "sudo" with an identical command)*

69
doc/source/developer/webapi/v2.rst
View File

@ -1,28 +1,30 @@
V2 API V2 API
====== ======
This API describes the ways of interacting with Mistral service via HTTP protocol using Representational State Transfer concept (ReST). This API describes the ways of interacting with Mistral service via HTTP protocol
using Representational State Transfer concept (ReST).
Basics Basics
------- -------
Media Types Media types
^^^^^^^^^^^ ^^^^^^^^^^^
Currently this API relies on JSON to represent states of REST resources. Currently this API relies on JSON to represent states of REST resources.
Error States Error states
^^^^^^^^^^^^ ^^^^^^^^^^^^
The common HTTP Response Status Codes (https://github.com/for-GET/know-your-http-well/blob/master/status-codes.md) are used. The common HTTP Response Status Codes (https://github.com/for-GET/know-your-http-well/blob/master/status-codes.md) are used.
Application Root [/] Application root [/]
^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^
Application Root provides links to all possible API methods for Mistral. URLs for other resources described below are relative to Application Root. Application Root provides links to all possible API methods for Mistral. URLs
for other resources described below are relative to Application Root.
API v2 Root [/v2/] API v2 root [/v2/]
^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^
All API v2 urls are relative to API v2 root. All API v2 urls are relative to API v2 root.
@ -32,7 +34,11 @@ Workbooks
.. autotype:: mistral.api.controllers.v2.resources.Workbook .. autotype:: mistral.api.controllers.v2.resources.Workbook
:members: :members:
`name` is immutable. tags is a list of values associated with a workbook that a user can use to group workbooks by some criteria (deployment workbooks, Big Data processing workbooks etc.). Note that name and tags get inferred from workbook definition when Mistral service receives a POST request. So they can't be changed in another way. `name` is immutable. tags is a list of values associated with a workbook that
a user can use to group workbooks by some criteria (deployment workbooks,
Big Data processing workbooks etc.). Note that name and tags get inferred from
workbook definition when Mistral service receives a POST request. So they
can't be changed in another way.
.. autotype:: mistral.api.controllers.v2.resources.Workbooks .. autotype:: mistral.api.controllers.v2.resources.Workbooks
:members: :members:
@ -47,7 +53,10 @@ Workflows
.. autotype:: mistral.api.controllers.v2.resources.Workflow .. autotype:: mistral.api.controllers.v2.resources.Workflow
:members: :members:
`name` is immutable. tags is a list of values associated with a workflow that a user can use to group workflows by some criteria. Note that name and tags get inferred from workflow definition when Mistral service receives a POST request. So they can't be changed in another way. `name` is immutable. tags is a list of values associated with a workflow that
a user can use to group workflows by some criteria. Note that name and tags get
inferred from workflow definition when Mistral service receives a POST request.
So they can't be changed in another way.
.. autotype:: mistral.api.controllers.v2.resources.Workflows .. autotype:: mistral.api.controllers.v2.resources.Workflows
:members: :members:
@ -84,7 +93,9 @@ Executions
Tasks Tasks
----- -----
When a workflow starts Mistral creates an execution. It in turn consists of a set of tasks. So Task is an instance of a task described in a Workflow that belongs to a particular execution. When a workflow starts Mistral creates an execution. It in turn consists of a
set of tasks. So Task is an instance of a task described in a Workflow that
belongs to a particular execution.
.. autotype:: mistral.api.controllers.v2.resources.Task .. autotype:: mistral.api.controllers.v2.resources.Task
@ -103,7 +114,9 @@ When a workflow starts Mistral creates an execution. It in turn consists of a se
Action Executions Action Executions
----------------- -----------------
When a Task starts Mistral creates a set of Action Executions. So Action Execution is an instance of an action call described in a Workflow Task that belongs to a particular execution. When a Task starts Mistral creates a set of Action Executions. So Action Execution
is an instance of an action call described in a Workflow Task that belongs to a
particular execution.
.. autotype:: mistral.api.controllers.v2.resources.ActionExecution .. autotype:: mistral.api.controllers.v2.resources.ActionExecution
@ -121,7 +134,10 @@ When a Task starts Mistral creates a set of Action Executions. So Action Executi
Cron Triggers Cron Triggers
------------- -------------
Cron trigger is an object that allows to run Mistral workflows according to a time pattern (Unix crontab patterns format). Once a trigger is created it will run a specified workflow according to its properties: pattern, first_execution_time and remaining_executions. Cron trigger is an object that allows to run Mistral workflows according to a time
pattern (Unix crontab patterns format). Once a trigger is created it will run a
specified workflow according to its properties: pattern, first_execution_time and
remaining_executions.
.. autotype:: mistral.api.controllers.v2.resources.CronTrigger .. autotype:: mistral.api.controllers.v2.resources.CronTrigger
@ -137,16 +153,23 @@ Cron trigger is an object that allows to run Mistral workflows according to a ti
Environments Environments
------------ ------------
Environment contains a set of variables which can be used in specific workflow. Using Environment possible to create and map action default values - just provide '__actions' key in 'variables'. All these variables can be accessed in workflow DSL by <% $.__env %> expression. Environment contains a set of variables which can be used in specific workflow.
Using Environment possible to create and map action default values - just provide
'__actions' key in 'variables'. All these variables can be accessed in workflow
DSL by <% $.__env %> expression.
Example of using in DSL:: Example of using in DSL
.. code-block:: yaml
workflow: workflow:
tasks: tasks:
task1: task1:
action: std.echo output=<% $.__env.my_echo_output %> action: std.echo output=<% $.__env.my_echo_output %>
Example of creating action defaults:: Example of creating action defaults
.. code-block:: yaml
...ENV... ...ENV...
@ -174,9 +197,17 @@ Note: using CLI, Environment can be created via JSON or YAML file.
Services Services
-------- --------
Through service management API, system administrator or operator can retrieve Mistral services information of the system, including service group and service identifier. The internal implementation of this feature make use of tooz library, which needs coordinator backend(the most commonly used at present is Zookeeper) installed, please refer to tooz official documentation for more detailed instruction. Through service management API, system administrator or operator can retrieve
Mistral services information of the system, including service group and service
identifier. The internal implementation of this feature make use of tooz library,
which needs coordinator backend(the most commonly used at present is Zookeeper)
installed, please refer to tooz official documentation for more detailed
instruction.
There are three service groups according to Mistral architecture currently, namely api_group, engine_group and executor_group. The service identifier contains name of the host the service is running on and the process identifier of the service on that host. There are three service groups according to Mistral architecture currently, namely
api_group, engine_group and executor_group. The service identifier contains name
of the host the service is running on and the process identifier of the service on
that host.
.. autotype:: mistral.api.controllers.v2.resources.Service .. autotype:: mistral.api.controllers.v2.resources.Service
:members: :members:
@ -190,7 +221,8 @@ There are three service groups according to Mistral architecture currently, name
Validation Validation
---------- ----------
Validation endpoints allow to check correctness of workbook, workflow and ad-hoc action DSL without having to upload them into Mistral. Validation endpoints allow to check correctness of workbook, workflow and ad-hoc
action DSL without having to upload them into Mistral.
**POST /v2/workbooks/validation** **POST /v2/workbooks/validation**
Validate workbook content (DSL grammar and semantics). Validate workbook content (DSL grammar and semantics).
@ -201,4 +233,5 @@ Validation endpoints allow to check correctness of workbook, workflow and ad-hoc
**POST /v2/actions/validation** **POST /v2/actions/validation**
Validate ad-hoc action content (DSL grammar and semantics). Validate ad-hoc action content (DSL grammar and semantics).
These endpoints expect workbook, workflow or ad-hoc action text (DSL) correspondingly in a request body. These endpoints expect workbook, workflow or ad-hoc action text (DSL) correspondingly
in a request body.

54
doc/source/dsl/dsl_v2.rst
View File

@ -103,7 +103,7 @@ This example workflow simply sends a command to OpenStack Compute
service Nova to start creating a virtual machine and wait till it's service Nova to start creating a virtual machine and wait till it's
created using special "retry" policy. created using special "retry" policy.
Workflow Types Workflow types
^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
Mistral DSL v2 introduces different workflow types and the structure of Mistral DSL v2 introduces different workflow types and the structure of
@ -117,10 +117,11 @@ Mistral provides two workflow types:
See corresponding sections for details. See corresponding sections for details.
Common Workflow Attributes Common workflow attributes
^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^
- **type** - Workflow type. Either 'direct' or 'reverse'. *Optional*. 'direct' by default. - **type** - Workflow type. Either 'direct' or 'reverse'. *Optional*. 'direct'
by default.
- **description** - Arbitrary text containing workflow description. *Optional*. - **description** - Arbitrary text containing workflow description. *Optional*.
- **input** - List defining required input parameter names and - **input** - List defining required input parameter names and
optionally their default values in a form "my_param: 123". *Optional*. optionally their default values in a form "my_param: 123". *Optional*.
@ -175,7 +176,7 @@ ahead it's worth saying that Mistral provides a lot of actions out of
the box (including actions for most of the core OpenStack services) and the box (including actions for most of the core OpenStack services) and
it's also easy to plug new actions into Mistral. it's also easy to plug new actions into Mistral.
Common Task Attributes Common task attributes
'''''''''''''''''''''' ''''''''''''''''''''''
All Mistral tasks regardless of workflow type have the following common All Mistral tasks regardless of workflow type have the following common
@ -183,7 +184,9 @@ attributes:
- **description** - Arbitrary text containing task description. - **description** - Arbitrary text containing task description.
*Optional*. *Optional*.
- **action** - Name of the action associated with the task. *Mutually exclusive with* **workflow**. If neither action nor workflow are provided then the action 'std.noop' will be used. - **action** - Name of the action associated with the task.
*Mutually exclusive with* **workflow**. If neither action nor workflow are
provided then the action 'std.noop' will be used.
- **workflow** - Name of the workflow associated with the task. - **workflow** - Name of the workflow associated with the task.
*Mutually exclusive with* **action**. *Mutually exclusive with* **action**.
- **input** - Actual input parameter values of the task. *Optional*. - **input** - Actual input parameter values of the task. *Optional*.
@ -266,8 +269,8 @@ automatically by engine if hasn't completed.
**concurrency** **concurrency**
Defines a max number of actions running simultaneously in a task. *Applicable* only for tasks that Defines a max number of actions running simultaneously in a task. *Applicable*
have *with-items*. only for tasks that have *with-items*.
**retry** **retry**
@ -283,7 +286,8 @@ Defines a pattern how task should be repeated in case of an error.
considered error. considered error.
- **continue-on** - Defines a YAQL expression that will continue iteration - **continue-on** - Defines a YAQL expression that will continue iteration
loop if it evaluates to 'true'. If it fires then the task is loop if it evaluates to 'true'. If it fires then the task is
considered successful. If it evaluates to 'false' then policy will break the iteration. considered successful. If it evaluates to 'false' then policy will break the
iteration.
Retry policy can also be configured on a single line as: Retry policy can also be configured on a single line as:
@ -295,7 +299,7 @@ Retry policy can also be configured on a single line as:
All parameter values for any policy can be defined as YAQL expressions. All parameter values for any policy can be defined as YAQL expressions.
Simplified Input Syntax Simplified input syntax
''''''''''''''''''''''' '''''''''''''''''''''''
When describing a workflow task it's possible to specify its input When describing a workflow task it's possible to specify its input
@ -343,7 +347,7 @@ keyword *input*. In this case all the parameters will be effectively
merged. If the same parameter is specified in both ways then the one merged. If the same parameter is specified in both ways then the one
under *input* keyword takes precedence. under *input* keyword takes precedence.
Direct Workflow Direct workflow
^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^
Direct workflow consists of tasks combined in a graph where every next Direct workflow consists of tasks combined in a graph where every next
@ -393,7 +397,7 @@ YAML example
   send_success_email:    send_success_email:
     action: send_email to_addrs=['admin@mysite.org'] body='Vm is successfully created and its id <% $.vm_id %>'      action: send_email to_addrs=['admin@mysite.org'] body='Vm is successfully created and its id <% $.vm_id %>'
Direct Workflow Task Attributes Direct workflow task attributes
''''''''''''''''''''''''''''''' '''''''''''''''''''''''''''''''
- **on-success** - List of tasks which will run after the task has - **on-success** - List of tasks which will run after the task has
@ -515,7 +519,7 @@ In this case instead of 1 it is possible to specify special
string value "one" which is introduced for symmetry with "all". However, string value "one" which is introduced for symmetry with "all". However,
it's up to the user whether to use "1" or "one". it's up to the user whether to use "1" or "one".
Reverse Workflow Reverse workflow
^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
In reverse workflow all relationships in workflow task graph are In reverse workflow all relationships in workflow task graph are
@ -571,13 +575,13 @@ YAML example
     action: send_email to='admin@mysite.org' body='Vm is created and id <% $.vm_id %> and ip address <% $.vm_ip %>'      action: send_email to='admin@mysite.org' body='Vm is created and id <% $.vm_id %> and ip address <% $.vm_ip %>'
     requires: [create_vm, associate_ip]      requires: [create_vm, associate_ip]
Reverse Workflow Task Attributes Reverse workflow task attributes
'''''''''''''''''''''''''''''''' ''''''''''''''''''''''''''''''''
- **requires** - List of tasks which should be executed before this - **requires** - List of tasks which should be executed before this
task. *Optional*. task. *Optional*.
Processing Collections Processing collections
^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^
YAML example YAML example
@ -657,7 +661,7 @@ similar to a regular function in general purpose programming language
like Python. It has a name and parameters. Mistral distinguishes 'system like Python. It has a name and parameters. Mistral distinguishes 'system
actions' and 'Ad-hoc actions'. actions' and 'Ad-hoc actions'.
System Actions System actions
^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
System actions are provided by Mistral out of the box and can be used by System actions are provided by Mistral out of the box and can be used by
@ -815,20 +819,20 @@ its wrapper - PyV8. For installing it, do the next steps:
1. Install required libraries - boost, g++, libtool, autoconf, subversion, libv8-legacy-dev: On Ubuntu:: 1. Install required libraries - boost, g++, libtool, autoconf, subversion, libv8-legacy-dev: On Ubuntu::
sudo apt-get install libboost-all-dev g++ libtool autoconf libv8-legacy-dev subversion make $ sudo apt-get install libboost-all-dev g++ libtool autoconf libv8-legacy-dev subversion make
2. Checkout last version of PyV8:: 2. Checkout last version of PyV8::
svn checkout http://pyv8.googlecode.com/svn/trunk/ pyv8 $ svn checkout http://pyv8.googlecode.com/svn/trunk/ pyv8
cd pyv8 $ cd pyv8
3. Build PyV8 - it will checkout last V8 trunk, build it, and then build PyV8:: 3. Build PyV8 - it will checkout last V8 trunk, build it, and then build PyV8::
sudo python setup.py build $ sudo python setup.py build
4. Install PyV8:: 4. Install PyV8::
sudo python setup.py install $ sudo python setup.py install
Example: Example:
@ -887,7 +891,7 @@ Another example for getting the current date and time:
        publish:         publish:
          current_date: <% task(get_date_task).result %>           current_date: <% task(get_date_task).result %>
Ad-hoc Actions Ad-hoc actions
^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
Ad-hoc action is a special type of action that can be created by user. Ad-hoc action is a special type of action that can be created by user.
@ -1042,7 +1046,7 @@ Attributes
- **actions** - Dictionary containing ad-hoc action definitions. - **actions** - Dictionary containing ad-hoc action definitions.
*Optional*. *Optional*.
Predefined Values/Functions in execution data context Predefined values/Functions in execution data context
----------------------------------------------------- -----------------------------------------------------
Using YAQL it is possible to use some predefined values in Mistral DSL. Using YAQL it is possible to use some predefined values in Mistral DSL.
@ -1062,8 +1066,10 @@ OpenStack context is available by **$.openstack**. It contains
Task result Task result
^^^^^^^^^^^ ^^^^^^^^^^^
Task result is available by **task(<task_name>).result**. It contains task result and directly depends on action output Task result is available by **task(<task_name>).result**. It contains task result
structure. Note that the *task(<task_name>)* function itself returns more than only task result. It returns the following fields of task executions: and directly depends on action output structure. Note that the *task(<task_name>)*
function itself returns more than only task result. It returns the following
fields of task executions:
* **id** - task execution UUID. * **id** - task execution UUID.
* **name** - task execution name. * **name** - task execution name.

48
doc/source/guides/configuration_guide.rst
View File

@ -4,15 +4,17 @@ Mistral Configuration Guide
Mistral configuration is needed for getting it work correctly Mistral configuration is needed for getting it work correctly
either with real OpenStack environment or without OpenStack environment. either with real OpenStack environment or without OpenStack environment.
**NOTE:** The most of the following operations should performed in mistral directory. **NOTE:** The most of the following operations should performed in mistral
directory.
1. Generate *mistral.conf* (if it does not already exist):: 1. Generate *mistral.conf* (if it does not already exist)::
oslo-config-generator --config-file tools/config/config-generator.mistral.conf --output-file etc/mistral.conf $ oslo-config-generator --config-file tools/config/config-generator.mistral.conf --output-file etc/mistral.conf
2. Edit file **etc/mistral.conf**. 2. Edit file **etc/mistral.conf**.
3. **If you are not using OpenStack, skip this item.** Provide valid keystone auth properties:: 3. **If you are not using OpenStack, skip this item.** Provide valid keystone
auth properties::
[keystone_authtoken] [keystone_authtoken]
auth_uri = http://<Keystone-host>:5000/v3 auth_uri = http://<Keystone-host>:5000/v3
@ -22,7 +24,8 @@ either with real OpenStack environment or without OpenStack environment.
admin_password = <password> admin_password = <password>
admin_tenant_name = <tenant> admin_tenant_name = <tenant>
4. If you want to configure SSL for Mistral API server, provide following options in config file:: 4. If you want to configure SSL for Mistral API server, provide following options
in config file::
[api] [api]
enable_ssl_api = True enable_ssl_api = True
@ -32,40 +35,46 @@ either with real OpenStack environment or without OpenStack environment.
cert_file = <path-to-certificate file> cert_file = <path-to-certificate file>
key_file = <path-to-key file> key_file = <path-to-key file>
5. **If you don't use OpenStack or you want to disable authentication for the Mistral service**, provide ``auth_enable = False`` in the config file:: 5. **If you don't use OpenStack or you want to disable authentication for the
Mistral service**, provide ``auth_enable = False`` in the config file::
[pecan] [pecan]
auth_enable = False auth_enable = False
6. **If you are not using OpenStack, skip this item**. Register Mistral service and Mistral endpoints on Keystone:: 6. **If you are not using OpenStack, skip this item**. Register Mistral service
and Mistral endpoints on Keystone::
$ MISTRAL_URL="http://[host]:[port]/v2" $ MISTRAL_URL="http://[host]:[port]/v2"
$ openstack service create workflow --name mistral --description 'OpenStack Workflow service' $ openstack service create workflow --name mistral --description 'OpenStack Workflow service'
$ openstack endpoint create workflow --publicurl $MISTRAL_URL --adminurl $MISTRAL_URL --internalurl $MISTRAL_URL $ openstack endpoint create workflow --publicurl $MISTRAL_URL --adminurl $MISTRAL_URL --internalurl $MISTRAL_URL
7. Configure transport properties in the corresponding config section: for RabbitMQ it is **oslo_messaging_rabbit**:: 7. Configure transport properties in the corresponding config section: for
RabbitMQ it is **oslo_messaging_rabbit**::
[oslo_messaging_rabbit] [oslo_messaging_rabbit]
rabbit_userid = <user_id> rabbit_userid = <user_id>
rabbit_password = <password> rabbit_password = <password>
rabbit_host = <host> rabbit_host = <host>
**NOTE**: Make sure that backend transport configuration is correct. Example for RabbitMQ:: **NOTE**: Make sure that backend transport configuration is correct. Example for
RabbitMQ::
[DEFAULT] [DEFAULT]
rpc_backend = rabbit rpc_backend = rabbit
8. Configure database. **SQLite can't be used in production**. Use *MySQL* or *PostgreSQL* instead. Here are the steps how to connect *MySQL* DB to Mistral: 8. Configure database. **SQLite can't be used in production**. Use *MySQL* or
*PostgreSQL* instead. Here are the steps how to connect *MySQL* DB to Mistral:
Make sure you have installed **mysql-server** package on your Database machine (it can be your Mistral machine as well). Make sure you have installed **mysql-server** package on your Database machine
(it can be your Mistral machine as well).
Install MySQL driver for python:: Install MySQL driver for python::
pip install mysql-python $ pip install mysql-python
Create the database and grant privileges:: Create the database and grant privileges::
mysql -u root -p $ mysql -u root -p
CREATE DATABASE mistral; CREATE DATABASE mistral;
USE mistral USE mistral
@ -80,14 +89,21 @@ either with real OpenStack environment or without OpenStack environment.
connection = postgresql://<user>:<password>@<database-host>:5432/mistral connection = postgresql://<user>:<password>@<database-host>:5432/mistral
9. **If you are not using OpenStack, skip this item.** Update mistral/actions/openstack/mapping.json file which contains all allowed OpenStack actions, according to the specific client versions of OpenStack projects in your deployment. Please find more detailed information in tools/get_action_list.py script. 9. **If you are not using OpenStack, skip this item.**
Update mistral/actions/openstack/mapping.json file which contains all allowed
OpenStack actions, according to the specific client versions of OpenStack
projects in your deployment. Please find more detailed information in
tools/get_action_list.py script.
10. Configure Task affinity feature if needed. It is needed for distinguishing either single task executor or one task executor from group of task executors:: 10. Configure Task affinity feature if needed. It is needed for distinguishing
either single task executor or one task executor from group of task executors::
[executor] [executor]
host = my_favorite_executor host = my_favorite_executor
Then, this executor can be referred in DSL by:: Then, this executor can be referred in DSL by
.. code-block:: yaml
...Workflow YAML... ...Workflow YAML...
my_task: my_task:
@ -104,5 +120,5 @@ either with real OpenStack environment or without OpenStack environment.
12. After that try to run mistral engine and see it is running without any error:: 12. After that try to run mistral engine and see it is running without any error::
mistral-server --config-file <path-to-config> --server engine $ mistral-server --config-file <path-to-config> --server engine

12
doc/source/guides/dashboard_guide.rst
View File

@ -31,7 +31,9 @@ The following should get you started:
$ cp -b <mistral-dashboard>/_50_mistral.py.example <horizon>/openstack_dashboard/local/enabled/_50_mistral.py $ cp -b <mistral-dashboard>/_50_mistral.py.example <horizon>/openstack_dashboard/local/enabled/_50_mistral.py
3. Since Mistral only supports Identity v3, you must ensure that the dashboard points the proper OPENSTACK_KEYSTONE_URL in <horizon>/openstack_dashboard/local/local_settings.py file:: 3. Since Mistral only supports Identity v3, you must ensure that the dashboard
points the proper OPENSTACK_KEYSTONE_URL in
<horizon>/openstack_dashboard/local/local_settings.py file::
OPENSTACK_API_VERSIONS = { OPENSTACK_API_VERSIONS = {
"identity": 3, "identity": 3,
@ -39,7 +41,9 @@ The following should get you started:
OPENSTACK_KEYSTONE_URL = "http://%s:5000/v3" % OPENSTACK_HOST OPENSTACK_KEYSTONE_URL = "http://%s:5000/v3" % OPENSTACK_HOST
4. Also, make sure you have changed OPENSTACK_HOST to point to your Keystone server and check all endpoints are accessible. You may want to change OPENSTACK_ENDPOINT_TYPE to "publicURL" if some of them are not. 4. Also, make sure you have changed OPENSTACK_HOST to point to your Keystone
server and check all endpoints are accessible. You may want to change
OPENSTACK_ENDPOINT_TYPE to "publicURL" if some of them are not.
5. When you're ready, you would need to either restart your apache:: 5. When you're ready, you would need to either restart your apache::
@ -50,6 +54,6 @@ The following should get you started:
$ cd ../horizon/ $ cd ../horizon/
$ tox -evenv -- python manage.py runserver $ tox -evenv -- python manage.py runserver
Debug Instructions Debug instructions
------------------ ------------------
Please refer to :doc:`Mistral Troubleshooting <../developer/troubleshooting>` Please refer to :doc:`Mistral Troubleshooting <../developer/troubleshooting>`

38
doc/source/guides/upgrade_guide.rst
View File

@ -1,75 +1,75 @@
Mistral Upgrade Guide Mistral Upgrade Guide
===================== =====================
Database Upgrade Database upgrade
---------------- ----------------
The migrations in ``alembic_migrations/versions`` contain the changes needed to migrate The migrations in ``alembic_migrations/versions`` contain the changes needed to
between Mistral database revisions. A migration occurs by executing a script that migrate between Mistral database revisions. A migration occurs by executing a
details the changes needed to upgrade the database. The migration scripts script that details the changes needed to upgrade the database. The migration
are ordered so that multiple scripts can run sequentially. The scripts are executed by scripts are ordered so that multiple scripts can run sequentially. The scripts are
Mistral's migration wrapper which uses the Alembic library to manage the migration. Mistral executed by Mistral's migration wrapper which uses the Alembic library to manage
supports migration from Kilo or later. the migration. Mistral supports migration from Kilo or later.
You can upgrade to the latest database version via: You can upgrade to the latest database version via:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf upgrade head $ mistral-db-manage --config-file /path/to/mistral.conf upgrade head
You can populate the database with standard actions and workflows: You can populate the database with standard actions and workflows:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf populate $ mistral-db-manage --config-file /path/to/mistral.conf populate
To check the current database version: To check the current database version:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf current $ mistral-db-manage --config-file /path/to/mistral.conf current
To create a script to run the migration offline: To create a script to run the migration offline:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf upgrade head --sql $ mistral-db-manage --config-file /path/to/mistral.conf upgrade head --sql
To run the offline migration between specific migration versions: To run the offline migration between specific migration versions:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf upgrade <start version>:<end version> --sql $ mistral-db-manage --config-file /path/to/mistral.conf upgrade <start version>:<end version> --sql
Upgrade the database incrementally: Upgrade the database incrementally:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf upgrade --delta <# of revs> $ mistral-db-manage --config-file /path/to/mistral.conf upgrade --delta <# of revs>
Or, upgrade the database to one newer revision: Or, upgrade the database to one newer revision:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf upgrade +1 $ mistral-db-manage --config-file /path/to/mistral.conf upgrade +1
Create new revision: Create new revision:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf revision -m "description of revision" --autogenerate $ mistral-db-manage --config-file /path/to/mistral.conf revision -m "description of revision" --autogenerate
Create a blank file: Create a blank file:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf revision -m "description of revision" $ mistral-db-manage --config-file /path/to/mistral.conf revision -m "description of revision"
This command does not perform any migrations, it only sets the revision. This command does not perform any migrations, it only sets the revision.
Revision may be any existing revision. Use this command carefully. Revision may be any existing revision. Use this command carefully.
:: ::
mistral-db-manage --config-file /path/to/mistral.conf stamp <revision> $ mistral-db-manage --config-file /path/to/mistral.conf stamp <revision>
To verify that the timeline does branch, you can run this command: To verify that the timeline does branch, you can run this command:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf check_migration $ mistral-db-manage --config-file /path/to/mistral.conf check_migration
If the migration path has branch, you can find the branch point via: If the migration path has branch, you can find the branch point via:
:: ::
mistral-db-manage --config-file /path/to/mistral.conf history $ mistral-db-manage --config-file /path/to/mistral.conf history

2
doc/source/guides/using_yaql.rst
View File

@ -1,4 +1,4 @@
How to use YAQL in Mistral How to use YAQL in Mistral
========================== ==========================
TBD TBD

139
doc/source/main_features.rst
View File

@ -2,20 +2,22 @@ Mistral Main Features
===================== =====================
Task result / Data Flow Task result / Data flow
----------------------- -----------------------
Mistral supports transferring data from one task to another. In other words, if *taskA* produces a value then Mistral supports transferring data from one task to another. In other words,
*taskB* which follows *taskA* can use it. In order to use this data Mistral relies on a query language called if *taskA* produces a value then *taskB* which follows *taskA* can use it.
`YAQL <https://github.com/openstack/yaql>`_. YAQL is a powerful yet simple tool In order to use this data Mistral relies on a query language called `YAQL <https://github.com/openstack/yaql>`_.
that allows the user to filter information, YAQL is a powerful yet simple tool that allows the user to filter information,
transform data and call functions. Find more information about it in the transform data and call functions. Find more information about it in the
`YAQL official documentation <http://yaql.readthedocs.org>`_ . This mechanism for transferring data plays a `YAQL official documentation <http://yaql.readthedocs.org>`_ . This mechanism
central role in the workflow concept and is referred to as Data Flow. for transferring data plays a central role in the workflow concept and is
referred to as Data Flow.
Below is a simple example of how Mistral Data Flow looks like from the DSL (workflow language) perspective: Below is a simple example of how Mistral Data Flow looks like from the DSL
(workflow language) perspective:
:: .. code-block:: yaml
version: '2.0' version: '2.0'
@ -35,10 +37,11 @@ Below is a simple example of how Mistral Data Flow looks like from the DSL (work
task2: task2:
action: do_something data=<% task(task1).result %> action: do_something data=<% task(task1).result %>
The task called "task1" produces a result that contains a list of the files in a user's home folder on a host The task called "task1" produces a result that contains a list of the files in
(both username and host are provided as workflow input) and the task "task2" uses this data using the YAQL a user's home folder on a host(both username and host are provided as workflow
expression "task(task1).result". input) and the task "task2" uses this data using the YAQLexpression
"task()" here is a function registered in YAQL by Mistral to get information about a task by its name. "task(task1).result". "task()" here is a function registered in YAQL by
Mistral to get information about a task by its name.
Task affinity Task affinity
------------- -------------
@ -49,8 +52,8 @@ tasks on specific Mistral executors. In fact, there are 2 cases:
1. You need to execute the task on a single executor. 1. You need to execute the task on a single executor.
2. You need to execute the task on any executor within a named group. 2. You need to execute the task on any executor within a named group.
To enable the task affinity feature, edit the "host" property in the "executor" section of the configuration To enable the task affinity feature, edit the "host" property in the
file:: "executor" section of the configuration file::
[executor] [executor]
host = my_favorite_executor host = my_favorite_executor
@ -67,13 +70,14 @@ this executor in Workflow DSL::
Task policies Task policies
------------- -------------
Any Mistral task regardless of its workflow type can optionally have configured policies. Any Mistral task regardless of its workflow type can optionally have
Policies control the flow of the task - for example, a policy can delay task execution before the task starts configured policies. Policies control the flow of the task - for example,
or after the task completes. a policy can delay task execution before the task starts or after the task
completes.
YAML example YAML example
^^^^^^^^^^^^ ^^^^^^^^^^^^
:: .. code-block:: yaml
my_task: my_task:
action: my_action action: my_action
@ -90,19 +94,24 @@ There are different types of policies in Mistral.
1. **pause-before** 1. **pause-before**
Specifies whether Mistral Engine should put the workflow on pause or not before starting a task. Specifies whether Mistral Engine should put the workflow on pause or not
before starting a task.
2. **wait-before** 2. **wait-before**
Specifies a delay in seconds that Mistral Engine should wait before starting a task. Specifies a delay in seconds that Mistral Engine should wait before starting
a task.
3. **wait-after** 3. **wait-after**
Specifies a delay in seconds that Mistral Engine should wait after a task has completed before starting the tasks specified in *'on-success'*, *'on-error'* or *'on-complete'*. Specifies a delay in seconds that Mistral Engine should wait after a task
has completed before starting the tasks specified in *'on-success'*,
*'on-error'* or *'on-complete'*.
4. **timeout** 4. **timeout**
Specifies a period of time in seconds after which a task will be failed automatically by the engine if it hasn't completed. Specifies a period of time in seconds after which a task will be failed
automatically by the engine if it hasn't completed.
5. **retry** 5. **retry**
@ -110,29 +119,34 @@ There are different types of policies in Mistral.
* *count* - Specifies a maximum number of times that a task can be repeated. * *count* - Specifies a maximum number of times that a task can be repeated.
* *delay* - Specifies a delay in seconds between subsequent task iterations. * *delay* - Specifies a delay in seconds between subsequent task iterations.
* *break-on* - Specifies a YAQL expression that will break the iteration loop if it evaluates to *'true'*. If * *break-on* - Specifies a YAQL expression that will break the iteration loop
it fires then the task is considered to have experienced an error. if it evaluates to *'true'*. If it fires then the task is considered to
* *continue-on* - Specifies a YAQL expression that will continue the iteration loop if it evaluates to have experienced an error.
*'true'*. If it fires then the task is considered successful. * *continue-on* - Specifies a YAQL expression that will continue the iteration
loop if it evaluates to *'true'*. If it fires then the task is considered
successful.
A retry policy can also be configured on a single line, as follows:: A retry policy can also be configured on a single line, as follows
.. code-block:: yaml
task1: task1:
action: my_action action: my_action
retry: count=10 delay=5 break-on=<% $.foo = 'bar' %> retry: count=10 delay=5 break-on=<% $.foo = 'bar' %>
All parameter values for any policy can be defined as YAQL expressions. All parameter values for any policy can be defined as YAQL expressions.
Join Join
---- ----
Join flow control allows to synchronize multiple parallel workflow branches and aggregate their data. Join flow control allows to synchronize multiple parallel workflow branches
and aggregate their data.
**Full Join (join: all)**. **Full join (join: all)**.
YAML example YAML example
^^^^^^^^^^^^ ^^^^^^^^^^^^
:: .. code-block:: yaml
register_vm_in_load_balancer: register_vm_in_load_balancer:
... ...
@ -153,17 +167,17 @@ YAML example
join: all join: all
action: send_email action: send_email
When a task has property *"join"* assigned with value *"all"* the task will run only When a task has property *"join"* assigned with value *"all"* the task will
if all upstream tasks (ones that lead to this task) are completed and corresponding run only if all upstream tasks (ones that lead to this task) are completed
conditions have triggered. Task A is considered an upstream task of Task B if Task A and corresponding conditions have triggered. Task A is considered an upstream
has Task B mentioned in any of its *"on-success"*, *"on-error"* and *"on-complete"* clauses task of Task B if Task A has Task B mentioned in any of its *"on-success"*,
regardless of YAQL guard expressions. *"on-error"* and *"on-complete"* clauses regardless of YAQL guard expressions.
**Partial Join (join: 2)** **Partial join (join: 2)**
YAML example YAML example
^^^^^^^^^^^^ ^^^^^^^^^^^^
:: .. code-block:: yaml
register_vm_in_load_balancer: register_vm_in_load_balancer:
... ...
@ -184,10 +198,11 @@ YAML example
join: 2 join: 2
action: send_email action: send_email
When a task has a numeric value assigned to the property *"join"*, then the task When a task has a numeric value assigned to the property *"join"*, then the
will run once at least this number of upstream tasks are completed and the task will run once at least this number of upstream tasks are completed and
corresponding conditions have triggered. In the example above, the task the corresponding conditions have triggered. In the example above, the task
"wait_for_two_registrations" will run if two any of the "register_vm_xxx" tasks are complete. "wait_for_two_registrations" will run if two any of the "register_vm_xxx"
tasks are complete.
**Discriminator (join: one)** **Discriminator (join: one)**
@ -196,12 +211,12 @@ In this case instead of 1 it is possible to specify the special string value *"o
which is introduced for symmetry with *"all"*. However, it's up to the user whether to use *"1"* or *"one"*. which is introduced for symmetry with *"all"*. However, it's up to the user whether to use *"1"* or *"one"*.
Processing Collections (with-items) Processing collections (with-items)
----------------------------------- -----------------------------------
YAML example YAML example
^^^^^^^^^^^^ ^^^^^^^^^^^^
:: .. code-block:: yaml
--- ---
version: '2.0' version: '2.0'
@ -231,14 +246,17 @@ YAML example
delay: 5 delay: 5
count: <% $.vm_names.len() * 10 %> count: <% $.vm_names.len() * 10 %>
The workflow *"create_vms"* in this example creates as many virtual servers as we The workflow *"create_vms"* in this example creates as many virtual servers
provide in the *"vm_names"* input parameter. E.g., if we specify *vm_names=["vm1", "vm2"]* as we provide in the *"vm_names"* input parameter. E.g., if we specify
then it'll create servers with these names based on the same image and flavor. *vm_names=["vm1", "vm2"]* then it'll create servers with these names based on
This is possible because we are using the *"with-items"* keyword that associates an action the same image and flavor. This is possible because we are using the *"with-items"*
or a workflow with a task run multiple times. The value of the *"with-items"* keyword that associates an action or a workflow with a task run multiple times.
task property contains an expression in the form: **<variable_name> in <% YAQL_expression %>**. The value of the *"with-items"* task property contains an expression in the
form: **<variable_name> in <% YAQL_expression %>**.
The most common form is:: The most common form is
.. code-block:: yaml
with-items: with-items:
- var1 in <% YAQL_expression_1 %> - var1 in <% YAQL_expression_1 %>
@ -248,16 +266,17 @@ The most common form is::
where collections expressed as YAQL_expression_1, YAQL_expression_2, where collections expressed as YAQL_expression_1, YAQL_expression_2,
YAQL_expression_N must have equal sizes. When a task gets started Mistral YAQL_expression_N must have equal sizes. When a task gets started Mistral
will iterate over all collections in parallel, i.e. the number of iterations will will iterate over all collections in parallel, i.e. the number of iterations
be equal to the length of any of the collections. will be equal to the length of any of the collections.
Note that in the *"with-items"* case, the task result (accessible in workflow Note that in the *"with-items"* case, the task result (accessible in workflow
context as <% $.task_name %>) will be a list containing results of corresponding context as <% $.task_name %>) will be a list containing results of
action/workflow calls. If at least one action/workflow call has failed then corresponding action/workflow calls. If at least one action/workflow call has
the whole task will get into *ERROR* state. It's also possible to apply retry failed then the whole task will get into *ERROR* state. It's also possible to
policy for tasks with a *"with-items"* property. In this case the retry policy will apply retry policy for tasks with a *"with-items"* property. In this case the
relaunch all action/workflow calls according to the *"with-items"* retry policy will relaunch all action/workflow calls according to the
configuration. Other policies can also be used in the same way as with regular non-*"with-items"* tasks. *"with-items"* configuration. Other policies can also be used in the same way
as with regular non-*"with-items"* tasks.
Execution expiration policy Execution expiration policy
--------------------------- ---------------------------

12
doc/source/quickstart.rst
View File

@ -7,17 +7,17 @@ Prerequisites
Before you start following this guide, make sure you have completed these Before you start following this guide, make sure you have completed these
three prerequisites. three prerequisites.
Install and Run Mistral Install and run Mistral
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
Go through the installation manual: :doc:`Mistral Installation Guide </guides/installation_guide>` Go through the installation manual: :doc:`Mistral Installation Guide </guides/installation_guide>`
Install Mistral Client Install Mistral client
~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
To install mistralclient, please refer to :doc:`Mistral Client / CLI Guide </guides/mistralclient_guide>` To install mistralclient, please refer to :doc:`Mistral Client / CLI Guide </guides/mistralclient_guide>`
Export Keystone Credentials Export Keystone credentials
~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
To use the OpenStack command line tools you should specify environment To use the OpenStack command line tools you should specify environment
@ -32,7 +32,7 @@ with a user ``admin`` in the ``admin`` tenant whose password is ``password``:
$ export OS_USERNAME=admin $ export OS_USERNAME=admin
$ export OS_PASSWORD=password $ export OS_PASSWORD=password
Write a Workflow Write a workflow
---------------- ----------------
For example, we have the following workflow. For example, we have the following workflow.
@ -64,7 +64,7 @@ then stores the word "Done" as a result of the second task (`task2`).
To learn more about the Mistral Workflows and what you can do, read the To learn more about the Mistral Workflows and what you can do, read the
:doc:`Mistral DSL specification </dsl/dsl_v2>` :doc:`Mistral DSL specification </dsl/dsl_v2>`
Upload the Workflow Upload the workflow
------------------- -------------------
Use the *Mistral CLI* to create the workflow:: Use the *Mistral CLI* to create the workflow::
@ -80,7 +80,7 @@ The output should look similar to this::
+-------------+--------+---------+---------------------+------------+ +-------------+--------+---------+---------------------+------------+
Run the Workflow and Check the Result Run the workflow and check the result
------------------------------------- -------------------------------------
Use the *Mistral CLI* to start the new workflow, passing in a list of names Use the *Mistral CLI* to start the new workflow, passing in a list of names