Update folder structure
* Added a folder for Pike * Moved implemented specs for Ocata into "implemented" Change-Id: I730ea9150137488a414c619a1b4c81a87b7c5be8
This commit is contained in:
parent
9f90f009d8
commit
4dd49460ac
|
@ -0,0 +1,268 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==================
|
||||
Custom Actions API
|
||||
==================
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/mistral/+spec/mistral-custom-actions-api
|
||||
|
||||
This specification sets a formal basis for those Mistral users who want to
|
||||
create their own actions and make them available to use as part of Mistral
|
||||
workflows. The number one question that the spec addresses is "What is
|
||||
available in Mistral code base in order to implement custom actions?"
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Custom actions are now possible to create and it's as simple as just
|
||||
implementing a class inherited from mistral.actions.base.Action that
|
||||
has 3 methods:
|
||||
|
||||
* run() - executes main action logic, **mandatory** to implement
|
||||
|
||||
* test() - execute action in test mode, related to future dry-run
|
||||
functionality, optional to implement
|
||||
|
||||
* is_sync() - must return **True** if action returns its result right from
|
||||
method run() or **False** if method run() only starts action logic and
|
||||
result is supposed to be delivered later via public Mistral API
|
||||
|
||||
There's also a mechanism based on stevedore library that allows to plug in
|
||||
new actions via adding new entry points in setup.cfg file.
|
||||
|
||||
If a custom action doesn't require any integration neither with Mistral
|
||||
nor with OpenStack this is enough to know in order to implement it.
|
||||
|
||||
However, if this action needs to leverage more advanced capabilities
|
||||
provided by Mistral and OpenStack then Action class itself doesn't
|
||||
give any knowledge about means that can be used to achieve that.
|
||||
A simple example of integration with OpenStack infrastructure is the need
|
||||
to call endpoints of OpenStack services. In this case, at minimum, action
|
||||
needs to be able to authenticate with Keystone, i.e., have access to
|
||||
Mistral security context.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
Simple OpenStack actions
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
As a user of Mistral I want to create actions that call OpenStack services.
|
||||
In this case action needs to be able to access Mistral security context
|
||||
that contains auth token to be able to pass it to a corresponding service.
|
||||
Note: This use case is generally implemented within Mistral but it needs
|
||||
to be rethought since OpenStack actions that are implemented now in Mistral
|
||||
use Mistral Python code that is not assumed to be a public API and hence
|
||||
stable.
|
||||
|
||||
Complex OpenStack actions
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
As a user of Mistral I want to create actions that call multiple OpenStack
|
||||
services from within one action.
|
||||
|
||||
For example, we may want to create action
|
||||
"create_cinder_volume_and_attach_to_vm" that creates a Cinder volume and
|
||||
attaches it to a virtual instance. In this case action needs to have access
|
||||
to Mistral security context that contains auth token so that it can pass
|
||||
that token to Cinder and Nova.
|
||||
|
||||
Reusing existing actions
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
As a user of Mistral I want to be able to reuse existing actions while
|
||||
implementing my new actions so that I don't have to reimplement similar
|
||||
functionality.
|
||||
|
||||
For example, I want to create action that checks if a certain virtual
|
||||
instance exists in the tenant by calling Nova and if it does the action
|
||||
runs a number of secure shell commands to configure it. In this scenario,
|
||||
we need to call Nova and do ssh. Both already exist in Mistral as actions
|
||||
"nova.servers_get" and "std.ssh". So there should be a mechanism allowing
|
||||
to reuse those actions while creating a new more complex action.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
General idea
|
||||
------------
|
||||
|
||||
We need to have one or more Python packages in Mistral that are designed
|
||||
and documented as a public Python API for developers that want to create
|
||||
custom actions. These packages should effectively provide a number of
|
||||
classes that can be used directly or inherited as needed. They should
|
||||
cover the following aspects of action development:
|
||||
|
||||
* Base class or a number of classes that can be extended in order to build
|
||||
new Mistral actions. Currently existing **mistral.actions.base.Action**
|
||||
is an example of such class.
|
||||
|
||||
* Module that provides access to security context associated with the
|
||||
current workflow that this action belongs to. Security context should
|
||||
at least include user, project/tenant, auth token.
|
||||
|
||||
* Module that provides access to current Mistral execution context. That
|
||||
context should include:
|
||||
|
||||
* Current workflow execution id
|
||||
|
||||
* Current task execution id
|
||||
|
||||
* Current action execution id
|
||||
|
||||
* Package with most frequently used utils and data types used during
|
||||
custom actions development. For example, class
|
||||
mistral.workflow.utils.Result that now exists in the code base is
|
||||
needed by actions but it's not clear that it's part of Python API.
|
||||
|
||||
* Module that allows to get and reuse existing actions
|
||||
|
||||
Since these Python entities must be available for both engine and
|
||||
executor they should be moved to a separate subproject of Mistral, for
|
||||
example, **mistral-actions-api**.
|
||||
|
||||
Existing OpenStack actions should be moved out of mistral project into
|
||||
a different Mistral subproject. The proposal is to use **mistral-extra**
|
||||
repo for this purpose because although we use it only for collecting
|
||||
Mistral examples its initial idea was also to have additional tools
|
||||
and extensions in it.
|
||||
|
||||
Specific entities
|
||||
-----------------
|
||||
|
||||
mistral.actions.api
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
Main Python package that contains all modules and classes which are part
|
||||
of Custom Actions API.
|
||||
|
||||
mistral.actions.api.base
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Python module that contains base classes for custom actions. Currently
|
||||
module **mistral.actions.base** performs similar function.
|
||||
|
||||
Note: Specific content of this module is out of scope of this spec and
|
||||
must be defined at implementation stage.
|
||||
|
||||
mistral.actions.api.security
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Python module that contains required functions to get all required
|
||||
information related to current OpenStack security context. At minimum:
|
||||
user, project, auth token.
|
||||
|
||||
Note: Specific content of this module is out of scope of this spec and
|
||||
must be defined at implementation stage.
|
||||
|
||||
mistral.actions.api.types
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Python module that contains all data types that custom actions need to
|
||||
use. One candidate to go to that module that now exists is
|
||||
**mistral.workflow.utils.Result**.
|
||||
|
||||
Note: Specific content of this module is out of scope of this spec and
|
||||
defined at implementation stage.
|
||||
|
||||
mistral.actions.api.utils
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Python module that contains additional functions helpful for creating
|
||||
new Mistral actions. At minimum: functions to get instances of existing
|
||||
actions so that action developers could re-use functionality of existing
|
||||
actions. Return type for these actions though must be rather a wrapper
|
||||
that doesn't just call **Action.run()** method but instead uses Mistral
|
||||
action execution machinery to actually call action just like as if it
|
||||
was called as part of workflow (taking care of data transformations,
|
||||
fulfilling security and execution context etc.)
|
||||
|
||||
Note: Specific content of this module is out of scope of this spec and
|
||||
must be defined at implementation stage.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
None.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None.
|
||||
|
||||
End user impact
|
||||
---------------
|
||||
|
||||
REST API users
|
||||
^^^^^^^^^^^^^^
|
||||
No impact.
|
||||
|
||||
Custom actions developers
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Having to use Custom Actions API described in this spec whereas now they
|
||||
can only use **mistral.actions.base** safely.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
No significant impact is expected. Minor is possible.
|
||||
|
||||
Deployer impact
|
||||
---------------
|
||||
|
||||
Deployers will need to make sure to install a new library containing
|
||||
Custom Action API packages, modules and classes. However, this impact
|
||||
is not supposed to be severe because all dependencies must be handled
|
||||
smoothly by Pip.
|
||||
|
||||
In case if there's an existing Mistral installation with installed
|
||||
actions, some DB migration might be required. Changes in DB schema are
|
||||
not expected though. If so, Mistral project should provide convenient
|
||||
tools to help make this transition to using new actions.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
To be found based on discussions around the spec.
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Create a new repo containing the code of Custom Actions API (e.g.
|
||||
**mistral-lib** or **mistral-common**, particular name is to be defined)
|
||||
* Design and implement modules listed in Specific Entities section
|
||||
* Provide deprecation mechanism so that during some period of time it
|
||||
would be possible to use the old approach for implementing Mistral
|
||||
actions (with **mistral.actions.base**) and the new one
|
||||
* Fix existing action implementations so that they use new API
|
||||
* Fix Mistral Executor accordingly
|
||||
* Fix Mistral Engine accordingly
|
||||
* Revisit and restructure repo **mistral-extra**
|
||||
* Move existing OpenStack actions into **mistral-extra**
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
No additional dependencies are required.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Custom Actions API can be tested on devstack based OpenStack CI gates
|
||||
such as gate-mistral-devstack-dsvm by creating and running custom
|
||||
actions that use this API.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
Initial patch for TripleO/Mistral integration:
|
||||
https://review.openstack.org/#/c/282366/
|
|
@ -0,0 +1,260 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=======================
|
||||
Workflow Global Context
|
||||
=======================
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/mistral/+spec/mistral-global-wf-context
|
||||
|
||||
Workflow global context will allow to store variables not associated with
|
||||
particular workflow branches.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Currently 'publish' keyword in Mistral saves variables into a storage
|
||||
(context) which is associated only with a branch.
|
||||
|
||||
Example:
|
||||
|
||||
::
|
||||
|
||||
---
|
||||
version: '2.0'
|
||||
|
||||
wf:
|
||||
tasks:
|
||||
A:
|
||||
action: std.noop
|
||||
publish:
|
||||
my_var: 1
|
||||
on-success: A1
|
||||
|
||||
A1:
|
||||
action: my_action param1=<% $.my_var %>
|
||||
|
||||
B:
|
||||
action: std.noop
|
||||
publish:
|
||||
my_var: 2
|
||||
on-success: B1
|
||||
|
||||
B1:
|
||||
action: my_action param1=<% $.my_var %>
|
||||
|
||||
|
||||
The expression "$.my_var" in the declaration of A1 will always evaluate to 1,
|
||||
for B1 it will always evaluate to 2. This doesn't depend on the order in which
|
||||
A and B will run. This is because we have two branches (A -> A1 and B -> B1)
|
||||
for which the variable "my_var" has its own different version.
|
||||
|
||||
Sometimes though we need to be able to share data across branches which is now
|
||||
impossible due to aforementioned semantics.
|
||||
The concept of workflow global context can help solve this problem. The word
|
||||
"global" here means "accessible from any workflow branch".
|
||||
|
||||
We also need an ability to make atomic updates of global workflow context.
|
||||
It's necessary when we, for example, want to create a global counter (e.g.
|
||||
counter of network calls to external systems performed by a workflow).
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
* Building conditions based on events happened in parallel workflow branches.
|
||||
Example: one branch needs to notify the other one that it should stop.
|
||||
* Passing data between branches. Example: one branch needs to wait till the
|
||||
other one produces some expected result. This is, essentially, creating
|
||||
a cross-branch mutex.
|
||||
* Counters that need to decrement or increment atomically.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
In order to achieve this goal the proposal is:
|
||||
|
||||
* Add the new keyword "publish-global" which is similar to "publish"
|
||||
with the difference that it publishes variables into workflow global
|
||||
context instead of branch workflow context. It's important to note
|
||||
that this is an unprotected way of modifying data because race
|
||||
conditions are possible when writing different values for same
|
||||
variables in the global context from parallel branches. In other
|
||||
words, if we have branches A and B and there are tasks in these
|
||||
branches writing different values to the variable X in the global
|
||||
context Mistral won't provide any guarantees as far as what value
|
||||
is going to be assigned to X and what value will be lost. Users need
|
||||
to understand possible consequences.
|
||||
For instance, using this keyword it's impossible to create an atomic
|
||||
counter since it doesn't assume acquiring a lock under which we can
|
||||
safely perform multiple operations (e.g. read and then write).
|
||||
However, for many scenarios even this model can be useful. For example,
|
||||
if there's only one branch writing values and others are only readers.
|
||||
* Add the new YAQL/Jinja function "global()" to explicitly access
|
||||
variables in workflow global context.
|
||||
* Make global variables also accessible using "$." in YAQL and "_." in
|
||||
Jinja in a way that branch variables can shadow them if they are
|
||||
published in the current branch.
|
||||
* Add the new keyword "publish-global-atomic" which is similar to
|
||||
"publish-global" but allows to atomically read and write variables
|
||||
in workflow global context by acquiring a temporary lock on it.
|
||||
Unlike 'publish-global' this will allow to create atomic counters
|
||||
when we need to perform multiple operations against the storage
|
||||
atomically.
|
||||
|
||||
Example #1 (writing and reading global variables):
|
||||
|
||||
::
|
||||
|
||||
---
|
||||
version: '2.0'
|
||||
|
||||
wf:
|
||||
tasks:
|
||||
A:
|
||||
action: std.noop
|
||||
publish:
|
||||
my_var: "branch value"
|
||||
publish-global:
|
||||
my_var: "global value"
|
||||
on-success: A1
|
||||
|
||||
A1:
|
||||
# $.my_var will always evaluate to "branch value" because A1 belongs
|
||||
# to the same branch as A and runs after A. When using "$" to access
|
||||
# context variables branch values have higher priority.
|
||||
# In order to access global context reliably we need to use YAQL/Jinja
|
||||
# function 'global'. So global(my_var) will always evaluate to
|
||||
# 'global value'.
|
||||
action: my_action1 param1=<% $.my_var %> param2=<% global(my_var) %>
|
||||
|
||||
B:
|
||||
# $.my_var will evaluate to "global value" if task A completes
|
||||
# before task B and "null", if not. It's because A and B are
|
||||
# parallel and 'publish' in A doesn't apply to B, only
|
||||
# 'publish-global' does. In this example global(my_var) has the same
|
||||
# meaning as $.my_var because there's no ambiguity from what context
|
||||
# we should take variable 'my_var'.
|
||||
action: my_action2 param1=<% $.my_var %> param2=<% global(my_var) %>
|
||||
|
||||
|
||||
Example #2 (writing global variables atomically):
|
||||
|
||||
::
|
||||
|
||||
---
|
||||
version: '2.0'
|
||||
|
||||
vars:
|
||||
- my_global_var: 0
|
||||
|
||||
wf:
|
||||
tasks:
|
||||
task1:
|
||||
action: std.noop
|
||||
publish-global-atomic:
|
||||
counter: <% global(my_global_var) + 1 %>
|
||||
|
||||
task2:
|
||||
action: std.noop
|
||||
publish-global-atomic:
|
||||
counter: <% global(my_global_var) + 1 %>
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
None.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
Workflow execution object already has the field "context" which is now
|
||||
immutable and initialized with openstack specific data, execution id and
|
||||
environment variables. In order to get the full context for evaluating a
|
||||
YAQL/Jinja expression in a task declaration we always build a context view
|
||||
merged from workflow input, workflow execution "context" field and branch
|
||||
specific context (e.g. task inbound context when evaluating action
|
||||
parameters). The field "context" can play the role of workflow global
|
||||
context. However, the idea to reuse this field can be revisited during
|
||||
the implementation phase.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None.
|
||||
|
||||
End user impact
|
||||
---------------
|
||||
|
||||
New workflow language feature that allows to store global variables into
|
||||
workflow context.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
When using "publish-global-atomic" we'll need to use locking in order
|
||||
to prevent concurrent modifications of global workflow context while
|
||||
reading and modifying it when processing a certain task. In fact, this is
|
||||
equal to locking the whole execution object and hence will have a serious
|
||||
performance impact in case of many parallel tasks. For this reason,
|
||||
"publish-global-atomic" needs to be well documented and used with
|
||||
precaution.
|
||||
|
||||
Deployer impact
|
||||
---------------
|
||||
|
||||
None.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
rakhmerov
|
||||
|
||||
Other contributors:
|
||||
melisha
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Add 'publish-global' and 'publish-global-atomic' into the direct workflow
|
||||
specification.
|
||||
* Make changes in Mistral engine to publish variables into global context
|
||||
(preliminarily it will be the field 'context' of workflow execution object).
|
||||
* Implement YAQL/Jinja function 'global' to explicitly read variables from
|
||||
workflow global context.
|
||||
* Add locking workflow global context (i.e. workflow execution) in case of
|
||||
using 'publish-global-atomic'. A thread that acquires a lock must first
|
||||
refresh state of workflow execution and then proceed with publishing etc.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None.
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
* Unit tests for 'publish-global' keyword and 'global' function in different
|
||||
cases: parallel branches, sequential branches.
|
||||
* Unit tests to check that branch-local variables take precedence when
|
||||
reading variables using '$.' in YAQL and '_.' in Jinja.
|
||||
* Unit tests for 'publish-global-atomic' that checks atomicity of reads and
|
||||
writes of global variables. Although unit tests can't fully test this
|
||||
feature. In order to fully test it we need to have a test with multiple
|
||||
Mistral engines to make sure we have concurrent access to workflow execution.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
None.
|
Loading…
Reference in New Issue