Update folder structure

* Added a folder for Pike
* Moved implemented specs for Ocata into "implemented"

Change-Id: I730ea9150137488a414c619a1b4c81a87b7c5be8
This commit is contained in:
Renat Akhmerov 2017-03-03 18:24:02 +07:00
parent 9f90f009d8
commit 4dd49460ac
4 changed files with 528 additions and 0 deletions

View File

@ -0,0 +1,268 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================
Custom Actions API
==================
Launchpad blueprint:
https://blueprints.launchpad.net/mistral/+spec/mistral-custom-actions-api
This specification sets a formal basis for those Mistral users who want to
create their own actions and make them available to use as part of Mistral
workflows. The number one question that the spec addresses is "What is
available in Mistral code base in order to implement custom actions?"
Problem description
===================
Custom actions are now possible to create and it's as simple as just
implementing a class inherited from mistral.actions.base.Action that
has 3 methods:
* run() - executes main action logic, **mandatory** to implement
* test() - execute action in test mode, related to future dry-run
functionality, optional to implement
* is_sync() - must return **True** if action returns its result right from
method run() or **False** if method run() only starts action logic and
result is supposed to be delivered later via public Mistral API
There's also a mechanism based on stevedore library that allows to plug in
new actions via adding new entry points in setup.cfg file.
If a custom action doesn't require any integration neither with Mistral
nor with OpenStack this is enough to know in order to implement it.
However, if this action needs to leverage more advanced capabilities
provided by Mistral and OpenStack then Action class itself doesn't
give any knowledge about means that can be used to achieve that.
A simple example of integration with OpenStack infrastructure is the need
to call endpoints of OpenStack services. In this case, at minimum, action
needs to be able to authenticate with Keystone, i.e., have access to
Mistral security context.
Use Cases
---------
Simple OpenStack actions
^^^^^^^^^^^^^^^^^^^^^^^^
As a user of Mistral I want to create actions that call OpenStack services.
In this case action needs to be able to access Mistral security context
that contains auth token to be able to pass it to a corresponding service.
Note: This use case is generally implemented within Mistral but it needs
to be rethought since OpenStack actions that are implemented now in Mistral
use Mistral Python code that is not assumed to be a public API and hence
stable.
Complex OpenStack actions
^^^^^^^^^^^^^^^^^^^^^^^^^
As a user of Mistral I want to create actions that call multiple OpenStack
services from within one action.
For example, we may want to create action
"create_cinder_volume_and_attach_to_vm" that creates a Cinder volume and
attaches it to a virtual instance. In this case action needs to have access
to Mistral security context that contains auth token so that it can pass
that token to Cinder and Nova.
Reusing existing actions
^^^^^^^^^^^^^^^^^^^^^^^^
As a user of Mistral I want to be able to reuse existing actions while
implementing my new actions so that I don't have to reimplement similar
functionality.
For example, I want to create action that checks if a certain virtual
instance exists in the tenant by calling Nova and if it does the action
runs a number of secure shell commands to configure it. In this scenario,
we need to call Nova and do ssh. Both already exist in Mistral as actions
"nova.servers_get" and "std.ssh". So there should be a mechanism allowing
to reuse those actions while creating a new more complex action.
Proposed change
===============
General idea
------------
We need to have one or more Python packages in Mistral that are designed
and documented as a public Python API for developers that want to create
custom actions. These packages should effectively provide a number of
classes that can be used directly or inherited as needed. They should
cover the following aspects of action development:
* Base class or a number of classes that can be extended in order to build
new Mistral actions. Currently existing **mistral.actions.base.Action**
is an example of such class.
* Module that provides access to security context associated with the
current workflow that this action belongs to. Security context should
at least include user, project/tenant, auth token.
* Module that provides access to current Mistral execution context. That
context should include:
* Current workflow execution id
* Current task execution id
* Current action execution id
* Package with most frequently used utils and data types used during
custom actions development. For example, class
mistral.workflow.utils.Result that now exists in the code base is
needed by actions but it's not clear that it's part of Python API.
* Module that allows to get and reuse existing actions
Since these Python entities must be available for both engine and
executor they should be moved to a separate subproject of Mistral, for
example, **mistral-actions-api**.
Existing OpenStack actions should be moved out of mistral project into
a different Mistral subproject. The proposal is to use **mistral-extra**
repo for this purpose because although we use it only for collecting
Mistral examples its initial idea was also to have additional tools
and extensions in it.
Specific entities
-----------------
mistral.actions.api
^^^^^^^^^^^^^^^^^^^
Main Python package that contains all modules and classes which are part
of Custom Actions API.
mistral.actions.api.base
^^^^^^^^^^^^^^^^^^^^^^^^
Python module that contains base classes for custom actions. Currently
module **mistral.actions.base** performs similar function.
Note: Specific content of this module is out of scope of this spec and
must be defined at implementation stage.
mistral.actions.api.security
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Python module that contains required functions to get all required
information related to current OpenStack security context. At minimum:
user, project, auth token.
Note: Specific content of this module is out of scope of this spec and
must be defined at implementation stage.
mistral.actions.api.types
^^^^^^^^^^^^^^^^^^^^^^^^^
Python module that contains all data types that custom actions need to
use. One candidate to go to that module that now exists is
**mistral.workflow.utils.Result**.
Note: Specific content of this module is out of scope of this spec and
defined at implementation stage.
mistral.actions.api.utils
^^^^^^^^^^^^^^^^^^^^^^^^^
Python module that contains additional functions helpful for creating
new Mistral actions. At minimum: functions to get instances of existing
actions so that action developers could re-use functionality of existing
actions. Return type for these actions though must be rather a wrapper
that doesn't just call **Action.run()** method but instead uses Mistral
action execution machinery to actually call action just like as if it
was called as part of workflow (taking care of data transformations,
fulfilling security and execution context etc.)
Note: Specific content of this module is out of scope of this spec and
must be defined at implementation stage.
Alternatives
------------
None.
Data model impact
-----------------
None.
REST API impact
---------------
None.
End user impact
---------------
REST API users
^^^^^^^^^^^^^^
No impact.
Custom actions developers
^^^^^^^^^^^^^^^^^^^^^^^^^
Having to use Custom Actions API described in this spec whereas now they
can only use **mistral.actions.base** safely.
Performance Impact
------------------
No significant impact is expected. Minor is possible.
Deployer impact
---------------
Deployers will need to make sure to install a new library containing
Custom Action API packages, modules and classes. However, this impact
is not supposed to be severe because all dependencies must be handled
smoothly by Pip.
In case if there's an existing Mistral installation with installed
actions, some DB migration might be required. Changes in DB schema are
not expected though. If so, Mistral project should provide convenient
tools to help make this transition to using new actions.
Implementation
==============
Assignee(s)
-----------
To be found based on discussions around the spec.
Work Items
----------
* Create a new repo containing the code of Custom Actions API (e.g.
**mistral-lib** or **mistral-common**, particular name is to be defined)
* Design and implement modules listed in Specific Entities section
* Provide deprecation mechanism so that during some period of time it
would be possible to use the old approach for implementing Mistral
actions (with **mistral.actions.base**) and the new one
* Fix existing action implementations so that they use new API
* Fix Mistral Executor accordingly
* Fix Mistral Engine accordingly
* Revisit and restructure repo **mistral-extra**
* Move existing OpenStack actions into **mistral-extra**
Dependencies
============
No additional dependencies are required.
Testing
=======
Custom Actions API can be tested on devstack based OpenStack CI gates
such as gate-mistral-devstack-dsvm by creating and running custom
actions that use this API.
References
==========
Initial patch for TripleO/Mistral integration:
https://review.openstack.org/#/c/282366/

View File

@ -0,0 +1,260 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=======================
Workflow Global Context
=======================
Launchpad blueprint:
https://blueprints.launchpad.net/mistral/+spec/mistral-global-wf-context
Workflow global context will allow to store variables not associated with
particular workflow branches.
Problem description
===================
Currently 'publish' keyword in Mistral saves variables into a storage
(context) which is associated only with a branch.
Example:
::
---
version: '2.0'
wf:
tasks:
A:
action: std.noop
publish:
my_var: 1
on-success: A1
A1:
action: my_action param1=<% $.my_var %>
B:
action: std.noop
publish:
my_var: 2
on-success: B1
B1:
action: my_action param1=<% $.my_var %>
The expression "$.my_var" in the declaration of A1 will always evaluate to 1,
for B1 it will always evaluate to 2. This doesn't depend on the order in which
A and B will run. This is because we have two branches (A -> A1 and B -> B1)
for which the variable "my_var" has its own different version.
Sometimes though we need to be able to share data across branches which is now
impossible due to aforementioned semantics.
The concept of workflow global context can help solve this problem. The word
"global" here means "accessible from any workflow branch".
We also need an ability to make atomic updates of global workflow context.
It's necessary when we, for example, want to create a global counter (e.g.
counter of network calls to external systems performed by a workflow).
Use Cases
---------
* Building conditions based on events happened in parallel workflow branches.
Example: one branch needs to notify the other one that it should stop.
* Passing data between branches. Example: one branch needs to wait till the
other one produces some expected result. This is, essentially, creating
a cross-branch mutex.
* Counters that need to decrement or increment atomically.
Proposed change
===============
In order to achieve this goal the proposal is:
* Add the new keyword "publish-global" which is similar to "publish"
with the difference that it publishes variables into workflow global
context instead of branch workflow context. It's important to note
that this is an unprotected way of modifying data because race
conditions are possible when writing different values for same
variables in the global context from parallel branches. In other
words, if we have branches A and B and there are tasks in these
branches writing different values to the variable X in the global
context Mistral won't provide any guarantees as far as what value
is going to be assigned to X and what value will be lost. Users need
to understand possible consequences.
For instance, using this keyword it's impossible to create an atomic
counter since it doesn't assume acquiring a lock under which we can
safely perform multiple operations (e.g. read and then write).
However, for many scenarios even this model can be useful. For example,
if there's only one branch writing values and others are only readers.
* Add the new YAQL/Jinja function "global()" to explicitly access
variables in workflow global context.
* Make global variables also accessible using "$." in YAQL and "_." in
Jinja in a way that branch variables can shadow them if they are
published in the current branch.
* Add the new keyword "publish-global-atomic" which is similar to
"publish-global" but allows to atomically read and write variables
in workflow global context by acquiring a temporary lock on it.
Unlike 'publish-global' this will allow to create atomic counters
when we need to perform multiple operations against the storage
atomically.
Example #1 (writing and reading global variables):
::
---
version: '2.0'
wf:
tasks:
A:
action: std.noop
publish:
my_var: "branch value"
publish-global:
my_var: "global value"
on-success: A1
A1:
# $.my_var will always evaluate to "branch value" because A1 belongs
# to the same branch as A and runs after A. When using "$" to access
# context variables branch values have higher priority.
# In order to access global context reliably we need to use YAQL/Jinja
# function 'global'. So global(my_var) will always evaluate to
# 'global value'.
action: my_action1 param1=<% $.my_var %> param2=<% global(my_var) %>
B:
# $.my_var will evaluate to "global value" if task A completes
# before task B and "null", if not. It's because A and B are
# parallel and 'publish' in A doesn't apply to B, only
# 'publish-global' does. In this example global(my_var) has the same
# meaning as $.my_var because there's no ambiguity from what context
# we should take variable 'my_var'.
action: my_action2 param1=<% $.my_var %> param2=<% global(my_var) %>
Example #2 (writing global variables atomically):
::
---
version: '2.0'
vars:
- my_global_var: 0
wf:
tasks:
task1:
action: std.noop
publish-global-atomic:
counter: <% global(my_global_var) + 1 %>
task2:
action: std.noop
publish-global-atomic:
counter: <% global(my_global_var) + 1 %>
Alternatives
------------
None.
Data model impact
-----------------
Workflow execution object already has the field "context" which is now
immutable and initialized with openstack specific data, execution id and
environment variables. In order to get the full context for evaluating a
YAQL/Jinja expression in a task declaration we always build a context view
merged from workflow input, workflow execution "context" field and branch
specific context (e.g. task inbound context when evaluating action
parameters). The field "context" can play the role of workflow global
context. However, the idea to reuse this field can be revisited during
the implementation phase.
REST API impact
---------------
None.
End user impact
---------------
New workflow language feature that allows to store global variables into
workflow context.
Performance Impact
------------------
When using "publish-global-atomic" we'll need to use locking in order
to prevent concurrent modifications of global workflow context while
reading and modifying it when processing a certain task. In fact, this is
equal to locking the whole execution object and hence will have a serious
performance impact in case of many parallel tasks. For this reason,
"publish-global-atomic" needs to be well documented and used with
precaution.
Deployer impact
---------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
rakhmerov
Other contributors:
melisha
Work Items
----------
* Add 'publish-global' and 'publish-global-atomic' into the direct workflow
specification.
* Make changes in Mistral engine to publish variables into global context
(preliminarily it will be the field 'context' of workflow execution object).
* Implement YAQL/Jinja function 'global' to explicitly read variables from
workflow global context.
* Add locking workflow global context (i.e. workflow execution) in case of
using 'publish-global-atomic'. A thread that acquires a lock must first
refresh state of workflow execution and then proceed with publishing etc.
Dependencies
============
None.
Testing
=======
* Unit tests for 'publish-global' keyword and 'global' function in different
cases: parallel branches, sequential branches.
* Unit tests to check that branch-local variables take precedence when
reading variables using '$.' in YAQL and '_.' in Jinja.
* Unit tests for 'publish-global-atomic' that checks atomicity of reads and
writes of global variables. Although unit tests can't fully test this
feature. In order to fully test it we need to have a test with multiple
Mistral engines to make sure we have concurrent access to workflow execution.
References
==========
None.