diff --git a/specs/ocata/approved/publish-on-failure.rst b/specs/ocata/implemented/publish-on-failure.rst similarity index 100% rename from specs/ocata/approved/publish-on-failure.rst rename to specs/ocata/implemented/publish-on-failure.rst diff --git a/specs/ocata/approved/yaql-tasks-function.rst b/specs/ocata/implemented/yaql-tasks-function.rst similarity index 100% rename from specs/ocata/approved/yaql-tasks-function.rst rename to specs/ocata/implemented/yaql-tasks-function.rst diff --git a/specs/pike/approved/mistral-custom-actions-api.rst b/specs/pike/approved/mistral-custom-actions-api.rst new file mode 100644 index 0000000..01f4874 --- /dev/null +++ b/specs/pike/approved/mistral-custom-actions-api.rst @@ -0,0 +1,268 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +================== +Custom Actions API +================== + +Launchpad blueprint: + +https://blueprints.launchpad.net/mistral/+spec/mistral-custom-actions-api + +This specification sets a formal basis for those Mistral users who want to +create their own actions and make them available to use as part of Mistral +workflows. The number one question that the spec addresses is "What is +available in Mistral code base in order to implement custom actions?" + + +Problem description +=================== + +Custom actions are now possible to create and it's as simple as just +implementing a class inherited from mistral.actions.base.Action that +has 3 methods: + +* run() - executes main action logic, **mandatory** to implement + +* test() - execute action in test mode, related to future dry-run + functionality, optional to implement + +* is_sync() - must return **True** if action returns its result right from + method run() or **False** if method run() only starts action logic and + result is supposed to be delivered later via public Mistral API + +There's also a mechanism based on stevedore library that allows to plug in +new actions via adding new entry points in setup.cfg file. + +If a custom action doesn't require any integration neither with Mistral +nor with OpenStack this is enough to know in order to implement it. + +However, if this action needs to leverage more advanced capabilities +provided by Mistral and OpenStack then Action class itself doesn't +give any knowledge about means that can be used to achieve that. +A simple example of integration with OpenStack infrastructure is the need +to call endpoints of OpenStack services. In this case, at minimum, action +needs to be able to authenticate with Keystone, i.e., have access to +Mistral security context. + +Use Cases +--------- + +Simple OpenStack actions +^^^^^^^^^^^^^^^^^^^^^^^^ +As a user of Mistral I want to create actions that call OpenStack services. +In this case action needs to be able to access Mistral security context +that contains auth token to be able to pass it to a corresponding service. +Note: This use case is generally implemented within Mistral but it needs +to be rethought since OpenStack actions that are implemented now in Mistral +use Mistral Python code that is not assumed to be a public API and hence +stable. + +Complex OpenStack actions +^^^^^^^^^^^^^^^^^^^^^^^^^ +As a user of Mistral I want to create actions that call multiple OpenStack +services from within one action. + +For example, we may want to create action +"create_cinder_volume_and_attach_to_vm" that creates a Cinder volume and +attaches it to a virtual instance. In this case action needs to have access +to Mistral security context that contains auth token so that it can pass +that token to Cinder and Nova. + +Reusing existing actions +^^^^^^^^^^^^^^^^^^^^^^^^ + +As a user of Mistral I want to be able to reuse existing actions while +implementing my new actions so that I don't have to reimplement similar +functionality. + +For example, I want to create action that checks if a certain virtual +instance exists in the tenant by calling Nova and if it does the action +runs a number of secure shell commands to configure it. In this scenario, +we need to call Nova and do ssh. Both already exist in Mistral as actions +"nova.servers_get" and "std.ssh". So there should be a mechanism allowing +to reuse those actions while creating a new more complex action. + +Proposed change +=============== + +General idea +------------ + +We need to have one or more Python packages in Mistral that are designed +and documented as a public Python API for developers that want to create +custom actions. These packages should effectively provide a number of +classes that can be used directly or inherited as needed. They should +cover the following aspects of action development: + +* Base class or a number of classes that can be extended in order to build + new Mistral actions. Currently existing **mistral.actions.base.Action** + is an example of such class. + +* Module that provides access to security context associated with the + current workflow that this action belongs to. Security context should + at least include user, project/tenant, auth token. + +* Module that provides access to current Mistral execution context. That + context should include: + + * Current workflow execution id + + * Current task execution id + + * Current action execution id + +* Package with most frequently used utils and data types used during + custom actions development. For example, class + mistral.workflow.utils.Result that now exists in the code base is + needed by actions but it's not clear that it's part of Python API. + +* Module that allows to get and reuse existing actions + +Since these Python entities must be available for both engine and +executor they should be moved to a separate subproject of Mistral, for +example, **mistral-actions-api**. + +Existing OpenStack actions should be moved out of mistral project into +a different Mistral subproject. The proposal is to use **mistral-extra** +repo for this purpose because although we use it only for collecting +Mistral examples its initial idea was also to have additional tools +and extensions in it. + +Specific entities +----------------- + +mistral.actions.api +^^^^^^^^^^^^^^^^^^^ +Main Python package that contains all modules and classes which are part +of Custom Actions API. + +mistral.actions.api.base +^^^^^^^^^^^^^^^^^^^^^^^^ +Python module that contains base classes for custom actions. Currently +module **mistral.actions.base** performs similar function. + +Note: Specific content of this module is out of scope of this spec and +must be defined at implementation stage. + +mistral.actions.api.security +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Python module that contains required functions to get all required +information related to current OpenStack security context. At minimum: +user, project, auth token. + +Note: Specific content of this module is out of scope of this spec and +must be defined at implementation stage. + +mistral.actions.api.types +^^^^^^^^^^^^^^^^^^^^^^^^^ +Python module that contains all data types that custom actions need to +use. One candidate to go to that module that now exists is +**mistral.workflow.utils.Result**. + +Note: Specific content of this module is out of scope of this spec and +defined at implementation stage. + +mistral.actions.api.utils +^^^^^^^^^^^^^^^^^^^^^^^^^ +Python module that contains additional functions helpful for creating +new Mistral actions. At minimum: functions to get instances of existing +actions so that action developers could re-use functionality of existing +actions. Return type for these actions though must be rather a wrapper +that doesn't just call **Action.run()** method but instead uses Mistral +action execution machinery to actually call action just like as if it +was called as part of workflow (taking care of data transformations, +fulfilling security and execution context etc.) + +Note: Specific content of this module is out of scope of this spec and +must be defined at implementation stage. + +Alternatives +------------ + +None. + +Data model impact +----------------- + +None. + +REST API impact +--------------- + +None. + +End user impact +--------------- + +REST API users +^^^^^^^^^^^^^^ +No impact. + +Custom actions developers +^^^^^^^^^^^^^^^^^^^^^^^^^ +Having to use Custom Actions API described in this spec whereas now they +can only use **mistral.actions.base** safely. + +Performance Impact +------------------ + +No significant impact is expected. Minor is possible. + +Deployer impact +--------------- + +Deployers will need to make sure to install a new library containing +Custom Action API packages, modules and classes. However, this impact +is not supposed to be severe because all dependencies must be handled +smoothly by Pip. + +In case if there's an existing Mistral installation with installed +actions, some DB migration might be required. Changes in DB schema are +not expected though. If so, Mistral project should provide convenient +tools to help make this transition to using new actions. + +Implementation +============== + +Assignee(s) +----------- + +To be found based on discussions around the spec. + +Work Items +---------- + +* Create a new repo containing the code of Custom Actions API (e.g. + **mistral-lib** or **mistral-common**, particular name is to be defined) +* Design and implement modules listed in Specific Entities section +* Provide deprecation mechanism so that during some period of time it + would be possible to use the old approach for implementing Mistral + actions (with **mistral.actions.base**) and the new one +* Fix existing action implementations so that they use new API +* Fix Mistral Executor accordingly +* Fix Mistral Engine accordingly +* Revisit and restructure repo **mistral-extra** +* Move existing OpenStack actions into **mistral-extra** + + +Dependencies +============ + +No additional dependencies are required. + +Testing +======= + +Custom Actions API can be tested on devstack based OpenStack CI gates +such as gate-mistral-devstack-dsvm by creating and running custom +actions that use this API. + +References +========== + +Initial patch for TripleO/Mistral integration: +https://review.openstack.org/#/c/282366/ diff --git a/specs/pike/approved/workflow-global-context.rst b/specs/pike/approved/workflow-global-context.rst new file mode 100644 index 0000000..8a15463 --- /dev/null +++ b/specs/pike/approved/workflow-global-context.rst @@ -0,0 +1,260 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +======================= +Workflow Global Context +======================= + +Launchpad blueprint: + +https://blueprints.launchpad.net/mistral/+spec/mistral-global-wf-context + +Workflow global context will allow to store variables not associated with +particular workflow branches. + +Problem description +=================== + +Currently 'publish' keyword in Mistral saves variables into a storage +(context) which is associated only with a branch. + +Example: + + :: + + --- + version: '2.0' + + wf: + tasks: + A: + action: std.noop + publish: + my_var: 1 + on-success: A1 + + A1: + action: my_action param1=<% $.my_var %> + + B: + action: std.noop + publish: + my_var: 2 + on-success: B1 + + B1: + action: my_action param1=<% $.my_var %> + + +The expression "$.my_var" in the declaration of A1 will always evaluate to 1, +for B1 it will always evaluate to 2. This doesn't depend on the order in which +A and B will run. This is because we have two branches (A -> A1 and B -> B1) +for which the variable "my_var" has its own different version. + +Sometimes though we need to be able to share data across branches which is now +impossible due to aforementioned semantics. +The concept of workflow global context can help solve this problem. The word +"global" here means "accessible from any workflow branch". + +We also need an ability to make atomic updates of global workflow context. +It's necessary when we, for example, want to create a global counter (e.g. +counter of network calls to external systems performed by a workflow). + +Use Cases +--------- + +* Building conditions based on events happened in parallel workflow branches. + Example: one branch needs to notify the other one that it should stop. +* Passing data between branches. Example: one branch needs to wait till the + other one produces some expected result. This is, essentially, creating + a cross-branch mutex. +* Counters that need to decrement or increment atomically. + +Proposed change +=============== + +In order to achieve this goal the proposal is: + +* Add the new keyword "publish-global" which is similar to "publish" + with the difference that it publishes variables into workflow global + context instead of branch workflow context. It's important to note + that this is an unprotected way of modifying data because race + conditions are possible when writing different values for same + variables in the global context from parallel branches. In other + words, if we have branches A and B and there are tasks in these + branches writing different values to the variable X in the global + context Mistral won't provide any guarantees as far as what value + is going to be assigned to X and what value will be lost. Users need + to understand possible consequences. + For instance, using this keyword it's impossible to create an atomic + counter since it doesn't assume acquiring a lock under which we can + safely perform multiple operations (e.g. read and then write). + However, for many scenarios even this model can be useful. For example, + if there's only one branch writing values and others are only readers. +* Add the new YAQL/Jinja function "global()" to explicitly access + variables in workflow global context. +* Make global variables also accessible using "$." in YAQL and "_." in + Jinja in a way that branch variables can shadow them if they are + published in the current branch. +* Add the new keyword "publish-global-atomic" which is similar to + "publish-global" but allows to atomically read and write variables + in workflow global context by acquiring a temporary lock on it. + Unlike 'publish-global' this will allow to create atomic counters + when we need to perform multiple operations against the storage + atomically. + +Example #1 (writing and reading global variables): + + :: + + --- + version: '2.0' + + wf: + tasks: + A: + action: std.noop + publish: + my_var: "branch value" + publish-global: + my_var: "global value" + on-success: A1 + + A1: + # $.my_var will always evaluate to "branch value" because A1 belongs + # to the same branch as A and runs after A. When using "$" to access + # context variables branch values have higher priority. + # In order to access global context reliably we need to use YAQL/Jinja + # function 'global'. So global(my_var) will always evaluate to + # 'global value'. + action: my_action1 param1=<% $.my_var %> param2=<% global(my_var) %> + + B: + # $.my_var will evaluate to "global value" if task A completes + # before task B and "null", if not. It's because A and B are + # parallel and 'publish' in A doesn't apply to B, only + # 'publish-global' does. In this example global(my_var) has the same + # meaning as $.my_var because there's no ambiguity from what context + # we should take variable 'my_var'. + action: my_action2 param1=<% $.my_var %> param2=<% global(my_var) %> + + +Example #2 (writing global variables atomically): + + :: + + --- + version: '2.0' + + vars: + - my_global_var: 0 + + wf: + tasks: + task1: + action: std.noop + publish-global-atomic: + counter: <% global(my_global_var) + 1 %> + + task2: + action: std.noop + publish-global-atomic: + counter: <% global(my_global_var) + 1 %> + + +Alternatives +------------ + +None. + +Data model impact +----------------- + +Workflow execution object already has the field "context" which is now +immutable and initialized with openstack specific data, execution id and +environment variables. In order to get the full context for evaluating a +YAQL/Jinja expression in a task declaration we always build a context view +merged from workflow input, workflow execution "context" field and branch +specific context (e.g. task inbound context when evaluating action +parameters). The field "context" can play the role of workflow global +context. However, the idea to reuse this field can be revisited during +the implementation phase. + +REST API impact +--------------- + +None. + +End user impact +--------------- + +New workflow language feature that allows to store global variables into +workflow context. + +Performance Impact +------------------ + +When using "publish-global-atomic" we'll need to use locking in order +to prevent concurrent modifications of global workflow context while +reading and modifying it when processing a certain task. In fact, this is +equal to locking the whole execution object and hence will have a serious +performance impact in case of many parallel tasks. For this reason, +"publish-global-atomic" needs to be well documented and used with +precaution. + +Deployer impact +--------------- + +None. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + rakhmerov + +Other contributors: + melisha + +Work Items +---------- + +* Add 'publish-global' and 'publish-global-atomic' into the direct workflow + specification. +* Make changes in Mistral engine to publish variables into global context + (preliminarily it will be the field 'context' of workflow execution object). +* Implement YAQL/Jinja function 'global' to explicitly read variables from + workflow global context. +* Add locking workflow global context (i.e. workflow execution) in case of + using 'publish-global-atomic'. A thread that acquires a lock must first + refresh state of workflow execution and then proceed with publishing etc. + +Dependencies +============ + +None. + + +Testing +======= + +* Unit tests for 'publish-global' keyword and 'global' function in different + cases: parallel branches, sequential branches. +* Unit tests to check that branch-local variables take precedence when + reading variables using '$.' in YAQL and '_.' in Jinja. +* Unit tests for 'publish-global-atomic' that checks atomicity of reads and + writes of global variables. Although unit tests can't fully test this + feature. In order to fully test it we need to have a test with multiple + Mistral engines to make sure we have concurrent access to workflow execution. + +References +========== + +None.