For many various support reasons, Mistral should have a special
endpoint to store all necessary info data. This endpoint will read
json from created by admin info file. To configure this you should
use mistral configuration:
[api]
enable_info_endpoint = True
info_json_file_path = info.json
Change-Id: I6f344dc15a4ca5c69a6b21841544a31f95eb393f
Implements: blueprint mistral-info-endpoint
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
Moved common logic in mistral/exceptions from
MistralError and MistralExceptions to MistralFailureBase
Change-Id: I79dc7b91c00a296a2c6d9c3b589ea3fa4fc7db48
* Also removed a custom initializer of SizeLimitExceededException with
3 parameters in favor of creating a static method because we
have to assume that all MistralException classes have an initializer
with only one parameter "message" to be able to restore an exception
on the RPC client which was raised on the RPC server.
Change-Id: Idd52bb7bb18168826644d8aaca3d96be46f8ae9e
Closes-Bug: #1750752
* Method _get_collection() in api.py wraps all DB exceptions with
DBQueryEntryError. It causes a problem because if it raises a
DBDeadlock exception then it's not handled by @retry_on_deadlock
decorator and the corresponding workflow operation gets lost.
This patch remove this wrapping and also the exception class
because it's not used anywhere else.
Closes-Bug: #1727152
Change-Id: If885bb63dc165aaf19001a4ff2c381732ee38982
Allow async action execution to be paused and resumed by
updating the action execution via API. When an action
execution is paused, the state transition will cascade up
to the task execution.
Implements: blueprint mistral-action-ex-pause-resume
Change-Id: I87233d27c46cfe86a23beb8dfdc96f77e58d24c1
Allows to use Jinja instead of or along with YAQL for expression
evaluation.
* Improved error reporting on API endpoints. Previously, Mistral API
tend to mute important logs related to errors during YAML parsing
or expression evaluation. The messages were shown in the http
response, but would not appear in logs.
* Renamed yaql_utils to evaluation_utils and added few more tests to
ensure evaluation functions can be safely reused between Jinja and
YAQL evaluators.
* Updated action_v2 example to reflect similarities between YAQL and
Jinja syntax.
Change-Id: Ie3cf8b4a6c068948d6dc051b12a02474689cf8a8
Implements: blueprint mistral-jinga-templates
This patch adds the Mistral changes to support the new event trigger
REST API.
Change-Id: I8190ce81d46cc8296db29f41442354cdfe1a5bbd
Implements: blueprint event-notification-trigger
Co-Authored-By: Lingxian Kong <anlin.kong@gmail.com>
* Changed AuthHook for Pecan that implements token validation
* Added another config option to disable SSL verification for
KeyCloak access tokens
* Added unit tests for successful and failed KeyCloak
authentication that use request_mock library
* Minor style changes
Change-Id: I87f8d54fc58f82952a4c68831547e6dab320230e
* Introduced new class Workflow that manages life-cycle of running
workflows and is responsible for managing workflow persistent state
* Moved all workflow level logic to workflow handler and Workflow class
* Changed semantics if how workflows start errors are handled.
Previously, in case of invalid user input Mistral engine would store
information about error in "state_info" field of workflow execution
and bubble up an exception to the user. This approach was incorrect
for a number of reasons including broken semantics: if an exception
was raised due to invalid input it's normal to expect that system
state has not changed. After this refactoring, engine only raises
an exception in case of bad user input. That way behavior is
consistent with the idea of exceptional situations.
* Fixed unit tests in according to the previous point
* Fixed a number of logical issues in tests. For example, in
test_default_engine.py we expected one type of errors (e.g. env not
found) but effectively received another one (invalid input).
Partially implements: blueprint mistral-engine-error-handling
Change-Id: I09070411fd833df8284cb80db69b8401a40eb6fe
* Clear separation for problems that can be handled so that the program
can continue and problems that can't handled automatically due to major
issues in configuration, environment or code itself
* Split YAQL exceptions into two types: grammar exception and evaluation
exception
* General NotFoundException is replaced with more specific DBEntryNotFoundException
for better consistency with other DB exceptions and more clear semantics
* Fixed corresponding tests
Change-Id: I07f495ab316b0f164caece78b1f101219199e68c
Implements: blueprint mistral-engine-error-handling
With this patch, users can perform operation as following:
DELETE http://127.0.0.1:8989/v2/action_executions/<action_execution_id>
NOTE:
* Added a new config item 'allow_action_execution_deletion', with default
value 'False'.
* The 'allow_action_execution_deletion' item is set to 'True' for
functional testing.
* Only ad-hoc action-execution can be deleted.
* Only completed action execution can be deleted.
Implements: blueprint mistral-action-execution-deletion
Closes-Bug: #1488157
Change-Id: I3729636911a42c273c5a7b2d7fbdaae0da398e31
Add query params for workflow list REST API:
* limit: return a maximun number of items at a time, default is None, the
query result will include all the resource items, which is backward
compatible.
* marker: the ID of the last item in the previous list.
* sort_keys: columns to sort results by. Default: created_at.
* sort_dirs: directions to sort corresponding to sort_keys, "asc" or
"desc" can be choosed. Default: asc. The length of sort_dirs can
be equal or less than that of sort_keys.
Change-Id: Ie73d4457193999555ce9886d4de1297b4d0bc51d
Partially-Implements: blueprint mistral-query-enhancement
Tooz is a Python library that provides a coordination API. Its primary
goal is to handle groups and membership of these groups in distributed
systems.
This patch adds coordination util to Mistral, which makes use of Tooz
libary.
Change-Id: Icbf3086d01649af813727f0972d6d5b0631d6afb
Partially-Implements: blueprint mistral-service-api
Allow the cloud provider to configure the
size limit of workflow input, action input,
action output, task publish and workflow params
This will allow users to execute workflows that
handle much bigger datasets from the ones which
are supported today.
The changes made in order to achieve that goal:
* Increase DB columns size so they will no longer
be the barrier
* Add configuration options to control the limit
* Add event listener on the columns to enforce
the size limitation
Change-Id: If7c29f9325e60ce456e23d5c7b6ceb3477a028d4
Implements: blueprint support-large-datasets
If the ad-hoc action input default value is defined in the spec, the input
param does not needed to be provided for action execution.
* rename validate_workflow_input function to validate_input for common
purpose of both workflow/action input validation.
* delete WorkflowInputException definition, use InputException instead.
* add unit test for ad-hoc action execution.
Partially Implements: blueprint mistral-default-input-values
Change-Id: I3e25c1ce9f7f93dfce3b5981221cf752603ec49e
* Fixed ProxyAwareDict according to YAQL
convert data when evaluating expression
* Fixed expression tests
* Fixed Engine tests where we refer to unexisting
variable in context
(now YAQL raises an error when it doesn't find a
value using given key)
* Fixed tests on join (new way to check test correction)
* YAQL 1.0 has its own builtin functions len() and str
therefore our ones were deleted.
* YAQL 1.0 uses '$.where' function instead of '$[$...]'
Implements: blueprint yaql-v1-0
Change-Id: I1b46c4e12aee6f25d1e9ca9daa1da059baf6fb68
Currently, syntax errors in the DSL do not surface until runtime while
in the middle of running a workflow. The change here hardens the v2
model and schema to catch obvious errors in the DSL prior to the
definition being created and updated. This is also the pre-requisite
for the API endpoint to validate workflow/workbook DSL.
Change-Id: I0efbfcc3a42446951fd3550892768a668c6283d1
Implements: blueprint api-validate-dsl
* Breaking DB api and DB models into separate versions
* Fixing unit tests and dependent code
* Preparing DB models for the new API/engine
TODO:
* Adjust v2 DB models according to the new spec
Change-Id: I763cc3f401b8040a182733750ce05577653e1d35
* Rewriting package 'workbook' to have v1 and v2 separately
* Refactoring all required places to use v1 spec explicitly
* Refactoring and making specs more consistent
TODO:
* Fully implement spec v2 when it's complete
Change-Id: I4d6949588635ff45775f68186ea43b16163b6a72
* Skeleton for the new engine
* Main modules and interfaces (Engine, WorkflowHandler, policies)
* Sketches of main engine methods
* Reverse workflow handler
* Function to check valid state transitions
* Unit tests
Change-Id: Id6ae293154bca7cab69965725ea6960d5f8d8790
Updated openstack-common.conf and remove outdated modules. Ran update script
in oslo-incubator to generate modules in the mistral/openstack and tools
directories. Sqlalchemy backend has changed significantly in oslo-incubator.
Refactored mistral db api to address these changes and updated the base test
class DbTestCase accordingly.
Change-Id: I61327c33c0c8bc616af5e7633634461a2d7ac691
Implements: blueprint mistral-sync-with-incubator
- Remove redundant constructors
- always use "code" as an http_status
- fix the default message strings so you can pass nothing into the
exception constructor and it will work.
- add some basic tests
Partial-bug: #1320136
Co-Authored-By: Nikolay Mahotkin <nmakhotkin@mirantis.com>
Change-Id: I9e21c6d5d39f1203324a37a39e5fd6069062b733
* Base class for actions (Action)
* Class for representing action namespaces (NameSpace)
* Module action_factory.py for registering action classes, accessing
them and creating action instances
* Unit tests for action_factory.py
* Standard actions (echo, http, email)
TODO:
* Ad-hoc per-workflow namespaces and actions
* Mistral HTTP standard action
* Refactoring executor to use new action infrastructure
* Remove old action infrastructure
* Action plugin architecture
Partially implements: blueprint mistral-actions-design
Partially implements: blueprint mistral-change-rest-api-to-http
Change-Id: I8e79b1f8173809584f0156e666e1675f5bd7bb37
* Implemented new object model
* Implemented spec validation
* Old DSL parser was deleted
* All code is replaced by new workbook spec
* Also tests are fixed
* events -> triggers
* Unit tests
Implements: blueprint mistral-dsl-model
Partially implements: blueprint mistral-dsl-validation
Partially implements: blueprint mistral-rename-event-to-trigger
Change-Id: I782b7983ddc724375229587f467c55ad9341478b
* Refactored tests hierarcy (moved get_resource to the base class)
* Fixed logic in abstract engine (transaction scoope was wrong)
* Added stub methods for context manipulations (not everywhere yet)
* Refactored actions
* Created ECHO action to be able to emulate required output
* Added simple data flow test (two dependent tasks)
TODO:
* More tests
* Refactor actions according to the latest discussions
Change-Id: I5ba5e330110889014eeca53501ddae54dc9a1236
Fixed several errors:
E1003
E1101
It is impossible to fix all E1101 errors: looks like we have
problems with outdated DataBase models.
Change-Id: I46863c553f033e5f9e4ae9cb9c20e297c099a114
Partial-Fixes: bug #1281164
1. Add couple of negative tests, find the way, more to go.
2. Add assertNotFound to the base test class for convenience.
3. MistralException - added default message to enable default instantiation - make mock.assertRaise fails when exception has no default constructor
Change-Id: I708637e3995597a819571a90f3292d6e34deab4e
* Moved pecan configuration to oslo config
* Refactored functional base test
* Got rig of thread local related code in mistral/context.py
* Added new exception ApplicationContextNotFoundException
* Fixed example configuration file
* Made minor cosmetic changes (blank lines, naming)
Change-Id: I1899ce2562a34ebafa20c5735bdf4f0c80dd0175
* Action definition was combined in actions module
* workflow from both single and scalable engines were deleted
* task from single engine was deleted
* unit test 'test_workflow' was deleted due to deleting workflow
* minor fixes to single engine
Next step will be extracting and combining common parts
of scalable and single engine.
Change-Id: I15fbbb7f9d5e453c17625423eeb1bf841a19db18
* Fixed several problems in scalable engine
* Fixed engine interface
* Fixed several problems in REST API
* Changed task DB model
* Fixed DSL parser
* Added scripts for end-to-end testing
Change-Id: Icbcc46d11a6c687d0ae3aad339ca5f5134c50195
* Added methods on DB API for explicit transaction demarcation
* Added @session_aware() decorator maker
* Refactoring SqlAlchemy implementation of DB API
* Added all tests for DB API
* Added methods to access thread local variables in mistral/utils.py
* Fixed API Resource class to ignore unset values in transformations
Change-Id: I5fae196f89c77a5e355207907bd08983969922c3
Vasudeo Nimbekar