We have no way to determine whether the workflow definition has
changed or not.
This complicates the work with Mistral for external services.
A 'checksum' field has been added to workflow_definitions_v2
to solve this issue.
Implements: blueprint mistral-add-checksum-field-to-wf-definition
Change-Id: I99d095c6c74c62f134aedfa3a63308a0712db38d
After this patch, user can update logging format to include root_execution_id in logs, which will be helpful to find and debug logs related to specific workflow execution.
- Logs about creation and status changes of Mistral entities(execution,
task, action execution, etc) are changed to INFO log level.
- User can update logging_context_format_string to include root_execution_id in logs.
Implements: Implements: blueprint improve-mistral-loggers
Change-Id: I54fe058e5451abba6ea7f69d03d498d78a90993e
This change introduces a new option, [healthcheck] enabled, which
enables the healthcheck middleware in mistral-api pipeline.
This middleware allows status check at /healthcheck path, which is
useful for load balancers or any monitoring services to validate health
of its backend services.
This change is created based on the same change proposed to ironic[1].
[1] 6f439414bdcef9fc02f844f475ec798d48d42558
Co-Authored-By: Jim Rollenhagen <jim@jimrollenhagen.com>
Change-Id: I9bf3de8a5ae6a8c9346285147732b773a3667f7e
There was a logical flaw where we compared types
that were different causing a TypeError.
There was also a flaw in that we default to the
name of the action when sorting if the key does
not exist, to compare the data both value should
come from the same key as well otherwise they
should be treated as not equal since the data
cannot be.
Change-Id: Idcb276912582bb097dc5c1c9692facde63d5886b
After this patch, it will validate execution-id exists in the DB while fetching tasks for the execution-id So, GET /v2/executions/execution_id/tasks API will return 404 for absent execution id.
Closes-Bug: #1998210
Change-Id: I6ac05f4b87c470a68cd67f19d1bd95a4d6cebe59
This patch adds an ability to rerun failed workflow by
skipping failed tasks. Workflow behavior in skip case could
be configured by new fields in task definition:
* on-skip
* publish-on-skip
Change-Id: Ib802a1b54e69c29b4d0361f048c2b9c076a4c176
Implements: blueprint mistral-task-skipping-feature
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
For many various support reasons, Mistral should have a special
endpoint to store all necessary info data. This endpoint will read
json from created by admin info file. To configure this you should
use mistral configuration:
[api]
enable_info_endpoint = True
info_json_file_path = info.json
Change-Id: I6f344dc15a4ca5c69a6b21841544a31f95eb393f
Implements: blueprint mistral-info-endpoint
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
When getting a cron trigger (but also any other object) by name, pecan
core will try to guess the content_type from the extension.
While for most name, the guess will fail, it may succeed, e.g. when a
name is ending with '.pl'.
Pecan guess it as a perl file and then reject the call (with a 404
error).
The fix is to tell pecan to not guess any content_type from extension by
setting guess_content_type_from_ext to False on initialisation.
Closes-Bug: 1987418
Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I1bc20b953787cae9b42d12bb5eb3c7a94ed4bbdd
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also replace policy.json to policy.yaml ref from doc and tests.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Change-Id: I3b9aeb3379a76f7e40dab0c46e27f4447a0c3d03
* Reworked /code_sources and /dynamic_actions API endpoints to
simplify them. For now they don't work with multiple objects and
they are consistent with other endpoints. If needed, we'll add
support for multiple objects (i.e. adding multiple dynamic actions
with a single request) later in a backwards compatible manner.
* Simplified unit tests.
* Got rid of services/*.py modules since they didn't do anything
useful. They were just wrappers around DB API calls.
Change-Id: Ib5a53f1f1a185f0395ffae1ab0c401633fcdd0fc
* Style improvements to make sure the code is compliant with the
coding guidelines
* Fixing small but important things like the mismatch between the
sinatures of the methods find_all() in the DynamicActionProvider
class and the base ActionProvider interface.
* Improved the tests.
* Simplified the implementation of DynamicActionProvider.
Change-Id: Idbfb15b4c3bb415e7fa9c7ece27eabfe674b6059
* added dynamic actions:
these actions are created and modified in runtime,
each action needs a code source to be imported from and a
class name.
- there are 2 new endpoints:
- /v2/code_sources/:
used to add new code sources to mistral.
- /v2/dynamic_actions/:
used to add dynamic actions to mistral in runtime
- a new Action provider (DynamicActionProvider) was added:
it provides the actions created from the dynamic actions api.
Change-Id: I9fe8c28ffdef71016d9dc13aea60a288c8ebaa0a
Signed-off-by: ali <ali.abdelal@nokia.com>
* This patch refactors Mistral with the action provider concept
that is responsible for delivering actions to the system. So
it takes all the burden of managing action definitions w/o
having to spread that across multiple subsystems like Engine
and API and w/o having to assume that action definitions are
always stored in DB.
* Added LegacyActionProvider that represents the old way of
delivering action definitions to the system. It pretty much just
analyses what entries are configured in the entry point
"mistral.actions" in setup.cfg and build a collection of
corresponding Python action classes in memory accessible by names.
* The module mistral/services/actions.py is now renamed to
adhoc_actions.py because it's effectively responsible only for
ad-hoc actions (those defined in YAML).
* Added the new entry point in setup.cfg "mistral.action.providers"
to register action provider classes
* Added the module mistral/services/actions.py that will be a facade
for action providers. Engine and other subsystems will need to
work with it.
* Other small code changes.
Depends-On: I13033253d5098655a001135c8702d1b1d13e76d4
Depends-On: Ic9108c9293731b3576081c75f2786e1156ba0ccd
Change-Id: I8e826657acb12bbd705668180f7a3305e1e597e2
*added a new parameter to the execution report Api,
statistics-only: the Api will only return the statistics without
the execution data.
*added a new field to statistics,
estimated_time: the estimated remaining time in seconds for the
execution,it is the average run time of the last 20 successful
executions of the same workflow
- if no previous successful executions are found -1 is returned
- if the runtime of the current execution is higher than the
average then 1 is returned
Change-Id: I7038d6d2a48f9f0455545f6be8dce33a48b25e1c
Partially-Implements: blueprint mistral-executions-report-return-statistics-only
Signed-off-by: ali <ali.abdelal@nokia.com>
*the new endpoints are:
v2/executions/{ex_id}/executions
v2/tasks/{task_id}/executions
it returns a list of execution resources, by default the API will
return all sub-executions
the API can take 3 parameters:
errors_only: returns only the error routes
- default is False
include_output: includes the output field for the returned executions
- default is False
max_depth: the max depth for the returned executions
- if a negative value is given, then the API will return
all sub-executions
- default is -1
Partially-Implements: blueprint mistral-execution-origin
Change-Id: I2a4c5e6890dbb4de868ce885d51341b15e359233
Signed-off-by: ali <ali.abdelal@nokia.com>
the input field of workflow-get API is ambiguous, and not well defined,
2 different inputs can have the same value for the input field.
for example :
input:
- first
- second: "value, third"
and:
input:
- first
- second: "value"
- third
for both it was: input ="first, second=value, third"
now it is:
for the first: input ="first, second=\"value, third\""
and for the second: input ="first, second=\"value\", third"
Closes-Bug: 1631541
Change-Id: I4b3fc7e0ad4552022d44c3bdd8c87004f5e8ff46
Signed-off-by: ali <ali.abdelal@nokia.com>
* Workflow resource(returned by workflow-get APIs) has a new JSON field
called 'interface', it contains the inputs and the outputs.
-Both input and output are lists.
-The output list contains the output param names.
-The input list contains the inputs, if the input has a default value,
the entry in the list will be a JSON object, with the name of the
input and the default value, otherwise its the input param name.
For example:
{"input": [ "input_with_no_default_value",
{ "string_input": "some_string" },
{"json_input": {
"defaultKey": "defaultValue"
}}],
"output":["output1","output2"]}
Change-Id: Id9be6c1d9bbe4a4c965530364833e71f8b7cacc6
Implements: blueprint mistral-workflow-outputs-in-wf-apis
Signed-off-by: ali <ali.abdelal@nokia.com>
added namespace for the actions, actions can have the same name if they
are not in the same namespace, when executing an action, if an action
with that name is not found in the workflow namespace or given
namespace mistral will look for that action in the default namespace.
* action base can only be in the same namespace,or in the
default namespace.
* Namespaces are not part of the mistral DSL.
* The default namespace is an empty string ''.
* all actions will be in a namespace, if not specified, they will be
under default namespace
Depends-On: I61acaed1658d291798e10229e81136259fcdb627
Change-Id: I07862e30adf28404ec70a473571a9213e53d8a08
Partially-Implements: blueprint create-and-run-workflows-within-a-namespace
Signed-off-by: ali <ali.abdelal@nokia.com>
This causes errors and is not recommended.
It also solves the problem of coordination not
working properly under wsgi
Change-Id: Ie3052d8cdc047f8a46e139f533a63f8c36d2906d
Closes-Bug: 18008953
This way we can test the service api later on gate
In order to use etcd in gate few changes were made:
* All identifiers must be byte type (group type, member id)
* Tooz has a built-in mechanizm for heartbeat no need to implement it
* Need to use eventlet monkey patch since etcd client uses blocking
methods
* Services name must be identical to LAUNCH_OPTIONS used in cli
* Gate coordination url should be define with a schema of etcd+http
which is the etcd gateway and works better then just etcd
Change-Id: I772651e33eada4a5c2149bfa867095c277eddeed
* Fields of type UUID with a filter prefix are denied because they
don't match the UUID string format. E.g.
"eq:6c07a453-c5e1-4bbe-97ed-3cb77b4f55ff" will throw an
InputException. This patch makes the change that allows to have:
"eq:" "neq:" "gt:" "gte:" "lt:" "lte:" "has:" "in:" "nin:" prefixes
in a value of such fields.
Closes-Bug: #1792875
Change-Id: I26667a82ec768c858f0282124864e377d8cf39f4
Signed-off-by: ali <ali.abdelal@nokia.com>
Workflow and task executions will inherit tags from
definition. Executions filtering by tag is included.
Change-Id: Id5d615b829901258af2be7ca99178ad92b60d1fb
Closes-Bug: #1853457
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
* Added a list argument to the method get_all() of the executions
API controller class that may contain names of columns that a
client wants to set to the null value (None internally in the
code) in the query string. Thereby we're getting the ability to
filter API entities (currently only workflow executions) fetched
with the get_all() by some fields that are assigned to the None
(null from the API perspective) value, i.e. typically it means
that a value of a field is not defined.
Closes-Bug: #1702421
Change-Id: I78fbf993519beb63ee9aef7058bdcb40f0a12ec3
* There's an issue with lazyly loaded field of WorkflowExecution
model occuring on GET /v2/execution/<id> because the logic
that calculates "published_global" of the execution rest resource
hits "root_execution" field out of transaction scope indirectly
within the "data_flow.get_workflow_environment_dict" method.
This patch makes refactoring of this logic and calculates
globally published variables of the workflow execution simply
as its context that doesn't contain all internal data like
"__execution" and "openstack".
* Other style change.
Closes-Bug: #1846152
Change-Id: Ic8609e55930e2ed13653e79e8ca7a31c951d9030
* For the sake of the service performance, it may make sense to
disable validation of the workflow language syntax if it is
affordable for a particular use case. For example, if all
workflows are auto-generated by a 3rd party system and tested
thoroughly (either by running them with Mistral or at least
validating them via the special Mistral endpoint) then we can
safely disable validation of the language syntax when uploading
workflow definitions. For production systems it makes a big
difference if workflow texts are large (thousands of tasks).
This patch adds the boolean parameter "skip_validation" for API
requests like "POST /v2/workflows" to disable validation, if
needed, and the new configuration property "validation_mode"
to set a desired validation mode.
The option is an enumeration and has the following valid values:
1) "enabled" - enabled for all API requests unless it's
explicitly disabled in the request itself
2) "mandatory" - enabled for all API requests regardless
of the flag in the request
3) "disabled" - disabled for all API requrests regardless
of the flag in the request
"mandatory" is choosen as the default value for this new
property to keep compatibility with the previous versions.
* Minor style changes.
Closes-Bug: #1844242
Change-Id: Ib509653d38254954f8449be3031457e5f636ccf2
* Various simple optimizations for creation of specs. Mostly
additional caching of non-growing data related to the
structure of the language spec classes. For huge workflows
with thousands of tasks these changes reduce execution time
by dozens of seconds.
* Minor style changes.
Partial-bug: #1844242
Change-Id: Ia700e25752d9d35ece18609f2977e6568062e4bf
* After the recent change that made all potentially heavy fields
of execution objects lazily loaded, some clients using Mistral
APIs started crashing because they expect to get "input" field
in JSON of execution objects that Mistral API returns. It wasn't
present though because we did not initialize it explicitly in
the API controller.
* Unfortunately, there's no easy way now to cover this change in
the API tests just for how they are organized: they mock all
DB calls and return all fields already initialized. We may want
to refactor these tests moving forward.
Change-Id: I683c79fa0a3ab23a16c493ce2314a506dfee9749
* Some fields of execution objects can be large and, what's even more
important, their size is defined by user input. Making these fields
lazy-loaded reduced the amount of data loaded from DB in those
methods where ther are not needed. For example, "spec" field of the
WorkflowExecution class is rarely needed because the corresponding
specification Python object gets cached. It's proven to reduce
the execution time in some cases by ~5-10%.
* Fixed failing test
Change-Id: Ica0ba2ffa312891483745d276d04c95985c7f4c2
Using default mutable parameter is bad.
Default parameters are evaluated only once
if you mutate it you will get unexpected results.
Since we don't mutate here the default paramter, make
sure it is unmutable.
Change-Id: Ib5c451a8c8cad7b6c9a009369c1c039563023368
This sets up the HTTPProxyToWSGI middleware in front of Mistral API. The
purpose of this middleware is to set up the request URL correctly in
the case there is a proxy (for instance, a loadbalancer such as HAProxy)
in front of the Mistral API.
The HTTPProxyToWSGI is off by default and needs to be enabled via a
configuration value.
It can be enabled with the option in mistral.conf:
[oslo_middleware]
enable_proxy_headers_parsing=True
Closes-Bug: #1590608
Closes-Bug: #1816364
Change-Id: I04ba85488b27cb05c3b81ad8c973c3cc3fe56d36
Vadim Zelenevsky