* This patch refactors Mistral with the action provider concept
that is responsible for delivering actions to the system. So
it takes all the burden of managing action definitions w/o
having to spread that across multiple subsystems like Engine
and API and w/o having to assume that action definitions are
always stored in DB.
* Added LegacyActionProvider that represents the old way of
delivering action definitions to the system. It pretty much just
analyses what entries are configured in the entry point
"mistral.actions" in setup.cfg and build a collection of
corresponding Python action classes in memory accessible by names.
* The module mistral/services/actions.py is now renamed to
adhoc_actions.py because it's effectively responsible only for
ad-hoc actions (those defined in YAML).
* Added the new entry point in setup.cfg "mistral.action.providers"
to register action provider classes
* Added the module mistral/services/actions.py that will be a facade
for action providers. Engine and other subsystems will need to
work with it.
* Other small code changes.
Depends-On: I13033253d5098655a001135c8702d1b1d13e76d4
Depends-On: Ic9108c9293731b3576081c75f2786e1156ba0ccd
Change-Id: I8e826657acb12bbd705668180f7a3305e1e597e2
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592
Change-Id: I98e80bcbc7637bbb6063935043e530718be87f7b
Closes-Bug: #1863021
* When Mistral is launched via launch.py script then YAQL engine class
initialization logic doesn't use the right properties from the config
file. This is because this initialization is caused by the chain of
imports taking its start in launch.py, and this happens before
the config file is parsed in the main() function of launch.py.
Using lazy initialization of YAQL engine class (YAQL_ENGINE in
yaql_expression.py module) solves this issue because now Mistral
doesn't initialize it immediately before parsing the config and
waits for the first usage of it.
* Minor style changes per Mistral coding guidelines.
Change-Id: If3367493803b57ef8bc281b1f64f2a223ac86f85
Closes-Bug: #1864785
* The functionality of graceful engine shutdown is now possible
due to correct calculation of the "graceful" flag in the engine
server's stop() method. Unfortunately, the Oslo Service framework
doesn't pass it correctly, it simply ignores it in the call chain.
So the only way to understand if the shutdown is graceful is to
peek at the configuration property "graceful_shutdown_timeout"
provided by Oslo Service. If it's greater than zero then we can
treat it as graceful.
* Oslo Service handles only four OS signals: SIGTERM, SIGINT,
SIGHUP and SIGALRM. Only sending SIGTERM to the process leads
to a graceful shutdown. For example, SIGINT (which is equal to
ctrl + C in a unix shell) interrupts the process immediately.
So the only way to do a graceful shutdown of an engine instance
using a unix shell is to run the "kill <PID>" command. This
needs to be taken into account when using it.
* The patch also changes the order in which the engine server
stops its inner services so that the underlying RPC server
(currently Oslo Messaging based or Kombu based) stops first.
This is needed to make sure that, first of all, no new RPC
calls can arrive, and thereby, let all active DB transactions
finish normally w/o starting new ones. Stopping the RPC server
may be a heavy operation if there are already lots of RPC
messages waiting for processing that are polled from the queue.
So to the great extent the entire functionality of graceful
shutdown will depend on whether an underlying RPC server
implements the corresponding functionality in the proper way,
i.e. after calling stop(graceful=True) it will stop receiving
new calls and wait till all buffered RPC messages are processed
normally.
* The maximum time given to graceful shutdown is controlled via
the "graceful_shutdown_timeout" configuration option, which is
60 seconds, by default.
* Minor refactoring
Implements blueprint: mistral-graceful-scale-in
Change-Id: I6d1234dfa21b1e3420ec9ca2c5235dee973748ee
pbr provides VersionInfo.version_string() as a method to determine
the version. mistral.version.version_string can hide this and
provide a uniform string interface to config and launch.
This fixes a problem with mistral-server --version generating
an exception because the version passed to argparse was a method
instead of a string.
Change-Id: Ie468685e4360bfaec5d82b02f8cf1a27a93bcd94
Closes-Bug: 1796921
New releases of oslo.config support a 'mutable' parameter to Opts.
oslo.service provides an option here Icec3e664f3fe72614e373b2938e8dee53cf8bc5e
allows services to tell oslo.service they want mutate_config_files to be
called by passing a parameter.
This commit is to use the same. This allows mistral to benefit from
I1e7a69de169cc85f4c09954b2f46ce2da7106d90, where the 'debug' option
(owned by oslo.log) is made mutable. we should be able to turn debug
logging on and off by changing the config.
tc goal:
https://governance.openstack.org/tc/goals/rocky/enable-mutable-configuration.html
Change-Id: I8ea7085b9343b07f5fe419d30a68c00caec1b745
Using this API is needed to correctly initialize the configuration.
[keystone_authtoken] group is used for keystonemiddleware to validating
token.
The new [keystone] group is used for keystoneauth to initialization
the keystone session.
Co-Authored-By: wangxiyuan<wangxiyuan@huawei.com>
Change-Id: Ie3ab512b0ab42c0f97b3099e0787f4edee08ddce
Partial-Bug: #1775140
* Using "passive_deletes=True" when configuring relationships
with SQLAlchemy significantly reduces memory footprint when
deleting a graph of objects (e.g. root workflow execution
with all dependent tasks, actions and sub workflows) from DB.
This happens because in this case SQLAlchemy doesn't load
those dependent objects that are not yet in the session and
rather relies on cascading mechanisms of databases which are
activated by the proper configuration of foreign key columns.
For example, "ON DELETE CASCADE" clause in the foreign key
definition in case of for MySQL.
Particularly, this change reduces memory footprint of API
processes when deleting large graphs of executions and
execution expiration policy mechanism where the same happens
but on the engine side.
* Added a test to make sure that cascaded deletion works well
with this new property.
Related-Bug: #1757966
Change-Id: I36988817fb8e7695094ef98958b8aa151208fadb
Now there is a discrepancy between how to display the string with
information about the server with the started with parameter "all"
and all other cases.
'all': [api,engine,executor,event-engine,notifier]
other: [['api', 'engine', 'executor']]
TrivialFix
This commit fixes this situation by unifying the string format.
Change-Id: If223fc87db8a48ad64868f5e847d54279f251e40
Introduce execution events and notification server and plugins for
publishing these events for consumers. Event notification is defined per
workflow execution and can be configured to notify on all the events or
only for specific events.
Change-Id: I9820bdc4792a374dad9ad5310f84cd7aaddab8ca
Implements: blueprint mistral-execution-event-subscription
* It turns out that update_on_match() from oslo_db breaks fields
of a persistent object (attached to the session) in some cases.
For example, when we use PostgreSQL as a DB it doesn't properly
merge fields of the given specimen onto the fields of the object
attached to the SQLAlchemy session. For some reason, it cleans up
the history of ORM events registered before this operation so
that if some update took place in this session and they are not
flushed yet to DB they simply get lost. Hence adding flush()
on the session right before this operation helps.
* NOTE: this is rather a workaround than a fix of the root cause.
Root cause still needs to be found.
Change-Id: I546badd8b1c96b68567287fc9d38b07738272503
Closes-Bug: #1736821
Using MySQLdb driver for MySQL is not compatible with
greenthreads provided by eventlet library. If we issue
an SQL query through it from within a green thread then
eventlet dispatches doesn't switch to another green
thread although it is an IO operation.
We need to apply monkey patching to MySQLdb explicitly
as it was fixed some time ago but doesn't apply by default.
Change-Id: Id32c65b628d8ffec5452cc89d7cf5a0a24f0312d
Closes-bug: #1721218
When Mistral services are configured with systemd, the services time out
on shutdown. The root cause is that each of the server thread is
assigned to a different instance of the oslo.service ServiceLauncher.
The ServiceLauncher can manage multiple server threads. When the server
threads are added to the same instance of the ServiceLauncher, SIGTERM
is handled properly for each server threads.
Change-Id: I8b739844f6073890324189aee028c0d7da3cc76e
Closes-Bug: #1714351
The rpc_backend with kombu and oslo are being used by the executor
and event engine as well. This patch move the rpc_backend up one
level so it's not engine specific. Also Event engine has its own module
and the EventEngine class is defined in the engine module. This patch
moves the EventEngine to it's own base file in the event_engine module.
Implements: blueprint mistral-actions-run-by-engine
Change-Id: Ie814a26e05f5ca6bfba10f20a7d5921836aa7602
Make executor pluggable and allow option to run the executor
locally on the engine or remotely over RPC.
Change-Id: I7cfb13068aa1d1f88136eaa092e629c34b78adf2
Implements: blueprint mistral-actions-run-by-engine
* Fixed the bug #1622534
* Introduced MistralService class extending oslo.service.Service
that all Mistral components running standalone should extend
* Refactored engine, executor and event engine with MistralService
* Moved most of the startup logic from launch.cmd to individual
corresponding components
* Cluster membership is now under control of MistralService
* Fixed test_join.test_full_join_with_conditions() which previously
had a bug and we were just 'lucky' that it passed due to a
different work of scheduler
* Fixed a number of test cases
* Other minor changes
TODO:
* We now use many launchers (from oslo.service) to launch
services whereas we could use just one which is recommended by
oslo. However, we can't do that because of the api service
that uses many workers. We'll need to look at how to refactor it
moving forward.
* Write tests for MistralService and its derrived classes
* Address a number of TODO comments.
Closes-Bug: 1622534
Change-Id: I34ba6a8b8caf8bea17109e0f259085b373eb6d45
TrivialFix: Similar [1] in Kolla project
As we known, Exceptions are raised by the sys.exit() function. When they
are not handled, no stack traceback is printed in the Python interpreter.
Therefore, when using sys.exit(main()) instead of main()
may be more readable and reasonable.
[1] https://review.openstack.org/#/c/349353/
Change-Id: Ic71d4bfb1085c68ea9eb7e1382e7263a81634ad1
This patch adds a new optional service in Mistral, event engine
service will listen to the message queue with exchange and topic users
configed, and trigger the corresponding workflow via rpc call to
mistral engine. The event engine service will receive rpc messages
from mistral api
Change-Id: I8bef24822575670ca90ef0479b9ce2e3e330cdeb
Implements: blueprint event-notification-trigger
Co-Authored-By: Lingxian Kong <anlin.kong@gmail.com>
There is a small issue during changing branch from current to Mitaka, where
there was module called rpc.py, and currently there is a package rpc. Problem
is, if Mistral from master was already at least run once, and changed back to
Mitaka, there is a chance, that directory rpc still exists, and contains pyc
files. Trying to use old rpc module will fail with unable to find
get_transport method due remains of bytecode from master branch.
Simple solution for this situation is to rename rpc module to rpc_backend.
Change-Id: Ib5d05930a9715caf2bb5ce1dba4e520a503bc0cd
At this point all tests pass, but there is no test for kombu driver.
Komu driver must be consider as expermiental before new tests are introduced.
TODO (next commits):
- tests
Partially implements blueprint mistral-alternative-rpc
Co-Authored-By: Dawid Deja <dawid.deja@intel.com>
Change-Id: I1f5ca1f1e8c741efcb61480ccbec8e50ad993cba
* Using new RPC interface OsloRPCServer and
OsloRPCClient are created.
TODO (next commits):
- integrate new RPC interface in Mistral
- make RPC implementation configurable
- unit tests
Partially implements blueprint mistral-alternative-rpc
Co-Authored-By: Dawid Deja <dawid.deja@intel.com>
Change-Id: I6c9770a5b84509529abc14dff2b4a9f6e3411951
This patch fixes the HA gap when executor dies.
Now if executor dies, next executor pick up previous executor's task.
* Currently it is almost impossible to write unit-test on
this bug. For now, I created a new config option for whether
to use this fix or to use original oslo.messaging. By default
it is False.
* For tests, need to wait creating of HA-gate.
Closes-Bug: #1502120
Change-Id: Ia6d25d039b1e8210b7e544540e4b527d28f6d394
Mistral cannot parse configure file from command line
like this:
--config-file="/path/to/config_file"
So update the code to recognize configure file rightly.
Change-Id: I9f3c23d0437578c9714019c480785198c940d181
Closes-Bug: #1552146
This patch allows the Mistral actions to be run synchronously from
the Mistral CLI.
When a Mistral action is run synchronously using the Mistral CLI
'run-action' command, it does not complete successfully. This is due
to the Mistral API server handling requests on a single thread.
The 'run-action' command performs a REST POST call to the
ActionExecution RestController in the API server. That in turn calls
back into the python-mistralclient which then performs another REST
call back to the appropriate REST controller to actually run the
action requested. That call hangs since the requests are handled on
a single thread because the first POST has not completed yet.
Eventually the RPC call between the engine and the executor servers
times out, and the 'run-action' command fails.
This patch changes the Mistral API server so that requests are
handled in separate threads.
Added a new functional test to the tempest test package to test
synchronous action execution of a mistral action from within
mistral.
Change-Id: I8e06d3ef6aab4b2009a8fff4aa4d1acc118eee3f
Implements: blueprint mistral-mistral-actions
Improved code style, fixed all H405 (Multi line docstring
summary not separated with an empty line) errors.
Change-Id: I6639a2e1a9dc5d3802cb1bda05c5bf9b302bc82f
A mechanism that remove old executions (expired) according to Admin policy: how old they are, when executing this policy.
By the default this task is disable.
Change-Id: I15433d176d8e4a499a4466fc9324ceef60ddc4b9
Implements: blueprint expiration-policies-for-executions
Add mistrai-api, mistral-engine, mistral-executor services to their
service group respectively when they start up, so that we can retrieve
their information through service api which will be implemented in
subsequent patch.
Change-Id: Ibfcc17b35b9d2e408888b7d1cb64e63f2b32a90d
Partially-Implements: blueprint mistral-service-api
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: I73addc2c144c76c60f046e83c97e3b6ffe09d879
* use oslo.log for logging functionality
* get rid of the file /mistral/openstack/common/log.py
Implements: blueprint mistral-log-enhancement
Partial-Bug: #1459188
Change-Id: I6b62cffda0a20b6caf59cc30bcc33c21ae1e5898
*This bug happened because of how oslo is handling the order of command line
*Make sure the command line parameter will always send last
Change-Id: Ia106ce2bd2994c494a6c25039d92358eeb0abdb9
Closes-Bug: #1454111