Commit Graph

49 Commits

Author SHA1 Message Date
Hieu LE 4469cacf39 Implement policy in code - event trigger (11)
This commit migrate all event trigger policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

This commit also remove policy.json file usage completely.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>

Change-Id: Icd674fdc248e5b9c63f9e7d0f47c53288291ff07
2017-11-27 02:47:50 +00:00
Hieu LE c73fb88c01 Implement policy in code - workflow (10)
This commit migrate all workflow policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>

Change-Id: I8ab332abb5ebd4c67f8baf62c43ac3e9a10d7ee1
2017-11-27 02:47:42 +00:00
Hieu LE 0f6241b031 Implement policy in code - workbook (9)
This commit migrate all workbook policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>

Change-Id: I267a528322cdf69f132503112bb7124ec882ce80
2017-11-27 02:47:35 +00:00
Hieu LE 643da24ff2 Implement policy in code - service and task (8)
This commit migrate all service and task policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>

Change-Id: I20e9be2705a41931309b5da4f114715485d1797e
2017-11-27 02:47:29 +00:00
Hieu LE 3c999d01b2 Implement policy in code - member (7)
This commit migrate all member policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>

Change-Id: I166d0d62489fa8230a2ba34ef09f9366ac321633
2017-11-27 02:47:21 +00:00
Hieu LE d2274af9a2 Implement policy in code - execution (6)
This commit migrate all execution policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>
Closes-Bug: #1723333
Change-Id: I3e1d9f5b61d9f2511190fb810726aeb22912ad7b
2017-11-27 02:47:11 +00:00
Hieu LE 518066b019 Implement policy in code - environment (5)
This commit migrate all environment policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>

Change-Id: I97307ad7f4c01d9b78ddf1b4cda8e59e736a7831
2017-11-27 02:47:03 +00:00
Hieu LE 22ff1ad400 Implement policy in code - cron trigger (4)
This commit migrate all cron trigger policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>

Change-Id: Ib877e4ac6ebfe4068e85705955923d162757689f
2017-11-27 02:46:56 +00:00
Hieu LE e32fa68bcb Implement policy in code - action (3)
This commit migrate all action policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>

Change-Id: I2ced708b492026c7561cdfc417c43cfa0e4bb503
2017-11-27 02:46:49 +00:00
Hieu LE 33407d6f44 Implement policy in code - action execution (2)
This commit migrate all action execution policies into code [1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Dai Dang-Van <daidv@vn.fujitsu.com>

Change-Id: Ic4b597995ae2ab49e793871c14060f0b9662f468
2017-11-27 02:46:42 +00:00
Dai Dang Van 49ed570cfe Implement policy in code (1)
This commit prepare for implementing policies in code[1].

Like oslo.config, with oslo.policy, we can define all of
default rules in code base and only change some rules
via policy file. Another thing that we should use yaml
format instead of json format.

[1] https://governance.openstack.org/tc/goals/queens/policy-in-code.html
Co-authored-By: Hieu LE <hieulq@vn.fujitsu.com>
Change-Id: I2051b6c25333c95aa9ea6786964d4ab710ea93e8
2017-11-27 02:46:32 +00:00
junboli 47fae66fe0 Add README.mistral.conf doc in etc directory
Currently, the mistral.conf can be generated by tox, So add the
README.mistral.conf is useful.

Change-Id: I657b0d193a8f53dd86ed7b5bf228310eddbecd75
2017-09-15 04:21:12 +00:00
Nikolay Mahotkin 7d8d4fba82 Allow to list all cron-triggers
Closes-Bug: #1707942

Change-Id: Ia69972493a070aac4d413e27b26b02f6c60b19de
2017-08-01 18:39:59 +03:00
Nikolay Mahotkin e780ffb06d [Event-triggers] Allow public triggers
* Allowed creating via API
 * Added the corresponding processing
   in the event engine
 * Corresponding test

Closes-Bug: #1704111

Change-Id: I73a1d10fe684f1ec962e42e9700064d06cb0bcbe
2017-07-26 12:21:21 +03:00
Nikolay Mahotkin c430a4620b Fix get event triggers
* Added get event triggers for all projects
   and the corresponding rule in policy.json
 * Fixed corresponding policy rule names for
   event triggers API

Change-Id: I74e65a1cee8d074cfd637e460465922b610237c5
2017-07-11 12:46:58 +03:00
Lingxian Kong 965db538aa Role based resource access control - get workflows
We already supported role based api access control, this series patches
will implement resource access control for mistral, so that
administrator could define the rules of resource accessibility, e.g.
admin user could get/delete/update the workflows of other tenants
according to the policy.

TODO:
- Implement update workflow by admin
- Implement delete workflow by admin
- Implement for other resources(workfbook/execution/task/action, etc.)

Partially implements: blueprint mistral-rbac

Change-Id: I8b00e8a260a74457ad037ee7322a7cba9ae34fab
2016-12-22 14:12:33 +13:00
Dawid Deja b1937191ee Delete unnecessary comma
Change-Id: I7049ec25870a148a78d619eb39c90d032a287af4
Closes-bug: 1623414
2016-09-14 12:14:16 +02:00
Lingxian Kong 4e1e358c8b Add event trigger REST API
This patch adds the Mistral changes to support the new event trigger
REST API.

Change-Id: I8190ce81d46cc8296db29f41442354cdfe1a5bbd
Implements: blueprint event-notification-trigger
Co-Authored-By: Lingxian Kong <anlin.kong@gmail.com>
2016-09-13 22:39:36 +12:00
Lingxian Kong c73edcbe1d Add event configuration for event trigger
When using event trigger feature, some of the notification data may be
useful to end users in their workflows. However, the whole data can not
be visible to end users due to security reasons.

This patch will give operators a chance to define what information will
be available to end users. If the definition is not found, a 'safe'
default setting will be used.

Change-Id: I8b6e9cbe318011d2d11a2bb91aaff4d37222c7eb
Implements: blueprint event-notification-trigger
2016-09-01 10:28:59 +12:00
Jenkins e951813b1a Merge "Enable osprofiler to measure performance" 2016-06-16 04:02:11 +00:00
hardik 11e443efeb Added role base authentication support
Mistral api access can be restricted base on role
by modifying /etc/mistral/policy.json file.

Change-Id: I4c78ca4cc04d25f46aea55948bce339cfe460ff0
Implements: blueprint mistral-customize-authorization
2016-06-14 05:15:00 +00:00
Winson Chan 45ac6d03b1 Enable osprofiler to measure performance
Add option to enable osprofiler to measure performance of workflow execution.

Change-Id: I98e66e3f721c134370848dc2b65fb37c49b0e8ee
Implements: blueprint mistral-osprofiler
2016-06-10 01:49:38 +00:00
Winson Chan d6e1f590fa Add config example for rotating logs
Add examples for logging.conf and wf_trace_logging.conf to setup rotating logs
for mistral.log and mistral_wf_trace.log.

Change-Id: I85be48f2e98f5b7e9b8dbc339fa8de2dd7e78449
Implements: blueprint mistral-conf-rotating-log
2015-08-25 02:04:38 +00:00
LingxianKong d364ef2639 Remove mistral.conf.sample
For two reasons:

* The configuration file can be generated by using oslo-config-generator
  CLI command provided by oslo.config module, which is generally adopted
  by OpenStack projects.
* Reduce maintenance effort for each configuration item change.

Change-Id: If2f2559d669275b5fd252790657521049296c156
2015-08-12 10:54:59 +08:00
Guy Paz a8436dc5e2 Expiration policy for expired Executions
A mechanism that remove old executions (expired) according to Admin policy: how old they are, when executing this policy.
By the default this task is disable.

Change-Id: I15433d176d8e4a499a4466fc9324ceef60ddc4b9
Implements: blueprint expiration-policies-for-executions
2015-08-11 15:24:53 +00:00
Moshe Elisha 30589cfa5f Support large datasets for execution objects
Allow the cloud provider to configure the
size limit of workflow input, action input,
action output, task publish and workflow params

This will allow users to execute workflows that
handle much bigger datasets from the ones which
are supported today.

The changes made in order to achieve that goal:
* Increase DB columns size so they will no longer
  be the barrier
* Add configuration options to control the limit
* Add event listener on the columns to enforce
  the size limitation

Change-Id: If7c29f9325e60ce456e23d5c7b6ceb3477a028d4
Implements: blueprint support-large-datasets
2015-07-27 12:29:07 +00:00
Winson Chan 71f74a93cb Modify run_tests.sh to support PostgreSQL
Modify run_tests.sh to run unit tests with PostgreSQL. Fix a number of
connection pool and deadlock related issues in the unit tests. Fix DB
clean up routine to remove records in the correct context and order.

Change-Id: If88c00a326c025af885d061ca95588939195b7c5
Implements: blueprint run-unit-tests-on-postgresql
2015-07-06 19:16:59 +00:00
Nikolay Mahotkin f34edb7e27 Make mistral use of oslo-config-generator
Change-Id: I01f36e4b0080e10b9d39a6adbfe4b6cd43957fc6
2015-05-26 13:06:09 +03:00
Nikolay Mahotkin 7499f55a26 Fix devstack back to rabbit
Found the root of the problem: 
  Heat-engine uses queue 'engine' in rabbit, therefore we
  need to rename our queue in order to increase uniqueness 

Closes-Bug: #1324967

Change-Id: Iaf8a8f9f0357ae15ee4fc47d0031572a162526e6
2015-05-20 08:39:38 +00:00
manasdk df5f04c58a Updates logging configuration samples
- Changed all formats to look more like the default formatters. This simplifies parsing
  in tools like logstash etc.
- Changed location of all log files to end up in /var/log instead of /tmp. /var/log is
  a better default than /tmp for log files.
- In wf_trace_conf_logging.conf.sample using separate formatters for different handlers.
  This accounts for the usecase of each handler having its own specific log format to
  reduce redundancy in log statements.

Change-Id: I467f4d493cd379c11cf7e5e872e4baa4faeac674
Closes-Bug: 1322740
2014-12-17 01:34:27 +00:00
Kirill Izotov 6d035709a7 Replace oslo-incubator's db with standalone oslo.db
Change-Id: Id78149faefb7e458f7b11de1b2d71a03a8d56091
2014-07-28 18:01:57 +07:00
Angus Salkeld fc46164d5f Make plugins easier to use
This removes the need for editing mistral.conf and
makes the naming of the plugin more obvious.

Change-Id: Ie6e043eacfd0e413d81690f7c5978fd7bbec171d
2014-07-03 18:24:57 -10:00
Renat Akhmerov 08e80b5a68 Renaming all example config files from *.conf.example to *.conf.sample
* It's needed for consistency with the main config example

Change-Id: I1154c2b2da2823e94d40014cdca4499ac4e272e8
2014-06-16 16:52:10 +07:00
Winson Chan c4a85ff431 Clean up configuration settings
Removed outdated configuration options from mistral.config. Used
cfg.CONF.import_opt in engine to avoid NoSuchOptError. Removed
the imports of mistral.config using importutils. Generated new
sample configuration file.

Change-Id: I4b5ea7ef2a9b6717e6f33256107409c228575048
Implements: blueprint mistral-config-cleanup
2014-06-09 16:50:11 -07:00
Winson Chan 30e29148c5 Fix duplicate keystone auth_token config options
Currently, Mistral registers the configuration options under
keystoneclient middleware auth_token twice: by default when the
keystoneclient.middleware.auth_token module is loaded and in mistral.config
under a separate keystone group. This change removes the keystone group
in mistral.config, modifies API to use the default opts registered under
keystone_authtoken, and adds unit tests to ensure the keystone middleware
is called when auth is enabled.

Change-Id: I18948f7fa8b93b458335eb08176427a75c568873
Implements: blueprint mistral-config-cleanup
2014-06-04 18:18:58 -07:00
Angus Salkeld ebf225553d Add support for plugin actions
implements blueprint mistral-pluggable-task-actions

Change-Id: If12a0c6835edcabd33027555501dea4f473fc1f5
2014-05-28 22:00:11 +10:00
Winson Chan 2b1a2ea7d9 Allow launch script to start any combination of servers
Currently, the launch options are only all, api, engine, and
executor. We want to allow launch options to start any combination
of servers such as api and engine or engine and executor. The
launch script is modified to the --server command line option to
accept a comma delimited list.

Change-Id: I4ce31f5fabd4de3b2f8f0e4c5ef661ed4ec7e5b2
Implements: blueprint mistral-launch-options
2014-05-15 07:59:28 -07:00
Winson Chan 6c5ca50b4c Refactor engine to use plugins
Refactor current engine process to instantiate the concrete engine using
stevedore. The current engine uses importutils to load the concrete engine.
Since the executor is tightly coupled to the engine, it is also refactored
to use stevedore. The ScalableEngine is renamed to DefaultEngine and moved
to the mistral.engine.drivers.default namespace. The setup.cfg file is
updated to include entry points for the engine and executor plugins.

Change-Id: Ia9d15b25ca96387e5ac22c1a86cffc7a816e92fd
Implements: blueprint mistral-engine-plugin
2014-04-30 01:28:14 -07:00
Nikolay Mahotkin 46c049d480 Replace rabbit config to 'default' section
* Rabbit configs are moved to 'default' section due to
   oslo.messaging doesn't read these parameters from
   section 'rabbit'

Change-Id: I39393a779076f36c6142d770efa268178437332b
2014-04-28 14:15:05 +04:00
Winson Chan e003849ecf Engine as a standalone process
Refactored engine to run as a standalone process. The engine is
modified to use the RPC client/server design pattern from
oslo.messaging. A new launch option is included to run the
engine separately. The API server communicates with the engine
via the RPC client. A pecan hook is used to pass a common client
instance with appropriate transport configuration to the API.

Change-Id: I9ca757aedad764fd38645a5b858800d69808bb95
Implements: blueprint mistral-engine-standalone-process
2014-04-24 17:46:05 -07:00
Jenkins c55623c96d Merge "Remove etc/logging.conf" 2014-04-23 21:17:48 +00:00
Nikolay Mahotkin 5a8442ab00 Remove etc/logging.conf
* We should not keep .conf files, only examples

Change-Id: If2e37dfb46048180cc0391b8ed9aa0d295d11148
2014-04-23 15:45:31 +04:00
Nikolay Mahotkin 8b4d2f6128 Add workflow logging
* Created new log configuration for pretty workflow output.
 * Since this config opt is used before importing
   executor.server, I include some refactor of unit tests.

Implements blueprint mistral-pretty-workflow-logging

Change-Id: Ia9b8b5d401afafe99d6133685a4d3abbcb8ba2e3
2014-04-23 15:43:29 +04:00
Winson Chan e9a0f14db2 Refactor the local engine to use an in process executor
Refactored launch script to start the API and executor on the same process
so the transport for a oslo.messaging fake driver can be shared. A transport
attribute is added to the abstract engine as a class attribute to reference
the transport object.

Change-Id: I4b8684ebded9eb993749f2503380e078087edd7d
Implements: blueprint mistral-inproc-executor
2014-03-30 18:05:54 -07:00
Dmitri Zimine b0fa4605ce Make engine configurable, make debugger show local variables
To make it easy to set Mistral for debugging.
1) Make engine configurable.
2) Enable debugger via command line. Passing --use-debugger will make PyDev or PyCharm show
 local variables in debugging session.

Also, a tiny fix in action_factory.

Change-Id: Id054ae5156649b1f6bbba7131e54bf21d71c659a
2014-03-03 10:21:03 -08:00
Renat Akhmerov 44e38c97b7 Fixing and refactoring authentication
* Moved pecan configuration to oslo config
* Refactored functional base test
* Got rig of thread local related code in mistral/context.py
* Added new exception ApplicationContextNotFoundException
* Fixed example configuration file
* Made minor cosmetic changes (blank lines, naming)

Change-Id: I1899ce2562a34ebafa20c5735bdf4f0c80dd0175
2014-01-27 12:11:28 -05:00
Alexander Kuznetsov cafa9fed0e Added trust for workbook runs
Change-Id: Id2afa200858ce1cf9481883134d3c3d524f44cd9
2013-12-24 16:54:56 +04:00
Renat Akhmerov bb6dd45ec4 Implementing scalable Mistral Engine
* Asynchronous tasks
* Separate stateless task executors
* Notification of task executors via RabbitMQ

TODO:
* DB transaction management
* Analyzing graph of tasks to identify what should be started
* Thorough automated tests

Partially implements blueprint: mistral-poc
Partially implements blueprint: mistral-workflow-execution
Partially implements blueprint: mistral-engine-ha

Change-Id: I022e1b73b9a3c58f1fbc259ba3e5995acd2bf6c3
2013-12-12 21:06:36 +07:00
Renat Akhmerov 039e899d8b Adding REST API application skeleton based on pecan/wsme
* Adding required dependencies to use and test pecan/wsme app
* Adding additional modules from oslo-incubator
* Adding configuration files mistral.conf and specific file
 loggin.conf for logging
* Adding dependencies for testing: fixtures, testtools, mock
* Updating tox.ini to run nosetests for py26 and py27 environments

Change-Id: I4fd63820aaaf3b50fb1c981031f60faa68a6d307
2013-11-27 19:34:43 +07:00