This change fixes the following two patterns of errors detected in
gate.
1)
sqlalchemy.exc.ArgumentError: Textual column expression '...' should be
explicitly declared with text('...'), or use column('...') for more
specificity
2)
testtools.matchers._impl.MismatchError: '('...',)' is not an instance
of tuple
Change-Id: Ie080bdcacf9aa2dd2fe0454bcb814285c08bbfa8
* Workflow resource(returned by workflow-get APIs) has a new JSON field
called 'interface', it contains the inputs and the outputs.
-Both input and output are lists.
-The output list contains the output param names.
-The input list contains the inputs, if the input has a default value,
the entry in the list will be a JSON object, with the name of the
input and the default value, otherwise its the input param name.
For example:
{"input": [ "input_with_no_default_value",
{ "string_input": "some_string" },
{"json_input": {
"defaultKey": "defaultValue"
}}],
"output":["output1","output2"]}
Change-Id: Id9be6c1d9bbe4a4c965530364833e71f8b7cacc6
Implements: blueprint mistral-workflow-outputs-in-wf-apis
Signed-off-by: ali <ali.abdelal@nokia.com>
added namespace for the actions, actions can have the same name if they
are not in the same namespace, when executing an action, if an action
with that name is not found in the workflow namespace or given
namespace mistral will look for that action in the default namespace.
* action base can only be in the same namespace,or in the
default namespace.
* Namespaces are not part of the mistral DSL.
* The default namespace is an empty string ''.
* all actions will be in a namespace, if not specified, they will be
under default namespace
Depends-On: I61acaed1658d291798e10229e81136259fcdb627
Change-Id: I07862e30adf28404ec70a473571a9213e53d8a08
Partially-Implements: blueprint create-and-run-workflows-within-a-namespace
Signed-off-by: ali <ali.abdelal@nokia.com>
We are now using drop_index to delete old constraint (name + project_id)
in namespace support migration. It works only for MySQL, for PostgreSQL
we should use drop_constraint to delete old constraint.
drop_index does not affect PostgreSQL and drop_constraint does not affect
MySQL, so we should use both for namespace support for both databases.
Added old constraint (name + project_id) deletion for
PostgreSQL with new migration.
Change-Id: I137162dd2b643b24f02799c6a13e3b0c0a8d3fb5
Closes-bug: #1838635
This commit adds a possibility to delete db objects with
advanced filters, like 'in', 'eq', 'has' and so on.
Change-Id: I97bd3a5627f62081a2cb9487cc91920c5a11561b
If the cascade deletion error is identified, then resort to
recursively deleting the subworkflows before deleting the
given execution.
Closes-Bug: https://bugs.launchpad.net/mistral/+bug/1832300
Change-Id: I6e0d9db93e103eaf3a329dfa1b4153975c135008
* Some fields of execution objects can be large and, what's even more
important, their size is defined by user input. Making these fields
lazy-loaded reduced the amount of data loaded from DB in those
methods where ther are not needed. For example, "spec" field of the
WorkflowExecution class is rarely needed because the corresponding
specification Python object gets cached. It's proven to reduce
the execution time in some cases by ~5-10%.
* Fixed failing test
Change-Id: Ica0ba2ffa312891483745d276d04c95985c7f4c2
New function update_action_execution_heartbeat was introduced in db api
to optimize updating of heartbeats.
Now to perform the update mistral downloads an action execution object,
then modifies it, and finally uploads back to the db using ORM.
This function removes initial object fetching and changes ORM update to
sqlalchemy core.
Change-Id: I4cb2d85c193531648540a37caca9ee868ede3a08
This patch brings namespace support to workbooks.
Namespace of the workbook is inherited by workflows.
Implements: blueprint mistral-namespace-for-actions-workbooks
Change-Id: I2c66b3961915f0f35a9c468eb6dd0c0c70995234
* Added the "fields" argument to all "get" and "load" DB API
methods where we potentially need partial loading
* Removed an unused method
* Refactored some methods taking "namespace" as an argument
Closes-Bug: #1771518
Change-Id: I3eb094c24bc5309d05bf859c0b4d43487d50f1df
* Added "fields" argument to db_api.get_workflow_execution() method
so that we could load particular fields instead of the full object.
In many cases we don't need all object fields.
* Used this argument in the REST API method that deletes a workflow
execution because we only need the "state" field.
Change-Id: I36fd93d9869eec9428959e6848f8b441136c644e
Fix `columns` parameter in call to `model_query` when
sqlalchemy.api._get_collection called with insecure to
true, and specific fields defined.
Change-Id: Ib46173d42d794775cfa302856cf9703ce8549389
Closes-Bug: #1763068
When python-mistralclient filters it passes a dictionary not just the
string. This clears the dictionary and fix the search parameters into
lists for multiple tags to be searched if separeted by commas
like:
openstack workflow list --filter tags=tripleo-common-managed,foo
Change-Id: Ie082901fd7eb8b61a6306da1275a79aeb6c4a6de
Closes-Bug: #1716648
This patchset updates the DuplicateEntry exception messages to
provide useful information about what exactly the duplicate values
are. It also adds a test for duplicate workflows within a namespace,
and methods for testing assert error message contents.
Change-Id: I186e58e0d7c93d328070f32bd41c5ac79dcbda70
Closes-Bug: 1756443
* Moved `created_at` and `updated_at` to MistralModelBase to generate a
default value without microseconds
* Increased time delay in scheduler tests. This change doesn't affect
a test duration
* Removed the PostgreSQL database generation commands because there
already are in the `tools/test-setup.sh` script
* Downloaded the python MySQL driver
Change-Id: I50c924ee94619c6622fc553f05a929607646f1fe
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
* Corrected the path to the mistral config file in a unit test
* Minor improvement in the run_tests.sh script
* Fixed unstable the `test_rerun_join_with_branch_errors` and
test_rerun_on_join_task test. task3 id is properly passed now
* Removed unused `insert_or_ignore` api and tests. This api
broke the test in PostgreSQL case
* Passed correct value to the filter in
`test_filter_action_definitions_by_has_filter` test
Change-Id: I2ce3d2e1be001d9c0c76ebe5293b2bf578a7160b
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
Currently we can only do a CURD action on a cron-trigger by name.
This patch refer to workflow implementation and re-encapsulate
DB API so that users can manage a cron-trigger by id or name.
Closes-Bug: 1684469
Change-Id: I9ff657b2604647e734b5539e9bd6a524a3a20efb
Allow adding many workflows with the same name.
The way this works is by adding the new workflows under another
namespace.
Notice that:
1. Namespaces are not part of the mistral DSL.
2. When executing a workflow, the namespace passes down to the
subworkflow.
3. When searching for the sub-workflow definition -
If no workflow was found in the given namespace, than the defualt
namespace will be used.
4. The default namespace is an empty string.
5. The namespace property or the workflow execution will be under
params
Partially-Implements: blueprint create-and-run-workflows-within-a-namespace
Change-Id: Id248ec5906a0899d188675d002b45f6249d36d90
* Added get event triggers for all projects
and the corresponding rule in policy.json
* Fixed corresponding policy rule names for
event triggers API
Change-Id: I74e65a1cee8d074cfd637e460465922b610237c5
* Before this patch, "state_info" field of execution objects
wasn't truncated properly before saving into DB which led to
the DB error like:
DBDataError: (_mysql_exceptions.DataError) (1406, "Data too
long for column 'state_info' at row 1")
It was happening because the method utils.cut() didn't work
accurately enough in case if we passed it with a dictionary.
This patch doesn't fix utils.cut() method but it just saves
space for possible method result difference with the expected
length so that we make sure that the truncated string
representation is always less than 65536 bytes (i.e. the size
of MySQL TEXT type). The total difference is not critical
anyway.
Closes-Bug: #1698125
Change-Id: I18449710ace6276224aaea564588c53a3e2c6adc
We already supported role based api access control, this series patches
will implement resource access control for mistral, so that
administrator could define the rules of resource accessibility, e.g.
admin user could get/delete/update the workflows of other tenants
according to the policy.
Delete executions by admin already supported after following patch
merged: https://review.openstack.org/#/c/451160/
This patch adds some tests.
Partially implements: blueprint mistral-rbac
Change-Id: I918708d5bf76abdf2c2c08bb147fa50bd715f526
We already supported role based api access control, this series patches
will implement resource access control for mistral, so that
administrator could define the rules of resource accessibility, e.g.
admin user could get/delete/update the workflows of other tenants
according to the policy.
This patch supports admin user to update executions of other tenants.
Partially implements: blueprint mistral-rbac
Change-Id: Id8445d28dcc8adfa12588ec59a4b143bd018899b
- Admin user could update other user's workflow.
- Admin user could update system workflow. Before this change, it is
impossible to update system workflow.
TODO:
- Implement delete workflow by admin
- Implement for other resources(workfbook/execution/task/action, etc.)
Partially implements: blueprint mistral-rbac
Change-Id: Ia9372c9a4ffe1904e489513912596b9052ab3142
* This filter is needed for search functionality when we don't need
to rely on full match. Instead, we should be able to find entities
by inclusion of a value into a specified field.
Change-Id: I2133493e5b79b59994c7ef6b0e50b6ba1e1d37c6
Mysql is rounding the microseconds to one second,
which leads to time inconsistency beetwen what
is returned to cllient and what is stored in the DB.
This patch changes the behaviour, so that no mircoseconds
gets genereated.
Closes-bug: 1644881
Change-Id: I514c1d5154b3c658ec74c88b800d2a3ded1fdad9
* Workflow completion check logic used heavy select query which
actually returned objects whereas all we need is to count them.
This patch adds get_incompleted_task_executions_count() methods
on DB API to address this.
Change-Id: I796ba175210f41ded7877f310efeb30bd5045531
* In order to avoid duplicates of same 'join' tasks we now use
named locks to exclusively create a task execution for 'join'
tasks in DB. Transaction that does that is always separate
from task completion logic and is very short. This is needed
to eliminate DB contention on same records of task execution
table. This is also a reason to use a separate mechanism such
as named locks, and additionally this reduces a number possible
scenarios for getting into deadlocks because for task executions
we have too many different access patterns that can lead to them
in case of doing locking on right on their table records.
So this approach guarantees that there's only one transaction
creates a new task execution object for 'join' task and schedules
'refresh_task_state' job that check 'join' completion.
* Dropped scheduler 'unique_key' column with unique constraint
because in practice it causes DB deadlocks (at least on MySQL)
while simultaneously inserting and updating the table
* Instead of 'unique_key' column we added non-unique 'key' column
that can potentially be used for squashing delayed calls
by scheduler itself (not implemented yet)
* Adjusted Scheduler implementation and tests accordingly
* Fixed task() YAQL function to work without precisely resolve
task execution object in case it's called for the current
task. Previously it was dependent on the luck and we were
lucky enough that tests were passing.
* Increased length of 'unique_key' column for task executions to
250 which is close to a limit for string fields participating
in unique constraints.
Change-Id: Ib7aaa20c2c8834ab0f2d9c90457677c9edb62805
* We need this kind of transactional semaphore implemented in a
separate table to serialize access to some DB objects based on
object identifiers (such as name or other string value uniquely
identifying an object).
This approach is tested on MySQL and Postgres and according to
http://blog.udby.com/archives/14 is expected to work on DB2,
Oracle and possibly other databases.
The plan is to use named locks to prevent multiple insertions
of same join tasks.
Change-Id: I9823b986caa8d528ddbad640b437f6199269b637
* Having different types of execution objects in different
tables will give less contention on DB tables and hence better
performance so DB schema was changed accordingly
* Fixed all unit tests and places in the code where we assumed
polymorphic access to execution objects
* Other minor fixes
TODO(in upcoming patches):
* DB migration script
Change-Id: Ibc8408e12dd85e143302d7fdddace32954551ac5
Also added some database tests for get and delete action
definition by id.
Change-Id: I71dde2b91423ba1850653a076e241787d1f79572
Partially-Implements: blueprint mistral-actions-identifier
This patch contains the Mistral database and db_api changes
required for the event triggers.
Change-Id: Ib930ef8aac3792ed415e2817f67226962e6f7a92
Implements: blueprint event-notification-trigger
Co-Authored-By: Lingxian Kong <anlin.kong@gmail.com>
* Introduced new class Workflow that manages life-cycle of running
workflows and is responsible for managing workflow persistent state
* Moved all workflow level logic to workflow handler and Workflow class
* Changed semantics if how workflows start errors are handled.
Previously, in case of invalid user input Mistral engine would store
information about error in "state_info" field of workflow execution
and bubble up an exception to the user. This approach was incorrect
for a number of reasons including broken semantics: if an exception
was raised due to invalid input it's normal to expect that system
state has not changed. After this refactoring, engine only raises
an exception in case of bad user input. That way behavior is
consistent with the idea of exceptional situations.
* Fixed unit tests in according to the previous point
* Fixed a number of logical issues in tests. For example, in
test_default_engine.py we expected one type of errors (e.g. env not
found) but effectively received another one (invalid input).
Partially implements: blueprint mistral-engine-error-handling
Change-Id: I09070411fd833df8284cb80db69b8401a40eb6fe
Takashi Kajinami