Commit Graph

26 Commits

Author SHA1 Message Date
Takashi Kajinami 20aa42b75b Replace deprecated import of ABCs from collections
ABCs in collections should be imported from collections.abc and direct
import from collections is deprecated since Python 3.3.

Closes-Bug: #1936667
Change-Id: Ide8aa0323d9713c1c2ea0abf3b671ca4dab95ef0
2022-03-02 09:30:29 +00:00
Q.hongtao 4bc6162515 Remove six library
Remove six-library Replace the following items with Python 3 style code.
- six.interger_types
- six.itervalues
- six.text_type
- six.string_types
- six.StringIO
- six.next
- six.b
- six.PY3

Change-Id: I299c90d5cbeb41be0132691265b8dcbeae65520e
2020-09-23 10:27:12 +08:00
Renat Akhmerov 2ad7bb829b Moving task type constants into Mistral task specifications module
* There were references to Mistral specific constants mistakenly
  placed into mistral-lib. This patch adds these constants into
  the module where task language specifications are declared and
  changes the corresponding references.

Change-Id: I8c7c6896f01894ac76cf9365abfdce17e7df7662
2020-06-08 13:40:45 +07:00
Renat Akhmerov 6fb17d8266 Add YAQL sanitizing for iterators
* This patch also adds handling for iterators in the sanitizing
  function for YAQL results. JSON serialization for iterators now
  works correctly but the issue is that they can be used only once.
  So, for example, if we got a result from YAQL and performed JSON
  serialization for it, then the iterator is already empty and
  the data structure is just corrupted. So we can't further pass
  to executors and other subsystems. The solution is just to sanitize
  iterators after they're just returned from YAQL.
* Added a test to make sure that action input reaches an executor
  not corrupted, although it gets saved into DB and hence serialized
  into JSON.
* Changed the Sphinx entry in requirements.txt so that the version
  3.0.0 is now excluded (it breaks the build).

Closes-Bug: #1871567
Change-Id: I47abe0904b49d72e33eb10080c71fb81980d44da
2020-04-08 15:44:13 +07:00
Renat Akhmerov 0dbab33c4c Fix serialization of structures that might contain YAQL types
* When YAQL output data conversion is disabled there's still
  an issue caused by presence of not JSON-compatible types within
  a YAQL result. The internal Mistral code is already able to deal
  with that (due to the previous changes) by checking that and
  converting them to what's needed. However, JSON serialization
  may still not work if it's done via the standard "json" library.
  The library simply doesn't handle those non-standard types and
  raises an exception. We have a sanitizing function that all YAQL
  results go through, however, it doesn't make sense to do recursive
  sanitizing for performance reasons. It does make sense to convert
  data as late as possible to avoid redundant data manipulations. So
  the sanitizing function handles only the root object in the object
  graph. The solution for this problem is to use our own utility
  function based on the "oslo_serialization.jsonutils" that is able
  to deal with at least part of the mentioned types, specifically
  FrozenDict and iterators. Generators are still a problem and this
  new function takes care of that separately, assuming that any
  generator is just a special iterator and hence represents a
  collection, i.e. a list in JSON terms. It works for all the cases
  we've encountered so far working with YAQL.
* Used the new function "utils.to_json_str()" everywhere for JSON
  serialization, including the action "std.http".
* Added necessary unit tests.

Closes-Bug: #1869168
Depends-On: I1081a44a6f305eb1dfe68a5bad30110385130725
Change-Id: I9e73ea7cbba215c3e1d174b5189be27c640c4d42
2020-03-31 18:42:11 +07:00
Renat Akhmerov 019cffb3ab Fix ContextView JSON serialization
* With disabled YAQL data output conversion, YAQL may return
  instances of ContextView which can't be properly saved into
  DB. This happens because Mistral serialization code doesn't
  turn on JSON conversion of custom objects, and they are just
  ignored by the "json" lib when it encounters them.
* Fixed how Mistral serializes context for Javascript evaluation
  to address the same problem.
* Implemented __repr__ method of ContextView.
* Removed logging of "data_context" from YAQL evaluation because
  previously it was always empty (because the string represetation
  of ContextView was always "{}") and now it may be very big, like
  megabytes, and the log gets populated too fast. It makes sense to
  log YAQL data context only when an error happened. In this case
  it helps to investigate an issue.
* Added all required unit tests.
* Fixed the tests for disabled YAQL conversion. In fact, they
  didn't test it properly because data conversion wasn't disabled.

Closes-Bug: #1867899
Change-Id: I12b4d0c5f1f49990d8ae09b72f73c0da96254a86
2020-03-19 17:07:42 +07:00
Renat Akhmerov 756bd5566a Add caching for YAQL/Jinja expression functions
* Functions like task() can be safely cached per one DB
  transaction. It will decrease the number of DB hits.
  This patch adds @tx_cached decorator that caches
  any function result using a thread local cache having
  same scope as a DB transaction.
* Style changes.

Closes-Bug: #1864823
Change-Id: Id00eae3cb8483dfe08f549ce7dd981b3c92df09a
2020-02-27 17:42:03 +07:00
Renat Akhmerov 592981f487 Refactor expressions
* This patch moves code related to YAQL and Jinja into their
  specific modules so that there isn't any module that works with
  both. It makes it easier to understand how code related to one
  of these technologies works.
* Custome built-in functions for YAQL and Jinja are now in a
  separate module. It's easier now to see what's related with
  the expression framework now and what's with integration part,
  i.e. functions themselves.
* Renamed the base module of expressions similar to other packages.
* Other style changes.

Change-Id: I94f57a6534b9c10e202205dfae4d039296c26407
2020-02-26 12:36:34 +07:00
Renat Akhmerov 753f1bc03f Fix YAQL engine initialization
* When Mistral is launched via launch.py script then YAQL engine class
  initialization logic doesn't use the right properties from the config
  file. This is because this initialization is caused by the chain of
  imports taking its start in launch.py, and this happens before
  the config file is parsed in the main() function of launch.py.
  Using lazy initialization of YAQL engine class (YAQL_ENGINE in
  yaql_expression.py module) solves this issue because now Mistral
  doesn't initialize it immediately before parsing the config and
  waits for the first usage of it.
* Minor style changes per Mistral coding guidelines.

Change-Id: If3367493803b57ef8bc281b1f64f2a223ac86f85
Closes-Bug: #1864785
2020-02-26 11:53:47 +07:00
Renat Akhmerov 8d75784356 Add "convert_output_data" config property for YAQL
* Adding "convert_output_data" config property gives an opportunity
  to increase overal performance. If YAQL always converts an expression
  result, it often takes significant CPU time and overall workflow
  execution time increases. It is especially important when a workflow
  publishes lots of data into the context and uses big workflow
  environments. It's been tested on a very big workflow (~7k tasks)
  with a big workflow environment (~2.5mb) that often uses the YAQL
  function "<% env() %>". This function basically just returns the
  workflow environment.
* Created all necessary unit tests.
* Other style fixes.

Change-Id: Ie3169ec884ec9a0e7e50327dd03cd78dcda0a39b
2020-02-13 17:31:41 +07:00
Renat Akhmerov 511be4f98f Initialize profiler for scheduler threads
* It turns out that osprofiler wasn't initialized properly for
  threads in which scheduler runs its jobs. So profiling simply
  didn't work for this threads and lots of important info didn't
  get to the profiler log. This patch fixes it.
* When evaluating a YAQL expression, sometimes we get a huge result
  value that we always put into the debug log. In practice, it doesn't
  make much sense and, moreover, it utilizes lots of CPU and disk
  space. It's better shrink it to some reasonable size that would
  allow to make necessary analysis, if needed.
* Other minor style fixes according to the Mistral coding guidelines.

Change-Id: I3df3ab96342c456429e20a905615b90bcb94818d
2020-01-30 16:23:56 +07:00
ali 7e7f1cb92b moved generic util functions from mistral to mistral-lib
Depends-On: I780c270e4b1a184d7d4dcc580d23697ba75edab1
Closes-bug: #1815183
Change-Id: I5a1d402baa3f69c37f9347c8b3d02a83b8f60423
2019-09-13 04:06:27 +00:00
Xavier Hardy fe0d441082 Add YAQL engine options
Change-Id: I2e2c85c8be90ee62c8f37f002e21098f17ba6f5c
Closes-Bug: #1700572
Closes-Bug: #1772864
2018-06-01 17:06:57 +07:00
Vitalii Solodilov e1f105685a Fixed Jinja error handling
Before changes `cls._env.from_string(expression).render(**ctx)` can
raise exception, for example the variable doesn't exist. And this
exception is not handled.
Jinja error handling was moved to cover this case.

Change-Id: Ib020550d0ec989ca382b738844a752ec9d02919a
Closes-bug: #1743649
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
2018-04-23 14:02:34 +04:00
Renat Akhmerov bb0f9f9fd1 Re-raise DB errors when evaluating expressions
* See details in the bug description

Change-Id: Id8c00c5d1196b72c08aa1d518b25a08290868693
Closes-Bug: #1729582
2017-11-03 11:24:29 +07:00
Nick Maludy 10b27fcdea Fixes issue rendering strings containing multiple jinja expressions
Closes-Bug: #1712133
Change-Id: I058e9858e4a4a9a94e0948df38758446e68bf5b9
2017-09-09 11:59:49 -04:00
Istvan Imre ff78d7f659 Dynamic workflow name evaluation.
Evaluate workflow names dynamically, so yaql or jinja expression
is allowed as sub-workflow name. Tasks names are not yet
dynamically evaluated.

Partially implements: blueprint mistral-dynamic-actions

Change-Id: Icfe591e27a4f45c2e3dcfa83512217f3b2122189
2017-08-16 13:17:10 +02:00
Sharat Sharma b38dceec50 Add a hacking rule for string interpolation at logging
From [1], string interpolation should be delayed to be handled by
the logging code, rather than being done at the point of the logging
call. Add a hacking rule, [M331], to delay the string interpolation.

[1] https://docs.openstack.org/oslo.i18n/latest/user/guidelines.html#adding-variables-to-log-messages
Change-Id: Ibf0ac960ae84e935979f93f3a0dec8d774aade78
Closes-Bug: #1705634
2017-07-26 11:13:15 +00:00
Xavier Hardy fc12891256 Use Jinja2 sandbox environment
Jinja2 non-sandbox environment is unsafe as it gives
access to unsafe Python methods

Change-Id: If8a96bb92f64c4226a3d02e3cf6e0dcb0e9156fd
Closes-Bug: #1680112
2017-04-10 15:36:23 +02:00
Lingxian Kong b563e5898f Trim yaql/jinja operation log
Only keep the top level validation/evaluation log is enough for
debugging.

Change-Id: I22d18aeeafcf68aaa6f55da1c943b282f001ef82
2017-03-22 11:08:03 +13:00
Istvan Imre 5267e1eb1c Invalid jinja pattern regex corrected
Due to this invalid regex it is not possible to run workflows
with Jinja2 expressions in some python2 environment.

Change-Id: I3d8fe9a617d5e3f916a8c102d0408354064b766c
Closes-bug: #1658958
2017-01-24 11:26:08 +01:00
Renat Akhmerov 764f5298dc Make Jinja evaluator catch and wrap all underlying exceptions
Change-Id: Ie45dcc51519a839a0983860711cba48fa4a112a1
Closes-Bug: 1642574
2016-11-18 17:11:54 +07:00
Renat Akhmerov 1ae33591fc Make YAQL evaluator catch and wrap all underlying exceptions
Change-Id: I8bdfd00dfd2ba9e86adc9e5961e5736cd792cf41
Partial-Bug: 1642574
2016-11-18 15:44:57 +07:00
Renat Akhmerov 07039d3ab3 Fix error message format for task run and continue
* Error message format is fixed so that an error goes before
  other contextual information

Change-Id: Ic8deed8679c6631fd0b797b46715e012033c7db4
2016-11-10 14:55:33 +07:00
Renat Akhmerov 91068b5f08 Change format of YAQL errors
* See the bug for details
* Style changes in Jinja expressions module

Partial-Bug: #1630473
Change-Id: Iad0720ce33aeefb99cc20d102535498203deb3d4
2016-10-19 17:46:51 +07:00
Kirill Izotov 362c2295e8 Add Jinja evaluator
Allows to use Jinja instead of or along with YAQL for expression
evaluation.

 * Improved error reporting on API endpoints. Previously, Mistral API
   tend to mute important logs related to errors during YAML parsing
   or expression evaluation. The messages were shown in the http
   response, but would not appear in logs.

 * Renamed yaql_utils to evaluation_utils and added few more tests to
   ensure evaluation functions can be safely reused between Jinja and
   YAQL evaluators.

 * Updated action_v2 example to reflect similarities between YAQL and
   Jinja syntax.

Change-Id: Ie3cf8b4a6c068948d6dc051b12a02474689cf8a8
Implements: blueprint mistral-jinga-templates
2016-10-05 11:27:29 +00:00