ABCs in collections should be imported from collections.abc and direct
import from collections is deprecated since Python 3.3.
Closes-Bug: #1936667
Change-Id: Ide8aa0323d9713c1c2ea0abf3b671ca4dab95ef0
* There were references to Mistral specific constants mistakenly
placed into mistral-lib. This patch adds these constants into
the module where task language specifications are declared and
changes the corresponding references.
Change-Id: I8c7c6896f01894ac76cf9365abfdce17e7df7662
* This patch also adds handling for iterators in the sanitizing
function for YAQL results. JSON serialization for iterators now
works correctly but the issue is that they can be used only once.
So, for example, if we got a result from YAQL and performed JSON
serialization for it, then the iterator is already empty and
the data structure is just corrupted. So we can't further pass
to executors and other subsystems. The solution is just to sanitize
iterators after they're just returned from YAQL.
* Added a test to make sure that action input reaches an executor
not corrupted, although it gets saved into DB and hence serialized
into JSON.
* Changed the Sphinx entry in requirements.txt so that the version
3.0.0 is now excluded (it breaks the build).
Closes-Bug: #1871567
Change-Id: I47abe0904b49d72e33eb10080c71fb81980d44da
* When YAQL output data conversion is disabled there's still
an issue caused by presence of not JSON-compatible types within
a YAQL result. The internal Mistral code is already able to deal
with that (due to the previous changes) by checking that and
converting them to what's needed. However, JSON serialization
may still not work if it's done via the standard "json" library.
The library simply doesn't handle those non-standard types and
raises an exception. We have a sanitizing function that all YAQL
results go through, however, it doesn't make sense to do recursive
sanitizing for performance reasons. It does make sense to convert
data as late as possible to avoid redundant data manipulations. So
the sanitizing function handles only the root object in the object
graph. The solution for this problem is to use our own utility
function based on the "oslo_serialization.jsonutils" that is able
to deal with at least part of the mentioned types, specifically
FrozenDict and iterators. Generators are still a problem and this
new function takes care of that separately, assuming that any
generator is just a special iterator and hence represents a
collection, i.e. a list in JSON terms. It works for all the cases
we've encountered so far working with YAQL.
* Used the new function "utils.to_json_str()" everywhere for JSON
serialization, including the action "std.http".
* Added necessary unit tests.
Closes-Bug: #1869168
Depends-On: I1081a44a6f305eb1dfe68a5bad30110385130725
Change-Id: I9e73ea7cbba215c3e1d174b5189be27c640c4d42
* With disabled YAQL data output conversion, YAQL may return
instances of ContextView which can't be properly saved into
DB. This happens because Mistral serialization code doesn't
turn on JSON conversion of custom objects, and they are just
ignored by the "json" lib when it encounters them.
* Fixed how Mistral serializes context for Javascript evaluation
to address the same problem.
* Implemented __repr__ method of ContextView.
* Removed logging of "data_context" from YAQL evaluation because
previously it was always empty (because the string represetation
of ContextView was always "{}") and now it may be very big, like
megabytes, and the log gets populated too fast. It makes sense to
log YAQL data context only when an error happened. In this case
it helps to investigate an issue.
* Added all required unit tests.
* Fixed the tests for disabled YAQL conversion. In fact, they
didn't test it properly because data conversion wasn't disabled.
Closes-Bug: #1867899
Change-Id: I12b4d0c5f1f49990d8ae09b72f73c0da96254a86
* Functions like task() can be safely cached per one DB
transaction. It will decrease the number of DB hits.
This patch adds @tx_cached decorator that caches
any function result using a thread local cache having
same scope as a DB transaction.
* Style changes.
Closes-Bug: #1864823
Change-Id: Id00eae3cb8483dfe08f549ce7dd981b3c92df09a
* This patch moves code related to YAQL and Jinja into their
specific modules so that there isn't any module that works with
both. It makes it easier to understand how code related to one
of these technologies works.
* Custome built-in functions for YAQL and Jinja are now in a
separate module. It's easier now to see what's related with
the expression framework now and what's with integration part,
i.e. functions themselves.
* Renamed the base module of expressions similar to other packages.
* Other style changes.
Change-Id: I94f57a6534b9c10e202205dfae4d039296c26407
* When Mistral is launched via launch.py script then YAQL engine class
initialization logic doesn't use the right properties from the config
file. This is because this initialization is caused by the chain of
imports taking its start in launch.py, and this happens before
the config file is parsed in the main() function of launch.py.
Using lazy initialization of YAQL engine class (YAQL_ENGINE in
yaql_expression.py module) solves this issue because now Mistral
doesn't initialize it immediately before parsing the config and
waits for the first usage of it.
* Minor style changes per Mistral coding guidelines.
Change-Id: If3367493803b57ef8bc281b1f64f2a223ac86f85
Closes-Bug: #1864785
* Adding "convert_output_data" config property gives an opportunity
to increase overal performance. If YAQL always converts an expression
result, it often takes significant CPU time and overall workflow
execution time increases. It is especially important when a workflow
publishes lots of data into the context and uses big workflow
environments. It's been tested on a very big workflow (~7k tasks)
with a big workflow environment (~2.5mb) that often uses the YAQL
function "<% env() %>". This function basically just returns the
workflow environment.
* Created all necessary unit tests.
* Other style fixes.
Change-Id: Ie3169ec884ec9a0e7e50327dd03cd78dcda0a39b
* It turns out that osprofiler wasn't initialized properly for
threads in which scheduler runs its jobs. So profiling simply
didn't work for this threads and lots of important info didn't
get to the profiler log. This patch fixes it.
* When evaluating a YAQL expression, sometimes we get a huge result
value that we always put into the debug log. In practice, it doesn't
make much sense and, moreover, it utilizes lots of CPU and disk
space. It's better shrink it to some reasonable size that would
allow to make necessary analysis, if needed.
* Other minor style fixes according to the Mistral coding guidelines.
Change-Id: I3df3ab96342c456429e20a905615b90bcb94818d
Before changes `cls._env.from_string(expression).render(**ctx)` can
raise exception, for example the variable doesn't exist. And this
exception is not handled.
Jinja error handling was moved to cover this case.
Change-Id: Ib020550d0ec989ca382b738844a752ec9d02919a
Closes-bug: #1743649
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
Evaluate workflow names dynamically, so yaql or jinja expression
is allowed as sub-workflow name. Tasks names are not yet
dynamically evaluated.
Partially implements: blueprint mistral-dynamic-actions
Change-Id: Icfe591e27a4f45c2e3dcfa83512217f3b2122189
Jinja2 non-sandbox environment is unsafe as it gives
access to unsafe Python methods
Change-Id: If8a96bb92f64c4226a3d02e3cf6e0dcb0e9156fd
Closes-Bug: #1680112
Due to this invalid regex it is not possible to run workflows
with Jinja2 expressions in some python2 environment.
Change-Id: I3d8fe9a617d5e3f916a8c102d0408354064b766c
Closes-bug: #1658958
Allows to use Jinja instead of or along with YAQL for expression
evaluation.
* Improved error reporting on API endpoints. Previously, Mistral API
tend to mute important logs related to errors during YAML parsing
or expression evaluation. The messages were shown in the http
response, but would not appear in logs.
* Renamed yaql_utils to evaluation_utils and added few more tests to
ensure evaluation functions can be safely reused between Jinja and
YAQL evaluators.
* Updated action_v2 example to reflect similarities between YAQL and
Jinja syntax.
Change-Id: Ie3cf8b4a6c068948d6dc051b12a02474689cf8a8
Implements: blueprint mistral-jinga-templates