openstack
/
mistral
Code Issues Proposed changes

Added JavaScript evaluator which doesn't require a compilation 

* Added new JavaScript evaluator py_mini_racer. Advantages:
    * is distributed as wheel package
    * supports differences platforms
    * live project
* BUILD_V8EVAL was removed because it was replaced by py_mini_racer in
Mistral Docker image
* Added stevedore integration to javascript evaluators
* Refreshed javascript tests. Add test for py_mini_racer evaluator
* Install py_mini_racer library in during mistral test
* Refreshed javascript action doc

Change-Id: Id9d558b9b8374a2c2639e10cb1868f4e67f96e86
Implements: blueprint mistral-add-py-mini-racer-javascript-evaluator
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
This commit is contained in:
Vitalii Solodilov 2018-04-22 13:48:13 +04:00 committed by Vitalii Solodilov
parent 7adea45910
commit 5f89e2e71f
12 changed files with 136 additions and 114 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
doc/source/user/wf_lang_v2.rst
View File

@ -1108,30 +1108,23 @@ Input parameters:
- **script** - The text of JavaScript snippet that needs to be
executed. *Required*.
- **context** - This object will be assigned to the *$* javascript variable.
The default value is None.
**To use std.javascript, it is needed to install a number of
dependencies and JS engine.** Currently Mistral uses only V8 Engine and its
wrapper - PyV8. For installing it, do the next steps:
To use std.javascript, it is needed to install the
`py_mini_racer <https://github.com/sqreen/PyMiniRacer>`__ and set
*py_mini_racer* to *js_implementation* parameter in *mistral.conf*:
1. Install required libraries - boost, g++, libtool, autoconf, subversion,
libv8-legacy-dev: On Ubuntu::
.. code-block:: bash
$ sudo apt-get install libboost-all-dev g++ libtool autoconf libv8-legacy-dev subversion make
pip install py_mini_racer
2. Checkout last version of PyV8::
Other available implementations:
$ svn checkout http://pyv8.googlecode.com/svn/trunk/ pyv8
$ cd pyv8
- `pyv8 <https://code.google.com/archive/p/pyv8>`__
- `v8eval <https://github.com/sony/v8eval>`__
3. Build PyV8 - it will checkout last V8 trunk, build it, and then build PyV8::
$ sudo python setup.py build
4. Install PyV8::
$ sudo python setup.py install
Example:
Example with *context*:
.. code-block:: mistral
@ -1141,8 +1134,6 @@ Example:
generate_uuid:
  description: Generates a Universal Unique ID
  type: direct
  input:
    - radix: 16
@ -1151,7 +1142,7 @@ Example:
  tasks:
    generate_uuid_task:
      action: std.javascript
      action: std.js
      input:
        context: <% $ %>
        script: |
@ -1160,7 +1151,7 @@ Example:
                  return v.toString($.radix);
          });
      publish:
        generated_uuid: <% task(generate_uuid_task).result %>
        generated_uuid: <% task().result %>
Another example for getting the current date and time:
@ -1172,21 +1163,18 @@ Another example for getting the current date and time:
  get_date_workflow:
    description: Get the current date
    type: direct
    output:
      current_date: <% $.current_date %>
    tasks:
      get_date_task:
        action: std.javascript
        action: std.js
        input:
          context: <% $ %>
          script: |
            var date = new Date();
            return date; # returns "2015-07-12T10:32:12.460000" or use date.toLocaleDateString() for "Sunday, July 12, 2015"
            return date; // returns "2015-07-12T10:32:12.460000" or use date.toLocaleDateString() for "Sunday, July 12, 2015"
        publish:
          current_date: <% task(get_date_task).result %>
          current_date: <% task().result %>
Ad-hoc actions
^^^^^^^^^^^^^^

2
mistral/config.py
View File

@ -90,7 +90,7 @@ api_opts = [
js_impl_opt = cfg.StrOpt(
'js_implementation',
default='pyv8',
choices=['pyv8', 'v8eval'],
choices=['pyv8', 'v8eval', 'py_mini_racer'],
help=_('The JavaScript implementation to be used by the std.javascript '
'action to evaluate scripts.')
)

109
mistral/tests/unit/engine/test_javascript_action.py
View File

@ -14,10 +14,11 @@
import mock
from oslo_config import cfg
from oslo_utils import importutils
import testtools
from mistral.db.v2 import api as db_api
from mistral.services import workbooks as wb_service
from mistral.services import workflows as wf_service
from mistral.tests.unit.engine import base
from mistral.utils import javascript
from mistral.workflow import states
@ -28,83 +29,81 @@ from mistral.workflow import states
cfg.CONF.set_default('auth_enable', False, group='pecan')
WORKBOOK = """
---
JAVASCRIPT_WORKFLOW = """
version: "2.0"
wf:
input:
- length
tasks:
task1:
action: std.javascript
input:
script: |
let numberSequence = Array.from({length: $['length']},
(x, i) => i);
let evenNumbers = numberSequence.filter(x => x % 2 === 0);
name: test_js
workflows:
js_test:
type: direct
input:
- num
tasks:
task1:
description: |
This task reads variable from context,
increasing its value 10 times, writes result to context and
returns 100 (expected result)
action: std.javascript
input:
script: |
return $['num'] * 10
context: <% $ %>
publish:
result: <% task(task1).result %>
return evenNumbers.length;
context: <% $ %>
publish:
res: <% task().result %>
"""
def fake_evaluate(_, context):
return context['num'] * 10
return context['length'] / 2
class JavaScriptEngineTest(base.EngineTestCase):
@testtools.skip('It requires installed JS engine.')
def test_javascript_action(self):
wb_service.create_workbook_v2(WORKBOOK)
# Start workflow.
wf_ex = self.engine.start_workflow(
'test_js.js_test',
wf_input={'num': 50}
@testtools.skipIf(not importutils.try_import('py_mini_racer'),
'This test requires that py_mini_racer library was '
'installed')
def test_py_mini_racer_javascript_action(self):
cfg.CONF.set_default(
'js_implementation',
'py_mini_racer'
)
length = 1000
self.await_workflow_success(wf_ex.id)
# Note: We need to reread execution to access related tasks.
wf_ex = db_api.get_workflow_execution(wf_ex.id)
task_ex = wf_ex.task_executions[0]
self.assertEqual(states.SUCCESS, task_ex.state)
self.assertDictEqual({}, task_ex.runtime_context)
self.assertEqual(500, task_ex.published['num_10_times'])
self.assertEqual(100, task_ex.published['result'])
@mock.patch.object(javascript, 'evaluate', fake_evaluate)
def test_fake_javascript_action_data_context(self):
wb_service.create_workbook_v2(WORKBOOK)
wf_service.create_workflows(JAVASCRIPT_WORKFLOW)
# Start workflow.
wf_ex = self.engine.start_workflow(
'test_js.js_test',
wf_input={'num': 50}
'wf',
wf_input={'length': length}
)
self.await_workflow_success(wf_ex.id)
with db_api.transaction():
# Note: We need to reread execution to access related tasks.
wf_ex = db_api.get_workflow_execution(wf_ex.id)
task_ex = wf_ex.task_executions[0]
self.assertEqual(states.SUCCESS, task_ex.state)
self.assertDictEqual({}, task_ex.runtime_context)
self.assertEqual(500, task_ex.published['result'])
self.assertEqual(length / 2, task_ex.published['res'])
@mock.patch.object(javascript, 'evaluate', fake_evaluate)
def test_fake_javascript_action_data_context(self):
length = 1000
wf_service.create_workflows(JAVASCRIPT_WORKFLOW)
# Start workflow.
wf_ex = self.engine.start_workflow(
'wf',
wf_input={'length': length}
)
self.await_workflow_success(wf_ex.id)
with db_api.transaction():
wf_ex = db_api.get_workflow_execution(wf_ex.id)
task_ex = wf_ex.task_executions[0]
self.assertEqual(states.SUCCESS, task_ex.state)
self.assertDictEqual({}, task_ex.runtime_context)
self.assertEqual(length / 2, task_ex.published['res'])

54
mistral/utils/javascript.py
View File

@ -15,20 +15,32 @@
import abc
import json
from oslo_utils import importutils
from mistral import config as cfg
from mistral import exceptions as exc
from oslo_utils import importutils
from stevedore import driver
from stevedore import extension
_PYV8 = importutils.try_import('PyV8')
_V8EVAL = importutils.try_import('v8eval')
_PY_MINI_RACER = importutils.try_import('py_mini_racer.py_mini_racer')
_EVALUATOR = None
class JSEvaluator(object):
@classmethod
@abc.abstractmethod
def evaluate(cls, script, context):
"""Executes given JavaScript."""
"""Executes given JavaScript.
:param script: The text of JavaScript snippet that needs to be
executed.
context: This object will be assigned to the $ javascript
variable.
:return result of evaluated javascript code.
:raise MistralException: if corresponding js library is not installed.
"""
pass
@ -61,9 +73,39 @@ class V8EvalEvaluator(JSEvaluator):
encoding='UTF-8'))
EVALUATOR = (V8EvalEvaluator if cfg.CONF.js_implementation == 'v8eval'
else PyV8Evaluator)
class PyMiniRacerEvaluator(JSEvaluator):
@classmethod
def evaluate(cls, script, context):
if not _PY_MINI_RACER:
raise exc.MistralException(
"PyMiniRacer module is not available. Please install "
"PyMiniRacer."
)
ctx = _PY_MINI_RACER.MiniRacer()
return ctx.eval(('$ = {}; {}'.format(json.dumps(context), script)))
_mgr = extension.ExtensionManager(
namespace='mistral.expression.evaluators',
invoke_on_load=False
)
def get_js_evaluator():
global _EVALUATOR
if not _EVALUATOR:
mgr = driver.DriverManager(
'mistral.js.implementation',
cfg.CONF.js_implementation,
invoke_on_load=True
)
_EVALUATOR = mgr.driver
return _EVALUATOR
def evaluate(script, context):
return EVALUATOR.evaluate(script, context)
return get_js_evaluator().evaluate(script, context)

6
releasenotes/notes/add-py-mini-racer-javascript-evaluator-9d8f9e0e36504d72.yaml Normal file
View File

@ -0,0 +1,6 @@
---
features:
- |
Added new JavaScript evaluator py_mini_racer. The py_mini_racer package
allows us to get a JavaScript evaluator that doesn't require compilation.
This is much lighter and easier to get started with.

5
setup.cfg
View File

@ -114,3 +114,8 @@ kombu_driver.executors =
pygments.lexers =
mistral = mistral.ext.pygmentplugin:MistralLexer
mistral.js.implementation =
pyv8 = mistral.utils.javascript:PyV8Evaluator
v8eval = mistral.utils.javascript:V8EvalEvaluator
py_mini_racer = mistral.utils.javascript:PyMiniRacerEvaluator

9
tools/docker/DOCKER_README.rst
View File

@ -24,18 +24,11 @@ On the other hand you could execute the following command::
docker build -t mistral -f tools/docker/Dockerfile .
The Mistral Docker image has set of build parameters. **Pay attention**, the
compile of 'V8EVAL' can take a long time.
The Mistral Docker image has a build parameter.
+-------------------------+-------------+--------------------------------------+
|Name |Default value| Description |
+=========================+=============+======================================+
|`BUILD_V8EVAL` |true |If the `BUILD_V8EVAL` equals `true`, |
| | |the `v8eval` library will be build for|
| | |std.javascript action. `Read more <ht |
| | |tps://docs.openstack.org/mistral/lates|
| | |t/user/dsl_v2.html#std-javascript>`_ |
+-------------------------+-------------+----------------------+---------------+
|`BUILD_TEST_DEPENDENCIES`|false |If the `BUILD_TEST_DEPENDENCIES` |
| | |equals `true`, the Mistral test |
| | |dependencies will be installed inside |

11
tools/docker/Dockerfile
View File

@ -18,20 +18,13 @@ RUN apt-get -qq update && \
crudini \
curl \
git \
cmake \
gcc \
libuv1 \
swig \
mc \
libuv1-dev && \
curl -f -o /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py && \
python /tmp/get-pip.py && rm /tmp/get-pip.py
RUN pip install pymysql psycopg2
ARG BUILD_V8EVAL="true"
RUN if ${BUILD_V8EVAL} ; then \
pip install -v v8eval && python -c 'import v8eval' ; \
fi
RUN pip install pymysql psycopg2 py_mini_racer
ENV MISTRAL_DIR="/opt/stack/mistral" \
TMP_CONSTRAINTS="/tmp/upper-constraints.txt" \

5
tools/docker/docker-compose/mistral-multi-node.yaml
View File

@ -5,7 +5,6 @@ services:
context: ../../..
dockerfile: tools/docker/Dockerfile
args:
BUILD_V8EVAL: "false"
BUILD_TEST_DEPENDENCIES: "false"
restart: always
ports:
@ -27,7 +26,6 @@ services:
context: ../../..
dockerfile: tools/docker/Dockerfile
args:
BUILD_V8EVAL: "false"
BUILD_TEST_DEPENDENCIES: "false"
restart: always
networks:
@ -45,7 +43,6 @@ services:
context: ../../..
dockerfile: tools/docker/Dockerfile
args:
BUILD_V8EVAL: "false"
BUILD_TEST_DEPENDENCIES: "false"
restart: always
networks:
@ -62,7 +59,6 @@ services:
context: ../../..
dockerfile: tools/docker/Dockerfile
args:
BUILD_V8EVAL: "false"
BUILD_TEST_DEPENDENCIES: "false"
restart: always
networks:
@ -80,7 +76,6 @@ services:
context: ../../..
dockerfile: tools/docker/Dockerfile
args:
BUILD_V8EVAL: "false"
BUILD_TEST_DEPENDENCIES: "false"
restart: always
networks:

1
tools/docker/docker-compose/mistral-single-node.yaml
View File

@ -5,7 +5,6 @@ services:
context: ../../..
dockerfile: "tools/docker/Dockerfile"
args:
BUILD_V8EVAL: "false"
BUILD_TEST_DEPENDENCIES: "false"
restart: always
ports:

2
tools/docker/start.sh
View File

@ -8,7 +8,7 @@ if [ ! -f ${CONFIG_FILE} ]; then
--config-file "${MISTRAL_DIR}/tools/config/config-generator.mistral.conf" \
--output-file "${CONFIG_FILE}"
${INI_SET} DEFAULT js_implementation v8eval
${INI_SET} DEFAULT js_implementation py_mini_racer
${INI_SET} oslo_policy policy_file "${MISTRAL_DIR}/etc/policy.json"
${INI_SET} pecan auth_enable false
${INI_SET} DEFAULT transport_url "${MESSAGE_BROKER_URL}"

2
tox.ini
View File

@ -14,6 +14,8 @@ deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}
-r{toxinidir}/test-requirements.txt
-r{toxinidir}/requirements.txt
# javascript engine
py_mini_racer
commands =
rm -f .testrepository/times.dbm
find . -type f -name "*.pyc" -delete