Improve the Docker integration

* Removed old scripts
* Added docker-compose files
* Added .dockerignore
* Updated Docker docs
* Added cloud-flow container to docker-compose
* PostgreSQL and MySQL were supported

Change-Id: Ia7b6d7faaff6d9aa0f02d64d569250f6a9ee59ea
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>

.dockerignore

@@ -0,0 +1,71 @@
+api-ref/
+devstack/
+doc/
+functionaltests/
+mistral.egg-info/
+playbooks/
+rally-jobs/
+releasenotes/
+envs/
+
+
+*.py[cod]
+*.sqlite
+
+# C extensions
+*.so
+
+# Packages
+*.egg*
+dist
+build
+.venv
+eggs
+parts
+bin
+var
+sdist
+develop-eggs
+.installed.cfg
+lib
+lib64
+
+# Installer logs
+pip-log.txt
+
+# Unit test / coverage reports
+.coverage
+.tox
+nosetests.xml
+cover/*
+.testrepository/
+subunit.log
+.mistral.conf
+AUTHORS
+ChangeLog
+
+# Translations
+*.mo
+
+# Mr Developer
+.mr.developer.cfg
+.project
+.pydevproject
+.idea
+.DS_Store
+etc/*.conf
+etc/mistral.conf.sample
+
+#Linux swap files range from .saa to .swp
+*.s[a-w][a-p]
+
+# Files created by releasenotes build
+releasenotes/build
+
+# Files created by doc build
+doc/build/
+doc/source/api
+doc/source/_static/
+
+# Files created by API build
+api-ref/build/

tools/docker/DOCKER_README.rst

@@ -1,4 +1,4 @@
-Using Mistral with docker
+Using Mistral with Docker
 =========================
 
 In order to minimize the work needed to run the current Mistral code, or
@@ -10,80 +10,125 @@ to launch an all-in-one Mistral container.
 Docker installation
 -------------------
 
-In order to install the latest docker engine, run::
+The links help you to install latest Docker software:
 
-  curl -fsSL https://get.docker.com/ | sh
+* `Docker Engine <https://docs.docker.com/engine/installation/>`_
-
+* `Docker Compose <https://docs.docker.com/compose/install/>`_
-If you are behind a proxy, additional configuration may be needed to be
-able to execute further steps in the setup process. For detailed information
-on this process, check out `the official guide at
-<http://www.sqlite.org/omitted.html>`_.
 
 
-Build the Mistral image
+Build the Mistral image manually
------------------------
+--------------------------------
 
 The `build.sh` script takes care of creating the `mistral-all` image locally.
-This is image is configured to use  RabbitMQ for transport and MySQL as database
+On the other hand you could execute the following command::
-backend. It is possible to run Mistral with Sqlite as database backend but
+
-it is very unreliable, thus, MySQL was selected as the default database backend
+  docker build -t mistral -f tools/docker/Dockerfile .
-for this image.
+
+The Mistral Docker image has set of build parameters. **Pay attention**, the
+compile of 'V8EVAL'  can take a long time.
+
++-------------------------+-------------+--------------------------------------+
+|Name                     |Default value| Description                          |
++=========================+=============+======================================+
+|`BUILD_V8EVAL`           |true         |If the `BUILD_V8EVAL` equals `true`,  |
+|                         |             |the `v8eval` library will be build for|
+|                         |             |std.javascript action. `Read more <ht |
+|                         |             |tps://docs.openstack.org/mistral/lates|
+|                         |             |t/user/dsl_v2.html#std-javascript>`_  |
++-------------------------+-------------+----------------------+---------------+
+|`BUILD_TEST_DEPENDENCIES`|false        |If the `BUILD_TEST_DEPENDENCIES`      |
+|                         |             |equals `true`, the Mistral test       |
+|                         |             |dependencies will be installed inside |
+|                         |             |the Docker image                      |
++-------------------------+-------------+----------------------+---------------+
 
 
-Running Mistral with MySQL
+Running Mistral using Docker Compose
---------------------------
+------------------------------------
 
-The `start_mistral_rabbit_mysql.sh` script sets up a rabbitmq container, a
+To launch Mistral in the single node configuration::
-mysql container and a mistral container to work together.
 
-The script can be invoked with::
+  docker-compose -f tools/docker/docker-compose/infrastructure.yaml \
+               -f tools/docker/docker-compose/mistral-single-node.yaml \
+               -p mistral up -d
 
-  start_mistral_rabbit_mysql.sh [single|multi]
+To launch Mistral in the multi node configuration::
 
-`single` mode (this is the default) will create
+  docker-compose -f tools/docker/docker-compose/infrastructure.yaml \
+               -f tools/docker/docker-compose/mistral-multi-node.yaml \
+               -p mistral up -d
 
- - rabbitmq container,
+The infrastructure docker-compose file contains a example of RabbitMQ,
- - the mysql container,
+PostgreSQL and MySQL. Fill free to modify docker-compose files.
- - and the mistral container that runs all Mistral services.
+
+Also the docker-compose contains the Clould-flow container.
+It is available by `link <http://localhost:8000/>`_
+
+If you want to rebuild image you should add `--build` option, for example::
+
+  docker-compose -f tools/docker/docker-compose/infrastructure.yaml \
+               -f tools/docker/docker-compose/mistral-single-node.yaml \
+               -p mistral up -d --build
+
+Configuring Mistral
+-------------------
+
+The Docker image contains the minimal set of Mistral configuration parameters
+by default:
+
++--------------------+------------------+--------------------------------------+
+|Name                |Default value     | Description                          |
++====================+==================+======================================+
+|`MESSAGE_BROKER_URL`|rabbit://guest:gu\|The message broker URL                |
+|                    |est@rabbitmq:5672 |                                      |
++--------------------+------------------+----------------------+---------------+
+|`DATABASE_URL`      |sqlite://mistral.\|The database URL                      |
+|                    |db                |                                      |
++--------------------+------------------+----------------------+---------------+
+|`UPGRADE_DB`        |false             |If the `UPGRADE_DB` equals `true`,    |
+|                    |                  |a database upgrade will be launched   |
+|                    |                  |before Mistral main process           |
++--------------------+------------------+----------------------+---------------+
+|`MISTRAL_SERVER`    |all               |Specifies which mistral server to     |
+|                    |                  |start by the launch script.           |
++--------------------+------------------+----------------------+---------------+
+|`LOG_DEBUG`         |false             |If set to true, the logging level will|
+|                    |                  |be set to DEBUG instead of the default|
+|                    |                  |INFO level.                           |
++--------------------+------------------+----------------------+---------------+
+|`RUN_TESTS`         |false             |If the `UPGRADE_DB` equals `true`,    |
+|                    |                  |the Mistral unit tests will be        |
+|                    |                  |launched inside container             |
++--------------------+------------------+----------------------+---------------+
+
+Other way you can mount the your config file to a Mistral Docker container.
+You should uncomment the volume sections in the docker-compose files.
 
 
-`multi` mode will create
+Launch tests inside Container
+-----------------------------
 
- - rabbitmq,
+Build mistral::
- - mysql,
- - mistral-api,
- - one mistral-engine,
- - two mistral-executors
 
-Check out the script for more detail and examples for different setup options.
+  docker build -t mistral -f tools/docker/Dockerfile \
+        --build-arg BUILD_TEST_DEPENDENCIES=true .
 
-Using Mistral
+Run tests using SQLite::
--------------
 
-Depending on the mode, you may need to use the `mistral` or the `mistral-api`
+  docker run -it -e RUN_TESTS=true mistral
-container.
 
-With the `multi` option execute commands inside the container::
+or PostgreSQL::
 
-  docker exec -it mistral-api bash
+  docker run -it \
+    -e DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres \
+    -e RUN_TESTS=true mistral
 
-E.g. to list workflows, issue::
 
-  mistral workflow-list
+Using Mistral Client
+--------------------
 
 The script also configures the containers so that the Mistral API will be
 accessible from the host machine on the default port 8989. So it is also
 possible to install the `mistral-pythonclient` to the host machine and
 execute commands there.
 
-Configuring Mistral
--------------------
-
-The Mistral configuration is stored in the Docker image. The changes to the
-configuration should be synchronized between all deployed containers to
-ensure consistent behavior. This can be achieved by mounting the configuration
-as a volume::
-
-  export EXTRA_OPTS='-v <path to local mistral.conf>:/etc/mistral/mistral.conf:ro'
-  start_mistral_rabbit_mysql.sh multi
-

tools/docker/Dockerfile

@@ -1,53 +1,70 @@
 FROM krallin/ubuntu-tini:16.04
 
-MAINTAINER Andras Kovi <akovi@nokia.com>
+LABEL name="Mistral" \
+      description="Workflow Service for OpenStack" \
+      maintainers="Andras Kovi <akovi@nokia.com> \
+                   Vitalii Solodilov <mcdkr@yandex.ru>"
 
-RUN export DEBIAN_FRONTEND=noninteractive && \
+RUN apt-get -qq update && \
-    apt-get -qq update && \
     apt-get install -y  \
-      curl \
-      git \
       libffi-dev \
       libssl-dev \
       libxml2-dev \
       libxslt1-dev \
       libyaml-dev \
-      mc \
+      libmysqlclient-dev \
+      python \
       python-dev \
-      python-pip \
-      python-setuptools \
-      swig \
-      cmake \
       crudini \
+      curl \
+      git \
+      cmake \
       libuv1 \
-      libuv1-dev
+      swig \
+      mc \
+      libuv1-dev && \
+      curl -f -o /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py && \
+      python /tmp/get-pip.py && rm /tmp/get-pip.py
 
-RUN pip install -v v8eval && python -c 'import v8eval'
+RUN pip install pymysql psycopg2
 
-RUN apt-get install -y libmysqlclient-dev && \
+ARG BUILD_V8EVAL="true"
-    pip install mysql-python
+RUN if ${BUILD_V8EVAL} ; then \
+        pip install -v v8eval && python -c 'import v8eval' ; \
+    fi
 
-RUN pip install -U tox python-mistralclient pip
+ENV MISTRAL_DIR="/opt/stack/mistral" \
+    TMP_CONSTRAINTS="/tmp/upper-constraints.txt" \
+    CONFIG_FILE="/etc/mistral/mistral.conf" \
+    INI_SET="crudini --set /etc/mistral/mistral.conf" \
+    MESSAGE_BROKER_URL="rabbit://guest:guest@rabbitmq:5672/" \
+    DATABASE_URL="sqlite://mistral.db" \
+    UPGRADE_DB="false" \
+    RUN_TESTS="false" \
+    DEBIAN_FRONTEND="noninteractive" \
+    MISTRAL_SERVER="all" \
+    LOG_DEBUG="false"
 
-COPY . /opt/stack/mistral
+# We install dependencies separatly for a caching purpose
+COPY requirements.txt "${MISTRAL_DIR}/"
+RUN curl -o "${TMP_CONSTRAINTS}" \
+    http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt && \
+    sed -i "/^mistral.*/d" "${TMP_CONSTRAINTS}" && \
+    pip install -r "${MISTRAL_DIR}/requirements.txt"
 
-RUN curl -o /tmp/upper-constraints.txt http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt && \
+ARG BUILD_TEST_DEPENDENCIES="false"
-    sed -i '/^mistral.*/d' /tmp/upper-constraints.txt &&\
+COPY test-requirements.txt "${MISTRAL_DIR}/"
-    pip install -e /opt/stack/mistral
+RUN if ${BUILD_TEST_DEPENDENCIES} ; then \
+        pip install -r "${MISTRAL_DIR}/test-requirements.txt" ; \
+    fi
 
-RUN mkdir -p /etc/mistral
+COPY . ${MISTRAL_DIR}
 
-RUN oslo-config-generator \
+RUN pip install -e "${MISTRAL_DIR}" && \
-      --config-file /opt/stack/mistral/tools/config/config-generator.mistral.conf \
+    mkdir /etc/mistral && \
-      --output-file /etc/mistral/mistral.conf
+    rm -rf /var/lib/apt/lists/* && \
-
+    find ${MISTRAL_DIR} -name "*.sh" -exec chmod +x {} \;
-RUN INI_SET="crudini --set /etc/mistral/mistral.conf" && \
-    $INI_SET DEFAULT js_implementation v8eval && \
-    $INI_SET DEFAULT transport_url rabbit://guest:guest@rabbitmq:5672/ && \
-    $INI_SET database connection mysql://root:strangehat@mysql:3306/mistral && \
-    $INI_SET oslo_policy policy_file /opt/stack/mistral/etc/policy.json && \
-    $INI_SET pecan auth_enable false
 
+WORKDIR "${MISTRAL_DIR}"
 EXPOSE 8989
-
+CMD "${MISTRAL_DIR}/tools/docker/start.sh"
-CMD mistral-server --server all

tools/docker/build.sh

@@ -1,9 +0,0 @@
-#!/bin/bash -xe
-
-SCRIPT_DIR="$(dirname "$(readlink -e "${BASH_SOURCE[0]}")")"
-
-(
-    cd "$SCRIPT_DIR"
-
-    docker build -t mistral-all -f Dockerfile ../..
-)

tools/docker/docker-compose/infrastructure.yaml

@@ -0,0 +1,53 @@
+version: '3'
+services:
+#  postgresql:
+#    image: postgres:10.1-alpine
+#    restart: always
+#    ports:
+#      - "5432:5432"
+#    volumes:
+#      - postgresql:/var/lib/postgresql/data
+#    networks:
+#      - database
+#    environment:
+#      - POSTGRES_PASSWORD=mistral
+#      - POSTGRES_USER=mistral
+#      - POSTGRES_DB=mistral
+
+  rabbitmq:
+    image: rabbitmq:3.7.2-management-alpine
+    restart: always
+    ports:
+      - "15672:15672"
+    networks:
+      - message-broker
+    hostname: rabbitmq
+    environment:
+      - RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.81
+      - RABBITMQ_DEFAULT_USER=mistral
+      - RABBITMQ_DEFAULT_PASS=mistral
+      - RABBITMQ_DEFAULT_VHOST=mistral
+
+  mysql:
+    image: mysql:8.0.3
+    restart: always
+    ports:
+      - "3306:3306"
+    volumes:
+      - mysql:/var/lib/mysql
+    networks:
+      - database
+    environment:
+      - MYSQL_ROOT_PASSWORD=mistral
+      - MYSQL_DATABASE=mistral
+      - MYSQL_USER=mistral
+      - MYSQL_PASSWORD=mistral
+
+volumes:
+  postgresql:
+  rabbitmq:
+  mysql:
+
+networks:
+  database:
+  message-broker:

tools/docker/docker-compose/mistral-multi-node.yaml

@@ -0,0 +1,111 @@
+version: '3'
+services:
+  mistral-api:
+    build:
+      context: ../../..
+      dockerfile: tools/docker/Dockerfile
+      args:
+        BUILD_V8EVAL: "false"
+        BUILD_TEST_DEPENDENCIES: "false"
+    restart: always
+    ports:
+      - "8989:8989"
+    networks:
+      - database
+      - message-broker
+      - mistral
+    env_file:
+     - mistral.env
+#    volumes:
+#      - "/path/to/mistral.conf:/etc/mistral/mistral.conf"
+    environment:
+      - MISTRAL_SERVER=api
+      - UPGRADE_DB=true
+
+  mistral-engine:
+    build:
+      context: ../../..
+      dockerfile: tools/docker/Dockerfile
+      args:
+        BUILD_V8EVAL: "false"
+        BUILD_TEST_DEPENDENCIES: "false"
+    restart: always
+    networks:
+      - database
+      - message-broker
+    env_file:
+     - mistral.env
+#    volumes:
+#      - "/path/to/mistral.conf:/etc/mistral/mistral.conf"
+    environment:
+      - MISTRAL_SERVER=engine
+
+  mistral-executor:
+    build:
+      context: ../../..
+      dockerfile: tools/docker/Dockerfile
+      args:
+        BUILD_V8EVAL: "false"
+        BUILD_TEST_DEPENDENCIES: "false"
+    restart: always
+    networks:
+      - message-broker
+    env_file:
+     - mistral.env
+#    volumes:
+#      - "/path/to/mistral.conf:/etc/mistral/mistral.conf"
+    environment:
+      - MISTRAL_SERVER=executor
+
+  mistral-event-engine:
+    build:
+      context: ../../..
+      dockerfile: tools/docker/Dockerfile
+      args:
+        BUILD_V8EVAL: "false"
+        BUILD_TEST_DEPENDENCIES: "false"
+    restart: always
+    networks:
+      - database
+      - message-broker
+    env_file:
+     - mistral.env
+#    volumes:
+#      - "/path/to/mistral.conf:/etc/mistral/mistral.conf"
+    environment:
+      - MISTRAL_SERVER=event-engine
+
+  mistral-notifier:
+    build:
+      context: ../../..
+      dockerfile: tools/docker/Dockerfile
+      args:
+        BUILD_V8EVAL: "false"
+        BUILD_TEST_DEPENDENCIES: "false"
+    restart: always
+    networks:
+      - database
+      - message-broker
+    env_file:
+     - mistral.env
+#    volumes:
+#      - "/path/to/mistral.conf:/etc/mistral/mistral.conf"
+    environment:
+      - MISTRAL_SERVER=notifier
+
+  cloud-flow:
+    image: mcdoker18/cloud-flow:0.5.0
+    restart: always
+    networks:
+      - mistral
+    ports:
+      - "8000:8000"
+    networks:
+      - mistral
+    environment:
+      - CF_MISTRAL_URL=http://mistral-api:8989
+
+networks:
+  database:
+  message-broker:
+  mistral:

tools/docker/docker-compose/mistral-single-node.yaml

@@ -0,0 +1,40 @@
+version: '3'
+services:
+  mistral:
+    build:
+      context: ../../..
+      dockerfile: "tools/docker/Dockerfile"
+      args:
+        BUILD_V8EVAL: "false"
+        BUILD_TEST_DEPENDENCIES: "false"
+    restart: always
+    ports:
+      - "8989:8989"
+    networks:
+      - database
+      - message-broker
+      - mistral
+    env_file:
+     - mistral.env
+#    volumes:
+#      - "/path/to/mistral.conf:/etc/mistral/mistral.conf"
+    environment:
+      - UPGRADE_DB=true
+
+  cloud-flow:
+    image: mcdoker18/cloud-flow:0.5.0
+    restart: always
+    networks:
+      - mistral
+    ports:
+      - "8000:8000"
+    networks:
+      - mistral
+    environment:
+      - CF_MISTRAL_URL=http://mistral:8989
+
+
+networks:
+  database:
+  message-broker:
+  mistral:

tools/docker/docker-compose/mistral.env

@@ -0,0 +1,3 @@
+MESSAGE_BROKER_URL=rabbit://mistral:mistral@rabbitmq:5672/mistral
+#DATABASE_URL=postgresql+psycopg2://mistral:mistral@postgresql:5432/mistral
+DATABASE_URL=mysql+pymysql://mistral:mistral@mysql:3306/mistral

tools/docker/start.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+set -e
+
+# If a Mistral config doesn't exist we should create it and fill in with
+# parameters
+if [ ! -f ${CONFIG_FILE} ]; then
+    oslo-config-generator \
+      --config-file "${MISTRAL_DIR}/tools/config/config-generator.mistral.conf" \
+      --output-file "${CONFIG_FILE}"
+
+    ${INI_SET} DEFAULT js_implementation v8eval
+    ${INI_SET} oslo_policy policy_file "${MISTRAL_DIR}/etc/policy.json"
+    ${INI_SET} pecan auth_enable false
+    ${INI_SET} DEFAULT transport_url "${MESSAGE_BROKER_URL}"
+    ${INI_SET} database connection "${DATABASE_URL}"
+    ${INI_SET} DEFAULT debug "${LOG_DEBUG}"
+fi
+
+if "${UPGRADE_DB}";
+then
+    mistral-db-manage --config-file "${CONFIG_FILE}" upgrade head
+    mistral-db-manage --config-file "${CONFIG_FILE}" populate
+fi
+
+if "${RUN_TESTS}";
+then
+    cp "${CONFIG_FILE}" .mistral.conf
+    "${MISTRAL_DIR}/run_tests.sh" -N
+else
+    mistral-server --config-file "${CONFIG_FILE}" --server ${MISTRAL_SERVER}
+fi

tools/docker/start_mistral_rabbit_mysql.sh

@@ -1,111 +0,0 @@
-#! /bin/bash -e
-
-if [ "${1}" == "--help" ]; then
-    echo '
-    Synopsis:
-
-       start_mistral_rabbit_mysql.sh [single|multi|clean]
-
-    Environment variables:
-
-    EXTRA_OPTS : extra parameters to be used for all mistral containers (e.g. -v)
-    MYSQL_ROOT_PASSWORD : password for the MySQL server
-    SCRATCH : remove all existing containers (RabbitMQ and MySQL are not removed by default)
-    '
-    exit 0
-fi
-
-MODE=${1:-single}
-
-export MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-strangehat}
-
-MISTRAL_CONTAINERS=$(docker ps -a --format '{{.ID}} {{.Names}}' | grep mistral || true)
-
-if [ -z "$SCRATCH" -a "$MODE" != 'clean' ]; then
-    MISTRAL_CONTAINERS=$(echo "$MISTRAL_CONTAINERS" | grep -v rabbitmq | grep -v mysql | cat)
-fi
-
-if [ -n "$MISTRAL_CONTAINERS" ]; then
-    echo "Removing existing containers: $MISTRAL_CONTAINERS"
-    KILLED_CONTAINERS=$(echo "$MISTRAL_CONTAINERS" | awk '{print $1}')
-    docker kill -s 9 $KILLED_CONTAINERS
-    docker rm $KILLED_CONTAINERS
-fi
-
-if [ "$MODE" == 'clean' ]; then
-    echo "Clean complete"
-    exit 0
-fi
-
-if [ -z "$(docker ps -aq --filter "Name=mistral-mysql")" ]; then
-    docker create --name mistral-mysql -e MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD} mysql
-fi
-docker start mistral-mysql
-if [ -z "$(docker ps -aq --filter "Name=mistral-rabbitmq")" ]; then
-    docker create --name mistral-rabbitmq rabbitmq
-fi
-docker start mistral-rabbitmq
-
-while true; do
-  sleep 5
-  docker exec mistral-mysql \
-    mysql -u root -pstrangehat \
-    -e "CREATE DATABASE IF NOT EXISTS mistral;
-        USE mistral;
-        GRANT ALL ON mistral.* TO 'root'@'%' IDENTIFIED BY '${MYSQL_ROOT_PASSWORD}'" \
-  && break || true
-done
-
-sleep 10
-
-docker run -dit --link mistral-mysql:mysql --name mistral-db-setup mistral-all cat
-docker exec mistral-db-setup python /opt/stack/mistral/tools/sync_db.py
-docker kill -s 9 mistral-db-setup
-docker rm mistral-db-setup
-
-function run_mistral() {
-    NAME=${1:-mistral}
-    shift || true
-    LINKS='--link mistral-mysql:mysql --link mistral-rabbitmq:rabbitmq'
-    docker run \
-            -d \
-            --name $NAME \
-	        $LINKS \
-	        ${EXTRA_OPTS} \
-            ${OPTS} \
-            mistral-all "$@"
-}
-
-unset OPTS
-
-case "$MODE" in
-
-single)
-    # Single node setup
-    # The CMD of the mistral-all image runs the `mistral-server --server all` command.
-    OPTS="-p 8989:8989" run_mistral
-
-    echo "
-    Enter the container:
-      docker exec -it mistral bash
-
-    List workflows
-      docker exec mistral mistral workflow-list
-
-    "
-    ;;
-
-multi)
-    # Multinode setup
-    OPTS="-p 8989:8989" run_mistral "mistral-api" mistral-server --server api
-    run_mistral "mistral-engine" mistral-server --server engine
-    run_mistral "mistral-executor-1" mistral-server --server executor
-    run_mistral "mistral-executor-2" mistral-server --server executor
-
-    echo "
-    List workflows
-      docker exec mistral-api mistral workflow-list
-    "
-    ;;
-esac
-