Merge pull request #411 from loles/example

Wordpress example

doc/source/index.rst

@@ -18,6 +18,7 @@ Contents:
    handler_ansible
    examples
    deployment_plan
+   tutorials/index
 
 
 Indices and tables

doc/source/tutorials/index.rst

@@ -0,0 +1,9 @@
+List of Solar tutorials
+=================================
+
+Contents:
+
+.. toctree::
+   :maxdepth: 1
+
+   wordpress

doc/source/tutorials/wordpress.rst

@@ -0,0 +1,222 @@
+Wordpress tutorial
+==================
+
+1. Introduction
+---------------
+In this tutorial we will create Worpdress site using docker containers. We will create one container with Mysql database, then we will create database and user for it. After that we will create Wordpress container which is running Apache.
+
+For now you can use Solar only in our Vagrant environment.  
+First checkout Solar repo and start vagrant. We need two virtual machines. One where Solar database and Orchestrator will run and one where we will install Wordpress and all components:
+
+2. Solar installation
+---------------------
+
+.. code-block:: bash
+
+   git clone https://github.com/Mirantis/solar.git
+   cd solar
+   vagrant up solar-dev solar-dev1
+   vagrant ssh solar-dev
+   cd /vagrant
+
+.. note::
+   For now please assume that all `solar` commands are run from dir `/vagrant`
+
+3. Config resource
+------------------
+
+First we need to create Solar Resource definition where global configuration will be stored. This will be a `data container` only, so it will not have any handler nor actions. Let's create base structure:
+
+.. code-block:: bash
+
+  mkdir /vagrant/tmp/wp_config
+  touch /vagrant/tmp/wp_config/meta.yaml
+
+Open meta file `/vagrant/tmp/wp_config/meta.yaml` with your favorite text editor and paste the following data:
+
+.. code-block:: yaml
+  
+  id: container
+  handler: none
+  version: 1.0.0
+  input:
+    db_root_pass:
+      schema: str!
+      value:
+    db_port:
+      schema: int!
+      value:
+    wp_db_name:
+      schema: str!
+      value:
+    wp_db_user:
+      schema: str!
+      value:
+    wp_db_pass:
+      schema: str!
+      value:
+
+Let's go through this document line by line. `id:container` is not used currently and may be removed in future. `handler: none` says that this resource has no handler and no actions. In next line we define version. Currently it's also not used. The most important part starts from line 4. We define there the inputs for this resource. It will be possible to configure following inputs: 
+
+* `db_root_pass` - Mysql root password
+* `db_port` - Mysql port
+* `wp_db_name` - database name for Wordpress
+* `wp_db_user` - database user name for Wordpress
+* `wp_db_pass` - database user password for Wordpress
+
+In schema it's defined if input will be string or integer, `!` at the end means that the input is mandatory and can not be empty.
+
+4. Virtual resource
+-------------------
+
+All other required resources are already available in solar repo in `resources` dir. We will use four more resources:
+
+* resources/docker - it installs docker 
+* resources/docker_container - it manages docker container
+* resources/mariadb_db - it creates database in MariaDB and Mysql
+* resources/mariadb_user - it creates user in MariaDB and Mysql
+
+There are three ways to create resources in Solar: Python API, CLI and Virtual Resources. We will use the last option. 
+Virtual Resource is just a simple yaml file where we define all needed resources and connections.
+
+Create new file `docker.yaml` in /vagrant dir, open it and past the following data:
+
+.. code-block:: yaml
+
+  resources:
+    - id: docker
+      from: resources/docker
+      location: node1
+
+    - id: config
+      from: tmp/wp_config
+      location: node1
+      values:
+        db_root_pass: 'r00tme'
+        db_port: 3306
+        wp_db_name: 'wp'
+        wp_db_user: 'wp'
+        wp_db_pass: 'h4ack'
+      
+    - id: mysql
+      from: resources/docker_container
+      location: node1
+      values:
+        ip: node1::ip
+        image: mysql:latest
+        ports:
+          - config::db_port
+        env:
+          MYSQL_ROOT_PASSWORD: config::db_root_pass
+
+    - id: wp_db
+      from: resources/mariadb_db
+      location: node1
+      values:
+        db_name: config::wp_db_name
+        db_host: mysql::ip
+        login_user: 'root'
+        login_password: config::db_root_pass
+        login_port: config::db_port
+
+    - id: wp_user
+      from: resources/mariadb_user
+      location: node1
+      values:
+        user_password: config::wp_db_pass
+        user_name: config::wp_db_user
+        db_name: wp_db::db_name
+        db_host: mysql::ip
+        login_user: 'root'
+        login_password: config::db_root_pass
+        login_port: config::db_port
+
+    - id: wordpress
+      from: resources/docker_container
+      location: node1
+      values:
+        ip: node1::ip
+        image: wordpress:latest
+        env:
+          WORDPRESS_DB_HOST: mysql::ip
+          WORDPRESS_DB_USER: wp_user::user_name
+          WORDPRESS_DB_PASSWORD: wp_user::user_password
+          WORDPRESS_DB_NAME: wp_db::db_name
+
+In block `resources` we define... resources. Each section is one resource. Each resource definition has a following structure:
+
+* id - resource name
+* from - path to resource dir
+* location - node where resource will be run
+* values: initialization of a Resource Inputs
+
+As you can see entries for `from` have relative paths. For now we do not have any resource repository. This is why it's safer to run all commands from /vagrant dir. In `location` we define `node1`. It's name of our virtual machine resource. It's not created yet, we will do it shortly.
+
+In our configuration there are two formats which we use to assign values to inputs. First:
+
+.. code-block:: yaml
+
+  db_port: 3306
+
+It just means that input `db_port` will be set to `3306`
+
+Another format is:
+
+.. code-block:: yaml
+
+  login_port: config::db_port
+
+This means that input `login_port` will have the same value as input `db_port` from resource `config`. In Solar we call it Connection. Now when value of `db_port` changes, value of `login_port` will also change.
+
+
+5. Deploying
+------------
+
+Now it's time to deploy our configuration. When running `vagrant up solar-dev solar-dev1` you started two virtual machines. We will deploy Wordpress on solar-dev1. To do it we need to create a resource for it. We already have in repo virtual resource which is doing it. Just run:
+
+.. code-block:: bash
+
+  solar resource create nodes templates/nodes.yaml count=1
+
+It will create all required resources to run actions on solar-dev1. You can analyze `templates/nodes.yaml` later. Now we create resources defined in `docker.yaml`
+
+.. code-block:: bash
+
+  solar resource create docker docker.yaml
+
+Command `create` requires name, but it's not used for VirtualResources.
+
+Now you can deploy all changes with:
+
+.. code-block:: bash
+
+  solar changes stage
+  solar changes process
+  solar orch run-once
+
+To see deployment progress run:
+
+.. code-block:: bash
+
+  solar orch report
+
+Wait until all task will return status `SUCCESS`. When it's done you should be able to open Wordpress site at http://10.0.0.3
+
+6. Update
+---------
+
+Now change password for Wordpress database user
+
+.. code-block:: bash
+
+  solar resource update config wp_db_pass=new_hacky_pass
+
+and deploy new changes
+
+.. code-block:: bash
+
+  solar changes stage
+  solar changes process
+  solar orch run-once
+
+Using `report` command wait until all tasks finish. Wordpress should still working and new password should be used.

resources/docker_container/actions/run.yaml

@@ -7,14 +7,17 @@
        image: {{ image }}
        state: running
        net: host
-       {% if ports.value %}
+       {% if ports %}
        ports:
-         {% for port in ports.value %}
-         {% for p in port['value'] %}
-         - {{ p['value'] }}:{{ p['value'] }}
+         {% for port in ports %}
+         - {{ port }}:{{ port }}
          {% endfor %}
+       expose: 
+         {% for port in ports %}
+         - {{ port }}
          {% endfor %}
        {% endif %}
+
        {% if host_binds.value %}
        volumes:
          # TODO: host_binds might need more work
@@ -25,3 +28,10 @@
          - {{ bind['value']['src'] }}:{{ bind['value']['dst'] }}:{{ bind['value'].get('mode', 'ro') }}
          {% endfor %}
        {% endif %}
+       
+       {% if env %}
+       env:
+         {% for key, value in env.iteritems() %}
+         {{ key }}: {{ value }}
+         {% endfor %}
+       {% endif %}

resources/docker_container/actions/update.yaml

@@ -0,0 +1,37 @@
+
+- hosts: [{{host}}]
+  sudo: yes
+  tasks:
+    - docker:
+       name: {{ resource_name }}
+       image: {{ image }}
+       state: reloaded
+       net: host
+       {% if ports %}
+       ports:
+         {% for port in ports %}
+         - {{ port }}:{{ port }}
+         {% endfor %}
+       expose: 
+         {% for port in ports %}
+         - {{ port }}
+         {% endfor %}
+       {% endif %}
+
+       {% if host_binds.value %}
+       volumes:
+         # TODO: host_binds might need more work
+         # Currently it's not that trivial to pass custom src: dst here
+         # (when a config variable is passed here from other resource)
+         # so we mount it to the same directory as on host
+         {% for bind in host_binds.value %}
+         - {{ bind['value']['src'] }}:{{ bind['value']['dst'] }}:{{ bind['value'].get('mode', 'ro') }}
+         {% endfor %}
+       {% endif %}
+       
+       {% if env %}
+       env:
+         {% for key, value in env.iteritems() %}
+         {{ key }}: {{ value }}
+         {% endfor %}
+       {% endif %}

resources/docker_container/meta.yaml

@@ -9,7 +9,7 @@ input:
       schema: str!
       value:
     ports:
-      schema: [{value: [{value: int}]}]
+      schema: [int]
       value: []
     host_binds:
       schema: [{value: {src: str, dst: str, mode: str}}]
@@ -17,6 +17,9 @@ input:
     volume_binds:
       schema: [{src: str, dst: str, mode: str}]
       value: []
+    env:
+      schema: {}
+      value: {}
     # ssh_user:
     #   schema: str!
     #   value: []

resources/mariadb_user/meta.yaml

@@ -3,7 +3,7 @@ handler: ansible
 version: 1.0.0
 actions:
     run: run.yaml
-    update: run.yaml
+    update: update.yaml
     remove: remove.yaml
 input:
     user_password:

solar/core/resource/virtual_resource.py

@@ -63,11 +63,16 @@ def create_resource(name, base_path, args=None, tags=None,
     if isinstance(base_path, provider.BaseProvider):
         base_path = base_path.directory
 
-    # List args init with empty list. Elements will be added later
+    # filter connections from lists and dicts
+    # will be added later
     def _filter(value):
-        if not isinstance(value, list):
+        if isinstance(value, list):
+            return filter(lambda res: not is_connection(res), value)
+        if isinstance(value, dict):
+            return {key: None if is_connection(value) else value
+                    for key, value in value.iteritems()}
+        else:
             return value
-        return filter(lambda res: not is_connection(res), value)
 
     args = {key: _filter(value) for key, value in args.items()}
     r = Resource(name, base_path, args=args,
@@ -244,6 +249,10 @@ def parse_inputs(args):
             c, a = parse_list_input(r_input, arg)
             connections.extend(c)
             assignments.update(a)
+        elif isinstance(arg, dict):
+            c, a = parse_dict_input(r_input, arg)
+            connections.extend(c)
+            assignments.update(a)
         else:
             if is_connection(arg):
                 c = parse_connection(r_input, arg)
@@ -268,6 +277,22 @@ def parse_list_input(r_input, args):
     return connections, assignments
 
 
+def parse_dict_input(r_input, args):
+    connections = []
+    assignments = {}
+    for key, value in args.iteritems():
+        if is_connection(value):
+            dict_input = '{}:{}'.format(r_input, key)
+            c = parse_connection(dict_input, value)
+            connections.append(c)
+        else:
+            try:
+                assignments[r_input][key] = value
+            except KeyError:
+                assignments[r_input] = {key: value}
+    return connections, assignments
+
+
 def is_connection(arg):
     if isinstance(arg, basestring) and '::' in arg:
         return True

solar/test/resource_fixtures/resource_with_dict.yaml.tmpl

@@ -0,0 +1,9 @@
+id: simple_resource_with_list
+resources:
+  - id: res1
+    from: {resource_path}
+    values:
+      ip: '10.0.0.3'
+      servers:
+        a: 1
+        b: 2

solar/test/test_virtual_resource.py

@@ -95,6 +95,22 @@ def test_create_virtual_resource_with_list(tmpdir):
     assert res.args['servers'] == [1, 2]
 
 
+def test_create_virtual_resource_with_dict(tmpdir):
+    base_path = os.path.join(
+        os.path.dirname(os.path.realpath(__file__)),
+        'resource_fixtures')
+    vr_tmpl_path = os.path.join(base_path, 'resource_with_dict.yaml.tmpl')
+    base_resource_path = os.path.join(base_path, 'base_service')
+    with open(vr_tmpl_path) as f:
+        vr_data = f.read().format(resource_path=base_resource_path)
+    vr_file = tmpdir.join('base.yaml')
+    vr_file.write(vr_data)
+    resources = vr.create('base', str(vr_file))
+    assert len(resources) == 1
+    res = resources[0]
+    assert res.args['servers'] == {'a': 1, 'b': 2}
+
+
 def test_update(tmpdir):
     # XXX: make helper for it
     base_path = os.path.join(
@@ -167,6 +183,15 @@ def test_parse_connection():
     assert correct_connection == connection
 
 
+def test_parse_dict_input():
+    correct_connections = [{'child_input': 'env:host',
+                            'events': None,
+                            'parent': 'node1',
+                            'parent_input': 'ip'}]
+    connections, assigments = vr.parse_dict_input('env', {'host': 'node1::ip'})
+    assert correct_connections == connections
+
+
 def test_parse_connection_disable_events():
     correct_connection = {'child_input': 'ip',
                           'parent': 'node1',