Better organization - seperated split setup and hlinux to their own subdirs.

Berksfile

@@ -4,7 +4,7 @@ if Gem::Specification::find_by_name('berkshelf').version.to_s[0] == '3'
   source 'https://api.berkshelf.com'
 end
 
-metadata
+cookbook 'mini-mon', path: './cookbooks/mini-mon'
 cookbook 'mon_api', git: 'https://github.com/hpcloud-mon/cookbooks-mon_api'
 cookbook 'kafka', git: 'https://github.com/hpcloud-mon/cookbooks-kafka'
 cookbook 'mon_agent', git: 'https://github.com/hpcloud-mon/cookbooks-mon_agent'

README.md

@@ -8,13 +8,10 @@
     - [Install Vagrant](#install-vagrant)
     - [Setup Berkshelf](#setup-berkshelf)
 - [Using mini-mon](#using-mini-mon)
-  - [Start mini-mon](#start-mini-mon)
-  - [Halt mini-mon](#halt-mini-mon)
-  - [Updating a VM](#updating-a-vm)
+  - [Updating](#updating)
   - [Improving Provisioning Speed](#improving-provisioning-speed)
   - [Cookbook Development](#cookbook-development)
-- [Running hLinux as the base OS](#running-hlinux-as-the-base-os)
-  - [Creating a new hLinux box](#creating-a-new-hlinux-box)
+- [Alternate Setups](#alternate-setups)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
@@ -54,40 +51,17 @@ gem install berkshelf
 
 # Using mini-mon
 
-- Your home dir is synced to `/vagrant_home` on each vm
-- Vms created
-  - `api` at `192.168.10.4`
-  - `kafka` at `192.168.10.10` - mon-notification runs on this box also
-  - `mysql` at `192.168.10.6`
-  - `persister` at `192.168.10.12`
-  - `thresh` at `192.168.10.14`
-  - `vertica` at `192.168.10.8`
-    - The management console is at https://192.168.10.8:5450
+- Your home dir is synced to `/vagrant_home` on the vm
+- The vm will have an ip of 196.168.10.4 that can be access from other services running on the host.
+- Run `vagrant ssh` to login
 - Run `vagrant help` for more info
-- Run `vagrant ssh <vm name>` to login to a particular vm
-- Can also run `ssh vagrant@<ip address>` to login 
-  - password is `vagrant`
-  
-## Start mini-mon
-From within the `mini-mon` directory, to start all the vms run:
-```
-bin/vup
-```
-The standard vagrant commands can also be used, the vup script just starts things up in a dependency order using parallel startup.
 
-## Halt mini-mon
-In some cases halting mini-mon can result in certain vms being left in an odd state, to avoid this a script has been made to halt boxes in the 
-correct order
-```
-bin/vhalt
-```
-
-## Updating a VM
-When someone updates the config for a vm this process should allow you to bring up an updated vm.
+## Updating
+When someone updates the config this process should allow you to bring up an updated vm.
 - `git pull`
 - `berks update`
-- `vagrant destroy vm` - Where vm is the name of the vm being updated, for example 'vertica'
-- `vagrant up vm`
+- `vagrant destroy` - Where vm is the name of the vm being updated, for example 'vertica'
+- `vagrant up`
 
 ## Improving Provisioning Speed
 The slowest part of the provisioning process is the downloading of deb packages. To speed this up a local apt-cacher-ng can be used.
@@ -109,23 +83,10 @@ To develop cookbook changes with Vagrant:
 - If the vagrant vm is already up run 'vagrant provision' if not run 'Vagrant up'
 - When finish testing commit and upload your cookbook as normal but don't forget to bump the cookbook version in the metadata.rb.
 
-# Running hLinux as the base OS
-hLinux can be installed and run as the base OS for all the vms defined in mini-mon. To this comment/uncomment the appropriate lines in the Vagrantfile.
-Also switch to the proper base apt repos in recipes/default.rb. There are a couple of minor problem which would slow down development and are why at
-this point hLinux has not been turned on by default:
-- The vboxsf filesystem driver is not working correctly in hLinux, this prevents home directory syncing.
-- Slow network performance of the hLinux vbox image makes some tasks annoying.
+# Alternate Vagrant Configurations
+To run any of these alternate configs, simply run the Vagrant commands from within the subdir, though note the vertica debs must be copied into
+the subdir also.
 
-## Creating a new hLinux box
-The [hLinux](http://hlinux-home.usa.hp.com/wiki/index.php/Main_Page) box used in mini-mon is created via [packer](http://www.packer.io/), config is in
-the templates directory.
-
-- Install packer
-  - `brew tap homebrew/binary`
-  - `brew install packer`
-- Run packer
-  - `cd templates`
-  - `packer build hlinux.json`
-- From the mini-mon directory run `vagrant box add hlinux templates/packer_virtualbox-iso_virtualbox.box`
-  - If you have an existing hLinux box you man need to first remove it `vagrant box remove hlinux`
-  - Also upload to a server for others to download.
+- A Vagrant config for running the various monitoring components split into their own vms is available in the split subdir.
+- A Vagrant config for testing hLinux is available in the hlinux subdir
+- Baremetal - actually not using Vagrant at all, see the baremetal fabric task in the utils directory.

cookbooks/mini-mon/README.md

@@ -0,0 +1,13 @@
+mini_mon cookbook
+=========================
+Various helper recipes for the mini-mon setup.
+
+Requirements
+------------
+apt cookbook
+
+Recipes
+---------
+- default - configures base apt repostories needed for mini-mon as well as option apt cache
+- mysql_schema - Defines the mysql_schema used by mini-mon
+- postfix - extremely simple postfix install

hlinux/README.md

@@ -0,0 +1,23 @@
+An experimental hLinux based mini-mon is available for testing.
+
+# Running hLinux as the base OS
+hLinux can be installed and run as the base OS for all the vm(s) defined in mini-mon. The vagrantfile in this sub-dir is setup for this purpose.
+Before running this you must create a hLinux box as describe below.
+
+As of the last testing there are a couple of minor problems:
+- The vboxsf filesystem driver is not working correctly in hLinux, this prevents home directory syncing.
+- Slow network performance of the hLinux vbox image makes some tasks annoying.
+
+## Creating a new hLinux box
+The [hLinux](http://hlinux-home.usa.hp.com/wiki/index.php/Main_Page) box used in mini-mon is created via [packer](http://www.packer.io/), config is in
+the templates sub-directory.
+
+- Install packer
+  - `brew tap homebrew/binary`
+  - `brew install packer`
+- Run packer
+  - `cd templates`
+  - `packer build hlinux.json`
+- From the mini-mon directory run `vagrant box add hlinux templates/packer_virtualbox-iso_virtualbox.box`
+  - If you have an existing hLinux box you man need to first remove it `vagrant box remove hlinux`
+  - Also upload to a server for others to download.

hlinux/Vagrantfile

@@ -0,0 +1,30 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+VAGRANTFILE_API_VERSION = "2" # Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  # Settings for all vms
+  config.berkshelf.enabled = true
+  config.berkshelf.berksfile_path = "../Berksfile"
+
+  config.vm.box = "hlinux"
+  config.vm.guest = :debian   # bypass vagrant guest type autodetection
+  # hlinux is not currently working with virtual box shared folders so minimize shared folders and switch to rsync
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+
+  config.vm.hostname = 'mini-mon'
+  config.vm.network :private_network, ip: "192.168.10.4"
+  config.vm.provider "virtualbox" do |vb|
+    vb.memory = 6144
+    vb.cpus = 4
+  end
+
+  config.vm.provision :chef_solo do |chef|
+    chef.synced_folder_type = "rsync"
+    chef.roles_path = "../roles"
+    chef.data_bags_path = "../data_bags"
+    chef.add_role "Mini-Mon"
+  end
+
+end

roles/Mini-Mon.json

@@ -3,20 +3,54 @@
   "description": "Sets up the entire monitoring stack on one box",
   "json_class": "Chef::Role",
   "default_attributes": {
+    "apt": {
+      "periodic_update_min_delay": 60
+    },
     "kafka": {
-      "listen_address": "127.0.0.1"
+      "listen_address": "127.0.0.1",
+      "topics": {
+        "metrics": { "replicas": 1, "partitions": 4 },
+        "events": { "replicas": 1, "partitions": 4 },
+        "alarm-state-transitions": { "replicas": 1, "partitions": 4 },
+        "alarm-notifications": { "replicas": 1, "partitions": 4 }
+      }
+    },
+    "percona": {
+      "backup": {
+        "password": "password"
+      },
+      "cluster": {
+        "package": "percona-xtradb-cluster-56"
+      },
+      "main_config_file": "/etc/mysql/my.cnf",
+      "server": {
+        "bind_address": "0.0.0.0",
+        "debain_password": "password",
+        "replication": {
+          "password": "password"
+        },
+        "root_password": "password",
+        "skip_name_resolve": true
+      }
     }
   },
   "override_attributes": {
   },
   "chef_type": "role",
   "run_list": [
-    "role[MySQL]",
-    "role[Kafka]",
-    "role[Vertica]",
-    "role[Api]",
-    "role[Persister]",
-    "role[Thresh]"
+    "recipe[mini-mon]",
+    "recipe[percona::cluster]",
+    "recipe[mini-mon::mysql_schema]",
+    "recipe[zookeeper]",
+    "recipe[kafka]",
+    "recipe[mini-mon::postfix]",
+    "recipe[mon_notification]",
+    "recipe[vertica]",
+    "recipe[sysctl]",
+    "recipe[mon_api]",
+    "recipe[mon_persister]",
+    "recipe[mon_agent]",
+    "recipe[mon_thresh]"
   ],
   "env_run_lists": {
   }

split/README.md

@@ -0,0 +1,32 @@
+The chef and Vagrant configuration here runs mini-mon split into 6 different VMs. This split demonstrates and provides a simple test of how the
+monitoring system can scale but also is useful for some development scenarios.
+
+# Using the split mini-mon
+- Your home dir is synced to `/vagrant_home` on each vm
+- Vms created
+  - `api` at `192.168.10.4`
+  - `kafka` at `192.168.10.10` - mon-notification runs on this box also
+  - `mysql` at `192.168.10.6`
+  - `persister` at `192.168.10.12`
+  - `thresh` at `192.168.10.14`
+  - `vertica` at `192.168.10.8`
+    - The management console is at https://192.168.10.8:5450
+- Run `vagrant help` for more info
+- Run `vagrant ssh <vm name>` to login to a particular vm
+- Can also run `ssh vagrant@<ip address>` to login 
+  - password is `vagrant`
+  
+## Start mini-mon
+From within this directory, run this simple scripts which aid in bringing up all the vms in the proper order.
+If desired, the standard vagrant commands can also be used.
+```
+bin/vup
+```
+
+## Halt mini-mon
+In some cases halting mini-mon can result in certain vms being left in an odd state, to avoid this a script has been made to halt boxes in the 
+correct order
+```
+bin/vhalt
+```
+

split/Vagrantfile

@@ -0,0 +1,88 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+VAGRANTFILE_API_VERSION = "2" # Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  # Settings for all vms
+  config.berkshelf.enabled = true
+  config.berkshelf.berksfile_path = "../Berksfile"
+
+  config.vm.box = "precise64"
+  config.vm.box_url = "http://files.vagrantup.com/precise64.box"
+  config.vm.synced_folder "~/", "/vagrant_home"
+  config.vm.provider "virtualbox" do |vb|
+    vb.customize ["modifyvm", :id, "--memory", "768"]
+  end
+
+  # VM specific settings, these machines come up in order they are specified.
+  config.vm.define "mysql" do |mysql|
+    mysql.vm.hostname = 'mysql'
+    mysql.vm.network :private_network, ip: "192.168.10.6"
+    mysql.vm.provision :chef_solo do |chef|
+      chef.roles_path = "roles"
+      chef.data_bags_path = "data_bags"
+      chef.add_role "MySQL"
+    end
+  end
+
+  config.vm.define "kafka" do |kafka|
+    kafka.vm.hostname = 'kafka'
+    kafka.vm.network :private_network, ip: "192.168.10.10"
+    kafka.vm.provision :chef_solo do |chef|
+      chef.roles_path = "roles"
+      chef.data_bags_path = "data_bags"
+      chef.add_role "Kafka"
+    end
+    kafka.vm.provider "virtualbox" do |vb|
+      vb.memory = 1024
+    end
+  end
+
+  config.vm.define "vertica" do |vertica|
+    vertica.vm.hostname = 'vertica'
+    vertica.vm.network :private_network, ip: "192.168.10.8"
+    vertica.vm.provision :chef_solo do |chef|
+      chef.roles_path = "roles"
+      chef.data_bags_path = "data_bags"
+      chef.add_role "Vertica"
+    end
+    vertica.vm.provider "virtualbox" do |vb|
+      vb.memory = 2048  # Vertica is pretty strict about its minimum
+    end
+  end
+
+  config.vm.define "api" do |api|
+    api.vm.hostname = 'api'
+    api.vm.network :private_network, ip: "192.168.10.4"
+    api.vm.provision :chef_solo do |chef|
+      chef.roles_path = "roles"
+      chef.data_bags_path = "data_bags"
+      chef.add_role "Api"
+    end
+  end
+
+  config.vm.define "persister" do |persister|
+    persister.vm.hostname = 'persister'
+    persister.vm.network :private_network, ip: "192.168.10.12"
+    persister.vm.provision :chef_solo do |chef|
+      chef.roles_path = "roles"
+      chef.data_bags_path = "data_bags"
+      chef.add_role "Persister"
+    end
+    persister.vm.provider "virtualbox" do |vb|
+      vb.memory = 1024
+    end
+  end
+
+  config.vm.define "thresh" do |thresh|
+    thresh.vm.hostname = 'thresh'
+    thresh.vm.network :private_network, ip: "192.168.10.14"
+    thresh.vm.provision :chef_solo do |chef|
+      chef.roles_path = "roles"
+      chef.data_bags_path = "data_bags"
+      chef.add_role "Thresh"
+    end
+  end
+
+end

split/data_bags/mon_agent/mon_agent.json

@@ -0,0 +1,15 @@
+{
+  "id" : "mon_agent",
+  "dimensions" : "Origin:Mini-mon, Environment:Dev",
+  "log_level" : "DEBUG",
+  "use_dd": "no",
+  "dd_url": "https://app.datadoghq.com",
+  "api_key": "",
+  "send_to_mon_api": "true",
+  "api" : {
+    "mon_api_url" : "http://192.168.10.4:8080/v2.0/metrics",
+    "mon_api_project_id" : "82510970543135",
+    "use_keystone" : "false",
+    "aggregate_metrics" : "true"
+  }
+}

split/data_bags/mon_api/mon_api.json

@@ -0,0 +1,17 @@
+{
+  "id" : "mon_api",
+  "vertica" : {
+    "dbname" : "mon",
+    "hostname" : "192.168.10.8"
+  },
+  "zookeeper" : {
+    "hostname" : "192.168.10.10"
+  },
+  "mysql": {
+   "hostname":"192.168.10.6",
+   "schema": "mon"
+  },
+  "kafka": {
+    "hostname": "192.168.10.10"
+  }
+}

split/data_bags/mon_api/mon_credentials.json

@@ -0,0 +1,21 @@
+{
+    "id": "mon_credentials",
+    "middleware": {
+        "keystore_password": "changeit",
+        "serverVip": "region-a.geo-1.identity-admin.hpcloudsvc.com",
+        "truststore_password": "changeit",
+	    "keystore_file":"hpmiddleware-keystore-production.jks"
+    },
+    "mysql": {
+        "hostname": "192.168.10.6",
+        "username": "monapi",
+        "password": "password",
+        "schema": "mon"
+    },
+    "vertica": {
+        "hostname": "192.168.10.8",
+        "username": "mon_api",
+        "password": "password",
+        "schema": "mon"
+    }
+}

split/data_bags/mon_notification/hosts.json

@@ -0,0 +1,7 @@
+{
+  "id" : "hosts",
+  "kafka": "192.168.10.10",
+  "mysql": "192.168.10.6",
+  "smtp": "localhost",
+  "zookeeper": "localhost"
+}

split/data_bags/mon_persister/mon_credentials.json

@@ -0,0 +1,7 @@
+{
+  "id" : "mon_credentials",
+  "vertica" : {
+    "user" : "dbadmin",
+    "password" : "password"
+  }
+}

split/data_bags/mon_persister/mon_persister.json

@@ -0,0 +1,29 @@
+{
+    "id": "mon_persister",
+    "kafka": {
+        "metrics_topic": "metrics",
+        "num_threads": "1",
+        "group_id": "1",
+        "consumer_id": "1"
+    },
+    "disruptor": {
+        "buffer_size": "1048576",
+        "num_processors": "1"
+    },
+    "vertica_output_processor": {
+        "batch_size": "10000"
+    },
+    "deduper_config": {
+        "dedupe_run_frequencey_seconds": "30"
+    },
+    "vertica_metric_repository_config": {
+        "max_cache_size": "2000000"
+    },
+    "vertica": {
+        "dbname": "mon",
+        "hostname": "192.168.10.8"
+    },
+    "zookeeper": {
+        "hostname": "192.168.10.10"
+    }
+}

split/data_bags/mon_thresh/mon_thresh.json

@@ -0,0 +1,22 @@
+{
+  "id" : "mon_thresh",
+  "kafka": {
+    "metric": {
+      "group": "thresh-metric",
+      "topic": "metrics"
+    },
+    "event": {
+      "group": "thresh-event",
+      "host": "192.168.10.10:9092",
+      "consumer_topic": "events",
+      "producer_topic": "alarm-state-transitions"
+    }
+  },
+  "mysql": {
+    "db": "mon",
+    "host": "192.168.10.6:3306"
+  },
+  "zookeeper": {
+    "host": "192.168.10.10:2181"
+  }
+}

utils/README.md

@@ -1 +1,4 @@
-Various utilities for leveraging mini-mon. These are written as [fabric](http://docs.fabfile.org/) scripts
+Various utilities for leveraging mini-mon. These are written as [fabric](http://docs.fabfile.org/) scripts.
+
+To deploy mini-mon to baremetal, choosing the right ip run
+  - `fab baremetal -H 10.0.0.0`

utils/baremetal.py

@@ -65,10 +65,6 @@ def install(install_dir='/vagrant', proxy=None):
         else:
             sudo('git clone https://github.com/hpcloud-mon/mon-vagrant.git %s' % install_dir)
 
-        # currently the one-vm setup is on a branch
-        with cd(install_dir):
-            sudo('git checkout feature/one-vm')
-
         # download cookbooks
         with cd(install_dir):
             sudo('berks vendor cookbooks')