Fuel stats

Project purpose

• collects stats information about OpenStack installations made by Fuel,
• generates stat reports in the CSV format,
• provides API for fetching raw data in the JSON format,
• provides Web UI for reports generation and basic stats charts/histograms.

Components

Collector is the service for collecting stats. It has REST API and DB storage. Analytics is the service for generating reports. It has REST API. Migrator is the tool for migrating data from the DB to the Elasticsearch.

The collector and analytics services are started by uWSGI. Migrator is started by cron to migrate the fresh data into Elasticsearch.

Collector

Data origin for collector is the Fuel master node. Stats collecting daemons collect and send data to the collector if allowed by the cloud operator.

Stats data is stored to the DB PostgreSQL.

Migrator

Migrator periodically migrates data from the fuel-stats DB to the Elasticsearch storage. This storage is used to generate basic stats charts and histograms for the Web UI.

Analytics

There are two sub-components in the analytics:

• analytics service
• Web UI

The analytics service API provides generation of CSV reports for installation info, plugins, and OpenStack workloads. The analytics API also provides export of data from DB as JSON.

The analytics Web UI provides basic summary stats charts and histograms with the possibility of filtering data by the Fuel release version. Also, in the Web UI we can generate and download stats reports on a selected time period.

How to configure development environment

To start fuel-stats on a localhost we need to have:

• PostgreSQL of version 9.3 or greater,
• Elasticsearch of version 1.3.4 or greater,
• Nginx.

Install PostgreSQL and development libraries: :

sudo apt-get install --yes postgresql postgresql-server-dev-all

Configure Elasticsearch repo as described in the elasticsearch docs and install Elasticsearch::

sudo apt-get install --yes elasticsearch

Install pip and development tools: :

sudo apt-get install --yes python-dev python-pip

Install virtualenv. This step increases flexibility when dealing with environment settings and package installation: :

sudo pip install virtualenv virtualenvwrapper

You can add '. /usr/local/bin/virtualenvwrapper.sh' to .bashrc or just execute it.:

. /usr/local/bin/virtualenvwrapper.sh

Create and activate virtual env for fuel-stats: :

mkvirtualenv stats
workon stats

You can use any name for the virtual env instead of 'stats'.

Install the fuel-stats requirements: :

pip install -r test-requirements.txt

Create a DB user for fuel-stats: :

sudo -u postgres psql
CREATE ROLE collector WITH NOSUPERUSER NOCREATEDB NOCREATEROLE LOGIN ENCRYPTED PASSWORD 'collector';

Create a DB and grant privileges to it: :

sudo -u postgres psql
CREATE DATABASE collector;
GRANT ALL ON DATABASE collector TO collector;

Check that all tests are passed: :

cd fuel-stats/collector && tox
cd fuel-stats/migration && tox
cd fuel-stats/analytics && tox

NOTE: The collector tests must be performed the first.

Now you are ready to develop fuel-stats.

How to configure Web UI

We assume that you already have configured virtual env as described in howto configure dev environment.

Install elsticsearch library and create sample data: :

workon stats
pip install elasticsearch
cd migration
nosetests migration.test.report.test_reports:Reports.test_libvirt_type_distribution

Install nodejs: :

sudo apt-get remove nodejs nodejs-legacy npm
sudo add-apt-repository -y ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs

Install nodejs and bower packages: :

cd fuel-stats/analytics/static
npm install
gulp bower

You can anytime lint your code by running: :

gulp lint

Add site configuration to Nginx: :

server {
    listen 8888;
    location / {
        root /your-path-to-fuel-stats/fuel-stats/analytics/static;
    }
    location ~ ^(/fuel)?(/[A-Za-z_0-9])?/(_count|_search) {
        proxy_pass http://127.0.0.1:9200;
    }
}

Then restart Nginx: :

service nginx restart

After this, your local server will be available at 0.0.0.0:8888 or any other port you've set up.

How to start local collector

You can use uWSGI to start the collector. Sample config can be found in collector/uwsgi/collector_example.yaml.

Or test web service to be used. To start the test web service, run: :

python collector/manage_collector.py --mode test runserver

How to start local analytics

You can use uWSGI to start analytics. Sample config can be found in analytics/uwsgi/analytics_example.yaml.

Or test the web service to be used. To start the test web service, run: :

python analytics/manage_analytics.py --mode test runserver

How to deal with DB migrations

Create new DB migration: :

python manage_collector.py --mode test db migrate -m "Migration comment" \
-d collector/api/db/migrations/

Apply all DB migrations: :

python manage_collector.py --mode test db upgrade -d collector/api/db/migrations/

Revert all migrations: :

python manage_collector.py --mode test db downgrade -d collector/api/db/migrations/

Switching off Elasticsearch

Elasticsearch was chosen as data storage for the dynamically generated statistics reports, but now only CSV reports are used for analytical purposes. Thus, Elasticsearch is an unnecessary complication of the infrastructure and data flow.

Without Elasticsearch, we are using memcached as cache for the web UI. Data expiration is configured by the parameter MEMCACHED_JSON_REPORTS_EXPIRATION for fuel_analytics.

Changes in the Nginx config: :

# Add this to the block 'server'
location /api/ {
    proxy_pass http://IP_OF_ANALYTICS_SERVICE:PORT_OF_ANALYTICS_SERVICE/api/;
}