Docs for fuel-stats were rewritten

README format changed from md to rst.
Project components description added.
Old README.md content moved to 'How to' recipes in the README.rst

Change-Id: I3f6471f859cb4f0998fe8c1bba784f4da5368b1a
Closes-Bug: #1507958
This commit is contained in:
Alexander Kislitsky 2016-03-18 18:16:29 +03:00
parent e9ec2111a7
commit 15944befb6
2 changed files with 211 additions and 134 deletions

134
README.md
View File

@ -1,134 +0,0 @@
collector
=========
Prototype of statistic collector
Requirements
----------
System requirements:
postgresql database server of version 9.3 or greater.
To install python requirements use command:
`pip install -r {corresponding_requirement_file}`
Operations
----------
By default manage_collector.py works with prod settings.
For working with test settings use `python manage_collector.py --mode` option.
For creating DB migration:
python manage_collector.py --mode test db migrate -m "Migration comment" \
-d collector/api/db/migrations/
Create DB user with password 'collector':
sudo -u postgres createuser -DES collector
or:
sudo -u postgres psql
CREATE ROLE collector WITH NOSUPERUSER NOCREATEDB NOCREATEROLE LOGIN ENCRYPTED PASSWORD 'collector';
Create DB and grant privileges to it:
sudo -u postgres psql
CREATE DATABASE collector;
GRANT ALL ON DATABASE collector TO collector;
For apply the latest migration:
python manage_collector.py --mode test db upgrade -d collector/api/db/migrations/
For revert all migrations:
python manage_collector.py --mode test db downgrade -d collector/api/db/migrations/
For starting test server:
python manage_collector.py --mode test runserver
Example config for uWSGI is located in collector/uwsgi/collector_test.yaml
Local setup
----------
To run statistics UI locally you need to do the following:
Install elasticsearch 1.3
pip install elasticsearch
Run test_report from NodesDistribution
prepare virtualenv:
cd fuel-stats
virtualenv .venv
source .venv/bin/activate
pip install -r collector/test-requirements.txt
cd migration
run tests:
nosetests migration.test.report.test_reports:Reports.test_libvirt_type_distribution
or
nosetests migration.test.report.test_os_distribution:OsDistribution
this will create demo data from elasticsearch
Install elasticsearch service
you can use this helpfull gist https://gist.github.com/wingdspur/2026107
And don't forget to start elasticsearch service
sudo service elasticsearch start
Nginx installation
sudo apt-get install nginx
fix Nginx config:
server {
listen 8888; // your free port
location / {
root /home/kpimenova/fuel/fuel-stats/analytics/static; // your path to fuel-stats/analytics/static
}
location ~ ^(/fuel)?(/[A-Za-z_0-9])?/(_count|_search) {
proxy_pass http://127.0.0.1:9200;
}
}
Then restart Nginx:
service nginx restart
After this your local server will be available at 0.0.0.0:8888 // or any other port you've set up :)
Also for correct UI work you need to setup a few things
Install nodejs packages
cd fuel-stats/analytics/static
npm install
Install bower packages
cd fuel-stats/analytics/static
gulp bower
That's all.
You can anytime lint your code by running
gulp lint

211
README.rst Normal file
View File

@ -0,0 +1,211 @@
==========
Fuel stats
==========
Project purpose
---------------
* collects stats information about OpenStack installations made by Fuel_,
* generates stat reports in the CSV format,
* provides API for fetching raw data in the JSON format,
* provides Web UI for reports generation and basic stats charts/histograms.
Components
----------
Collector is the service for collecting stats. It has REST API and DB storage.
Analytics is the service for generating reports. It has REST API.
Migrator is the tool for migrating data from the DB to the Elasticsearch_.
The collector and analytics services are started by uWSGI_. Migrator is
started by cron to migrate the fresh data into Elasticsearch_.
Collector
---------
Data origin for collector is the Fuel_ master node. Stats collecting daemons
collect and send data to the collector if allowed by the cloud operator.
Stats data is stored to the DB PostgreSQL_.
Migrator
--------
Migrator periodically migrates data from the fuel-stats DB to the
Elasticsearch_ storage. This storage is used to generate basic stats charts
and histograms for the Web UI.
Analytics
---------
There are two sub-components in the analytics:
* analytics service
* Web UI
The analytics service API provides generation of CSV reports for installation
info, plugins, and OpenStack workloads. The analytics API also provides
export of data from DB as JSON.
The analytics Web UI provides basic summary stats charts and histograms with
the possibility of filtering data by the Fuel_ release version. Also, in the
Web UI we can generate and download stats reports on a selected time period.
.. _howto configure dev environment:
How to configure development environment
----------------------------------------
To start fuel-stats on a localhost we need to have:
* PostgreSQL_ of version 9.3 or greater,
* Elasticsearch_ of version 1.3.4 or greater,
* Nginx_.
Install PostgreSQL_ and development libraries: ::
sudo apt-get install --yes postgresql postgresql-server-dev-all
Configure Elasticsearch_ repo as described in the `elasticsearch docs`_ and
install Elasticsearch_:::
sudo apt-get install --yes elasticsearch
Install pip and development tools: ::
sudo apt-get install --yes python-dev python-pip
Install virtualenv. This step increases flexibility when dealing with
environment settings and package installation: ::
sudo pip install virtualenv virtualenvwrapper
You can add '. /usr/local/bin/virtualenvwrapper.sh' to .bashrc or just
execute it.::
. /usr/local/bin/virtualenvwrapper.sh
Create and activate virtual env for fuel-stats: ::
mkvirtualenv stats
workon stats
You can use any name for the virtual env instead of 'stats'.
Install the fuel-stats requirements: ::
pip install -r test-requirements.txt
Create a DB user for fuel-stats: ::
sudo -u postgres psql
CREATE ROLE collector WITH NOSUPERUSER NOCREATEDB NOCREATEROLE LOGIN ENCRYPTED PASSWORD 'collector';
Create a DB and grant privileges to it: ::
sudo -u postgres psql
CREATE DATABASE collector;
GRANT ALL ON DATABASE collector TO collector;
Check that all tests are passed: ::
cd fuel-stats/collector && tox
cd fuel-stats/migration && tox
cd fuel-stats/analytics && tox
**NOTE:** The collector tests must be performed the first.
Now you are ready to develop fuel-stats.
How to configure Web UI
-----------------------
We assume that you already have configured virtual env as described in
`howto configure dev environment`_.
Install elsticsearch library and create sample data: ::
workon stats
pip install elasticsearch
cd migration
nosetests migration.test.report.test_reports:Reports.test_libvirt_type_distribution
Install nodejs: ::
sudo apt-get remove nodejs nodejs-legacy npm
sudo add-apt-repository -y ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs
Install nodejs and bower packages: ::
cd fuel-stats/analytics/static
npm install
gulp bower
You can anytime lint your code by running: ::
gulp lint
Add site configuration to Nginx_: ::
server {
listen 8888;
location / {
root /your-path-to-fuel-stats/fuel-stats/analytics/static;
}
location ~ ^(/fuel)?(/[A-Za-z_0-9])?/(_count|_search) {
proxy_pass http://127.0.0.1:9200;
}
}
Then restart Nginx: ::
service nginx restart
After this, your local server will be available at 0.0.0.0:8888
or any other port you've set up.
How to start local collector
----------------------------
You can use uWSGI_ to start the collector. Sample config can be found in
collector/uwsgi/collector_example.yaml.
Or test web service to be used. To start the test web service, run: ::
python collector/manage_collector.py --mode test runserver
How to start local analytics
----------------------------
You can use uWSGI_ to start analytics. Sample config can be found in
analytics/uwsgi/analytics_example.yaml.
Or test the web service to be used. To start the test web service, run: ::
python analytics/manage_analytics.py --mode test runserver
How to deal with DB migrations
------------------------------
Create new DB migration: ::
python manage_collector.py --mode test db migrate -m "Migration comment" \
-d collector/api/db/migrations/
Apply all DB migrations: ::
python manage_collector.py --mode test db upgrade -d collector/api/db/migrations/
Revert all migrations: ::
python manage_collector.py --mode test db downgrade -d collector/api/db/migrations/
.. _Fuel: https://docs.mirantis.com/openstack/fuel/
.. _Elasticsearch: https://www.elastic.co/
.. _uWSGI: https://pypi.python.org/pypi/uWSGI/
.. _PostgreSQL: http://www.postgresql.org/
.. _Nginx: http://nginx.org/
.. _elasticsearch docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html