1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: I8432a85b025a46d20484a0d8a3ec72dca54d23cb
Following OpenStack Style Guidelines[1]: http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H234] assertEquals() logs a DeprecationWarning in Python 3.x,
use assertEqual() instead. The same goes for assertNotEquals().
Change-Id: Iaa4b2fd04d2aa661bb196c4b9f2494a281a9294a
We don't use Elasticsearch for flexible reports generation on the
fuel-stats web UI, only for five fixed reports. Thus using of
Elasticsearch is overhead and it can be removed from the servers
Instead of Elasticsearch we use fuel-stats json api calls and
PostgreSQL + Memcached.
Changes list:
- api call added to fuel-stats json api for data required on the web UI page,
- column release added to DB installation_structures table schema,
- memcached is used for caching data for the web UI page,
- elasticsearch client removed from js requirement,
- web UI page rewritten to use fuel-stats json api instead Elaticsearch.
Co-Authored-By: Kate Pimenova <kpimenova@mirantis.com>
Change-Id: Ie752e0d0a3c80933888f986e2497b45adce730c9
Closes-Bug: #1595548
Some files were committed or created with executable permissions 755.
Theese permissions changed to 644.
Change-Id: Ic7a99eda0518de89ff72f13936f0d131fd1eb121
Closes-Bug: #1598746
In py3 ,
Xrange is renamed to range,
so it needs to be modified to make it compatible.
the code:
print (xrange(5))
Traceback (most recent call last):
File "code", line 4, in <module>
print (xrange(5))
NameError: name 'xrange' is not defined.
Change-Id: I3a0e22876c35ced5c961c092fcda25db33d1c599
Closes-Bug: #1596127
On huge reports data we can have 500 HTTP error by timeout limitation.
In case of streaming data from the DB cursor we had hanged
DB operation inside the open transaction. For fix the issue releasing of
SQLAlchemy session was added to collector and analytics apps.
Additional changes:
- We stop to migrate action logs data to Elasticsearch due to
no one UI report uses this data.
- Parameters fixed in OSWLs 'export' call in 'all_reports'.
Change-Id: I009081a2304b36fb4ab3705b6fa09a8a3df5710b
Closes-Bug: #1564427
On the Fuel CI we have no rights to drop and create DB.
In the tests we are downgrading and upgrading DB schema by Alembic
migrations. In case of adding migrations in one review we had
all other reviews broken due to Alembic failures on attempt to
downgrade DB schema.
As fix we are fallback to the cleaning DB without migrations
in case of Alembic error on the schema downgrade.
Change-Id: I65faf43c28427594495c67696ca190e7fe29f083
Closes-Bug: #1556876
For deploying collector service on the production nodes
DevOps team uses external collector configs, saved in the JSON.
JSON doesn't support tuple type and dict with tuple as key also
can't be serialized.
We are introducing new format for filtering rules configuration.
The following dicts can be used as filtering rule:
- {'packages_list': ['a', 'b']}
- {'packages_list': ['a', 'b']: 'from_date': None}
- {'packages_list': ['a', 'b']: 'from_date': '2016-03-10T22:34:39'}
- {'build_id': 'build_id_value'}
- {'build_id': 'build_id_value', 'from_date': None}
- {'build_id': 'build_id_value', 'from_date': '2016-03-10T22:34:39'}
The old filtering rules format is backward compatible:
- {'build_id_value': None}
- {'build_id_value': '2016-03-10T22:34:39'}
Change-Id: I1be9760bb700be5b8e20c0e27689a6b017ba75f1
Partial-Bug: #1550376
On the 1M number of rows in the table JSON index begins to
work too slow. We are copy action_name and action_type values
from action_logs.body to separate indexed columns in the
action_logs table.
Redundant JSON serialization/deserealization on the python
level is removed by changing column 'structure' type from
TEXT to JSON in the InstallationStrurcure model. We already
had JSON data in 'structure' in the DB, thus only SQLAlchemy
model is changed.
Change-Id: I415f4fe607f44ac670f63f961af7fb4998f75dd6
Closes-Bug: #1540289
From release 8.0 Fuel sends fuel_packages list as build marker
instead fuel_release->build_id. Now we are filtering installation
info by build_id and fuel_packages.
Change-Id: I798ad8261e9982e71afb1c53e5ce39a67b9fe097
Closes-Bug: #1533660
Closes-Bug: #1515579
Version info data handled for OpenStackWorkloadStats. If master node
updated and the version info in InstallationStructure changed we have
correct version info in OSWLs stats report.
Fixes in collector:
- version_info added to DB model
- version_info added to API protocol
- tests for version_info added
Fixes in analytics:
- version_info added to DB model
- version_info fields added to CSV
- field installation structure release info removed from CSV
- version_info building on the fly for old OSWLs added
Partial-Bug: #1525902
Change-Id: I27d9d65517b4d5a7c6125b889fb0d1ba1ea213cf
Instead sha checksums and build_id fuel packages info handled
in the stats collector api.
Change-Id: I05e841d4279bc61645cefb68f411a93b67e57189
Partial-Bug: #1513737
We have 3 services: analytics, collector, migration.
Migration is frozen and going to be removed. Also
migration requires Elasticsearch server for tests,
thus it can't be tested on OpenStack CI.
Pep8 checked on all services.
Project works only on python2.7, thus
gate-fuel-stats-python26 should be removed.
Change-Id: Ic5b6f15903918b4b0850ce0cdc12543925de0cfa
Closes-Bug: #1500073
In FUEL 8.0 release info data structure was changed. Field 'fuel-nailgun_sha'
was used instead 'nailgun_sha'. Information about nailgun build is
required for stats, so we change required fields list for FUEL 8.0
Change-Id: Ib37502663c0dd05a2ec1cc3df182cdb87c400059
Closes-Bug: #1500470
For increasing speed of creating clusters report indexes on
body->>'action_name' and body->>'action_type' were added.
Query execution plan cost was optimized from 21000 to 60.
Collector base DB test fixed. Alembic constant 'base' used
for downgrade all alembic migrations instead of current
revisions number calculation.
Change-Id: Ief06aa9bbb94ebad2c2791617df8fb96cf979118
Closes-Bug: #1488092
- Rules of filtering installation info added to the application config.
- All data is saved as before, but filtered installation info is marked
as filtered in the DB.
- Boolean field added to installation info DB model for saving status
of data filtration.
- DB migration for adding filtration flag added.
- Log level to applictaion logger set explicitly.
- Optional build_id parameter added to installation info JSON schema.
- Adding of migration files handled in the collector base test.
- Requirement for python-dateutil added to the collector requirements.txt
Change-Id: I99626f7f9495a0f36ed70af487d41a647250355b
Partial-Bug: #1443347
ostf_sha was renamed to fuel-ostf_sha.
fuellib_sha was renamed to fuel-library_sha.
python-fuelclient_sha was added.
Change-Id: Ica4a7d479e1db3f417ec84785aa303f4e12efd41
Closes-Bug: #1447962
Fields names mapping from dict keys to object attributes was ignored.
This causes changing of id fields values in the updating DB objects
and IntegrityError was raised.
Change-Id: I35d4ec13d28ec35bcecd1bba9aeb809eec93ad85
Closes-Bug: #1427575
Fuel-stats part of implementation.
In case of cluster reset we can lose information about adding and
removing workloads. For prevention format of resource_data is
changed. Now we have lists of dicts in 'added', 'remove', 'modified'
and 'current'. On cluster reset we can have several records in the
'added' with different time of adding. For 'removed' we add full
data only if it doesn't already added. In other case we add only
id with removing time.
Change-Id: I2d57f53fcb1489f177772875464056ef301b16a6
Closes-Bug: #1427757
Implements: blueprint openstack-workload-statistics
Db model created.
Json scema for API created.
Blueprints urls moved to app.
Insert and update for lists of data generalized.
SQL logging enabled for test mode.
Implements: blueprint openstack-workload-statistics
Change-Id: If74e4bea0bc9c6a28ce859d775d1b33f36b547a1
Flask application added for export clusters statstics in CSV format.
Export process streams data by the generators.
Closes-Bug: #1410262
Blueprint: export-stats-to-csv
Change-Id: I265b617e78de142f8f10f22e85f734d0df7979c2
DB model fixed
Action logs JSON schema fixed
Tasks generation added into tests
Fixed Elasticsearch mapping scheme
Fixed collector manager running script
Set of fields for es id generation is used
Closes-Bug: #1397326
Change-Id: Ifdb47d5af6a75d4306711e3929146a9a82406dcc
MANIFEST.in used for include json schemas.
Keyword 'anonymous' removed from collector package description.
Creation of log directory added into logger intialization.
Blueprint: send-anon-usage
Closes-Bug: 1395703
Change-Id: I4bce4db8484d74103b177e9063e123c319c90d65
Loading from external configs is required for auto deployment
fuel-stats packages by the DevOps team.
For collector COLLECTOR_SETTINGS environment param is used.
Value of COLLECTOR_SETTINGS should be path to external config.
Config is regular python file. For collector standard Flask
external config file feature is used.
From collector config removed unused HOST and PORT params
For migrator parameter '--config' is added to manage_migration.py.
External config is yaml config file.
Blueprint: send-anon-usage
Change-Id: I3667d22d55b6edbbaa8f81e14dcd3d199e7a87a6
Non-completed tasks are verified properly and skipped with 'failed' status
Blueprint: send-anon-usage
Change-Id: I876daccbc3bc8510b8d7cc5df3c949be907c509f
Add script wich can be used in CLI to generate
'bullets' for yandex-tank, which will be used for
perfomance testing.
Change-Id: I31dea6a8aa490b5f23c5cba6e1ac1c1e5ecd95ad
Implements: blueprint anonymous-statistics-performance-testing
* migration status saved into Elasticsearch
* tox handled for migration
* JSON field added into migration
* reports tests moved into migration
* migration run script implemented
* manager for migration run implemented
* production wsgi application added for collector
* logging error in collector fixed
* installation struct model and table renamed
Change-Id: I001010a13f4798505b4d806728b3376da57c90fb
Blueprint: send-anon-usage
Field 'body' is added to 'action_logs' table with its processing in handler.
'body' contains action log record, its structure is checked with json scheme.
Blueprint: send-anon-usage
Change-Id: I2ec40a9904899a72f62c1f29b58ccbe0a1d6a02d