Yaml.load() return Python object may be dangerous if you receive
a YAML document from an untrusted source such as the Internet.
The function yaml.safe_load() limits this ability to simple Python
objects like integers or lists.
Reference:
https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html
Change-Id: I9841e688e16e9bebef5eb278ee2f3a12f2ca61f4
1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: I8432a85b025a46d20484a0d8a3ec72dca54d23cb
Following OpenStack Style Guidelines[1]: http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H234] assertEquals() logs a DeprecationWarning in Python 3.x,
use assertEqual() instead. The same goes for assertNotEquals().
Change-Id: Iaa4b2fd04d2aa661bb196c4b9f2494a281a9294a
Some files were committed or created with executable permissions 755.
Theese permissions changed to 644.
Change-Id: Ic7a99eda0518de89ff72f13936f0d131fd1eb121
Closes-Bug: #1598746
In py3 ,
Xrange is renamed to range,
so it needs to be modified to make it compatible.
the code:
print (xrange(5))
Traceback (most recent call last):
File "code", line 4, in <module>
print (xrange(5))
NameError: name 'xrange' is not defined.
Change-Id: I3a0e22876c35ced5c961c092fcda25db33d1c599
Closes-Bug: #1596127
On huge reports data we can have 500 HTTP error by timeout limitation.
In case of streaming data from the DB cursor we had hanged
DB operation inside the open transaction. For fix the issue releasing of
SQLAlchemy session was added to collector and analytics apps.
Additional changes:
- We stop to migrate action logs data to Elasticsearch due to
no one UI report uses this data.
- Parameters fixed in OSWLs 'export' call in 'all_reports'.
Change-Id: I009081a2304b36fb4ab3705b6fa09a8a3df5710b
Closes-Bug: #1564427
We have 3 services: analytics, collector, migration.
Migration is frozen and going to be removed. Also
migration requires Elasticsearch server for tests,
thus it can't be tested on OpenStack CI.
Pep8 checked on all services.
Project works only on python2.7, thus
gate-fuel-stats-python26 should be removed.
Change-Id: Ic5b6f15903918b4b0850ce0cdc12543925de0cfa
Closes-Bug: #1500073
Filtered by application stats is marked through is_filtered
field in the installation_structure.
Filtering by is_filtered added into UI queriesto Elasticsearch. Queries
tested in migration unit tests.
Queries to Elasticsearch creation in migration tests refactored.
Change-Id: Icdaa65f856aa7ea8349c39aa1bc5b8aca7368aca
Closes-Bug: #1443347
Flask application added for export clusters statstics in CSV format.
Export process streams data by the generators.
Closes-Bug: #1410262
Blueprint: export-stats-to-csv
Change-Id: I265b617e78de142f8f10f22e85f734d0df7979c2
Example of request for fetching releases added.
Exapmle of request for filtering by releases added.
Change-Id: Ide31a9166c7f1fc4eb1f8d3f314097de0722dc39
Closes-Bug: #1405748
In old style action logs we have mixing of
field types in the lists inside request_data and response_data.
Format of action_logs is already changed. So we should just
migrate old-style action_logs into Elasticsearch.
Closes-Bug: #1401120
Change-Id: I416e9815b8b4ace7b14c671b643d033e334b8326
DB model fixed
Action logs JSON schema fixed
Tasks generation added into tests
Fixed Elasticsearch mapping scheme
Fixed collector manager running script
Set of fields for es id generation is used
Closes-Bug: #1397326
Change-Id: Ifdb47d5af6a75d4306711e3929146a9a82406dcc
Loading from external configs is required for auto deployment
fuel-stats packages by the DevOps team.
For collector COLLECTOR_SETTINGS environment param is used.
Value of COLLECTOR_SETTINGS should be path to external config.
Config is regular python file. For collector standard Flask
external config file feature is used.
From collector config removed unused HOST and PORT params
For migrator parameter '--config' is added to manage_migration.py.
External config is yaml config file.
Blueprint: send-anon-usage
Change-Id: I3667d22d55b6edbbaa8f81e14dcd3d199e7a87a6
Tests improved
Test data generation added
Checking of aggregation requests added
Blueprint: send-anon-usage
Change-Id: I0b76ff368a8d6e5a2bb8d598195257d052151f45
Sync strategy changed for installation structures: modification_date
field is used as migration db sync field.
Field db_sync_field_name added into migration data.
Migration algorithm is refactored.
Change-Id: I070546ad39922008b213ec58a740cddc745c9b07
Blueprint: send-anon-usage
* migration status saved into Elasticsearch
* tox handled for migration
* JSON field added into migration
* reports tests moved into migration
* migration run script implemented
* manager for migration run implemented
* production wsgi application added for collector
* logging error in collector fixed
* installation struct model and table renamed
Change-Id: I001010a13f4798505b4d806728b3376da57c90fb
Blueprint: send-anon-usage
Andreas Jaeger