Some of tests use different method of
assertTrue(isinstance(A, B)) or assertEqual(type(A), B).
The correct way is to use assertIsInstance(A, B) provided
by testtools
Change-Id: I6bb51ecec188847f67b579694bd47bf7599bfbd6
Following OpenStack Style Guidelines[1]: http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H234] assertEquals() logs a DeprecationWarning in Python 3.x,
use assertEqual() instead. The same goes for assertNotEquals().
Change-Id: Iaa4b2fd04d2aa661bb196c4b9f2494a281a9294a
Changes list:
- CSV exporter for nodes added
- link on nodes report added to the UI
- export of lists and tuples to CSV implemented. As result
enumeration functions removed from export_utils
- aggregate functions handled in the skeleton
- lists and tuples handled in the skeleton generation util
- workaround for counting of volume attachments removed
- new fields added into installation info and OSWL skeletons
- unnecessary fields removed from installation info and OSWL skeletons
- lists definitions changed in skeletons
- common logic for plugins and nodes reports extracted to helper functions
- node related data removed from cluster report (manufacturer, platform name)
Change-Id: Iacf5421895f6803acf071111ef04a13c53eba6ac
Closes-Bug: #1572490
We had huge memory consumption on fetching OSWLs due to fetching
clusters data in each OSWL row.
Joined clusters data removed from each OSWL row. Explicit
fetching of version info from clusters is introduced to CSV
exporter.
We are using SQLAlchemy yield_per for fetching large query results.
Yeld_per page parameter reduced from 1000 to 100.
We still have issues with memory consumption in all
reports generation. Thus we are temporary removing link
on 'All reports' from web UI.
Example uWSGI configuration changed to start http service.
Change-Id: I66c97058c0f15f8a15b626ab1fa222106754a455
Partial-Bug: #1564427
Proper name for fuel_version info is used in the CSV reports
generation process.
If version_info is not provided by OSWL, than OpenStack version,
OS name and release are fetched from cluster data.
Change-Id: If3e7442a5b336af5c1a4f5c257f280e332ab81cf
Closes-Bug: #1547565
Wrong source for Fuel version was used in the OSWL report.
We used the version of Fuel installed on master node, instead of
cluster version. Thus in case of upgrade master node we had
different fuel versions in the clusters and OSWL reports for
the same master node uid.
Now we are using Fuel version from OSWL.version_info or from
the cluster data. Fuel version from cluster is used also in the
plugins report.
Change-Id: I114beae261686fb6e4d77504dd102b722a43e611
Closes-Bug: #1547565
If resource removed and added we show only one record
in the report with status True in is_added, is_removed.
Change-Id: I783414ee4c23d07e02d13e76eb19fe3a2b3db80e
Closes-Bug: #1526320
Version info data handled for OpenStackWorkloadStats. If master node
updated and the version info in InstallationStructure changed we have
correct version info in OSWLs stats report.
Fixes in collector:
- version_info added to DB model
- version_info added to API protocol
- tests for version_info added
Fixes in analytics:
- version_info added to DB model
- version_info fields added to CSV
- field installation structure release info removed from CSV
- version_info building on the fly for old OSWLs added
Partial-Bug: #1525902
Change-Id: I27d9d65517b4d5a7c6125b889fb0d1ba1ea213cf
We can have duplicated OSWLs. They have the same checksum but
different external_ids. Whe should ignore such 'new' OSWLs in the
report generation process.
Change-Id: I71b5ef91eb66a9ce475a78da2485f1ec439acd4e
Closes-Bug: #1524820
Mocking of werkzeug request args raises error on exiting from
'with' block and was removed from analytics tests.
Issues in tests caused by unbound to session SQLAlchemy objects
inside test request were fixed.
Extracting of dict params from HTTP GET request fixed.
Change-Id: I7215247909a8fcd2e4c74b915670ff373538b186
Closes-Bug: #1514337
Stats info with is_filtered == True excluded from InstallationInfo
and OpenStack workloads reports.
OSWLs info for filtered InstallationInfo excluded from CSV reports
Value is_filtered is set according to FILTERING_RULES in collector
config file.
Web UI javascript libraries files added to .gitignore.
Change-Id: I9a4e2e9e953e3424139e8fa31e0b0f10d61a394a
Partial-Bug: #1443347
Json export of filtered installation infos implemented.
Limit, offset, total info added into json exporter output for
group operations.
Usage of DB commit decreased in analytics tests.
Change-Id: Ied35132ce3d194b584c8af40576a61e608862c3e
Closes-Bug: #1458528
test_dates_filtering from test_oswl_stats_to_csv.py mixed dates
hardcoded into test and current date in csv_exporter. Test now uses
current date as a base.
Change-Id: Ic57444181b02dbbb39c938ea709f3d4593da0e85
Closes-Bug: #1460590
We should generate CSV reports in case of errors in the flatten
data processing. Invalid data is skipped.
Change-Id: I22ea7cb4f22024329f3bf55fd855e04b8f4f285d
Closes-Bug: #1440080
Volume contains list of attachments. Export of attachments into CSV as enumerated fields implemented.
Number of enumrations is added into config as CSV_VOLUME_ATTACHMENTS_NUM parameter.
Enumeration of keys paths implemented in export_utils.
Volume attachment skeleton added.
Change-Id: I63eb2217c3564457a8ff4a5396eb7e7c40a76321
Closes-Bug: #1434086
Closes-Bug: #1435224
Keystone users report available by url: /api/v1/csv/keystone_user.
Test for resource types structure added.
Implements: blueprint openstack-workload-statistics
Change-Id: I1d62d0e905c49af7dbd703fe6f0f514cb4280dbf
Fuel-stats part of implementation.
In case of cluster reset we can lose information about adding and
removing workloads. For prevention format of resource_data is
changed. Now we have lists of dicts in 'added', 'remove', 'modified'
and 'current'. On cluster reset we can have several records in the
'added' with different time of adding. For 'removed' we add full
data only if it doesn't already added. In other case we add only
id with removing time.
Change-Id: I2d57f53fcb1489f177772875464056ef301b16a6
Closes-Bug: #1427757
Implements: blueprint openstack-workload-statistics
Separate test for csv_exporter added.
Exceptions classes added.
Errors handling added into app.
DB yield_per param moved to config.
Change-Id: I8b962a62c6f414b6030cca7a39e8cb2b0aafab37
Implements: blueprint openstack-workload-statistics
If resource is not changed during time period we have only
one record in DB. But in CSV time series report we should have
rows on each date for resource. For such case we are duplicating
not changed row in the CSV report. But we should remove from
resource data 'added', 'modified', 'removed' in case of duplication
of the resource row.
We are duplicating resource rows only until masternode had sent
stats. For indication of sending stat InstallationStructure creation_date
and modification_date are used.
Implements: blueprint openstack-workload-statistics
Change-Id: Ide21e17d5ebf8359fe2913a5458c9c8d499b8da5
Creation and modification dates of stats from
installation_structures table are added into get_oswls function.
Now we are selecting only required fields in get_oswls funciton.
Oswl resource_checksum and id fields removed from CSV report.
Change-Id: Ib0f00ed2bac8758988b512cc067ce74a66b7f3d8
Blueprint: export-stats-to-csv
CSV export implementation generalized for resource types
CSV export tests generalized for resource types
DB operations in tests wrapped into transaction
Transaction rollback added into DbTest.tearDown
Result file name format changed to resource_type.csv
Blueprint: export-stats-to-csv
Change-Id: I36cefec099d551d320c57abdc19fcdf29bc2c2ad
VMs info export to CSV implemented. Export uses DB. Export
uses data streaming.
Installation info exporter renamed.
Common CSV export logic extracted to export_utils module.
SQLAlchemy and psycopg added to requirements.
DB credentials added into config.
SQL debug enabled for test configuration.
Implements: blueprint openstack-workload-statistics
Change-Id: I8a9a04e1a2462a3a8d3950e20daa92025a7de11b