Yaml.load() return Python object may be dangerous if you receive
a YAML document from an untrusted source such as the Internet.
The function yaml.safe_load() limits this ability to simple Python
objects like integers or lists.
Reference:
https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html
Change-Id: I9841e688e16e9bebef5eb278ee2f3a12f2ca61f4
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests. This is exactly what is
currently done by OpenStack CI in project-config.
This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.
See also
http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html
Change-Id: I04f7fcd6e7031e59f3efc5c351f1f2d036bdac7d
The method assertEquals has been deprecated since python 2.7.
http://docs.python.org/2/library/unittest.html#deprecated-aliases
Also in Python 3, a deprecated warning is raised when using assertEquals
therefore we should use assertEqual instead.
Change-Id: I4e99cd5d58b676f300eddcff673c26d4769e5af7
Closes-Bug: #1218185
1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: I8432a85b025a46d20484a0d8a3ec72dca54d23cb
Some of tests use different method of
assertTrue(isinstance(A, B)) or assertEqual(type(A), B).
The correct way is to use assertIsInstance(A, B) provided
by testtools
Change-Id: I6bb51ecec188847f67b579694bd47bf7599bfbd6
Following OpenStack Style Guidelines[1]: http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H234] assertEquals() logs a DeprecationWarning in Python 3.x,
use assertEqual() instead. The same goes for assertNotEquals().
Change-Id: Iaa4b2fd04d2aa661bb196c4b9f2494a281a9294a
Instead fetching of all clusters data we are selecting only
required fields for report.
Change-Id: Id5555d0d0c29a20a3d03e640398a85b961fe37ec
Closes-Bug: #1608889
We don't use Elasticsearch for flexible reports generation on the
fuel-stats web UI, only for five fixed reports. Thus using of
Elasticsearch is overhead and it can be removed from the servers
Instead of Elasticsearch we use fuel-stats json api calls and
PostgreSQL + Memcached.
Changes list:
- api call added to fuel-stats json api for data required on the web UI page,
- column release added to DB installation_structures table schema,
- memcached is used for caching data for the web UI page,
- elasticsearch client removed from js requirement,
- web UI page rewritten to use fuel-stats json api instead Elaticsearch.
Co-Authored-By: Kate Pimenova <kpimenova@mirantis.com>
Change-Id: Ie752e0d0a3c80933888f986e2497b45adce730c9
Closes-Bug: #1595548
Some files were committed or created with executable permissions 755.
Theese permissions changed to 644.
Change-Id: Ic7a99eda0518de89ff72f13936f0d131fd1eb121
Closes-Bug: #1598746
In py3 ,
Xrange is renamed to range,
so it needs to be modified to make it compatible.
the code:
print (xrange(5))
Traceback (most recent call last):
File "code", line 4, in <module>
print (xrange(5))
NameError: name 'xrange' is not defined.
Change-Id: I3a0e22876c35ced5c961c092fcda25db33d1c599
Closes-Bug: #1596127
We had data duplication in the nodes CSV report:
- platform_name is identical to meta.system.product
- manufacturer is identical to meta.system.manufacturer
Product Management team uses data from meta in their reports, thus
we are removig platform_name and manufacturer fields from the nodes
CSV report.
Change-Id: I890d6e651de056b3ca3a1f2ab977652ca8d68ef9
Closes-Bug: #1585564
Interfaces info changes:
- interfaces details added
- offloading modes added
- interface properties added: SR-IOV, mtu, dpdk, numa_node
Meta info about numa topology added: numa nodes, hugepages, distances
Node error_type field removed. This field doesn't reflect actual
node state.
Check that all node attributes were exported added into test.
Change-Id: I9b5f45dad43fefb0e60a563123dab833293e4332
Closes-Bug: #1575669
Closes-Bug: #1575666
Changes list:
- CSV exporter for nodes added
- link on nodes report added to the UI
- export of lists and tuples to CSV implemented. As result
enumeration functions removed from export_utils
- aggregate functions handled in the skeleton
- lists and tuples handled in the skeleton generation util
- workaround for counting of volume attachments removed
- new fields added into installation info and OSWL skeletons
- unnecessary fields removed from installation info and OSWL skeletons
- lists definitions changed in skeletons
- common logic for plugins and nodes reports extracted to helper functions
- node related data removed from cluster report (manufacturer, platform name)
Change-Id: Iacf5421895f6803acf071111ef04a13c53eba6ac
Closes-Bug: #1572490
On huge reports data we can have 500 HTTP error by timeout limitation.
In case of streaming data from the DB cursor we had hanged
DB operation inside the open transaction. For fix the issue releasing of
SQLAlchemy session was added to collector and analytics apps.
Additional changes:
- We stop to migrate action logs data to Elasticsearch due to
no one UI report uses this data.
- Parameters fixed in OSWLs 'export' call in 'all_reports'.
Change-Id: I009081a2304b36fb4ab3705b6fa09a8a3df5710b
Closes-Bug: #1564427
We had huge memory consumption on fetching OSWLs due to fetching
clusters data in each OSWL row.
Joined clusters data removed from each OSWL row. Explicit
fetching of version info from clusters is introduced to CSV
exporter.
We are using SQLAlchemy yield_per for fetching large query results.
Yeld_per page parameter reduced from 1000 to 100.
We still have issues with memory consumption in all
reports generation. Thus we are temporary removing link
on 'All reports' from web UI.
Example uWSGI configuration changed to start http service.
Change-Id: I66c97058c0f15f8a15b626ab1fa222106754a455
Partial-Bug: #1564427
On the Fuel CI we have no rights to drop and create DB.
In the tests we are downgrading and upgrading DB schema by Alembic
migrations. In case of adding migrations in one review we had
all other reviews broken due to Alembic failures on attempt to
downgrade DB schema.
As fix we are fallback to the cleaning DB without migrations
in case of Alembic error on the schema downgrade.
Change-Id: I65faf43c28427594495c67696ca190e7fe29f083
Closes-Bug: #1556876
README format changed from md to rst.
Project components description added.
Old README.md content moved to 'How to' recipes in the README.rst
Change-Id: I3f6471f859cb4f0998fe8c1bba784f4da5368b1a
Closes-Bug: #1507958
Proper name for fuel_version info is used in the CSV reports
generation process.
If version_info is not provided by OSWL, than OpenStack version,
OS name and release are fetched from cluster data.
Change-Id: If3e7442a5b336af5c1a4f5c257f280e332ab81cf
Closes-Bug: #1547565
Cluster attributes list synchronized with actual Nailgun attributes
list.
PostgreSQL setup provided by the OpenStack infra is used.
Change-Id: Id317f4db80459e55e62ab7885e0514d024bb7296
Closes-Bug: #1551249
For deploying collector service on the production nodes
DevOps team uses external collector configs, saved in the JSON.
JSON doesn't support tuple type and dict with tuple as key also
can't be serialized.
We are introducing new format for filtering rules configuration.
The following dicts can be used as filtering rule:
- {'packages_list': ['a', 'b']}
- {'packages_list': ['a', 'b']: 'from_date': None}
- {'packages_list': ['a', 'b']: 'from_date': '2016-03-10T22:34:39'}
- {'build_id': 'build_id_value'}
- {'build_id': 'build_id_value', 'from_date': None}
- {'build_id': 'build_id_value', 'from_date': '2016-03-10T22:34:39'}
The old filtering rules format is backward compatible:
- {'build_id_value': None}
- {'build_id_value': '2016-03-10T22:34:39'}
Change-Id: I1be9760bb700be5b8e20c0e27689a6b017ba75f1
Partial-Bug: #1550376
Wrong source for Fuel version was used in the OSWL report.
We used the version of Fuel installed on master node, instead of
cluster version. Thus in case of upgrade master node we had
different fuel versions in the clusters and OSWL reports for
the same master node uid.
Now we are using Fuel version from OSWL.version_info or from
the cluster data. Fuel version from cluster is used also in the
plugins report.
Change-Id: I114beae261686fb6e4d77504dd102b722a43e611
Closes-Bug: #1547565
On the 1M number of rows in the table JSON index begins to
work too slow. We are copy action_name and action_type values
from action_logs.body to separate indexed columns in the
action_logs table.
Redundant JSON serialization/deserealization on the python
level is removed by changing column 'structure' type from
TEXT to JSON in the InstallationStrurcure model. We already
had JSON data in 'structure' in the DB, thus only SQLAlchemy
model is changed.
Change-Id: I415f4fe607f44ac670f63f961af7fb4998f75dd6
Closes-Bug: #1540289
From release 8.0 Fuel sends fuel_packages list as build marker
instead fuel_release->build_id. Now we are filtering installation
info by build_id and fuel_packages.
Change-Id: I798ad8261e9982e71afb1c53e5ce39a67b9fe097
Closes-Bug: #1533660
Closes-Bug: #1515579
If resource removed and added we show only one record
in the report with status True in is_added, is_removed.
Change-Id: I783414ee4c23d07e02d13e76eb19fe3a2b3db80e
Closes-Bug: #1526320
Version info data handled for OpenStackWorkloadStats. If master node
updated and the version info in InstallationStructure changed we have
correct version info in OSWLs stats report.
Fixes in collector:
- version_info added to DB model
- version_info added to API protocol
- tests for version_info added
Fixes in analytics:
- version_info added to DB model
- version_info fields added to CSV
- field installation structure release info removed from CSV
- version_info building on the fly for old OSWLs added
Partial-Bug: #1525902
Change-Id: I27d9d65517b4d5a7c6125b889fb0d1ba1ea213cf