Commit Graph

235 Commits

Author SHA1 Message Date
Andreas Jaeger 754d7b048c Retire repository
Fuel (from openstack namespace) and fuel-ccp (in x namespace)
repositories are unused and ready to retire.

This change removes all content from the repository and adds the usual
README file to point out that the repository is retired following the
process from
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project

See also
http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011647.html

Depends-On: https://review.opendev.org/699362
Change-Id: I6a9128d5c87d7d4a2ca77db96ba6c8d9d2d91b17
2019-12-18 09:50:44 +01:00
OpenDev Sysadmins 1c38de5a04 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:31:59 +00:00
Jenkins ff88c5dc6b Merge "Removes unnecessary utf-8 encoding" 2017-02-13 19:39:33 +00:00
Jenkins 86573025b2 Merge "Replaces yaml.load() with yaml.safe_load()" 2017-02-13 19:37:44 +00:00
M V P Nitesh 46063059ea Removes unnecessary utf-8 encoding
Change-Id: Ia70c1719ce3853391dfbdc0ce4a3469902724fbb
2017-02-10 06:35:12 +00:00
gengchc2 99ad4fba53 Replaces yaml.load() with yaml.safe_load()
Yaml.load() return Python object may be dangerous if you receive
a YAML document from an untrusted source such as the Internet.
The function yaml.safe_load() limits this ability to simple Python
objects like integers or lists.

Reference:
https://security.openstack.org/guidelines/dg_avoid-dangerous-input-parsing-libraries.html

Change-Id: I9841e688e16e9bebef5eb278ee2f3a12f2ca61f4
2017-02-04 11:50:24 +08:00
Andreas Jaeger 2cf45e737a Prepare for using standard python tests
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests. This is exactly what is
currently done by OpenStack CI in project-config.

This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.

See also
http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html

Change-Id: I04f7fcd6e7031e59f3efc5c351f1f2d036bdac7d
2017-02-02 20:09:57 +01:00
Jenkins 37d1855c0f Merge "Replace assertEquals with assertEqual" 2016-12-16 09:19:45 +00:00
Jenkins 9fc79a11fa Merge "Show team and repo badges on README" 2016-12-16 09:18:03 +00:00
Jenkins 146b0bd05d Merge "Change assertTrue(isinstance()) by optimal assert" 2016-12-16 09:17:44 +00:00
Luong Anh Tuan bcf340eff3 Replace assertEquals with assertEqual
The method assertEquals has been deprecated since python 2.7.
http://docs.python.org/2/library/unittest.html#deprecated-aliases

Also in Python 3, a deprecated warning is raised when using assertEquals
therefore we should use assertEqual instead.

Change-Id: I4e99cd5d58b676f300eddcff673c26d4769e5af7
Closes-Bug: #1218185
2016-12-16 07:09:18 +00:00
gengchc2 a6ad471636 Replace six.iteritems() with .items()
1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

Change-Id: I8432a85b025a46d20484a0d8a3ec72dca54d23cb
2016-12-09 10:41:06 +08:00
Flavio Percoco 9414505b82 Show team and repo badges on README
This patch adds the team's and repository's badges to the README file.
The motivation behind this is to communicate the project status and
features at first glance.

For more information about this effort, please read this email thread:

http://lists.openstack.org/pipermail/openstack-dev/2016-October/105562.html

To see an example of how this would look like check:

https://gist.github.com/ed71c01f88810894253f80ad98f6e7a3

Change-Id: Ia2e09c233e37700ea913fa1edd28444f85ddc9d5
2016-11-25 17:36:59 +01:00
gecong1973 819f93fe0a Change assertTrue(isinstance()) by optimal assert
Some of tests use different method of
assertTrue(isinstance(A, B)) or assertEqual(type(A), B).
The correct way is to use assertIsInstance(A, B) provided
by testtools

Change-Id: I6bb51ecec188847f67b579694bd47bf7599bfbd6
2016-11-17 10:50:57 +08:00
Nam Nguyen Hoai b2f25e97a4 TrivialFix: Using assertEqual instead of assertEquals
Following OpenStack Style Guidelines[1]: http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H234] assertEquals() logs a DeprecationWarning in Python 3.x,
use assertEqual() instead. The same goes for assertNotEquals().

Change-Id: Iaa4b2fd04d2aa661bb196c4b9f2494a281a9294a
2016-09-28 11:54:37 +07:00
Jenkins 0a96410af7 Merge "Memory consumption decreased for JSON reports" 2016-08-05 16:09:16 +00:00
Alexander Kislitsky 601e81cbf8 Memory consumption decreased for JSON reports
Instead fetching of all clusters data we are selecting only
required fields for report.

Change-Id: Id5555d0d0c29a20a3d03e640398a85b961fe37ec
Closes-Bug: #1608889
2016-08-04 18:17:37 +03:00
Alexander Kislitsky dd70b95d56 Neutron QoS added to installations info report
Change-Id: If3c61a4bb1c8bb81ca4a03d20a536acf083375cc
Closes-Bug: #1604746
2016-07-22 19:15:40 +03:00
Alexander Kislitsky 8e16219249 Elasticsearch removing from fuel-stats analytics
We don't use Elasticsearch for flexible reports generation on the
fuel-stats web UI, only for five fixed reports. Thus using of
Elasticsearch is overhead and it can be removed from the servers
Instead of Elasticsearch we use fuel-stats json api calls and
PostgreSQL + Memcached.

Changes list:

 - api call added to fuel-stats json api for data required on the web UI page,
 - column release added to DB installation_structures table schema,
 - memcached is used for caching data for the web UI page,
 - elasticsearch client removed from js requirement,
 - web UI page rewritten to use fuel-stats json api instead Elaticsearch.

Co-Authored-By: Kate Pimenova <kpimenova@mirantis.com>
Change-Id: Ie752e0d0a3c80933888f986e2497b45adce730c9
Closes-Bug: #1595548
2016-07-13 14:33:24 +03:00
Alexander Kislitsky 78abf5f70c Files permmission are fixed
Some files were committed or created with executable permissions 755.
Theese permissions changed to 644.

Change-Id: Ic7a99eda0518de89ff72f13936f0d131fd1eb121
Closes-Bug: #1598746
2016-07-06 09:11:01 +00:00
Ji-Wei a3102f6fc6 Make xrange py3 compatible
In py3 ,
Xrange is renamed to range,
so it needs to be modified to make it compatible.
the code:
print (xrange(5))
Traceback (most recent call last):
  File "code", line 4, in <module>
    print (xrange(5))
NameError: name 'xrange' is not defined.

Change-Id: I3a0e22876c35ced5c961c092fcda25db33d1c599
Closes-Bug: #1596127
2016-07-01 03:15:46 +00:00
Alexander Kislitsky ce4e26fc12 Filter for 9.0 release added to web UI
Change-Id: I90341540fc40badc6dc5187e47fd9d772b43239a
Closes-Bug: #1594750
2016-06-22 16:37:16 +03:00
Alexander Kislitsky 1aa6135c1e manufacturer and platform_name removed from nodes CSV report
We had data duplication in the nodes CSV report:

 - platform_name is identical to meta.system.product
 - manufacturer is identical to meta.system.manufacturer

Product Management team uses data from meta in their reports, thus
we are removig platform_name and manufacturer fields from the nodes
CSV report.

Change-Id: I890d6e651de056b3ca3a1f2ab977652ca8d68ef9
Closes-Bug: #1585564
2016-05-27 12:03:01 +03:00
Alexander Kislitsky 85178d53c1 Interfaces detailed info added into nodes report
Interfaces info changes:

- interfaces details added
- offloading modes added
- interface properties added: SR-IOV, mtu, dpdk, numa_node

Meta info about numa topology added: numa nodes, hugepages, distances
Node error_type field removed. This field doesn't reflect actual
node state.

Check that all node attributes were exported added into test.

Change-Id: I9b5f45dad43fefb0e60a563123dab833293e4332
Closes-Bug: #1575669
Closes-Bug: #1575666
2016-05-03 17:53:25 +03:00
Alexander Kislitsky 51da64b7db Nodes CSV report implemented
Changes list:

- CSV exporter for nodes added
- link on nodes report added to the UI
- export of lists and tuples to CSV implemented. As result
  enumeration functions removed from export_utils
- aggregate functions handled in the skeleton
- lists and tuples handled in the skeleton generation util
- workaround for counting of volume attachments removed
- new fields added into installation info and OSWL skeletons
- unnecessary fields removed from installation info and OSWL skeletons
- lists definitions changed in skeletons
- common logic for plugins and nodes reports extracted to helper functions
- node related data removed from cluster report (manufacturer, platform name)

Change-Id: Iacf5421895f6803acf071111ef04a13c53eba6ac
Closes-Bug: #1572490
2016-04-21 18:29:19 +03:00
Alexander Kislitsky e200196347 Connection hanging issue fixed
On huge reports data we can have 500 HTTP error by timeout limitation.
In case of streaming data from the DB cursor we had hanged
DB operation inside the open transaction. For fix the issue releasing of
SQLAlchemy session was added to collector and analytics apps.

Additional changes:

- We stop to migrate action logs data to Elasticsearch due to
  no one UI report uses this data.
- Parameters fixed in OSWLs 'export' call in 'all_reports'.

Change-Id: I009081a2304b36fb4ab3705b6fa09a8a3df5710b
Closes-Bug: #1564427
2016-04-11 13:00:49 +03:00
Jenkins 1a1940b8dd Merge "Docs for fuel-stats were rewritten" 2016-04-06 11:58:01 +00:00
Alexander Kislitsky ebe8d4eb46 Memory consumption reduced in fuel-stats analytics
We had huge memory consumption on fetching OSWLs due to fetching
clusters data in each OSWL row.

Joined clusters data removed from each OSWL row. Explicit
fetching of version info from clusters is introduced to CSV
exporter.

We are using SQLAlchemy yield_per for fetching large query results.
Yeld_per page parameter reduced from 1000 to 100.

We still have issues with memory consumption in all
reports generation. Thus we are temporary removing link
on 'All reports' from web UI.

Example uWSGI configuration changed to start http service.

Change-Id: I66c97058c0f15f8a15b626ab1fa222106754a455
Partial-Bug: #1564427
2016-04-06 12:41:36 +03:00
Jenkins dce427279e Merge "Version info from OSWL is used in the CSV report" 2016-03-30 12:31:01 +00:00
Alexander Kislitsky ea6263af70 Cleaning test DB without Alembic migrations
On the Fuel CI we have no rights to drop and create DB.
In the tests we are downgrading and upgrading DB schema by Alembic
migrations. In case of adding migrations in one review we had
all other reviews broken due to Alembic failures on attempt to
downgrade DB schema.
As fix we are fallback to the cleaning DB without migrations
in case of Alembic error on the schema downgrade.

Change-Id: I65faf43c28427594495c67696ca190e7fe29f083
Closes-Bug: #1556876
2016-03-29 14:23:13 +00:00
Alexander Kislitsky 15944befb6 Docs for fuel-stats were rewritten
README format changed from md to rst.
Project components description added.
Old README.md content moved to 'How to' recipes in the README.rst

Change-Id: I3f6471f859cb4f0998fe8c1bba784f4da5368b1a
Closes-Bug: #1507958
2016-03-23 12:12:50 +00:00
Alexander Kislitsky 91ea0c8876 Version info from OSWL is used in the CSV report
Proper name for fuel_version info is used in the CSV reports
generation process.
If version_info is not provided by OSWL, than OpenStack version,
OS name and release are fetched from cluster data.

Change-Id: If3e7442a5b336af5c1a4f5c257f280e332ab81cf
Closes-Bug: #1547565
2016-03-21 17:40:33 +03:00
Jenkins e9ec2111a7 Merge "Config format for packages based filtering rules changed" 2016-03-15 07:46:46 +00:00
Jenkins 019ef0c6e7 Merge "Fuel version from cluster used in OSWL reports" 2016-03-10 14:57:40 +00:00
Alexander Kislitsky cd121801e6 Cluster attributes added to CSV report
Cluster attributes list synchronized with actual Nailgun attributes
list.
PostgreSQL setup provided by the OpenStack infra is used.

Change-Id: Id317f4db80459e55e62ab7885e0514d024bb7296
Closes-Bug: #1551249
2016-03-09 18:23:50 +03:00
Alexander Kislitsky fab9e71234 Config format for packages based filtering rules changed
For deploying collector service on the production nodes
DevOps team uses external collector configs, saved in the JSON.
JSON doesn't support tuple type and dict with tuple as key also
can't be serialized.
We are introducing new format for filtering rules configuration.
The following dicts can be used as filtering rule:

- {'packages_list': ['a', 'b']}
- {'packages_list': ['a', 'b']: 'from_date': None}
- {'packages_list': ['a', 'b']: 'from_date': '2016-03-10T22:34:39'}
- {'build_id': 'build_id_value'}
- {'build_id': 'build_id_value', 'from_date': None}
- {'build_id': 'build_id_value', 'from_date': '2016-03-10T22:34:39'}

The old filtering rules format is backward compatible:

- {'build_id_value': None}
- {'build_id_value': '2016-03-10T22:34:39'}

Change-Id: I1be9760bb700be5b8e20c0e27689a6b017ba75f1
Partial-Bug: #1550376
2016-03-02 18:30:52 +03:00
Alexander Kislitsky 9ce6b3f412 Fuel version from cluster used in OSWL reports
Wrong source for Fuel version was used in the OSWL report.
We used the version of Fuel installed on master node, instead of
cluster version. Thus in case of upgrade master node we had
different fuel versions in the clusters and OSWL reports for
the same master node uid.

Now we are using Fuel version from OSWL.version_info or from
the cluster data. Fuel version from cluster is used also in the
plugins report.

Change-Id: I114beae261686fb6e4d77504dd102b722a43e611
Closes-Bug: #1547565
2016-02-29 13:19:07 +03:00
Alexander Kislitsky 154a24df0f Fuel release info added into plugins CSV report
Packages info also added into plugins CSV report.

Change-Id: I2b28acfb21ff9de56b41c5b5ac52f4dd69acfa9f
Closes-Bug: #1536639
2016-02-05 15:45:21 +00:00
Alexander Kislitsky 820770807f DB fields added to action_logs for SQL speedup
On the 1M number of rows in the table JSON index begins to
work too slow. We are copy action_name and action_type values
from action_logs.body to separate indexed columns in the
action_logs table.

Redundant JSON serialization/deserealization on the python
level is removed by changing column 'structure' type from
TEXT to JSON in the InstallationStrurcure model. We already
had JSON data in 'structure' in the DB, thus only SQLAlchemy
model is changed.

Change-Id: I415f4fe607f44ac670f63f961af7fb4998f75dd6
Closes-Bug: #1540289
2016-02-05 15:32:10 +03:00
Alexander Kislitsky 4203f56d39 Filtering inst info by packages list implemented
From release 8.0 Fuel sends fuel_packages list as build marker
instead fuel_release->build_id. Now we are filtering installation
info by build_id and fuel_packages.

Change-Id: I798ad8261e9982e71afb1c53e5ce39a67b9fe097
Closes-Bug: #1533660
Closes-Bug: #1515579
2016-01-29 17:32:44 +03:00
Alexander Kislitsky 7fccde7be7 Release 8.0 added to filter in fuel-stats UI
Change-Id: I53c4d60714ec69ff700abdc2a533bbe863f58e39
Partial-Bug: #1533660
2016-01-21 10:32:52 +00:00
Jenkins 01a03ede12 Merge "OSWL resource status representation in report fixed" 2015-12-28 09:46:05 +00:00
Jenkins ff3318c1bb Merge "Tooltips fixed for hypervisor pie chart" 2015-12-26 10:43:57 +00:00
Jenkins 00542a7a78 Merge "Inner labels for hypervisor pie chart fixed" 2015-12-26 10:43:20 +00:00
Alexander Kislitsky 564d89d134 OSWL resource status representation in report fixed
If resource removed and added we show only one record
in the report with status True in is_added, is_removed.

Change-Id: I783414ee4c23d07e02d13e76eb19fe3a2b3db80e
Closes-Bug: #1526320
2015-12-25 13:01:25 +03:00
Jenkins 5311031c1f Merge "Deprecated tox -downloadcache option removed" 2015-12-25 10:00:53 +00:00
Kate Pimenova c791474713 Inner labels for hypervisor pie chart fixed
- labels with values lower than 5% are hidden now

Closes-Bug: #1516660

Change-Id: I6e2637dda5f283c1fac0bd82c08b058cbe2c36f1
2015-12-24 15:23:37 +00:00
Kate Pimenova a155462122 Tooltips fixed for hypervisor pie chart
- just playing with chart size to get enough free space for tooltips

Closes-Bug: #1516674

Change-Id: Ia9dc1109eb1ecbe9dd2a5e43404e855755ab907a
2015-12-24 18:06:41 +03:00
Alexander Kislitsky c1fe2d34d9 Version info handled for OSWLs
Version info data handled for OpenStackWorkloadStats. If master node
updated and the version info in InstallationStructure changed we have
correct version info in OSWLs stats report.

Fixes in collector:

- version_info added to DB model
- version_info added to API protocol
- tests for version_info added

Fixes in analytics:

- version_info added to DB model
- version_info fields added to CSV
- field installation structure release info removed from CSV
- version_info building on the fly for old OSWLs added

Partial-Bug: #1525902
Change-Id: I27d9d65517b4d5a7c6125b889fb0d1ba1ea213cf
2015-12-16 13:58:27 +03:00
Jenkins 4c6b98d126 Merge "Pass environment variables of proxy to tox" 2015-12-15 07:34:52 +00:00