Commit Graph

77 Commits

Author SHA1 Message Date
Andreas Jaeger 754d7b048c Retire repository
Fuel (from openstack namespace) and fuel-ccp (in x namespace)
repositories are unused and ready to retire.

This change removes all content from the repository and adds the usual
README file to point out that the repository is retired following the
process from
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project

See also
http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011647.html

Depends-On: https://review.opendev.org/699362
Change-Id: I6a9128d5c87d7d4a2ca77db96ba6c8d9d2d91b17
2019-12-18 09:50:44 +01:00
gengchc2 a6ad471636 Replace six.iteritems() with .items()
1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

Change-Id: I8432a85b025a46d20484a0d8a3ec72dca54d23cb
2016-12-09 10:41:06 +08:00
Nam Nguyen Hoai b2f25e97a4 TrivialFix: Using assertEqual instead of assertEquals
Following OpenStack Style Guidelines[1]: http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H234] assertEquals() logs a DeprecationWarning in Python 3.x,
use assertEqual() instead. The same goes for assertNotEquals().

Change-Id: Iaa4b2fd04d2aa661bb196c4b9f2494a281a9294a
2016-09-28 11:54:37 +07:00
Alexander Kislitsky 8e16219249 Elasticsearch removing from fuel-stats analytics
We don't use Elasticsearch for flexible reports generation on the
fuel-stats web UI, only for five fixed reports. Thus using of
Elasticsearch is overhead and it can be removed from the servers
Instead of Elasticsearch we use fuel-stats json api calls and
PostgreSQL + Memcached.

Changes list:

 - api call added to fuel-stats json api for data required on the web UI page,
 - column release added to DB installation_structures table schema,
 - memcached is used for caching data for the web UI page,
 - elasticsearch client removed from js requirement,
 - web UI page rewritten to use fuel-stats json api instead Elaticsearch.

Co-Authored-By: Kate Pimenova <kpimenova@mirantis.com>
Change-Id: Ie752e0d0a3c80933888f986e2497b45adce730c9
Closes-Bug: #1595548
2016-07-13 14:33:24 +03:00
Alexander Kislitsky 78abf5f70c Files permmission are fixed
Some files were committed or created with executable permissions 755.
Theese permissions changed to 644.

Change-Id: Ic7a99eda0518de89ff72f13936f0d131fd1eb121
Closes-Bug: #1598746
2016-07-06 09:11:01 +00:00
Ji-Wei a3102f6fc6 Make xrange py3 compatible
In py3 ,
Xrange is renamed to range,
so it needs to be modified to make it compatible.
the code:
print (xrange(5))
Traceback (most recent call last):
  File "code", line 4, in <module>
    print (xrange(5))
NameError: name 'xrange' is not defined.

Change-Id: I3a0e22876c35ced5c961c092fcda25db33d1c599
Closes-Bug: #1596127
2016-07-01 03:15:46 +00:00
Alexander Kislitsky e200196347 Connection hanging issue fixed
On huge reports data we can have 500 HTTP error by timeout limitation.
In case of streaming data from the DB cursor we had hanged
DB operation inside the open transaction. For fix the issue releasing of
SQLAlchemy session was added to collector and analytics apps.

Additional changes:

- We stop to migrate action logs data to Elasticsearch due to
  no one UI report uses this data.
- Parameters fixed in OSWLs 'export' call in 'all_reports'.

Change-Id: I009081a2304b36fb4ab3705b6fa09a8a3df5710b
Closes-Bug: #1564427
2016-04-11 13:00:49 +03:00
Alexander Kislitsky ea6263af70 Cleaning test DB without Alembic migrations
On the Fuel CI we have no rights to drop and create DB.
In the tests we are downgrading and upgrading DB schema by Alembic
migrations. In case of adding migrations in one review we had
all other reviews broken due to Alembic failures on attempt to
downgrade DB schema.
As fix we are fallback to the cleaning DB without migrations
in case of Alembic error on the schema downgrade.

Change-Id: I65faf43c28427594495c67696ca190e7fe29f083
Closes-Bug: #1556876
2016-03-29 14:23:13 +00:00
Alexander Kislitsky fab9e71234 Config format for packages based filtering rules changed
For deploying collector service on the production nodes
DevOps team uses external collector configs, saved in the JSON.
JSON doesn't support tuple type and dict with tuple as key also
can't be serialized.
We are introducing new format for filtering rules configuration.
The following dicts can be used as filtering rule:

- {'packages_list': ['a', 'b']}
- {'packages_list': ['a', 'b']: 'from_date': None}
- {'packages_list': ['a', 'b']: 'from_date': '2016-03-10T22:34:39'}
- {'build_id': 'build_id_value'}
- {'build_id': 'build_id_value', 'from_date': None}
- {'build_id': 'build_id_value', 'from_date': '2016-03-10T22:34:39'}

The old filtering rules format is backward compatible:

- {'build_id_value': None}
- {'build_id_value': '2016-03-10T22:34:39'}

Change-Id: I1be9760bb700be5b8e20c0e27689a6b017ba75f1
Partial-Bug: #1550376
2016-03-02 18:30:52 +03:00
Alexander Kislitsky 820770807f DB fields added to action_logs for SQL speedup
On the 1M number of rows in the table JSON index begins to
work too slow. We are copy action_name and action_type values
from action_logs.body to separate indexed columns in the
action_logs table.

Redundant JSON serialization/deserealization on the python
level is removed by changing column 'structure' type from
TEXT to JSON in the InstallationStrurcure model. We already
had JSON data in 'structure' in the DB, thus only SQLAlchemy
model is changed.

Change-Id: I415f4fe607f44ac670f63f961af7fb4998f75dd6
Closes-Bug: #1540289
2016-02-05 15:32:10 +03:00
Alexander Kislitsky 4203f56d39 Filtering inst info by packages list implemented
From release 8.0 Fuel sends fuel_packages list as build marker
instead fuel_release->build_id. Now we are filtering installation
info by build_id and fuel_packages.

Change-Id: I798ad8261e9982e71afb1c53e5ce39a67b9fe097
Closes-Bug: #1533660
Closes-Bug: #1515579
2016-01-29 17:32:44 +03:00
Alexander Kislitsky c1fe2d34d9 Version info handled for OSWLs
Version info data handled for OpenStackWorkloadStats. If master node
updated and the version info in InstallationStructure changed we have
correct version info in OSWLs stats report.

Fixes in collector:

- version_info added to DB model
- version_info added to API protocol
- tests for version_info added

Fixes in analytics:

- version_info added to DB model
- version_info fields added to CSV
- field installation structure release info removed from CSV
- version_info building on the fly for old OSWLs added

Partial-Bug: #1525902
Change-Id: I27d9d65517b4d5a7c6125b889fb0d1ba1ea213cf
2015-12-16 13:58:27 +03:00
Alexander Kislitsky 3146d40e4d Sha checksums removed from fuel version info
Instead sha checksums and build_id fuel packages info handled
in the stats collector api.

Change-Id: I05e841d4279bc61645cefb68f411a93b67e57189
Partial-Bug: #1513737
2015-11-12 14:49:12 +03:00
Alexander Kislitsky c75c0ea3d3 FILTERING_RULES handling fixed
Empty FILTERING_RULES dict should not filter any release info.

Change-Id: Ie7304efa52a5c78f3ce188f0fb5c63d1938642bd
Closes-Bug: #1513547
2015-11-11 13:31:12 +03:00
Alexander Kislitsky db1fb068b8 Fuel-stats prepared for OpenStack CI
We have 3 services: analytics, collector, migration.
Migration is frozen and going to be removed. Also
migration requires Elasticsearch server for tests,
thus it can't be tested on OpenStack CI.

Pep8 checked on all services.

Project works only on python2.7, thus
gate-fuel-stats-python26 should be removed.

Change-Id: Ic5b6f15903918b4b0850ce0cdc12543925de0cfa
Closes-Bug: #1500073
2015-10-21 14:40:44 +03:00
Alexander Kislitsky 13593e77ef Field 'fuel-nailgun_sha' added to collector protocol
In FUEL 8.0 release info data structure was changed. Field 'fuel-nailgun_sha'
was used instead 'nailgun_sha'. Information about nailgun build is
required for stats, so we change required fields list for FUEL 8.0

Change-Id: Ib37502663c0dd05a2ec1cc3df182cdb87c400059
Closes-Bug: #1500470
2015-09-29 13:06:54 +03:00
Alexander Kislitsky 2419e947b6 Indexes added on action_logs.body
For increasing speed of creating clusters report indexes on
body->>'action_name' and body->>'action_type' were added.
Query execution plan cost was optimized from 21000 to 60.

Collector base DB test fixed. Alembic constant 'base' used
for downgrade all alembic migrations instead of current
revisions number calculation.

Change-Id: Ief06aa9bbb94ebad2c2791617df8fb96cf979118
Closes-Bug: #1488092
2015-08-24 16:32:24 +03:00
Alexander Kislitsky 4b224453cc Installation info filtering rules added to collector
- Rules of filtering installation info added to the application config.
- All data is saved as before, but filtered installation info is marked
  as filtered in the DB.
- Boolean field added to installation info DB model for saving status
  of data filtration.
- DB migration for adding filtration flag added.
- Log level to applictaion logger set explicitly.
- Optional build_id parameter added to installation info JSON schema.
- Adding of migration files handled in the collector base test.
- Requirement for python-dateutil added to the collector requirements.txt

Change-Id: I99626f7f9495a0f36ed70af487d41a647250355b
Partial-Bug: #1443347
2015-05-21 11:45:54 +03:00
Jenkins 7d3e6c5f39 Merge "New fields names in the fuel-release handled" 2015-04-28 10:18:06 +00:00
Alexander Kislitsky 3ffe67073d New fields names in the fuel-release handled
ostf_sha was renamed to fuel-ostf_sha.
fuellib_sha was renamed to fuel-library_sha.
python-fuelclient_sha was added.

Change-Id: Ica4a7d479e1db3f417ec84785aa303f4e12efd41
Closes-Bug: #1447962
2015-04-27 13:07:42 +03:00
asledzinskiy 049b37e0ae Add request for OSWL collector
Change-Id: Icff7e97423b0893dc447d7fda4ac3a4ed6174faa
Implements: blueprint anonymous-statistics-performance-testing
2015-04-15 11:55:14 +03:00
Alexander Kislitsky 661ebe71c9 Fix integrity error on updating OSWLs
Fields names mapping from dict keys to object attributes was ignored.
This causes changing of id fields values in the updating DB objects
and IntegrityError was raised.

Change-Id: I35d4ec13d28ec35bcecd1bba9aeb809eec93ad85
Closes-Bug: #1427575
2015-03-06 15:35:31 +03:00
Alexander Kislitsky 32a1eb4c93 Data format of workload resource_data changed
Fuel-stats part of implementation.
In case of cluster reset we can lose information about adding and
removing workloads. For prevention format of resource_data is
changed. Now we have lists of dicts in 'added', 'remove', 'modified'
and 'current'. On cluster reset we can have several records in the
'added' with different time of adding. For 'removed' we add full
data only if it doesn't already added. In other case we add only
id with removing time.

Change-Id: I2d57f53fcb1489f177772875464056ef301b16a6
Closes-Bug: #1427757
Implements: blueprint openstack-workload-statistics
2015-03-05 15:17:03 +03:00
Alexander Kislitsky 8d16631e88 Fix for fuel-stats DB migration
Unique constraint fixed on oswl_stats

Change-Id: I6dc453e82b96677e93333d7e1123b3364d49b6de
2015-02-24 12:05:37 +03:00
Artem Roma 79be1945d3 List of collected OSWLs is extended. Images key is added
Change-Id: Ifabcfe441053865fc684943d6c7bb68add0eebe8
Implements: blueprint openstack-workload-statistic
2015-02-18 16:32:03 +02:00
Jenkins e58a5cde40 Merge "Network configuration info added to stats" 2015-02-10 13:48:14 +00:00
Alexander Kislitsky b82d267fbf Skipped status added to action_logs protocol
Closes-Bug: #1418589
Change-Id: Iee057607c2ca1c307c1aa3823f58d65bc6290e79
2015-02-10 12:04:00 +03:00
Alexander Kislitsky d60bf4e852 Network configuration info added to stats
Change-Id: Ieffcdd66e1b1adf36ba375aeb2398216eba93b42
Closes-Bug: #1410274
2015-02-06 18:41:59 +03:00
Alexander Charykov 741c39d625 Remove unneeded uwsgi requirement
Change-Id: Ib0b417738a1c31ea39aa674115a772e564a84493
2015-02-06 18:09:42 +03:00
Alexander Charykov 77f02ba38d Added db migrations to MANIFEST.in
Change-Id: Ibb714219601134a63fc08063fffbc5d9a7e92c5a
Closes-bug: #1418900
2015-02-06 10:07:34 +00:00
Alexander Kislitsky 5321a4e4b1 Collecting OSWL
Db model created.
Json scema for API created.
Blueprints urls moved to app.
Insert and update for lists of data generalized.
SQL logging enabled for test mode.

Implements: blueprint openstack-workload-statistics
Change-Id: If74e4bea0bc9c6a28ce859d775d1b33f36b547a1
2015-02-02 16:29:37 +03:00
Alexander Kislitsky 2f4eb79233 Http hander for fetching clusters stats as CSV
Flask application added for export clusters statstics in CSV format.
Export process streams data by the generators.

Closes-Bug: #1410262
Blueprint: export-stats-to-csv
Change-Id: I265b617e78de142f8f10f22e85f734d0df7979c2
2015-01-22 13:18:53 +03:00
Jenkins 1e5faf15f7 Merge "Add requests templates for performance testing" 2014-12-08 16:29:55 +00:00
Alexander Kislitsky 7117e58c92 Fix for external configs for uwsgi process
Change-Id: Icf6fa48c03adcdea0277e044a2e5629cb4e0ad18
Closes-Bug: #1398889
2014-12-03 19:43:08 +03:00
Jenkins 58e89a2d01 Merge "Fix for action_logs migration" 2014-12-03 12:33:15 +00:00
Jenkins cfcb8013fc Merge "Json schemas added into collector python package" 2014-12-02 11:45:17 +00:00
Alexander Kislitsky 77bdc11b22 Fix for action_logs migration
DB model fixed
Action logs JSON schema fixed
Tasks generation added into tests
Fixed Elasticsearch mapping scheme
Fixed collector manager running script
Set of fields for es id generation is used

Closes-Bug: #1397326
Change-Id: Ifdb47d5af6a75d4306711e3929146a9a82406dcc
2014-12-01 12:54:07 +03:00
Alexander Kislitsky 59d0b5eaa4 Json schemas added into collector python package
MANIFEST.in used for include json schemas.
Keyword 'anonymous' removed from collector package description.
Creation of log directory added into logger intialization.

Blueprint: send-anon-usage
Closes-Bug: 1395703
Change-Id: I4bce4db8484d74103b177e9063e123c319c90d65
2014-11-25 19:07:40 +04:00
Alexander Kislitsky 6201c2bf17 Config params substitution handled in managers
Loading from external configs is required for auto deployment
fuel-stats packages by the DevOps team.

For collector COLLECTOR_SETTINGS environment param is used.
Value of COLLECTOR_SETTINGS should be path to external config.
Config is regular python file. For collector standard Flask
external config file feature is used.

From collector config removed unused HOST and PORT params

For migrator parameter '--config' is added to manage_migration.py.
External config is yaml config file.

Blueprint: send-anon-usage
Change-Id: I3667d22d55b6edbbaa8f81e14dcd3d199e7a87a6
2014-11-25 16:13:56 +03:00
Aleksey Kasatkin 79516ab7d4 Filter incomplete tasks
Non-completed tasks are verified properly and skipped with 'failed' status

Blueprint: send-anon-usage

Change-Id: I876daccbc3bc8510b8d7cc5df3c949be907c509f
2014-11-13 19:21:40 +02:00
Aleksey Kasatkin b148903113 Add nodes' manufacturers to statistics snapshot
Blueprint: send-anon-usage

Change-Id: I161599955e4fdc7523bedf85f320da6242e3d857
2014-11-12 18:26:12 +02:00
Artem Panchenko 874bc3e934 Add requests templates for performance testing
Add script wich can be used in CLI to generate
'bullets' for yandex-tank, which will be used for
perfomance testing.

Change-Id: I31dea6a8aa490b5f23c5cba6e1ac1c1e5ecd95ad
Implements: blueprint anonymous-statistics-performance-testing
2014-11-06 17:45:06 +02:00
Aleksey Kasatkin 9ffa6c92f9 Extend installation_structure validation
Change-Id: I0debc8f1d0d920054d99b6d0e1a9b8f6f316cdcb
2014-10-29 19:00:57 +02:00
Jenkins edc03d4168 Merge "Add validation for action_logs.additional_info field" 2014-10-29 15:51:26 +00:00
Alexander Kislitsky 52a79a8182 Migration from PostgreSQL to Elasticsearch
* migration status saved into Elasticsearch
* tox handled for migration
* JSON field added into migration
* reports tests moved into migration
* migration run script implemented
* manager for migration run implemented
* production wsgi application added for collector
* logging error in collector fixed
* installation struct model and table renamed

Change-Id: I001010a13f4798505b4d806728b3376da57c90fb
Blueprint: send-anon-usage
2014-10-28 14:44:20 +03:00
Aleksey Kasatkin 19fd3bde08 Add validation for action_logs.additional_info field
blueprint send-anon-usage

Change-Id: I3eaf5ba0a454f2b6ec6d2ea9c798d0324090ac7c
2014-10-27 15:20:10 +02:00
Aleksey Kasatkin 8a52420d3b Replace 'aid's with 'uid's
Blueprint: send-anon-usage

Change-Id: I5da1747dac391d3e4fc818b86d7e1d2b1ea19df1
2014-10-22 11:50:13 +03:00
Jenkins d7d2bd7105 Merge "Action logs payload collection added" 2014-10-20 13:57:56 +00:00
Artem Roma 26a656d509 Requiremnts file updated
six package added to requirements of the project.

Change-Id: Ic77f3f38e042650371febdf8560a0c1f3a5dfb18
2014-10-15 14:01:26 +03:00
Aleksey Kasatkin 96b6bc7c0b Action logs payload collection added
Field 'body' is added to 'action_logs' table with its processing in handler.
'body' contains action log record, its structure is checked with json scheme.

Blueprint: send-anon-usage

Change-Id: I2ec40a9904899a72f62c1f29b58ccbe0a1d6a02d
2014-10-14 12:43:20 +03:00