Connection hanging issue fixed

On huge reports data we can have 500 HTTP error by timeout limitation.
In case of streaming data from the DB cursor we had hanged
DB operation inside the open transaction. For fix the issue releasing of
SQLAlchemy session was added to collector and analytics apps.

Additional changes:

- We stop to migrate action logs data to Elasticsearch due to
  no one UI report uses this data.
- Parameters fixed in OSWLs 'export' call in 'all_reports'.

Change-Id: I009081a2304b36fb4ab3705b6fa09a8a3df5710b
Closes-Bug: #1564427
This commit is contained in:
Alexander Kislitsky 2016-04-08 14:57:14 +03:00
parent 1a1940b8dd
commit e200196347
4 changed files with 18 additions and 3 deletions

View File

@ -39,3 +39,8 @@ def date_parsing_error(error):
@app.errorhandler(NoResultFound)
def db_object_not_found(error):
return make_response(six.text_type(error), 404)
@app.teardown_appcontext
def shutdown_session(exception=None):
db.session.remove()

View File

@ -312,8 +312,8 @@ def get_all_reports(from_date, to_date, clusters_version_info):
app.logger.debug("Getting report '%s'", resource_type)
oswls = get_oswls_query(resource_type, from_date=from_date,
to_date=to_date)
report = oswl_exporter.export(resource_type, oswls,
clusters_version_info, to_date)
report = oswl_exporter.export(resource_type, oswls, to_date,
clusters_version_info)
app.logger.debug("Report '%s' got", resource_type)
yield report, '{}.csv'.format(resource_type)

View File

@ -100,3 +100,8 @@ def server_error(error):
),
500
)
@app.teardown_appcontext
def shutdown_session(exception=None):
db.session.remove()

View File

@ -44,7 +44,12 @@ def execute(params):
from migration.migrator import Migrator
migrator = Migrator()
if params.action == 'migrate':
migrator.migrate_action_logs()
# We don't use action logs in the Elasticsearch reports.
# Action logs table contains huge number of action logs,
# thus we temporarily comment this migration.
# TODO (akislitsky): uncomment this when ActionLogs
# reports will be introduced into Fuel stats web UI.
# migrator.migrate_action_logs()
migrator.migrate_installation_structure()
elif params.action == 'remove_indices':
migrator.remove_indices()