The contents of this repository are still available in the Git source
code management system. To see the contents of this repository before it
reached its end of life, please check out the previous commit with
"git checkout HEAD^1".
For any further questions, please email
openstack-discuss@lists.openstack.org or join #openstack-telemetry on OFTC.
Change-Id: Ic8a7001e708d8da7cf8951eefd0a96762ea5fa46
When thare're a lot of events in the database panko-expire
consumes a lot of memory to delete all records with a single
call. To avoid this behaviour a new config option
`events_delete_batch_size` was introduced.
Change-Id: Icf83ffe089301b3782273923f18efd4d209131c2
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Fix problems found.
Update local hacking checks to work with new flake8 version.
Change-Id: Id5d224358b44fd7ab4bd3d08f1db6b6faf54042f
Configuration options must be registered before using them.
In impl_elasticsearch.py, using conf.database.es_index_name
or using conf.database.es_index_name are all wrong that will
lead the api server to be invalid.
Change-Id: I6b1364954c231a477d493d4a518c080f87058123
Signed-off-by: yuanrunsen <yuanrunsen@unitedstack.com>
oslo project provide jsonutils, the others project use it
this PS to update the json moudule to oslo jsonutils.
Change-Id: I311d4f9dec7b685a5bccb9725de6010926375c7d
* Add osprofiler wsgi middleware. This middleware is used for 2 things:
1) It checks that person who wants to trace is trusted and knows
secret HMAC key.
2) It starts tracing in case of proper trace headers
and adds first wsgi trace point, with info about HTTP request
* Traces HTTP API calls
* Traces DB (SQLAlchemy) calls
Demo: https://tovin07.github.io/panko/openstack-event-list.html
HOW TO TEST?
1. Install devstack with panko as usual
2. Install osprofiler
pip install osprofiler
3. Add these line to /etc/panko/panko.conf
[profiler]
enabled = true
hmac_keys = SECRET_KEY
connection_string = redis://localhost:6379 # example when using redis
4. Pass HMAC_KEYS to client commands
- panko client with `--profile <HMAC_KEYS>`
- openstack client with `--os-profiler <HMAC_KEYS>`
Output will look like this:
To display trace use the command:
osprofiler trace show --html <TRACE_ID>
5. Use osprofiler to get the trace
osprofiler trace show \
--connection-string redis://localhost:6379 \
--out out.html \
--html <TRACE_ID>
6. Open that html file with browser and view the result
Change-Id: Ic934acbe1340a3e00361f2709f34725e0e4aa3ba
currently when there is a problem with configuration, the logging is
insufficient, especially when running panko-dbsync command.
Change-Id: I22ef74eb45f324fc6e374c4a3e03fbe9d7268eed
Logging the sqlalchemy DuplicateEntry error message on each event
is useless. And people may though it's an error.
This change moves the log level to debug.
Change-Id: I86b64e5cf370386dab6f962b09029feeb35a9dbf
Closes-Bug: #1710812
Original problem:
Event storage backend is sqlalchemy(mariaDB).
When listing events by:
openstack event list [--limit n]
or ceilometer event-list [--limit n]
using non-admin user, the number of events returned does not match the
limit 'n'. It has much fewer events than it should have. Even when
limit is not given, the default limit (100 if not changed) will apply
and that the returned result is still wrong.
Problems in code:
In the file 'panko/storage/impl_sqlalchemy.py
The subquery for trait filtering(project_id, user_id mostly) which
generated in function 'get_events' into variable 'trait_subq' (start
around line 269) does not seem behave correctly.
In the original code, the subquery is being wrapped into a deeper
layered subquery when a new trait is applied. The resulting
complicated subquery makes results have duplicate 'event_id's. And
duplicate ids still count for the amount of the result, resulting in
much fewer number of events than the one 'limit' flag sets.
Fixing:
This part(line 269-299) which generates the subquery is modified.
For each trait, the table the trait is in is adding to the 'from' part
of the query. As for the 'where' section, the trait is filtered, and
the event_id should be the same as other ones.
The function at the beginning is changed from '_build_trait_query' to
'_get_model_and_conditions' which used by the part mentioned above.
'sqlalchemy.orm.aliased' is imported to aliase tables since the same
table may be selected multiple times, and it is required to apply
'aliased' to the table model before adding them to the query.
Test:
$ tox -e py27-mysql
The difference is at function 'test_get_event_multiple_trait_filter'
in file '/tests/functional/storage/test_storage_scenarios.py'.
The order of the traits to filter can affect how the original code
produce the subquery, which is the cause of duplicate event_id.
Patch:
1. Fix coding style according to Julien Danjou's suggestion.
2. Add test code. Actually, modify existing one.
Change-Id: Ic303e4ed7af43c1d31559895c56a29b2c5ea85fc
Allow to:
- Use ssl connection to the ES cluster
- Parametrize the index name (which was hardcoded to 'events')
Change-Id: I1c72ba6b4c216a5ab7fad10d53545d15f240a26d
If adding the database fails, not using a nested transaction will make the
whole transaction passed from the caller fail. The code does not handle that at
all.
This switches to using a nested transaction, so only the insert is rolled-back
if it fails.
Change-Id: I5196147524e5fdd0d46d4c1995a8afb964ce3f6f
- we don't need to prefix 'event/storage' because that's all panko
does. cleanup inline with aodh.
- test_hbase_table_utils is not used anywhere and last reference
was from icehouse.
- drop a few places that use 'event_*' prefix since it wastes line
space
Change-Id: I348f5a3dfb59682589b4912eaed4e4b8abb15ebb
sorting and marker options are enabled for event listing (for sqlalchemy only)
using oslob.db.
sort -- list of sorting criteria. Each sorting option has to format
<sort key>:<sort direction>
Valid sort keys: message_id, generated
Valid sort directions: asc (ascending), desc (descending)
This defaults to asc if unspecified
If sort list is unspecified, this defaults to
['generated:asc', 'message_id:asc']
marker -- If specified, assumed to be an integer and assumed to be the
message id of the last object on the previous page of the results
limit is still available and supported for non sqlalchemy databases
Closes-bug: #1615699
Change-Id: I0441ca6283be5859c09c3d381ffbb52ca4a79eca
Class base.Model defines __eq__() built-in function, but does
not define __ne__() built-in function, so self.assertEqual works
but self.assertNotEqual does not work at all in this test case in
python2. This patch fixes it.
Change-Id: I22e6b8e067638e148923e7a72aa8bfdc5f29b6df
Closes-Bug: #1586268
Matthias Runge