This patch will Introduce the Topic resource into Zaqar
and implement the storage in mongodb.
Change-Id: I6d37c93aa75d7df78d3939044b0e8fefa5d9a5f5
Implements: bp introduce-topic-resource-for-notification
This is the openstack governance goal that the config files should
be merged into one folder to make the code to be better understood
and managed.
Change-Id: I33dd85145d9f5b0384323ffa51b4d68de0aeb5b7
The use of plain SQLAlchemy create_engine in Zaqar bypasses
lots of oslo_db features that are associated with its
version of create_engine, some useful and some critical.
Among favorable behaviors that are added:
* The SQLite PRAGMA FOREIGN KEYS call is abstracted into the
sqlite_fk flag
* The engine is given a pessimistic "ping" event, which emits
a "SELECT 1" upon connection checkout which will then recycle
the connection if the backing database is no longer connected.
As we are using a connection pool, if the database has been
restarted, or in the case of MySQL a default timeout of eight
hours idle has passed, existing pooled connections will be stale.
Upcoming SQLAlchemy 1.2 includes this feature as a simple flag
but for now, oslo_db implements the event listener as directed
by SQLA docs
* The errors raised by the DBAPI, and then wrapped by SQLAlchemy,
are further sub-classified. In particular, Zaqar seems to use
a lot of IntegrityError catches to detect duplicate entry and
foreign key constraint conditions. oslo_db parses these out on
a per-database-basis into individual DBDuplicateEntry,
DBReferenceError, and other error classes, allowing Zaqar to
respond specifically to the sub-class of IntegrityError or
allow it to propagate if the condition is unhandled.
* checks for MySQL SQL_MODE and best practice driver (pymysql)
will emit warnings if these conditions are not met.
* Gets Zaqar ready for further oslo.db integration and guides
new features into oslo_db, including a potential "mysql timezone"
setting
Change-Id: I16c3ed89e006e132bbd0295be1dfd0b561b2037c
Resolves-bug: https://bugs.launchpad.net/tripleo/+bug/1691951
Currently, Zaqar is using only Core SQL in an autocommit
context, meaning, no ORM, and when a statement is run,
it's on a new pool checkout, autocommit, returned.
In that case there's no reason to deal with the ORM
Session or thread-local scope of scoped_session() since there
is no state being used across requests here, other than the
"engine" itself.
Keeping in mind Zaqar wants to move to a transactionalized
pattern w/ context managers, for now they are doing the equivalent
of connectionless execution. Just switch to that for now.
Change-Id: I06bfb96381864448a874278364f06489bdc76ddd
Table creation must happen outside of API requests. This changes the
devstack plugin to create the tables properly.
Change-Id: I57c4fba06dc6725ea977477b777e4ffd6856adde
*) Add osprofiler wsgi middleware
This middleware is used for 2 things:
1) It checks that person who want to trace is trusted and knows
secret HMAC key.
2) It start tracing in case of proper trace headers
and add first wsgi trace point, with info about HTTP request.
*) Add initialization of osprofiler at start of server
Initialize and set an oslo.messaging based notifier instance
to osprofiler which be used to send notifications to Ceilometer.
*) Enable profile on existing useful storage backends
Change controller creation logic of data and control panel for
mongodb, redis and sqlalchemy storage backends, as well as
an aggregative pooling driver.
*) Add options to allow operator control profiles separately
NOTE to test this:
1) You have to enable necessary profiler option(s) base on your needed.
2) You need to enable follow services in localrc for devstack:
CEILOMETER_NOTIFICATION_TOPICS=notifications,profiler
ENABLED_SERVICES+=,ceilometer-acompute,ceilometer-acentral
ENABLED_SERVICES+=,ceilometer-anotification,ceilometer-collector
ENABLED_SERVICES+=,ceilometer-alarm-evaluator,ceilometer-alarm-notifier
ENABLED_SERVICES+=,ceilometer-api
3) You should use python-zaqarclient with this change:
I880c003511e9e4ef99806ba5b19d0ef6996be80b
Run any command with --os-profile <SECRET_KEY>
$ openstack --os-profile <SECRET_KEY> queue list
# it will print <Trace ID>
Get pretty HTML with traces:
$ osprofiler trace show --html <Trace ID>
note that osprofiler should be run from admin user name & tenant.
DocImpact
Partially-implements BP: osprofiler
Change-Id: I32565de6c447cd5e95a0ef54a9fbd4e571c2d820
Co-Authored-By: wangxiyuan <wangxiyuan@huawei.com>
Now zaqar's sql backend always use only one connection
to db. It will lead an multi-thread problem:
pymysql.err.InternalError) Packet sequence number wrong
So we should use session in sqlalchemy to avoid this happen.
Change-Id: I84e2f9899e141aed151f7c06fc4b1f603481886e
Closes-bug: #1588117
Tests creates connections to backends that we need to explicitly close
so that they don't pile up. This adds new methods to drivers,
implementation on the various backends, and cleanup calls in tests to do
so.
Change-Id: I48fb31f21403d5af56b13979eb8710cbf42948e4
This adds a new FlavorsController for sqlalchemy. It follows closely
what mongodb's controller does and it was forked from sqlalchemy's pool
controller.
Change-Id: I22d01fc93d030fbd8c97c481616f0f76bd620505
Closes-bug: #1489883
Sqlalchemy management store was missing its queue_controller property,
causing the server to fail on startup.
Change-Id: I590be382cb783918e969b62ae4b177db4eb5afef
Remove data modules from the sqlalchemy package since we don't support
them anymore.
Closes-bug: #1461962
Change-Id: I28abb510ea918ece65e5b28271b4d921221e6877
As discussed in the Liberty Design Summit "Moving apps to Python 3"
cross-project workshop, the way forward in the near future is to
switch to the pure-python PyMySQL library as a default.
https://etherpad.openstack.org/p/liberty-cross-project-python3
Change-Id: I35ac53ec8a51ef0037c81aa939b871c9d037272d
This patch removes the sqlalchemy as a valid data driver. In order to
make this work properly, we also need the data plane to be properly
split into 2 completely independent pipelines.
This patch, unfortunately, disables some of the sqlalchemy's tests for
queues and pools. The reason is that, until these planes are correctly
separated and pooling is enabled by default, we won't be able to
properly run those tests. A follow-up patch, expected to land in kilo,
will do this.
DocImpact
Implements blueprint: disable-sqlalchemy-as-pool
Change-Id: I5ca73102241692b0e1b23b3eaaaf1938f0106dfa
This patch adds the base API for notification to the storage layer,
and an init mongodb driver. The API follows pretty much the leads
of other APIs and supports basic operations that allow users to
create, delete, update and retrieve subscriptions for notification.
Partially-Implements blueprint: notifications
Change-Id: Ie065aaec16b9e6dbf9d31d93320bf6e691a4ed37
This patch adds the capabilities property to DataDrivers as a first step
to support storage capabilities. For now, capabilities are static for
storage drivers, whereas they are calculated dynamically for pooled
drivers.
Follow up patches will make drivers capabilities dynamic.
Change-Id: I29602ab5c8b3857caa3a5a6b1e9e37a5f88b2eef
Partial-blueprint: expose-storage-capabilities
To split Data/Control planes first separate groups for options are created.
Accordingly registered data and control plane options in the new group.
Modified drivers modules and many test cases.
Change-Id: I413b30b14ea605a7b3f22d6a51059a601af71e76
Given we are going to implement notification, so the 'queues'
package is not suitable for current scope of zaqar. This
patch will remove the 'queues' package.
Partially implements: blueprint notifications
Change-Id: I6984f31f4bd1e646b585c45c088ed239b58587c4
wanghao