Merge "Create table publishing middleware"

This commit is contained in:
Jenkins 2015-06-09 14:25:18 +00:00 committed by Gerrit Code Review
commit 0622286937
1 changed files with 174 additions and 0 deletions

View File

@ -0,0 +1,174 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Create table publishing middleware
==========================================
https://blueprints.launchpad.net/congress/+spec/table-publishing-middleware
To lower the barrier to entry for other projects to publish data to Congress
on the (extended) DSE, create middleware that automates as much of the process
as possible.
Problem description
===================
Currently Congress periodically polls the services it is managing to get their
data. It would be preferable if services sent data to Congress only when that
data changes. For example, instead of Congress pulling the list of Nova's
servers every minute, Nova would send its servers to Congress whenever a
new server is created, an existing server is deleted, or an existing server
is updated.
oslo.messaging already makes it possible for projects to publish data
to other projects, but that mechanism is underutilized today.
Proposed change
===============
Solving this problem means making it even easier for other projects to send their
data to Congress than oslo.messaging does. This change will include middleware,
perhaps made available via oslo, that publishes all changes to the underlying
database tables on the bus. Moreover, it will
send not the entire table but rather the delta on the table that occurred.
It will be tightly integrated into existing oslo.db so that existing projects
need make no code changes; they need only include and configure the code.
There will need to be a synchronization mechanism: Congress pulls all the data
once and then needs deltas published on the bus from that point on. The deltas
sent on the bus will include timestamps that allow Congress to synchronize
the initial pull of data with the deltas.
Alternatives
------------
Another option is to have a separate effort, say oslo.publish_tables, that
is not so tightly integrated with oslo.db. The upside to this approach is that
projects not using oslo.db (even those outside of OpenStack) could leverage
the code. The downside to this approach is that projects already using oslo.db
would need to write and maintain a bunch of special-purpose code for publishing
their data--an approach that has seen little success til now.
Perhaps we will find that we can implement both approaches with little extra
effort, but the primary goal of this spec is the tight integration with
oslo.db.
Policy
------
None
Policy Actions
--------------
None
Data Sources
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
The goal of this spec is to improve the ability of other projects to
utilize the notifications mechanism provided by oslo.messaging.
Other end user impact
---------------------
None
Performance Impact
------------------
There will be minimal performance impact on the projects that utilize this
code because only the deltas that hit the database will be published on
the bus. Moreover, because each project can configure which tables
are published on the bus, the project owner can tune any
possible performance impact.
Other Deployer Impacts
----------------------
When configuring the middleware, we propose one key configuration option:
which tables should be published on the bus.
Developer Impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work Items
----------
- Write basic middleware functionality
- Integrate it into oslo.db
- Install it into an existing service, say Nova.
- Standalone documentation
Dependencies
============
enable-delta-driven-datasources: enable the datasources/DatasourceDriver
class to connect to datasources that send deltas.
Testing
=======
- Unit testing the basic functionality
- Tempest tests
o Verify that changes to tables configured to be published actually
get published
o Verify that changes to tables configured NOT to be published do not
get published
Documentation Impact
====================
Need documentation in oslo for the basic middleware.
Need documentation update to oslo.db
References
==========
None