From 5e22802920b918a81db0cc2a4a78d6df055b1119 Mon Sep 17 00:00:00 2001 From: Tim Hinrichs Date: Fri, 14 Nov 2014 13:09:28 -0800 Subject: [PATCH] Create table publishing middleware Make it easy for other projects like Nova/Neutron to publish data in the format that Congress expects. Change-Id: Ifdebf0f7150cd04ac27eeea7dec7a0c1ef1de341 --- specs/liberty/table-publishing-middleware.rst | 174 ++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 specs/liberty/table-publishing-middleware.rst diff --git a/specs/liberty/table-publishing-middleware.rst b/specs/liberty/table-publishing-middleware.rst new file mode 100644 index 0000000..9529305 --- /dev/null +++ b/specs/liberty/table-publishing-middleware.rst @@ -0,0 +1,174 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +========================================== +Create table publishing middleware +========================================== + +https://blueprints.launchpad.net/congress/+spec/table-publishing-middleware + +To lower the barrier to entry for other projects to publish data to Congress +on the (extended) DSE, create middleware that automates as much of the process +as possible. + + +Problem description +=================== + +Currently Congress periodically polls the services it is managing to get their +data. It would be preferable if services sent data to Congress only when that +data changes. For example, instead of Congress pulling the list of Nova's +servers every minute, Nova would send its servers to Congress whenever a +new server is created, an existing server is deleted, or an existing server +is updated. + +oslo.messaging already makes it possible for projects to publish data +to other projects, but that mechanism is underutilized today. + + +Proposed change +=============== + +Solving this problem means making it even easier for other projects to send their +data to Congress than oslo.messaging does. This change will include middleware, +perhaps made available via oslo, that publishes all changes to the underlying +database tables on the bus. Moreover, it will +send not the entire table but rather the delta on the table that occurred. +It will be tightly integrated into existing oslo.db so that existing projects +need make no code changes; they need only include and configure the code. + +There will need to be a synchronization mechanism: Congress pulls all the data +once and then needs deltas published on the bus from that point on. The deltas +sent on the bus will include timestamps that allow Congress to synchronize +the initial pull of data with the deltas. + + +Alternatives +------------ + +Another option is to have a separate effort, say oslo.publish_tables, that +is not so tightly integrated with oslo.db. The upside to this approach is that +projects not using oslo.db (even those outside of OpenStack) could leverage +the code. The downside to this approach is that projects already using oslo.db +would need to write and maintain a bunch of special-purpose code for publishing +their data--an approach that has seen little success til now. + +Perhaps we will find that we can implement both approaches with little extra +effort, but the primary goal of this spec is the tight integration with +oslo.db. + + +Policy +------ + +None + +Policy Actions +-------------- + +None + +Data Sources +------------ + +None + +Data model impact +----------------- + +None + +REST API impact +--------------- + +None + + +Security impact +--------------- + +None + +Notifications impact +-------------------- + +The goal of this spec is to improve the ability of other projects to +utilize the notifications mechanism provided by oslo.messaging. + + +Other end user impact +--------------------- + +None + +Performance Impact +------------------ + +There will be minimal performance impact on the projects that utilize this +code because only the deltas that hit the database will be published on +the bus. Moreover, because each project can configure which tables +are published on the bus, the project owner can tune any +possible performance impact. + + +Other Deployer Impacts +---------------------- + +When configuring the middleware, we propose one key configuration option: +which tables should be published on the bus. + +Developer Impact +---------------- +None + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + + +Other contributors: + + +Work Items +---------- + +- Write basic middleware functionality +- Integrate it into oslo.db +- Install it into an existing service, say Nova. +- Standalone documentation + +Dependencies +============ + +enable-delta-driven-datasources: enable the datasources/DatasourceDriver +class to connect to datasources that send deltas. + + +Testing +======= + +- Unit testing the basic functionality +- Tempest tests + o Verify that changes to tables configured to be published actually + get published + o Verify that changes to tables configured NOT to be published do not + get published + +Documentation Impact +==================== + +Need documentation in oslo for the basic middleware. +Need documentation update to oslo.db + + +References +========== + +None