From b285ca6486ad3cedfc182170d7970cc5d0ce590e Mon Sep 17 00:00:00 2001 From: Doug Szumski Date: Thu, 4 Oct 2018 08:47:31 +0100 Subject: [PATCH] Merge Monasca APIs into a single unified API This commit serves as a straw man for merging the Monasca APIs into a single unified API. APIImpact Story: 2003881 Task: 26742 Change-Id: Id08a46bb5b54db5baa5c3595100b73bab85ea8ff --- specs/stein/approved/merge_apis.rst | 251 ++++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) create mode 100644 specs/stein/approved/merge_apis.rst diff --git a/specs/stein/approved/merge_apis.rst b/specs/stein/approved/merge_apis.rst new file mode 100644 index 0000000..b1e9209 --- /dev/null +++ b/specs/stein/approved/merge_apis.rst @@ -0,0 +1,251 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +================== +Merge Monasca APIs +================== + +https://storyboard.openstack.org/#!/story/2003881 + +Monasca currently has 3 APIs; one for metrics, one for logs and one for +events. Historically, this has created additional overhead for developers +upgrading or adding new features to the APIs, for system administrators +configuring the APIs, and for packagers generating Monasca distributions. +The purpose of this spec is to consider whether the Monasca APIs should +be merged into a single, unified API. + +Problem description +=================== + +The Monasca APIs are written in Python and share a lot of functionality, +some, but not all of which has been factored out to the Monasca common +library. Here lies the first problem: + +1. There exists significant technical debt. + +There is a significant amount of common functionality that has not been +factored out to Monasca common. For example, the Monasca API repo contains +code for validating metrics and so does the Monasca common library. The +same is true for query authorisation, where validation code lives in +both repos. This technical debt could of course be addressed, but how +did it come about? + +2. Updating the APIs to support common features incurs significant + developer and reviewer overhead compared to updating a single API. + +The APIs are all written slightly differently. Adding support for +Oslo Policy or uWSGI to one API is not the same as adding it to another +API. Furthermore, changes frequently have to be made across multiple +repos. Whilst Zuul is well suited to coordinating this task, it requires +careful synchronisation of versions by developers and meticulous +attention from reviewers. If one modifies code which is common to all +three APIs, they must systematically verify that in each case the changes +work as intended. Of course, automated testing can provide relief here, but +it doesn't make the burden disappear entirely. + +3. Historically it has been difficult to maintain a standard experience + across multiple APIs. + +At times, APIs have diverged when in an ideal world they should not have. +For example, when adding support for a common feature, work has traditionally +been focused on one API to keep the task simple, with the view to adding +support for the other APIs at a later date. If works stops, for example due +to a release, or due to a developer moving on, the APIs can be left in a +diverged state for a significant amount of time. Of course, one solution +is to block merging of a common feature until the work is complete +in all three APIs, and all common code is added to the Monasca +common library. In practice, this has been prevented by the number of man +hours available. For example, a contributor may only be interested in +metrics. They may not be able to justify spending time working on the other +APIs. + +4. Packaging, deploying, and configuring 3 APIs is more complex than + deploying a single, unified API. + +This is an obvious point, but it can be made worse by 3). For example, +historically, it has not always been possible to run all APIs in the same +way. Configuration of common functionality such as Kafka has not +always been uniform. There is extra build system configuration, +additional keystone endpoints that need to be registered and monitoring +the availability of 3 APIs is more difficult than one. + +Use Cases +--------- + +As a developer I would save time by implementing a feature common +to all APIs in a single repo, rather than across four repos. + +As a reviewer I will save time by not having to think about how +a change may impact multiple repos. + +As a deployer I will save time by having two fewer services to deploy +and configure. + +As a security analyst I can focus my efforts reviewing a single API, +rather than three. + +As a user of Monasca I would like a consistent experience, including usability +and documentation. + +As a packager I would like to have a single Monasca API, rather than +three to save time configuring and maintaining my build system. + +Proposed change +=============== + +Merge all APIs into a single unified API. Merge all common API code from the +Monasca common repo into the unified API repository. Specifically, it is +proposed to merge all relevant code into the Monasca API. + +Alternatives +------------ + +1. Refactor all APIs so that the code is standard. Prevent merging of common + features until the work is complete across all APIs, and all common code + has been factored out into the Monasca common library. From historical + experience this will be difficult without additional developers. + +2. Don't do anything and carry on working around the technical debt. In the + long term this is likely to make it more difficult to add new features, and + require more time for maintenance. + +Data model impact +----------------- + +None + +REST API impact +--------------- + +Aside from the fact that a single service will implement the combined schema +from all APIs, the calls should not change. We should be careful when merging, +for example dimension validation code, that we do not break things which were +accepted in one API, but not another. An ideal result would be that we use +the same code for parsing common fields such as dimensions for all API calls. + +Security impact +--------------- + +If a single API was previously exposed publicly, deploying the unified API +will increase the surface area for attack. However, in general, it will be +easier to review changes to make security improvements due to the reduced +developer and reviewer overhead. + +During the deprecation period of the Events and Log API, security fixes made +to the unified API may need to be backported. + +Other end user impact +--------------------- + +All services talking to the Monasca Events and Log APIs will need +to be directed at the unified API. For services which use Keystone +to lookup the Monasca endpoints this should occur automatically. Other +services such as the Fluentd Monasca output plugin will need to be +reconfigured manually. A grace period where the Monasca Events and Log +APIs are still supported is one possibility to make this transition +easier. + +Performance Impact +------------------ + +No direct impact. + +In general, it will be less effort to profile and optimise a single API than +it would be to do the same for all three APIs. + +In clustered deployments with specialised nodes (for example a dedicated +logging node, hosting the Monasca Log API) the unified API can be deployed +as a direct replacement, and the additional functionality can simply be +ignored. + +Other deployer impact +--------------------- + +For Monasca users supporting legacy releases, any security or bug fixes +made to the unified API may need to be backported to the individual APIs. + +All actively maintained deployment solutions will need to be updated to +deploy the unified API. For example, Monasca-Docker, DevStack, Kolla, +OpenStack Ansible, and Helm. + +In the case of DevStack we should merge the three existing plugins into +one. The resulting plugin should have options like `log_pipeline_enabled` +and `metrics_pipeline_enabled` to support enabling those pipelines +separately. This is useful, for example, when DevStack is used in OpenStack +CI to allow testing changes localised to specific areas more efficiently. + +Developer impact +---------------- + +The motiviation behind this change is to reduce the burden placed on +developers and reviewers when making improvements to the Monasca APIs. It +is hoped that this will lead to an increase in developer productivity. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + + +Other contributors: + + +Work Items +---------- + +* Review test coverage of APIs, and add coverage for any missing areas. +* Review test coverage of Monasca common and add coverage for any missing areas. +* Merge Monasca common Python code into the Monasca API. Common functionality + should include: + + * authorisation + * validation + * OpenStack policy + * WSGI deployment + +* Implement Log API schema in the Monasca API and port tests. +* Implement Events API schema and port tests. +* Merge DevStack plugins into a single plugin and add support for enabling + pipelines individually. +* Deprecate Monasca Log and Event APIs. +* Merge and update documentation + +Dependencies +============ + +No additional dependencies are added. The dependency on Monasca common can +be removed. + +Testing +======= + +The Monasca API and Log API Tempest test plugins have already been merged into +one plugin. Any Tempest tests which exist for the Events API should also be +merged into the unified plugin. The Tempest plugin will need to be updated to +use the unified API. + +Unit tests from the Log API and Events API repo will need to be ported to the +Monasca API (unified) repo. Some of these tests may be redundant. + +Documentation Impact +==================== + +Three sets of documentation will be reduced to one. Whilst it will take +some effort to merge the documentation, it should hopefully be more +consistent. + +References +========== + +None + +History +======= + +None