summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorJenkins <jenkins@review.openstack.org>2017-01-24 20:11:29 +0000
committerGerrit Code Review <review@openstack.org>2017-01-24 20:11:29 +0000
commit7c0bf43318ad3d2cda34d9cc033099ea35b8dead (patch)
treef774eaf3a0441963dc3915f2a5bf0c1676a84e51
parentdf4488d4620d2395c37cfb2cb05862f3caf929d4 (diff)
parent334eae80808621e7d9c4a4a14f925df7f9cb622a (diff)
Merge "Implementing pipeline architecture for Searchlight"
-rw-r--r--specs/newton/pipeline-architecture.rst109
1 files changed, 109 insertions, 0 deletions
diff --git a/specs/newton/pipeline-architecture.rst b/specs/newton/pipeline-architecture.rst
new file mode 100644
index 0000000..8629eb8
--- /dev/null
+++ b/specs/newton/pipeline-architecture.rst
@@ -0,0 +1,109 @@
1
2..
3 c) Copyright 2016 Intel Corp.
4
5 Licensed under the Apache License, Version 2.0 (the License); you may
6 not use this file except in compliance with the License. You may obtain
7 a copy of the License at
8
9 http://www.apache.org/licenses/LICENSE-2.0
10
11 Unless required by applicable law or agreed to in writing, software
12 distributed under the License is distributed on an AS IS BASIS, WITHOUT
13 WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
14 License for the specific language governing permissions and limitations
15 under the License.
16
17================================================
18Pipeline Architecture
19================================================
20
21https://blueprints.launchpad.net/searchlight/+spec/pipeline-architecture
22
23This feature enables flexible pipeline architecture to allow Searchlight to
24configure multiple publishers to consume enhanced data.
25
26Problem Description
27===================
28
29Currently when a notification comes to Searchlight, it gets processed in
30a notification handler. The handler enriches notifications in a number of ways
31and indexes enriched data into Elasticsearch. This is a simple and
32straightforward approach because Elasticsearch is the only storage backend in
33Searchlight now. As we are going to introduce notification forwarder [1], this
34process becomes inflexible. A pipeline architecture is needed to
35provide extra flexibility for users to define where enriched data is published
36to. It also allows Searchlight to support other non-elasticsearch backends
37in the future.
38
39Proposed Change
40===============
41We propose that Searchlight change to a pipeline architecture to provide
42flexibility for forwarding notifications.
43
44The current main message flow in Searchlight is looking like the below:
45Source(Notifications) -> Enrichment&Index to Elasticsearch.
46
47Currently notification handlers wait for notifications, transform them and
48index data into Elasticsearch. It's tight coupled and not modular. A
49consistent workflow is needed.
50
51With pipeline in place, the main message flow would look like:
52Source(Notifications) -> Data Transformer(Enrichment) -> Publishers(Elasticsearch, Zaqar).
53
54To achieve this, some refactors to Searchlight are needed. Notification
55handlers focus solely on capturing supported notification events.
56After that, notifications are forwarded to data transformers.
57There are mainly two kinds of data transformation. One is to normalize
58a notification into an OpenStack resource payload. The payload is api
59compatible format without publisher additional metadata. The
60normalization is always done by either calling OpenStack API services
61or updating existing Elasticsearch data. These resource transformers
62are plugin-dependent. For example, a nova create instance notification
63could be normalized into a server info document. Besides resource data
64enrichment, there might be some publisher metadata to be attached,
65like user role field, parent child relationship, version in
66Elasticsearch. These transformations should be separated from resource
67data enrichment.
68
69Publishers should implement a method to accept enriched data, notification
70information, as well as an action indicated resources CURD. For example, if a
71nova server has been updated, publishers in the pipeline will receive server
72full info, server update notification and an 'update' action. It is entirely
73up to the publisher to decide how to deal with those information.
74
75We see Elasticsearch indexing as a case of publisher. It could be the default
76publisher because for some plugins resource update needs to fetch old documents
77from Elasticsearch, partial update doesn't work without Elasticsearch, though in
78the future we may solve this issue. The order of publishers in the pipeline
79doesn't matter. Publisher can choose how to deal with errors, either request a
80requeue or just ignore the exception. The requeue operation is especially
81useful for Elasticsearch publisher, because data integrity is important for
82search functions of Searchlight. Requeue should not affect other configured
83publishers, thus a filter is needed to make sure publishers won't deliver
84same message twice.
85
86Currently Searchlight gets its data in two ways, one is incremental updates via
87notifications, the other is full indexing to ElasticSearch via API calls.
88Incremental updates are notified to all the publishers configured in the
89pipeline. For reindexing, it is up to publishers to decide if they want
90reindexing or not.
91
92There are two alternatives of pipeline design. One is the pipeline only
93consists of publishers. Notifications are normalized by resource transformer,
94then passed to configured publishers. Publishers could attach specific metadata
95inside themselves. Users can only control what publishers Searchlight data is
96heading for. Another alternative is make both transformers and publishers
97configurable. By combining different transformers and publishers, one can
98produce different pipeline on same notification.
99
100
101Alternatives
102------------
103
104
105References
106==========
107
108[1] https://blueprints.launchpad.net/searchlight/+spec/notification-forwarding
109