diff --git a/README.rst b/README.rst index ee3f8d2..11bfa73 100644 --- a/README.rst +++ b/README.rst @@ -25,11 +25,14 @@ The layout of this repository is:: priorities// specs// -Where there are two sub-directories in ``specs``: +Where there are three sub-directories in ``specs``: specs//approved - specifications approved, but not yet implemented + Specifications approved, but not yet implemented specs//implemented - implemented specifications + Implemented specifications +specs//not-implemented + Specifications that were approved but are not expected to be implemented. + These are kept for historical reference. diff --git a/doc/source/specs/rocky/index.rst b/doc/source/specs/rocky/index.rst index ccb3149..3c27fee 100644 --- a/doc/source/specs/rocky/index.rst +++ b/doc/source/specs/rocky/index.rst @@ -24,3 +24,11 @@ Rocky approved (but not implemented) specs: :maxdepth: 1 approved/* + +Rocky deprecated specs: + +.. toctree:: + :glob: + :maxdepth: 1 + + not-implemented/* diff --git a/doc/source/specs/rocky/not-implemented b/doc/source/specs/rocky/not-implemented new file mode 120000 index 0000000..93d2bbc --- /dev/null +++ b/doc/source/specs/rocky/not-implemented @@ -0,0 +1 @@ +../../../../specs/rocky/not-implemented \ No newline at end of file diff --git a/specs/rocky/not-implemented/monasca-events-publishing.rst b/specs/rocky/not-implemented/monasca-events-publishing.rst new file mode 100644 index 0000000..db67291 --- /dev/null +++ b/specs/rocky/not-implemented/monasca-events-publishing.rst @@ -0,0 +1,705 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +================================================ +Monasca Events Publishing - from Ceilometer +================================================ + +https://storyboard.openstack.org/#!/story/2003023 + +Monasca Events API [1] was developed to store Openstack Notification data in Elasticsearch. There is +still a need to collect and publish Openstack Notifications to Monasca Events API. Monasca +Ceilometer project[2] currently publishes ceilometer samples[3] to Monasca API. We are proposing to +extend Monasca Ceilometer project and add a new events publisher which will publish Openstack +notifications (or events)[3] to Monasca Events API. + +UPDATE: This spec is being superceded by the ../../stein/approved/monasca-events-listener.rst spec, +but is kept here for reference. + + +Problem description +=================== + +All Openstack services generate a lot of notifications or events which contain large amounts of +operational and state information about the service and its resources. This notification data is not +currently available in Monasca. + +Ceilometer data processing pipeline[3] provides an extensible mechanism of publishing samples and +events using a custom publisher. Ceilometer samples represent a quantity that can be measured (for +e.g. the size of a volume) and events represent an occurrence of an event and do not have any +associated quantity (e.g. volume was created). + +Monasca Ceilometer project currently provides a samples publisher. Monasca Ceilometer samples +publisher converts Ceilometer samples to Monasca Metrics format which are then published to Monasca +API. There is no corresponding events publisher to Monasca yet. + +By adding an event publisher to Monasca Ceilometer project we could take advantage of Ceilometer's +event publishing mechanism to publish events to Monasca Events API. + +Ceilometer consists of different data collection components - namely Polling Agent, Notification +Agent and Compute Agent. (Please see [7] for System Architecture diagram) Ceilometer also has a data +storage and retrieval component, which would be Monasca in our case. + +Samples publisher and new proposed events publisher run within Ceilometer's Notification Agent +component and are part of Notifcation Agent's data processing pipeline. Monasca +Ceilometer presumes the need to install and deploy Ceilometer Notification Agent component (doesn't +need Polling Agent or Compute Agent deployed) on all the control nodes. Ceilometer Notification +Agent is Highly Available (HA) and can run on multiple nodes. We will have to evaluate its +performance in terms of scaling for events, but we haven't run into performance/scale problems +with current samples publisher. + + +Use Cases +--------- + +#. Openstack notification data would be stored in Elasticsearch + via the Monasca Events API + + Example sequence from Nova notification to Monasca API + #. Nova completes the creation of a VM + #. Nova generates a Notification message to oslo.messaging + #. Ceilometer Notification agent receives the Notification message + #. Ceilometer translates the Notification to a Monasca API format according to the configuration + #. Ceilometer Event Publisher publishes formatted Notification to Monasca Events API + #. Monasca Events API receives and validates formatted Notification + #. Monasca Events stores event Notification in configured Elasticsearch instance + + +Proposed change +=============== + +#. Openstack Notifications consist of envelope and payload fields + + Example Openstack Notification data format: + + .. code-block:: javascript + + { + "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1", + "_context_domain": null, + "_context_instance_lock_checked": false, + "_context_is_admin": true, + "_context_project_domain": null, + "_context_project_id": "a4f77", + "_context_project_name": "admin", + "_context_quota_class": null, + "_context_read_deleted": "no", + "_context_read_only": false, + "_context_remote_address": "192.168.245.4", + "_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460", + "_context_resource_uuid": null, + "_context_roles": [ + "monasca-user", + "admin", + "KeystoneAdmin" + ], + "_context_service_catalog": [ + { + "endpoints": [ + { + "adminURL": "http://192.168.245.8:8776/v2/a4f77", + "internalURL": "http://192.168.245.8:8776/v2/a4f77", + "publicURL": "http://192.168.245.9:8776/v2/a4f77", + "region": "region1" + } + ], + "name": "cinderv2", + "type": "volumev2" + }, + { + "endpoints": [ + { + "adminURL": "http://192.168.245.8:8776/v1/a4f77", + "internalURL": "http://192.168.245.8:8776/v1/a4f77", + "publicURL": "http://192.168.245.9:8776/v1/a4f77", + "region": "region1" + } + ], + "name": "cinder", + "type": "volume" + } + ], + "_context_show_deleted": false, + "_context_tenant": "a4f77", + "_context_timestamp": "2015-09-18T20:54:23.468522", + "_context_user": "be396488c7034811a200a3cb1d103a28", + "_context_user_domain": null, + "_context_user_id": "be396488c7034811a200a3cb1d103a28", + "_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -", + "_context_user_name": "admin", + "_unique_id": "ff9699d587bf4283a3c367ab88be1541", + "event_type": "compute.instance.create.start", + "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d", + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + }, + "priority": "INFO", + "publisher_id": "compute.ccp-compute0001-mgmt", + "timestamp": "2015-09-18 20:55:37.639023" + } + +#. All the fields with the prefix of '_context" are the envelope fields, the + other interesting fields are + + #. 'message_id' - notification identifier + #. 'payload' - contains most of the relevant and useful information in JSON format + #. 'priority' - notification priority + #. 'publisher_id' - notification publisher + #. 'timestamp' - notification timestamp + +#. Ceilometer event publishing framework converts the Openstack notifications to events format[4]. + Event publishing framework also has the ability to extract only some of the 'payload' data into + a flat set of key-value pairs called 'traits' and publish the normalized 'event' with 'traits' + extracted from the payload using a custom publisher. + + Extraction of certain fields into traits from the payload is + driven by configuration file, but by default "publisher_id", + 'request_id', 'tenant_id', 'user_id' and 'project_id' + fields are always extracted and added as 'traits'. + + The event can also have an optional field called 'raw' which has original + notification, provided 'store_raw' option is set in ceilometer.conf + + Questions/TODO: + + * Q1: Does the store_raw field only apply to events, or to all notifications processed by + Ceilometer? + * We will have to find it out if it has any adverse impact on sample publisher. Though in the + case of samples, monasca sample publisher definitely does not submit raw payload, so it must + be getting dropped. + + Example Ceilometer Event data format: + + .. code-block:: javascript + + { + "event_type": "compute.instance.create.start", + "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d", + "generated": "2015-09-18 20:55:37.639023", + "traits": { + "publisher_id": "compute.ccp-compute0001-mgmt", + "request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460", + "tenant_id": "a4f77", + "project_id": "a4f77", + "user_id": "be396488c7034811a200a3cb1d103a28" + }, + "raw": { "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1", + "_context_domain": null, + ... + ... + "event_type": "compute.instance.create.start", + "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d", + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + } + } + } + + +#. Key-Value pairs that can be extracted from 'payload' in form of traits + can be defined in events definitions file. + + For example the following events definitions yaml specifies that for + all events which have a prefix of "compute.instance.*" then + add "user_id", "instance_id", and "instance_type_id" as traits, + after extracting values from "payload.user_id", "payload.instance_id", + and "payload.instance_type_id" respectively. + + .. code-block:: yaml + + --- + - event_type: compute.instance.* + + traits: &instance_traits + user_id: + fields: payload.user_id + instance_id: + fields: payload.instance_id + instance_type_id: + type: int + fields: payload.instance_type_id + + We are for now proposing not to use this feature, of defining traits for each event + extracting since we have the ability to store entire payload, via + Monasca Events API. + + We can certainly look at enabling this feature in the future if we run into trouble storing + entire JSON "payload" in Elasticsearch. This is certainly a nifty way to trim the amount + of data that will be stored. + +#. The proposed new Custom Monasca Ceilometer event publisher will run within Ceilometer's + Notification Agent component. It will leverage Ceilometer's data processing pipeline[3] which + converts notifications to Ceilometer's event format. At the end of its processing, Monasca + Ceilometer event publisher will convert Ceilometer Event data into Monasca Event format[6] and + publish the monasca event to Monasca Events API. + +#. Monasca Events API allows a field called 'payload' which can be in an arbitrary + nested JSON format. Monasca-Ceilometer event publisher will extract JSON field called + 'payload' from 'raw' (JSON path notation: 'raw.payload'), publish the payload from the + original notification to Monasca Events API. + + Example Monasca Event Format: + + .. code-block:: javascript + + events: [ + { + dimensions": { + "service": "compute.ccp-compute0001-mgmt", + "topic": "notification.sample", + "hostname": "nova-compute:compute + }, + event: { + + "event_type": "compute.instance.create.start", + + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + } + }, + publisher_id: "compute.ccp-compute0001-mgmt", + priority: "INFO" + } + ] + +#. If no traits are specified in events pipeline yaml configuration file for an event + Ceilometer's data processing pipeline will add the following default traits: + + * service: (All notifications should have this) notification’s publisher + * tenant_id + * request_id + * project_id + * user_id + + Note: "service" is not the service that produced the event as in say "compute", "glance", + "cinder" but rather notification RabbitMQ publisher that produced the event + e.g. "compute.ccp-compute0001-mgmt" so is not very useful. + +#. Ceilometer event data is converted to Monasca event data format before being published to Monasca + Event API. Following fields in Monasca Event data are not available in current Ceilometer Event + data format: + + * "service" + * "dimensions.topic" + * "event.priority" + + We are proposing removing these fields from Monasca Event format (will be done as a separate + spec/implementation process) for the following reasons: + + "service": Currently Openstack notifications do not specify a service, that + generated the notification in a consistent way. It might be possible to create an external + mapping file which maps event name to a service but its hard to maintain such mapping over a + period of time. + + "dimensions.topic": This field is not available in the source Openstack notification + + "event.priority": This field is not currently available in Ceilometer Event format. It is + available in the source Openstack notification. Note: If we think this field can be useful we can + propose adding it to the Ceilometer Event format. + +#. Following new fields will be added to Monasca Event data as dimensions: + + * "dimensions.publisher_id": Identifier for the publisher that generated the event. Maps to + "traits.publisher_id" in Ceilometer event data. + * "dimensions.user_id": Identifier for user that generated the event. Maps to "traits.user_id" in + Ceilometer event data. + * "dimensions.project_id": Identifier of the project that generated the event. Maps to + "traits.project_id" or "traits.tenant_id" in Ceilometer event data. + +#. hostname is available in the event payload, but its location might differ from event to event. We + can use Ceilometer's event definitions config to always add a trait called "hostname" to all + events. e.g. for compute.instance.* will have a trait called "hostname", which grabs data from + "payload.hostname" + + .. code-block:: yaml + + --- + - event_type: compute.instance.* + + traits: &instance_traits + user_id: + fields: payload.hostname + +#. The proposed new Monasca Ceilometer event publisher will have the ability to submit event + data in a batch and at a configurable frequency (similar to current samples publisher). The + event data will be published if the items in the current batch reach their maximum size + (config setting) or if certain time interval has elapsed since the last publish + (config setting). This will make sure that the batch does not get huge at the same time + there is no significant delay in publishing of the events to Monasca Events API. + +#. Monasca Ceilometer event publisher will use service credentials from ceilometer configuration + file (in "[monasca]" section) to get keystone token. + + Example "[monasca]" section in ceilometer config file + .. code-block:: text + + [monasca] + service_auth_url = https://localhost:5000/v3 + service_password = secretpassword + service_username = ceilometer + service_interface = internalURL + service_auth_type = password + # service_project_id may also be used + service_project_name = admin + service_domain_name = Default + service_region_name = RegionOne + + The publisher will then make a POST request to Monasca Events /v1.0/events REST api[8] to publish + events to Monasca Events API. The URL for the instance of Monasca Events API will be configured + in the Ceilometer 'events-pipeline.yaml' file. This has the added advantage of allowing + different events to be published differently (see Ceilometer pipeline documentation [10]). + +#. "tenant_id" and "user_id" that the notification relates to are available in "payload" section + of the notification, and these notifications are generated by each service itself. + + There is no additional "Openstack-operator-agent" like component or functionality required to + fetch that data from the service and publish to monasca event api on behalf of the original + tenant. + Ceilometer publishing pipeline simply extracts these "tenant_id" and "user_id" fields from the + "payload" and makes those fields available as "tenant_id" and "user_id" traits, which would then + be mapped to "dimensions.project_id" and "dimensions.user_id" fields in monasca events format. + + In other words, original "tenant_id" and "user_id" values are available in + the payload of the notification, and will make its way to "dimensions.tenant_id" + and "dimensions.user_id" in Monasca Event. + + Questions/TODO: + * Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being + done for metrics[9] ? Would original user_id and tenant_id in "dimensions.user_id" and + "dimensions.tenant_id" fields in dimensions serve this purpose? + * Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role + "admin" could access data for all tenants, whereas a user with "ceilometer-admin" role could + access only data for a particular tenant. Can we implement something like this for + monasca-events api when querying for data? + +#. Monasca Ceilometer event publisher will also retry submitting a batch, in case Monasca + Events API is temporarily unavailable or down. The retry frequency, the number of retries + and the number of items that can be in the retry batch will also be set via configuration. + + +Alternative Solutions +--------------------- + +#. Standalone monasca event agent which reads Openstack notifications published to RabbitMQ + (on "notification" topic) and publishes them to Monasca Events API. + Pro: + * No dependency on Telemetry project. + * May be simple to develop if leverage the oslo.messaging functionality. + * Ceilometer has *deprecated* the events functionality in the Stein release. [13] + Con: + * Another agent to convince users to install on their systems. + * Reinventing work already done in the Ceilometer agent. The OpenStack community already uses Ceilometer and contributes updates when something fails. + This alternate solution will be detailed in a separate spec, as it is likely + the long term solution Monasca will need. + +#. Openstack Panko [5] is a event storage and REST API for Ceilometer. + Pro: + * An 'official' subproject within Telemetry, so there is some community recognition. + Con: + * Its primary storage is in a relational database which has problems with scale. + * It is not maintained actively and not ready for production. [11] + * It will be deprecated eventually. [12] + +Data model impact +----------------- + +None + +REST API impact +--------------- + +#. We are proposing to tweak the Monasca Event data format by removing and adding following + fields as mentioned in "Proposed change" section above. + + Remove fields (JSON path notation): "service", "dimensions.topic", + "dimensions.hostname" and "event.priority" + + Add fields (JSON path notation): "dimensions.publisher_id", "dimensions.user_id" and + "dimensions.project_id" + + This change will have an impact on Monasca Events API. + +Security impact +--------------- + +The proposed Monasca Ceilometer events publisher will collect and publish +Openstack event (notification) data to Monasca API. Openstack notification +data does not have any sensitive data like 'tokens'. +Notifications do contain 'user_id' and 'project_id' fields but do not +contain any Personally Identifiable Information (PII) for the user or +the project. + + +Other end user impact +--------------------- + +None. + +Performance Impact +------------------ + +#. The number of notifications(events) generated by different services will depend on the capacity + of the cloud along with the number of resources being created by the users. + + For example, if there was a large number of compute VM's being created or destroyed it could + lead to a surge in number of notifications (events) that would have to be published. Optimum + configuration options related to say event batch size and event batch interval would have to be + documented, to reduce any adverse affect on performance. + +#. Monasca Ceilometer publisher runs within Ceilometer Notification Agent component and invoked as a + last step in its data processing pipeline. It is an additional component that will have to to be + deployed on all the controller nodes. We will have to evaluate the performance impact of + Ceilometer Notification Agent when publishing events to Monasca Events API. + + +Other deployer impact +--------------------- + +#. The proposed new Monasca-Ceilometer events publisher will introduce + few new configuration options like + * events api endpoint + * events batch interval + * events batch size + * events retry interval + +#. Monasca Ceilometer Events publisher will have to to be added to Ceilometer's + "[ceilometer.event.publisher]" section entry_points.txt + + For example: + + [ceilometer.event.publisher] + monasca = ceilometer.publisher.monclient:MonascaEventsPublisher + +#. As part of developing new Monasca Ceilometer Events publisher devstack plugin would be updated to + add the above configuration changes. + + +Developer impact +---------------- + +#. The proposed change to Monasca Event Format will have an impact on existing Monasca Event API, + since Monasca Event Format will have to be tweaked. (See REST API Impact section above) + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + joadavis, aagate + +Other contributors: + + + +Work Items +---------- + +#. Implement new Monasca Ceilometer Events publisher. + +#. Implement monasca-ceilometer devstack plugin changes to deploy + new events publisher. + +#. Implement unit tests for Events publisher. + +#. Implement change to Monasca Event format in Monasca Events API. + + +Dependencies +============ + +#. Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654 + +#. Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer + +#. Ceilometer Data processing and pipelines: +https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html + +Testing +======= + +#. New Monasca Ceilometer Event publisher unit tests will be added, which can test publishing with + various config options events batch size, events batch interval, handling retry when Monasca + Event API is not available. + +#. Adding tempest tests for Monasca Ceilometer events publisher could be looked at as part of + separate effort. + + Please note that current Monasca Ceilometer samples publisher does not have tempest tests either + so having tempest tests for both events and samples publisher could be considered in the future. + +Documentation Impact +==================== + +#. New Monasca Events Publisher config options will be documented + + * events api endpoint + * events batch interval + * events batch size + * events retry interval + +#. Recommended values for each of the config options will also be documented based on the size of + the cloud and resources for Cloud Operators. + +References +========== + +[1] Monasca Events API 1.0: https://storyboard.openstack.org/#!/story/2001654 + +[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer + +[3] Ceilometer Data processing and pipelines: +https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html + +[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html + +[5] Openstack Panko: https://github.com/openstack/panko + +[6] Monasca Event Format: +https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json + +[7] Ceilometer System Architecture Diagram: +https://docs.openstack.org/ceilometer/ocata/architecture.html + +[8] Monasca Events POST v1.0 API: +https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc + +[9] Cross-Tenant Metric Submission: +https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission + +[10] Ceilometer pipeline yaml documentation: +https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html + +[11] No future for Panko or Aodh: +https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/ + +[12] Ceilometer Events deprecated means Panko also deprecated: +http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html + +[13] Ceilometer Events marked as deprecated in Stein: +https://review.openstack.org/#/c/603336/ \ No newline at end of file diff --git a/specs/stein/approved/monasca-events-listener.rst b/specs/stein/approved/monasca-events-listener.rst new file mode 100644 index 0000000..c01d5b1 --- /dev/null +++ b/specs/stein/approved/monasca-events-listener.rst @@ -0,0 +1,586 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +======================= +Monasca Events Listener +======================= + +https://storyboard.openstack.org/#!/story/2003023 + +Monasca Events API [1]_ was developed to store event data in Elasticsearch. +A new application could use the Monasca Events API to post an event directly for processing +and storage. +However, a general collection service is needed to capture the existing OpenStack Notifications +already generated by OpenStack Services and pass them to Monasca for storage and processing. +This specification proposes creating a new Monasca Events Listener to capture events and pass +them to Monasca services. + + +Problem description +=================== + +All Openstack services generate a lot of notifications or events which contain large amounts of +operational and state information about the service and its resources. Services such as Nova [14]_ +already publish to a RabbitMQ topic. +This notification data is not currently available in Monasca. + +Ceilometer data processing pipeline [3]_ provides an extensible mechanism of publishing samples and +events using a custom publisher. Ceilometer samples represent a quantity that can be measured +(e.g. the size of a volume) and events represent an occurrence of an event and do not have any +associated quantity (e.g. volume was created). The Telemetry project also has the Panko [5]_ +service for indexing and storing these events. + +A previous `spec <../../rocky/not-implemented/monasca-events-publishing.html>`_ was created to +specify an enhancement to Ceilometer to allow collected events to be published to the Monasca +Events API. +However, with the recent deprecation of Ceilometer's Event functionality [13]_, this is no longer +a long-term option. + +This spec is for creating a new Monasca service which would listen for OpenStack Event messages +(often called notifications) and process them through Monasca by consuming the RabbitMQ message +and producing a post to the Monasca Events API with that event. + +This service could use Ceilometer, Vitrage, Watcher, or another service as an example of how to +listen to notifications from OpenStack services such as Nova [14]_. + +It is being proposed to make the Monasca Events Listener service a part of the Monasca Agent +code base and reuse code, including the existing monasca_setup script and config. + + +Use Cases +--------- + +#. Openstack notification data would be stored in Elasticsearch via the Monasca services + + Example sequence: + + #. Nova completes the creation of a VM + #. Nova generates a Notification message to oslo.messaging + #. oslo.messaging posts the message to RabbitMQ + #. Monasca Event Listener receives the Notification message from RabbitMQ + #. Monasca Event Listener validates and translates the Notification to a Monasca Events + API format according to the configuration + #. Monasca Event Listener publishes formatted Notification to Monasca Events API + #. Monasca Events API receives and validates formatted Notification + #. Monasca Events API stores event Notification in configured Elasticsearch instance + + +Proposed change +=============== + +#. Monasca Event Listener will be a new service which can be run in a Highly + Available configuration by running an instance of Monasca Event Listener + on each Controller node in a cloud. Each node will listen to OpenStack + notifications in RabbitMQ and convert notifications to a post to the Monasca + Events API. The Monasca Events API then passes the notification on as Kafka + messages on the 'monevents' topic, where the Monasca Persister will receive + and store them. By the nature of RabbitMQ clients, the load will be + distributed between the Monasca Events Listener instances (only one listener + will process each RabbitMQ message). + +#. Monasca Event Listener will filter messages from OpenStack services based on + specification of event_types to be collected. This will reduce 'noise' and + focus event collection on those events that are deemed valuable. + This filtering specification can be from a configuration file, or optionally + could be controlled through a new API implemented as part of the Monasca + Events API service. + +#. OpenStack Notifications consist of envelope and payload fields. See [15]_ and [16]_ for + examples. + + Example OpenStack Notification data format: + + .. code-block:: javascript + + { + "_context_auth_token": "42630b3ea13242fcad20e0a92d0207f1", + "_context_domain": null, + "_context_instance_lock_checked": false, + "_context_is_admin": true, + "_context_project_domain": null, + "_context_project_id": "a4f77", + "_context_project_name": "admin", + "_context_quota_class": null, + "_context_read_deleted": "no", + "_context_read_only": false, + "_context_remote_address": "192.168.245.4", + "_context_request_id": "req-5948338c-f223-4fd8-9249-8769f7a3e460", + "_context_resource_uuid": null, + "_context_roles": [ + "monasca-user", + "admin", + "KeystoneAdmin" + ], + "_context_service_catalog": [ + { + "endpoints": [ + { + "adminURL": "http://192.168.245.8:8776/v2/a4f77", + "internalURL": "http://192.168.245.8:8776/v2/a4f77", + "publicURL": "http://192.168.245.9:8776/v2/a4f77", + "region": "region1" + } + ], + "name": "cinderv2", + "type": "volumev2" + }, + { + "endpoints": [ + { + "adminURL": "http://192.168.245.8:8776/v1/a4f77", + "internalURL": "http://192.168.245.8:8776/v1/a4f77", + "publicURL": "http://192.168.245.9:8776/v1/a4f77", + "region": "region1" + } + ], + "name": "cinder", + "type": "volume" + } + ], + "_context_show_deleted": false, + "_context_tenant": "a4f77", + "_context_timestamp": "2015-09-18T20:54:23.468522", + "_context_user": "be396488c7034811a200a3cb1d103a28", + "_context_user_domain": null, + "_context_user_id": "be396488c7034811a200a3cb1d103a28", + "_context_user_identity": "be396488c7034811a200a3cb1d103a28 a4f77 - - -", + "_context_user_name": "admin", + "_unique_id": "ff9699d587bf4283a3c367ab88be1541", + "event_type": "compute.instance.create.start", + "message_id": "c6149ba1-34b3-4367-b8c2-b1d6f073742d", + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + }, + "priority": "INFO", + "publisher_id": "compute.ccp-compute0001-mgmt", + "timestamp": "2015-09-18 20:55:37.639023" + } + +#. All the fields with the prefix of '_context' are the envelope fields, the + other interesting fields are + + #. 'message_id' - notification identifier + #. 'payload' - contains most of the relevant and useful information in JSON format + #. 'priority' - notification priority + #. 'publisher_id' - notification publisher + #. 'timestamp' - notification timestamp + +#. Monasca Event Listener converts the OpenStack notifications to Monasca events format. + This format will be suitable for Kafka messaging and will match the expected data fields + of the Monasca Persister. This conversion and validation should be common between the + Monasca Event Listener and Monasca Event API. + +#. The Kafka client connection will handle communication issues such as reconnections and + resending as needed. + +#. Monasca Events API allows a field called 'payload' which can be in an arbitrary + nested JSON format. + TODO: mappings + + Example Monasca Event Format: + + .. code-block:: javascript + + events: [ + { + dimensions": { + "service": "compute.ccp-compute0001-mgmt", + "topic": "notification.sample", + "hostname": "nova-compute:compute + }, + event: { + + "event_type": "compute.instance.create.start", + + "payload": { + "access_ip_v4": null, + "access_ip_v6": null, + "architecture": null, + "availability_zone": null, + "cell_name": "", + "created_at": "2015-09-18 20:55:25+00:00", + "deleted_at": "", + "disk_gb": 1, + "display_name": "testeee", + "ephemeral_gb": 0, + "host": null, + "hostname": "testeee", + "image_meta": { + "base_image_ref": "df0c8", + "container_format": "bare", + "disk_format": "qcow2", + "min_disk": "1", + "min_ram": "0" + }, + "image_name": "glanceaaa3", + "image_ref_url": "http://192.168.245.5:9292/images/df0c8", + "instance_flavor_id": "1", + "instance_id": "abd2ef5c-0381-434a-8efc-d7b39b28a2b6", + "instance_type": "m1.tiny", + "instance_type_id": 4, + "kernel_id": "", + "launched_at": "", + "memory_mb": 512, + "metadata": {}, + "node": null, + "os_type": null, + "progress": "", + "ramdisk_id": "", + "reservation_id": "r-1ghilddw", + "root_gb": 1, + "state": "building", + "state_description": "", + "tenant_id": "a4f77", + "terminated_at": "", + "user_id": "be396488c7034811a200a3cb1d103a28", + "vcpus": 1 + } + }, + publisher_id: "compute.ccp-compute0001-mgmt", + priority: "INFO" + } + ] + +#. Following fields in Monasca Event data may not be available in the OpenStack notification + data format: + + * "service" + * "dimensions.topic" + * "event.priority" + + We are proposing removing these fields from Monasca Event format (will be done as a separate + spec/implementation process) for the following reasons: + + "service": Currently OpenStack notifications do not specify a service, that + generated the notification in a consistent way. It might be possible to create an external + mapping file which maps event name to a service but its hard to maintain such mapping over a + period of time. + + "dimensions.topic": This field is not available in the source OpenStack notification. + However, the Monasca Event Listener may be able to save the RabbitMQ topic that the notification + was collected from. In that case, this field should be used. + + "event.priority": This field is not currently available in Ceilometer Event format. It is + available in the source OpenStack notification. Note: If we think this field can be useful we can + propose adding it to the Monasca Event Listener format. + +#. Following new fields will be added to Monasca Event data as dimensions: + + * "dimensions.publisher_id": Identifier for the publisher that generated the event. + * "dimensions.user_id": Identifier for user that generated the event. + * "dimensions.project_id": Identifier of the project that generated the event. + +#. 'hostname' is available in the event payload, but its location might differ from event to event. + +#. The proposed new Monasca Event Listener will have the ability to submit event + data in a batch and at a configurable frequency (similar to current samples publisher). The + event data will be published if the items in the current batch reach their maximum size + (config setting) or if certain time interval has elapsed since the last publish + (config setting). This will make sure that the batch does not get huge at the same time + there is no significant delay in publishing of the events to Monasca Events API. + +#. Monasca Event Listener will have a configuration file to configure connection information for + both RabbitMQ and Monasca Events API. + +#. The "tenant_id" and "user_id" that the notification relates to are available in "payload" + section of the notification, and these notifications are generated by each service itself. + + There is no additional "OpenStack-operator-agent" like component or functionality required to + fetch that data from the service and publish to monasca event api on behalf of the original + tenant. + (Ceilometer publishing pipeline simply extracts these "tenant_id" and "user_id" fields from the + "payload" and makes those fields available as "tenant_id" and "user_id" traits, which would then + be mapped to "dimensions.project_id" and "dimensions.user_id" fields in monasca events format.) + + In other words, original "tenant_id" and "user_id" values are available in + the payload of the notification, and will make its way to "dimensions.tenant_id" + and "dimensions.user_id" in Monasca Event. + + Questions/TODO: + + * Q: Do we need to do anything special to handle multi-tenancy in monasca-events api like being + done for metrics [9]_ ? Would original user_id and tenant_id in "dimensions.user_id" and + "dimensions.tenant_id" fields in dimensions serve this purpose? + + * A: Monasca Events Listener can start with sending all events to a single "admin" tenant + and if required in the future some other process could copy select metrics back to tenant + projects. + + * Q: In Ceilometer V2 API (which has been deprecated and removed), when querying data the role + "admin" could access data for all tenants, whereas a user with "ceilometer-admin" role could + access only data for a particular tenant. Can we implement something like this for + monasca-events api when querying for data? + + * A: In Monasca API every request is scoped to the project, so there is no equivalent of + Ceilometer's "admin" role to query data for all projects. So placing all events in to + an "admin" project may be the best approach. + + * Q: How should services which generate notifications but do not include a tenant_id be + handled? For example Keystone [16]_. + How does Ceilometer handle such events? + + * A: If all events are in an "admin" project then admin metrics like shared ceph cluster + load or provider network load can be copied back to tenants so they may understand how + infrastructure is affecting their workload. + + * Note: Configuration of Elasticsearch cluster is out of scope for this spec. If needed + could assign a separate Elasticsearch cluster to the events API to avoid overloads. + + +Alternative Solutions +--------------------- + +#. Reuse the Ceilometer functionality to collect and publish events to the + Monasca Events API. While this may be less work initially, Ceilometer has + deprecated the Events functionality as of Stein. [13]_ + +#. An alternate Events Listener was proposed that would listen for RabbitMQ events + then publish them directly to the 'monevents' topic in Kafka. Discussion on this + can be seen in the git history for this document and in IRC logs [18]_. + + Pro: + + * A much simpler approach, more efficient that HTTP hop through another service. + * No need for batching in service, as RabbitMQ and Kafka clients would handle fast + throughput and short network interruptions. + + Con: + + * Nova Cells v2 each have their own RabbitMQ instances. While most deployments + would likely have a centralized RabbitMQ, it is not required in the documented + architecture. + * Regions may also cause separation of RabbitMQ instances that need to be monitored. + * While it might be possible to have a service/agent in each Cell publish back to + a centralized to Kafka directly, our authentication and networking for Kafka was + not designed to support that. + +#. OpenStack Panko [5]_ is a event storage and REST API for Ceilometer and could + be used instead of a Monasca solution. + + Pro: + + * An 'official' subproject within Telemetry, so there is some community recognition. + + Con: + + * Its primary storage is in a relational database which has problems with scale. + * It is not maintained actively and not ready for production. [11]_ + * It will be deprecated eventually. [12]_ + + +Data model impact +----------------- + +None + +REST API impact +--------------- + +#. We are proposing to tweak the Monasca Event data format by removing and adding following + fields as mentioned in "Proposed change" section above. + + Remove fields or make them optional (JSON path notation): "service", "dimensions.topic", + "dimensions.hostname" and "event.priority" + + Add fields (JSON path notation): "dimensions.publisher_id", "dimensions.user_id" and + "dimensions.project_id" + + This change will have an impact on Monasca Events API. + +Security impact +--------------- + +The proposed Monasca Event Listener will collect and publish OpenStack event +(notification) data to Monasca Events API. OpenStack notification data does not +have any sensitive data like 'tokens'. +Notifications do contain 'user_id' and 'project_id' fields but do not +contain any Personally Identifiable Information (PII) for the user or +the project. + + +Other end user impact +--------------------- + +None. + +Performance Impact +------------------ + +#. The number of notifications(events) generated by different services will depend on the capacity + of the cloud along with the number of resources being created by the users. + + For example, if there was a large number of compute VM's being created or destroyed it could + lead to a surge in number of notifications (events) that would have to be published. Optimum + configuration options related to say event batch size and event batch interval would have to be + documented, to reduce any adverse affect on performance. + +#. The proposed Monasca Event Listener is a new service, so performance is unknown. However, the + Monasca API has been shown to have a high performance throughput. + + +Other deployer impact +--------------------- + +#. The proposed Monasca Event Listener will introduce a + few new configuration options like + + * RabbitMQ connection information + * Monasca Events API endpoint URL + * events batch interval + * events batch size + * events retry interval + * Keystone credentials for obtaining a token + * Conversion options for OpenStack notifications to the Monasca Event format. This may + be stored in separate pipeline configuration files, similar to how transform specs + are configured in Monasca Transform. + +#. As part of developing new Monasca Event Listener devstack plugin would be + updated to add the above configuration changes. + + +Developer impact +---------------- + +#. The proposed change to Monasca Event Format will have an impact on existing Monasca Event API, + since Monasca Event Format will have to be tweaked. (See REST API Impact section above) + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + joadavis, aagate + +Other contributors: + + + +Work Items +---------- + +#. Implement new Monasca Event Listener. + + * Connection to RabbitMQ for OpenStack Notifications + * Add filtering of notifications for configured event_types + + * Specification in configuration file + * (Optional) Creation of a new API to configure event_type subscriptions + + * Validation of OpenStack Notification data and format + * Conversion of data format to meet Monasca Events requirements + * Publishing to Monasca Events API + * Configuration of conversion specifications per-event type + +#. Implement monasca devstack plugin changes to deploy + new Events Listener service. + +#. Implement unit tests for Monasca Event Listener. + + +Dependencies +============ + +None + +Testing +======= + +#. New Monasca Event Listener unit tests will be added, which can test publishing with + various config options events batch size, events batch interval, handling retry when Monasca + Event API is not available. + +#. Adding tempest tests for Monasca Event Listener could be looked at as part of + separate effort. + + +Documentation Impact +==================== + +#. New Monasca Event Listener config options will be documented + +#. Recommended values for each of the config options will also be documented based on the size of + the cloud and resources for Cloud Operators. + +References +========== + +.. [1] Monasca Events API 1.0: https://git.openstack.org/cgit/openstack/monasca-events-api/ + +[2] Monasca Ceilometer project: https://github.com/openstack/monasca-ceilometer + +.. [3] Ceilometer Data processing and pipelines: https://docs.openstack.org/ceilometer/pike/admin/telemetry-data-pipelines.html + +[4] Ceilometer Events: https://docs.openstack.org/ceilometer/latest/admin/telemetry-events.html + +.. [5] Openstack Panko: https://github.com/openstack/panko + +[6] Monasca Event Format: https://github.com/openstack/monasca-events-api/blob/master/doc/api-samples/v1/req_simple_event.json + +[7] Ceilometer System Architecture Diagram: https://docs.openstack.org/ceilometer/ocata/architecture.html + +[8] Monasca Events POST v1.0 API: https://github.com/openstack/monasca-events-api/blob/master/api-ref/source/events.inc + +.. [9] Cross-Tenant Metric Submission: https://github.com/openstack/monasca-agent/blob/master/docs/MonascaMetrics.md#cross-tenant-metric-submission + +[10] Ceilometer pipeline yaml documentation: https://docs.openstack.org/ceilometer/latest/admin/telemetry-data-pipelines.html + +.. [11] No future for Panko or Aodh: https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/ + +.. [12] Ceilometer Events deprecated means Panko also deprecated: http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2018-10-10.log.html + +.. [13] Ceilometer Events marked as deprecated in Stein: https://review.openstack.org/#/c/603336/ + +.. [14] Nova notification version update lists services effected (see "Deprecating legacy notifications"): https://etherpad.openstack.org/p/nova-ptg-stein + +.. [15] Nova notification reference: https://docs.openstack.org/nova/latest/reference/notifications.html#existing-versioned-notifications + +.. [16] Keystone notification reference: https://docs.openstack.org/keystone/latest/advanced-topics/event_notifications.html#example-notification-project-create + +[17] Monasca Events API publisher to Kafka: https://github.com/openstack/monasca-events-api/blob/master/monasca_events_api/app/common/events_publisher.py + +.. [18] Monasca IRC meeting Dec 15, 2018: http://eavesdrop.openstack.org/meetings/monasca/2018/monasca.2018-12-12-15.00.log.html \ No newline at end of file