Add Mirantis Stacklight plugin and serializer

Change-Id: I747bd9fcb44c2fbb303aeae0f4b44da08069183b
This commit is contained in:
Vitaliy Kharechko 2016-05-16 13:48:57 +03:00
parent 2287e165aa
commit d9c0b5f106
4 changed files with 2538 additions and 0 deletions

View File

@ -44,3 +44,13 @@ port: 8082
#auth_url: http://127.0.0.1:35357/v3
#endpoint: http://127.0.0.1:8070/v2.0
#api_version: 2_0
# uncomment for stacklight
#[stacklight]
# ip address and port of Heka HttpListenInput plugin
#ip_address: 172.16.170.184
#port: 8088

View File

@ -0,0 +1,66 @@
# (C) Copyright Broadcom Corporation 2016
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
#
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from broadviewpublisherbase import BroadViewPublisherBase
from broadview_collector.serializers.bst_to_stacklight import BSTToStacklight
import json
import ConfigParser
import requests
from oslo_log import log
LOG = log.getLogger(__name__)
class BroadViewPublisher(BroadViewPublisherBase):
def readConfig(self):
self._ipaddr = "127.0.0.1"
self._port = 8088
self._timeout = 0.5
try:
bvcfg = ConfigParser.ConfigParser()
bvcfg.read("/etc/broadviewcollector.conf")
self._ipaddr = bvcfg.get("stacklight", "ip_address")
self._port = bvcfg.get("stacklight", "port")
self._timeout = bvcfg.get("stacklight", "timeout")
except:
LOG.error("BroadViewPublisher: unable to read configuration")
def __init__(self):
self.readConfig()
def publish(self, host, data):
code = 500
success, sdata = BSTToStacklight().serialize(host, data)
if success:
sdata = json.loads(sdata)
for x in sdata:
try:
r = requests.post('http://{}:{}'.format(self._ipaddr, self._port), json=sdata, timeout=self._timeout)
code = r.status_code
except requests.exceptions.Timeout:
LOG.error('BroadViewPublisher timeout {} seconds {}:{}'.format(self._timeout, self._ipaddr, self._port))
code = 500
break
except requests.exceptions.RequestException as e:
LOG.error('BroadViewPublisher {} {}:{}'.format(e, self._ipaddr, self._port))
code = 500
break
if r.status_code != 200:
LOG.error('BroadViewPublisher received {}'.format(r.status_code))
break
return code
def __repr__(self):
return "BroadView Stacklight Publisher"

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,209 @@
Stacklight BST Serializer Format
================================
Stacklight serializer generates a JSON object with the following common
key/value pairs:
* entity - for BST statistics, the string "broadview-bst"
* name - the statistic name (see below for the set of possible names)
* timestamp - event time in microseconds (floating point)
* value - metric value
* asic-id - the number of the ASIC for which the stat is being reported
* metric - metric name (not present for "device" statistic)
* bv-agent - the IP address of the BroadView agent that reported the data
The data is hierarchical - the entity "broadview-bst" is the parent to a
number of statistics, and a statistic can have 0 or more metrics.
This document details the format of Stacklight metrics as generated by the
BST to Stacklight serializer (serializers/bst_to_stacklight.py)
BST
---
This section describes in more detail the format currently used for BST
metrics published to Stacklight.
# entity
The name field consists of the string "broadview-bst".
# name
BST statistic names consist of the following:
* device
* ingress-port-priority-group
* ingress-port-service-pool
* ingress-service-pool
* egress-cpu-queue
* egress-mc-queue
* egress-port-service-pool
* egress-rqe-queue
* egress-service-pool
* egress-uc-queue
* egress-uc-queue-group
###Example: "name": "egress-port-service-pool"
# timestamp
The timestamp field is a floating point value which represents that time in
microseconds of the statistic.
###Example: "timestamp": 1459361118000.0
# value
The value field represents that reported value for the statistic.
###Example: "value": 366
# asic-id
The ID of the ASIC on the device to which the report corresponds.
###Example: "asic-id": 3
# bv-agent
This is the IPV4 internet address of the device (e.g., switch) that hosts
the BroadView agent from which the report originated.
###Example: "bv-agent": "192.168.3.47"
# metric
Note the structure of the "name" field values for BST. Except for "device",
the name consists of the string "ingress" or "egress", followed by a '-'
separated list of strings. These names imply some of the metadata that
is associated with the named stat. For example, names which contain
the "-port-" substring will have "port" metadata. Similarly, a name with
"-service-pool-" substring will have "service-pool" metadata.
Finally, be aware that a statistic reported by BroadView often comes in
the form of an aggregate. For a given named statistic, there might be
multiple metrics reported. In the case of ingress-port-priority-group,
for example, both um-share-buffer-count and um-headroom-count are reported
together in an array. They share the same metadata: asic-id, port, and
priority-group (as implied by the name of the statistic). broadview-lib
flattens this aggregate report into two separate metrics, one for
um-share-bufffer-count, and one for um-headroom-count. This flattening
aids indexers of the data; both metrics can be retrieved by searching
on ingress-port-priority-group, or a more refined search can obtain just
the um-share-bufffer-count data by providing that metadata as a search
key.
The following sections define the metrics and related metadata that are
generated by the BST to Stacklight serializer for each BST statistic.
## device
The device "name" has no metric field.
###Example:
{"asic-id": "20", "name": "device", "timestamp": 1463014303000.0, "bv-agent": "172.16.170.184", "value": 46000, "entity": "broadview-bst"}
## ingress-port-priority-group
* priority-group - the priority group associated with the stat
* port - the port impacted by the statistic
* metric - either um-share-buffer-count or um-headroom-buffer-count
###Example:
{"asic-id": "20", "name": "ingress-port-priority-group", "bv-agent": "172.16.170.184", "metric": "um-share-buffer-count", "priority-group": 5, "value": 45500, "entity": "broadview-bst", "timestamp": 1463014303000.0, "port": "2"}
{"asic-id": "20", "name": "ingress-port-priority-group", "bv-agent": "172.16.170.184", "metric": "um-headroom-buffer-count", "priority-group": 5, "value": 44450, "entity": "broadview-bst", "timestamp": 1463014303000.0, "port": "2"}
## ingress-port-service-pool
* service-pool - the service pool associated with the stat
* port - the port impacted by the statistic
* metric - um-share-buffer-count
###Example:
{"asic-id": "20", "service-pool": 5, "name": "ingress-port-service-pool", "timestamp": 1463014303000.0, "metric": "um-share-buffer-count", "bv-agent": "172.16.170.184", "value": 10000, "entity": "broadview-bst", "port": "2"}
## ingress-service-pool
* service-pool - the service pool associated with the stat
* metric - um-share-buffer-count
###Example:
{"asic-id": "20", "service-pool": 1, "name": "ingress-service-pool", "timestamp": 1463014303000.0, "metric": "um-share-buffer-count", "bv-agent": "172.16.170.184", "value": 3240, "entity": "broadview-bst"}
## egress-cpu-queue
* queue - the queue associated with the statistic
* metric - either cpu-buffer-count or cpu-queue-entries
###Example:
{"asic-id": "20", "name": "egress-cpu-queue", "timestamp": 1463014303000.0, "metric": "cpu-buffer-count", "bv-agent": "172.16.170.184", "value": 4566, "entity": "broadview-bst", "queue": 3}
{"asic-id": "20", "name": "egress-cpu-queue", "timestamp": 1463014303000.0, "metric": "cpu-queue-entries", "bv-agent": "172.16.170.184", "value": 0, "entity": "broadview-bst", "queue": 3}
## egress-mc-queue
* queue - the queue associated with the statistic
* port - the port associated with the statistic
* metric - either mc-buffer-count or mc-queue-entries
###Example:
{"asic-id": "20", "name": "egress-mc-queue", "timestamp": 1463014303000.0, "metric": "mc-buffer-count", "bv-agent": "172.16.170.184", "value": 34, "entity": "broadview-bst", "queue": 1, "port": "1"}
{"asic-id": "20", "name": "egress-mc-queue", "timestamp": 1463014303000.0, "metric": "mc-queue-entries", "bv-agent": "172.16.170.184", "value": 89, "entity": "broadview-bst", "queue": 1, "port": "1"}
## egress-port-service-pool
* service-pool - the service pool associated with the statistic
* port - the port associated with the statistic
* metric - either um-share-buffer-count, mc-share-buffer-count or mc-share-queue-entries
###Example:
{"asic-id": "20", "service-pool": 5, "name": "egress-port-service-pool", "timestamp": 1463014303000.0, "metric": "um-share-buffer-count", "bv-agent": "172.16.170.184", "value": 0, "entity": "broadview-bst", "port": "2"}
{"asic-id": "20", "service-pool": 5, "name": "egress-port-service-pool", "timestamp": 1463014303000.0, "metric": "mc-share-buffer-count", "bv-agent": "172.16.170.184", "value": 24000, "entity": "broadview-bst", "port": "2"}
{"asic-id": "20", "service-pool": 5, "name": "egress-port-service-pool", "timestamp": 1463014303000.0, "metric": "mc-share-queue-entries", "bv-agent": "172.16.170.184", "value": 0, "entity": "broadview-bst", "port": "2"}
## egress-rqe-queue
* metric - either rqe-buffer-count or rqe-queue-entries
###Example:
{"asic-id": "20", "name": "egress-rqe-queue", "timestamp": 1463014303000.0, "metric": "rqe-buffer-count", "bv-agent": "172.16.170.184", "value": 3333, "entity": "broadview-bst", "queue": 2}
{"asic-id": "20", "name": "egress-rqe-queue", "timestamp": 1463014303000.0, "metric": "rqe-queue-entries", "bv-agent": "172.16.170.184", "value": 4444, "entity": "broadview-bst", "queue": 2}
## egress-service-pool
* service-pool - the service pool associated with the statistic
* metric - either um-share-buffer-count, mc-share-buffer-count or mc-share-queue-entries
###Example:
{"asic-id": "20", "service-pool": 2, "name": "egress-service-pool", "timestamp": 1463014303000.0, "metric": "um-share-buffer-count", "bv-agent": "172.16.170.184", "value": 5700, "entity": "broadview-bst"}
{"asic-id": "20", "service-pool": 2, "name": "egress-service-pool", "timestamp": 1463014303000.0, "metric": "mc-share-buffer-count", "bv-agent": "172.16.170.184", "value": 4567, "entity": "broadview-bst"}
{"asic-id": "20", "service-pool": 2, "name": "egress-service-pool", "timestamp": 1463014303000.0, "metric": "mc-share-queue-entries", "bv-agent": "172.16.170.184", "value": 3240, "entity": "broadview-bst"}
## egress-uc-queue
* queue - the queue associated with the statistic
* port - the port associated with the statistic
* metric - uc-queue-buffer-count
###Example:
{"asic-id": "20", "name": "egress-uc-queue", "timestamp": 1463014303000.0, "metric": "uc-queue-buffer-count", "bv-agent": "172.16.170.184", "value": 1111, "entity": "broadview-bst", "queue": 6, "port": "0"}
## egress-uc-queue-group
* queue-group - the queue group associated with the statistic
* metric - uc-buffer-count
###Example:
{"asic-id": "20", "name": "egress-uc-queue-group", "queue-group": 6, "timestamp": 1463014303000.0, "metric": "uc-buffer-count", "bv-agent": "172.16.170.184", "value": 2222, "entity": "broadview-bst"}