326483ee4c
assertItemsEqual was removed from Python's unittest.TestCase in Python 3.3 [1][2]. We have been able to use them since then, because testtools required unittest2, which still included it. With testtools removing Python 2.7 support [3][4], we will lose support for assertItemsEqual, so we should switch to use assertCountEqual. [1] - https://bugs.python.org/issue17866 [2] - https://hg.python.org/cpython/rev/d9921cb6e3cd [3] - testing-cabal/testtools#286 [4] - testing-cabal/testtools#277 Change-Id: I67ec6c5ea47b927e2ef44864afa9313e0956a90b |
||
|---|---|---|
| devstack | ||
| doc/source | ||
| docs | ||
| etc | ||
| monasca_transform | ||
| releasenotes/notes | ||
| scripts | ||
| tests | ||
| tools/vagrant | ||
| .gitignore | ||
| .gitreview | ||
| .stestr.conf | ||
| .zuul.yaml | ||
| LICENSE | ||
| README.rst | ||
| lower-constraints.txt | ||
| requirements.txt | ||
| setup.cfg | ||
| setup.py | ||
| test-requirements.txt | ||
| tox.ini | ||
README.rst
Team and repository tags
Monasca Transform
monasca-transform is a data driven aggregation engine which collects, groups and aggregates existing individual Monasca metrics according to business requirements and publishes new transformed (derived) metrics to the Monasca Kafka queue.
- Since the new transformed metrics are published as any other metric in Monasca, alarms can be set and triggered on the transformed metric.
- Monasca Transform uses Apache Spark to aggregate data. Apache Spark is a highly scalable, fast, in-memory, fault tolerant and parallel data processing framework. All monasca-transform components are implemented in Python and use Spark’s PySpark Python API to interact with Spark.
- Monasca Transform does transformation and aggregation of incoming
metrics in two phases.
- In the first phase spark streaming application is set to retrieve in data from kafka at a configurable stream interval (default stream_inteval is 10 minutes) and write the data aggregated for stream interval to pre_hourly_metrics topic in kafka.
- In the second phase, which is kicked off every hour, all metrics in metrics_pre_hourly topic in Kafka are aggregated again, this time over a larger interval of an hour. These hourly aggregated metrics published to metrics topic in kafka.
Use Cases handled by Monasca Transform
Please refer to Problem Description section on the Monasca/Transform wiki
Operation
Please refer to How Monasca Transform Operates section on the Monasca/Transform wiki
Architecture
Please refer to Architecture and Logical processing data flow sections on the Monasca/Transform wiki
To set up the development environment
The monasca-transform uses DevStack as a common dev environment. See the README.md in the devstack directory for details on how to include monasca-transform in a DevStack deployment.
Generic aggregation components
Monasca Transform uses a set of generic aggregation components which can be assembled in to an aggregation pipeline.
Please refer to the generic-aggregation-components document for information on list of generic aggregation components available.
Create a new aggregation pipeline example
Generic aggregation components make it easy to build new aggregation pipelines for different Monasca metrics.
This create a new aggregation pipeline example shows how to create pre_transform_specs and transform_specs to create an aggregation pipeline for a new set of Monasca metrics, while leveraging existing set of generic aggregation components.
Original proposal and blueprint
Original proposal: Monasca/Transform-proposal
Blueprint: monasca-transform blueprint