openstack
/
monasca-analytics
Code Issues Proposed changes

Remove project content 

This is step 2 of the project retirement process as described in [1].
Project retirement has been anounced here [2].

[1] https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#step-2-remove-project-content
[2] http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016830.html

Depends-On: https://review.opendev.org/751987
Change-Id: Id4c4f95fee1787e23a3156933c40c491f298128e
This commit is contained in:
Witek Bedyk 2020-09-15 10:29:46 +02:00
parent e1f3608242
commit aea6e4858d
299 changed files with 10 additions and 22549 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
.gitignore vendored
View File

@ -1,14 +0,0 @@
*~
*.egg-info
*.swp
*.log
*.pyc
*.pyo
.eggs
.project
.pydevproject
.vagrant
.idea
.tox
.coverage
cover

175
LICENSE
View File

@ -1,175 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

30
Makefile
View File

@ -1,30 +0,0 @@
ifeq (testspec,$(firstword $(MAKECMDGOALS)))
# use the rest as arguments for "run"
TEST_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
# ...and turn them into do-nothing targets
$(eval $(TEST_ARGS):;@:)
endif
MONANAS_SRC=monasca_analytics
PYTHON=python
all: test style
test:
$(PYTHON) -m unittest discover -v
testspec:
$(PYTHON) -m unittest -v $(TEST_ARGS)
clean:
find $(MONANAS_SRC) -type f -name '*.pyc' -exec rm {} +
style:
find $(MONANAS_SRC) -type f -name '*.py' -exec pep8 --max-line-length 79 {} +
start:
bash -c "sleep 7; curl -H \"Content-Type: application/json\" -d '{\"action\": \"start_streaming\"}' http://localhost:3000/" &
$(PYTHON) run.py -p ~/spark/spark-1.6.1 -c ./config/metric_experiments.json -l ./config/logging.json
.PHONY: all test clean style testspec

110
README.md
View File

@ -1,110 +0,0 @@
Team and repository tags
========================
[![Team and repository tags](https://governance.openstack.org/tc/badges/monasca-analytics.svg)](https://governance.openstack.org/tc/reference/tags/index.html)
<!-- Change things from this point on -->
# MoNanas - Monasca Analytics Framework
![MoNanas Logo](doc/images/monanas-logo.png)
## Overview
Monasca Analytics (MoNanas) is a statistical/machine-learning
([SML](doc/design.md#sml)) [flow](doc/design.md#flow) composition
engine. Users can compose a sequence of algorithms to be executed by just
providing a description as an input to MoNanas. The data flow is automatically
handled by the framework.
Easy [flow](doc/design.md#flow) composition and reusability means that
we can speed up the extraction of actionable infrastructure insight.
### Advantages
:thumbsup: Decouple algorithm design from execution.
:thumbsup: Reusable specification of the desired [flow](doc/design.md#flow).
:thumbsup: Language independent [flow](doc/design.md#flow) definition.
:thumbsup: Data source and format independent.
:thumbsup: Easy to add new [SML](doc/design.md#sml) algorithms and # combine them with pre-existing ones in the [flow](doc/design.md#flow).
:thumbsup: Transparently exploit data parallelism.
### Documentation
* [MoNanas/GettingStarted](doc/getting_started.md): A starting point for users
and developers of MoNanas.
### Repositories
Core: https://github.com/openstack/monasca-analytics.git
## Example Use Cases
### Integrate with Monasca
See: [MoNanas/IntegrateMonasca](doc/monasca.md) for more details integration with Monasca
### Relevant to OpenStack
See: [MoNanas/UseCases](doc/use_cases.md) for more details use cases that are relevant to OpenStack.
## MoNanas Design
See: [MoNanas/Design](doc/design.md) for details on MoNanas's architecture,
its functional requirements and core concepts.
## Technologies
MoNanas uses a number of third-party technologies:
* Apache Spark (https://spark.apache.org/): Apache Spark is a fast and general
engine for large-scale data processing.
* Apache Kafka (https://kafka.apache.org/): Used by Monasca and MoNanas's Kafka
`source` and `sink`.
* Apache ZooKeeper (https://zookeeper.apache.org/): Used by Kafka.
## Feature Release Schedule
- [x] Basic SML flow.
- [x] New algorithm "add-on" ability.
- [x] Example datasets and SML flows.
- [ ] Support end-to-end learning + data processing flows (currently, the latter part does not get updated due to Spark's immutability.)
- [ ] Refactor codes to be consistent with terms used in the documentation.
- [ ] Add a source, ingestor and transformer for Monasca.
- [ ] Model connections as objects rather than references and have driver specifics in one place.
- [ ] Expanded orchestration abilities/expressiveness.
- [ ] Container-enabled testing/deployment for non-production environments.
- [ ] Add Vitrage Sink.
- [ ] Add a ready-to-use virtual machine image (get rid of the fetch-deps.sh).
## Contributing
There are multiple ways to contribute to the project. All are equally important
to us!
* You can have a look at the
[Monasca launchpad](https://launchpad.net/monasca) for problems that
needs to be solved (bugs/issues), and blueprints.
* You can also help us to add
[new learning algorithms](doc/dev_guide.md#add_new_algorithms).
* Finally, we are very interested in having more data sources to experiment
with. The source can either be from an existing data provider or randomly
generated. The more, the better! :) If you are interested to work on that
aspect, [you are welcome as well](doc/dev_guide.md#add_new_sources).
For more information on setting up your development environment, see
[MoNanas/DevGuide](doc/dev_guide.md).
For more information about Monanas, please visit the wiki page:
[Monanas wiki](https://wiki.openstack.org/wiki/Monasca/Analytics).
And for more information about Monasca, please visit the wiki page:
[Monasca wiki](https://wiki.openstack.org/wiki/Monasca).
## License
Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
Licensed under the Apache License, Version 2.0 (the "License"); you may not
used this file except in compliance with the License. You may obtain a copy of
the License at:
```
http://www.apache.org/licenses/LICENSE-2.0
```
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.

10
README.rst Normal file
View File

@ -0,0 +1,10 @@
This project is no longer maintained.
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
For any further questions, please email
openstack-discuss@lists.openstack.org or join #openstack-monasca on
Freenode.

7
SUMMARY.md
View File

@ -1,7 +0,0 @@
# Summary
* [MoNanas](README.md)
* [Design](doc/design.md)
* [Getting Started](doc/getting_started.md)
* [Developer Guide](doc/dev_guide.md)
* [Examples](doc/examples.md)

43
Vagrantfile vendored
View File

@ -1,43 +0,0 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure(2) do |config|
# The most common configuration options are documented and commented below.
# For a complete reference, please see the online documentation at
# https://docs.vagrantup.com.
# Default box
config.vm.box = "ubuntu/trusty64"
# Setup port for Machine Learning Framework
config.vm.network "forwarded_port", guest: 3000, host: 3000
config.vm.network "forwarded_port", guest: 4040, host: 4040
# Shell provisioning
config.vm.provision "shell",
inline: "echo 'export VAGRANT_ENV=1' >> /home/vagrant/.profile"
config.vm.provision "shell" do |s|
# s.env is only available since Vagrant 1.8
s.path = "fetch-deps.sh"
s.privileged = false
end
# Virtual box settings
config.vm.provider "virtualbox" do |v|
v.memory = 4096
v.cpus = 3
end
# Proxy settings
if Vagrant.has_plugin?("vagrant-proxyconf")
config.proxy.http = ENV['HTTP_PROXY'] || ENV['http_proxy']
config.proxy.https = ENV['HTTPS_PROXY'] || ENV['https_proxy']
config.proxy.no_proxy = "localhost,127.0.0.1"
end
end

0
__init__.py
View File

52
config/config_template.json
View File

@ -1,52 +0,0 @@
{
"id": "my_id",
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": false
},
"sources": {
"src1": {
"module": "my_src_module"
}
},
"ingestors": {
"ing1": {
"module": "my_ingestor_module"
}
},
"smls": {
"sml1": {
"module": "my_sml_module"
}
},
"voters": {
"vot1": {
"module": "my_voter_module"
}
},
"sinks": {
"snk1": {
"module": "my_sink_module"
}
},
"ldps": {
"ldp1": {
"module": "my_ldp_module"
}
},
"connections": {
"src1": ["ing1", "ldp1"],
"sml1": ["vot1"],
"ing1": [],
"vot1": ["ldp1"],
"ldp1": ["snk1"],
"snk1": []
},
"feedback": {}
}

16
config/iptables_anomalies.banana
View File

@ -1,16 +0,0 @@
##############################
# IP Tables anomalies
#
sleep = 0.01
src = IPTablesSource(sleep=sleep)
ing1 = IptablesIngestor()
svm = SvmOneClass()
voter = PickIndexVoter(0)
ldp1 = IptablesLDP()
stdout = StdoutSink()
sqlite = IptablesSQLiteSink()
src -> [ing1, ldp1]
ing1 -> svm -> voter -> ldp1
ldp1 -> [sqlite, stdout]

58
config/iptables_anomalies.json
View File

@ -1,58 +0,0 @@
{
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": false
},
"sources": {
"src1": {
"module": "IPTablesSource",
"sleep": 0.01
}
},
"ingestors": {
"ing1": {
"module": "IptablesIngestor"
}
},
"smls": {
"sml1": {
"module": "SvmOneClass",
"nb_samples": 500
}
},
"voters": {
"vot1": {
"module": "PickIndexVoter",
"index": 0
}
},
"sinks": {
"snk1": {
"module": "IptablesSQLiteSink"
},
"snk2": {
"module": "StdoutSink"
}
},
"ldps": {
"ldp1": {
"module": "IptablesLDP"
}
},
"connections": {
"src1": ["ing1", "ldp1"],
"ing1": [],
"sml1": ["vot1"],
"vot1": ["ldp1"],
"ldp1": ["snk1", "snk2"],
"snk1": [],
"snk2": []
},
"feedback": {}
}

23
config/logging.json
View File

@ -1,23 +0,0 @@
{
"version": 1,
"disable_existing_loggers": false,
"formatters": {
"standard": {
"format": "%(asctime)s [%(levelname)s] %(name)s: %(message)s"
}
},
"handlers": {
"default": {
"level": "INFO",
"class": "logging.StreamHandler",
"formatter": "standard"
}
},
"loggers": {
"": {
"handlers": ["default"],
"level": "INFO",
"propagate": true
}
}
}

57
config/markov_source_config.banana
View File

@ -1,57 +0,0 @@
##############################
# Markov source config example
# (cloud-like data model)
#
src = CloudMarkovChainSource(sleep=0.01, min_event_per_burst=500)
src.transitions.web_service = {
"run=>slow": {
"0": 0.001,
"8": 0.02,
"12": 0.07,
"14": 0.07,
"22": 0.03,
"24": 0.00
},
"slow=>run": {
"0": 0.99,
"8": 0.7,
"12": 0.1,
"14": 0.1,
"22": 0.8,
"24": 0.9
},
"stop=>run": 0.7
}
src.transitions.host = {
"on=>off": 0.005,
"off=>on": 0.5
}
src.transitions.switch = {
"on=>off": 0.01,
"off=>on": 0.7
}
src.triggers.support = {
"get_called" : {
"0": 0.1,
"8": 0.2,
"12": 0.8,
"14": 0.8,
"22": 0.5,
"24": 0.0
}
}
ing1 = CloudIngestor()
ling = LiNGAM(threshold=0.5)
voter = PickIndexVoter(0)
sink = KafkaSink(host="localhost", port=9092, topic="transformed_alerts")
ldp = CloudCausalityLDP()
# Connections
src -> [ing1 -> ling, ldp]
ling -> voter -> ldp -> sink

107
config/markov_source_config.json
View File

@ -1,107 +0,0 @@
{
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": false
},
"sources": {
"src1": {
"module": "CloudMarkovChainSource",
"sleep": 0.01,
"min_event_per_burst": 500,
"transitions": {
"web_service": {
"run=>slow": {
"0": 0.001,
"8": 0.02,
"12": 0.07,
"14": 0.07,
"22": 0.03,
"24": 0.001
},
"slow=>run": {
"0": 0.99,
"8": 0.7,
"12": 0.1,
"14": 0.1,
"22": 0.8,
"24": 0.99
},
"stop=>run": 0.7
},
"host": {
"on=>off": 0.005,
"off=>on": 0.5
},
"switch": {
"on=>off": 0.01,
"off=>on": 0.7
}
},
"triggers": {
"support": {
"get_called": {
"0": 0.1,
"8": 0.2,
"12": 0.8,
"14": 0.8,
"22": 0.5,
"24": 0.0
}
}
},
"graph": {
"h1:host": ["s1"],
"h2:host": ["s1"],
"s1:switch": [],
"w1:web_service": ["h1"],
"w2:web_service": ["h2"]
}
}
},
"ingestors": {
"ing1": {
"module": "CloudIngestor"
}
},
"smls": {
"sml1": {
"module": "LiNGAM",
"threshold": 0.5
}
},
"voters": {
"vot1": {
"module": "PickIndexVoter",
"index": 0
}
},
"sinks": {
"snk1": {
"module": "KafkaSink",
"host": "localhost",
"port": 9092,
"topic": "transformed_alerts"
}
},
"ldps": {
"ldp1": {
"module": "CloudCausalityLDP"
}
},
"connections": {
"src1": ["ing1", "ldp1"],
"sml1": ["vot1"],
"ing1": [],
"vot1": ["ldp1"],
"ldp1": ["snk1"],
"snk1": []
},
"feedback": {}
}

30
config/metric_experiments.banana
View File

@ -1,30 +0,0 @@
#######################
# Metric experiments
#
# Sources
src = MonascaMarkovChainSource(sleep=0.01)
# Sinks
stdout = StdoutSink()
file = FileSink(path="~/monasca-aggregate.log")
# Live data processors
period = 0.1 * 2
aggregator = MonascaAggregateLDP(func="cnt", period=period)
combiner = MonascaCombineLDP(
metric= "cpu.logical_cores_actives",
bindings= {
a: "cpu.idle_perc",
b: "cpu.total_logical_cores",
},
lambda= "a * b",
period= period
)
derivator = MonascaDerivativeLDP(period=period)
# Connections
src -> aggregator -> stdout
src -> [combiner, derivator] -> stdout
[combiner, derivator] -> file

60
config/metric_experiments.json
View File

@ -1,60 +0,0 @@
{
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": false
},
"sources": {
"src1": {
"module": "MonascaMarkovChainSource",
"sleep": 0.01
}
},
"ingestors": {},
"smls": {},
"voters": {},
"sinks": {
"snk2": {
"module": "StdoutSink"
},
"snk3": {
"module": "FileSink",
"path": "~/monasca-aggregate.log"
}
},
"ldps": {
"ldp3": {
"module": "MonascaAggregateLDP",
"period": 2,
"func": "cnt"
},
"ldp4": {
"module": "MonascaCombineLDP",
"metric": "cpu.logical_cores_actives",
"period": 1,
"lambda": "a * b",
"bindings": {
"a": "cpu.idle_perc",
"b": "cpu.total_logical_cores"
}
},
"ldp5": {
"module": "MonascaDerivativeLDP",
"period": 1
}
},
"connections": {
"src1": ["ldp5"],
"ldp3": [],
"ldp4": [],
"ldp5": ["snk2"],
"snk2": [],
"snk3": []
},
"feedback": {}
}

13
config/monasca_aggregate_all_fn.banana
View File

@ -1,13 +0,0 @@
##############################
# Monasca aggregate all functions
src = MonascaMarkovChainSource(sleep=0.01)
snk = KafkaSink(host="localhost", port=9092, topic="experiments")
stdout = StdoutSink()
ldp1 = MonascaAggregateLDP(func="cnt")
ldp2 = MonascaAggregateLDP(func="max")
ldp3 = MonascaAggregateLDP(func="min")
ldp4 = MonascaAggregateLDP(func="avg")
src -> [ldp1, ldp2, ldp3, ldp4] -> [snk, stdout]

64
config/monasca_aggregate_all_fn.json
View File

@ -1,64 +0,0 @@
{
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": false
},
"sources": {
"src1": {
"module": "MonascaMarkovChainSource",
"sleep": 0.01
}
},
"ingestors": {},
"smls": {},
"voters": {},
"sinks": {
"snk1": {
"module": "KafkaSink",
"host": "localhost",
"port": 9092,
"topic": "monasca_experiments"
},
"snk2": {
"module": "StdoutSink"
}
},
"ldps": {
"ldp1": {
"module": "MonascaAggregateLDP",
"period": 2,
"func": "max"
},
"ldp2": {
"module": "MonascaAggregateLDP",
"period": 2,
"func": "min"
},
"ldp3": {
"module": "MonascaAggregateLDP",
"period": 2,
"func": "avg"
},
"ldp4": {
"module": "MonascaAggregateLDP",
"period": 2,
"func": "sum"
}
},
"connections": {
"src1": ["ldp1", "ldp2", "ldp3", "ldp4"],
"ldp1": ["snk2"],
"ldp2": ["snk2"],
"ldp3": ["snk2"],
"ldp4": ["snk2"],
"snk1": [],
"snk2": []
},
"feedback": {}
}

54
config_dsl.py
View File

@ -1,54 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import json
import os
import six
from logging import config as log_conf
from monasca_analytics.banana.cli import interpreter
DEFAULT_LOGGING_CONFIG_FILE = "config/logging.json"
def setup_logging():
current_dir = os.path.dirname(__file__)
logging_config_file = os.path.join(current_dir,
DEFAULT_LOGGING_CONFIG_FILE)
with open(logging_config_file, "rt") as f:
config = json.load(f)
log_conf.dictConfig(config)
def main():
setup_logging()
print(">>>>> DEPRECATED TOOL <<<<<")
print(">>>>> Use the banana language instead <<<<<")
print("")
print("Welcome to Monanas config command line")
print("Type help for help about commands")
inter = interpreter.DSLInterpreter()
cmd = ""
while "exit" != cmd.lower():
cmd = six.moves.input(">> ")
if cmd != "":
try:
print(inter.execute_string(cmd))
except Exception as e:
print("Failed : " + str(e))
if __name__ == "__main__":
main()

71
devstack/README.md
View File

@ -1,71 +0,0 @@
# Monasca Analytics DevStack Plugin
The Monasca Analytics DevStack plugin currently only works on Ubuntu 16.04 (Xenial).
More Linux Distributions will be supported in the future.
Running the Monasca Analytics DevStack plugin requires a machine with 8GB of RAM.
Directions for installing and running Devstack can be found here:
http://docs.openstack.org/developer/devstack/
To run Monasca Analytics in DevStack, do the following three steps.
1. Clone the DevStack repo.
```
git clone https://opendev.org/openstack/devstack
```
2. Add the following to the DevStack local.conf file in the root of the devstack directory. You may
need to create the local.conf if it does not already exist.
# BEGIN DEVSTACK LOCAL.CONF CONTENTS
```
[[local|localrc]]
MYSQL_PASSWORD=secretmysql
DATABASE_PASSWORD=secretdatabase
RABBIT_PASSWORD=secretrabbit
ADMIN_PASSWORD=secretadmin
SERVICE_PASSWORD=secretservice
SERVICE_TOKEN=111222333444
LOGFILE=$DEST/logs/stack.sh.log
LOGDIR=$DEST/logs
LOG_COLOR=False
# This line will enable all of Monasca Analytics.
enable_plugin monasca-analytics https://opendev.org/openstack/monasca-analytics
```
# END DEVSTACK LOCAL.CONF CONTENTS
3. Run './stack.sh' from the root of the devstack directory.
To run Monasca Analytics with the bare mininum of OpenStack components you can
add the following two lines to the local.conf file.
```
disable_all_services
enable_service rabbit mysql key
```
# License
```
# (C) Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

40
devstack/files/kafka/kafka-server-start.sh
View File

@ -1,40 +0,0 @@
#!/bin/bash
#
# Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
if [ $# -lt 1 ];
then
echo "USAGE: $0 [-daemon] server.properties"
exit 1
fi
base_dir=$(dirname $0)
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
export KAFKA_HEAP_OPTS="-Xms256m -Xmx256m"
EXTRA_ARGS="-name kafkaServer -loggc"
COMMAND=$1
case $COMMAND in
-daemon)
EXTRA_ARGS="-daemon "$EXTRA_ARGS
shift
;;
*)
;;
esac
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka $@

35
devstack/files/kafka/kafka.conf
View File

@ -1,35 +0,0 @@
#
# Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
description "Kafka"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
limit nofile 32768 32768
# If zookeeper is running on this box also give it time to start up properly
pre-start script
if [ -e /etc/init.d/zookeeper ]; then
/etc/init.d/zookeeper start || true
fi
end script
# Rather than using setuid/setgid sudo is used because the pre-start task must run as root
exec sudo -Hu kafka -g kafka KAFKA_HEAP_OPTS="-Xmx128m" /opt/kafka/bin/kafka-server-start.sh /etc/kafka/server.properties

12
devstack/files/kafka/kafka.service
View File

@ -1,12 +0,0 @@
[Unit]
Description=Apache Kafka
Requires=network.target
After=zookeepepr.service
[Service]
Type=simple
ExecStart=/opt/kafka/bin/kafka-server-start.sh /etc/kafka/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target

75
devstack/files/kafka/log4j.properties
View File

@ -1,75 +0,0 @@
#
# Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
log4j.rootLogger=WARN, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
log4j.appender.kafkaAppender.MaxFileSize=50MB
log4j.appender.kafkaAppender.MaxBackupIndex=4
log4j.appender.kafkaAppender.File=/var/log/kafka/server.log
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.stateChangeAppender=org.apache.log4j.RollingFileAppender
log4j.appender.stateChangeAppender.MaxFileSize=50MB
log4j.appender.stateChangeAppender.MaxBackupIndex=4
log4j.appender.stateChangeAppender.File=/var/log/kafka/state-change.log
log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.controllerAppender=org.apache.log4j.RollingFileAppender
log4j.appender.controllerAppender.MaxFileSize=50MB
log4j.appender.controllerAppender.MaxBackupIndex=4
log4j.appender.controllerAppender.File=/var/log/kafka/controller.log
log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
# Turn on all our debugging info
#log4j.logger.kafka.producer.async.DefaultEventHandler=DEBUG, kafkaAppender
#log4j.logger.kafka.client.ClientUtils=DEBUG, kafkaAppender
#log4j.logger.kafka.perf=DEBUG, kafkaAppender
#log4j.logger.kafka.perf.ProducerPerformance$ProducerThread=DEBUG, kafkaAppender
#log4j.logger.org.I0Itec.zkclient.ZkClient=DEBUG
log4j.logger.kafka=WARN, kafkaAppender
# Tracing requests results in large logs
#log4j.appender.requestAppender=org.apache.log4j.RollingFileAppender
#log4j.appender.requestAppender.MaxFileSize=50MB
#log4j.appender.requestAppender.MaxBackupIndex=4
#log4j.appender.requestAppender.File=/var/log/kafka/kafka-request.log
#log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
#log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
#
#log4j.logger.kafka.network.RequestChannel$=TRACE, requestAppender
#log4j.additivity.kafka.network.RequestChannel$=false
#
#log4j.logger.kafka.network.Processor=TRACE, requestAppender
#log4j.logger.kafka.server.KafkaApis=TRACE, requestAppender
#log4j.additivity.kafka.server.KafkaApis=false
#log4j.logger.kafka.request.logger=TRACE, requestAppender
#log4j.additivity.kafka.request.logger=false
log4j.logger.kafka.controller=TRACE, controllerAppender
log4j.additivity.kafka.controller=false
log4j.logger.state.change.logger=TRACE, stateChangeAppender
log4j.additivity.state.change.logger=false

119
devstack/files/kafka/server.properties
View File

@ -1,119 +0,0 @@
#
# Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
############################# Server Basics #############################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################# Socket Server Settings #############################
# The port the socket server listens on
port=9092
# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=127.0.0.1
# Hostname the broker will advertise to producers and consumers. If not set, it uses the
# value for "host.name" if configured. Otherwise, it will use the value returned from
# java.net.InetAddress.getCanonicalHostName().
#advertised.host.name=<hostname routable by clients>
# The port to publish to ZooKeeper for clients to use. If this is not set,
# it will publish the same port that the broker binds to.
#advertised.port=<port accessible by clients>
# The number of threads handling network requests
num.network.threads=2
# The number of threads doing disk I/O
num.io.threads=2
# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=1048576
# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=1048576
# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/var/kafka
auto.create.topics.enable=false
# The number of logical partitions per topic per server. More partitions allow greater parallelism
# for consumption, but also mean more files.
num.partitions=2
############################# Log Flush Policy #############################
# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.
# The number of messages to accept before forcing a flush of data to disk
log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a log before we force a flush
log.flush.interval.ms=1000
############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.
# The minimum age of a log file to be eligible for deletion
log.retention.hours=24
# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.bytes.
log.retention.bytes=104857600
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=104857600
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=60000
# By default the log cleaner is disabled and the log retention policy will default to just delete segments after their retention expires.
# If log.cleaner.enable=true is set the cleaner will be enabled and individual logs can then be marked for log compaction.
log.cleaner.enable=false
############################# Zookeeper #############################
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=127.0.0.1:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=1000000

14
devstack/files/maven/settings.xml
View File

@ -1,14 +0,0 @@
<settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
<proxies>
<proxy>
<active>true</active>
<protocol>http</protocol>
<username></username>
<password></password>
<host></host>
<port>8080</port>
<nonProxyHosts>localhost</nonProxyHosts>
</proxy>
</proxies>
</settings>

36
devstack/files/zookeeper/environment
View File

@ -1,36 +0,0 @@
#
# Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Modified from http://packages.ubuntu.com/saucy/zookeeperd
NAME=zookeeper
ZOOCFGDIR=/etc/zookeeper/conf
# seems, that log4j requires the log4j.properties file to be in the classpath
CLASSPATH="$ZOOCFGDIR:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar"
ZOOCFG="$ZOOCFGDIR/zoo.cfg"
ZOO_LOG_DIR=/var/log/zookeeper
USER=$NAME
GROUP=$NAME
PIDDIR=/var/run/$NAME
PIDFILE=$PIDDIR/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
JAVA=/usr/bin/java
ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"
ZOO_LOG4J_PROP="INFO,ROLLINGFILE"
JMXLOCALONLY=false
JAVA_OPTS=""

69
devstack/files/zookeeper/log4j.properties
View File

@ -1,69 +0,0 @@
#
# Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# From http://packages.ubuntu.com/saucy/zookeeperd
# ZooKeeper Logging Configuration
#
# Format is "<default threshold> (, <appender>)+
log4j.rootLogger=${zookeeper.root.logger}
# Example: console appender only
# log4j.rootLogger=INFO, CONSOLE
# Example with rolling log file
#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE
# Example with rolling log file and tracing
#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE
#
# Log INFO level and above messages to the console
#
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=INFO
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
#
# Add ROLLINGFILE to rootLogger to get log file output
# Log DEBUG level and above messages to a log file
log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=WARN
log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/zookeeper.log
# Max log file size of 10MB
log4j.appender.ROLLINGFILE.MaxFileSize=10MB
# uncomment the next line to limit number of backup files
#log4j.appender.ROLLINGFILE.MaxBackupIndex=10
log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
#
# Add TRACEFILE to rootLogger to get log file output
# Log DEBUG level and above messages to a log file
log4j.appender.TRACEFILE=org.apache.log4j.FileAppender
log4j.appender.TRACEFILE.Threshold=TRACE
log4j.appender.TRACEFILE.File=${zookeeper.log.dir}/zookeeper_trace.log
log4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout
### Notice we are including log4j's NDC here (%x)
log4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n

1
devstack/files/zookeeper/myid
View File

@ -1 +0,0 @@
0

74
devstack/files/zookeeper/zoo.cfg
View File

@ -1,74 +0,0 @@
#
# Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/var/lib/zookeeper
# Place the dataLogDir to a separate physical disc for better performance
# dataLogDir=/disk2/zookeeper
# the port at which the clients will connect
clientPort=2181
# Maximum number of clients that can connect from one client
maxClientCnxns=60
# specify all zookeeper servers
# The fist port is used by followers to connect to the leader
# The second one is used for leader election
server.0=127.0.0.1:2888:3888
# To avoid seeks ZooKeeper allocates space in the transaction log file in
# blocks of preAllocSize kilobytes. The default block size is 64M. One reason
# for changing the size of the blocks is to reduce the block size if snapshots
# are taken more often. (Also, see snapCount).
#preAllocSize=65536
# Clients can submit requests faster than ZooKeeper can process them,
# especially if there are a lot of clients. To prevent ZooKeeper from running
# out of memory due to queued requests, ZooKeeper will throttle clients so that
# there is no more than globalOutstandingLimit outstanding requests in the
# system. The default limit is 1,000.ZooKeeper logs transactions to a
# transaction log. After snapCount transactions are written to a log file a
# snapshot is started and a new transaction log file is started. The default
# snapCount is 10,000.
#snapCount=1000
# If this option is defined, requests will be will logged to a trace file named
# traceFile.year.month.day.
#traceFile=
# Leader accepts client connections. Default value is "yes". The leader machine
# coordinates updates. For higher update throughput at thes slight expense of
# read throughput the leader can be configured to not accept clients and focus
# on coordination.
#leaderServes=yes
# Autopurge every hour to avoid using lots of disk in bursts
# Order of the next 2 properties matters.
# autopurge.snapRetainCount must be before autopurge.purgeInterval.
autopurge.snapRetainCount=3
autopurge.purgeInterval=1

362
devstack/plugin.sh
View File

@ -1,362 +0,0 @@
#
# Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Monasca-analytics DevStack plugin
#
# Install and start Monasca-analytics service in devstack
#
# To enable Monasca-analytics in devstack add an entry to local.conf that
# looks like
#
# [[local|localrc]]
# enable_plugin monasca-analytics https://git.openstack.org/openstack/monasca-analytics
#
# By default all Monasca services are started (see
# devstack/settings). To disable a specific service use the
# disable_service function. For example to turn off notification:
#
# disable_service monasca-notification
#
# Several variables set in the localrc section adjust common behaviors
# of Monasca (see within for additional settings):
#
# EXAMPLE VARS HERE
# Save trace setting
XTRACE=$(set +o | grep xtrace)
set -o xtrace
ERREXIT=$(set +o | grep errexit)
set -o errexit
# Determine if we are running in devstack-gate or devstack.
if [[ $DEST ]]; then
# We are running in devstack-gate.
export MONASCA_BASE=${MONASCA_BASE:-"${DEST}"}
else
# We are running in devstack.
export MONASCA_BASE=${MONASCA_BASE:-"/opt/stack"}
fi
export MONASCA_ANALYTICS_BASE=${MONASCA_ANALYTICS_BASE:-"${MONASCA_BASE}/monasca-analytics"}
###
function pre_install_spark {
:
}
###
function pre_install_monasca_analytics {
:
}
###
function unstack_monasca_analytics {
echo_summary "Unstack Monasca-analytics"
delete_monasca_analytics_files
sudo userdel monasca-analytics || true
sudo groupdel monasca-analytics || true
unstack_spark
}
###
function delete_monasca_analytics_files {
sudo rm -rf /opt/monasca/analytics || true
sudo rm -rf /etc/monasca/analytics || true
sudo rm /etc/init/monasca-analytics.conf || true
MONASCA_ANALYTICS_DIRECTORIES=("/var/log/monasca/analytics" "/var/run/monasca/analytics" "/etc/monasca/analytics/init")
for MONASCA_ANALYTICS_DIRECTORY in "${MONASCA_ANALYTICS_DIRECTORIES[@]}"
do
sudo rm -rf ${MONASCA_ANALYTICS_DIRECTORY} || true
done
}
###
function unstack_spark {
echo_summary "Unstack Spark"
delete_spark_directories
sudo rm -rf /opt/spark/download || true
}
###
function clean_monasca_analytics {
set +o errexit
unstack_monasca_analytics
unistall_pkgs
set -o errexit
}
###
function delete_spark_directories {
for SPARK_DIRECTORY in "${SPARK_DIRECTORIES[@]}"
do
sudo rm -rf ${SPARK_DIRECTORY} || true
done
sudo rm -rf /var/log/spark-events || true
}
###
function unistall_pkgs {
sudo apt-get -y purge ipython python-scipy python-numpy
sudo apt-get -y purge python-setuptools
sudo apt-get -y purge sbt
sudo apt-key del $KEYID
sudo sed -i -e '/deb https\:\/\/dl.bintray.com\/sbt\/debian \//d' /etc/apt/sources.list.d/sbt.list
sudo dpkg -r scala
sudo apt-get -y purge $JDK_PKG
sudo rm -rf ~/.m2
sudo rm -rf $SPARK_DIR
}
###
function install_pkg {
## JDK
sudo -E apt-get -y install $JDK_PKG
## SCALA
download_through_cache $SCALA_URL $SCALA $SPARK_DOWNLOAD
sudo -E dpkg -i $SPARK_DOWNLOAD/$SCALA
echo "deb https://dl.bintray.com/sbt/debian /" | sudo -E tee -a /etc/apt/sources.list.d/sbt.list
sudo -E apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv $KEYID
sudo -E apt-get update
sudo -E apt-get -y install sbt
## other pkg
sudo -E apt-get -y install python-setuptools
sudo -E apt-get -y install python-numpy python-scipy ipython
}
###
function build_spark {
## install maven
download_through_cache $MAVEN_URL $MAVEN_TARBALL $SPARK_DOWNLOAD
sudo chown stack:stack $SPARK_DOWNLOAD/$MAVEN_TARBALL
sudo -u stack -g stack tar -xzf $SPARK_DOWNLOAD/$MAVEN_TARBALL -C $SPARK_DIR
if [ ${http_proxy} ];then
read HTTP_PROXY_USER_NAME HTTP_PROXY_PASSWORD HTTP_PROXY_HOST<< END
`echo ${http_proxy:7} | awk -F: '{sub("@", ":");print $1, $2, $3}'`
END
if [ -z $HTTP_PROXY_HOST ];then
LENGTH_FOR_HOST=`expr match "$http_proxy" 'http://[\.A-Za-z\-]*'`-7
sed -e '7,8d' \
-e "s/<host><\/host>/<host>${http_proxy:7:$LENGTH_FOR_HOST}<\/host>/g" \
${MONASCA_ANALYTICS_BASE}/devstack/files/maven/settings.xml > ~/.m2/settings.xml
else
sed -e "s/<username><\/username>/<username>${HTTP_PROXY_USER_NAME}<\/username>/g" \
-e "s/<password><\/password>/<password>${HTTP_PROXY_PASSWORD}<\/password>/g" \
-e "s/<host><\/host>/<host>${HTTP_PROXY_HOST}<\/host>/g" \
${MONASCA_ANALYTICS_BASE}/devstack/files/maven/settings.xml > ~/.m2/settings.xml
fi
fi
## Build Spark
download_through_cache $SPARK_URL ${SPARK_TARBALL_NAME} $SPARK_DOWNLOAD
sudo chown stack:stack $SPARK_DOWNLOAD/${SPARK_TARBALL_NAME}
sudo -u stack -g stack tar -xzf $SPARK_DOWNLOAD/${SPARK_TARBALL_NAME} -C $SPARK_DIR
DEVSTACK_DIR=`pwd`
cd $SPARK_DIR/spark-${SPARK_VERSION}
$SPARK_DIR/$MAVEN/bin/mvn -DskipTests clean package
sudo cp -pf ./conf/log4j.properties.template ./conf/log4j.properties
sudo sed -i 's/log4j.rootCategory=INFO/log4j.rootCategory=ERROR/g' ./conf/log4j.properties
cd $DEVSTACK_DIR
}
###
function install_zookeeper {
if [ ! -e /etc/init.d/zookeeper ]; then
echo_summary "Install Monasca Zookeeper"
sudo apt-get -y install zookeeperd
sudo cp "${MONASCA_ANALYTICS_BASE}"/devstack/files/zookeeper/myid /etc/zookeeper/conf/myid
sudo cp "${MONASCA_ANALYTICS_BASE}"/devstack/files/zookeeper/environment /etc/zookeeper/conf/environment
sudo cp "${MONASCA_ANALYTICS_BASE}"/devstack/files/zookeeper/log4j.properties /etc/zookeeper/conf/log4j.properties
sudo cp "${MONASCA_ANALYTICS_BASE}"/devstack/files/zookeeper/zoo.cfg /etc/zookeeper/conf/zoo.cfg
if [[ ${SERVICE_HOST} ]]; then
sudo sed -i "s/server\.0=127\.0\.0\.1/server.0=${SERVICE_HOST}/g" /etc/zookeeper/conf/zoo.cfg
fi
sudo mkdir -p /var/log/zookeeper || true
sudo chmod 755 /var/log/zookeeper
sudo systemctl start zookeeper || sudo systemctl restart zookeeper
else
echo_summary "SKIP:Install Monasca Zookeeper"
fi
}
###
function install_kafka {
if [ ! -e /etc/init/kafka.conf ];then
echo_summary "Install Monasca Kafka"
sudo groupadd --system kafka || true
sudo useradd --system -g kafka kafka || true
download_through_cache $KAFKA_URL $KAFKA_TARBALL $SPARK_DOWNLOAD
sudo tar -xzf $SPARK_DOWNLOAD/$KAFKA_TARBALL -C /opt
sudo ln -sf /opt/kafka_${KAFKA_VERSION} /opt/kafka
sudo cp -f "${MONASCA_ANALYTICS_BASE}"/devstack/files/kafka/kafka-server-start.sh /opt/kafka_${KAFKA_VERSION}/bin/kafka-server-start.sh
sudo cp -f "${MONASCA_ANALYTICS_BASE}"/devstack/files/kafka/kafka.conf /etc/init/kafka.conf
sudo chown root:root /etc/init/kafka.conf
sudo chmod 644 /etc/init/kafka.conf
sudo mkdir -p /var/kafka || true
sudo chown kafka:kafka /var/kafka
sudo chmod 755 /var/kafka
sudo rm -rf /var/kafka/lost+found
sudo mkdir -p /var/log/kafka || true
sudo chown kafka:kafka /var/log/kafka
sudo chmod 755 /var/log/kafka
sudo ln -sf /opt/kafka/config /etc/kafka
sudo cp -f "${MONASCA_ANALYTICS_BASE}"/devstack/files/kafka/log4j.properties /etc/kafka/log4j.properties
sudo chown kafka:kafka /etc/kafka/log4j.properties
sudo chmod 644 /etc/kafka/log4j.properties
sudo cp -f "${MONASCA_ANALYTICS_BASE}"/devstack/files/kafka/server.properties /etc/kafka/server.properties
sudo chown kafka:kafka /etc/kafka/server.properties
sudo chmod 644 /etc/kafka/server.properties
if [[ ${SERVICE_HOST} ]]; then
sudo sed -i "s/host\.name=127\.0\.0\.1/host.name=${SERVICE_HOST}/g" /etc/kafka/server.properties
sudo sed -i "s/zookeeper\.connect=127\.0\.0\.1:2181/zookeeper.connect=${SERVICE_HOST}:2181/g" /etc/kafka/server.properties
fi
sudo cp -f "${MONASCA_ANALYTICS_BASE}"/devstack/files/kafka/kafka.service /etc/systemd/system/kafka.service
sudo chmod 644 /etc/systemd/system/kafka.service
sudo systemctl enable kafka
sudo systemctl start kafka || sudo systemctl restart kafka
else
echo_summary "SKIP:Install Monasca Kafka"
fi
}
###
function install_spark {
echo_summary "Install Spark"
sudo mkdir -p $SPARK_DOWNLOAD
sudo chown -R stack:stack $SPARK_DIR
sudo chmod -R 755 $SPARK_DIR
mkdir -p ~/.m2
sudo -E apt-get update
install_pkg
build_spark
install_zookeeper
install_kafka
}
###
function post_config_monasca_analytics {
:
}
###
function extra_monasca_analytics {
:
}
function download_through_cache {
local resource_location=$1
local resource_name=$2
local download_dir=$3
if [[ ! -d ${download_dir} ]]; then
_safe_permission_operation mkdir -p ${download_dir}
_safe_permission_operation chown stack ${download_dir}
fi
pushd ${download_dir}
if [[ ! -f ${resource_name} ]]; then
sudo -E curl -m ${DOWNLOAD_FILE_TIMEOUT} --retry 3 --retry-delay 5 ${resource_location} -o ${resource_name}
fi
popd
}
# check for service enabled
echo_summary "Monasca-analytics plugin with service enabled = `is_service_enabled monasca-analytics`"
if is_service_enabled monasca-analytics; then
if [[ "$1" == "stack" && "$2" == "pre-install" ]]; then
# Set up system services
echo_summary "Configuring Spark system services"
pre_install_spark
echo_summary "Configuring Monasca-analytics system services"
pre_install_monasca_analytics
elif [[ "$1" == "stack" && "$2" == "install" ]]; then
# Perform installation of service source
echo_summary "Installing Spark"
install_spark
elif [[ "$1" == "stack" && "$2" == "post-config" ]]; then
# Configure after the other layer 1 and 2 services have been configured
echo_summary "Configuring Monasca-analytics"
post_config_monasca_analytics
elif [[ "$1" == "stack" && "$2" == "extra" ]]; then
# Initialize and start the Monasca service
echo_summary "Initializing Monasca-analytics"
extra_monasca_analytics
fi
if [[ "$1" == "unstack" ]]; then
echo_summary "Unstacking Monasca-analytics"
unstack_monasca_analytics
fi
if [[ "$1" == "clean" ]]; then
# Remove state and transient data
# Remember clean.sh first calls unstack.sh
echo_summary "Cleaning Monasca-analytics"
clean_monasca_analytics
fi
else
echo_summary "Monasca-analytics not enabled"
fi
#Restore errexit
$ERREXIT
# Restore xtrace
$XTRACE

59
devstack/settings
View File

@ -1,59 +0,0 @@
#
# Copyright 2016 FUJITSU LIMITED
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
enable_service monasca-analytics
#
# Monasca infrastructure services
#
# databases
# MySQL is already enabled in devstack
#
# Dependent Software Versions
#
DOWNLOAD_FILE_TIMEOUT=${DOWNLOAD_FILE_TIMEOUT:-1800}
# spark vars
SPARK_DIRECTORIES=("/var/spark" "/var/log/spark" "/var/run/spark/work" "/etc/spark/conf" "/etc/spark/init" )
JDK_PKG="openjdk-8-jre-headless openjdk-8-jdk"
MAVEN="apache-maven-3.5.4"
MAVEN_TARBALL="$MAVEN-bin.tar.gz"
MAVEN_URL="https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/$MAVEN_TARBALL"
SCALA_VERSION=${SCALA_VERSION:-2.12}
SCALA_MIN_VERSION=${SCALA_MIN_VERSION:-.8}
SCALA="scala-${SCALA_VERSION}${SCALA_MIN_VERSION}.deb"
SCALA_URL="https://downloads.lightbend.com/scala/${SCALA_VERSION}${SCALA_MIN_VERSION}/$SCALA"
KEYID=642AC823
SPARK_DIR="/opt/spark"
SPARK_DOWNLOAD="$SPARK_DIR/download"
SPARK_VERSION=${SPARK_VERSION:-2.4.4}
SPARK_TARBALL_NAME="spark-${SPARK_VERSION}.tgz"
SPARK_URL="http://archive.apache.org/dist/spark/spark-$SPARK_VERSION/$SPARK_TARBALL_NAME"
BASE_KAFKA_VERSION=${BASE_KAFKA_VERSION:-2.1.1}
KAFKA_DIR="/opt/kafka"
KAFKA_DOWNLOAD="$KAFKA_DIR/download"
KAFKA_VERSION=${KAFKA_VERSION:-${SCALA_VERSION}-${BASE_KAFKA_VERSION}}
KAFKA_TARBALL="kafka_$KAFKA_VERSION.tgz"
KAFKA_URL="https://archive.apache.org/dist/kafka/$BASE_KAFKA_VERSION/$KAFKA_TARBALL"

141
doc/banana.md
View File

@ -1,141 +0,0 @@
# Banana: a configuration language for Monanas
Welcome to Banana, a configuration language for Monanas. The language is the
key of "recipes" allowing users to reuse it to tailor solutions for their
use-cases. In other words, Banana allows us to write a recipe(s) that will be
ingested by Monanas.
The language is fully typed. It uses type inference to avoid having to add any
typing annotation. It is still in its early stages, so more features will be
added to solve common problems.
> Note: a valid `banana` recipe (or file) might still contain errors that
> can only be discovered at run-time. The type system, will remove
> most of them though.
To get you started, we provide an example below, which would allow us to
understand most parts of the language.
> TODO: Once we have a specified interface, we should use it instead as it will
> provide syntax highlighting, in-editor error indication as well as other
> features such as autocompletion.
## Part 1: Creating components
Here is how we create a component:
```python
# Create an IPTablesSource with sleep parameter set to 0.01
src = IPTablesSource(sleep=0.01)
```
We could also create a component without any parameter. In that case, each
parameter is initialized with a default value.
In order to get something interesting we first create the following components:
```python
src = IPTablesSource(sleep=0.01)
ing1 = IptablesIngestor()
svm = SvmOneClass()
voter = PickIndexVoter(0)
ldp1 = IptablesLDP()
stdout = StdoutSink()
sqlite = IptablesSQLiteSink()
```
## Part 2: Connecting components
Connections can be placed anywhere in the file. They will always be processed
after everything else.
We have created five components so far, but note that, some components can only
connected to certain types of components). For instance, a source can only
be connected to an ingestor or a live data processor. However, you can't
connect it to a statistical or machine learning algorithms as those need to get
curated data only. Try to add the following line:
```py
src -> alg
```
You should see an error, saying that this is not possible:
```
Error: Can't connect src to alg, src can only be connected to Ingestor.
```
What we want is to have those connections:
```
+---------+ +---------+ +---------+ +---------+ +---------+ +------------+
| src +-----> | ing1 +------> | alg +-----> | vot +---> | ldp +-----> | stdout |
+----+----+ +---------+ +---------+ +---------+ +----+----+ +------------+
| ^
| |
+----------------------------------------------------------------------+
```
Here is how we can achieve that:
```
src -> [ing1 -> alg -> vot -> ldp, ldp]
ldp -> stdout
```
We could also write it like this:
```
src -> ing1 -> alg -> vot -> ldp
src -> ldp -> stdout
```
Or like this:
```
src -> ing1
src -> ldp
ing1 -> alg
alg -> vot
vot -> ldp
ldp -> stdout
```
The main difference is readability and this is subjective. Use the version that
you think is more readable.
Banana will treat all of them as being semantically identical.
## Part 3: Sharing settings between components
From what we described above, it is possible that we could end up with many
similar parameters across components. It would be great if we could share them.
In Banana we can declare a variable not only for components, but also for
`string`, `number` and json-like `object`.
For instance, this is valid in banana:
```python
sleep = 0.01
# ...
```
You can also make use of arithmetic operators to perform any computation you
might require with previously declared variables or anything else.
Some global variables, defined by the execution environment are also available.
For instance, you could define `sleep` like this:
```python
sleep = 0.01 / $spark.BATCH_INTERVAL
# ...
```
> TODO: the above feature has not yet been implemented.
Finally, Banana supports string interpolation to mix many types together:
```python
port = 9645
host = "localhost:" + port
```

187
doc/cli.md
View File

@ -1,187 +0,0 @@
# Command line interface to generate a JSON configuration
A simple command line tool has been implemented in order to manage
configurations in an easier way. It is not expected to be maintained in the
long term. It is introduced to experiment with the creation of `banana`, the
configuration language of Monanas.
This section explains what operations are currently available and how to use
them.
> NOTE: Please refer to [Monasca/Configuration](configuration.md) for the
> structure of the JSON configuration.
> NOTE: This tool is DEPRECATED, use [BANANA](banana.md) configuration language
> instead.
## Usage
* Start the Monanas cli
```bash
python $MONANAS_HOME/config_dsl.py
```
After running this command, a simple empty configuration will be created. You
can then modify it by running the commands listed below:
## Available commands
You can run the following operations from the CLI in order to create, remove, connect,
disconnect and modify components from the configuration.
### Create component
Create a component and assign it to a variable:
```python
>> A = IPTablesSource
```
This command adds a new source of type `IPTablesSource` to the configuration,
assigning it with a default configuration. It links the source component to
variable A and returns a unique ID. You can either use the variable or the ID
to refer to the instance of the `IPTablesSource` created.
### Remove component
Remove a component using an ID or a variable name:
```python
>> rm A
```
This command removes the component referenced by `A` from the configuration.
The parameter can either be a variable or an ID associated to a component
in the configuration. The component will only be removed if it is not connected
to any other component.
### Connect components
Connect two components in the configuration:
```python
>> A -> B
```
This command connects the component referenced by `A` with the component
referenced by `B`. Both `A` and `B` can be variables or IDs, and the connection
is directional from `A` to `B`. The connection is valid and considered only if
the associated components exist and allowed for connection. For example,
connecting a source with an ingestor is allowed, but connecting a source with
a voter is not.
### Disconnect components
Disconnect two components in the configuration:
```python
>> A !-> B
```
This command disconnects the component `A` from component `B`. Both `A` and `B`
can be variables or IDs and the connection is directional from `A` to `B`. If
the connection doesn't exist, nothing will happen.
### Modify component
Modify values of the configuration of a component:
```python
>> A.params.subparam1.subparam2 = value
```
This command modifies the value of the configuration parameter at the end of
the path defined by a dot notation. The configuration is validated before being
modified; hence, if the modification results in an invalid configuration, it
will not be executed. `A` can either be a variable or an ID.
## Config presentation operations
The following operations can be run using the tool in order to view the
current configuration, sub-configurations, and available components that can be
instantiated.
### Print
Print the full configuration:
```python
>> print
```
This command displays the full configuration in JSON format on the screen.
Print component type sub-configuration:
```python
>> print connections
```
If a parameter is passed to the print command that corresponds to a component
type, or in general, a first level key of the configuration, only the relevant
sub-configuration that is selected will be displayed on the screen.
Print a particular component sub-configuration:
```python
>> print A
```
If a parameter is passed to the print command that corresponds to a variable
or an ID associated to a particular component, only its configuration will be
displayed on the screen.
### List
Print all available components:
```python
>> list
```
This command displays all available components that can be add to the
configuration, organized by type.
Print all available components of a particular type:
```python
>> list smls
```
If a type is passed as a parameter to the list command, only the available
components of that type will be listed.
## Config storage operations
The following operations can be run from the tool in order to load and save
configurations from/to files.
### Load
Load a configuration from a file:
```python
>> load filename
```
This command loads the configuration stored in the file 'filename', overriding
the configuration currently being manipulated in memory.
### Save
Save a configuration to a file:
```python
>> save filename
```
This command saves the configuration being currently handled to the file
'filename', overriding the file if it existed previously.
Save a configuration to the last file:
```python
>> save
```
If no parameter is provided, the save operation saves the current
configuration being handled to the last file loaded from or saved to.

123
doc/configuration.md
View File

@ -1,123 +0,0 @@
# Configurations
MoNanas consumes two configuration files, one for orchestrating data execution
and the other for logging.
## Orchestrating Configuration
> Note: Please refer to [Monasca/Design](design.md) to understand the concept
of each component before creating or modifying a configuration.
With the current implementation, a JSON file is used to configure how MoNanas
orchestrates the data execution pipeline. This comes in different parts as
follows.
### ID (`id`)
A unique identifier for the configuration.
### Spark (`spark_config`)
* `appName`: A string describing the Spark's application name.
* `streaming`: Attributes for data streaming.
* `batch_interval`: DStream's batch interval.
### Server (`server`)
* `port`: Port number to listen.
* `debug`: Debug mode.
### Sources (`sources`)
`sources` is a JSON object (equivalent to python dictionary in this case) where
the keys represent unique identifiers of sources. Each source has the following
attributes:
* `module`: The name of the python module used for connecting to the source.
* `params`: A JSON object representing the source's parameters (e.g. data model
used).
### Ingestors (`ingestors`)
`ingestors` is a JSON object (equivalent to python dictionary in this case)
where the keys represent unique identifiers of ingestors. Each ingestor has
the following attributes:
* `module`: The name of the python module implementing the ingestor.
* `params`: A JSON object representing the ingestor's parameters (e.g.
parameters for data conversion).
### Aggregators (`aggregators`)
`aggregators` is a JSON object (equivalent to python dictionary in this case)
where the keys represent unique identifiers of aggregators. Each aggregator
has the following attributes:
* `module`: The name of the python module implementing the aggregator.
* `params`: A JSON object representing the aggregator's parameters (e.g. how
data streams are aggregated).
> Note: Currently, only one aggregator is supported - the python module is
`default_aggregator`.
### SML Functions (`sml`)
`sml` is a JSON object (equivalent to python dictionary in this case)
where the keys represent unique identifiers of SML functions. Each SML function
has the following attributes:
* `module`: The name of the python module implementing the SML function.
* `params`: A JSON object representing the SML function's parameters (e.g.
number of samples, confidence interval).
### Voters (`voters`)
`voters` is a JSON object (equivalent to python dictionary in this case) where
the keys represent unique identifiers of voters. Each voter has the following
attributes:
* `module`: The name of the python module implementing the voter.
* `params`: A JSON object representing the voter's parameters (e.g. weights,
topology).
### Live Data Processors (`ldp`)
`ldp` is a JSON object (equivalent to python dictionary in this case) where
the keys represent unique identifiers of live data processors. Each live data
processor has the following attributes:
* `module`: The name of the python module implementing the live data processor.
* `params`: A JSON object representing the live data processor's parameters
(e.g. mode).
### Sinks (`sinks`)
`sinks` is a JSON object (equivalent to python dictionary in this case) where
the keys represent unique identifiers of sinks. Each sink has the following
attributes:
* `module`: The name of the python module used for connecting to the sink.
* `params`: A JSON object representing the sink's parameters (e.g. server's
details, data format).
### Connections
`connections` is a JSON object (equivalent to python dictionary in this case)
where each key represents a unique identifier of the component acting as the
originating end of the data flow where its associated value is an list of
unique identifiers of components acting as terminating ends of the data flow.
The information described by `connections` can be used to represent the flow
of data execution end-to-end.
## Logging Configuration
### MoNanas
MoNanas comes with a default logging configuration. To change the logging
properties including format, level, etc., simply override
`$MONANAS_HOME/config/logging.json`.
### Spark
By default, Spark's logging is at an INFO level. To avoid unnecessary console
output, change `log4j.rootCategory=INFO, console` to `log4j.rootCategory=ERROR,
console` in `$SPARK_HOME/conf/log4j.properties` or as preferred.
> Note: Not required for a Vagrant generated VM.
### Other Software/Tools
For logging configuration of other software/tools, please refer to their
user guide.

106
doc/design.md
View File

@ -1,106 +0,0 @@
# MoNanas/Design
## Overview
MoNanas enables users to compose their data execution flow through
configuration. Each part of the configuration describes how MoNanas should
handle the data.
![Tags](images/tags.png)
The needs for processing data from heterogeneous sources and a way to describe
the execution declaratively (i.e. how data flow and get processed) are
increasing. An implicit cause is the requirement to get a better insight by
utilizing the enormous amount of data we have. By taking several concepts
required to achieve that, MoNanas aims to be an statistical/analytic framework
for processing data from arbitrary sources with Monasca being the core user
to makes sense out of alerts data generated or coming into the system.
### Functional Requirements
To understand the design principle of MoNanas, first we need to understand what
the functional requirements are.
* A machine learning framework that ingests events/data from arbitrary sources
(single or multiple, homogeneous or heterogeneous), processes it and persists
into arbitrary data sinks.
* Composable components for events/data processing through configurations,
where additional machine learning algorithms can be easily integrated. For
example, a stream of events can be aggregated using a causality graph and
motif comparisons.
* An easy to use API for controlling and manipulating events/data streams as
well as querying for processed data.
> #### Additional Requirements
> * Built on open-source technologies.
## MoNanas Components
![MoNanas Architecture](images/architecture.png)
The diagram above illustrates MoNanas architecture with the following
components.
### <a name="sources"></a>Data Sources
A data source in MoNanas refers to the source of information including alerts,
raw events or any other data that can be pushed or pulled into the system.
This can be a streaming source or readily available data available for
accessing. MoNanas provides a few commonly used connectors including Kafka.
Since MoNanas uses Spark, any sources provided by Spark can be easily utilized
in MoNanas as well.
### <a name="ingestors"></a>Data Ingestors
Machine learning algorithms may not be implemented in such a way that it can
use data with arbitrary format as inputs. Therefore, an ingestor is required
for a specific type to convert the data into a common format, in this case,
numpy array, which can be understood by the SML functions. We provide each data
ingestor for each source available in MoNanas.
### <a name="aggregators"></a>Data Aggregators
Currently, MoNanas supports only one aggregator (`default_aggregator`). Its
responsibility is to aggregate data from different streams (from ingestors).
However, the support for multiple aggregators and multilayered aggregations
can be easily developed in the future.
### <a name="sml"></a>Statistical/Machine Learning (SML) Functions
An SML function may be a single machine learning algorithm or a specifically
composed sequence of algorithms that consumes numpy arrays and processes the
data to extract insights. Few SML functions are readily available for use
including LiNGAM and LibPGM for finding causality structures.
### <a name="voters"></a>Voters
In case where multiple SML functions are used in the orchestration of the
SML flow, voters are used to determine which outcome(s) of the SML functions
are used to process the data streamed from sources/ingestors. MoNanas's default
voter is `pick` where the user simply indicate the index of the SML function
that finishes the learning process.
### <a name="ldp"></a>Live Data Processors
A live data processor is optional in the data execution pipeline. It takes the
outcome of a voted SML function and uses it to process the data coming from sources
or ingestors.
### <a name="sinks"></a>Data Sinks
In MoNanas, data sinks are not necessarily persistent storage. For example, we
can push processed data back to Kafka with a different topic for Monasca or
Vitrage consumption. Any type of data can be pushed to data sink including raw
data, ingested data, processed data and outcomes of the SML functions.
## <a name="sml"></a>Statistical/Machine-Learning Algorithms
Statistical/Machine Learning (SML) algorithms are used to gain insights of
data ingested into MoNanas. It can be an arbitrary algorithm, a simple filter
or a composition of multiple algorithms (run in sequence, in parallel or a
combination of both) as long as it satisfies the data processing requirement.
The outputs of these algorithms are then discarded, aggregated or utilized,
usually by one or more live data processors to process the raw or ingested
data.
## <a name="flow"></a>Data Flow Composition
Data flow composition is a essential concept in MoNanas. It allows users to
compose a data flow and execution arbitrarily based on available components as
long as the composition is valid i.e. the connections defined by the
composition are all valid. For example, connecting a source to a voter
directly is invalid but connecting a source to a live data processor or an
ingestor is valid. Each component used in the composition can be one of many
provided by MoNanas or one of the custom developed components. Diagrams of
example compositions can be found in [MoNanas/Examples](examples.md).

89
doc/dev_guide.md
View File

@ -1,89 +0,0 @@
# MoNanas/DevGuide
## Development Environment
MoNanas's repository comes with a Vagrantfile for a quick deployment on the
client system. To use vagrant, simple do the following.
* Install [vagrant](https://www.vagrantup.com/) on your local machine.
* Clone the project from https://github.hpe.com/labs/monanas/.
* From `$MONANAS_HOME`, run `vagrant up && vagrant ssh`. This will set up a VM
using VMWare or VirtualBox and ssh onto it. Once you have logged into the
instance, following the guideline provided in
[MoNanas/GettingStarted](getting_started.md).
At the root of the project there is a `Makefile` which provides common tasks:
* `make test`: Run the test suite.
* `make style`: Check for pep8 compliance on the entire code base.
* `make testspec <python.path.to.TestCase>` A handy way to test only a
`TestCase` subclass.
* `make all`: Run both `test` and `style`.
* `make start`: Start Monanas and send the `start_streaming` action via the
REST interface.
## Adding Custom Components
As illustrated in [Monasca/Design](design.md), MoNanas's architecture was
designed to be pluggable. Therefore, integrating a new component or a new
statistical/machine learning algorithm is very simple. The following shows a
step-by-step guide on how to add a new data source and a new learning
algorithm. A custom implementation of other components can be added in similar
fashion.
### <a name="add_new_sources"></a>Add New Data Sources
When creating a new data source, everything you need is located in `main/source` package, and new sources should be contained in that package in order to keep the convention. All you need to do is extend the class `BaseSource` in `main/source/base.py` module.
#### Default configuration and Validation
The first step in the Data Source life-cycle is its creation and configuration validation. Also, a default configuration is needed by DSL in order to add a component to the configuration, and it can be very convenient for users to have a default configuration. Please, implement the following methods:
* `validate_config`
It should validate the schema of the configuration passed as parameter, checking that expected parameters are there, and values have the expected type and/or values. Please, check other classes validate_config implementations in order to have examples on how to use the Schema library. Please, make this method static by annotating it with the @staticmethod decorator.
* `get_default_config`
It should return a dictionary containing the default schema of this component. This method will be called by DSL when creating a component of this type. Please, make this method static by annotating it with the @staticmethod decorator.
#### Main logic functions
The aim of a source class is to provide data which will then be consumed by ingestors. When MoNanas is ordered to start streaming data, the source classes will be asked to create a stream of data, and other components in the pipeline may be interested in the features of the data provided by the source class.
* `create_dstream`
It should create a spark dstream using the Spark Streaming Context passed as parameter. Please, refer to spark documentation if you want more details about dstream object, and feel free to view implementations of this function by other source classes.
* `get_feature_list`
It should return a list of strings in order representing the features provided by the dstream.
#### Termination functions
When MoNanas is ordered to stop streaming data, it will call terminate_source in all the sources that are streaming.
* `terminate_source`
It should do any necessary cleanup of the source when it is terminated. For example, if the source was running a TCP server generating traffic, at this point it may want to stop it.
### <a name="add_new_algorithms"></a>Add New Learning Algorithms
When adding a new algorithm, everything you need is located in:
`main/sml` package, and new algorithms should be contained in that package in order to keep the convention. All you need to do is extend the class `BaseSML` in `main/sml/base.py` module.
#### Default configuration and Validation
Please, refer to the 'Add New Data Sources' section.
#### Main logic functions
The aim of a SML class is to train a machine learning algorithm, or do statistics to learn something, using a batch of data provided by the aggregator. When data is available, it will be manipulated by the logic implemented in the learn_structure function; the data flow will be stopped by MoNanas when all the SMLs have consumed at least the number of samples provided by the number_of_samples_required function.
* `learn_structure`
This is the function that implements the logic of the algorithm. The data is provided as a parameter, and it should return the structure learned from the data (e.g. causality matrix, or trained classifier object).
* `number_of_samples_required`
this function should return the number of samples that the algorithm requires in order to provide reliable results.
## Coding Standards
Python: All Python code conforms to the OpenStack standards at:
https://docs.openstack.org/hacking/latest/
* Developers must add unit tests for all logical components before merging to
master.
* Pull Requests are welcome and encouraged to ensure good code quality. Label
the PR as `ready for review` when all the features are completed.

556
doc/examples.md
View File

@ -1,556 +0,0 @@
# Examples
This section shows a step-by-step guide for running examples using MoNanas.
## Alert Fatigue Management
When there are too many alerts, devops become less productive due to alert
fatigue. Using MoNanas helps devops to focus on only what matters, in many
cases, "the root causes" of those alerts. This example shows how MoNanas
ingests alerts, aggregates them based on LiNGAM, a non-gaussian methods for
estimating causal structures, and pushes aggregated alerts back to a Kafka
server.
![Alert Fatigue Pipeline](images/alert_fatigue_pipeline.png)
The diagram above shows a simple data execution pipeline for consuming alerts
from Kafka server used by Monasca for messaging. However, to demonstrate this
scenario without integrating into a production environment, we use a Markov
Chain process to generate alerts (as illustrated in the diagram below) similar
to that would have been pushed to the queue by Monasca.
![Alert Fatigue Pipeline with Markov Chain Process](images/alert_fatigue_pipeline_with_markov.png)
### Running the Example
To test this example, perform the following steps.
#### Configuration File
Before running MoNanas, we need to create a configuration file describing how
we want MoNanas to orchestrate the data execution (creating a pipeline). You can find the following configuration in `$MONANAS_HOME/config/markov_source_config.json`:
```json
{
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": false
},
"sources": {
"src1": {
"module": "CloudMarkovChainSource",
"params": {
"server_sleep_in_seconds": 0.01
},
"transitions": {
"web_service": {
"run=>slow": {
"0": 0.001,
"8": 0.02,
"12": 0.07,
"14": 0.07,
"22": 0.03,
"24": 0.001
},
"slow=>run": {
"0": 0.99,
"8": 0.7,
"12": 0.1,
"14": 0.1,
"22": 0.8,
"24": 0.99
},
"stop=>run": 0.7
},
"host": {
"on=>off": 0.005,
"off=>on": 0.5
},
"switch": {
"on=>off": 0.01,
"off=>on": 0.7
}
},
"triggers": {
"support": {
"get_called": {
"0": 0.1,
"8": 0.2,
"12": 0.8,
"14": 0.8,
"22": 0.5,
"24": 0.0
}
}
},
"graph": {
"h1:host": ["s1"],
"h2:host": ["s1"],
"s1:switch": [],
"w1:web_service": ["h1"],
"w2:web_service": ["h2"]
}
}
},
"ingestors": {
"ing1": {
"module": "CloudIngestor"
}
},
"smls": {
"sml1": {
"module": "LiNGAM",
"params": {
"threshold": 0.5
}
}
},
"voters": {
"vot1": {
"module": "PickIndexVoter",
"params": {
"index": 0
}
}
},
"sinks": {
"snk1": {
"module": "KafkaSink",
"params": {
"host": "localhost",
"port": 9092,
"topic": "transformed_alerts"
}
}
},
"ldps": {
"ldp1": {
"module": "CloudCausalityLDP"
}
},
"connections": {
"src1": ["ing1", "ldp1"],
"sml1": ["vot1"],
"ing1": [],
"vot1": ["ldp1"],
"ldp1": ["snk1"],
"snk1": []
},
"feedback": {}
}
```
The flow of data execution is defined in `connections`. In this case, data
are ingested from `src1` where Monasca-like alerts are being generated
using Markov Chain process. The data are then ingested by `ing1` where
each entry is converted into a format suitable for machine learning algorithm.
MoNanas uses `numpy.array` as a standard format. Typically, an aggregator is
responsible for aggregating data from different ingestors but in this scenario,
there is only one ingestor, hence the implicit aggregator (not defined in the
configuration) simply forwards the data to `sml1`, which simply deduplicates
the data then uses LiNGAM algorithm to find a causality structure of the
aggregated data then passes the result (structure) to `vot1`. The voter is
configured to pick the output of the first SML function and forwards that to
`ldp1`. Here, the live data processor transforms data streamed from `src1`
using the causality structure and pushes it to standard output as well as the
specified Kafka server.
#### Run MoNanas
As the sink of this orchestration is Kafka, we need to run Apache Zookeeper
and Apache Kafka first.
```bash
$KAFKA_HOME/bin/zookeeper-server-start.sh \
$KAFKA_HOME/config/zookeeper.properties
$KAFKA_HOME/bin/kafka-server-start.sh \
$KAFKA_HOME/config/server.properties
```
After that, start MoNanas as follows.
```bash
python $MONANAS_HOME/run.py -p $SPARK_HOME -c $MONANAS_HOME/config/markov_source_config.json \
-l $MONANAS_HOME/config/logging.json
```
A default logging configuration file is located in `$MONANAS_HOME/config`.
Users can override this option. Currently, `-l` option is mandatory as MoNanas
does not assume a default configuration.
#### Start Data Execution
MoNanas exposes REST API for controlling the data execution. Use any HTTP
client to POST with the following request body to MoNanas server to start
data streaming.
```json
{
"action": "start_streaming"
}
```
e.g. using curl from terminal to make a request assuming the server is running
locally.
```bash
curl -H "Content-Type: application/json" -X POST \
-d '{"action": "start_streaming"}' \
http://localhost:3000/api/v1/actions
```
#### Results
The sinks for the transformed (processed) alerts defined in the configuration
are via standard output and Kafka server. Therefore, the output will be
displayed in the console. Alternatively, users can subscribe to a queue
with the topic `transformed_alerts` using any Kafka client. A result example
(live data processing with aggregation mode based on causality structure) found
by LiNGAM algorithm is shown below.
```json
{
"alarm_id": "219e65b8-395a-4588-a792-799fdbeef455",
"id": "1452900361688",
"metrics": [
"..."
],
"new_state": "OK",
"old_state": "UNDETERMINED",
"reason": "The alarm threshold(s) have not been exceeded.",
"reason_data": "{}",
"sub_alarms": [
"..."
],
"timestamp": "2016-01-15T23:26:01.688Z",
"__monanas__": {
"causes": [
"9b0129fa-7869-4436-8319-68bfa8bc88a1",
"3f4b2651-41f5-4f2a-8093-e9ad8cd5b7bf"
]
}
}
```
The alert shown above has been annotated with MoNanas's specific metadata,
`__monanas__`. In this case where the mode of live data processing is
`aggregation`, IDs of the alerts caused by this one are added onto a list,
`causes`.
> Note: The underlying causality structure is a directed acyclic graph.
In contrast, if the `filter` mode is used, the result format would be identical to
the source input format. So you can typically use it anywhere in existing data
flow. In this mode, only root cause alerts are preserved. A result example
under this mode is shown below.
## Disk Failure prediction using SMART data
Hard disk drives (HDDs) are most fragile part of systems and consequences of disk failures can be difficult to recover, or even unrecoverable.
To ensure the reliability and stability of systems, it is crucial to monitor the working conditions of HDDs in real time and detect soon-to-fail HDDs by sensors.
This example shows how MoNanas ingests SMART (self-monitoring and repair technology) data, trains a classification algorithm(Random forest), and then uses the trained algorithm to detect the likelihood failure of the HDDs.
To train the model, we use publically available [SMART dataset](https://www.backblaze.com/b2/hard-drive-test-data.html), which is collected in the Backblaze data center. Backblaze run tens of thousands hard drives from 2013 and representing the largest public SMART dataset.
### Running the Example
To test this example, perform the following steps.
#### Configuration File
Before running MoNanas, we need to create a configuration file describing how
we want MoNanas to orchestrate the data execution (creating a pipeline). You can find the following configuration in `$MONANAS_HOME/config/smart.json`:
```json
{
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": false
},
"sources": {
"src1": {
"module": "SmartFile",
"params": {
"dir": "/var/tmp/source_data/"
}
}
},
"ingestors": {
"ing1": {
"module": "SmartIngestor"
}
},
"smls": {
"sml1": {
"module": "RandomForestClassifier",
}
},
"voters": {
"vot1": {
"module": "PickIndexVoter",
"index": 0
}
},
"sinks": {
"snk3": {
"module": "KafkaSink",
"host": "127.0.0.1",
"port": 9092,
"topic": "my_topic"
},
"snk2": {
"module": "StdoutSink"
}
},
"ldps": {
"ldp1": {
"module": "Smart"
}
},
"connections": {
"src1": ["ing1", "ldp1"],
"ing1": [],
"sml1": ["vot1"],
"vot1": ["ldp1"],
"ldp1": ["snk1", "snk2"],
"snk1": [],
"snk2": [],
"snk3": []
},
"feedback": {}
}
```
The flow of data execution is defined in `connections`. In this case, data
are ingested from `src1` where smart data with csv format smart-data are being generated
using open dataxxxx. The data are then ingested by `ing1` where
each entry is converted into a format suitable for machine learning algorithm.
MoNanas uses `numpy.array` as a standard format. Typically, an aggregator is
responsible for aggregating data from different ingestors but in this scenario,
there is only one ingestor, hence the implicit aggregator (not defined in the
configuration) simply forwards the data to `sml1`, which uses Random Forest algorithm
to find the disk failures then passes the result to `vot1`.
The voter is configured to pick the output of the first SML function and
forwards that to `ldp1`. Here, the live data processor transforms data streamed from `src1`
using prediction result and pushes it to standard output as well as thespecified Kafka server.
#### Run MoNanas
Start MoNanas as follows:
```bash
python $MONANAS_HOME/run.py -p $SPARK_HOME -c $MONANAS_HOME/config/iptables_anomalies.json \
-l $MONANAS_HOME/config/logging.json
```
If you want to run your own configuration you will need to change the value of the -c parameter.
#### Start Data Execution
MoNanas exposes a REST API for controlling the data execution. Use any HTTP
client to POST with the following request body to MoNanas server (running on your localhost, at port 3000 by default) to start data streaming.
```json
{
"action": "start_streaming"
}
```
e.g. using curl from terminal to make a request assuming the server is running
locally.
```bash
curl -H "Content-Type: application/json" -X POST \
-d '{"action": "start_streaming"}' \
http://localhost:3000/api/v1/actions
```
#### Stop Data Execution
When you want to stop the example, you can send another HTTP POST to order MoNanas to stop streaming data. In this case, the request body should be:
```json
{
"action": "stop_streaming"
}
```
e.g. using curl from terminal to make a request assuming the server is running
locally.
```bash
curl -H "Content-Type: application/json" -X POST \
-d '{"action": "stop_streaming"}' \
http://localhost:3000/api/v1/actions
```
#### Results
The sinks for the transformed (processed) alerts defined in the configuration are via standard output and Kafka server.
Therefore, if the hard drive failure is predicted, the output will be displayed in the console. Alternatively, users can subscribe to a queue with the topic "my_topic" using any Kafka client.
A result example is shown below. The value of the `DiskFailure` is identifier specified in input, so you can change the value as you like, as long as it's constant and unique across all data.
```json
{
"DiskFailure": "ST3000DM001_disk001"
}
```
## Anomalies in Rule Firing Patterns
Some attacks can be recognized by patterns of rules being triggered in an anomalous fashion. For example, a Ping flood, a denial of service attack that consist in sending multiple pings to a system, would trigger IPTable rules handling the Ping much more often than if the system is not being attacked.
This example shows how MoNanas ingests iptables firing data, trains an anomaly detection algorithm (SVM One Class), and then uses the trained algorithm to detect when the system is being attacked, which is detected as an anomaly. Finally the data is stored in a SQLite database via the SQLite sink.
![IPTables Anomaly Detection Pipeline](images/iptables_arch.png)
The diagram above shows a simple data execution pipeline for generating and consuming IPTables triggering data. The source is a TCP server that generates mock data, triggering IPTables according to the logic provided by this Markov Chain:
![IPTables Markov Chain Source](images/iptables_src.png)
As you can see, there are 3 states: `stopped`, `normal`, and `attack`. Each transition marked in the diagram can happen with some probability each time period: we made sure that the system stays in `normal` state most of the time.
The system triggers 12 possible IPTables, classified in four families: `ssh`, `concrete IPs`, `http`, and `ping` with different probabilities according to the state of the system, in order to model possible ping flood attacks:
1) State `stopped`: nothing is triggered
2) State `normal`: higher probability for http traffic, but some of the other rules will be triggered.
3) State `attack`: very high probability of triggering `ping` traffic in every timestamp, keeping the rest of probabilities as they were in `normal` state.
The ingestor consumes data from the markov chain IPTable source, and counts the number of times that an IPTable of each family has been triggered in every time window, generating a 4D array per time window (one dimension per IPTable type).
The SML block is an anomaly detection algorithm which is based in SVM (Support Vector Machines). In the first phase of MoNanas execution, it trains the classifier, learning the border in the space of arrays generated by the ingestor between anomalies and non-anomalies.
After the pick voter selects the only available SML, the LDP module is ready to find anomalies in real time in phase 2. It uses the feature names returned by the IPTables Markov chain source class, which are the IPTable type represented in each position of the arrays. When new data of triggered IPTables arrive to the IPTables LDP, it is classified as anomaly or non-anomaly, and the dstream is modified with this information.
Finally the SQLite sink saves the classified data to a SQLite database.
### Running the Example
To test this example, perform the following steps.
#### Configuration File
Before running MoNanas, we need to create a configuration file describing how
we want MoNanas to orchestrate the data execution (creating a pipeline). You can find the following configuration in `$MONANAS_HOME/config/iptables_anomalies.json`:
```json
{
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": false
},
"sources": {
"src1": {
"module": "IPTablesSource",
"params": {
"server_sleep_in_seconds": 0.01
}
}
},
"ingestors": {
"ing1": {
"module": "IptablesIngestor"
}
},
"smls": {
"sml1": {
"module": "SvmOneClass"
}
},
"voters": {
"vot1": {
"module": "PickIndexVoter",
"params": {
"index": 0
}
}
},
"sinks": {
"snk1": {
"module": "IptablesSQLiteSink"
},
"snk2": {
"module": "StdoutSink"
}
},
"ldps": {
"ldp1": {
"module": "IptablesLDP"
}
},
"connections": {
"src1": ["ing1", "ldp1"],
"ing1": [],
"sml1": ["vot1"],
"vot1": ["ldp1"],
"ldp1": ["snk1", "snk2"],
"snk1": [],
"snk2": []
},
"feedback": {}
}
```
The flow of data execution is defined in `connections`, with the origin being the keys of the dictionary, and the destinations are each element of the list associated to each origin.
As you can see, the flow described in the previous section is mapped to the configuration, except for the ingestor, which is not explicitly connected; the reason for this is that all ingestors are connected to the aggregator, which is connected to all SMLs, so no explicit connection needs to be defined.
#### Run MoNanas
Start MoNanas as follows:
```bash
python $MONANAS_HOME/run.py -p $SPARK_HOME -c $MONANAS_HOME/config/iptables_anomalies.json \
-l $MONANAS_HOME/config/logging.json
```
If you want to run your own configuration you will need to change the value of the -c parameter.
#### Start Data Execution
MoNanas exposes a REST API for controlling the data execution. Use any HTTP
client to POST with the following request body to MoNanas server (running on your localhost, at port 3000 by default) to start data streaming.
```json
{
"action": "start_streaming"
}
```
e.g. using curl from terminal to make a request assuming the server is running
locally.
```bash
curl -H "Content-Type: application/json" -X POST \
-d '{"action": "start_streaming"}' \
http://localhost:3000/api/v1/actions
```
#### Stop Data Execution
When you want to stop the example, you can send another HTTP POST to order MoNanas to stop streaming data. In this case, the request body should be:
```json
{
"action": "stop_streaming"
}
```
e.g. using curl from terminal to make a request assuming the server is running
locally.
```bash
curl -H "Content-Type: application/json" -X POST \
-d '{"action": "stop_streaming"}' \
http://localhost:3000/api/v1/actions
```
#### Results
By default, the pre-selected logging level defined in `$MONANAS_HOME/config/logging.json` lets you see relevant information in the screen during this execution of this example.
The classified data generated by the LDP module is saved in a SQLite database located by default in `sqlite_sink.db` file; please feel free to inspect the database to see what was classified as anomaly and what was classified as normal traffic.

68
doc/getting_started.md
View File

@ -1,68 +0,0 @@
# MoNanas/GettingStarted
This section serves as a user guide and helps you to get started with MoNanas.
For developer guide, see: [MoNanas/DevGuide](dev_guide.md).
> Note: `$XXX_HOME` indicates the root directory where the corresponding
project resides. For example, `$KAFKA_HOME` refers to where Kafka's source
files have been extracted to.
## Pre-requisites
* Python (>= v.2.7.6)
* Apache Spark (>= v.1.6.1)
* Apache Kafka (>= v.0.8.2.1) - this comes with Apache Zookeeper.
* Vagrant (for development environment or testing MoNanas only).
> Note: Newer versions of these software/tools should be compatible, but are
not tested so nothing is guaranteed.
## Installation
* Clone MoNanas repo: https://github.com/openstack/monasca-analytics
### Everything on Host
The easiest way to install everything on a physical host is through our
provided script, `fetch-deps.sh` located in `$MONANAS_HOME`.
### Everything on VM
MoNanas comes with a Vagrantfile for quick deployment. For more information,
see: [MoNanas/DevGuide](dev_guide.md).
## Usage
* Start Apache ZooKeeper<sup>1</sup>
```bash
$KAFKA_HOME/bin/zookeeper-server-start.sh \
$KAFKA_HOME/config/zookeeper.properties
```
* Start Apache Kafka<sup>1</sup>
```bash
$KAFKA_HOME/bin/kafka-server-start.sh \
$KAFKA_HOME/config/server.properties
```
* Start MoNanas with your configuration
```bash
python $MONANAS_HOME/run.py -p $SPARK_HOME -c <config_file> \
-l <log_config_file>
```
e.g.
```bash
python $HOME/monanas/run.py -p $HOME/spark -c $HOME/monanas/config/config.json \
-l $HOME/monanas/config/logging.json
```
> Note:
1. Only when `KafkaSource` or `KafkaSink` is used in your processes.
## Configurations
MoNanas consumes two configuration files, one for orchestrating data execution
and the other for logging. A Domain Specific Language (DSL) has been implemented in order to manipulate configurations.
Please, see [MoNanas/Configuration](configuration.md) for more details on MoNanas configuration;
and [MoNanas/Banana](banana.md) for more details on `banana` the language to write recipes.
## Examples
To run examples and see how MoNanas works step-by-step, see: [MoNanas/Examples](examples.md)

BIN
doc/images/alert_fatigue_pipeline.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 30 KiB

BIN
doc/images/alert_fatigue_pipeline_with_markov.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 31 KiB

BIN
doc/images/architecture.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 85 KiB

BIN
doc/images/disk_failure_prediction.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 115 KiB

BIN
doc/images/disk_failure_prediction_with_data.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 120 KiB

BIN
doc/images/got_tons_of_alarms.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 120 KiB

BIN
doc/images/iptables_arch.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 15 KiB

BIN
doc/images/iptables_src.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 21 KiB

BIN
doc/images/monanas-logo.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 18 KiB

BIN
doc/images/save_long_investigation_works.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 115 KiB

BIN
doc/images/tags.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 937 KiB

213
doc/monasca.md
View File

@ -1,213 +0,0 @@
# Integrate with Monasca
This section shows a step-by-step guide for integration MoNanas with Monasca.
## Alarm management
For example, let's say you have a some cloud system built on OpenStack,
and all services have been monitored by Monasca.
Operators got tons of alarms when a trouble happened
even if there is actual trouble in one service.
![Got tons of alarms](images/got_tons_of_alarms.png)
Integrating MoNanas and Monasca eliminates unnecessary alarms.
And save the operator from long investigation works!
![Save long investigation works](images/save_long_investigation_works.png)
### Running the MoNanas
To manage alarms from Monasca, perform the following steps.
#### Alarm Definition in Monasca.
Refer to [Monanas API](https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md).
#### Configuration File in MoNanas
Before running MoNanas, we need to create a configuration file describing how
we want MoNanas to orchestrate the data execution (creating a pipeline).
You can edit configuration in `$MONANAS_HOME/config/markov_source_config.json` like the following.
```json
{
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": false
},
"sources": {
"src1": {
"module": "TextFileSource",
"params": {
"dir": "/tmp/source_data"
},
"features": [
"api_response.time.5sec.cinder1",
"api_response.time.5sec.cinder2",
"api_response.time.5sec.heat1",
"api_response.time.5sec.heat2",
"api_response.time.5sec.keystone1",
"api_response.time.5sec.keystone2",
"api_response.time.5sec.neutron1",
"api_response.time.5sec.neutron2",
"api_response.time.5sec.nova1",
"api_response.time.5sec.nova2"
]
}
},
"ingestors": {
"ing1": {
"module": "CloudIngestor"
}
},
"smls": {
"sml1": {
"module": "LiNGAM",
"threshold": 0.5
}
},
"voters": {
"vot1": {
"module": "PickIndexVoter",
"index": 0
}
},
"sinks": {
"snk1": {
"module": "FileSink2",
"params": {
"dir": "/var/tmp/result/markov",
"file_prefix": "result"
}
},
"snk2": {
"module": "StdoutSink"
}
},
"ldps": {
"ldp1": {
"module": "CloudCausalityLDP"
}
},
"connections": {
"src1": ["ing1", "ldp1"],
"sml1": ["vot1"],
"ing1": [],
"vot1": ["ldp1"],
"ldp1": ["snk1"],
"snk1": [],
"snk2": []
},
"feedback": {}
}
```
The flow of data execution is defined in `connections`. In this case, data
are ingested from `src1`. The data are then ingested by `ing1` where
each entry is converted into a format suitable for machine learning algorithm.
MoNanas uses `numpy.array` as a standard format. Typically, an aggregator is
responsible for aggregating data from different ingestors but in this scenario,
there is only one ingestor, hence the implicit aggregator (not defined in the
configuration) simply forwards the data to `sml1`, which simply deduplicates
the data then uses LiNGAM algorithm to find a causality structure of the
aggregated data then passes the result (structure) to `vot1`. The voter is
configured to pick the output of the first SML function and forwards that to
`ldp1`. Here, the live data processor transforms data streamed from `src1`
using the causality structure and pushes it to standard output as well as the
specified Kafka server.
#### Run MoNanas
As the sink of this orchestration is Kafka, we need to run Apache Zookeeper
and Apache Kafka first.
```bash
$KAFKA_HOME/bin/zookeeper-server-start.sh \
$KAFKA_HOME/config/zookeeper.properties
$KAFKA_HOME/bin/kafka-server-start.sh \
$KAFKA_HOME/config/server.properties
```
After that, start MoNanas as follows.
```bash
python $MONANAS_HOME/run.py -p $SPARK_HOME -c $MONANAS_HOME/config/markov_source_config.json \
-l $MONANAS_HOME/config/logging.json
```
A default logging configuration file is located in `$MONANAS_HOME/config`.
Users can override this option. Currently, `-l` option is mandatory as MoNanas
does not assume a default configuration.
#### Start Data Execution
MoNanas exposes REST API for controlling the data execution. Use any HTTP
client to POST with the following request body to MoNanas server to start
data streaming.
```json
{
"action": "start_streaming"
}
```
e.g. using curl from terminal to make a request assuming the server is running
locally.
```bash
curl -H "Content-Type: application/json" -X POST \
-d '{"action": "start_streaming"}' \
http://localhost:3000/api/v1/actions
```
#### Results
The sinks for the transformed (processed) alerts defined in the configuration
are via standard output and Kafka server. Therefore, the output will be
displayed in the console. Alternatively, users can subscribe to a queue
with the topic `transformed_alerts` using any Kafka client. A result example
(live data processing with aggregation mode based on causality structure) found
by LiNGAM algorithm is shown below.
```json
{
"alarm": {
"state": "ALARM",
"alarm_definition": {
"name": "api_response.time.5sec.nova2"
}
},
"__monanas__":
[
{
"causes": [
"api_response.time.5sec.neutron2",
"api_response.time.5sec.cinder1",
"api_response.time.5sec.nova1",
"api_response.time.5sec.keystone1",
"api_response.time.5sec.keystone2"
]
},
{
"results": [
"api_response.time.5sec.cinder2",
"api_response.time.5sec.heat1",
"api_response.time.5sec.heat2",
"api_response.time.5sec.nova1",
"api_response.time.5sec.neutron1",
"api_response.time.5sec.keystone1"
]
}
]
}
```

20
doc/use_cases.md
View File

@ -1,20 +0,0 @@
# Example Use Cases
Below are few use cases that are relevant to OpenStack. However, MoNanas
enables you to add your own [data ingestors](doc/dev_guide.md#ingestors).
| Example | Alert Fatigue Management | Anomaly Detection |
|:------------------------------|:-------------------------|:------------------|
| **Dataset** | Synthetic, but representative, set of Monasca alerts that are processed in a stream manner. This alert set represents alerts that are seen in a data center consisting of several racks, enclosures and nodes. | `iptables` rules together with the number of times they are fired in a time period. |
| **Parsing** | Monasca alert parser. | Simple parser extracting period and number of fire events per rule. |
| **SML algorithm flow** | `filter(bad_formatted) -> filter(duplicates) -> aggregate() >> aggregator` aggregation can utilize conditional independence causality, score-based causality, linear algebra causality. | `detect_anomaly() >> aggregator` anomaly detection could be based on SVM, trend, etc. |
| **Output** | Directed acyclic alert graph with potential root causes at the top. | Rule set with an anomalous number of firing times in a time period. |
| **:information_source: Note** | Even though this could be consumed directly by devops, the usage of [Vitrage MoNanas Sink](doc/getting_started.md#vitrage_sink) is recommended. The output of this module can speed up creation of a [Vitrage](https://wiki.openstack.org/wiki/Vitrage) entity graph to do further analysis on it. | None. |
`->` indicates a sequential operation in the flow.
`//` indicates beginning of group of operations running in parallel.
`-` indicates operations running in parallel.
`>>` indicates end of group of operations running in parallel.

92
fetch-deps.sh
View File

@ -1,92 +0,0 @@
# This file is intended to be used by vagrant only.
if [ "$VAGRANT_ENV" ]; then
# Create a temporary folder
mkdir ~/tmp
cd ~/tmp
# Java 8
sudo -E apt-get update
sudo -E apt-get -y install openjdk-7-jre-headless openjdk-7-jdk
# Maven
curl ftp://mirror.reverse.net/pub/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz > mvn.tar.gz
tar -xzf mvn.tar.gz
sudo mv apache-maven-3.3.9 /usr/local/apache-maven-3.3.9
echo 'export PATH=/usr/local/apache-maven-3.3.9/bin:$PATH' >> $HOME/.profile
source $HOME/.profile
mkdir ~/.m2
LENGTH_FOR_HOST=`expr match "$HTTP_PROXY" 'http://[\.A-Za-z\-]*'`-7
echo '<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
<proxies>
<proxy>'"
<active>true</active>
<protocol>http</protocol>
<host>${HTTP_PROXY:7:$LENGTH_FOR_HOST}</host>
<port>8080</port>
<nonProxyHosts>localhost</nonProxyHosts>
</proxy>
</proxies></settings>" > ~/.m2/settings.xml
# Scala
curl https://downloads.typesafe.com/scala/2.11.7/scala-2.11.7.deb > scala.deb
sudo dpkg -i scala.deb
# Scala build tool
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo -E apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
sudo -E apt-get update
sudo -E apt-get -y install sbt
# Spark
curl http://apache.claz.org/spark/spark-1.6.1/spark-1.6.1.tgz > spark.tgz
echo "-------------------------"
echo "unzip spark to ~/spark"
mkdir ~/spark/
tar -xzf spark.tgz
mv spark-1.6.1/ ~/spark/spark-1.6.1
cd ~/spark/spark-1.6.1
mvn -DskipTests clean package
cd ~/tmp
# config for logging in spark
cp ~/spark/spark-1.6.1/conf/log4j.properties.template ~/spark/spark-1.6.1/conf/log4j.properties
sed -i 's/log4j.rootCategory=INFO/log4j.rootCategory=ERROR/g' ~/spark/spark-1.6.1/conf/log4j.properties
# Kafka
mkdir ~/kafka
curl http://apache.arvixe.com/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz > kafka.tgz
echo "-------------------------"
echo "unzip kafka to ~/kafka"
tar -xzf kafka.tgz -C ~/kafka
mv ~/kafka/kafka_2.11-0.9.0.0/* ~/kafka
rm -r ~/kafka/kafka_2.11-0.9.0.0
# Python dependencies
sudo -E apt-get -y install python-pip python-setuptools
sudo -E pip install pep8
# Remove temporary folder
echo "-------------------------"
echo "Removing temporary folder"
rm -r ~/tmp
# Monanas dependencies
echo "-------------------------"
echo "Get Monanas deps"
cd /vagrant
sudo -E apt-get -y install python-numpy python-scipy ipython
sudo -E -H python setup.py develop
# Environment setup
set -v
echo 'export SPARK_HOME=~/spark/spark-1.6.1' >> $HOME/.profile
echo 'export KAFKA_HOME=~/kafka' >> $HOME/.profile
echo 'export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH' >> $HOME/.profile
set +v
else
echo "This file is intended to be used by a vagrant vm only."
fi

0
monasca_analytics/__init__.py
View File

40
monasca_analytics/banana/README.md
View File

@ -1,40 +0,0 @@
## Banana configuration language
This module contains everything related to Banana. In each
sub-module (sub-folder) you will find a `README.md` file
that describes:
* Purpose of the module.
* The current status of the implementation.
* How testing is done.
The compiler is split in passes. Each pass performs some
transformations and / or generates more data. Only the last
step has side-effects on the Monanas instance.
Each sub-module roughly maps to one pass run by the compiler.
### Passes
The Banana compiler runs the following passes:
* `parse`, parse the input and build an [AST](./grammar/README.md).
* `typeck`, type check the input.
* `deadpathck`, remove dead path in the connections.
* `eval`, evaluate the AST generated.
Each pass makes some assumptions about the state of
the data, and in particular that the previous passes
have run successfully. While this is made obvious by
the arguments required to run some passes, it is less
so for others.
Generally, things to remember:
* Changing the ordering of passes is more likely to
break things.
* New passes are free to modify the AST / TypeTable.
* New passes should not break invariants.
For more information on passes, have a look in their
specific `README.md` file.

0
monasca_analytics/banana/__init__.py
View File

0
monasca_analytics/banana/bytecode/__init__.py
View File

1030
monasca_analytics/banana/bytecode/assembler.py
View File

File diff suppressed because it is too large Load Diff

104
monasca_analytics/banana/bytecode/decorators.py
View File

@ -1,104 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# This file is an adaptation of BytecodeAssembler:
#
# http://peak.telecommunity.com/DevCenter/BytecodeAssembler#toc
#
# It has been adapted to match the requirements of monasca_analytics
import sys
def enclosing_frame(frame=None, level=3):
"""Get an enclosing frame that skips DecoratorTools callback code"""
frame = frame or sys._getframe(level)
while frame.f_globals.get('__name__') == __name__:
frame = frame.f_back
return frame
def decorate_assignment(callback, depth=2, frame=None):
"""Invoke 'callback(frame,name,value,old_locals)' on next assign in 'frame'
The frame monitored is determined by the 'depth' argument, which gets
passed to 'sys._getframe()'. When 'callback' is invoked, 'old_locals'
contains a copy of the frame's local variables as they were before the
assignment took place, allowing the callback to access the previous value
of the assigned variable, if any. The callback's return value will become
the new value of the variable. 'name' is the name of the variable being
created or modified, and 'value' is its value (the same as
'frame.f_locals[name]').
This function also returns a decorator function for forward-compatibility
with Python 2.4 '@' syntax. Note, however, that if the returned decorator
is used with Python 2.4 '@' syntax, the callback 'name' argument may be
'None' or incorrect, if the 'value' is not the original function (e.g.
when multiple decorators are used).
"""
frame = enclosing_frame(frame, depth + 1)
oldtrace = [frame.f_trace]
old_locals = frame.f_locals.copy()
def tracer(frm, event, arg):
if event == 'call':
# We don't want to trace into any calls
if oldtrace[0]:
# ...but give the previous tracer a chance to, if it wants
return oldtrace[0](frm, event, arg)
else:
return None
try:
if frm is frame and event != 'exception':
# Aha, time to check for an assignment...
for k, v in frm.f_locals.items():
if k not in old_locals or old_locals[k] is not v:
break
else:
# No luck, keep tracing
return tracer
# Got it, fire the callback, then get the heck outta here...
frm.f_locals[k] = callback(frm, k, v, old_locals)
finally:
# Give the previous tracer a chance to run before we return
if oldtrace[0]:
# And allow it to replace our idea of the "previous" tracer
oldtrace[0] = oldtrace[0](frm, event, arg)
uninstall()
return oldtrace[0]
def uninstall():
# Unlink ourselves from the trace chain.
frame.f_trace = oldtrace[0]
sys.settrace(oldtrace[0])
# Install the trace function
frame.f_trace = tracer
sys.settrace(tracer)
def do_decorate(f):
# Python 2.4 '@' compatibility; call the callback
uninstall()
frame = sys._getframe(1)
return callback(
frame, getattr(f, '__name__', None), f, frame.f_locals
)
return do_decorate

51
monasca_analytics/banana/bytecode/symbols.py
View File

@ -1,51 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# This file is an adaptation of BytecodeAssembler:
#
# http://peak.telecommunity.com/DevCenter/BytecodeAssembler#toc
#
# It has been adapted to match the requirements of monasca_analytics
"""Symbolic global constants, like 'None', 'NOT_FOUND', etc."""
class Symbol(object):
"""Symbolic global constant"""
__slots__ = ['_name', '_module']
__name__ = property(lambda self: self._name)
__module__ = property(lambda self: self._module)
def __init__(self, symbol, module_name):
self.__class__._name.__set__(self, symbol)
self.__class__._module.__set__(self, module_name)
def __reduce__(self):
return self._name
def __setattr__(self, attr, val):
raise TypeError("Symbols are immutable")
def __repr__(self):
return self.__name__
__str__ = __repr__
NOT_GIVEN = Symbol("NOT_GIVEN", __name__)
NOT_FOUND = Symbol("NOT_FOUND", __name__)

0
monasca_analytics/banana/cli/__init__.py
View File

27
monasca_analytics/banana/cli/const.py
View File

@ -1,27 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
CREATE = "create"
CONNECT = "connect"
DISCONNECT = "disconnect"
LOAD = "load"
SAVE_AS = "save_as"
SAVE = "save"
REMOVE = "remove"
MODIFY = "modify"
PRINT = "print"
LIST = "list"
HELP = "help"

367
monasca_analytics/banana/cli/dsl.py
View File

@ -1,367 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import copy
import json
import logging
import os
import six
import voluptuous
from monasca_analytics.config import const
from monasca_analytics.config import validation
from monasca_analytics.exception import dsl as err
import monasca_analytics.util.common_util as cu
logger = logging.getLogger(__name__)
MODULE = "module"
"""
DEPRECATED: Preferred way is to now use the Banana language instead.
"""
class MonanasDSL(object):
def __init__(self, config_file_path=None):
"""Constructor with an optional configuration file path
If the configuration file is provided, it will be loaded at
MonanasDLS creation time.
If no configuration file path is provided, the default base
configuration defined in config.const will be used.
:type config_file_path: str
:param config_file_path: path to the file containing the configuration
that will be loaded at MonansDSL creation
"""
if config_file_path:
self.load_configuration(config_file_path)
else:
self._config = const.get_default_base_config()
self._init_ids_dictionary()
def add_component(self, _component_config):
"""Add the component configuration passed as parameter
Add it to MonanasDSL configuration, and return a new unique ID
generated for it.
The configuration passed as parameter is validated, raising exceptions
if the module does not exist or the configuration is invalid.
:type _component_config: dict
:param _component_config: configuration of the component to be added
:rtype: str
:returns: Component ID for the added component
:raises: MonanasNoSuchClassError -- if the defined class doesn't
exist or is not of a valid type
:raises: SchemaError -- if the configuration is not valid for
the class.
"""
if type(_component_config) == str:
_component_config = json.loads(_component_config)
clz = cu.get_class_by_name(_component_config[MODULE])
clz.validate_config(_component_config)
comp_type = cu.get_component_type(_component_config[MODULE])
comp_id = self._generate_id(comp_type)
self._config[comp_type][comp_id] = _component_config
self._config[const.CONNECTIONS][comp_id] = []
return comp_id
def modify_component(self, comp_id, params_path, value):
"""Overrides the value of the configuration path of a component
Modifies the configuration of the component defined by comp_id,
following the path in the dictionary defined by params_path, and
assigning the value value.
:type comp_id: str
:param comp_id: ID of the component to be modified
:type params_path: list
:param params_path: parameters path to modify in the config
:type value: str | int | float
:param value: new value to be assigned, will be parsed
according to the expected configuration
:rtype: bool
:returns: True if the component was modified (or if the modification
result was the same as the existing configuration),
False otherwise
:raises: SchemaError -- if the new configuration would not be valid
"""
comp_type = self._get_type_by_id(comp_id)
if not comp_type:
return False
new_conf = copy.deepcopy(self._config[comp_type][comp_id])
logger.debug("Modifying " + comp_id + ", existing config = " +
str(new_conf))
clz = cu.get_class_by_name(new_conf[MODULE])
for var_type in [str, int, float]:
try:
parsed_value = var_type(value)
except ValueError as e:
logger.debug(str(e))
continue
new_conf = self._modify_dictionary(new_conf, params_path,
parsed_value)
try:
clz.validate_config(new_conf)
logger.debug("New validated config = " + str(new_conf))
self._config[comp_type][comp_id] = new_conf
return True
except voluptuous.Invalid as e:
logger.debug(str(e))
continue
return False
def remove_component(self, component_id):
"""Remove a component from the configuration.
Removes from the configuration the component whose ID matches the
one passed as parameter.
:type component_id: str
:param: component_id: ID of the component to be removed
:rtype: bool
:returns: True if the component was removed, False otherwise
:raises: DSLExistingConnection -- if at least a connection exists with
the component as origin or destination.
"""
if self._is_connected(component_id):
raise err.DSLExistingConnection("Cannot remove component " +
component_id +
" because it is still connected")
for comp_type in const.components_types:
if component_id in self._config[comp_type].keys():
del self._config[comp_type][component_id]
del self._config[const.CONNECTIONS][component_id]
return True
return False
def connect_components(self, origin_id, dest_id):
"""Connect the two components passed as parameter.
Being the origin the first one and the destination the second one.
If the connection already existed, False is returned. If we created a
new connection, True is returned.
If any of the components is not defined, a DSLInexistentComponent
exception is raised.
:type origin_id: str
:param origin_id: ID of the component at the origin of the connection
:type dest_id: str
:param dest_id: ID of the component at the destination of the
connection
:rtype: bool
:returns: True if the components were connected, False if
the connection already existed
:raises: DSLInexistentComponent -- if either the origin or the
destination are not defined in the configuration
"""
if not self._component_defined(origin_id):
raise err.DSLInexistentComponent(origin_id)
if not self._component_defined(dest_id):
raise err.DSLInexistentComponent(dest_id)
if dest_id in self._config[const.CONNECTIONS][origin_id]:
return False
if not self._validate_connection_by_ids(origin_id, dest_id):
raise err.DSLInvalidConnection("Connection from " + origin_id +
" to " + dest_id +
" is not allowed")
self._config[const.CONNECTIONS][origin_id].append(dest_id)
return True
def disconnect_components(self, origin_id, dest_id):
"""Disconnect the component dest_id from origin_id
If the connection from origin_id to dest_id exists, it is removed,
and the function returns true.
If it didn't exist, the function returns false and nothing happens
:type origin_id: str
:param origin_id: ID of the component at the origin of the connection
:type dest_id: str
:param dest_id: ID of the component at the destination of the
connection
:rtype: bool
:returns: True if the components were already disconnected,
False if the connection didn't exist in the first place
"""
if origin_id in self._config[const.CONNECTIONS]:
if dest_id in self._config[const.CONNECTIONS][origin_id]:
self._config[const.CONNECTIONS][origin_id].remove(dest_id)
return True
return False
def load_configuration(self, config_file_path):
"""Load a configuration from the file passed as parameter
:type config_file_path: str
:param config_file_path: file path containing the
configuration that will be loaded
"""
self._config = cu.parse_json_file(config_file_path)
self._init_ids_dictionary()
def _init_ids_dictionary(self):
self.ids_by_type = {
const.SOURCES: ("src", 0),
const.INGESTORS: ("ing", 0),
const.SMLS: ("sml", 0),
const.VOTERS: ("vot", 0),
const.SINKS: ("snk", 0),
const.LDPS: ("ldp", 0),
const.CONNECTIONS: ("connections", 0)}
def save_configuration(self, config_file_path, overwrite_file=True):
"""Save the configuration to the file passed as parameter
:type config_file_path: str
:param config_file_path: file path where the configuration
will be saved
:type overwrite_file: bool
:param overwrite_file: True will overwrite the file if it exists,
False will make the function return without
saving.
:rtype: bool
:returns: True if the configuration was saved,
False otherwise
"""
if os.path.exists(config_file_path) and\
os.stat(config_file_path).st_size > 0 and\
overwrite_file is False:
return False
with open(config_file_path, "w") as f:
str_config = json.dumps(self._config)
f.write(str_config)
return True
def _generate_id(self, comp_type):
"""Generate a new ID for the component type passed as parameter
After the ID is generated, the last max checked number is stored
in the ids_by_type dictionary
:type comp_type: str
:param comp_type: type of component for which the ID will
be created
:raises: KeyError -- if the comp_type does not correspond to a
component type of the configuration
"""
typ, num = self.ids_by_type[comp_type]
num += 1
while(typ + str(num) in self._config[comp_type].keys()):
num += 1
self.ids_by_type[comp_type] = (typ, num)
return typ + str(num)
def _get_type_by_id(self, component_id):
"""Gets the type of a copmonent from its ID
:type component_id: str
:param component_id: ID of a component
:rtype: str
:returns: type of component for the ID passed as parameter
"""
for comp_type in const.components_types:
if component_id in self._config[comp_type]:
return comp_type
def _component_defined(self, component_id):
"""Check if a component is defined in the configuration
:type component_id: str
:param component_id: ID of a component
:rtype: bool
:returns: True if the component is defined in the configuration
"""
comp_type = self._get_type_by_id(component_id)
if not comp_type:
return False
return True
def _is_connected(self, component_id):
"""Check if a component is connected
Both in and out connections will be considered: if there is any
connection with component_id as either source or destination, this
function will return true. Empty connections lists are ignored.
:type component_id: str
:param component_id: ID of a component
:rtype: bool
:returns: True if the component is connected to another component
according to the configuration, False otherwise
"""
cons = self._config[const.CONNECTIONS]
for origin_id, dest_ids in six.iteritems(cons):
if dest_ids == []:
continue
if origin_id == component_id:
return True
for dest_id in dest_ids:
if dest_id == component_id:
return True
return False
def _validate_connection_by_ids(self, origin_id, dest_id):
"""Validate the connection from origin_id to dest_id
The connection from the component with ID = origin_id
to the component with ID = dest_id is validated according to the
valid connections dictionary defined in the validation module.
:type origin_id: str
:param origin_id: ID of the origin component
:type dest_id: str
:param dest_id: ID of the destination component
:rtype: bool
:returns: True if the connection is allowed, False otherwise
"""
origin_type = self._get_type_by_id(origin_id)
dest_type = self._get_type_by_id(dest_id)
if dest_type in validation.valid_connection_types[origin_type]:
return True
return False
def _modify_dictionary(self, target_dict, params_path, value):
"""Override the value at the end of a path in the target_dictionary
Modify the dictionary passed as first parameter, assigning the value
passed as last parameter to the key path defined by params_path
:type target_dict: dict
:param target_dict: target to be modified
:type params_path: list[str]
:param params_path: hierarchy of keys to navigate in
the dictionary, pointing the leave to modify
:type value: object
:param value: Value that will be assigned to the path defined
by params_path in the dictionary
:rtype dict
:returns: Modified dictionary
"""
aux = target_dict
for i, param in enumerate(params_path):
if param not in aux.keys():
aux[param] = {}
if i == len(params_path) - 1:
aux[param] = value
else:
aux = aux[param]
return target_dict

275
monasca_analytics/banana/cli/interpreter.py
View File

@ -1,275 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import copy
import json
import logging
from monasca_analytics.banana.cli import const as dsl_const
from monasca_analytics.banana.cli import dsl
from monasca_analytics.banana.cli import parser
from monasca_analytics.config import const as config_const
from monasca_analytics.exception import dsl as err
import monasca_analytics.util.common_util as cu
logger = logging.getLogger(__name__)
class DSLInterpreter(object):
def __init__(self):
self.file_in_use = None
self.dsl = dsl.MonanasDSL()
self.mappings = {}
def execute_string(self, str_program):
"""Parse and execute the command/s in the string passed as parameter
:type str_program: str
:param str_program: command to be executed
:rtype: str
:returns: execution result
"""
info = parser.get_parser().parseString(str_program)
return self.execute(info)
def execute_file(self, file_program):
"""Parse and execute the command/s in the file passed as parameter
:type file_program: str
:param file_program: path to the file containing the
command to be executed
:rtype: str
:returns: execution result
"""
info = parser.get_parser().parseFile(file_program)
return self.execute(info)
def execute(self, info):
"""Execute parsed command/s
:type info: dict
:param info: containing the parsed instructions
:rtype: str
:returns: execution result
"""
for cmd in info:
for key in cmd.keys():
if key == dsl_const.CREATE:
return self.create(cmd[key][0], cmd[key][1])
elif key == dsl_const.CONNECT:
return self.connect(cmd[key][0], cmd[key][1])
elif key == dsl_const.DISCONNECT:
return self.disconnect(cmd[key][0], cmd[key][1])
elif key == dsl_const.LOAD:
return self.load(cmd[key][0])
elif key == dsl_const.SAVE_AS:
return self.save(cmd[key][0])
elif key == dsl_const.SAVE:
return self.save()
elif key == dsl_const.REMOVE:
return self.remove(cmd[key][0])
elif key == dsl_const.MODIFY:
return self.modify(
cmd[key][0], cmd[key][1:-1], cmd[key][-1])
elif key == dsl_const.PRINT:
if len(cmd[key]) > 0:
return self.prnt(cmd[key][0])
else:
return self.prnt_all()
elif key == dsl_const.LIST:
if len(cmd[key]) > 0:
return self.list(cmd[key][0])
else:
return self.list_all()
elif key == dsl_const.HELP:
return self.help()
else:
return logger.warn("Wrong command" + str(cmd))
def create(self, varname, modulename):
"""Add a module defined by modulename in the configuration
:type varname: str
:param varname: name of the variable representing
the new component
:rtype: str
:returns: new component ID
"""
clz = cu.get_class_by_name(modulename)
conf = copy.deepcopy(clz.get_default_config())
comp_id = self.dsl.add_component(conf)
self.mappings[varname] = comp_id
return comp_id
def connect(self, origin_varname, dest_varname):
"""Connect two components
:type origin_varname: str
:param origin_varname: variable name or ID of the source
component of the connection
:type dest_varname: str
:param dest_varname: variable name or ID of the destination
component of the connection
:rtype: bool
:returns: True if the connection was performed,
False otherwise
"""
origin_id = self._get_id(origin_varname)
dest_id = self._get_id(dest_varname)
return self.dsl.connect_components(origin_id, dest_id)
def _get_id(self, name_or_id):
"""Get the ID from a name or ID
:type name_or_id: str
:param name_or_id: variable name or ID
:rtype: str
:returns: ID
"""
if name_or_id in self.mappings.keys():
return self.mappings[name_or_id]
for comp_type in config_const.components_types:
if name_or_id in self.dsl._config[comp_type]:
return name_or_id
raise err.DSLInterpreterException("undefined variable: " + name_or_id)
def disconnect(self, origin_varname, dest_varname):
"""Disconnect two components
:type origin_varname: str
:param origin_varname: variable name or ID of the source
component of the connection
:type dest_varname: str
:param dest_varname: variable name or ID of the destination
component of the connection
:rtype: bool
:returns: True if the components were disconnected,
False otherwise
"""
origin_id = self._get_id(origin_varname)
dest_id = self._get_id(dest_varname)
return self.dsl.disconnect_components(origin_id, dest_id)
def load(self, filepath):
"""Load configuration from a file
:type filepath: str
:param filepath: path to the file to be loaded
"""
self.dsl.load_configuration(filepath)
self.file_in_use = filepath
def save(self, filepath=None):
"""Save configuration to a file
:type filepath: str
:param filepath: (Optional) path to the file where the configuration
will be saved. If the path is not provided, the last
file used for saving or loading will be used.
"""
if not filepath:
filepath = self.file_in_use
saved = self.dsl.save_configuration(filepath, overwrite_file=True)
if saved:
self.file_in_use = filepath
return saved
def remove(self, varname):
"""Remove a variable or ID from the configuration
:type varname: str
:param varname: variable name or ID mapped to the component
that will be removed from the configuration
"""
remove_id = self._get_id(varname)
return self.dsl.remove_component(remove_id)
def modify(self, varname, params, value):
"""Override the value of the configuration path of a component
:type varname: str
:param varname: variable name or ID mapped to the component
:type params: list
:param params: path to be modified in the configuration
:type value: float | int | str
:param value: value to assign
"""
modify_id = self._get_id(varname)
return self.dsl.modify_component(modify_id, params, value)
def prnt(self, varname):
"""Print the configuration of the module/s defined by varname
If varname is a variable or ID associated to a particular component,
the configuration of that component will be printed. If it is a type
of components, the configurations of all components of that type
will be printed.
:type varname: str
:param varname: variable, ID or type to be printed
:rtype: str
:returns: requested configuration in string format
"""
if varname in self.dsl._config.keys():
return self._json_print(self.dsl._config[varname])
itemId = self._get_id(varname)
for k in config_const.components_types:
if itemId in self.dsl._config[k]:
return self._json_print(self.dsl._config[k][itemId])
def prnt_all(self):
"""Print the the whole configuration
:rtype: str
:returns: whole configuration in string format
"""
return self._json_print(self.dsl._config)
def _json_print(self, jstr):
"""Format Json as a clean string"""
return json.dumps(jstr, indent=4, separators=(',', ': '))
def list(self, key):
"""List the available components of the type passed as parameter"""
ret = ""
if key in config_const.components_types:
for name in cu.get_available_class_names(key):
ret += "- " + name + "\n"
return ret
def list_all(self):
"""List all available components grouped by type"""
ret = ""
for key in config_const.components_types:
ret += "- " + key + "\n"
for name in cu.get_available_class_names(key):
ret += " - " + name + "\n"
return ret
def help(self):
return """
Available commands
- print: prints current configuration
- list: shows available modules
- load: loads a config from a file
- save: saves a config to a file
- <var> = <module>: instantiates module <module>, referenced by <var>
- <var1>-><var2>: connects the module <var1> to the module <var2>
- <var1>!-><var2>: disconnects the module <var1> from the module <var2>
- rm <var>: removes the module corresponding to <var>
- exit: finishes the execution of monanas command line
"""

66
monasca_analytics/banana/cli/parser.py
View File

@ -1,66 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import pyparsing as p
from monasca_analytics.banana.cli import const
EQUALS = p.Literal("=").suppress()
CONNECT = p.Literal("->").suppress()
DISCONNECT = p.Literal("!->").suppress()
LPAREN = p.Literal("(").suppress()
RPAREN = p.Literal(")").suppress()
LOAD = p.CaselessKeyword("load").suppress()
SAVE = p.CaselessKeyword("save").suppress()
REMOVE = p.CaselessKeyword("rm").suppress()
PRINT = p.CaselessKeyword("print").suppress()
LIST = p.CaselessKeyword("list").suppress()
HELP = p.CaselessKeyword("help").suppress()
DOT = p.Literal(".").suppress()
VARNAME = p.Word(p.alphas + "_", p.alphanums + "_")
PARAMETER = p.Word(p.alphanums + "_-")
MODULE_NAME = p.Word(p.alphanums + "_-")
VALUE = p.Word(p.alphanums + "_-.")
PATH = p.Word(p.alphanums + "_-/\.")
cmd_create = (VARNAME + EQUALS + MODULE_NAME)
cmd_connect = (VARNAME + CONNECT + VARNAME)
cmd_disconnect = (VARNAME + DISCONNECT + VARNAME)
cmd_modify = (VARNAME + p.OneOrMore(DOT + PARAMETER) + EQUALS + VALUE)
cmd_load = (LOAD + p.Optional(LPAREN) + PATH + p.Optional(RPAREN))
cmd_save = (SAVE + p.Optional(LPAREN) + p.Optional(RPAREN))
cmd_save_as = (SAVE + p.Optional(LPAREN) + PATH + p.Optional(RPAREN))
cmd_remove = (REMOVE + p.Optional(LPAREN) + VARNAME + p.Optional(RPAREN))
cmd_print = (PRINT + p.Optional(LPAREN) + p.Optional(VARNAME) +
p.Optional(RPAREN))
cmd_list = (LIST + p.Optional(LPAREN) + p.Optional(VARNAME) +
p.Optional(RPAREN))
cmd_help = (HELP + p.Optional(LPAREN) + p.Optional(RPAREN))
bnfLine = cmd_create(const.CREATE) | cmd_connect(const.CONNECT) | \
cmd_disconnect(const.DISCONNECT) | cmd_load(const.LOAD) | \
cmd_save_as(const.SAVE_AS) | cmd_save(const.SAVE) | \
cmd_remove(const.REMOVE) | cmd_modify(const.MODIFY) | \
cmd_print(const.PRINT) | cmd_list(const.LIST) | cmd_help(const.HELP)
bnfComment = "#" + p.restOfLine
bnf = p.OneOrMore(p.Group(bnfLine))
bnf.ignore(bnfComment)
def get_parser():
return bnf

22
monasca_analytics/banana/deadpathck/README.md
View File

@ -1,22 +0,0 @@
## Dead path checker
Dead path checking is about removing paths in the pipeline that
lead to nothing. For instance, if there's no source or no sink
in a path. This pass is the only one that modifies the AST.
This is the third step of the pipeline:
```
+---------------------+ +---------------------+
| | | |
---> | AST & TypeTable | ---- deadpathck ---> | AST' & TypeTable' | --->
| | | |
+---------------------+ +---------------------+
```
### Current status:
* [x] Remove branches that are dead from the list of connections.
* [x] Remove the components from the collected list of components.
* [ ] Remove statements that are dead code:
- [ ] Do not instantiate components.
- [ ] Do not compute expressions for unused variables.

0
monasca_analytics/banana/deadpathck/__init__.py
View File

118
monasca_analytics/banana/deadpathck/config.py
View File

@ -1,118 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import monasca_analytics.banana.deadpathck.dag as dag
import monasca_analytics.banana.emitter as emit
import monasca_analytics.banana.grammar.ast as ast
import monasca_analytics.banana.typeck.type_util as type_util
import monasca_analytics.exception.banana as exception
import six
def deadpathck(banana_file, type_table, emitter=emit.PrintEmitter()):
"""
Perform dead path elimination on the provided AST.
This allow to remove branches and components that
are not connected to a Sink.
:type banana_file: ast.BananaFile
:param banana_file: The AST tree we will clean.
:type type_table monasca_analytics.banana.typeck.type_table.TypeTable
:param type_table: The TypeTable of the provided AST.
:type emitter: emit.Emitter
:param emitter: Emitter for reporting warnings.
"""
# Check that first argument is a banana file. Mainly
# an excuse to remove the F401 warning.
if not isinstance(banana_file, ast.BananaFile):
raise Exception("Expected BananaFile as first argument.")
# Look first for all branch that are "dead"
connections = banana_file.connections # type: ast.Connection
# If there are no connections everything is considered
# as dead.
if connections is None:
class EmptyConnections(object):
connections = []
connections = EmptyConnections()
# Collect the nodes and connect them.
dag_nodes = {}
# Create all the nodes
for ident in banana_file.components.keys():
dag_nodes[ident] = dag.DagNode(type_table.get_type(ident))
# Connect them
for ident_from, ident_to in connections.connections:
dag_from = dag_nodes[ident_from]
dag_to = dag_nodes[ident_to]
dag_from.children.append(dag_to)
dag_to.parents.append(dag_from)
# Start from every sources and for each, check if the path is dead
for node in dag_nodes.values():
if isinstance(node.typec, type_util.Source):
node.visit()
# We can now remove all the components that are "dead"
# from the list of connections
for ident, node in six.iteritems(dag_nodes):
if not node.is_alive():
emitter.emit_warning(
ident.span,
"Dead code found, this component is not in a path "
"starting from a 'Source' and ending with a 'Sink'."
)
banana_file.components.pop(ident)
connections.connections = [edge for edge in connections.connections
if edge[0] != ident and
edge[1] != ident]
# TODO(Joan): We could also remove them from the statements.
# TODO(Joan): But for this we need a dependency graph between
# TODO(Joan): statements to make sure we don't break the code.
def contains_at_least_one_path_to_a_sink(banana_file, type_table):
"""
Check that there's at least one path to a sink in the list
of components.
To run this pass, you need to make sure you
have eliminated all dead path first.
:type banana_file: ast.BananaFile
:param banana_file: The AST to check.
:type type_table monasca_analytics.banana.typeck.type_table.TypeTable
:param type_table: The TypeTable of the provided AST.
:raise: Raise an exception if there's no Sink.
"""
def is_sink(comp):
type_comp = type_table.get_type(comp)
return isinstance(type_comp, type_util.Sink)
def is_src(comp):
type_comp = type_table.get_type(comp)
return isinstance(type_comp, type_util.Source)
comp_vars = banana_file.components.keys()
at_least_one_sink = len(list(filter(is_sink, comp_vars))) > 0
at_least_one_source = len(list(filter(is_src, comp_vars))) > 0
if not at_least_one_sink:
raise exception.BananaNoFullPath("Sink")
if not at_least_one_source:
raise exception.BananaNoFullPath("Source")

56
monasca_analytics/banana/deadpathck/dag.py
View File

@ -1,56 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import monasca_analytics.banana.typeck.type_util as type_util
class DagNode(object):
def __init__(self, typec):
"""
Create a DAGNode.
:param typec: The type of the node.
"""
self.parents = []
self.children = []
self.typec = typec
self._visited = False
self._seen_sink = False
def visit(self):
"""
Visit this nodes and all of its connections.
"""
if not self._visited:
self._visited = True
if isinstance(self.typec, type_util.Sink):
self.visit_parents()
return
for child in self.children:
child.visit()
def visit_parents(self):
"""
Visit the parent to tell them that we've seen a Sink.
"""
if not self._seen_sink:
self._seen_sink = True
for parent in self.parents:
parent.visit_parents()
def is_alive(self):
return self._visited and self._seen_sink

94
monasca_analytics/banana/emitter.py
View File

@ -1,94 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import six
@six.add_metaclass(abc.ABCMeta)
class Emitter(object):
"""
Emitter base class to emit errors and warnings.
Typically errors will be collected and then send
over the network as an http response but for tests
and debugging, a `PrintEmitter` can be used instead.
"""
@abc.abstractmethod
def emit_warning(self, span, message):
"""
Emit a warning.
:type span: monasca_analytics.banana.grammar.base_ast.Span
:param span: Span associated with the message.
:type message: str
:param message: message to emit with the warning level.
"""
pass
@abc.abstractmethod
def emit_error(self, span, message):
"""
Emit an error
:type span: monasca_analytics.banana.grammar.base_ast.Span
:param span: Span associated with the message.
:type message: str
:param message: message to emit with the error level.
"""
pass
class PrintEmitter(Emitter):
"""
Print warnings and errors to the console.
"""
def emit_warning(self, span, message):
print("WARNING at line:{}".format(span.get_lineno()))
print("WARNING: {}".format(message))
def emit_error(self, span, message):
print("ERROR at line:{}".format(span.get_lineno()))
print("ERROR: {}".format(message))
class JsonEmitter(Emitter):
"""
Print warnings and errors in a Json object.
"""
def __init__(self):
self.result = {
"errors": [],
"warnings": [],
}
def emit_error(self, span, message):
error = JsonEmitter._gen_message_structure(span, message)
self.result["errors"].append(error)
def emit_warning(self, span, message):
warning = JsonEmitter._gen_message_structure(span, message)
self.result["warnings"].append(warning)
@staticmethod
def _gen_message_structure(span, message):
spanrange = span.get_range()
return {
"startLineNumber": spanrange[0][0],
"startColumn": spanrange[0][1],
"endLineNumber": spanrange[1][0],
"endColumn": spanrange[1][1],
"byteRange": [span.lo, span.hi],
"message": message
}

50
monasca_analytics/banana/eval/README.md
View File

@ -1,50 +0,0 @@
## Interpreter / Evaluator
This folder contains everything related to the evaluation
of banana files.
This pass makes some assumptions: it is valid to create
all the components and connecting them won't throw any errors.
Some components might need to be created in order to
check if they are valid. For instance, when a DNS lookup is
involved. In such cases, an error will be thrown during
the interpretation. However, the general intention is to move
the checks out of the evaluation as much as possible. We want to
avoid at all cost an half-working pipeline as it could have
side-effects on external data sources by corrupting them or
feeding them with incorrect data.
The execution environment (e.g Spark) might also reject the
pipeline during an evaluation for some reason. However, this is
less likely to happen as the `deadpathck` pass removes
components and paths that would lead to errors.
This is the last step of the pipeline:
```
+---------------------+
| |
---> | AST & TypeTable | ---- interpret ---> Done
| |
+---------------------+
```
### Current status
* [x] Evaluate expressions
* [x] Create components
* [x] Connect components
* [x] Restart the pipeline
### Tests
All tests are located in `test/banana/eval`. We only try
to evaluate valid files, so for this pass there's only a
`should_pass` directory.
#### Available instruction
* `# LHS_EQ <string-version-of-the-value>`: This instruction
compares the evaluation of the left hand side of the previous
expression with the provided string. If they are not equal,
the test will fail.

0
monasca_analytics/banana/eval/__init__.py
View File

265
monasca_analytics/banana/eval/config.py
View File

@ -1,265 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import operator
import monasca_analytics.banana.eval.ctx as ctx
import monasca_analytics.banana.eval.old_style as into
import monasca_analytics.banana.grammar.ast as ast
import monasca_analytics.banana.typeck.type_util as type_util
import monasca_analytics.config.connection as connection
import monasca_analytics.config.const as conf_const
import monasca_analytics.exception.banana as exception
import monasca_analytics.util.common_util as introspect
import six
logger = logging.getLogger(__name__)
def eval_ast(ast_root, type_table, driver):
"""
Evaluate the provided AST by instantiating
the appropriate components and connecting them
together using the Driver interface.
:type ast_root: ast.BananaFile
:param ast_root: AST to evaluate.
:type type_table: monasca_analytics.banana.typeck.type_table.TypeTable
:param type_table: the TypeTable (used to create configurations)
:type driver: monasca_analytics.spark.driver.DriverExecutor
:param driver: Driver that will manage the created
components and connect them together.
"""
logger.debug("Creating the config dictionary from the AST...")
_config = conf_const.get_default_base_config()
try:
logger.debug("Creating components according to banana config...")
components = eval_create_components(ast_root.statements, type_table)
convert_connections(ast_root.connections, _config)
logger.debug("Done creating components. Creating link data...")
# Pre-process to convert to old style components
components_old_style = into.into_old_conf_dict(components)
links = connection.connect_components(components_old_style, _config)
logger.debug("Done connecting components. Successful instantiation")
except Exception as ex:
logger.error("Failed to instantiate components")
logger.error("Reason : " + str(ex))
return
# Restart Spark using the new config
logger.debug("Stop pipeline")
driver.stop_pipeline()
logger.debug("Set new links")
driver.set_links(links)
logger.debug("Start pipeline")
driver.start_pipeline()
def convert_connections(connections, output_config):
"""
Augment the output_config object with the list of
connections
:type connections: ast.Connection
:param connections: The list of connections.
:type output_config: dict
:param output_config: Config where the links will be written.
"""
output_config[conf_const.CONNECTIONS] = connections.connections_cache
def eval_create_components(statements, type_table):
"""
Convert the provided AST into the old dict configuration.
:type statements: list[(ast.ASTNode, ast.ASTNode)]
:param statements: The AST to process
:type type_table: monasca_analytics.banana.typeck.type_table.TypeTable
:param type_table: the type table.
:rtype: dict[str, Component]
:return: Returns the component keyed by name.
"""
context = ctx.EvaluationContext()
eval_statements_generic(
statements,
type_table,
context
)
return context.get_components()
def eval_statements_generic(
statements,
type_table,
context,
cb=lambda *a, **k: None):
"""
Eval the list of statements, and call the cb after evaluating
each statement providing it with the type of the value, the
left hand side ast node, and the computed value.
:type statements: list[(ast.ASTNode, ast.ASTNode)]
:param statements: The AST to process
:type type_table: monasca_analytics.banana.typeck.type_table.TypeTable
:param type_table: the type table.
:type context: ctx.EvaluationContext
:param context: evaluation context that will collect
all intermediary results.
:type cb: (type_util.IsType, ast.ASTNode, object) -> None
:param cb: Callback called after each statement evaluation.
"""
stmt_index = 0
for stmt in statements:
lhs, rhs = stmt
expected_type = type_table.get_type(lhs, stmt_index + 1)
stmt_index += 1
# Provide the expected type
value = eval_rhs(context, rhs, expected_type)
# Call the cb with the expected_type of the value
# The lhs node and the value
cb(expected_type, lhs, value)
# Store result if referenced later.
context.set_variable(lhs, value)
def eval_rhs(context, ast_node, expected_type):
"""
Eval the right hand side node.
:type context: ctx.EvaluationContext
:param context: Evaluation context.
:type ast_node: ast.ASTNode
:param ast_node: the node to evaluate.
:type expected_type: type_util.IsType
:param expected_type: The expected type of this computation.
:return: Returns the result of this evaluation.
"""
if isinstance(ast_node, ast.StringLit):
return ast_node.inner_val()
if isinstance(ast_node, ast.Ident):
return context.get_variable(ast_node.inner_val())
if isinstance(ast_node, ast.JsonObj):
return eval_object(context, ast_node, expected_type)
if isinstance(ast_node, ast.Number):
return ast_node.val
if isinstance(ast_node, ast.DotPath):
variable_name = ast_node.varname.inner_val()
prop = map(lambda x: x.inner_val(), ast_node.properties)
return context.get_prop_of_variable(variable_name, prop)
if isinstance(ast_node, ast.Expr):
return eval_expr(context, ast_node, expected_type)
if isinstance(ast_node, ast.Component):
return eval_comp(context, ast_node, expected_type)
raise Exception("Unhandled ast value type {}!!".format(ast_node))
def eval_comp(context, comp, expected_type):
"""
Instantiate the given component, computing
the required config.
:type context: ctx.EvaluationContext
:param context: Evaluation context.
:type comp: ast.Component
:param comp: the node to evaluate.
:type expected_type: type_util.IsType
:param expected_type: The expected type of this computation.
:return: Returns the instantiated component.
"""
arguments = {}
# Compute arguments
for arg in comp.args:
arg_name = ast.DotPath(arg.arg_name.span, arg.arg_name, [])
arg_value = eval_rhs(context, arg.value, expected_type[arg_name])
arguments[arg.arg_name.inner_val()] = arg_value
# Lookup component
component_type = introspect.get_class_by_name(comp.type_name.val)
# Get default config for the component
conf = component_type.get_default_config()
# Update modified params
for k, val in six.iteritems(arguments):
conf[k] = val
# Delay evaluation until we do the assign
return component_type, conf
def eval_object(context, obj, expected_type):
"""
Evaluate the provided object
:type context: ctx.EvaluationContext
:param context: Evaluation context.
:type obj: ast.JsonObj
:param obj: The expression to evaluate
:type expected_type: type_util.IsType
:param expected_type: The expected type of this computation.
:return: Returns the computed value
"""
result = expected_type.default_value()
for name, val in six.iteritems(obj.props):
subtype = expected_type[name]
ctx.set_property(result, name, eval_rhs(context, val, subtype))
return result
def eval_expr(context, expr, expected_type):
"""
Eval the provided expression
:type context: ctx.EvaluationContext
:param context: Evaluation context.
:type expr: ast.Expr
:param expr: The expression to evaluate
:type expected_type: type_util.IsType
:param expected_type: The expected type of this computation.
:rtype: str | float
:return: Returns the computed value
"""
if len(expr.expr_tree) == 1:
return eval_rhs(context, expr.expr_tree[0], expected_type)
if isinstance(expected_type, type_util.Number):
result = 0
cast_func = float
elif isinstance(expected_type, type_util.String):
result = ""
cast_func = str
else:
raise exception.BananaEvalBug(
"Expected type for an expression can only be "
"'TypeNumber' or 'TypeString', got '{}'".format(
str(expected_type))
)
current_operator = operator.add
for el in expr.expr_tree:
if isinstance(el, six.string_types) and el in ['+', '-', '*', '/']:
current_operator = get_op_func(el)
else:
value = eval_rhs(context, el, expected_type)
value = cast_func(value)
result = current_operator(result, value)
return result
def get_op_func(op_str):
if op_str == '+':
return operator.add
if op_str == '-':
return operator.sub
if op_str == '*':
return operator.mul
if op_str == '/':
return operator.div
raise exception.BananaEvalBug(
"Unknown operator '{}'".format(op_str)
)

119
monasca_analytics/banana/eval/ctx.py
View File

@ -1,119 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import monasca_analytics.banana.grammar.ast as ast
import monasca_analytics.exception.banana as exception
class EvaluationContext(object):
"""Evalutation context for an AST evaluation"""
def __init__(self):
"""
Construct an evaluation context using the type table
as based for the variable needed to be created.
"""
self._variables = {}
self._components = {}
def set_variable(self, name, value):
"""Set the variable value."""
if isinstance(value, tuple) and len(value) == 2:
comp_type, config = value
comp_name = name.varname.inner_val()
self._components[comp_name] = comp_type(
comp_name,
config
)
self._variables[comp_name] = config
elif not set_property(self._variables, name, value):
raise exception.BananaEvalBug(
"set_variable can only be used with DotPath or Ident."
)
def get_components(self):
return self._components
def get_variable(self, name):
"""Returns the variable value."""
return self._variables[name]
def get_prop_of_variable(self, name, prop):
"""Returns the sub property of the given variable name."""
variable = self._variables[name]
for el in prop:
variable = variable[el]
return variable
def set_property(root_object, name, value):
"""
Set the property name of the root_object to value.
:type root_object: dict
:param root_object: The root object
:type name: ast.DotPath | ast.Ident | ast.StringLit
:param name: Name of
:param value:
:return: Returns true if succeeded.
"""
if isinstance(name, ast.Ident) or isinstance(name, ast.StringLit):
root_object[name.inner_val()] = value
return True
elif isinstance(name, ast.DotPath):
_create_as_many_object_as_needed(root_object, name, value)
return True
return False
def _create_as_many_object_as_needed(root_object, dot_path, value):
"""
Create as many object as needed to be able to access the
nested property.
:type root_object: dict
:param root_object: The root object
:type dot_path: ast.DotPath
:param dot_path: Dot Path to use.
:type value: object
:param value: Any value to set.
"""
name = dot_path.varname.inner_val()
if len(dot_path.properties) == 0:
root_object[name] = value
return
if name in root_object:
variable = root_object[name]
else:
variable = {}
current_var = variable
last_index = len(dot_path.properties) - 1
for index, subpath in enumerate(dot_path.properties):
subpath_name = subpath.inner_val()
if index != last_index:
if subpath_name in current_var:
current_var = current_var[subpath_name]
else:
new_object = {}
current_var[subpath_name] = new_object
current_var = new_object
else:
current_var[subpath_name] = value
root_object[name] = variable

59
monasca_analytics/banana/eval/old_style.py
View File

@ -1,59 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import monasca_analytics.config.const as conf_const
import monasca_analytics.ingestor.base as ingestor
import monasca_analytics.ldp.base as ldp
import monasca_analytics.sink.base as sink
import monasca_analytics.sml.base as sml
import monasca_analytics.source.base as source
import monasca_analytics.voter.base as voter
import six
def into_old_conf_dict(components):
"""
Convert the provided dict of components
keyed by ids into a dict keyed by component
type. This is the data structure used to do
the validation of JSON configuration (the old format).
:type components: dict[str, object]
:param components: The dictionary of components.
:return: Returns the old conf dictionary.
"""
return {
conf_const.INGESTORS:
dict(filter(lambda x: isinstance(x[1], ingestor.BaseIngestor),
six.iteritems(components))),
conf_const.VOTERS:
dict(filter(lambda x: isinstance(x[1], voter.BaseVoter),
six.iteritems(components))),
conf_const.SINKS:
dict(filter(lambda x: isinstance(x[1], sink.BaseSink),
six.iteritems(components))),
conf_const.LDPS:
dict(filter(lambda x: isinstance(x[1], ldp.BaseLDP),
six.iteritems(components))),
conf_const.SOURCES:
dict(filter(lambda x: isinstance(x[1], source.BaseSource),
six.iteritems(components))),
conf_const.SMLS:
dict(filter(lambda x: isinstance(x[1], sml.BaseSML),
six.iteritems(components))),
}

63
monasca_analytics/banana/grammar/README.md
View File

@ -1,63 +0,0 @@
## Grammar
This folder is all about the definition of the `banana` grammar.
The grammar purpose is to convert the input, text, into an
abstract syntax tree (AST).
This is the first step of the pipeline:
```
+--------+ +---------+
| | | |
| Text | --- grammar ---> | AST | --->
| | | |
+--------+ +---------+
```
The module `ast.py` contains all the possible `ASTNode` which
itself is defined in `base_ast.py`.
### Current status
* [x] Parsing connections such as `a -> b`, `a -> [b, c]`,
`[a, b] -> [c, d]`
* [x] Parsing numbers
* [x] Parsing string literals
* [ ] Parsing booleans
* [x] Parsing assignments where the left hand side can be a property
or an identifier.
* [x] Parsing assignments where the right hand side is a number, a
string literal, a property or an identifier.
* [x] Parsing components arguments using a constructor-like syntax.
* [ ] Parsing ingestors generators (for JSON dialect)
* [ ] Parsing imports such as `from ldp.monasca import *`
* [ ] Parsing disconnections such as `a !-> b` *(requires imports)*
### Tests
All test regarding the grammar (i.e. the syntax and the way
the AST is built) is defined in `test/banana/grammar`.
This folder looks like this:
```
test/banana/grammar
├── should_fail
│   ├── ...
│   └── file.banana
├── should_pass
│   ├── ...
│   └── file.banana
└── test_config.py
```
The `test_config` generates one test for each file in the
`should_pass` and `should_fail` directories.
Test can assert various things using instructions below.
#### Available instruction
* `# RAISE <exception-name>`: Check that `exception-name` is raised.
* `# STMT_EQ <ast-of-statements>` Check the AST of statements.
* `# AST_EQ <full-ast>` Check the full AST.
* `# CONN_EQ <ast-of-connections>` Check the AST of connections.

0
monasca_analytics/banana/grammar/__init__.py
View File

704
monasca_analytics/banana/grammar/ast.py
View File

@ -1,704 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import monasca_analytics.banana.emitter as emit
import monasca_analytics.banana.grammar.base_ast as base
import monasca_analytics.exception.banana as exception
import monasca_analytics.util.string_util as strut
import pyparsing as p
import six
ASTNode = base.ASTNode
Span = base.Span
class BananaFile(object):
def __init__(self, emitter=emit.PrintEmitter()):
self.imports = []
self._emitter = emitter
# Components is a dict where keys
# are the name of the var and
# values are of type Component
self.components = dict()
# Statement are component
# creation or variable creation
self.statements = []
self.connections = None
def add_component_ctor(self, dot_path, component):
"""
Add a component if the component
does not already exists. If it does exists,
then this raises an Error.
:type dot_path: DotPath
:param dot_path: Name of the variable or property path
:type component: Component
:param component: AST part that will contains all properties
"""
if not isinstance(dot_path, DotPath):
raise exception.BananaAssignmentError(
dot_path.span, component.span)
if not len(dot_path.properties) == 0:
raise exception.BananaAssignmentError(
dot_path.span, component.span)
if dot_path.varname in self.components:
other_dot_path = filter(
lambda x: x == dot_path,
self.components.keys())[0]
no_effect_str = dot_path.span.str_from_to(component.span)
collision_str = other_dot_path.span.str_from_to(
self.components[other_dot_path].span)
self._emitter.emit_warning(
dot_path.span,
"Statement has no effect: '{}'".format(no_effect_str)
)
self._emitter.emit_warning(
other_dot_path.span,
"It collides with: '{}'".format(collision_str)
)
else:
self.components[dot_path.varname] = component
self.statements.append((dot_path, component))
def add_assignment(self, dot_path, ast_value):
"""
Add an assignment to a property or a variable.
We don't check at this point whether or not the variable
has collision with a component.
This will be done during the name resolution pass.
:type dot_path: DotPath
:param dot_path: Name of the variable or property
:type ast_value: Ident | JsonObj | StringLit | Number | DotPath
:param ast_value: Ast node this variable is assigned to.
"""
if not isinstance(dot_path, DotPath):
raise exception.BananaAssignmentError(
dot_path.span,
ast_value.span
)
self.statements.append((dot_path, ast_value))
def add_connections(self, connections):
"""
Add a new set of connections between components
This function performs the same checks as the one
performs when components are connected. It warns
on redundant connections.
:type connections: Connection
:param connections: AST node that contains the collected connections.
"""
if self.connections is None:
self.connections = connections
else:
self.connections.merge_and_reset_inputs_outputs(
connections,
self._emitter
)
def statements_to_str(self):
return "{ " + ', '.join(
['{} = {}'.format(x[0], x[1]) for x in self.statements]
) + ' }'
def __str__(self):
res = "BananaFile { "
res += 'components: { '
res += strut.dict_to_str(self.components)
res += ' }, '
res += 'statements: ' + self.statements_to_str()
res += 'connections: { '
res += str(self.connections)
res += " } }"
return res
def make_span(s, l, t):
def compute_hi(init_loc, tokens):
hi = init_loc
for tok in tokens:
if isinstance(tok, ASTNode):
hi = max(hi, tok.span.hi)
elif isinstance(tok, six.string_types):
hi += len(tok)
elif isinstance(tok, p.ParseResults):
hi = max(hi, compute_hi(init_loc, tok))
else:
raise exception.BananaGrammarBug(
"Couldn't create span for: {}".format(tok)
)
return hi
if len(t) > 0:
span_hi = compute_hi(l, t)
return Span(s, l, span_hi)
else:
return Span(s, l, 2)
class Number(ASTNode):
def __init__(self, span, val):
"""
Construct a Number ast node.
:type span: Span
:param span: Span for this number
:type val: str
:param val: Value for this number
"""
super(Number, self).__init__(span)
self.val = float(val)
def __str__(self):
return "Number< {} >".format(self.val)
def into_unmodified_str(self):
return str(self.val)
class StringLit(ASTNode):
def __init__(self, span, val):
"""
Construct a StringLit ast node.
:type span: Span
:param span: Span for this string
:type val: str
:param val: Value for this string
"""
super(StringLit, self).__init__(span)
self.val = val
def __hash__(self):
return hash(self.inner_val())
def __eq__(self, other):
return (isinstance(other, StringLit) or isinstance(other, Ident))\
and self.inner_val() == other.inner_val()
def __ne__(self, other):
return not self.__eq__(other)
def __str__(self):
return "StringLit< {} >".format(self.val)
def into_unmodified_str(self):
return self.val
def inner_val(self):
return self.val.strip()[1:-1]
class Ident(ASTNode):
def __init__(self, span, val):
"""
Construct an Ident ast node.
:type span: Span
:param span: Span for this identifier
:type val: str
:param val: Value of this identifier
"""
super(Ident, self).__init__(span)
self.val = val
def __hash__(self):
return hash(self.val)
def __eq__(self, other):
return (isinstance(other, StringLit) or isinstance(other, Ident))\
and self.val == other.inner_val()
def __ne__(self, other):
return not self.__eq__(other)
def __str__(self):
return "Ident< {} >".format(self.val)
def into_unmodified_str(self):
return self.val
def inner_val(self):
return self.val
class DotPath(ASTNode):
def __init__(self, span, varname, properties):
"""
:type span: Span
:param span: Span for this dotpath.
:type varname: Ident | StringLit
:param varname: Name of the variable being changed.
:type properties: list[StringLit | Ident]
:param properties: Properties being accessed.
"""
super(DotPath, self).__init__(span)
self.varname = varname
self.properties = properties
def next_dot_path(self):
"""
Assuming the properties length is more than zero.
This returns the dot path where the varname has been
dropped. So given 'a.b.c' the returned dot path will
be 'b.c'.
:rtype: DotPath
:return: Returns the next dot path.
"""
return DotPath(
self.span.new_with_lo(self.properties[0].span.lo),
self.properties[0],
self.properties[1:]
)
def into_unmodified_str(self):
arr = [self.varname.into_unmodified_str()]
arr.extend([x.into_unmodified_str() for x in self.properties])
return '.'.join(arr)
def __str__(self):
arr = [str(self.varname)]
arr.extend([str(x) for x in self.properties])
return 'DotPath< {} >'.format('.'.join(arr))
def __key(self):
return self.into_unmodified_str().replace('"', '')
def __eq__(self, other):
return self.__key() == other.__key()
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.__key())
class Expr(ASTNode):
def __init__(self, span, expr_tree):
"""
Construct an expression
:type span: Span
:param span: Span for the expression.
:type expr_tree: p.ParseResults
;:param expr_tree: The tree generated by pyparsing.infixNotation
"""
super(Expr, self).__init__(span)
# We don't use this tree at this point.
# During typecheck we will make sure
# that the expression can evaluate
# Finally during evaluation, we will evaluate
# the final result.
if isinstance(expr_tree, p.ParseResults):
expr_tree = expr_tree.asList()
if isinstance(expr_tree, list):
for i in range(0, len(expr_tree)):
if isinstance(expr_tree[i], list):
expr_tree[i] = Expr(span, expr_tree[i])
self.expr_tree = expr_tree
else:
self.expr_tree = [expr_tree]
def into_unmodified_str(self):
# TODO(Joan): reconstruct the original expression
return 'expression'
def __str__(self):
return "Expr< {} >".format(strut.array_to_str(self.expr_tree))
class JsonObj(ASTNode):
def __init__(self, span, tokens):
super(JsonObj, self).__init__(span)
self.props = {}
last_prop = None
if len(tokens) > 0:
for toks in tokens:
for tok in toks:
if isinstance(tok, DotPath):
if last_prop is None:
last_prop = tok
else:
self._set_new_prop(last_prop, tok)
last_prop = None
elif isinstance(tok, StringLit):
if last_prop is None:
last_prop = tok
else:
self._set_new_prop(last_prop, tok)
last_prop = None
elif isinstance(tok, list):
if last_prop is None:
raise p.ParseFatalException(
"Bug Found in JsonObj!"
)
self._set_new_prop(
last_prop,
JsonObj.dictify_array(tok)
)
last_prop = None
else:
if last_prop is None:
raise p.ParseFatalException(
"Bug Found in JsonObj!"
)
self._set_new_prop(last_prop, tok)
last_prop = None
def _set_new_prop(self, prop, token):
if prop in self.props:
raise exception.BananaJsonObjShadowingError(self.span, prop.span)
else:
self.props[prop] = token
def into_unmodified_str(self):
# TODO(Joan): improve this for error reporting
return str(self.props)
def __str__(self):
return "JsonObj< {} >".format(strut.dict_to_str(self.props))
@staticmethod
def dictify_array(tok):
new_array = []
for el in tok:
if isinstance(el, list):
new_array.append(JsonObj.dictify_array(el))
else:
new_array.append(el)
return new_array
def into_connection(ast_node):
"""
Convert an ast node into a Connection node.
:type ast_node: Connection | Ident
:param ast_node: The ast node to convert.
:rtype: Connection
:return: Returns a Connection node
"""
if isinstance(ast_node, Connection):
return ast_node
elif isinstance(ast_node, Ident):
return Connection(
ast_node.span,
[ast_node],
[ast_node]
)
else:
raise p.ParseFatalException("Bug found!")
class Connection(ASTNode):
def __init__(self, span, inputs=None, outputs=None, connections=None):
"""
Create a connection object.
:type span: Span
:param span: Span for this connection
:type inputs: list[Ident]
:param inputs: Input ast nodes of the connection
:type outputs: list[Ident]
:param outputs: Outputs nodes
:type connections: list[(Ident, Ident)]
:param connections: The list of connections aggregated so far.
"""
super(Connection, self).__init__(span)
if inputs is None:
inputs = []
if outputs is None:
outputs = []
if connections is None:
connections = []
self.inputs = inputs
self.outputs = outputs
self.connections = connections
self.connections_cache = {}
self._build_connection_cache()
def connect_to(self, other_con, emitter):
"""
Connect this connection to the other one.
After this function has been executed, the other_con
object can be dropped.
:type other_con: Connection
:param other_con: Other connection to connect to.
:type emitter: emit.Emitter
:param emitter: Emitter.
"""
self.span.hi = max(other_con.span.hi, self.span.hi)
self.span.lo = min(other_con.span.lo, self.span.lo)
old_outputs = self.outputs
self.outputs = other_con.outputs
# Generate new connections
for old_output in old_outputs:
for other_input in other_con.inputs:
self._check_and_connect(old_output, other_input, emitter)
# Preserve old connections
self._merge_connections(other_con, emitter)
def merge_all(self, tokens, emitter):
"""
Merge all the tokens with this class
:type tokens: list[list[Connection | Ident]]
:param tokens: List of list of tokens
:type emitter: emit.Emitter
:param emitter: Emitter to report errors
"""
if len(tokens) == 1:
if len(tokens[0]) > 0:
for tok in tokens[0]:
other_con = into_connection(tok)
self.merge_with(other_con, emitter)
def merge_and_reset_inputs_outputs(self, other_con, emitter):
"""
Merge this connection with other_con and reset inputs / outputs
as they're no longer necessary.
:type other_con: Connection
:param other_con: the other connection we are gonna merge with.
:type emitter: emit.Emitter
:param emitter: Emitter to report errors
"""
self.inputs = []
self.outputs = []
self._merge_connections(other_con, emitter)
def merge_with(self, other_con, emitter):
"""
Merge the provided connection with this one.
:type other_con: Connection
:param other_con: Connection to merge with self.
:type emitter: emit.Emitter
:param emitter: Emitter to report errors
"""
def extend(into, iterable, what):
for other_thing in iterable:
if len(list(filter(lambda x: x.val == other_thing.val,
into))) > 0:
emitter.emit_warning(
other_thing.span,
"{} {} already present".format(
what, other_thing.val
)
)
else:
into.append(other_thing)
extend(self.inputs, other_con.inputs, 'Input')
extend(self.outputs, other_con.outputs, 'Output')
self._merge_connections(other_con, emitter)
def _merge_connections(self, other_con, emitter):
"""
Merge only the connections field from other_con into self.
:type other_con: Connection
:param other_con: Connection to merge with self.
:type emitter: emit.Emitter
:param emitter: Emitter to report errors
"""
for ident_from, ident_to in other_con.connections:
self._check_and_connect(ident_from, ident_to, emitter)
def _check_and_connect(self, ident_from, ident_to, emitter):
"""
Check if the connection does not already exists and if it does not,
add it to the list of connections. Otherwise report a warning and
do nothing.
:type ident_from: Ident
:param ident_from: The 'From' node of the directed edge.
:type ident_to: Ident
:param ident_to: The 'To' node of the directed edge we are creating.
:type emitter: emit.Emitter
:param emitter: Emitter to report errors.
"""
if ident_from.val in self.connections_cache:
if ident_to.val in self.connections_cache[ident_from.val]:
emitter.emit_warning(
ident_to.span,
"Connection from '{}' to '{}'"
" is already present"
.format(ident_from.val, ident_to.val)
)
return
self.connections_cache[ident_from.val].append(ident_to.val)
else:
self.connections_cache[ident_from.val] = [ident_to.val]
self.connections.append((ident_from, ident_to))
def _build_connection_cache(self):
"""
Build a cache of connections keyed by where they start from.
"""
for ident_from, ident_to in self.connections:
if ident_from.val not in self.connections_cache:
self.connections_cache[ident_from.val] = []
if ident_to.val not in self.connections_cache:
self.connections_cache[ident_to.val] = []
self.connections_cache[ident_from.val].append(ident_to.val)
# Sanity check
for _, vals in self.connections_cache:
if len(set(vals)) != len(vals):
raise p.ParseFatalException("Bug found in Connection!!")
def into_unmodified_str(self):
# TODO(Joan): improve this
return "connection"
def __str__(self):
res = "Connection<"
res += " {} ".format([(str(x[0]), str(x[1]))
for x in self.connections])
res += ">"
return res
class Assignment(ASTNode):
def __init__(self, span, dot_path, value):
"""
Construct an assignment AST node.
:type span: Span
:param span: the span of the assignment.
:type dot_path: DotPath
:param dot_path: the left hand side of the assignment.
:type value: Component | Number | StringLit | JsonObj | DotPath | Expr
:param value: the right hand side of the assignment.
"""
super(Assignment, self).__init__(span)
if (isinstance(value, Component) or
isinstance(value, JsonObj) or
isinstance(value, Number) or
isinstance(value, StringLit) or
isinstance(value, Ident) or
isinstance(value, DotPath) or
isinstance(value, Expr)) and\
isinstance(dot_path, DotPath):
self.lhs = dot_path
self.rhs = value
else:
raise exception.BananaGrammarBug(
'Impossible assignment found with'
' left hand side: {} and'
' right hand side: {}'
.format(type(dot_path), type(value))
)
def into_unmodified_str(self):
return "{} = {}".format(self.lhs.into_unmodified_str(),
self.rhs.into_unmodified_str())
def __str__(self):
return "{} = {}".format(str(self.lhs), str(self.rhs))
class ComponentCtorArg(ASTNode):
def __init__(self, span, value, arg_name=None):
"""
Construct an argument for a component ctor
:type span: Span
:param span: Span of the argument.
:type value: Number | StringLit | JsonObj | DotPath | Expr
:param value: Value for the argument
:type arg_name: Ident
:param arg_name: Name of the argument
"""
super(ComponentCtorArg, self).__init__(span)
if (isinstance(value, JsonObj) or
isinstance(value, Number) or
isinstance(value, StringLit) or
isinstance(value, Ident) or
isinstance(value, DotPath) or
isinstance(value, Expr)) and (
isinstance(arg_name, Ident) or
arg_name is None):
self.arg_name = arg_name
self.value = value
else:
# This code should be unreachable.
# The grammar as defined should prevent us from
# seeing an arg value or a value of the incorrect type
raise exception.BananaGrammarBug(
'Impossible constructor argument found with'
' left hand side: {} and'
' right hand side: {}'
.format(type(arg_name), type(value))
)
def into_unmodified_str(self):
return "{} = {}".format(self.arg_name.into_unmodified_str(),
self.value.into_unmodified_str())
def __str__(self):
if self.arg_name is not None:
return "{} = {}".format(self.arg_name, self.value)
else:
return "{}".format(self.value)
class Component(ASTNode):
def __init__(self, span, type_name=None, args=None):
"""
Construct a component
:type span: Span
:param span: Span of this component
:type type_name: Ident
:param type_name: Name of this component
:type args: list[ComponentCtorArg]
:param args: List of arguments
"""
super(Component, self).__init__(span)
if args is None:
args = []
self.type_name = type_name
self.args = args
def set_ctor(self, type_name):
"""
Set the constructor name of that component
:type type_name: Ident
:param type_name: Name of that constructor
"""
self.type_name = type_name
def add_arg(self, arg):
"""
Add an argument to that component constructor.
:type arg: ComponentCtorArg
:param arg: Argument to add to that component.
"""
self.args.append(arg)
def into_unmodified_str(self):
return self.type_name.into_unmodified_str() + "(" + \
', '.join(map(lambda x: x.into_unmodified_str(), self.args)) +\
")"
def __str__(self):
res = ""
res += "Component {"
res += " type_name: {},".format(self.type_name)
arr = ', '.join(map(lambda x: str(x), self.args))
res += " args: {}".format("[" + arr + "]")
res += "}"
return res

169
monasca_analytics/banana/grammar/base_ast.py
View File

@ -1,169 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import six
@six.add_metaclass(abc.ABCMeta)
class ASTNode(object):
"""
Base class of all ast nodes
"""
def __init__(self, span):
"""
Construct an ASTNode
:type span: Span
:param span: span for this AST node.
"""
self.span = span
@abc.abstractmethod
def into_unmodified_str(self):
"""
Returns a simple name for this ASTNode. It should be minimalist
and user oriented. No span info, no debug info.
:rtype: str
:returns: A simple name for that ast node.
"""
pass
def __ne__(self, other):
return not self.__eq__(other)
class Span(object):
"""
Represent a region of code, used for error reporting.
Position are absolute within the file.
"""
def __init__(self, text, lo, hi):
"""
:type text: str | None
:param text: Full text of the file
:type lo: int
:param lo: position of the beginning of the region
:type hi: int
:param hi: position of the end of the region
"""
self._text = text
self.lo = lo
self.hi = hi
def __str__(self):
if self._text is not None:
return self._text[self.lo:self.hi]
else:
return '?SPAN?'
def new_with_lo(self, lo_val):
"""
Construct a new Span with an new value for
lo.
:type lo_val: int
:param lo_val: New value for lo.
:rtype: Span
:return: Returns a new span
"""
return Span(self._text, lo_val, self.hi)
def str_from_to(self, to_span):
"""
Returns a string that start at self and stops at to_span.
:type to_span: Span
:param to_span: Span to stop at.
:rtype: six.string_types
:return: Returns the string encapsulating both
"""
return self._text[self.lo:to_span.hi]
def get_line(self):
"""
Returns the line for associated with this span.
"""
if self._text is not None:
splitted = self._text.splitlines()
current_pos = 0
for line in splitted:
if current_pos < self.lo < len(line) + current_pos:
return line.strip()
else:
current_pos += len(line)
else:
return '?LINE?'
def get_range(self):
"""
Returns the start and end (line number, column number) of this span.
"""
if self._text is not None:
splitted = self._text.splitlines()
current_pos = 0
startlineno = 0
startcolno = 0
endlineno = 0
endcolno = 0
for lineno in range(0, len(splitted)):
line = splitted[lineno]
if current_pos <= self.lo <= len(line) + current_pos:
startlineno = lineno + 1
startcolno = self.lo - current_pos + 1
if current_pos <= self.hi <= len(line) + current_pos:
endlineno = lineno + 1
endcolno = self.hi - current_pos + 1
current_pos += len(line) + 1
return (startlineno, startcolno), (endlineno, endcolno)
else:
return (0, 0), (0, 0)
def get_lineno(self):
"""
Returns the line number of this span.
"""
if self._text is not None:
splitted = self._text.splitlines()
current_pos = 0
lineno = 0
for _ in range(0, len(splitted)):
line = splitted[lineno]
if current_pos < self.lo < len(line) + current_pos:
return lineno + 1
else:
current_pos += len(line)
lineno += 1
return lineno
else:
return -1
DUMMY_SPAN = Span(None, 0, 0)
def from_pyparsing_exception(parse_fatal_exception):
"""
Convert the provided ParseFatalException into a Span.
:type parse_fatal_exception: pyparsing.ParseBaseException
:param parse_fatal_exception: Exception to convert.
:rtype: Span
:return: Returns the span mapping to that fatal exception.
"""
return Span(
parse_fatal_exception.pstr,
parse_fatal_exception.loc,
parse_fatal_exception.loc + 1
)

285
monasca_analytics/banana/grammar/config.py
View File

@ -1,285 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import pyparsing as p
import monasca_analytics.banana.emitter as emit
import monasca_analytics.banana.grammar.ast as ast
import monasca_analytics.banana.grammar.const as const
import monasca_analytics.exception.banana as exception
# This file describe the grammar for the banana config file.
# It make use of one sub grammar for certain configuration
# that requires expressions (see expression.py file)
def banana_grammar(emitter=emit.PrintEmitter()):
"""
Generate a banana parser that can be then used to
parse a banana content. It build an AST on which
operation can then be applied.
:return: Return a banana parser
:rtype: BananaScopeParser
"""
# Should debug
debug_grammar = False
# Actions
def action_str_lit(s, l, t):
return ast.StringLit(ast.make_span(s, l, t), t[0])
def action_num_lit(s, l, t):
return ast.Number(ast.make_span(s, l, t), t[0])
def action_ident(s, l, t):
return ast.Ident(ast.make_span(s, l, t), t[0])
def action_expr(s, l, t):
if len(t) != 1:
raise exception.BananaGrammarBug(
'Bug found in the grammar for expression,'
' Please report this bug.'
)
if isinstance(t[0], ast.Expr):
return t[0]
return ast.Expr(ast.make_span(s, l, t), t[0])
def action_dot_path(s, l, t):
# First token is the name of the variable
# The rest is the property path
if isinstance(t[0], ast.StringLit) and len(t[1:]) == 0:
return t[0]
return ast.DotPath(ast.make_span(s, l, t), t[0], t[1:])
def action_json_obj(s, l, t):
return ast.JsonObj(ast.make_span(s, l, t), t)
def action_parse_ctor_arg(s, l, t):
if len(t) > 1:
return ast.ComponentCtorArg(ast.make_span(s, l, t), t[1], t[0])
else:
return ast.ComponentCtorArg(ast.make_span(s, l, t), t[0])
def action_parse_comp_ctor(s, l, tokens):
comp = ast.Component(ast.make_span(s, l, tokens))
for tok in tokens:
if isinstance(tok, ast.Ident):
comp.set_ctor(tok)
elif isinstance(tok, ast.ComponentCtorArg):
comp.add_arg(tok)
else:
raise exception.BananaGrammarBug(
'Bug found in the grammar, Please report this bug'
)
return comp
def action_assignment(s, l, t):
return ast.Assignment(ast.make_span(s, l, t), t[0], t[1])
def action_create_connections(s, l, t):
ast_conn = ast.into_connection(t[0])
ast_conn.span = ast.make_span(s, l, t)
for i in range(1, len(t)):
next_conn = ast.into_connection(t[i])
ast_conn.connect_to(next_conn, emitter)
return ast_conn
def action_merge_connections(s, l, t):
ast_conn = ast.Connection(ast.make_span(s, l, t))
ast_conn.merge_all(t, emitter)
return ast_conn
def action_root_ast(s, l, tokens):
root = ast.BananaFile(emitter)
for tok in tokens:
if isinstance(tok, ast.Assignment):
if isinstance(tok.rhs, ast.Component):
root.add_component_ctor(tok.lhs, tok.rhs)
else:
root.add_assignment(tok.lhs, tok.rhs)
elif isinstance(tok, ast.Connection):
root.add_connections(tok)
else:
raise exception.BananaGrammarBug(
'Bug found in the grammar, Please report this bug.'
)
return root
# TODO(Joan): Remove once it is no longer needed
def print_stmt(s, l, t):
print("\nPRINT AST")
print((l, [str(x) for x in t]))
print("END PRINT AST\n")
def action_unimplemented(s, l, t):
raise exception.BananaGrammarBug("unimplemented code reached")
# Tokens
equals = p.Literal("=").suppress().setName('"="').setDebug(debug_grammar)
arrow = p.Literal("->").suppress().setName('"->"').setDebug(debug_grammar)
lbra = p.Literal("[").suppress().setName('"["').setDebug(debug_grammar)
rbra = p.Literal("]").suppress().setName('"]"').setDebug(debug_grammar)
colon = p.Literal(":").suppress().setName('":"')
comma = p.Literal(",").suppress().setName(",")
less = p.Literal("<").suppress().setName('"<"')
greater = p.Literal(">").suppress().setName('">"')
lbrace = p.Literal("{").suppress().setName('"{"').setDebug(debug_grammar)
rbrace = p.Literal("}").suppress().setName('"}"').setDebug(debug_grammar)
lpar = p.Literal("(").suppress().setName('"("')
rpar = p.Literal(")").suppress().setName('")"')
# Keywords
ing = p.Literal("ing").suppress()
imp = p.Literal("import").suppress()
fro = p.Literal("from").suppress()
# String Literal, Numbers, Identifiers
string_lit = p.quotedString()\
.setParseAction(action_str_lit)\
.setName(const.STRING_LIT)
number_lit = p.Regex(r'\d+(\.\d*)?([eE]\d+)?')\
.setParseAction(action_num_lit)\
.setName(const.NUMBER)
ident = p.Word(p.alphas + "_", p.alphanums + "_")\
.setParseAction(action_ident)\
.setName(const.IDENT)
# Path for properties
dot_prop = ident | string_lit
dot_path = p.delimitedList(dot_prop, ".")\
.setParseAction(action_dot_path)\
.setName(const.DOT_PATH)\
.setDebug(debug_grammar)
# Expressions
# Here to simplify the logic, we can match directly
# against ident and string_lit to avoid having to deal
# only with dot_path. It also allow to remove the confusion
# where '"a"' could be interpreted as a dot_path and would thus
# be the same as 'a'. With the following, the first we
# always be type-checked as a String whereas the latter will
# be as the type of the variable.
expr = p.infixNotation(number_lit | dot_path, [
(p.oneOf('* /'), 2, p.opAssoc.LEFT),
(p.oneOf('+ -'), 2, p.opAssoc.LEFT),
], lpar=lpar, rpar=rpar)
expr.setParseAction(action_expr)\
.setName(const.EXPR)\
.setDebug(debug_grammar)
# Json-like object (value are much more)
json_obj = p.Forward()
json_value = p.Forward()
json_array = p.Group(
lbra + p.Optional(p.delimitedList(json_value)) + rbra
)
json_array.setDebug(debug_grammar)
json_array.setName(const.JSON_ARRAY)
json_value <<= expr | json_obj | json_array
json_value.setDebug(debug_grammar)\
.setName(const.JSON_VALUE)
json_members = p.delimitedList(p.Group(dot_path + colon - json_value)) +\
p.Optional(comma)
json_members.setDebug(debug_grammar)\
.setName(const.JSON_MEMBERS)
json_obj <<= p.Dict(lbrace + p.Optional(json_members) - rbrace)
json_obj.setParseAction(action_json_obj)\
.setName(const.JSON_OBJ)\
.setDebug(debug_grammar)
# Component constructor
arg = (ident + equals - (expr | json_obj)) | expr | json_obj
arg.setParseAction(action_parse_ctor_arg)
params = p.delimitedList(arg)
comp_ctor = ident + lpar - p.Optional(params) + rpar
comp_ctor.setParseAction(action_parse_comp_ctor)\
.setName(const.COMP_CTOR)\
.setDebug(debug_grammar)
# Assignments
assignment = dot_path + equals - (comp_ctor | expr | json_obj)
assignment.setParseAction(action_assignment)
# Connections
connection = p.Forward()
array_of_connection = p.Group(
lbra + p.Optional(p.delimitedList(connection)) + rbra
)
array_of_connection.setParseAction(action_merge_connections)
last_expr = ident | array_of_connection
this_expr = p.Forward()
match_expr = p.FollowedBy(last_expr + arrow - last_expr) + \
(last_expr + p.OneOrMore(arrow - last_expr))
this_expr <<= match_expr | last_expr
connection <<= this_expr
match_expr.setDebug(debug_grammar)\
.setName(const.CONNECTION) \
.setParseAction(action_create_connections)
# Definitions
definition = ing - less - string_lit - greater - ident - lbrace - rbrace
definition.setDebug(debug_grammar)\
.setName(const.DEFINITION)\
.setParseAction(action_unimplemented)
# Import directive
module_def = (imp - ident) | fro - ident - imp - ident
module_def.setDebug(debug_grammar)\
.setName(const.MOD_IMPORT)\
.setParseAction(action_unimplemented)
# Comments
comments = "#" + p.restOfLine
statement = assignment | \
match_expr | \
definition | \
module_def
statement.setName(const.STATEMENT)
statement.setDebug(debug_grammar)
statement.setParseAction(print_stmt)
# Grammar
grammar = p.OneOrMore(statement).ignore(comments)
grammar.setParseAction(action_root_ast)
return BananaScopeParser(grammar)
class BananaScopeParser(object):
"""
Aggregate and resolve conflicts as everything was define
within the same scope. Usefull for have cpp "include"-like
functionality when importing another file.
"""
def __init__(self, grammar):
self._grammar = grammar
def parse(self, string):
"""
Parse the given input string.
:type string: str
:param string: Input string.
:rtype: ast.BananaFile
:return: Returns the ast root.
"""
tree = self._grammar.parseString(string, parseAll=True)[0]
return tree

31
monasca_analytics/banana/grammar/const.py
View File

@ -1,31 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
ASSIGNMENT = "assignment"
CONNECTION = "connection"
DEFINITION = "definition"
MOD_IMPORT = "mod_import"
DOT_PATH = "path"
NUMBER = "number"
STRING_LIT = "string_lit"
IDENT = "ident"
EXPR = "expression"
JSON_OBJ = "jsonlike_obj"
STATEMENT = "statement"
JSON_ARRAY = "jsonlike_array"
JSON_VALUE = "jsonlike_value"
JSON_MEMBERS = "jsonlike_members"
COMP_CTOR = "component_ctor"

117
monasca_analytics/banana/pass_manager.py
View File

@ -1,117 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import pyparsing as p
import monasca_analytics.banana.deadpathck.config as deadpathck
import monasca_analytics.banana.emitter as emit
import monasca_analytics.banana.eval.config as ev
import monasca_analytics.banana.eval.ctx as ctx
import monasca_analytics.banana.grammar.base_ast as span_util
import monasca_analytics.banana.grammar.config as grammar
import monasca_analytics.banana.typeck.config as typeck
import monasca_analytics.exception.banana as exception
def execute_banana_string(banana, driver=None, emitter=emit.PrintEmitter()):
"""
Execute the provided banana string.
It will run the parse phase, and the typechecker.
:type banana: str
:param banana: The string to parse and type check.
:type driver: monasca_analytics.spark.driver.DriverExecutor | None
:param driver: Driver that will manage the created
components and connect them together.
:type emitter: emit.Emitter
:param emitter: Emitter for reporting errors/warning.
"""
try:
# Convert the grammar into an AST
parser = grammar.banana_grammar(emitter)
ast = parser.parse(banana)
# Compute the type table for the given AST
type_table = typeck.typeck(ast)
# Remove from the tree path that are "dead"
deadpathck.deadpathck(ast, type_table, emitter)
# Check that there's at least one path to be executed
deadpathck.contains_at_least_one_path_to_a_sink(ast, type_table)
# Evaluate the script
if driver is not None:
ev.eval_ast(ast, type_table, driver)
except exception.BananaException as err:
emitter.emit_error(err.get_span(), str(err))
except p.ParseSyntaxException as err:
emitter.emit_error(span_util.from_pyparsing_exception(err), err.msg)
except p.ParseFatalException as err:
emitter.emit_error(span_util.from_pyparsing_exception(err), err.msg)
except p.ParseException as err:
emitter.emit_error(span_util.from_pyparsing_exception(err), err.msg)
def try_compute_type_table(banana):
"""
Compute the type table for the provided banana string
if possible. Does not throw any exception if it fails.
:type banana: str
:param banana: The string to parse and type check.
"""
try:
# Convert the grammar into an AST
parser = grammar.banana_grammar()
ast = parser.parse(banana)
# Compute the type table for the given AST
return typeck.typeck(ast)
except exception.BananaException:
return None
except p.ParseSyntaxException:
return None
except p.ParseFatalException:
return None
except p.ParseException:
return None
def compute_type_table(banana):
"""
Compute the type table for the provided banana string
if possible.
:type banana: str
:param banana: The string to parse and type check.
"""
# Convert the grammar into an AST
parser = grammar.banana_grammar()
ast = parser.parse(banana)
# Compute the type table for the given AST
return typeck.typeck(ast)
def compute_evaluation_context(banana, cb=lambda *a, **k: None):
"""
Compute the evaluation context for the provided
banana string.
:type banana_str: str
:param banana_str: The string to parse and type check.
:param cb: Callback called after each statement
"""
parser = grammar.banana_grammar()
ast = parser.parse(banana)
type_table = typeck.typeck(ast)
context = ctx.EvaluationContext()
def custom_cb(_type, lhs, value):
cb(context, _type, lhs, value)
ev.eval_statements_generic(ast.statements, type_table, context, custom_cb)

85
monasca_analytics/banana/typeck/README.md
View File

@ -1,85 +0,0 @@
## Type-checker
This folder is all about the type checking of `banana` files.
The type checker purpose is to verify that components exist,
the type of local variable matches the requirements of components
parameters and assignments between them are correct. It also
checks that connections between components are valid.
The biggest difference between the old `validation` of the JSON
format is that we have more information available. We can warn
users when they make mistakes and point at the exact locations
using `Span`. Also, the type table generated is used by other passes
to perform other static analyses.
This is the second step of the pipeline:
```
+-------+ +---------------------+
| | | |
---> | AST | ---- typeck ---> | AST & TypeTable | --->
| | | |
+-------+ +---------------------+
```
The module `type_util.py` contains all the possible types that are
known by the type checker. The `TypeTable` built lives in the
`type_table.py` module.
### Current status
* [x] Type check numbers
* [x] Type check string literals
* [x] Type check variable assignments
* [x] Type check component assignments
* [x] Type check component parameters
* [x] Type check connections
* [x] Resolve variable names
* [ ] Resolve imports
* [ ] Type check disconnections
### Tests
All tests for the type checker (i.e. making sure that
inferred types are correct and that errors are raised in
appropriate situation) lives in `test/banana/typeck`.
This folder looks like this:
```
test/banana/typeck
├── should_fail
│   ├── ...
│   └── file.banana
├── should_pass
│   ├── ...
│   └── file.banana
└── test_typeck_config.py
```
The `test_typeck_config`, generates one test for each file
in the `should_pass` and `should_fail` directories.
For each generated test, we basically run the following passes:
* `grammar`: convert the input text into an AST.
* `typeck`: run the type checker.
Tests can assert various things in `banana` comments:
- In the `should_fail` directory, a test is expected to use
the `RAISE` instruction to specify a type of exceptions
that should be raised by the test.
- In the `should_pass` directory, a test is expected to not
raised any exception **and** to specify the state of the
`TypeTable` when the type checker has type checked everything
in the file. This is done with the `TYPE_TABLE_EQ` instruction.
#### Available instruction
* `# RAISE <exception-name>`: Check that `exception-name` is raised.
* `# TYPE_TABLE_EQ <string-version-of-the-type-table>`
* `# NEW_TEST`: This instruction splits the file into two tests. However,
the expected exception or type table should still be the same. It
allows us to verify what should be semantically equivalent in the
grammar.

0
monasca_analytics/banana/typeck/__init__.py
View File

285
monasca_analytics/banana/typeck/config.py
View File

@ -1,285 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import monasca_analytics.ingestor.base as ingestor
import monasca_analytics.ldp.base as ldp
import monasca_analytics.sink.base as sink
import monasca_analytics.sml.base as sml
import monasca_analytics.source.base as source
import monasca_analytics.voter.base as voter
import monasca_analytics.banana.grammar.ast as ast
import monasca_analytics.banana.typeck.connections as conn
import monasca_analytics.banana.typeck.type_table as typetbl
import monasca_analytics.banana.typeck.type_util as u
import monasca_analytics.exception.banana as exception
import monasca_analytics.exception.monanas as exception_monanas
import monasca_analytics.util.common_util as introspect
import six
def typeck(banana_file):
"""
Type-check the provided BananaFile instance.
If it type check, it returns the associated TypeTable.
:type banana_file: ast.BananaFile
:param banana_file: The file to typecheck.
:rtype: typetbl.TypeTable
:return: Returns the TypeTable for this BananaFile
"""
type_table = typetbl.TypeTable()
statement_index = 0
for stmt in banana_file.statements:
lhs, rhs = stmt
type_computed = typeck_rhs(rhs, type_table)
type_table.set_type(lhs, type_computed, statement_index)
statement_index += 1
conn.typeck_connections(banana_file.connections, type_table)
return type_table
def typeck_rhs(ast_value, type_table):
"""
Type-check the provided ast value. And returns its type.
This function does not support assignment,
:type ast_value: ast.ASTNode
:param ast_value: The ast_value to type check.
:type type_table: typetbl.TypeTable
:param type_table: The type table. Used for type lookup.
:rtype: u.Component | u.Object | u.Number | u.String
:return: Returns the computed type.
"""
if isinstance(ast_value, ast.Number):
return u.Number()
if isinstance(ast_value, ast.StringLit):
return u.String()
if isinstance(ast_value, ast.Ident):
return type_table.get_type(ast_value)
if isinstance(ast_value, ast.DotPath):
return type_table.get_type(ast_value)
if isinstance(ast_value, ast.Expr):
return typeck_expr(ast_value, type_table)
if isinstance(ast_value, ast.JsonObj):
return typeck_jsonobj(ast_value, type_table)
if isinstance(ast_value, ast.Component):
return typeck_component(ast_value, type_table)
raise Exception("Unhandled ast value type {}!!".format(ast_value))
def typeck_jsonobj(json_obj, type_table):
"""
Type-check a json-like object. If it succeeds
it return the appropriate type describing this
json like object. Raise an exception otherwise.
:type json_obj: ast.JsonObj
:param json_obj: The JsonObj ast node.
:type type_table: typetbl.TypeTable
:param type_table: The type table.
:rtype: u.Object
:return: Returns an instance of util.Object describing
the full type of this json object.
"""
root_type = u.Object(strict_checking=False)
for k, v in six.iteritems(json_obj.props):
sub_type = u.create_object_tree(k, typeck_rhs(v, type_table))
u.attach_to_root(root_type, sub_type, json_obj.span)
return root_type
def typeck_expr(expr, type_table):
"""
Type-check the given expression. If the typecheck
pass, the resulting type will be used for the strategy
to use when evaluating this expression.
:type expr: ast.Expr
:param expr: The expression to typecheck.
:type type_table: typetbl.TypeTable
:param type_table: Type of the table
:rtype: u.Number | u.String
:return: Returns the type of the expression if possible
:raise: Raise an exception
"""
# In the case where we are just wrapping around
# only one expression, the logic below
# needs to be skipped.
if len(expr.expr_tree) == 1:
return typeck_rhs(expr.expr_tree[0], type_table)
_type = None
must_be_number = False
def check_type(old_type, new_type):
if new_type == old_type:
return old_type
elif new_type == u.String():
if must_be_number:
raise exception.BananaTypeError(
expected_type=u.Number,
found_type=new_type
)
if old_type is None:
return new_type
elif u.can_to_str(old_type):
return new_type
else:
raise exception.BananaTypeError(
expected_type=old_type,
found_type=new_type
)
elif new_type == u.Number():
if old_type is None:
return new_type
elif old_type == u.String():
return old_type
elif not old_type == u.Number():
raise exception.BananaTypeError(
expected_type=old_type,
found_type=new_type
)
else:
raise exception.BananaTypeError(
expected_type=old_type,
found_type=new_type
)
def allowed_symbol(current_type):
if current_type == u.String():
return ['+']
else:
return ['+', '-', '*', '/']
for el in expr.expr_tree:
if isinstance(el, ast.StringLit):
_type = check_type(_type, u.String())
elif isinstance(el, ast.Number):
_type = check_type(_type, u.Number())
elif isinstance(el, ast.Ident):
ident_type = type_table.get_type(el)
_type = check_type(_type, ident_type)
elif isinstance(el, ast.DotPath):
dotpath_type = type_table.get_type(el)
_type = check_type(_type, dotpath_type)
elif isinstance(el, ast.Expr):
_type = check_type(_type, typeck_expr(el, type_table))
elif isinstance(el, six.string_types):
if el not in allowed_symbol(_type):
raise exception.BananaUnknownOperator(expr.span, el, _type)
if el in ['-', '*', '/']:
must_be_number = True
else:
raise exception.BananaTypeError(
expected_type=[u.Number.__name__, u.String.__name__,
u.Object.__name__],
)
# The final type if we made until here!
return _type
def typeck_component(component, type_table):
"""
Type-check the provided component. Returns
the appropriate subclass of util.Component if
successful, or raise an exception if there's
an error.
:type component: ast.Component
:param component: The component ast node.
:type type_table: typetbl.TypeTable
:param type_table: the type table.
:rtype: u.Source | u.Sink | u.Voter | u.Ldp | u.Sml | u.Ingestor
:return: Returns the appropriate type for the component.
"""
# TODO(Joan): This wont't work for type that are defined
# TODO(Joan): at the language level. We need a registration service
# TODO(Joan): to manage the Types of component that we can create
# TODO(Joan): instead of this hacky function call.
try:
component_type = introspect.get_class_by_name(component.type_name.val)
comp_params = component_type.get_params()
except exception_monanas.MonanasNoSuchClassError:
raise exception.BananaUnknown(
component
)
# Compute the type of the component
if issubclass(component_type, source.BaseSource):
comp_type = u.Source(component_type.__name__, comp_params)
elif issubclass(component_type, sink.BaseSink):
comp_type = u.Sink(component_type.__name__, comp_params)
elif issubclass(component_type, sml.BaseSML):
comp_type = u.Sml(component_type.__name__, comp_params)
elif issubclass(component_type, voter.BaseVoter):
comp_type = u.Voter(component_type.__name__, comp_params)
elif issubclass(component_type, ldp.BaseLDP):
comp_type = u.Ldp(component_type.__name__, comp_params)
elif issubclass(component_type, ingestor.BaseIngestor):
comp_type = u.Ingestor(component_type.__name__, comp_params)
else:
raise exception.BananaTypeCheckerBug("Couldn't find a type for '{}'"
.format(component.type_name.val))
# Type check the parameters
if len(component.args) > len(comp_params):
raise exception.BananaComponentTooManyParams(component.span)
# Does saying that parameter should either all have a name
# or non at all satisfying? -> Yes
# Are parameter all named?
all_named = -1
for arg in component.args:
if arg.arg_name is not None:
if all_named == 0:
raise exception.BananaComponentMixingParams(arg.span, False)
all_named = 1
else:
if all_named == 1:
raise exception.BananaComponentMixingParams(arg.span, True)
all_named = 0
if all_named == 1:
for arg in component.args:
param = list(filter(lambda x:
x.param_name == arg.arg_name.inner_val(),
comp_params))
if len(param) != 1:
raise exception.BananaComponentIncorrectParamName(
component=component.type_name,
found=arg.arg_name
)
param = param[0]
expr_type = typeck_rhs(arg.value, type_table)
if not u.can_be_cast_to(expr_type, param.param_type):
raise exception.BananaArgumentTypeError(
where=arg,
expected_type=param.param_type,
received_type=expr_type
)
else:
for arg, param in zip(component.args, comp_params):
arg.arg_name = ast.Ident(arg.span, param.param_name)
expr_type = typeck_rhs(arg.value, type_table)
if not u.can_be_cast_to(expr_type, param.param_type):
raise exception.BananaArgumentTypeError(
where=arg,
expected_type=param.param_type,
received_type=expr_type
)
return comp_type

60
monasca_analytics/banana/typeck/connections.py
View File

@ -1,60 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import monasca_analytics.banana.typeck.type_util as util
import monasca_analytics.exception.banana as exception
valid_connections_types = {
util.Source: [util.Ingestor, util.Ldp],
util.Ingestor: [util.Sml, util.Sink],
util.Sml: [util.Voter, util.Sink],
util.Voter: [util.Ldp, util.Sink],
util.Ldp: [util.Sink],
util.Sink: []
}
def typeck_connections(connection, type_table):
"""
Once all variable have been type-checked, we can
try to type-check connections.
:type connection: monasca_analytics.banana.grammar.ast.Connection
:param connection: The connection to type-check
:type type_table: monasca_analytics.banana.typeck.type_table.TypeTable
:param type_table: The table with all variable already type-checked.
:raise Raise an exception if there's a type error in connections.
"""
if connection is not None:
for ident_from, ident_to in connection.connections:
type_from = type_table.get_type(ident_from)
type_to = type_table.get_type(ident_to)
if not util.is_comp(type_from):
raise exception.BananaTypeError(
expected_type=util.Component(),
found_type=type_from
)
if not util.is_comp(type_to):
raise exception.BananaTypeError(
expected_type=util.Component(),
found_type=type_to
)
if type(type_to) not in valid_connections_types[type(type_from)]:
possible_types = map(lambda x: x.__name__,
valid_connections_types[type(type_from)])
raise exception.BananaConnectionError(
connection.span,
ident_from, ident_to, type_from, possible_types
)

215
monasca_analytics/banana/typeck/type_table.py
View File

@ -1,215 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import copy
import monasca_analytics.banana.grammar.ast as ast
import monasca_analytics.banana.typeck.type_util as util
import monasca_analytics.exception.banana as exception
import monasca_analytics.util.string_util as strut
import six
class TypeTable(object):
"""
Type table. Support lookup for JsonLike object.
Json-like object have properties that needs to be
type-checked. The TypeTable allows to store
that information as well. All type values are
rooted by their variable name.
Every-time a variable type is erased, we create a new
snapshot of the variables types. This allow to have
variable where the type change as the statement are
being executed.
"""
def __init__(self):
self._variables_snapshots = [(0, {})]
self._variables = self._variables_snapshots[0][1]
def get_type(self, var, statement_index=None):
variables = self.get_variables(statement_index)
if isinstance(var, ast.Ident):
if var in variables:
return variables[var]
else:
raise exception.BananaUnknown(var)
# If we encounter a dot path:
if isinstance(var, ast.DotPath):
if var.varname in variables:
if len(var.properties) > 0:
return variables[var.varname][var.next_dot_path()]
else:
return variables[var.varname]
else:
raise exception.BananaUnknown(var.varname)
raise exception.BananaTypeCheckerBug("Unkown type for {}".format(var))
def set_type(self, var, _type, statement_index):
"""
Set the type for the given var to _type.
:type var: ast.Ident | ast.DotPath
:param var: The var to set a type.
:type _type: util.Object | util.Component | util.String | util.Number
:param _type: The type for the var.
:type statement_index: int
:param statement_index: The statement at which this assignment was
made.
"""
if _type is None:
raise exception.BananaTypeCheckerBug(
"'None' is not a valid banana type"
)
if isinstance(var, ast.Ident):
self._check_needs_for_snapshot(var, _type, statement_index)
self._variables[var] = _type
return
if isinstance(var, ast.DotPath):
if util.is_comp(_type) and len(var.properties) > 0:
raise exception.BananaAssignCompError(var.span)
if len(var.properties) == 0:
self._check_needs_for_snapshot(
var.varname,
_type,
statement_index
)
self._variables[var.varname] = _type
else:
if var.varname in self._variables:
var_type = self._variables[var.varname]
if isinstance(var_type, util.Object):
new_type = util.create_object_tree(
var.next_dot_path(), _type)
util.attach_to_root(var_type, new_type, var.span,
erase_existing=True)
elif isinstance(var_type, util.Component):
var_type[var.next_dot_path()] = _type
else:
raise exception.BananaTypeError(
expected_type=util.Object,
found_type=type(var)
)
# Var undeclared, declare its own type
else:
new_type = util.create_object_tree(var.next_dot_path(),
_type)
self._variables[var.varname] = new_type
return
raise exception.BananaTypeCheckerBug("Unreachable code reached.")
def get_variables(self, statement_index=None):
"""
Returns the list of variables with their associated type.
:type statement_index: int
:param: Statement index.
:rtype: dict[str, util.Object|util.Component|util.String|util.Number]
"""
if statement_index is None:
return self._variables
variables = {}
for created_at, snap in self._variables_snapshots:
if created_at < statement_index:
variables = snap
else:
break
return variables
def get_variables_snapshots(self):
return self._variables_snapshots
def _check_needs_for_snapshot(self, var, _type, statement_index):
if var in self._variables:
# If we shadow a component, we need to raise an error
if util.is_comp(self._variables[var]):
raise exception.BananaShadowingComponentError(
where=var.span,
comp=self._variables[var].class_name
)
# If we change the type of the variable, we create a new snapshot:
# This is very strict but will allow to know exactly how
# the type of a variable (or a property) changed.
if self._variables[var] != _type:
self._create_snapshot(statement_index)
def _create_snapshot(self, statement_index):
"""
Create a new snapshot of the variables.
:type statement_index: int
:param statement_index: index of the statement
(should be strictly positive)
"""
new_snapshot = copy.deepcopy(self._variables)
self._variables_snapshots.append((
statement_index, new_snapshot
))
self._variables = new_snapshot
def to_json(self):
"""
Convert this type table into a dictionary.
Useful to serialize the type table.
:rtype: dict
:return: Returns this type table as a dict.
"""
res = {}
for key, val in six.iteritems(self._variables):
res[key.inner_val()] = val.to_json()
return res
def __contains__(self, key):
"""
Test if the type table contains or not the provided
path. This function is more permissive than the other two.
It will never raise any exception (or should aim not to).
:type key: six.string_types | ast.Ident | ast.DothPath
:param key: The key to test.
:return: Returns True if the TypeTable contains a type for the
given path or identifier.
"""
if isinstance(key, six.string_types):
return key in self._variables
if isinstance(key, ast.Ident):
return key.val in self._variables
if isinstance(key, ast.DotPath):
res = key.varname in self._variables
if not res:
return False
val = self._variables[key.varname]
for prop in key.properties:
if isinstance(val, util.Object):
if prop in val.props:
val = val[prop]
else:
return False
return True
return False
def __str__(self):
return strut.dict_to_str(self._variables)

626
monasca_analytics/banana/typeck/type_util.py
View File

@ -1,626 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
Util files to manipulates banana types.
The list of possible types is as follow:
* `Number`
* `Boolean`
* `String`
* `Object` (Json-like object)
* `Component.Source.<class-name>`
* `Component.Ingestor.<class-name>`
* `Component.Sink.<class-name>`
* `Component.Voter.<class-name>`
* `Component.Ldp.<class-name>`
* `Component.Sml.<class-name>`
where <class-name> will be the component class name defined
in the code base.
For type defined in banana such as Json parsers, <class-name>
refers the name they are defined with.
"""
import abc
import six
import monasca_analytics.banana.grammar.ast as ast
import monasca_analytics.exception.banana as exception
import monasca_analytics.util.string_util as strut
@six.add_metaclass(abc.ABCMeta)
class IsType(object):
"""
Any class that represents a Banana type should inherit
from this class.
"""
def __ne__(self, other):
# Dispatch to eq function
return not self.__eq__(other)
@abc.abstractmethod
def default_value(self):
pass
@abc.abstractmethod
def to_json(self):
pass
class Any(IsType):
"""
Any type. This type should be used by component's writer when
they have a complex handling of parameters. This is not
recommended though as it move the error handling to
the component writer.
"""
def __str__(self):
return "TypeAny"
def __eq__(self, _):
# Any type is equal to nothing not even itself.
return False
def __ne__(self, other):
return not self.__eq__(other)
def __getitem__(self, _):
return Any()
def __hash__(self):
raise Exception("Any type should not be used in dictionaries.")
def default_value(self):
return {}
def to_json(self):
return {"id": "any"}
class String(IsType):
"""
String Type.
"""
def __str__(self):
return "TypeString"
def __eq__(self, other):
return isinstance(other, String)
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(str(self))
def default_value(self):
return ""
def to_json(self):
return {"id": "string"}
class Number(String):
"""
Number type. Banana has only floating point value.
"""
def __str__(self):
return "TypeNumber"
def __eq__(self, other):
return isinstance(other, Number)
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(str(self))
def default_value(self):
return 0
def to_json(self):
return {"id": "number"}
class Enum(String):
"""
Enum type. This type is a way to constraint a string or number,
to a specific set of values.
"""
def __init__(self, variants):
self.variants = variants
def __eq__(self, other):
return isinstance(other, Enum) and self.variants == other.variants
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.variants)
def __str__(self):
return "TypeEnum < {} >".format(','.join(self.variants))
def default_value(self):
return ""
def to_json(self):
return {
"id": "enum",
"variants": self.variants
}
def attach_to_root(root_obj, obj1, span, erase_existing=False):
"""
Attach the object obj1 to the root_obj object type.
:type root_obj: Object
:param root_obj: The root object
:type obj1: Object
:param obj1: The object to attach.
:type span: Span
:param span: The span for this change.
:type erase_existing: bool
:param erase_existing: Set to true if the root type should
always be erased.
"""
for key, child_type in six.iteritems(obj1.props):
if key in root_obj.props:
root_sub_type = root_obj.props[key]
# Both are object -> recurse
if isinstance(root_sub_type, Object) and\
isinstance(child_type, Object):
attach_to_root(root_sub_type, child_type, span, erase_existing)
elif erase_existing:
root_obj.props[key] = child_type
else:
raise exception.BananaTypeError(
expected_type=root_sub_type,
found_type=child_type,
span=span
)
else:
# We can simply attach the new type!
root_obj.props[key] = child_type
def create_object_tree(dot_path, value):
"""
Create a linear tree of object type from the dot_path.
Also work when dot_path is an Ident or StringLit.
:type dot_path: ast.DotPath | ast.Ident | ast.StringLit
:param dot_path: The ast node that forms a linear tree of type.
:type value: Object | String | Number
:param value: the value to set at the end of the linear tree.
:rtype: Object
:return: Returns the created object
"""
if is_comp(value):
raise exception.BananaAssignCompError(dot_path.span)
# {a.b.c: value}
root_object = Object(strict_checking=False)
if isinstance(dot_path, ast.DotPath):
# {a: value}
if len(dot_path.properties) == 0:
root_object.props[dot_path.varname.inner_val()] = value
else:
# {a: <Object>}
root_object.props[dot_path.varname.inner_val()] = \
Object(strict_checking=False)
# {b.c: value}
current_obj = root_object.props[dot_path.varname.inner_val()]
last_index = len(dot_path.properties) - 1
for index, sub_prop in enumerate(dot_path.properties):
sub_prop_name = sub_prop.inner_val()
if index != last_index:
current_obj.props[sub_prop_name] = \
Object(strict_checking=False)
current_obj = current_obj.props[sub_prop_name]
else:
current_obj.props[sub_prop_name] = value
else:
# Ident and StringLit are captured here.
root_object.props[dot_path.inner_val()] = value
return root_object
class Object(String):
"""
Object Type. The value that are dictionary-like have this type.
"""
def __init__(self, props=None, strict_checking=True):
if props is None:
props = {}
self.props = props
# Strict checking is off for all objects defined within the banana
# language. It is on by default for components so that they can
# force the type checker to throw errors when we try to access
# or to modify unknown properties
self.strict_checking = strict_checking
def __getitem__(self, key):
# a.b or a."b"
if isinstance(key, ast.Ident) or isinstance(key, ast.StringLit):
if key.inner_val() not in self.props:
raise exception.BananaPropertyDoesNotExists(key,
on_type=self)
return self.props[key.inner_val()]
# a.b.c
if isinstance(key, ast.DotPath):
if key.varname.inner_val() not in self.props:
raise exception.BananaPropertyDoesNotExists(key.varname,
on_type=self)
sub_object = self.props[key.varname.inner_val()]
if len(key.properties) == 0:
return sub_object
# Recurse
if isinstance(sub_object, Object):
return sub_object[key.next_dot_path()]
if isinstance(sub_object, Any):
return sub_object
raise exception.BananaPropertyDoesNotExists(key.next_dot_path(),
on_type=sub_object)
raise exception.BananaTypeCheckerBug(
"Unreachable code in Object.__getitem__ reached."
)
def __str__(self):
if self.strict_checking:
return "TypeStruct < {} >".format(strut.dict_to_str(self.props))
else:
return "TypeObject < {} >".format(strut.dict_to_str(self.props))
def __eq__(self, other):
return self.props == other
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.props)
def default_value(self):
default_value = {}
for key, val in six.iteritems(self.props):
default_value[key] = val.default_value()
return default_value
def to_json(self):
res = {"id": "object", "props": {}}
for key, val in six.iteritems(self.props):
res["props"][key] = val.to_json()
return res
class Component(IsType):
"""
Type of all components. While not strictly used directly, it
is very useful to performs checks on variable that are supposed
to be any of the available components.
"""
def __init__(self, ctor_properties=None, class_name=None):
"""
Component type
:type ctor_properties:
list[monasca_analytics.component.params.ParamDescriptor]
:param ctor_properties:
:type class_name: str
:param class_name: Name of the class if there's any.
"""
self.ctor_properties = ctor_properties
self.class_name = class_name
def __str__(self):
if self.class_name is None:
return "TypeComponent"
else:
return self.class_name + "(" +\
",".join(map(lambda x: x.param_name + "=" + str(x.param_type),
self.ctor_properties))\
+ ")"
def __setitem__(self, dot_path, value):
"""
Attempt to set the value at 'dot_path' to 'value'.
:type dot_path: ast.DotPath
:param dot_path: The path of the property
:type value: String | Enum | Object | Number
:param value: The new type to set.
"""
if self.ctor_properties is None:
raise exception.BananaTypeCheckerBug(
"Component type can't have properties"
)
if len(dot_path.properties) == 0:
for arg in self.ctor_properties:
if arg.param_name == dot_path.varname.inner_val():
if not can_be_cast_to(value, arg.param_type):
raise exception.BananaArgumentTypeError(
expected_type=arg.param_type,
received_type=value,
where=dot_path.span
)
else:
return
else:
for arg in self.ctor_properties:
if arg.param_name == dot_path.varname.inner_val():
if isinstance(arg.param_type, Any):
return
elif isinstance(arg.param_type, Object):
next_dot_path = dot_path.next_dot_path()
sub_arg_type = arg.param_type[next_dot_path]
if not can_be_cast_to(value, sub_arg_type):
raise exception.BananaArgumentTypeError(
expected_type=sub_arg_type,
received_type=value,
where=next_dot_path.span
)
else:
return
else:
raise exception.BananaPropertyDoesNotExists(
dot_path.next_dot_path(),
arg.param_type
)
raise exception.BananaPropertyDoesNotExists(dot_path, on_type=self)
def __getitem__(self, dot_path):
"""
Return the type of the given item.
:type dot_path: ast.DotPath
:param dot_path: The path to follow
:return:
"""
if self.ctor_properties is None:
raise exception.BananaTypeCheckerBug(
"Component type can't have properties"
)
if len(dot_path.properties) == 0:
for arg in self.ctor_properties:
if arg.param_name == dot_path.varname.inner_val():
return arg.param_type
else:
for arg in self.ctor_properties:
if arg.param_name == dot_path.varname.inner_val():
if isinstance(arg.param_type, Object):
return arg.param_type[dot_path.next_dot_path()]
else:
raise exception.BananaPropertyDoesNotExists(
dot_path.next_dot_path(),
arg.param_type
)
raise exception.BananaPropertyDoesNotExists(dot_path, on_type=self)
def __eq__(self, other):
return isinstance(other, Component)
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(str(self))
def default_value(self):
return None
def to_json(self):
res = {"id": "component", "name": self.class_name, "args": []}
for arg in self.ctor_properties:
res["args"].append(arg.to_json())
return res
class Source(Component):
"""
Source type. All component that inherits from BaseSource have
this type in Banana.
"""
def __init__(self, class_name, ctor_properties):
super(Source, self).__init__(ctor_properties, class_name)
def __eq__(self, other):
return self.class_name == other.class_name
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.class_name)
class Ingestor(Component):
"""
Ingestor type. All component that inherits from BaseIngestor have
this type in Banana.
"""
def __init__(self, class_name, ctor_properties):
super(Ingestor, self).__init__(ctor_properties, class_name)
def __eq__(self, other):
return self.class_name == other.class_name
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.class_name)
class Sink(Component):
"""
Sink type. All component that inherits from BaseSink have
this type in Banana.
"""
def __init__(self, class_name, ctor_properties):
super(Sink, self).__init__(ctor_properties, class_name)
def __eq__(self, other):
return self.class_name == other.class_name
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.class_name)
class Voter(Component):
"""
Voter type. All component that inherits from BaseVoter have
this type in Banana.
"""
def __init__(self, class_name, ctor_properties):
super(Voter, self).__init__(ctor_properties, class_name)
def __eq__(self, other):
return self.class_name == other.class_name
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.class_name)
class Ldp(Component):
"""
Ldp type. All component that inherits from BaseLdp have
this type in Banana.
"""
def __init__(self, class_name, ctor_properties):
super(Ldp, self).__init__(ctor_properties, class_name)
def __eq__(self, other):
return self.class_name == other.class_name
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.class_name)
class Sml(Component):
"""
Sml type. All component that inherits from BaseSml have
this type in Banana.
"""
def __init__(self, class_name, ctor_properties):
super(Sml, self).__init__(ctor_properties, class_name)
def __eq__(self, other):
return self.class_name == other.class_name
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.class_name)
def get_type(ast_node):
"""
Returns the type for the given ast node.
This function only works for literal node such as
Number, StringLit and JsonObj.
:type ast_node: ast.Number | ast.StringLit | ast.JsonObj | ast.Component
:param ast_node: the node.
:return: Returns the appropriate type.
"""
if isinstance(ast_node, ast.Number):
return Number()
if isinstance(ast_node, ast.StringLit):
return String()
if isinstance(ast_node, ast.JsonObj):
return Object(strict_checking=False)
if isinstance(ast_node, ast.Component):
return Component()
return None
def can_to_str(_type):
"""
Check if we the type can be cast to str.
:param _type: Type to check
:return: Returns True if it can be casted
"""
return isinstance(_type, String)
def is_comp(_type):
"""
:type _type: String | Number | Object | Component
:param _type: Type to check.
:rtype: bool
:return: Returns True if the provided _type is a component
"""
return isinstance(_type, Component)
def can_be_cast_to(_type1, _type2):
"""
Check if the given type `_type1` can be cast into `_type2`.
:type _type1: String | Number | Enum | Object
:param _type1: Type to try to change into _type2
:type _type2: String | Number | Enum | Object
:param _type2: Type reference.
:return: Returns true if the conversion can be done.
"""
if isinstance(_type2, Any):
return True
elif _type1 == _type2:
return True
elif _type2 == String():
return can_to_str(_type1)
elif isinstance(_type2, Enum):
return isinstance(_type1, String) or isinstance(_type2, Enum)
elif isinstance(_type1, Object) and isinstance(_type2, Object):
if not _type2.strict_checking:
return True
else:
for prop_name, prop_type in six.iteritems(_type2.props):
if prop_name not in _type1.props:
return False
if not can_be_cast_to(_type1.props[prop_name], prop_type):
return False
return True
return False

0
monasca_analytics/component/__init__.py
View File

126
monasca_analytics/component/base.py
View File

@ -1,126 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import abc
import six
class abstractstatic(staticmethod):
"""
See http://stackoverflow.com/a/4474495 for
more details on this class purpose
"""
__slots__ = ()
def __init__(self, function):
super(abstractstatic, self).__init__(function)
function.__isabstractmethod__ = True
__isabstractmethod__ = True
@six.add_metaclass(abc.ABCMeta)
class BaseComponent(object):
"""Base class for all component types.
Should be as lightweight as possible, becuase any data here will be sent
to all workers every time a component is added.
"""
def __init__(self, _id, _config):
"""BaseComponent constructor.
:type _id: str
:param _id: identifier of this Source
:type _config: dict
:param _config: configuration of this Source
"""
self.validate_config(_config)
self._id = _id
@abstractstatic
def validate_config(_config): # @NoSelf
"""Abstract static method for configuration validation.
To be implemented by BaseComponent children.
It is called at creation time, and it should validate the
schema of the configuration passed as parameter
(i.e. check for expected keys and value format).
This function should raise exceptions if the validation fails,
otherwise it is assumed the validation was successful.
:type _config: dict
:param _config: configuration of this module to be validated.
"""
pass
@abstractstatic
def get_default_config(): # @NoSelf
"""Abstract static method that returns the default configuration.
To be implemented by BaseComponent children. It has to return a default
valid configuration for the module.
:rtype: dict
:returns: default configuration
"""
pass
@abstractstatic
def get_params(): # @NoSelf
"""Abstract static method that returns the description of the params.
To be implemented by BaseComponent children. It has to return
a list of the params description such as:
return [
ParamDescriptor('param1', type_util.String(), 'default value'),
ParamDescriptor('param2', type_util.Object({
'a': type_util.Number()
), {'a': 123}),
...
]
This function must be kept in sync with `get_default_config` and
`validate_config`, otherwise banana scripts using this component
will get runtime errors when being evaluated.
The order in the list maps to the order the parameter must be
passed when the component would be created, in banana:
a = MyComponent(param1, param2)
`param1` and `param2` would be type-checked respectively against
the first and the second element of the returned list.
:rtype: list[monasca_analytics.component.params.ParamDescriptor]
:return: Returns the list of parameters accepted by this component.
"""
pass
def id(self):
return self._id
def __hash__(self):
return hash(self._id)
def __eq__(self, other):
return self._id == other.id()
def __ne__(self, other):
return not(self == other)
def __str__(self):
return self._id

54
monasca_analytics/component/params.py
View File

@ -1,54 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import monasca_analytics.banana.typeck.type_util as u
class ParamDescriptor(object):
"""
Description of a component parameter. This object contains
information such as the name of the parameter, the type,
the default value and a validator that will be evaluated
when the component is instantiated.
"""
def __init__(self, name, _type, default=None, validator=None):
"""
Construct a parameter descriptor.
:type name: str
:param name: The name of the parameter
:type _type: u.String | u.Number | u.Object | u.Enum | u.Any
:param _type: The type of the parameter
:type default: str | float | int | dict
:param default: The default value for the parameter.
:param validator: Additional validator for the parameter.
"""
if not isinstance(_type, u.String) and\
not isinstance(_type, u.Number) and\
not isinstance(_type, u.Object) and\
not isinstance(_type, u.Enum) and\
not isinstance(_type, u.Any):
raise Exception("ParamDescriptor incorrectly defined")
self.param_name = name
self.default_value = default
self.param_type = _type
self.validator = validator
def to_json(self):
return {
"name": self.param_name,
"default_value": self.default_value,
"type": self.param_type.to_json(),
}

0
monasca_analytics/config/__init__.py
View File

63
monasca_analytics/config/config.py
View File

@ -1,63 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import sys
import monasca_analytics.config.connection as connection
import monasca_analytics.config.creation as creation
import monasca_analytics.config.validation as validation
import monasca_analytics.source.base as msource
logger = logging.getLogger(__name__)
def instantiate_components(_config):
"""Instantiate the components from the configuration.
:type _config: dict
:param _config: configuration of the module to instantiate
:rtype: dict
:returns: the list of components
"""
try:
validation.validate_config(_config)
logger.debug("Creating components according to configuration...")
components = creation.create_components(_config)
logger.debug("Done creating components. Creating link data...")
links = connection.connect_components(components, _config)
logger.debug("Done connecting components. Validating links...")
validation.validate_links(links)
logger.debug("Done validating links. Successful instantiation")
return links
except Exception as ex:
logger.error("Failed to instantiate components")
logger.error("Reason : " + str(ex))
sys.exit(-1)
def collect_sources(links):
"""Collect the sources from the links and return them in a list
:type links: dict
:returns: List of sources
:rtype: list[msource.BaseSource]
"""
sources = []
for key in links.keys():
if isinstance(key, msource.BaseSource):
sources.append(key)
return sources

107
monasca_analytics/config/connection.py
View File

@ -1,107 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import monasca_analytics.config.const as const
import monasca_analytics.exception.monanas as err
logger = logging.getLogger(__name__)
def connect_components(components, _config):
"""Connects the components using the information from the configuration
:type components: dict
:param components: components to connect
:type _config: dict
:param _config: configuration containing the
connections definitions
:rtype: dict
:returns: Dictionary where keys are IDs of sources of connections,
and values are their destinations IDs.
"""
return _perform_all_connections(const.CONNECTIONS, _config, components)
def _perform_all_connections(connection_kind, _config, components):
"""Connect all component that are of the given kind.
:type connection_kind: str
:param connection_kind: Kind of connection (feedback or flow)
:type _config: dict
:param _config: configuration containing the connections
:type components: dict
:param components: components to connect.
:rtype: dict
:returns: Dictionay where keys are IDs of sources of connections,
and values are their destinations IDs.
"""
links = {}
for origin_id in _config[connection_kind].keys():
for comp_type in const.components_types:
if origin_id in components[comp_type]:
component = components[comp_type][origin_id]
connections_list = _config[connection_kind][origin_id]
_perform_connections(
component, connections_list,
components, links)
return links
def _perform_connections(origin, connections_list, components, links):
"""Connect 'origin' with all destinations in connections_list
:type origin: str
:param origin: origin component
:type connections_list: list[str]
:param connections_list: destinations IDs to be connected
:type components: dict
:param components: all components
:type links: dict
:param links: links that we are going to represent
"""
if origin not in links:
links[origin] = []
if not connections_list:
logger.debug("No connections for {}".format(origin))
return
for dest_id in connections_list:
if dest_id in components[const.INGESTORS].keys():
ingestor = components[const.INGESTORS][dest_id]
links[origin].append(ingestor)
continue
if dest_id in components[const.SMLS].keys():
sml = components[const.SMLS][dest_id]
links[origin].append(sml)
continue
if dest_id in components[const.VOTERS].keys():
voter = components[const.VOTERS][dest_id]
links[origin].append(voter)
continue
if dest_id in components[const.SINKS].keys():
sink = components[const.SINKS][dest_id]
links[origin].append(sink)
continue
if dest_id in components[const.LDPS].keys():
ldp = components[const.LDPS][dest_id]
links[origin].append(ldp)
continue
raise err.MonanasWrongConnectoinError(
"wrong component defined in connection : " + dest_id)

54
monasca_analytics/config/const.py
View File

@ -1,54 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import copy
SOURCES = "sources"
INGESTORS = "ingestors"
SMLS = "smls"
VOTERS = "voters"
SINKS = "sinks"
LDPS = "ldps"
CONNECTIONS = "connections"
FEEDBACK = "feedback"
components_types = [SOURCES, INGESTORS, SMLS, VOTERS, SINKS, LDPS]
_default_base_config = {
"spark_config": {
"appName": "testApp",
"streaming": {
"batch_interval": 1
}
},
"server": {
"port": 3000,
"debug": False
},
"sources": {},
"ingestors": {},
"smls": {},
"voters": {},
"sinks": {},
"ldps": {},
"connections": {},
"feedback": {}
}
def get_default_base_config():
return copy.deepcopy(_default_base_config)

82
monasca_analytics/config/creation.py
View File

@ -1,82 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
from monasca_analytics.config import const
from monasca_analytics.util import common_util
import six
logger = logging.getLogger(__name__)
def create_components(_config):
"""Creates the components defined by the configuration
:type _config: dict
:param _config: configuration containing components
:rtype: dict
:returns: Created components indexed by type and ID
"""
components = {}
for component_type in const.components_types:
components[component_type] = \
_create_comps_by_module(component_type, _config)
return components
def _create_comps_by_module(comp_type, _config):
"""Instantiates all the components of a type defined in the configuration
:type comp_type: str
:param comp_type: type of the components to be deployed
(e.g. source, ingestor, ...)
:type _config: dict
:param _config: Configuration containing components
:rtype: dict
:returns: deployed components, keyed by ID
:raises: MonanasNoSuchSourceError -- if no source class found.
"""
logger.debug("Creating components of type : " + comp_type)
ret = {}
for comp_id, comp_config in six.iteritems(_config[comp_type]):
comp = _create_component_by_module(
comp_id, comp_config, comp_type)
ret[comp_id] = comp
return ret
def _create_component_by_module(comp_id, comp_config, comp_type):
"""Create a single component matching the past configuration.
The id assigned to that component will be comp_id.
:type comp_id: str
:param comp_id: ID of the component to create
:type comp_config: dict
:param comp_config: Configuration of the component to create
:type comp_type: str
:param comp_type: type of component to create
:rtype: monasca_analytics.component.base.BaseComponent
:returns: Instantiated component object
"""
logger.debug("deploying " + comp_config["module"] + " object")
clazz = common_util.get_class_by_name(comp_config["module"],
comp_type)
_comp = clazz(comp_id, comp_config)
return _comp

251
monasca_analytics/config/validation.py
View File

@ -1,251 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""A list of functions to validate config models."""
import logging
import six
import voluptuous
from monasca_analytics.config import const
logger = logging.getLogger(__name__)
valid_connection_types = {
const.SOURCES: [const.INGESTORS, const.LDPS],
const.INGESTORS: [const.SINKS],
const.SMLS: [const.VOTERS, const.SINKS],
const.VOTERS: [const.LDPS, const.SINKS],
const.LDPS: [const.SINKS],
const.SINKS: []
}
valid_feedback_connection_types = {
const.SOURCES: [],
const.INGESTORS: [],
const.SMLS: [],
const.VOTERS: [],
const.LDPS: [],
const.SINKS: [const.VOTERS, const.SMLS]
}
def validate_config(config):
"""Perform the whole validation: schema, uniqueness and connections
:type config: dict
:param config: configuration to validate
:raises: SchemaError -- if the configuration is not valid for any reason
"""
_validate_schema(config)
_validate_only_one_voter(config)
_validate_ids_uniqueness(config)
_validate_connections(config)
def validate_links(links):
"""Validate links to make sure, nothing is missing
:type links: dict
:param links: connection links to validate
:raises: SchemaError -- if any link is missing
"""
missing = set([])
all_keys = set(links.keys())
for connections in links.values():
for component in connections:
if component not in all_keys:
missing.add(component.id())
if len(missing) > 0:
raise voluptuous.Invalid([
"In connections section, the following components are not "
"connected\n\t{}\n"
"please modify the configuration so that their list of "
"connections is at least '[]'".format(", ".join(missing))], [])
def _validate_schema(config):
"""Validate the configuration, with spark, up to the orchestration level
Checks that hte spark configuration is valid, as well as the modules
structure in the configuration up to the orchestration level.
Each module will be responsible to validate its own sub-configuration.
:type config: dict
:param config: configuration model for the whole system
:raises: SchemaError -- if the configuration, up to the
orchestration level, is not valid
"""
config_schema = voluptuous.Schema({
"spark_config": {
"appName": six.string_types[0],
"streaming": {
"batch_interval": voluptuous.And(int, voluptuous.Range(min=1))
}
},
"server": {
"port": int,
"debug": bool
},
"sources": {
voluptuous.Optional(six.string_types[0]): {six.string_types[0]:
object}
},
"ingestors": {
voluptuous.Optional(six.string_types[0]): {six.string_types[0]:
object}
},
"smls": {
voluptuous.Optional(six.string_types[0]): {six.string_types[0]:
object}
},
"voters": {
voluptuous.Optional(six.string_types[0]): {six.string_types[0]:
object}
},
"sinks": {
voluptuous.Optional(six.string_types[0]): {six.string_types[0]:
object}
},
"ldps": {
voluptuous.Optional(six.string_types[0]): {six.string_types[0]:
object}
},
"connections": {
voluptuous.Optional(six.string_types[0]): [six.string_types[0]]
},
"feedback": {
voluptuous.Optional(six.string_types[0]): [six.string_types[0]]
}
}, required=True)
return config_schema(config)
def _validate_only_one_voter(config):
"""Check that the configuration defines only a single voter
:type config: dict
:param config: configuration model for the whole system
:raises: SchemaError -- if there is more than one voter defined in config
"""
def _raise(comp):
raise voluptuous.Invalid([
"More than one {} found in the config, please modify " +
"it specifying only one {}".format(comp, comp)], [])
if len(config["voters"]) > 1:
_raise("voter")
def _validate_ids_uniqueness(config):
"""Validate that the IDs of the components are unique
:type config: dict
:param config: configuration model for the whole system
:raises: SchemaError -- if there is any duplicated ID in the configuration
"""
all_ids = set()
for comp_type in valid_connection_types.keys():
for com_id in config[comp_type].keys():
if com_id in all_ids:
raise voluptuous.Invalid(
["Duplicated component ID : " + com_id], [])
all_ids.add(com_id)
def _validate_expected_dest_type(config, from_id, to_ids, expected_types):
"""Check that the connection is valid according to expected_types.
:type config: dict
:param config: configuration model for the whole system
:type from_id: str
:param from_id: ID of the component which is the
origin point of the connection
:type to_ids: list
:param to_ids: IDs of the components which are the
destination points of the connections
:type expected_types: list
:param expected_types: types of components that are allowed
as destination points
"""
for to_id in to_ids:
logger.debug("validating connection " + from_id + " --> " + to_id)
valid_connection = False
for expected_type in expected_types:
if to_id in config[expected_type].keys():
valid_connection = True
break
if not valid_connection:
raise voluptuous.Invalid([
from_id + " connected to a wrong component: " + to_id +
". It should be connected only to any of : " +
str(expected_types)], [])
def _validate_existing_id(config, component_id):
"""Check that the id passed as parameter is defined in the configuration
:type config: dict
:param config: configuration model for the whole system
:type component_id: str
:param component_id: component ID to be found in configuration
"""
found_id = False
for comp_type in valid_connection_types.keys():
if component_id in config[comp_type].keys():
found_id = True
if not found_id:
raise voluptuous.Invalid([
'In "connections", component `{}` hasn\'t been defined'
.format(component_id)
], [])
def _validate_from_dictionary(config, conf_key, validation_dict):
"""Validate connections in config[conf_key] according to validation_dict
:type config: dict
:param config: configuration model for the whole system
:type conf_key: str
:param conf_key: key of the configuration dictionary where
the connections to be checked are defined
:type validation_dict: dict
:param validation_dict: keys are source types, and values
are lists of allowed destination types
for that particular source type
"""
for from_id in config[conf_key].keys():
_validate_existing_id(config, from_id)
to_ids = config[conf_key][from_id]
for comp_type in validation_dict.keys():
if from_id in config[comp_type].keys():
_validate_expected_dest_type(
config, from_id, to_ids, validation_dict[comp_type])
def _validate_connections(config):
"""Validate that the connections defined in config are allowed
:type config: dict
:param config: configuration model for the whole system
"""
_validate_from_dictionary(config, "connections", valid_connection_types)
_validate_from_dictionary(config, "feedback",
valid_feedback_connection_types)

0
monasca_analytics/exception/__init__.py
View File

353
monasca_analytics/exception/banana.py
View File

@ -1,353 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Banana Error classes."""
import abc
import pyparsing as p
import six
import monasca_analytics.banana.grammar.base_ast as ast
@six.add_metaclass(abc.ABCMeta)
class BananaException(Exception):
@abc.abstractmethod
def __str__(self):
pass
@abc.abstractmethod
def get_span(self):
"""
:rtype: ast.Span
:return: Returns the span where the error occured if appropriate
"""
pass
class BananaInvalidExpression(BananaException):
def __init__(self, value):
self._value = value
def __str__(self):
return repr(self._value)
def get_span(self):
return ast.DUMMY_SPAN
class BananaEnvironmentError(BananaException):
def __init__(self, value):
self._value = value
def __str__(self):
return repr(self._value)
def get_span(self):
return ast.DUMMY_SPAN
class BananaNoFullPath(BananaException):
def __init__(self, missing):
self._value = "None of the paths can be executed. Missing at least" \
" one {}.".format(missing)
def __str__(self):
return self._value
def get_span(self):
return ast.DUMMY_SPAN
class BananaArgumentTypeError(BananaException):
def __init__(self, where, expected_type, received_type):
if isinstance(where, ast.ASTNode):
self._span = where.span
else:
self._span = where
self._value = "Wrong type of argument. Expected '{}' got '{}'."\
.format(expected_type, received_type)
def __str__(self):
return self._value
def get_span(self):
return self._span
class BananaComponentTooManyParams(BananaException):
def __init__(self, span):
self._span = span
self._value = "Too many params provided to '{}'.".format(
span, span.get_lineno()
)
def __str__(self):
return self._value
def get_span(self):
return self._span
class BananaComponentMixingParams(BananaException):
def __init__(self, span, named_is_wrong):
self._span = span
if named_is_wrong:
self._value = "'{}' should be named as " \
"previous parameters are.".format(span)
else:
self._value = "'{}' should not be named as " \
"previous parameters are.".format(span)
def __str__(self):
return self._value
def get_span(self):
return self._span
class BananaComponentIncorrectParamName(BananaException):
def __init__(self, found, component):
if isinstance(component, ast.ASTNode):
component = component.span
if isinstance(found, ast.ASTNode):
self._span = found.span
found = found.span
else:
self._span = found
self._value = "Incorrect parameter name. Parameter '{}' " \
"does not exists on component {}."\
.format(found, component)
def __str__(self):
return self._value
def get_span(self):
return self._span
class BananaComponentAlreadyDefined(BananaException):
def __init__(self, first_def, second_def):
self._value = "Component already defined!\n" \
" First definition: '{}'\n" \
" Second definition: '{}'."\
.format(first_def, second_def)
def __str__(self):
return self._value
def get_span(self):
# TODO(Joan): This could be a real span instead of this one.
return ast.DUMMY_SPAN
class BananaShadowingComponentError(BananaException):
def __init__(self, where, comp):
self._span = where
self._value = "Shadowing component '{}'. " \
"Please use another variable name.".format(comp)
def __str__(self):
return self._value
def get_span(self):
return self._span
class BananaAssignmentError(BananaException):
def __init__(self, lhs, rhs):
self._value = "You can't assign '{}' to '{}'.".format(lhs, rhs)
def __str__(self):
return self._value
def get_span(self):
return ast.DUMMY_SPAN
class BananaGrammarBug(BananaException, p.ParseFatalException):
def __init__(self, error):
super(BananaGrammarBug, self).__init__(pstr=error)
self._value = "Bug found in the grammar!" \
" Please report this error: {}".format(error)
def __str__(self):
return self._value
def get_span(self):
return ast.DUMMY_SPAN
class BananaJsonObjShadowingError(BananaException, p.ParseFatalException):
def __init__(self, span, error):
self._span = span
error = "Can't shadow property already defined in {}.".format(error)
super(BananaJsonObjShadowingError, self).__init__(pstr=error)
def __str__(self):
return self.msg
def get_span(self):
return self._span
class BananaTypeCheckerBug(BananaException):
def __init__(self, error):
self._value = "Bug found in the TypeChecker!" \
" Please report this error: {}".format(error)
def __str__(self):
return self._value
def get_span(self):
return ast.DUMMY_SPAN
class BananaEvalBug(BananaException):
def __init__(self, error):
self._value = "Bug found in the evaluator!" \
" Please report this error: {}".format(error)
def __str__(self):
return self._value
def get_span(self):
return ast.DUMMY_SPAN
class BananaUnknown(BananaException):
def __init__(self, ident):
self._span = ident.span
self._value = "Unknown '{}'.".format(
ident.into_unmodified_str()
)
def __str__(self):
return self._value
def get_span(self):
return self._span
class BananaUnknownOperator(BananaException):
def __init__(self, span, operator, for_type):
self._span = span
self._value = "Unknown operator '{}' for type '{}'.".format(
operator,
for_type
)
def __str__(self):
return self._value
def get_span(self):
return self._span
class BananaPropertyDoesNotExists(BananaException):
def __init__(self, dotpath, on_type=None):
self._span = dotpath.span
if on_type is None:
self._value = "Property '{}' " \
"does not exists."\
.format(
dotpath.into_unmodified_str()
)
else:
self._value = "Property '{}' " \
"does not exists on type '{}'."\
.format(
dotpath.into_unmodified_str(),
str(on_type)
)
def __str__(self):
return self._value
def get_span(self):
return self._span
class BananaTypeError(BananaException):
def __init__(self, expected_type, found_type=None, span=None):
self._span = span
if expected_type is None:
class DummyType(object):
def __str__(self):
return "_"
expected_type = DummyType
if found_type is None:
if isinstance(expected_type, list):
self._value = "Type error found. Expected" \
" one among '{}'."\
.format(', '.join(map(lambda x: str(x), expected_type)))
else:
self._value = "Type error found. Expected '{}'.".format(
str(expected_type)
)
else:
if isinstance(expected_type, list):
self._value = "Type error found. Expected" \
" one among '{}', found '{}'."\
.format(', '.join(map(lambda x: str(x), expected_type)),
str(found_type))
else:
self._value = "Type error found. Expected" \
" '{}', found '{}'."\
.format(str(expected_type), str(found_type))
def __str__(self):
return self._value
def get_span(self):
if self._span is None:
return ast.DUMMY_SPAN
return self._span
class BananaAssignCompError(BananaException):
def __init__(self, span):
self._span = span
self._value = "Component objects " \
"can't be assigned to " \
"properties of other objects."
def __str__(self):
return self._value
def get_span(self):
return self._span
class BananaConnectionError(BananaException):
def __init__(self, span, ident_from, ident_to, type_from,
possible_connections):
self._value = "Can't connect '{}'" \
" to '{}'," \
" '{}' can only be connected to a {}."\
.format(
ident_from.val,
ident_to.val,
type_from.class_name, ' or a '.join(possible_connections))
self._span = span
def __str__(self):
return self._value
def get_span(self):
return self._span

64
monasca_analytics/exception/dsl.py
View File

@ -1,64 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2016 Hewlett Packard Enterprise Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""DSL Error classes."""
import abc
import six
@six.add_metaclass(abc.ABCMeta)
class DSLException(Exception):
@abc.abstractmethod
def __str__(self):
pass
class DSLExistingConnection(DSLException):
def __init__(self, value):
self._value = value
def __str__(self):
DSLException.__str__(self)
return repr(self._value)
class DSLInexistentComponent(DSLException):
def __init__(self, value):
self._value = value
def __str__(self):
DSLException.__str__(self)
return repr(self._value)
class DSLInvalidConnection(DSLException):
def __init__(self, value):
self._value = value
def __str__(self):
DSLException.__str__(self)
return repr(self._value)
class DSLInterpreterException(DSLException):
def __init__(self, value):
self._value = value
def __str__(self):
DSLException.__str__(self)
return repr(self._value)

Some files were not shown because too many files have changed in this diff Show More