assertItemsEqual was removed from Python's unittest.TestCase in
Python 3.3 [1][2]. We have been able to use them since then, because
testtools required unittest2, which still included it. With testtools
removing Python 2.7 support [3][4], we will lose support for
assertItemsEqual, so we should switch to use assertCountEqual.
[1] - https://bugs.python.org/issue17866
[2] - https://hg.python.org/cpython/rev/d9921cb6e3cd
[3] - testing-cabal/testtools#286
[4] - testing-cabal/testtools#277
Change-Id: I67ec6c5ea47b927e2ef44864afa9313e0956a90b
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I5868501da8cfc975c62e63a106f0fcae2384112a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
* To find the oldest and newest quantities
for calculating rate, Spark Data Frame
is converted to a Spark RDD, which makes it
easy to do a group by and sort. The syntax to pull
the value from a map column type in an
RDD is rdd.map_column['key'] which
is different from dataframe.map_column.key
which was causing 'dict' object has no attribute
exception
* Renamed first_attempt_at_spark_test.py to
test_first_attempt_at_spark, because all the tests
in that file were not being run.
* Changed the pre transform and transform specs
which were being used by test_first_attempt_at_spark
to newer 'dimensions#field_name that was introduced
in rocky release.
Story: 2005449
Task: 30501
Change-Id: I6adeee54fe261c535372b8f5f3580e7d3261259b
Remove unused field service_id from pre-transform spec.
service_id's original purpose was identify the service
that is genarating metric, but this information should
be provided by the source, as a dimension rather
than assigning its value in pre-transform spec.
Change-Id: I223eb2296df438b139e3d9b5aaf4b1b679f70797
Depends-on: I81a35e048e6bd5649c6b3031ac2722be6a309088
Story: 2001815
Task: 12556
* Removed unused fields "event_status", "event_version",
"record_type", "mount", "device", "pod_name", "container_name"
"app", "interface", "deployment" and "daemon_set"
from record_store data. Now it is not required to add
new dimension, meta or value_meta fields to record store data
instead use special notation, e.g. "dimension#" to refer to any
dimension field in the incoming metric.
* Refactor and eliminate need to add any new metric.dimensions
field in multiple places e.g. add to record store and
instance usage dataframe schema and in all generic
aggregation components. Added a new Map type column
called "extra_data_map" to store any new fields, in
instance usage data format. Map type column eliminates the
need to add new columns to instance usage data.
* Allow users to define any fields in "meta",
"metric.dimensions" and "metric.value_meta" fields
for aggregation in "aggregation_group_by_list" or
"setter_group_by_list" using "dimensions#{$field_name}"
or "meta#{$field_name}" or "value_meta#{$field_name}"
* Updated generic aggregation components and data formats docs.
Change-Id: I81a35e048e6bd5649c6b3031ac2722be6a309088
Story: 2001815
Task: 19605
* set the maximum line length to 100
* cleaned up the codes for pep8
Change-Id: Iab260a4e77584aae31c0596f39146dd5092b807a
Signed-off-by: Amir Mofakhar <amofakhar@op5.com>
Following changes were required:
1.)
By default the pre-built distribution
for Spark 2.2.0 is compiled with Scala 2.11.
monasca-transform requires Spark compiled with
Scala 2.10 since we use spark streaming to
pull data from Kafka and the version of Kafka
is compatible with Scala 2.10.
The recommended way is to compile Spark
with Scala 2.10, but for purposes of devstack
plugin made changes to pull the required jars
from mvn directly.
(see SPARK_JARS and SPARK_JAVA_LIB variables in
settings)
All jars get moved to
<SPARK_HOME>/assembly/target/assembly/
target/scala_2.10/jars/
Note: <SPARK_HOME>/jars gets renamed
to <SPARK_HOME>/jars_original.
spark-submit defaults to assembly location
if <SPARK_HOME>/jars directory is missing.
2.) Updated start up scripts for spark
worker and spark master with a new env variable
SPARK_SCALA_VERSIOn=2.10. Also updated
PYTHONPATH variable to add new
py4j-0.10.4-src.zip file
3.) Some changes to adhere to deprecated pyspark
function calls which were removed in Spark 2.0
Change-Id: I8f8393bb91307d55f156b2ebf45225a16ae9d8f4
With this change pre hourly processor which does the
hourly aggregation (second stage) and writes the
final aggregated metrics to metris topic in kafka
now accounts for any early arriving metrics.
This change along with two previous changes
to pre hourly processor that added
1.) configurable late metrics slack time
(https://review.openstack.org/#/c/394497/), and
2.) batch filtering
(https://review.openstack.org/#/c/363100/)
will make sure all late arriving or early
arriving metrics for an hour are aggregated
appropriately.
Also made improvement in MySQL offset to call
delete excess revisions only once.
Change-Id: I919cddf343821fe52ad6a1d4170362311f84c0e4
Changing devstack environment vagrant box and
also rename the devstack VM to 'devstack'
from 'pg-tips'
Also fixing all the tests that were broken when
they were moved from tests/unit to tests/functional
with this review
https://review.openstack.org/#/c/400237/
Update devstack README with a section called
Development workflow for monasca-transform with
steps developers can take to develop and run
tests.
Change-Id: I11678148ba2bcb96eb3e2a522176683dc8bca30a
Openstack common has a wrapper for generating uuids.
We should only use that function when generating uuids for consistency.
Fixes bug #1082248
Change-Id: I0dbd2c7e2a03f3932a963ea2196f32574d3a2306
There is a desire to use Monasca Transform to aggregate
kubernetes metrics. This change is a start in that
direction. The transformation specs in the tests
folder now include some representative aggregations
of some kubernetes metrics.
This commit also includes some changes to get
first_attempt_at_spark_test.py working again
after being moved from the unit test folder to the
functional test folder.
Change-Id: I038ecaf42e67d5c994980991232a2a8428a4f4e3
Added configuration option to allow the pre-hourly transformation to be
done at a specified period past the hour. This includes a check to
ensure that if not done yet for the hour but overdue processing is done
at the earliest time.
Change-Id: I8882f3089ca748ce435b4e9a92196a72a0a8e63f
Moved what were called unit tests to the functional directory as there is
more work to be done to make them self supporting.
Added sphinx to the test-requirements along with testr deps.
Adjusted the tox.ini to suit.
Added doc source tree with the minimal set of files to allow sphinx
success.
Change-Id: Ic64c2ccd01778bb0d1142d12ca43b2346bdc7aa7
Witek Bedyk