agatea 1579d8b9e5 Reuse existing spark sql context ... Prevent creating a new spark sql context object with every batch. Profiling of java heap for the driver indicated that there is a steady increase (~12MB over 5 days) of org.apache.spark.sql.execution.metric.LongSQLMetricValue and org.apache.spark.sql.execution.ui.SQLTaskMetrics with each batch execution. These are used by the spark streaming ui and were not being garbage collected. See https://issues.apache.org/jira/browse/SPARK-17381 with a similar issue. This change along with setting spark.sql.ui.retainedExecutions to a low number in sparks-defaults.conf will reduce gradual increase in heap size. Also made a change to catch unhandled MemberNotJoined exception because of whichthe transform service thread went into a unresponsive state. Change-Id: Ibf244cbfc00a90ada66f492b473719c25fa17fd2		2017-02-27 14:06:53 -08:00
..
__init__.py	monasca-transform initial commit	2016-05-26 00:10:37 +00:00
mon_metrics_kafka.py	Reuse existing spark sql context	2017-02-27 14:06:53 -08:00