openstack
/
sahara-image-elements
Code Issues Proposed changes
sahara-image-elements/elements/hadoop-cloudera
History
Daniele Venzano f376b0f480 Use Cloudera element for Spark HDFS 
Update the Spark element to use the existing hadoop-cloudera element for HDFS
for Spark versions > 1.0, instead of the ad-hoc cloudera-cdh one. For Spark 1.0.2,
CDH4 via the old hadoop-cdh element is used, since a precompiled binary for CDH5
is not available.

This change also makes it possible to specify an arbitrary Spark version via the
new -s commandline switch, reducing the amount of code for supporting future
versions of Spark. The defaults for Spark are 1.3.1 and CDH 5.3, a combination
that works well in our deployments.

A small change is needed in the cloudera element: when creating a Spark image,
only the HDFS packages have to be installed.

README files have been updated to clarify that default versions are tested, while
other combinations are not. A reference to the SparkPlugin wiki page was added
to point to a table of supported versions.

Change-Id: Ifc2a0c8729981e1e1df79b556a4c2e6bd1ba893a
Implements: blueprint support-spark-1-3
Depends-On: I8fa482b6d1d6abaa6633aec309a3ba826a8b7ebb
 		2015-07-10 14:12:37 +00:00
..
environment.d Switch to the extjs element 2015-04-08 18:59:11 +02:00
install.d Use Cloudera element for Spark HDFS 2015-07-10 14:12:37 +00:00
post-install.d Make almost all the element scripts as e/u/pipefail 2015-04-07 12:46:39 +02:00
pre-install.d Add installing keytrustee-keyprovider package 2015-06-09 00:16:35 +08:00
README.rst Use Cloudera element for Spark HDFS 2015-07-10 14:12:37 +00:00
element-deps Start switching to declarative package-installs 2015-04-15 00:44:54 +08:00
package-installs.yaml Use Cloudera element for Spark HDFS 2015-07-10 14:12:37 +00:00

README.rst

hadoop-cloudera

Installs cloudera (cloudera-manager-agent cloudera-manager-daemons cloudera-manager-server cloudera-manager-server-db-2 hadoop-hdfs-namenode hadoop-hdfs-secondarynamenode hadoop-hdfs-datanode hadoop-yarn-resourcemanager hadoop-yarn-nodemanager hadoop-mapreduce hadoop-mapreduce-historyserver) and Java (oracle-j2sdk1.7) packages from cloudera repositories cdh5 and cm5.

In order to create the Cloudera images with diskimage-create.sh, use the following syntax to select the cloudera plugin:

diskimage-create.sh -p cloudera

Environment Variables

The element can be configured by exporting variables using a environment.d script.

CDH_HDFS_ONLY
Required

No

Description

If set will install only the namenode and datanode packages with their dependencies.