Add spark element to vanilla images.
It provides run spark jobs on vanilla
clusters.
bp spark-jobs-for-vanilla-hadoop
Change-Id: Ie5eb9ec10b0052c9d1f6284b312edfee0ddba4f0
This change request adds deprecation warnings when generating
images containing Spark versions that have been deprecated in the
Sahara Spark plugin.
The idea is also to deprecate all Spark versions that depend on
the hadoop-cdh element, so that it can be removed later on.
Change-Id: Iefed31c8f22a40eb4e3eed835d7bb10f4fa20750
Implements: blueprint deprecate-spark-100
Update the Spark element to use the existing hadoop-cloudera element for HDFS
for Spark versions > 1.0, instead of the ad-hoc cloudera-cdh one. For Spark 1.0.2,
CDH4 via the old hadoop-cdh element is used, since a precompiled binary for CDH5
is not available.
This change also makes it possible to specify an arbitrary Spark version via the
new -s commandline switch, reducing the amount of code for supporting future
versions of Spark. The defaults for Spark are 1.3.1 and CDH 5.3, a combination
that works well in our deployments.
A small change is needed in the cloudera element: when creating a Spark image,
only the HDFS packages have to be installed.
README files have been updated to clarify that default versions are tested, while
other combinations are not. A reference to the SparkPlugin wiki page was added
to point to a table of supported versions.
Change-Id: Ifc2a0c8729981e1e1df79b556a4c2e6bd1ba893a
Implements: blueprint support-spark-1-3
Depends-On: I8fa482b6d1d6abaa6633aec309a3ba826a8b7ebb
Following the latest dib-lint reporting, make sure almost all the
scripts are enabling -e, -u, and pipefail. This eases the discovery of
failing commands, and the usage of unset variables.
There are few exceptions with flags not set:
* elements/hadoop-hdp/install.d/40-install-hdp, set -e
when installing HDP 2.0, it tries to install tez, which does not seem
to exist in Hortonworks' repositories
* elements/ssh/install.d/33-ssh, set -e
the version of augtool (part of augeas 0.10) in older Ubuntu versions
(like Precise, needed by the cloudera plugin) exits with wrong return
values in autosave mode
* elements/storm/install.d/60-storm, set -e
It tries to change the permission of /etc/supervisord.conf, which does
not seem to exist
Change-Id: Ic1314639dfc6a66c48ca87b6820707a2b0cb1dbd
Partial-Bug: #1435306
Make sure that, even when set -u, scripts can run handling unset
variables correctly.
All the changes involve variables optionally set by the user.
Change-Id: If74051fb5882eacdaa49ca7e35d2808f3989bd17
Mimic commit 36b59c001c1643217449646b371df46d2cb11b91 in
diskimage-builder, by adopting the usage of $DIB_DEBUG_TRACE to check
whether enable tracing in scripts.
Unlike with the diskimage-builder commit, the default is to not enable
tracing even in the few scripts that used to unconditionally "set -x".
Enabling tracing can be done by either:
- passing -x to disk-image-create
- exporting DIB_DEBUG_TRACE=N, with N=0/1
Change-Id: I56ccd6753df31f7ddda641640cdb1985b2d9e856
Partial-Bug: #1435306
Add a space before ending ], otherwise it will produce the error:
[...]/root.d/0-check-spark: line 2: [: missing `]'
Change-Id: I178dff2d66310056bce05747094c8b912d575b19
Rename all the non-mirror 0-check scripts to avoid conflicting names
with newer diskimage-builder.
Kind of followup of commit de118b5ab1.
Change-Id: I01eb5575ffc5c40e278f0d275580ed9f541a64dd
Related-Bug: #1424715
Mikhail Lelyakin