Commit Graph

2 Commits

Author SHA1 Message Date
Jeremy Freudberg 3ee20cbc09 Revise s3_hadoop
* Handle Hadoop classpath better
* Include proper support for Spark classpath
* Formally limit element's use to Vanilla and Spark

Change-Id: I65abd7e375dba11599a4ab943d24f878235cd71d
Closes-Bug: #1727757
Closes-Bug: #1728061
2018-02-07 05:58:44 +00:00
Jeremy Freudberg a77a9a978a Add S3 jar to Hadoop classpath
As prereq of support for S3 datasource, the hadoop-aws jar needs to be
in the Hadoop classpath. The jar is copied into the proper folder when
possible on the appropriate plugins, and otherwise can be provided from
a download URL by the user.

Additionally, set the correct value of DIB_HDFS_LIB_DIR on the Vanilla
plugin to avoid any unnecessary simlinking.

Partially-Implements: bp sahara-support-s3

Change-Id: I94c5b0055b87f6a4e1382118d0718e588fccfe87
2017-07-28 14:01:01 +00:00