History

Jeremy Freudberg a77a9a978a Add S3 jar to Hadoop classpath As prereq of support for S3 datasource, the hadoop-aws jar needs to be in the Hadoop classpath. The jar is copied into the proper folder when possible on the appropriate plugins, and otherwise can be provided from a download URL by the user. Additionally, set the correct value of DIB_HDFS_LIB_DIR on the Vanilla plugin to avoid any unnecessary simlinking. Partially-Implements: bp sahara-support-s3 Change-Id: I94c5b0055b87f6a4e1382118d0718e588fccfe87		2017-07-28 14:01:01 +00:00
..
post-install.d	Add S3 jar to Hadoop classpath	2017-07-28 14:01:01 +00:00
README.rst	Add S3 jar to Hadoop classpath	2017-07-28 14:01:01 +00:00
element-deps	Add S3 jar to Hadoop classpath	2017-07-28 14:01:01 +00:00
package-installs.yaml	Add S3 jar to Hadoop classpath	2017-07-28 14:01:01 +00:00

README.rst

s3_hadoop

Copy the Hadoop S3 connector jar file into the Hadoop classpath.

Environment Variables

HADOOP_S3_JAR_ORIGIN

Required: No
Default: Depends on plugin.
Description: Path to where the S3 jar is (already) located.

HADOOP_S3_JAR_DOWNLOAD

Required: No
Default: None.
Description: If set, use a download a specific S3 jar instead of one already available on the image.

DIB_HDFS_LIB_DIR

Required: No
Default: /usr/share/hadoop/lib
Description: Directory in the guest where to save the S3 jar. Shared with swift_hadoop.