Commit Graph

26 Commits

Author SHA1 Message Date
Zhuang Changkun 83224a6c5e Add support to build 3.0.1 image
Change-Id: I1d90099b22ed9a0ede08df45059e6d7848c237b8
2018-06-08 14:11:37 +08:00
Zhuang Changkun 5707f5cc5a Add support to build 2.7.5 image
Change-Id: Ia16692632cbe6380d4021dd031eed09362873905
2018-04-17 19:26:08 +08:00
Shu Yingya 12efcf84e2 Add support to build 2.8.2 image
Also tweak Hive a bit and refer to artifacts in a new (but not totally
ideal) location.

Co-Authored-By: Jeremy Freudberg <jeremyfreudberg@gmail.com>
Change-Id: I3a25ee8c282849911089adf6c3593b1bb50fd067
2018-03-01 09:51:01 +08:00
Luigi Toscano 8cdff5d71f Update the links to artifacts (removing sahara-files too)
The canonical location for the artifacts going on is
tarballs.openstack.org/sahara-extra/, so fix the link to use that
and also use https.

Moreover, since the last tarball required for building images
is available on tarballs.openstack.org, remove the last references
to sahara-files for artifacts and documentation
(the new location was used already for a while in few places).
there are still few references to sahara-files,
but they are all about CentOS6 which is no more supported
by diskimage-builder, and should be removed separately.

Change-Id: Iab5a4d50a0abc6ab278837b6a9efd5e30f31c44a
2018-02-27 12:14:09 +01:00
Evgeny Sikachov 8f2caee358 Add support to create CDH 5.11 images
Change-Id: Ibd61ca89bb24e369cca5d06e338edef2f0c6f6d6
2017-07-25 17:26:40 +00:00
Shu Yingya 20126fbde6 Remove some codes of older version builder
Building CDH image under version 5.5.0 is no longer support.
Remove these useless code.
Also, adding the ambari usage info in sahara-image-create
command.

Change-Id: I6fffe25ee9daf651355611be675137babb67e2a8
2017-04-14 11:24:05 +08:00
Luigi Toscano 3dfcb641c0 Set DIB_CDH_MINOR_VERSION when required
Set DIB_CDH_MINOR_VERSION in diskimage-create so that it could not
be an empty value. The order of the lines where DIB_CDH_VERSION is
assigned for other CDH versions where DIB_CDH_MINOR_VERSION is not
used was changed as well for consistency.
Also, in order to try to prevent future errors, set the value of
DIB_CDH_MINOR_VERSION based on DIB_CDH_VERSION in all elements where
it is used (thanks to the ${VAR:-value} syntax which does not fail
with set -u even if DIB_CDH_MINOR_VERSION is not assigned).

Closes-Bug: #1657482
Change-Id: I31b25fd4ba886d051b9b57902cd72349a4a2dbfa
2017-01-19 19:46:57 +01:00
yingya.shu 4cc289bc0c [CDH] Support CDH 5.9
Added support of CDH 5.9 for ubuntu, centos and centos7.

Change-Id: I14aad25829d011e14f37bf0ff50e50b157681163
bp: cdh-5-9-support
2016-12-09 11:05:54 +00:00
Vitaly Gridnev 9da9f50326 use latest build of hadoop openstack from tarballs
we migrated hadoop openstack jars to tarballs.openstack.org,
so we should switch to use these files.

Change-Id: I2b8a773c29649113fa85a9e0260b54b77bda6a3f
2016-09-21 16:04:56 +03:00
Vitaly Gridnev ec50d50387 drop vanilla 2.6.0 support from elements
we don't have vanilla 2.6.0 in supported list
in all current branches of sahara. we can
just drop that. if needed, stable/mitaka
branch should be used for building that image.

Change-Id: I81ed8209f2154f112fe7f6718029b84548793380
2016-08-22 17:41:05 +03:00
Michael Ionkin 038ea3a6f1 [CDH] Support CDH 5.7
Added support of CDH 5.7 for ubuntu, centos and centos7.
Also added support of CDH minor versions (5.7.0 and 5.7.1) that we
can set with DIB_CDH_MINOR_VERSION variable
By default it builds 5.7.0 version.

note: for Spark plugin we use HDFS part of CDH so we don't need to
create symbolic links for oozie (so for this we check variable
DIB_CDH_HDFS_ONLY)

depends-on: I1167d0d98ae6fb6fabaf7f1f9a344691d459b50b
Change-Id: I1d71ffbdf78373d27f7e1304164b32da786ac10b
bp: cdh-5-7-support
2016-07-07 12:26:09 +03:00
Michael Ionkin d56039497b Add hadoop openstack swift jar to plugins images
This patch adds our custom hadoop swiftfs implementation to Vanilla,
Spark, Ambari and CDH plugins images in order to allow usage of
swift with Keystone API v3

Partial-bug: 1558064
Depends-on: Ie6df4a542a16b4417b505b4a621e8b4b921364d3
Change-Id: Icd4b62bd4293bc9b40dba171a22285c7d0ac75c7
2016-03-24 19:02:11 +03:00
Michael McCune c00560e4db update swift_hadoop element
This change replaces the hardcoded value for the hadoop swift jar
filename with a variable that defaults to the old value
"hadoop-swift.jar" but also allows redefinition.

* add variables for hadoop swift jar file name
* add documentation for new variable
* correct documentation for the lib path variable
* add more to the basic description documentation

Change-Id: I44eaaa8b23ae30231b7b4087740d8dd5d050ec00
Closes-Bug: 1498967
2015-10-17 18:55:21 +00:00
Pino Toscano c8cbe25f1b Start switching to declarative package-installs
Make use of the package-installs element to declare in yaml format the
packages to be installed at the beginning of a phase.

Other than reducing the amont of explicit 'install-packages pkg1 ...'
invocations, they can also be installed just once per-phase.

Change-Id: I1f1acfb2bd74fed5cf4c0b48bc739f7f75c35d83
2015-04-15 00:44:54 +08:00
Pino Toscano 2fb2422da8 Make almost all the element scripts as e/u/pipefail
Following the latest dib-lint reporting, make sure almost all the
scripts are enabling -e, -u, and pipefail.  This eases the discovery of
failing commands, and the usage of unset variables.

There are few exceptions with flags not set:

* elements/hadoop-hdp/install.d/40-install-hdp, set -e
  when installing HDP 2.0, it tries to install tez, which does not seem
  to exist in Hortonworks' repositories

* elements/ssh/install.d/33-ssh, set -e
  the version of augtool (part of augeas 0.10) in older Ubuntu versions
  (like Precise, needed by the cloudera plugin) exits with wrong return
  values in autosave mode

* elements/storm/install.d/60-storm, set -e
  It tries to change the permission of /etc/supervisord.conf, which does
  not seem to exist

Change-Id: Ic1314639dfc6a66c48ca87b6820707a2b0cb1dbd
Partial-Bug: #1435306
2015-04-07 12:46:39 +02:00
Pino Toscano 157dd805e6 Fix more unbound variables
Followup of 7d189ed603.

Change-Id: Ia2aefe0b1dd3427c039bd8d3166d53f5dc6e4bd6
2015-04-07 12:46:31 +02:00
Pino Toscano 55a21cd746 Adopt DIB_DEBUG_TRACE for tracing in elements
Mimic commit 36b59c001c1643217449646b371df46d2cb11b91 in
diskimage-builder, by adopting the usage of $DIB_DEBUG_TRACE to check
whether enable tracing in scripts.

Unlike with the diskimage-builder commit, the default is to not enable
tracing even in the few scripts that used to unconditionally "set -x".

Enabling tracing can be done by either:
- passing -x to disk-image-create
- exporting DIB_DEBUG_TRACE=N, with N=0/1

Change-Id: I56ccd6753df31f7ddda641640cdb1985b2d9e856
Partial-Bug: #1435306
2015-03-23 20:12:41 +01:00
Trevor McKay 4560d069f2 Add the swift_hadoop element to Spark images
The Spark image needs the hadoop-swift.jar in /usr/lib/hadoop
in order for Spark jobs to access Swift URLs.

Change-Id: I1c47cbc877e6c6628dc3fd2181152b2c4d4cd3f9
Partial-Implements: blueprint edp-spark-swift-integration
2015-01-16 15:49:03 -05:00
Sergey Reshetnyak 59cd245f13 Fix bashate errors
Change-Id: Icee255fc3393bad79ca02457395170af42c61a32
2014-10-14 12:49:42 +04:00
Sergey Reshetnyak 8020b1b4fb Disable hadoop swift element for vanilla 2 plugin
In Hadoop 2 already have a library to support swiftfs

Change-Id: Id3a69fa3556dc18219aad029ba115ca3a0c0dff9
2014-03-21 12:13:47 +04:00
Michael McCune 0e1241d923 Renaming all Savanna references to Sahara
WARNING:
----
Before merging this commit the alias for
http://sahara-files.mirantis.com needs to be in place.

Also before merging this commit the new openstack git project must be
available at https://git.openstack.org/openstack/sahara-image-elements/

NOTE:
----
The file 'elements/hadoop-hdp/source-repository-hadoopswift' contains a
link to the HortonWorks repository that holds the Hadoop Swift rpm, this
link needs to be updated when HortonWorks makes the change.

Implements: blueprint savanna-renaming-image-elements
Change-Id: Icb9a992f8545535af3a111580ce7c9622d754c67
2014-03-13 15:26:47 +04:00
Sergey Reshetnyak 7a59589e22 Add hadoop v2 support to hadoop element
Partially implements bp: vanilla-plugin-hadoop-2

Change-Id: Ifc87c6c702d8198a1840620de3d85c575da6c98e
2014-02-28 11:24:55 +04:00
Sergey Lukjanov fa750f5a0e Hadoop-Swift integration jar moved to the CDN
All related docs are already duplicated at savanna.rtfd.org

Change-Id: I9c7988bd9cffba6cca2ab4125cbb81b28b52e799
2013-09-02 13:00:05 +04:00
Matthew Farrellee 06b88336e2 Add missing #! to 81-add-jar
Change-Id: Icbdfbb107a6e4326680105f0b83ecb6055cb7674
2013-09-02 12:59:05 +04:00
Ivan Berezovskiy 129ac229c7 Add check of swift downloading
Change-Id: Ic7049f914a5917a71546552b7108a576a8588a93
2013-09-02 12:59:04 +04:00
Ivan Berezovskiy 329bbb606d Add install-packages 'wget' to Swift element (for Fedora).
Change-Id: Ie64676bfa7d878ebc6fd6966a5747b13d2373417
2013-09-02 12:59:00 +04:00