Commit Graph

276 Commits

Author SHA1 Message Date
Elod Illes c80594592a DNM: test build with latest setuptools
Change-Id: I0e489c0d1b0b1c641164d64f1bc0120d4b5da21e
2022-03-26 11:32:32 +01:00
jorgefrancoibanez 24a0ea0195 Allow qemu-utils to be used to build image
This patch allows build sahara images in ubuntu/debian
distribution with qemu-utils package.

Change-Id: Ibfa5c2550898a0d36b51057f5cd2b85f434b57f5
Story: 2006505
Task: 36483
2019-09-10 21:03:10 -04:00
Luigi Toscano 29c817d217 Plain Ubuntu image are still based on Xenial
Even if we are not really testing plain images much, let's make clear
that we didn't test any hadoop stuff on bionic yet.

Change-Id: I684a98338945d4449e37d9f652bddddf827838bf
2018-12-03 11:57:58 +01:00
Luigi Toscano 5eeec72859 Use the correct "or" operator to check the spark version
It clearly slipped through the review...

Story: 2003281
Task: 24222
Change-Id: I094045a665a4441029fa4930e278e94d0cff50f8
2018-08-02 09:58:34 +02:00
Telles Nobrega 34e8f4a27a Updating Storm version list
We are adding Storm 1.2.0 and 1.2.1 and removing 0.9.2

Change-Id: I7e82464dda5d7d9f03e634eef814c63b661bb0c4
2018-07-12 15:24:42 -03:00
Zuul 93f57b7e3c Merge "Adding Spark 2.3" 2018-07-11 23:22:29 +00:00
Jeremy Freudberg 7910521a7e Totally rewrite s3_hadoop
Remedying of patchings, version conflicts, classpath issues, etc.

ALSO: Switch the Hadoop libraries used on the Spark standalone plugin to
Hadoop 2.7.3. The version was previously 2.6.5, to match Cloudera's
so-called "Hadoop 2.6.0", but in fact this concordance is not at all
necessary...

Change-Id: Iafafb64fd60a1ae585375a68173c84fbb82c7e1f
2018-07-02 11:51:28 -04:00
Telles Nobrega 0421dfb8e5 Adding Spark 2.3
Adding ability to create Spark 2.3

Change-Id: I5fcfd050121b563960a48a3c8dbad9dda19b2a3e
2018-06-26 09:14:44 -03:00
Zhuang Changkun 83224a6c5e Add support to build 3.0.1 image
Change-Id: I1d90099b22ed9a0ede08df45059e6d7848c237b8
2018-06-08 14:11:37 +08:00
Luigi Toscano 5d05a3db2f Remove CentOS6/RHEL6 support
Deprecated in Pike, and removed from diskimage-builder for a while.

Story: 2001996
Task: 19626
Change-Id: I9b4eab56fcbfcb2e97805fba805eafeeafa5d540
2018-05-15 18:22:50 +02:00
Zhuang Changkun 5707f5cc5a Add support to build 2.7.5 image
Change-Id: Ia16692632cbe6380d4021dd031eed09362873905
2018-04-17 19:26:08 +08:00
Luigi Toscano d987717768 Migration to Storyboard
Sahara projects migrated to storyboard.openstack.org.
Replace the references to Launchpad, including the bugs now
available as stories.
Fix a reference to github.
Change-Id: Iadba69efc1e310b6a19463d3398bf5c6549acd73
2018-03-17 14:40:54 +01:00
Shu Yingya 12efcf84e2 Add support to build 2.8.2 image
Also tweak Hive a bit and refer to artifacts in a new (but not totally
ideal) location.

Co-Authored-By: Jeremy Freudberg <jeremyfreudberg@gmail.com>
Change-Id: I3a25ee8c282849911089adf6c3593b1bb50fd067
2018-03-01 09:51:01 +08:00
Luigi Toscano 8cdff5d71f Update the links to artifacts (removing sahara-files too)
The canonical location for the artifacts going on is
tarballs.openstack.org/sahara-extra/, so fix the link to use that
and also use https.

Moreover, since the last tarball required for building images
is available on tarballs.openstack.org, remove the last references
to sahara-files for artifacts and documentation
(the new location was used already for a while in few places).
there are still few references to sahara-files,
but they are all about CentOS6 which is no more supported
by diskimage-builder, and should be removed separately.

Change-Id: Iab5a4d50a0abc6ab278837b6a9efd5e30f31c44a
2018-02-27 12:14:09 +01:00
Zuul 8a9185373f Merge "Revise s3_hadoop" 2018-02-08 18:47:14 +00:00
Jeremy Freudberg 3ee20cbc09 Revise s3_hadoop
* Handle Hadoop classpath better
* Include proper support for Spark classpath
* Formally limit element's use to Vanilla and Spark

Change-Id: I65abd7e375dba11599a4ab943d24f878235cd71d
Closes-Bug: #1727757
Closes-Bug: #1728061
2018-02-07 05:58:44 +00:00
Telles Nobrega 05085a81db Adding Spark 2.2.0
Adding newest version of spark.

Change-Id: Ib2894d5d93d3ecfd17e0fb1eba4d687a97406027
2017-12-13 09:55:41 -03:00
Telles Nobrega 589abcce31 Adding generation of Storm 1.1.1
We are adding the newest version of Storm to image generation.

Change-Id: I3e8459edc6fa94675f5b8f4b8519f11f8f2e3436
2017-12-11 10:31:49 -03:00
Mohammed Naser 5b011dcbcc
Allow control of image output format
In certain cases, it is preferable to build a different format of
image such as using Ceph RBD backend which prefers a RAW image
format.

This option allows passing that value over to DIB in order to
control the final image format.

Change-Id: I851fc222b5e8a77d148c5f9d53c2688b17e6e96f
2017-08-25 14:09:34 -04:00
Jenkins d12f6167bb Merge "Add S3 jar to Hadoop classpath" 2017-07-28 17:49:18 +00:00
Jeremy Freudberg a77a9a978a Add S3 jar to Hadoop classpath
As prereq of support for S3 datasource, the hadoop-aws jar needs to be
in the Hadoop classpath. The jar is copied into the proper folder when
possible on the appropriate plugins, and otherwise can be provided from
a download URL by the user.

Additionally, set the correct value of DIB_HDFS_LIB_DIR on the Vanilla
plugin to avoid any unnecessary simlinking.

Partially-Implements: bp sahara-support-s3

Change-Id: I94c5b0055b87f6a4e1382118d0718e588fccfe87
2017-07-28 14:01:01 +00:00
Jenkins 796a48490b Merge "Configurable Spark version for Vanilla image" 2017-07-27 12:06:42 +00:00
Jenkins 5c64e8e345 Merge "Add support to create CDH 5.11 images" 2017-07-25 18:24:32 +00:00
Evgeny Sikachov 8f2caee358 Add support to create CDH 5.11 images
Change-Id: Ibd61ca89bb24e369cca5d06e338edef2f0c6f6d6
2017-07-25 17:26:40 +00:00
Jenkins 588e1c7d69 Merge "Change default link for Hadoop native libs" 2017-07-25 16:28:38 +00:00
Jeremy Freudberg a90fde8cc6 Change default link for Hadoop native libs
Now hosted on tarballs.o.o. Plus, for Hadoop 2.7.1, correct the content
of this tarball.

Change-Id: Ib42df14dcb62548082d38fb36f3928d632ee4da5
Closes-Bug: #1705942
2017-07-24 20:34:37 +00:00
Jeremy Freudberg c450ba9b18 Build Xenial images for Vanilla and Storm
Support for Ubuntu 16.04 is simple. Implement this and make it the
default building behavior.

Change-Id: I0e0f279afb88521f119365c2ea2cb9ce6320f58a
2017-07-19 19:49:17 +00:00
Jeremy Freudberg ca8767104b Configurable Spark version for Vanilla image
Instead of hard-coded Spark 1.6.0, allow use of DIB_SPARK_VERSION (and
its corresponding CLI argument `-s`) to specify which version of Spark
should be included on images for Vanilla plugin.

Change-Id: Ia7c7027c9eadfc9d724733a503990ca78e487ee9
2017-07-18 13:00:11 +00:00
Jenkins 3425012a84 Merge "Adding Spark 2.1.0" 2017-07-15 19:31:07 +00:00
Luigi Toscano 802379aba6 Deprecate and don't build CentOS 6 images by default
Following the approved spec:
- do not automatically build centos (CentOS 6) images when no
  operating system is specified for a specific plugin;
- if 'centos' is explicitly specified, print out a warning message.
- whenever 'centos' is explicitly specified by the gate script,
  do not build it anymore.
Blueprint: deprecate-centos6-images
Change-Id: I4e11b97061d6e1f9804bae0157a345ed484d7dbe
2017-07-13 10:05:19 +02:00
Telles Nobrega d6cf14a3f3 Adding Spark 2.1.0
Support to Spark 2.1.0 was accepted on the sahara side, now we need to
create the image for it.

Change-Id: I9ea30414811b272fb7bb236111aeca5961818653
2017-07-11 09:49:01 -03:00
Vitaly Gridnev 972ce02c76 Fixing spark and CDH refinements
Actually there is no much difference between using CDH 5.4 and 5.5
for spark, because both are using Hadoop 2.6 as a base.
Also removing redundant case in installing packages for CDH.

Change-Id: Ie24bb72365352edb22d94a461df0a0af6cd71806
Closes-bug: 1686400
2017-06-07 20:51:34 +00:00
Jenkins 7b1e5fb03d Merge "Adding ability to create Storm 1.1.0 image" 2017-05-02 10:07:21 +00:00
Shu Yingya 20126fbde6 Remove some codes of older version builder
Building CDH image under version 5.5.0 is no longer support.
Remove these useless code.
Also, adding the ambari usage info in sahara-image-create
command.

Change-Id: I6fffe25ee9daf651355611be675137babb67e2a8
2017-04-14 11:24:05 +08:00
Telles Nobrega 5e1f45c373 Adding ability to create Storm 1.1.0 image
Change-Id: I5d0c5c600fda5d54d2b6bb8cac9b0f44c24f691a
2017-04-12 09:48:08 -03:00
Vitaliy Levitski 35f7276131 Remove support for MapR 5.0.0 images
Implements: blueprint remove-mapr-500
Change-Id: I6ce676240d053941de6eaafdddd18d3190f353f7
2017-02-01 15:54:44 +00:00
Luigi Toscano 3dfcb641c0 Set DIB_CDH_MINOR_VERSION when required
Set DIB_CDH_MINOR_VERSION in diskimage-create so that it could not
be an empty value. The order of the lines where DIB_CDH_VERSION is
assigned for other CDH versions where DIB_CDH_MINOR_VERSION is not
used was changed as well for consistency.
Also, in order to try to prevent future errors, set the value of
DIB_CDH_MINOR_VERSION based on DIB_CDH_VERSION in all elements where
it is used (thanks to the ${VAR:-value} syntax which does not fail
with set -u even if DIB_CDH_MINOR_VERSION is not assigned).

Closes-Bug: #1657482
Change-Id: I31b25fd4ba886d051b9b57902cd72349a4a2dbfa
2017-01-19 19:46:57 +01:00
Jenkins a94b9bbb85 Merge "use latest ambari in images" 2017-01-10 14:37:21 +00:00
Jenkins e6845532bf Merge "Remove the older versions from the builder" 2016-12-26 13:34:03 +00:00
Vitaly Gridnev 6e4afa43d6 include netcat package for centos images
sahara is using netcat for indirect access from
cluster's instances. so, additionally include netcat package
for proper work of indirect access

Closes-bug: 1649644
Change-Id: Ifb255b3c51fd7810730d5809dab913ef0fcd0f17
2016-12-19 14:34:44 +00:00
Luigi Toscano 6a293ea6db Remove the older versions from the builder
Those versions are removed or disabled in the current master, and anyway
the older branches of sahara-image-elements still can build them.

Change-Id: I4e14662465e96ac223792a2f40e3925a7ecbebd5
2016-12-14 10:58:21 +01:00
Vitaly Gridnev a71277b9a1 use latest ambari in images
that will be required to support HDP 2.5 deployment.

Change-Id: Iab44589e4a641763cb3f0dc56ffa821a1d65c2e2
2016-12-14 13:36:57 +04:00
Jenkins 8aa0ce16ca Merge "[DOC] Update support plugin version into doc" 2016-12-14 08:59:27 +00:00
Shu Yingya 604857af49 [DOC] Update support plugin version into doc
Update the plugin version we support(5.7) and will support(5.9) into doc.

Though "DIB_LOCAL_IMAGE" is defined in project "diskimage-builder",
add a specification in docs will be helpful.

Partially-Implements: blueprint cdh-5-9-support
Change-Id: I11ada0c05d426ea483f7c3b7ffa5b247eb77133c
2016-12-13 15:20:18 +00:00
yingya.shu 4cc289bc0c [CDH] Support CDH 5.9
Added support of CDH 5.9 for ubuntu, centos and centos7.

Change-Id: I14aad25829d011e14f37bf0ff50e50b157681163
bp: cdh-5-9-support
2016-12-09 11:05:54 +00:00
Luigi Toscano 9ac38b939a Force trusty on Spark, Vanilla and Storm Ubuntu images
The other images (Ambari, CDH, MapR) do not support Xenial.
Spark depends on CDH, and while Vanilla and Storm may support Xenial,
there are other packages and dependencies that need to be fixed first.

Change-Id: Ied720e35d2a5372f228861b104cca1afddaf26f9
2016-11-25 15:47:44 +01:00
Jenkins 6561879886 Merge "Add support for building MapR 5.2 prebuilt images" 2016-09-14 21:10:12 +00:00
Jenkins 74403e7ed2 Merge "add element for kerberos artifacts" 2016-09-08 19:25:03 +00:00
Jenkins 87d7000744 Merge "Merge Vanilla and Spark plugins" 2016-09-07 13:48:27 +00:00
Mikhail Lelyakin b7d83da204 Merge Vanilla and Spark plugins
Add spark element to vanilla images.
It provides run spark jobs on vanilla
clusters.

bp spark-jobs-for-vanilla-hadoop
Change-Id: Ie5eb9ec10b0052c9d1f6284b312edfee0ddba4f0
2016-09-07 12:12:58 +00:00