Commit Graph

420 Commits

Author SHA1 Message Date
Jeremy Freudberg ded80db266 Fix URL of Maven Central Repository
Change-Id: I7accc23d6e91ff4a525c31616033f76c07dec21e
Story: 2007996
Task: 40639
2020-08-10 18:21:16 +00:00
Luigi Toscano d0b1d24be4 Fix linting (both dib-lint and bashate)
- dib-lint: make sure every element has a README.rst file
- bashate: ignore E006 (line too long).

Change-Id: I53a334b05a30e98f1df19504cee616e3d145e9e3
2020-04-17 01:09:43 +02:00
Luigi Toscano 48491edb72 archive-primary.cloudera.com -> archive.cloudera.com
The former is no more active.

Story: 2004936
Task: 29329
Change-Id: I260bf4cd3840ba2c44b85df87b89b717831b3b5c
2019-02-04 22:07:02 +01:00
Luigi Toscano 578cb4824b firstboot: make rc-local start after cloud-init
The change is required because the cloud-init user (centos, ubuntu,
cloud-user, ...) is used in the firstboot code.
All distributions where vanilla can be deployed are based
on systemd.

Story: 2004479
Task: 28194

Change-Id: I9d8a626b84d5d3c2a91348895cded5fd32ded52a
2018-11-30 23:20:01 +01:00
Telles Nobrega 34e8f4a27a Updating Storm version list
We are adding Storm 1.2.0 and 1.2.1 and removing 0.9.2

Change-Id: I7e82464dda5d7d9f03e634eef814c63b661bb0c4
2018-07-12 15:24:42 -03:00
Jeremy Freudberg 7910521a7e Totally rewrite s3_hadoop
Remedying of patchings, version conflicts, classpath issues, etc.

ALSO: Switch the Hadoop libraries used on the Spark standalone plugin to
Hadoop 2.7.3. The version was previously 2.6.5, to match Cloudera's
so-called "Hadoop 2.6.0", but in fact this concordance is not at all
necessary...

Change-Id: Iafafb64fd60a1ae585375a68173c84fbb82c7e1f
2018-07-02 11:51:28 -04:00
Zhuang Changkun 83224a6c5e Add support to build 3.0.1 image
Change-Id: I1d90099b22ed9a0ede08df45059e6d7848c237b8
2018-06-08 14:11:37 +08:00
Luigi Toscano 5d05a3db2f Remove CentOS6/RHEL6 support
Deprecated in Pike, and removed from diskimage-builder for a while.

Story: 2001996
Task: 19626
Change-Id: I9b4eab56fcbfcb2e97805fba805eafeeafa5d540
2018-05-15 18:22:50 +02:00
Zuul aa3ff84f2b Merge "Add support to build 2.7.5 image" 2018-05-10 14:41:00 +00:00
Luigi Toscano 30717f2516 Fix (again) the detection of scala version (now https)
The www.scala-lang.org switched to https and the curl call did not
account for this scenario, so it failed.
Switch the address to https and as extra measure add also the -L
argument, which forces curl to follow the redirects.

Switch also the download URL to https.

Story: 2001964
Task: 15089
Change-Id: I82d3dd0bd000284fb5404483bf3ee34a6fbb147b
2018-05-04 17:55:50 +02:00
Zhuang Changkun 5707f5cc5a Add support to build 2.7.5 image
Change-Id: Ia16692632cbe6380d4021dd031eed09362873905
2018-04-17 19:26:08 +08:00
Zuul ce56817439 Merge "Add support to build 2.8.2 image" 2018-03-15 14:40:31 +00:00
Elise Gafford 859baa5b0a Preload soci-mysql and soci on RHEL7 images
Change-Id: I00d78b45dffd785e956058ca7d76fb9e32bfac60
Closes-bug: #1754313
2018-03-09 20:49:35 -03:00
Shu Yingya 12efcf84e2 Add support to build 2.8.2 image
Also tweak Hive a bit and refer to artifacts in a new (but not totally
ideal) location.

Co-Authored-By: Jeremy Freudberg <jeremyfreudberg@gmail.com>
Change-Id: I3a25ee8c282849911089adf6c3593b1bb50fd067
2018-03-01 09:51:01 +08:00
Luigi Toscano 8cdff5d71f Update the links to artifacts (removing sahara-files too)
The canonical location for the artifacts going on is
tarballs.openstack.org/sahara-extra/, so fix the link to use that
and also use https.

Moreover, since the last tarball required for building images
is available on tarballs.openstack.org, remove the last references
to sahara-files for artifacts and documentation
(the new location was used already for a while in few places).
there are still few references to sahara-files,
but they are all about CentOS6 which is no more supported
by diskimage-builder, and should be removed separately.

Change-Id: Iab5a4d50a0abc6ab278837b6a9efd5e30f31c44a
2018-02-27 12:14:09 +01:00
Telles Nobrega 6229ee0de9 Disables CA checking for Ambari on Centos/RHEL
The default SSL certificate, which is generated when the Ambari server is
installed, is invalid.

We are disabling check for now.

Change-Id: Ifcbc931e2ca23cb1fe221d509f57750e7e060aea
Partial-bug: #1748507
2018-02-13 10:43:11 +01:00
Zuul 8a9185373f Merge "Revise s3_hadoop" 2018-02-08 18:47:14 +00:00
Jeremy Freudberg 3ee20cbc09 Revise s3_hadoop
* Handle Hadoop classpath better
* Include proper support for Spark classpath
* Formally limit element's use to Vanilla and Spark

Change-Id: I65abd7e375dba11599a4ab943d24f878235cd71d
Closes-Bug: #1727757
Closes-Bug: #1728061
2018-02-07 05:58:44 +00:00
Telles Nobrega dfc914ccfe Adding fuse package to Ambari
In order for Ambari to work properly on Rhel we need to install fuse package
on the image.

Change-Id: I27b661b2da83776e382da962c89b9098599524d0
Closes-bug: #1746712
2018-02-01 08:38:48 -03:00
Shu Yingya e5f50ed163 Fix starting hive failure in Ubuntu xenial
The mysql configuration file name and directory change in
Ubuntu 16.04. my.cnf becomes an entry for mysql configuration.
The new file locates here:
  "/etc/mysql/mysql.conf.d/mysqld.cnf"

Change-Id: I8f70589cfc904a7d30081172b313ad8e30914935
Closes-Bug: #1738124
2018-01-18 21:59:33 +08:00
Telles Nobrega 05085a81db Adding Spark 2.2.0
Adding newest version of spark.

Change-Id: Ib2894d5d93d3ecfd17e0fb1eba4d687a97406027
2017-12-13 09:55:41 -03:00
Jeremy Freudberg a77a9a978a Add S3 jar to Hadoop classpath
As prereq of support for S3 datasource, the hadoop-aws jar needs to be
in the Hadoop classpath. The jar is copied into the proper folder when
possible on the appropriate plugins, and otherwise can be provided from
a download URL by the user.

Additionally, set the correct value of DIB_HDFS_LIB_DIR on the Vanilla
plugin to avoid any unnecessary simlinking.

Partially-Implements: bp sahara-support-s3

Change-Id: I94c5b0055b87f6a4e1382118d0718e588fccfe87
2017-07-28 14:01:01 +00:00
Jenkins 5c64e8e345 Merge "Add support to create CDH 5.11 images" 2017-07-25 18:24:32 +00:00
Evgeny Sikachov 8f2caee358 Add support to create CDH 5.11 images
Change-Id: Ibd61ca89bb24e369cca5d06e338edef2f0c6f6d6
2017-07-25 17:26:40 +00:00
Jeremy Freudberg c450ba9b18 Build Xenial images for Vanilla and Storm
Support for Ubuntu 16.04 is simple. Implement this and make it the
default building behavior.

Change-Id: I0e0f279afb88521f119365c2ea2cb9ce6320f58a
2017-07-19 19:49:17 +00:00
Luigi Toscano 83c3011f69 mapr: fix the discovery of the version of Scala
The structure of the page changed slightly between July 8th
and now (thanks Internet Archive).
Adapt the extraction code. including a more robust regex.

Also update the download location (lightbend.com instead of
typesafe.com, even if the latter is still a redirect).

Finally, expose a new variable which allows users to
force a specific version of Scala, disabling the autodetection.
This is useful to workaround possible future similar breakages
without changing the code.

Closes-Bug: #1704337

Change-Id: Ib105d08816d6833f05301e59f64163de5f070109
2017-07-14 18:12:37 +02:00
Luigi Toscano 00f9124e3c Get files from tarballs.o.o if possible (extjs, policy)
Few artifacts are already available on tarballs.openstack.org; use
them instead of sahara-files (the others should be migrated as well).

Change-Id: I5151b43fab56a071793097dba72aca6e9fb19196
2017-07-10 16:37:56 +02:00
Luigi Toscano 52c39ec6d0 Remove the usage of 'which' (following dib)
diskimage-builder deprecated the usage of which (dib-lint marks its
occurrences as error by default):
https://review.openstack.org/#/c/466063/

Replace all the occurences of 'which' with 'type -p'.
Also, remove the (now useless) installation of which from
the disable-firewalld element.

Change-Id: I00bda062767070e46133cc800e82c9470923f572
2017-07-10 16:37:41 +02:00
Jenkins d0bfaa0fa9 Merge "Fix an outdated link" 2017-06-08 09:14:31 +00:00
Vitaly Gridnev 972ce02c76 Fixing spark and CDH refinements
Actually there is no much difference between using CDH 5.4 and 5.5
for spark, because both are using Hadoop 2.6 as a base.
Also removing redundant case in installing packages for CDH.

Change-Id: Ie24bb72365352edb22d94a461df0a0af6cd71806
Closes-bug: 1686400
2017-06-07 20:51:34 +00:00
Jeremy Freudberg 844f5d7adb Fix an outdated link
In [1], we changed the default link
to ExtJS to be on sahara-files.

Now, update the README to reflect it.

[1] https://review.openstack.org/#/c/336024/
Depends-On: I6b8e12f0d33752e7790c5f40ebe6a3940ff1b0c1
Change-Id: I473257476b53e50ad0659f058cddf9bb2f78d917
2017-05-30 16:11:12 -04:00
Jenkins 7b1e5fb03d Merge "Adding ability to create Storm 1.1.0 image" 2017-05-02 10:07:21 +00:00
Luigi Toscano ffb7f68895 Fix: set the Fedora-specific ssh_config file for augeas
The Host entry in /etc/ssh/ssh_config file on Fedora 25+ is commented
because there is an Include directive and the custom settings have
been moved to a file there.
This leads to a failure when the augeas lens tries to write to that
files. If I read the schema correctly, the lens assumes that the Host
entry is available/uncommented (this it was the case before in Fedora,
and in CentOS and RHEL, and in Debian/Ubuntu, where a customization
is a applied to the upstream default file, which is used as it is now
in Fedora).
This maybe could be an issue of the lens which may not assume that
key, but I'm not sure whether is possible.
In the meantime, use the existing customization file on Fedora,
instead of /etc/ssh/ssh_config.

Change-Id: Iee5c109b1265b54a75f675d26c9fec49c777d298
2017-04-27 15:23:07 +02:00
Luigi Toscano 8ec0c00ae0 MapR: do not match "all" if a specific version is available
The "all" entry is a fallback if an entry/override for a specific
version has not been specified.

Change-Id: I5e57a00023cddaed691a2c701dc2e7fbd4726fe4
Closes-Bug: #1686343
2017-04-26 10:44:01 +02:00
Jenkins b9cf0fc089 Merge "Remove some codes of older version builder" 2017-04-21 10:06:01 +00:00
Shu Yingya 105577b6ca Fix 'navigator-keytrustee' repo not found error
Build an image with DIB_CDH_MINOR_VERSION may fail
because `navigator-keytrustee` doesn't release
minor version sometimes, and then the repo will not
be found.
This patch determines whether the repo exists before
writing repo url.

Change-Id: I6bcd66264019847a08c2ba4c15da401567e57574
Closes-Bug: 1682754
2017-04-17 06:57:25 +00:00
Shu Yingya 20126fbde6 Remove some codes of older version builder
Building CDH image under version 5.5.0 is no longer support.
Remove these useless code.
Also, adding the ambari usage info in sahara-image-create
command.

Change-Id: I6fffe25ee9daf651355611be675137babb67e2a8
2017-04-14 11:24:05 +08:00
Telles Nobrega 5e1f45c373 Adding ability to create Storm 1.1.0 image
Change-Id: I5d0c5c600fda5d54d2b6bb8cac9b0f44c24f691a
2017-04-12 09:48:08 -03:00
Jenkins 2113ac4861 Merge "MapR 5.1: remove non-existing version of hue" 2017-04-06 18:52:25 +00:00
Luigi Toscano 0498254076 mysql: fix the packages mapping on RHEL7 (mariadb)
RHEL7 configuration is the same as CentOS7 (or the other way around).
uses mariadb like CentOS7

Closes-Bug: #1675119
Change-Id: I7ea7a6f327f69a7271b605733ba95ef296bebef7
2017-03-22 19:15:57 +01:00
Luigi Toscano 5541a5c764 MapR 5.1: remove non-existing version of hue
The version 3.7.0 of Hue (or better, its packages) are not available
and the build fails on CentOS 7 and RHEL7.

Closes-Bug: #1675047
Change-Id: Ifd5d8c96dd62cc6def8b785c932221c4987aac62
2017-03-22 19:15:28 +01:00
Jenkins 09298bdae4 Merge "Fix documentation for default Ambari version value" 2017-03-10 10:30:38 +00:00
Luigi Toscano 591bfed8f0 Fix: DIB_RELEASE is not defined on CentOS
Otherwise an "unbound variable" error is thrown.

Change-Id: I1e29cce101e44fc82e27b5e7986101847be0da8d
2017-02-27 11:22:34 +01:00
Mohammed Naser e68607193f Fix documentation for default Ambari version value
The documented default version is much older than the real
default value which was bumped a while back but the README.rst
was not updated.  This patch fixes this issue.

Change-Id: I7a8aa7a0ed2234fdf48a2dec5c300fbaf1783dd3
2017-02-23 09:28:44 -05:00
Vitaliy Levitski 35f7276131 Remove support for MapR 5.0.0 images
Implements: blueprint remove-mapr-500
Change-Id: I6ce676240d053941de6eaafdddd18d3190f353f7
2017-02-01 15:54:44 +00:00
Vitaliy Levitski c7045335ec Upgrade packages for MapR 5.2 according to latest versions pack MEP 2.0
Implements: blueprint add-new-mapr-services
Change-Id: I8998178edd10f06b75980d98697a205ee0b75400
2017-01-31 15:02:31 +00:00
Luigi Toscano 3dfcb641c0 Set DIB_CDH_MINOR_VERSION when required
Set DIB_CDH_MINOR_VERSION in diskimage-create so that it could not
be an empty value. The order of the lines where DIB_CDH_VERSION is
assigned for other CDH versions where DIB_CDH_MINOR_VERSION is not
used was changed as well for consistency.
Also, in order to try to prevent future errors, set the value of
DIB_CDH_MINOR_VERSION based on DIB_CDH_VERSION in all elements where
it is used (thanks to the ${VAR:-value} syntax which does not fail
with set -u even if DIB_CDH_MINOR_VERSION is not assigned).

Closes-Bug: #1657482
Change-Id: I31b25fd4ba886d051b9b57902cd72349a4a2dbfa
2017-01-19 19:46:57 +01:00
Jenkins a94b9bbb85 Merge "use latest ambari in images" 2017-01-10 14:37:21 +00:00
Luigi Toscano 42d79a9828 Disable firewalld (also on centos7) when it is enabled
Calling systemctl disable on a non-existing unit leads to a failure due
to `set -e`. This checks prevents it.
Also applies the same codepath to centos7 (as it is for rhel7, also
because centos7 has currently test coverage).

Closes-Bug: #1653535
Change-Id: I21f9cfd1959dcde80ed6abbf410bd0c4e93cc803
2017-01-02 18:38:30 +01:00
Vitaly Gridnev 3ba71becd2 fix firstboot script
implement required changes to adopt firstboot script for
CentOS 7.x. Since lsb_release command is not available on
CentOS 7.x, we will include redhat-lsb-core package to image.

Change-Id: I2e5f2d598fbb9b3af0fcfd6e69a09e1ebce8e52d
2016-12-22 08:58:20 +00:00