Remove IDH support

Partially implements blueprint: remove-idh-plugin

Change-Id: Iac90c3ec88e47f225500617afc55e98d0a719940
This commit is contained in:
Matthew Farrellee 2014-04-11 14:37:42 -04:00
parent 3f9f45929c
commit 89c44f543c
9 changed files with 5 additions and 509 deletions

View File

@ -1,7 +1,7 @@
Diskimage-builder script for creation cloud images
==================================================
This script builds Ubuntu, Fedora, CentOS cloud images for use in Sahara. By default the all plugin are targeted, all images will be built. The '-p' option can be used to select plugin (vanilla, spark, idh or hdp). The '-i' option can be used to select image type (ubuntu, fedora or centos). The '-v' option can be used to select hadoop version (1, 2 or plain).
This script builds Ubuntu, Fedora, CentOS cloud images for use in Sahara. By default the all plugin are targeted, all images will be built. The '-p' option can be used to select plugin (vanilla, spark or hdp). The '-i' option can be used to select image type (ubuntu, fedora or centos). The '-v' option can be used to select hadoop version (1, 2 or plain).
NOTE: You should use Ubuntu or Fedora host OS for building images, CentOS as a host OS has not been tested well.
@ -25,7 +25,7 @@ For users:
.. sourcecode:: bash
sudo bash sahara-image-elements/diskimage-create/diskimage-create.sh -p [vanilla|spark|hdp|idh]
sudo bash sahara-image-elements/diskimage-create/diskimage-create.sh -p [vanilla|spark|hdp]
5. To select which hadoop version to target use the '-v' commandline option like this:
@ -44,9 +44,8 @@ NOTE for 4, 5, 6:
For Vanilla you can create ubuntu, fedora and centos cloud image with hadoop 1.x.x and 2.x.x versions. Use environment variables 'DIB_HADOOP_VERSION_1' and 'DIB_HADOOP_VERSION_2' to change defaults.
For Spark you can create only ubuntu image with one hadoop version. You shouldn't specify image type and hadoop version.
For HDP you can create only centos image with hadoop 1.3.0 or 2.0 and without hadoop ('plain' image). You shouldn't specify image type.
For IDH you can create only centos image with one hadoop version. You shouldn't specify image type and hadoop version.
NOTE for CentOS images (for vanilla, hdp and idh plugins):
NOTE for CentOS images (for vanilla and hdp plugins):
Resizing disk space during firstboot on that images fails with errors (https://bugs.launchpad.net/sahara/+bug/1304100). So, you will get an instance that will have a small available disk space. To solve this problem we build images with 10G available disk space as default. If you need in more available disk space you should export parameter DIB_IMAGE_SIZE:

View File

@ -26,7 +26,7 @@ while getopts "p:i:v:d:" opt; do
*)
echo
echo "Usage: $(basename $0)"
echo " [-p vanilla|spark|hdp|idh]"
echo " [-p vanilla|spark|hdp]"
echo " [-i ubuntu|fedora|centos]"
echo " [-v 1|2|plain]"
echo " [-d true|false]"
@ -48,7 +48,7 @@ done
# Checks of input
if [ -n "$PLUGIN" -a "$PLUGIN" != "vanilla" -a "$PLUGIN" != "spark" -a "$PLUGIN" != "hdp" -a "$PLUGIN" != "idh" ]; then
if [ -n "$PLUGIN" -a "$PLUGIN" != "vanilla" -a "$PLUGIN" != "spark" -a "$PLUGIN" != "hdp" ]; then
echo -e "Unknown plugin selected.\nAborting"
exit 1
fi
@ -77,10 +77,6 @@ if [ "$PLUGIN" = "vanilla" -a "$HADOOP_VERSION" = "plain" ]; then
exit 1
fi
if [ "$PLUGIN" = "idh" -a "$HADOOP_VERSION" = "plain" ]; then
echo "Impossible combination.\nAborting"
exit 1
fi
#################
if [ -e /etc/os-release ]; then
@ -331,46 +327,5 @@ if [ -z "$PLUGIN" -o "$PLUGIN" = "hdp" ]; then
fi
fi
########################
# Image for IDH plugin #
########################
if [ -z "$PLUGIN" -o "$PLUGIN" = "idh" ]; then
# Ignoring image type and hadoop version options
echo "For idh plugin option -i is ignored"
export DIB_IMAGE_SIZE=${IMAGE_SIZE:-"10"}
export BASE_IMAGE_FILE="CentOS-6.4-cloud-init.qcow2"
export DIB_CLOUD_IMAGES="http://sahara-files.mirantis.com"
centos_elements_sequence="vm rhel hadoop-idh"
if [ "$platform" = 'NAME="Ubuntu"' ]; then
echo "**************************************************************"
echo "WARNING: As a workaround for DIB bug 1204824, you are about to"
echo " create a CentOS image that has SELinux "
echo " disabled. Do not use these images in production. "
echo "**************************************************************"
centos_elements_sequence="$centos_elements_sequence selinux-permissive"
suffix=".selinux-permissive"
fi
if [ -z "$HADOOP_VERSION" -o "$HADOOP_VERSION" = "1" ]; then
export HADOOP_VERSION="1"
export DIB_HADOOP_VERSION="2.5.1"
export centos_image_name_idh="centos_sahara_idh_${DIB_HADOOP_VERSION}${suffix}"
disk-image-create $centos_elements_sequence -o $centos_image_name_idh
mv $centos_image_name_idh.qcow2 ../
fi
if [ -z "$HADOOP_VERSION" -o "$HADOOP_VERSION" = "2" ]; then
export HADOOP_VERSION="2"
export DIB_HADOOP_VERSION="3.0.2"
export centos_image_name_idh="centos_sahara_idh_${DIB_HADOOP_VERSION}${suffix}"
disk-image-create $centos_elements_sequence -o $centos_image_name_idh
mv $centos_image_name_idh.qcow2 ../
fi
fi
popd # out of $TEMP
rm -rf $TEMP

View File

@ -1,13 +0,0 @@
Install the Intel Distribution Hadoop
The following script:
sahara-image-elements/diskimage-create/diskimage-create.sh
is the default script to use for creating CentOS images with IDH installed/configured.
The available versions of Intel Hadoop are 2.5.1 and 3.0.2 now. If you want build 2.5.1 version then you should specify -v as 1. If you want build 3.0.2 version then you should specify -v as 2.
In order to create the IDH images with the diskimage-create.sh script, use the following syntax to select the "idh" plugin:
diskimage-create.sh -p idh -v 1|2

View File

@ -1,3 +0,0 @@
#!/bin/bash
cat $(dirname $0)/packages-$HADOOP_VERSION | xargs install-packages

View File

@ -1,195 +0,0 @@
GeoIP
alsa-lib
apr
apr-util
apr-util-ldap
at
atk
augeas-libs
avahi-libs
bc
cairo
cdparanoia-libs
cloog-ppl
compat-readline5
cpp
createrepo
cups
cups-libs
cvs
db4-cxx
db4-devel
dbus
deltarpm
ed
expect
facter
fontconfig
foomatic
foomatic-db
foomatic-db-filesystem
foomatic-db-ppds
freetype
fuse
fuse-libs
gcc
gd
gdbm-devel
gettext
ghostscript
ghostscript-fonts
glib2
glibc
glibc-common
glibc-devel
glibc-headers
gnuplot
gnuplot-common
gnutls
gstreamer
gstreamer-plugins-base
gstreamer-tools
gtk2
hicolor-icon-theme
httpd
httpd-tools
iptraf
iso-codes
jasper-libs
kernel-headers
lcms-libs
libICE
libSM
libX11
libX11-common
libXau
libXcomposite
libXcursor
libXdamage
libXext
libXfixes
libXfont
libXft
libXi
libXinerama
libXpm
libXrandr
libXrender
libXt
libXtst
libXv
libXxf86vm
libconfuse
libfontenc
libgcc
libgenders
libgomp
libgudev1
libjpeg-turbo
libmng
libogg
liboil
libpng
libselinux-ruby
libtasn1
libthai
libtheora
libtiff
libudev
libvisual
libvorbis
libxcb
libxml2
libxml2-python
libxslt
mailcap
mailx
man
mesa-dri-drivers
mesa-dri-filesystem
mesa-dri1-drivers
mesa-libGL
mesa-libGLU
mesa-private-llvm
mpfr
mtools
nagios
nagios-common
nagios-plugins
nginx
ntp
ntpdate
openjpeg-libs
openssl
pango
patch
pax
pciutils
perl
perl-CGI
perl-ExtUtils-MakeMaker
perl-ExtUtils-ParseXS
perl-Module-Pluggable
perl-Pod-Escapes
perl-Pod-Simple
perl-Test-Harness
perl-Test-Simple
perl-Time-HiRes
perl-devel
perl-libs
perl-version
phonon-backend-gstreamer
php
php-cli
php-common
php-gd
php-xml
pixman
pkgconfig
poppler
poppler-data
poppler-utils
portreserve
ppl
puppet
puppet-server
python-deltarpm
qt
qt-sqlite
qt-x11
qt3
redhat-lsb
redhat-lsb-compat
redhat-lsb-core
redhat-lsb-graphics
redhat-lsb-printing
ruby
ruby-augeas
ruby-irb
ruby-libs
ruby-rdoc
ruby-shadow
rubygem-daemons
rubygem-fastthread
rubygem-gem_plugin
rubygem-mongrel
rubygem-rake
rubygems
sdparm
shared-mime-info
syslinux
sysstat
tcl
tftp-server
time
tmpwatch
unzip
urw-fonts
virt-what
vsftpd
xinetd
xml-common
xorg-x11-font-utils
xz
xz-lzma-compat
zip

View File

@ -1,189 +0,0 @@
1:perl-Pod-Escapes-1.04-136.el6.x86_64
1:perl-Module-Pluggable-3.90-136.el6.x86_64
1:perl-Pod-Simple-3.13-136.el6.x86_64
4:perl-libs-5.10.1-136.el6.x86_64
3:perl-version-0.77-136.el6.x86_64
4:perl-5.10.1-136.el6.x86_64
freetype-2.3.11-14.el6_3.1.x86_64
fontconfig-2.8.0-3.el6.x86_64
libjpeg-turbo-1.2.1-3.el6_5.x86_64
2:libpng-1.2.49-1.el6_2.x86_64
libICE-1.0.6-1.el6.x86_64
libSM-1.2.1-2.el6.x86_64
libtiff-3.9.4-10.el6_5.x86_64
2:libogg-1.1.4-2.1.el6.x86_64
lcms-libs-1.19-1.el6.x86_64
1:pkgconfig-0.23-9.1.el6.x86_64
alsa-lib-1.0.22-3.el6.x86_64
libmng-1.0.10-4.1.el6.x86_64
jasper-libs-1.900.1-15.el6_1.1.x86_64
openjpeg-libs-1.3-10.el6_5.x86_64
avahi-libs-0.6.25-12.el6.x86_64
libfontenc-1.0.5-2.el6.x86_64
foomatic-db-filesystem-4.0-7.20091126.el6.noarch
mesa-dri-filesystem-9.2-0.5.el6_5.2.x86_64
libXfont-1.4.5-3.el6_5.x86_64
1:xorg-x11-font-utils-7.2-11.el6.x86_64
ghostscript-fonts-5.50-23.1.el6.noarch
shared-mime-info-0.70-4.el6.x86_64
atk-1.30.0-1.el6.x86_64
nc-1.84-22.el6.x86_64
gstreamer-tools-0.10.29-1.el6.x86_64
gstreamer-0.10.29-1.el6.x86_64
1:libtheora-1.1.0-2.el6.x86_64
1:libvorbis-1.2.3-4.el6_2.1.x86_64
urw-fonts-2.4-10.el6.noarch
perl-CGI-3.51-136.el6.x86_64
cvs-1.11.23-16.el6.x86_64
at-3.1.10-43.el6_2.1.x86_64
libtasn1-2.3-3.el6_2.1.x86_64
gnutls-2.8.5-13.el6_5.x86_64
1:cups-libs-1.4.2-50.el6_4.5.x86_64
time-1.7-37.1.el6.x86_64
1:qt-4.6.2-28.el6_5.x86_64
1:qt-sqlite-4.6.2-28.el6_5.x86_64
db4-cxx-4.7.25-18.el6_4.x86_64
db4-devel-4.7.25-18.el6_4.x86_64
cdparanoia-libs-10.2-5.1.el6.x86_64
ed-1.1-3.3.el6.x86_64
liboil-0.3.16-4.1.el6.x86_64
xz-4.999.9-0.3.beta.20091007git.el6.x86_64
xz-lzma-compat-4.999.9-0.3.beta.20091007git.el6.x86_64
man-1.6f-32.el6.x86_64
1:dbus-1.2.24-7.el6_3.x86_64
portreserve-0.0.4-9.el6.x86_64
mailx-12.4-7.el6.x86_64
libXau-1.0.6-4.el6.x86_64
libxcb-1.8.1-1.el6.x86_64
tmpwatch-2.9.16-4.el6.x86_64
bc-1.06.95-1.el6.x86_64
pax-3.4-10.1.el6.x86_64
libvisual-0.4.0-9.1.el6.x86_64
libudev-147-2.51.el6.x86_64
libgudev1-147-2.51.el6.x86_64
patch-2.6-6.el6.x86_64
pixman-0.26.2-5.1.el6_5.x86_64
libgomp-4.4.7-4.el6.x86_64
gettext-0.17-16.el6.x86_64
mesa-private-llvm-3.3-0.3.rc3.el6.x86_64
libthai-0.1.12-3.el6.x86_64
xml-common-0.6.3-32.el6.noarch
iso-codes-3.16-2.el6.noarch
kernel-headers-2.6.32-431.5.1.el6.x86_64
glibc-headers-2.12-1.132.el6.x86_64
glibc-devel-2.12-1.132.el6.x86_64
gdbm-devel-1.8.0-36.el6.x86_64
perl-Test-Harness-3.17-136.el6.x86_64
perl-ExtUtils-MakeMaker-6.55-136.el6.x86_64
4:perl-devel-5.10.1-136.el6.x86_64
1:perl-ExtUtils-ParseXS-2.2003.0-136.el6.x86_64
perl-Test-Simple-0.92-136.el6.x86_64
redhat-lsb-core-4.0-7.el6.centos.x86_64
poppler-data-0.4.0-1.el6.noarch
poppler-0.12.4-3.el6_0.1.x86_64
poppler-utils-0.12.4-3.el6_0.1.x86_64
1:cups-1.4.2-50.el6_4.5.x86_64
foomatic-db-ppds-4.0-7.20091126.el6.noarch
foomatic-db-4.0-7.20091126.el6.noarch
libX11-common-1.5.0-4.el6.noarch
libX11-1.5.0-4.el6.x86_64
libXext-1.3.1-2.el6.x86_64
libXrender-0.9.7-2.el6.x86_64
cairo-1.8.8-3.1.el6.x86_64
libXi-1.6.1-3.el6.x86_64
libXfixes-5.0-3.el6.x86_64
libXcursor-1.1.13-6.20130524git8f677eaea.el6.x86_64
libXft-2.3.1-2.el6.x86_64
pango-1.28.1-7.el6_3.x86_64
libXrandr-1.4.0-1.el6.x86_64
libXinerama-1.1.2-2.el6.x86_64
libXdamage-1.1.3-4.el6.x86_64
libXt-1.1.3-1.el6.x86_64
ghostscript-8.70-19.el6.x86_64
foomatic-4.0.4-3.el6.x86_64
redhat-lsb-printing-4.0-7.el6.centos.x86_64
qt3-3.3.8b-30.el6.x86_64
libXtst-1.2.1-2.el6.x86_64
libXxf86vm-1.1.2-2.el6.x86_64
mesa-dri-drivers-9.2-0.5.el6_5.2.x86_64
mesa-libGL-9.2-0.5.el6_5.2.x86_64
mesa-dri1-drivers-7.11-8.el6.x86_64
mesa-libGLU-9.2-0.5.el6_5.2.x86_64
libXv-1.0.7-2.el6.x86_64
gstreamer-plugins-base-0.10.29-2.el6.x86_64
1:phonon-backend-gstreamer-4.6.2-28.el6_5.x86_64
1:qt-x11-4.6.2-28.el6_5.x86_64
libXcomposite-0.4.3-4.el6.x86_64
hicolor-icon-theme-0.11-1.1.el6.noarch
gtk2-2.20.1-4.el6.x86_64
redhat-lsb-graphics-4.0-7.el6.centos.x86_64
redhat-lsb-compat-4.0-7.el6.centos.x86_64
redhat-lsb-4.0-7.el6.centos.x86_64
ntpdate-4.2.6p5-1.el6.centos.x86_64
ntp-4.2.6p5-1.el6.centos.x86_64
libxslt-1.1.26-2.el6_3.1.x86_64
GeoIP-1.4.8-1.el6.x86_64
libselinux-ruby-2.0.94-5.3.el6_4.1.x86_64
augeas-libs-1.0.0-5.el6_5.1.x86_64
pciutils-3.1.10-2.el6.x86_64
compat-readline5-5.2-17.1.el6.x86_64
ruby-libs-1.8.7.352-13.el6.x86_64
ruby-1.8.7.352-13.el6.x86_64
ruby-irb-1.8.7.352-13.el6.x86_64
ruby-rdoc-1.8.7.352-13.el6.x86_64
rubygems-1.3.7-5.el6.noarch
rubygem-daemons-1.0.10-2.el6.noarch
rubygem-fastthread-1.0.7-2.el6.x86_64
rubygem-rake-0.8.7-2.1.el6.noarch
rubygem-gem_plugin-0.2.3-3.el6.noarch
ruby-augeas-0.4.1-1.el6.x86_64
ruby-shadow-1.4.1-13.el6.x86_64
virt-what-1.11-1.2.el6.x86_64
facter-1.6.18-3.el6.x86_64
puppet-2.7.23-1.el6.noarch
1:tcl-8.5.7-6.el6.x86_64
libXpm-3.5.10-2.el6.x86_64
gd-2.0.35-11.el6.x86_64
nginx-1.0.15-5.el6.x86_64
expect-5.44.1.15-5.el6_4.x86_64
puppet-server-2.7.23-1.el6.noarch
rubygem-mongrel-1.1.5-3.el6.x86_64
libconfuse-2.7-4.el6.x86_64
apr-1.3.9-5.el6_2.x86_64
apr-util-1.3.9-3.el6_0.1.x86_64
nagios-common-3.5.1-1.el6.x86_64
php-common-5.3.3-27.el6_5.x86_64
php-cli-5.3.3-27.el6_5.x86_64
httpd-tools-2.2.15-29.el6.centos.x86_64
apr-util-ldap-1.3.9-3.el6_0.1.x86_64
mailcap-2.1.31-2.el6.noarch
httpd-2.2.15-29.el6.centos.x86_64
php-5.3.3-27.el6_5.x86_64
vsftpd-2.2.2-11.el6_4.1.x86_64
libxml2-python-2.7.6-14.el6.x86_64
deltarpm-3.5-0.5.20090913git.el6.x86_64
python-deltarpm-3.5-0.5.20090913git.el6.x86_64
createrepo-0.9.9-18.el6.noarch
iptraf-3.0.1-14.el6.x86_64
php-gd-5.3.3-27.el6_5.x86_64
sysstat-9.0.4-22.el6.x86_64
php-xml-5.3.3-27.el6_5.x86_64
4:perl-Time-HiRes-1.9721-136.el6.x86_64
rrdtool-perl-1.4.5-1.wrl.x86_64
rrdtool-1.4.5-1.wrl.x86_64
sdparm-1.04-1.1.el6.x86_64
2:xinetd-2.3.14-39.el6_4.x86_64
tftp-server-0.49-7.el6.x86_64
mtools-4.0.12-1.el6.x86_64
syslinux-4.02-8.el6.x86_64
unzip-6.0-1.el6.x86_64
zip-3.0-1.el6.x86_64
fuse-2.8.3-4.el6.x86_64
fuse-libs-2.8.3-4.el6.x86_64
ppl-0.10.2-11.el6.x86_64
cloog-ppl-0.15.7-1.2.el6.x86_64
mpfr-2.4.1-6.el6.x86_64
cpp-4.4.7-4.el6.x86_64
gcc-4.4.7-4.el6.x86_64
gnuplot-common-4.2.6-2.el6.x86_64
gnuplot-4.2.6-2.el6.x86_64

View File

@ -1,12 +0,0 @@
#!/bin/bash
mkdir /root/old-repo
mv /etc/yum.repos.d/* /root/old-repo
install -D -g root -o root -m 644 $(dirname $0)/Centos-Vault.repo /etc/yum.repos.d/
install-packages wget createrepo
mkdir -p /var/repo/intel
wget -P /var/repo/intel --no-parent -r -l inf --content-disposition -nH --cut-dirs=5 http://repo2.intelhadoop.com/evaluation/en/RHEL/$DIB_HADOOP_VERSION/rpm/
createrepo /var/repo/intel
install -D -g root -o root -m 644 $(dirname $0)/Intel.repo /etc/yum.repos.d/

View File

@ -1,34 +0,0 @@
[C6.4-base]
name=CentOS-6.4 - Base
baseurl=http://vault.centos.org/6.4/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=1
[C6.4-updates]
name=CentOS-6.4 - Updates
baseurl=http://vault.centos.org/6.4/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=1
[C6.4-extras]
name=CentOS-6.4 - Extras
baseurl=http://vault.centos.org/6.4/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=1
[C6.4-contrib]
name=CentOS-6.4 - Contrib
baseurl=http://vault.centos.org/6.4/contrib/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=1
[C6.4-centosplus]
name=CentOS-6.4 - CentOSPlus
baseurl=http://vault.centos.org/6.4/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
enabled=1

View File

@ -1,12 +0,0 @@
[intel]
name = Intel-Distribution-for-Apache-Hadoop
baseurl = file:///var/repo/intel
proxy = _none_
gpgcheck = 0
metadata_expire = 300
fastestmirror_enabled = 1
failovermethod = roundrobin
priority = 1
enabled = 1
retries = 5
timeout = 10