openstack
/
sahara
Code Issues Proposed changes

doc: restructure the image building documentation 

Main goal: consolidate the information about image
building under the same documentation page, and move
plugin-specific details inside plugin pages.
No plugin-specific information should live outside
those pages.

More details:
- move the detailed documentation about sahara-image-pack
  from the contributor guide to the new dedicated page
  in the user manual;
- remove the vanilla and cdh pages which describes building
  images with sahara-image-create, and move the common
  information to new sahara-image-create page
  in the user manual;
- add the matrix of supported plugin versions and
  supported building technology for each plugin inside
  the respective <plugin>-plugin.rst;
- add the redirects for the removed pages (only for master
  and rocky, where this change should be backported).
- remove few details not really needed (e.g. how to convert
  to VMDK images, location of cloud-init packages, etc,
  which do not really belong here).

Change-Id: I8398a7ad625276d8f11d743688ba71902a7e1adc
This commit is contained in:
Luigi Toscano 2018-10-18 15:15:35 +02:00
parent 6fc5a7b8ba
commit 4a44010f31
20 changed files with 521 additions and 352 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
doc/source/_extra/.htaccess
View File

@ -1,2 +1,6 @@
# renamed after the switch to Storyboard
redirectmatch 301 ^/sahara/([^/]+)/contributor/launchpad.html$ /sahara/$1/contributor/project.html
# renamed after some documentation reshuffling
redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/vanilla-imagebuilder.html$ /sahara/$1/user/vanilla-plugin.html
redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/cdh-imagebuilder.html$ /sahara/$1/user/cdh-plugin.html
redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/guest-requirements.html$ /sahara/$1/user/building-guest-images.html

2
doc/source/admin/advanced-configuration-guide.rst
View File

@ -512,7 +512,7 @@ sahara have the latest version of the Hadoop swift filesystem plugin
installed. The sources for this plugin can be found in the
`sahara extra repository`_. For more information on images or swift
integration see the sahara documentation sections
:ref:`diskimage-builder-label` and :ref:`swift-integration-label`.
:ref:`building-guest-images-label` and :ref:`swift-integration-label`.
.. _Sahara extra repository: http://git.openstack.org/cgit/openstack/sahara-extra

78
doc/source/contributor/image-gen.rst
View File

@ -24,82 +24,8 @@ The key user-facing interface to this feature is the CLI script
``sahara-image-pack``. This script will be installed with all other Sahara
binaries.
Interface
~~~~~~~~~
The script ``sahara-image-pack`` takes the following primary arguments:
::
--config-file PATH Path to a config file to use. Multiple config files
can be specified, with values in later files taking
precedence. Defaults to None.
--image IMAGE The path to an image to modify. This image will be
modified in-place: be sure to target a copy if you
wish to maintain a clean master image.
--root-filesystem ROOT_FS
The filesystem to mount as the root volume on the
image. Novalue is required if only one filesystem is
detected.
--test-only If this flag is set, no changes will be made to the
image; instead, the script will fail if discrepancies
are found between the image and the intended state.
After these arguments, the script takes ``PLUGIN`` and ``VERSION`` arguments.
These arguments will allow any plugin and version combination which supports
the image packing feature. Plugins may require their own arguments at specific
versions; use the ``--help`` feature with ``PLUGIN`` and ``VERSION`` to see
the appropriate argument structure.
a plausible command-line invocation would be:
::
sahara-image-pack --image CentOS.qcow2 \
--config-file etc/sahara/sahara.conf \
cdh 5.7.0 [cdh 5.7.0 specific arguments, if any]
This script will modify the target image in-place. Please copy your image
if you want a backup or if you wish to create multiple images from a single
base image.
This CLI will automatically populate the set of available plugins and
versions from the plugin set loaded in Sahara, and will show any plugin for
which the image packing feature is available. The next sections of this guide
will first describe how to modify an image packing specification for one
of the plugins, and second, how to enable the image packing feature for new
or existing plugins.
Note: In case of a RHEL 7 images, it is necessary to register the image before
starting to pack it, also enable some required repos.
::
virt-customize -v -a $SAHARA_RHEL_IMAGE --sm-register \
--sm-credentials ${REG_USER}:password:${REG_PASSWORD} --sm-attach \
pool:${REG_POOL_ID} --run-command 'subscription-manager repos \
--disable=* --enable=$REPO_A \ --enable=$REPO_B \ --enable=$REPO_C'
Dev notes on the CLI itself
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The script depends on a python library which is not packaged
in pip, but is available through yum, dnf, and apt. If you have installed
Sahara through yum, dnf, or apt, you should have appropriate dependencies,
but if you wish to use the script but are working with Sahara from source,
run whichever of the following is appropriate to your OS:
::
sudo yum install libguestfs python-libguestfs libguestfs-tools
sudo dnf install libguestfs python-libguestfs libguestfs-tools
sudo apt-get install libguestfs python-guestfs libguestfs-tools
If you are using tox to create virtual environments for your Sahara work,
please use the ``images`` environment to run sahara-image-pack. This
environment is configured to use system site packages, and will thus
be able to find its dependency on python-libguestfs.
The usage of the CLI script ``sahara-image-pack`` is documented in
the :ref:`sahara-image-pack-label` section of the user guide.
The Image Manifest

58
doc/source/user/ambari-plugin.rst
View File

@ -23,10 +23,59 @@ variety of environments.
Images
------
The sahara Ambari plugin is using minimal (operating system only) images.
For more information about Ambari images, refer to
`<https://git.openstack.org/cgit/openstack/sahara-image-elements>`_.
For cluster provisioning, prepared images should be used.
.. list-table:: Support matrix for the `ambari` plugin
:widths: 15 15 20 15 35
:header-rows: 1
* - Version
(image tag)
- Distribution
- Build method
- Version
(build parameter)
- Notes
* - 2.6
- Ubuntu 16.04, CentOS 7
- sahara-image-pack
- 2.6
- uses Ambari 2.6
* - 2.5
- Ubuntu 16.04, CentOS 7
- sahara-image-pack
- 2.5
- uses Ambari 2.6
* - 2.4
- Ubuntu 14.04, CentOS 7
- sahara-image-pack
- 2.4
- uses Ambari 2.6
* - 2.4
- Ubuntu 14.04, CentOS 7
- sahara-image-create
- 2.4
- uses Ambari 2.2.1.0
* - 2.3
- Ubuntu 14.04, CentOS 7
- sahara-image-pack
- 2.3
- uses Ambari 2.4
* - 2.3
- Ubuntu 14.04, CentOS 7
- sahara-image-create
- 2.3
- uses Ambari 2.2.0.0
For more information about building image, refer to
:doc:`building-guest-images`.
HDP plugin requires an image to be tagged in sahara Image Registry with two
tags: 'ambari' and '<plugin version>' (e.g. '2.5').
@ -34,6 +83,9 @@ tags: 'ambari' and '<plugin version>' (e.g. '2.5').
The image requires a username. For more information, refer to the
:doc:`registering-image` section.
To speed up provisioning, the HDP packages can be pre-installed on the image
used. The packages' versions depend on the HDP version required.
High Availability for HDFS and YARN
-----------------------------------
High Availability (Using the Quorum Journal Manager) can be

49
doc/source/user/building-guest-images.rst Normal file
View File

@ -0,0 +1,49 @@
.. _building-guest-images-label:
Building guest images
=====================
Sahara plugins represent different Hadoop or other Big Data platforms
and requires specific guest images.
While it is possible to use cloud images which only contain the basic
software requirements (also called *plain images*), their usage slows down
the cluster provisioning process and was not throughly tested recently.
It is strongly advised to build images which contain
the software required to create the clusters for the various plugins
and use them instead of *plain images*.
Sahara currently provides two different tools for building
guest images:
- ``sahara-image-pack`` is newer and support more recent images;
- ``sahara-image-create`` is the older tool.
Both tools are described in the details in the next sections.
The documentation of each plugin describes which method is supported
for the various versions. If both are supported, ``sahara-image-pack``
is recommended.
General requirements for guest images
-------------------------------------
There are few common requirements for all guest images,
which must be based on GNU/Linux distributions.
* cloud-init must be installed
* the ssh server must be installed
* the firewall, if enabled, must allow connections on port 22 (ssh)
The cloud images provided by the GNU/Linux distributions respect
those requirements.
Each plugin specifies additional requirements.
The image building tools provided by Sahara take care of preparing the images
with those additional requirements.
.. toctree::
building-guest-images/sahara-image-pack
building-guest-images/sahara-image-create
building-guest-images/baremetal

14
doc/source/user/building-guest-images/baremetal.rst Normal file
View File

@ -0,0 +1,14 @@
.. _building-baremetal-images-label:
Bare metal images
-----------------
Images that can be used for bare metal deployment through Ironic
can be generated using both image building tools:
sahara-image-create:
pass the -b parameters to the command
sahara-image-pack:
use `virt-get-kernel` on the generated image to extract the kernel and
the initramfs file

80
doc/source/user/building-guest-images/sahara-image-create.rst Normal file
View File

@ -0,0 +1,80 @@
sahara-image-create
-------------------
The historical tool for building images, ``sahara-image-create``, is based on
`Disk Image Builder <https://git.openstack.org/cgit/openstack/diskimage-builder>`_.
`Disk Image Builder` builds disk images using elements. An element is a
particular set of code that alters how the image is built, or runs within the
chroot to prepare the image.
The additional elements required by Sahara images and the ``sahara-image-create``
command itself are stored in the
`Sahara image elements repository <https://git.openstack.org/cgit/openstack/sahara-image-elements>`_
To create images for a specific plugin follow these steps:
1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements"
locally.
2. Use tox to build images.
You can run the command below in sahara-image-elements
directory to build images. By default this script will attempt to create
cloud images for all versions of supported plugins and all operating systems
(subset of Ubuntu, Fedora, and CentOS depending on plugin).
.. sourcecode:: console
tox -e venv -- sahara-image-create -u
If you want to build a image for ``<plugin>`` with ``<version>`` on a specific
``<distribution>`` just execute:
.. sourcecode:: console
tox -e venv -- sahara-image-create -p <plugin> -v <version> -i <distribution>
Tox will create a virtualenv and install required python packages in it,
clone the repositories "https://git.openstack.org/cgit/openstack/diskimage-builder" and
"https://git.openstack.org/cgit/openstack/sahara-image-elements" and export necessary
parameters.
The valid values for the ``<distribution>`` argument are:
- Ubuntu (all versions): ``ubuntu``
- CentOS 7: ``centos7``
- Fedora: ``fedora``
``sahara-image-create`` will then create the required cloud images
using image elements that install all the necessary packages
and configure them.
You will find created images in the parent directory.
Variables
~~~~~~~~~
The following environment variables can be used to change the behavior of the
image building:
* ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin)
* ``DIB_IMAGE_SIZE`` - parameter that specifies a volume of hard disk
of instance. You need to specify it only for Fedora because Fedora
doesn't use all available volume
The following variables can be used to change the name of the output
image:
* ``centos7_image_name``
* ``ubuntu_image_name``
* ``fedora_image_name``
.. note::
Disk Image Builder will generate QCOW2 images, used with the default
OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different
hypervisor, the generated image should be converted to an appropriate
format.
For finer control of ``sahara-image-create`` see the `official documentation
<https://git.openstack.org/cgit/openstack/sahara-image-elements/tree/diskimage-create/README.rst>`_

90
doc/source/user/building-guest-images/sahara-image-pack.rst Normal file
View File

@ -0,0 +1,90 @@
.. _sahara-image-pack-label:
sahara-image-pack
-----------------
The CLI command ``sahara-image-pack`` operates in-place on an existing image
and installs and configures the software required for the plugin.
The script ``sahara-image-pack`` takes the following primary arguments:
::
--config-file PATH Path to a config file to use. Multiple config files
can be specified, with values in later files taking
precedence. Defaults to None.
--image IMAGE The path to an image to modify. This image will be
modified in-place: be sure to target a copy if you
wish to maintain a clean master image.
--root-filesystem ROOT_FS
The filesystem to mount as the root volume on the
image. Novalue is required if only one filesystem is
detected.
--test-only If this flag is set, no changes will be made to the
image; instead, the script will fail if discrepancies
are found between the image and the intended state.
After these arguments, the script takes ``PLUGIN`` and ``VERSION`` arguments.
These arguments will allow any plugin and version combination which supports
the image packing feature. Plugins may require their own arguments at specific
versions; use the ``--help`` feature with ``PLUGIN`` and ``VERSION`` to see
the appropriate argument structure.
a plausible command-line invocation would be:
::
sahara-image-pack --image CentOS.qcow2 \
--config-file etc/sahara/sahara.conf \
cdh 5.7.0 [cdh 5.7.0 specific arguments, if any]
This script will modify the target image in-place. Please copy your image
if you want a backup or if you wish to create multiple images from a single
base image.
This CLI will automatically populate the set of available plugins and
versions from the plugin set loaded in Sahara, and will show any plugin for
which the image packing feature is available. The next sections of this guide
will first describe how to modify an image packing specification for one
of the plugins, and second, how to enable the image packing feature for new
or existing plugins.
Note: In case of a RHEL 7 images, it is necessary to register the image before
starting to pack it, also enable some required repos.
::
virt-customize -v -a $SAHARA_RHEL_IMAGE --sm-register \
--sm-credentials ${REG_USER}:password:${REG_PASSWORD} --sm-attach \
pool:${REG_POOL_ID} --run-command 'subscription-manager repos \
--disable=* --enable=$REPO_A \ --enable=$REPO_B \ --enable=$REPO_C'
Installation and developer notes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The script is part of of the Sahara repository, but it does not depend
on the Sahara services. In order to use its development version,
clone the `Sahara repository <https://git.openstack.org/openstack/sahara>`_,
check out the branch which matches the Sahara version used, and
install the repository in a virtualenv.
The script is also provided by binary distributions of OpenStack.
For example, RDO ships it in the ``openstack-sahara-image-pack`` package.
The script depends on a python library which is not packaged
in pip, but is available through yum, dnf, and apt. If you have installed
Sahara through yum, dnf, or apt, you should have appropriate dependencies,
but if you wish to use the script but are working with Sahara from source,
run whichever of the following is appropriate to your OS:
::
sudo yum install libguestfs python-libguestfs libguestfs-tools
sudo dnf install libguestfs python-libguestfs libguestfs-tools
sudo apt-get install libguestfs python-guestfs libguestfs-tools
If you are using tox to create virtual environments for your Sahara work,
please use the ``images`` environment to run sahara-image-pack. This
environment is configured to use system site packages, and will thus
be able to find its dependency on python-libguestfs.

89
doc/source/user/cdh-imagebuilder.rst
View File

@ -1,89 +0,0 @@
.. _cdh-diskimage-builder-label:
Building Images for Cloudera Plugin
===================================
In this document you will find instructions on how to build Ubuntu and CentOS
images with Cloudera Express (now only versions {5.0.0, 5.3.0 5.4.0, 5.5.0,
5.7.x, 5.9.x} are supported).
To simplify the task of building such images we use
`Disk Image Builder <https://git.openstack.org/cgit/openstack/diskimage-builder>`_.
`Disk Image Builder` builds disk images using elements. An element is a
particular set of code that alters how the image is built, or runs within the
chroot to prepare the image.
Elements for building Cloudera images are stored in
`Sahara extra repository <https://git.openstack.org/cgit/openstack/sahara-image-elements>`_
.. note::
Sahara requires images with cloud-init package installed:
* `For CentOS 7 <http://mirror.centos.org/centos/7/os/x86_64/Packages/cloud-init-0.7.9-24.el7.centos.x86_64.rpm>`_
* `For Ubuntu 14.04 <http://packages.ubuntu.com/trusty/cloud-init>`_
* `For Ubuntu 16.04 <http://packages.ubuntu.com/xenial/cloud-init>`_
To create cloudera images follow these steps:
1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements" locally.
2. Use tox to build images.
You can run "tox -e venv -- sahara-image-create" command in
sahara-image-elements directory to build images. By default this script will
attempt to create cloud images for all versions of supported plugins and all
operating systems (subset of Ubuntu, Fedora, and CentOS depending on
plugin). To only create Cloudera images, you should use the "-p cloudera"
parameter in the command line. If you want to create the image only for a
specific operating system, you should use the "-i ubuntu|centos|centos7"
parameter to assign the operating system (the cloudera plugin only supports
Ubuntu and Centos). If you want to create the image only for a specific
Cloudera version, you should use the "-v 5.0|5.3|5.4|5.5|5.7|5.9" parameter
to assign the version. Note that Centos 7 can only be used with CDH 5.5 and
higher. Below is an example to create Cloudera images for both Ubuntu and
CentOS with Cloudera Express 5.5.0 version.
.. sourcecode:: console
tox -e venv -- sahara-image-create -p cloudera -v 5.5
If you want to create only an Ubuntu image, you may use following example
for that:
.. sourcecode:: console
tox -e venv -- sahara-image-create -p cloudera -i ubuntu -v 5.5
For CDH 5.7 and higher we support minor versions. If you want to build a minor
version just export DIB_CDH_MINOR_VERSION before sahara-image-create launch, e.g.:
.. sourcecode:: console
export DIB_CDH_MINOR_VERSION=5.7.1
NOTE: If you don't want to use default values, you should explicitly set the
values of your required parameters.
The script will create required cloud images using image elements that
install all the necessary packages and configure them. You will find the
created images in the parent directory.
.. note::
Disk Image Builder will generate QCOW2 images, used with the default
OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different
hypervisor, the generated image should be converted to an appropriate
format.
The VMware Nova backend requires the VMDK image format. You may use
qemu-img utility to convert a QCOW2 image to VMDK.
.. sourcecode:: console
qemu-img convert -O vmdk <original_image>.qcow2 <converted_image>.vmdk
For finer control of diskimage-create.sh see the `official documentation
<https://git.openstack.org/cgit/openstack/sahara-image-elements/tree/diskimage-create/README.rst>`_

61
doc/source/user/cdh-plugin.rst
View File

@ -8,17 +8,70 @@ The Cloudera plugin is enabled in Sahara by default. You can manually
modify the Sahara configuration file (default /etc/sahara/sahara.conf) to
explicitly enable or disable it in "plugins" line.
You need to build images using :doc:`cdh-imagebuilder` to produce images used
to provision cluster.
Images
------
For cluster provisioning, prepared images should be used.
.. list-table:: Support matrix for the `vanilla` plugin
:widths: 15 15 20 15 35
:header-rows: 1
* - Version
(image tag)
- Distribution
- Build method
- Version
(build parameter)
- Notes
* - 5.13.0
- Ubuntu 16.04, CentOS 7
- sahara-image-pack
- 5.13.0
-
* - 5.11.0
- Ubuntu 16.04, CentOS 7
- sahara-image-pack, sahara-image-create
- 5.11.0
-
* - 5.9.0
- Ubuntu 14.04, CentOS 7
- sahara-image-pack, sahara-image-create
- 5.9.0
-
* - 5.7.0
- Ubuntu 14.04, CentOS 7
- sahara-image-pack, sahara-image-create
- 5.7.0
-
For more information about building image, refer to
:doc:`building-guest-images`.
The cloudera plugin requires an image to be tagged in Sahara Image Registry
with two tags: 'cdh' and '<cloudera version>' (e.g. '5', '5.3.0', '5.4.0',
'5.5.0', '5.7.0', '5.9.0' or '5.9.1', here '5' stands for '5.0.0').
with two tags: 'cdh' and '<cloudera version>' (e.g. '5.13.0', '5.11.0',
'5.9.0', etc).
The default username specified for these images is different for each
distribution. For more information, refer to the
:doc:`registering-image` section.
Build settings
~~~~~~~~~~~~~~
It is possible to specify minor versions of CDH when ``sahara-image-create``
is used.
If you want to use a minor versions, export ``DIB_CDH_MINOR_VERSION``
before starting the build command, e.g.:
.. sourcecode:: console
export DIB_CDH_MINOR_VERSION=5.7.1
Services Supported
------------------

57
doc/source/user/guest-requirements.rst
View File

@ -1,57 +0,0 @@
Requirements for Guests
=======================
Sahara manages guests of various platforms (for example Ubuntu, Fedora, RHEL,
and CentOS) with various versions of the Hadoop ecosystem projects installed.
There are common requirements for all guests, and additional requirements based
on the plugin that is used for cluster deployment.
Common Requirements
-------------------
* The operating system must be Linux
* cloud-init must be installed
* ssh-server must be installed
+ if a firewall is active it must allow connections on port 22 to enable ssh
Vanilla Plugin Requirements
---------------------------
If the Vanilla Plugin is used for cluster deployment the guest is required to
have
* ssh-client installed
* Java (version >= 6)
* Apache Hadoop installed
* 'hadoop' user created
See :doc:`hadoop-swift` for information on using Swift with your sahara cluster
(for EDP support Swift integration is currently required).
To support EDP, the following components must also be installed on the guest:
* Oozie version 4 or higher
* mysql
* hive
See :doc:`vanilla-imagebuilder` for instructions on building images for this
plugin.
Hortonworks Plugin Requirements
-------------------------------
This plugin does not have any additional requirements. Currently, only the
CentOS Linux and Ubuntu distributions are supported but other distributions
will be supported in the future.
To speed up provisioning, the HDP packages can be pre-installed on the image
used. The packages' versions depend on the HDP version being used.
Cloudera Plugin Requirements
----------------------------
Cloudera Plugin does not have any additional requirements, just build a CDH
image to deploy the cluster.
See :doc:`cdh-imagebuilder` for instructions on building images for this
plugin.

4
doc/source/user/index.rst
View File

@ -48,7 +48,5 @@ Guest Images
.. toctree::
:maxdepth: 2
guest-requirements
building-guest-images
hadoop-swift
vanilla-imagebuilder
cdh-imagebuilder

31
doc/source/user/mapr-plugin.rst
View File

@ -30,12 +30,33 @@ spent in the provisioning cycle. In addition, provisioning large clusters will
put a burden on the network as packages for all nodes need to be downloaded
from the package repository.
For more information about MapR images, refer to
`<https://git.openstack.org/cgit/openstack/sahara-image-elements>`_.
There are VM images provided for use with the MapR Plugin, that can also be
built using the tools available in sahara-image-elements:
https://s3-us-west-2.amazonaws.com/sahara-images/index.html
.. list-table:: Support matrix for the `mapr` plugin
:widths: 15 15 20 15 35
:header-rows: 1
* - Version
(image tag)
- Distribution
- Build method
- Version
(build parameter)
- Notes
* - 5.2.0.mrv2
- Ubuntu 14.04, CentOS 7
- sahara-image-pack
- 5.2.0.mrv2
-
* - 5.2.0.mrv2
- Ubuntu 14.04, CentOS 7
- sahara-image-create
- 5.2.0
-
For more information about building image, refer to
:doc:`building-guest-images`.
MapR plugin needs an image to be tagged in Sahara Image Registry with
two tags: 'mapr' and '<MapR version>' (e.g. '5.2.0.mrv2').

8
doc/source/user/quickstart.rst
View File

@ -44,11 +44,9 @@ generated Ubuntu vanilla image, referred to as
``sahara-vanilla-latest-ubuntu.qcow2``,
and the latest version of vanilla plugin as an example.
* Download a pre-built image
* Build the image using `sahara-image-elements
<https://git.openstack.org/cgit/openstack/sahara-image-elements/tree/README.rst>`_
Build an image which works for the specific plugin.
Please refer to :ref:`building-guest-images-label` and
to the plugin-specific documentation.
Upload the generated image into the OpenStack Image service:

4
doc/source/user/sahara-on-ironic.rst
View File

@ -19,8 +19,8 @@ from the bare metal performance with self-service resource provisioning.
4. Build the Sahara image and prepare it for uploading to Glance:
- Build an image for Sahara plugin with the ``-b`` flag. Use sahara image elements
when building the image. See `<https://git.openstack.org/cgit/openstack/sahara-image-elements>`_
- Build an image for Sahara plugin which supports baremetal deployment.
Refer to the :ref:`building-baremetal-images-label` section.
- Convert the qcow2 image format to the raw format. For example:

41
doc/source/user/spark-plugin.rst
View File

@ -10,14 +10,34 @@ support.
Images
------
For cluster provisioning, prepared images should be used. The Spark plugin
has been developed and tested with the images generated by
sahara-image-elements:
For cluster provisioning, prepared images should be used.
* https://git.openstack.org/cgit/openstack/sahara-image-elements
.. list-table:: Support matrix for the `spark` plugin
:widths: 15 15 20 15 35
:header-rows: 1
The latest Ubuntu images generated by sahara-image-elements have Cloudera
CDH 5.4.0 HDFS and Apache Spark installed.
* - Version
(image tag)
- Distribution
- Build method
- Version
(build parameter)
- Notes
* - 2.3
- Ubuntu 16.04
- sahara-image-create
- 2.3.0
- based on CDH 5.11
* - 2.2
- Ubuntu 16.04
- sahara-image-create
- 2.2.0
- based on CDH 5.11
For more information about building image, refer to
:doc:`building-guest-images`.
The Spark plugin requires an image to be tagged in the sahara image registry
with two tags: 'spark' and '<Spark version>' (e.g. '1.6.0').
@ -31,6 +51,15 @@ slaves), stop all services and so on. As such (and as opposed to CDH HDFS
daemons), Spark is not deployed as a standard Ubuntu service and if the
virtual machines are rebooted, Spark will not be restarted.
Build settings
~~~~~~~~~~~~~~
When ``sahara-image-create`` is used, you can override few settings
by exporting the corresponding environment variables
before starting the build command:
* ``SPARK_DOWNLOAD_URL`` - download link for Spark
Spark configuration
-------------------

31
doc/source/user/storm-plugin.rst
View File

@ -9,11 +9,34 @@ Currently Storm is installed in standalone mode, with no YARN support.
Images
------
For cluster provisioning, prepared images should be used. The Storm plugin
has been developed and tested with the images generated by
sahara-image-elements:
For cluster provisioning, prepared images should be used.
* https://git.openstack.org/cgit/openstack/sahara-image-elements
.. list-table:: Support matrix for the `storm` plugin
:widths: 15 15 20 15 35
:header-rows: 1
* - Version
(image tag)
- Distribution
- Build method
- Version
(build parameter)
- Notes
* - 1.2
- Ubuntu 16.04
- sahara-image-create
- 1.2.1, 1.2.0
- both versions are supported by the same image tag
* - 1.1.0
- Ubuntu 16.04
- sahara-image-create
- 1.1.1, 1.1.0
- both versions are supported by the same image tag
For more information about building image, refer to
:doc:`building-guest-images`.
The Storm plugin requires an image to be tagged in the sahara image registry
with two tags: 'storm' and '<Storm version>' (e.g. '1.1.0').

94
doc/source/user/vanilla-imagebuilder.rst
View File

@ -1,94 +0,0 @@
.. _diskimage-builder-label:
Building Images for Vanilla Plugin
==================================
In this document you will find instruction on how to build Ubuntu, Fedora, and
CentOS images with Apache Hadoop version 2.x.x.
As of now the vanilla plugin works with images with pre-installed versions of
Apache Hadoop. To simplify the task of building such images we use
`Disk Image Builder <https://git.openstack.org/cgit/openstack/diskimage-builder>`_.
`Disk Image Builder` builds disk images using elements. An element is a
particular set of code that alters how the image is built, or runs within the
chroot to prepare the image.
Elements for building vanilla images are stored in the
`Sahara image elements repository <https://git.openstack.org/cgit/openstack/sahara-image-elements>`_
.. note::
Sahara requires images with cloud-init package installed:
* `For Ubuntu 16.04 <http://packages.ubuntu.com/xenial/cloud-init>`_
* `For CentOS 7 <http://mirror.centos.org/centos/7/os/x86_64/Packages/cloud-init-0.7.9-24.el7.centos.x86_64.rpm>`_
* `For Fedora <https://src.fedoraproject.org/rpms/cloud-init>`_
To create vanilla images follow these steps:
1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements"
locally.
2. Use tox to build images.
You can run the command below in sahara-image-elements
directory to build images. By default this script will attempt to create
cloud images for all versions of supported plugins and all operating systems
(subset of Ubuntu, Fedora, and CentOS depending on plugin).
.. sourcecode:: console
tox -e venv -- sahara-image-create -u
If you want to build Vanilla 2.7.1 image with centos 7 just execute:
.. sourcecode:: console
tox -e venv -- sahara-image-create -p vanilla -v 2.7.1 -i centos7
Tox will create a virtualenv and install required python packages in it,
clone the repositories "https://git.openstack.org/cgit/openstack/diskimage-builder" and
"https://git.openstack.org/cgit/openstack/sahara-image-elements" and export necessary
parameters.
* ``DIB_HADOOP_VERSION`` - version of Hadoop to install
* ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin)
* ``OOZIE_DOWNLOAD_URL`` - download link for OOZIE (we have built
Oozie libs here: ``https://tarballs.openstack.org/sahara-extra/dist/oozie/oozie-4.2.0-hadoop-2.7.1.tar.gz``)
* ``SPARK_DOWNLOAD_URL`` - download link for Spark
* ``HIVE_VERSION`` - version of Hive to install
(currently supports only 0.11.0)
* ``ubuntu_image_name``
* ``fedora_image_name``
* ``DIB_IMAGE_SIZE`` - parameter that specifies a volume of hard disk
of instance. You need to specify it only for Fedora because Fedora
doesn't use all available volume
* ``DIB_COMMIT_ID`` - latest commit id of diskimage-builder project
* ``SAHARA_ELEMENTS_COMMIT_ID`` - latest commit id of
sahara-image-elements project
NOTE: If you don't want to use default values, you should set your values
of parameters.
Then it will create required cloud images using image elements that install
all the necessary packages and configure them. You will find created images
in the parent directory.
.. note::
Disk Image Builder will generate QCOW2 images, used with the default
OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different
hypervisor, the generated image should be converted to an appropriate
format.
VMware Nova backend requires VMDK image format. You may use qemu-img
utility to convert a QCOW2 image to VMDK.
.. sourcecode:: console
qemu-img convert -O vmdk <original_image>.qcow2 <converted_image>.vmdk
For finer control of diskimage-create.sh see the `official documentation
<https://git.openstack.org/cgit/openstack/sahara-image-elements/tree/diskimage-create/README.rst>`_

74
doc/source/user/vanilla-plugin.rst
View File

@ -7,10 +7,43 @@ a cluster with Apache Hadoop.
Since the Newton release Spark is integrated into the Vanilla plugin so you
can launch Spark jobs on a Vanilla cluster.
For cluster provisioning prepared images should be used. They already have
Apache Hadoop 2.7.1 installed.
Images
------
You may build images by yourself using :doc:`vanilla-imagebuilder`.
For cluster provisioning, prepared images should be used.
.. list-table:: Support matrix for the `vanilla` plugin
:widths: 15 15 20 15 35
:header-rows: 1
* - Version
(image tag)
- Distribution
- Build method
- Version
(build parameter)
- Notes
* - 2.8.2
- Ubuntu 16.04, CentOS 7
- sahara-image-create
- 2.8.2
- Hive 2.3.2, Oozie 4.3.0
* - 2.7.5
- Ubuntu 16.04, CentOS 7
- sahara-image-create
- 2.7.5
- Hive 2.3.2, Oozie 4.3.0
* - 2.7.1
- Ubuntu 16.04, CentOS 7
- sahara-image-create
- 2.7.1
- Hive 0.11.0, Oozie 4.2.0
For more information about building image, refer to
:doc:`building-guest-images`.
Vanilla plugin requires an image to be tagged in Sahara Image Registry with
two tags: 'vanilla' and '<hadoop version>' (e.g. '2.7.1').
@ -18,6 +51,41 @@ two tags: 'vanilla' and '<hadoop version>' (e.g. '2.7.1').
The image requires a username. For more information, refer to the
:doc:`registering-image` section.
Build settings
~~~~~~~~~~~~~~
When ``sahara-image-create`` is used, you can override few settings
by exporting the corresponding environment variables
before starting the build command:
* ``DIB_HADOOP_VERSION`` - version of Hadoop to install
* ``HIVE_VERSION`` - version of Hive to install
* ``OOZIE_DOWNLOAD_URL`` - download link for Oozie (we have built
Oozie libs here: https://tarballs.openstack.org/sahara-extra/dist/oozie/)
* ``SPARK_DOWNLOAD_URL`` - download link for Spark
Vanilla Plugin Requirements
---------------------------
The image building tools described in :ref:`building-guest-images-label`
add the required software to the image and their usage is strongly suggested.
Nevertheless, here are listed the software that should be pre-loaded
on the guest image so that it can be used to create Vanilla clusters:
* ssh-client installed
* Java (version >= 7)
* Apache Hadoop installed
* 'hadoop' user created
See :doc:`hadoop-swift` for information on using Swift with your sahara cluster
(for EDP support Swift integration is currently required).
To support EDP, the following components must also be installed on the guest:
* Oozie version 4 or higher
* mysql/mariadb
* hive
Cluster Validation
------------------

4
doc/test/redirect-tests.txt
View File

@ -1,3 +1,7 @@
/sahara/pike/contributor/launchpad.html 301 /sahara/pike/contributor/project.html
/sahara/queens/contributor/launchpad.html 301 /sahara/queens/contributor/project.html
/sahara/latest/contributor/launchpad.html 301 /sahara/latest/contributor/project.html
/sahara/latest/user/vanilla-imagebuilder.html 301 /sahara/latest/user/vanilla-plugin.html
/sahara/latest/user/cdh-imagebuilder.html 301 /sahara/latest/user/cdh-plugin.html
/sahara/latest/user/guest-requirements.html 301 /sahara/latest/user/building-guest-images.html
/sahara/rocky/user/guest-requirements.html 301 /sahara/rocky/user/building-guest-images.html