From 4a44010f3124c0e0d20d3850a6ceb0e2cc7d643d Mon Sep 17 00:00:00 2001 From: Luigi Toscano Date: Thu, 18 Oct 2018 15:15:35 +0200 Subject: [PATCH] doc: restructure the image building documentation Main goal: consolidate the information about image building under the same documentation page, and move plugin-specific details inside plugin pages. No plugin-specific information should live outside those pages. More details: - move the detailed documentation about sahara-image-pack from the contributor guide to the new dedicated page in the user manual; - remove the vanilla and cdh pages which describes building images with sahara-image-create, and move the common information to new sahara-image-create page in the user manual; - add the matrix of supported plugin versions and supported building technology for each plugin inside the respective -plugin.rst; - add the redirects for the removed pages (only for master and rocky, where this change should be backported). - remove few details not really needed (e.g. how to convert to VMDK images, location of cloud-init packages, etc, which do not really belong here). Change-Id: I8398a7ad625276d8f11d743688ba71902a7e1adc --- doc/source/_extra/.htaccess | 4 + .../admin/advanced-configuration-guide.rst | 2 +- doc/source/contributor/image-gen.rst | 78 +-------------- doc/source/user/ambari-plugin.rst | 58 +++++++++++- doc/source/user/building-guest-images.rst | 49 ++++++++++ .../user/building-guest-images/baremetal.rst | 14 +++ .../sahara-image-create.rst | 80 ++++++++++++++++ .../sahara-image-pack.rst | 90 ++++++++++++++++++ doc/source/user/cdh-imagebuilder.rst | 89 ------------------ doc/source/user/cdh-plugin.rst | 61 +++++++++++- doc/source/user/guest-requirements.rst | 57 ----------- doc/source/user/index.rst | 4 +- doc/source/user/mapr-plugin.rst | 31 +++++- doc/source/user/quickstart.rst | 8 +- doc/source/user/sahara-on-ironic.rst | 4 +- doc/source/user/spark-plugin.rst | 41 ++++++-- doc/source/user/storm-plugin.rst | 31 +++++- doc/source/user/vanilla-imagebuilder.rst | 94 ------------------- doc/source/user/vanilla-plugin.rst | 74 ++++++++++++++- doc/test/redirect-tests.txt | 4 + 20 files changed, 521 insertions(+), 352 deletions(-) create mode 100644 doc/source/user/building-guest-images.rst create mode 100644 doc/source/user/building-guest-images/baremetal.rst create mode 100644 doc/source/user/building-guest-images/sahara-image-create.rst create mode 100644 doc/source/user/building-guest-images/sahara-image-pack.rst delete mode 100644 doc/source/user/cdh-imagebuilder.rst delete mode 100644 doc/source/user/guest-requirements.rst delete mode 100644 doc/source/user/vanilla-imagebuilder.rst diff --git a/doc/source/_extra/.htaccess b/doc/source/_extra/.htaccess index 6e39f755fc..ca9b57a1bb 100644 --- a/doc/source/_extra/.htaccess +++ b/doc/source/_extra/.htaccess @@ -1,2 +1,6 @@ # renamed after the switch to Storyboard redirectmatch 301 ^/sahara/([^/]+)/contributor/launchpad.html$ /sahara/$1/contributor/project.html +# renamed after some documentation reshuffling +redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/vanilla-imagebuilder.html$ /sahara/$1/user/vanilla-plugin.html +redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/cdh-imagebuilder.html$ /sahara/$1/user/cdh-plugin.html +redirectmatch 301 ^/sahara/(?!ocata|pike|queens)([^/]+)/user/guest-requirements.html$ /sahara/$1/user/building-guest-images.html diff --git a/doc/source/admin/advanced-configuration-guide.rst b/doc/source/admin/advanced-configuration-guide.rst index 457e7a509a..17911dea33 100644 --- a/doc/source/admin/advanced-configuration-guide.rst +++ b/doc/source/admin/advanced-configuration-guide.rst @@ -512,7 +512,7 @@ sahara have the latest version of the Hadoop swift filesystem plugin installed. The sources for this plugin can be found in the `sahara extra repository`_. For more information on images or swift integration see the sahara documentation sections -:ref:`diskimage-builder-label` and :ref:`swift-integration-label`. +:ref:`building-guest-images-label` and :ref:`swift-integration-label`. .. _Sahara extra repository: http://git.openstack.org/cgit/openstack/sahara-extra diff --git a/doc/source/contributor/image-gen.rst b/doc/source/contributor/image-gen.rst index c36c1da095..2f22e4b4cc 100644 --- a/doc/source/contributor/image-gen.rst +++ b/doc/source/contributor/image-gen.rst @@ -24,82 +24,8 @@ The key user-facing interface to this feature is the CLI script ``sahara-image-pack``. This script will be installed with all other Sahara binaries. -Interface -~~~~~~~~~ - -The script ``sahara-image-pack`` takes the following primary arguments: - -:: - - --config-file PATH Path to a config file to use. Multiple config files - can be specified, with values in later files taking - precedence. Defaults to None. - --image IMAGE The path to an image to modify. This image will be - modified in-place: be sure to target a copy if you - wish to maintain a clean master image. - --root-filesystem ROOT_FS - The filesystem to mount as the root volume on the - image. Novalue is required if only one filesystem is - detected. - --test-only If this flag is set, no changes will be made to the - image; instead, the script will fail if discrepancies - are found between the image and the intended state. - -After these arguments, the script takes ``PLUGIN`` and ``VERSION`` arguments. -These arguments will allow any plugin and version combination which supports -the image packing feature. Plugins may require their own arguments at specific -versions; use the ``--help`` feature with ``PLUGIN`` and ``VERSION`` to see -the appropriate argument structure. - - -a plausible command-line invocation would be: - -:: - - sahara-image-pack --image CentOS.qcow2 \ - --config-file etc/sahara/sahara.conf \ - cdh 5.7.0 [cdh 5.7.0 specific arguments, if any] - -This script will modify the target image in-place. Please copy your image -if you want a backup or if you wish to create multiple images from a single -base image. - -This CLI will automatically populate the set of available plugins and -versions from the plugin set loaded in Sahara, and will show any plugin for -which the image packing feature is available. The next sections of this guide -will first describe how to modify an image packing specification for one -of the plugins, and second, how to enable the image packing feature for new -or existing plugins. - -Note: In case of a RHEL 7 images, it is necessary to register the image before -starting to pack it, also enable some required repos. - -:: - - virt-customize -v -a $SAHARA_RHEL_IMAGE --sm-register \ - --sm-credentials ${REG_USER}:password:${REG_PASSWORD} --sm-attach \ - pool:${REG_POOL_ID} --run-command 'subscription-manager repos \ - --disable=* --enable=$REPO_A \ --enable=$REPO_B \ --enable=$REPO_C' - -Dev notes on the CLI itself -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The script depends on a python library which is not packaged -in pip, but is available through yum, dnf, and apt. If you have installed -Sahara through yum, dnf, or apt, you should have appropriate dependencies, -but if you wish to use the script but are working with Sahara from source, -run whichever of the following is appropriate to your OS: - -:: - - sudo yum install libguestfs python-libguestfs libguestfs-tools - sudo dnf install libguestfs python-libguestfs libguestfs-tools - sudo apt-get install libguestfs python-guestfs libguestfs-tools - -If you are using tox to create virtual environments for your Sahara work, -please use the ``images`` environment to run sahara-image-pack. This -environment is configured to use system site packages, and will thus -be able to find its dependency on python-libguestfs. +The usage of the CLI script ``sahara-image-pack`` is documented in +the :ref:`sahara-image-pack-label` section of the user guide. The Image Manifest diff --git a/doc/source/user/ambari-plugin.rst b/doc/source/user/ambari-plugin.rst index be3451384a..57bf6d7b35 100644 --- a/doc/source/user/ambari-plugin.rst +++ b/doc/source/user/ambari-plugin.rst @@ -23,10 +23,59 @@ variety of environments. Images ------ -The sahara Ambari plugin is using minimal (operating system only) images. -For more information about Ambari images, refer to -``_. +For cluster provisioning, prepared images should be used. + +.. list-table:: Support matrix for the `ambari` plugin + :widths: 15 15 20 15 35 + :header-rows: 1 + + * - Version + (image tag) + - Distribution + - Build method + - Version + (build parameter) + - Notes + + * - 2.6 + - Ubuntu 16.04, CentOS 7 + - sahara-image-pack + - 2.6 + - uses Ambari 2.6 + + * - 2.5 + - Ubuntu 16.04, CentOS 7 + - sahara-image-pack + - 2.5 + - uses Ambari 2.6 + + * - 2.4 + - Ubuntu 14.04, CentOS 7 + - sahara-image-pack + - 2.4 + - uses Ambari 2.6 + + * - 2.4 + - Ubuntu 14.04, CentOS 7 + - sahara-image-create + - 2.4 + - uses Ambari 2.2.1.0 + + * - 2.3 + - Ubuntu 14.04, CentOS 7 + - sahara-image-pack + - 2.3 + - uses Ambari 2.4 + + * - 2.3 + - Ubuntu 14.04, CentOS 7 + - sahara-image-create + - 2.3 + - uses Ambari 2.2.0.0 + +For more information about building image, refer to +:doc:`building-guest-images`. HDP plugin requires an image to be tagged in sahara Image Registry with two tags: 'ambari' and '' (e.g. '2.5'). @@ -34,6 +83,9 @@ tags: 'ambari' and '' (e.g. '2.5'). The image requires a username. For more information, refer to the :doc:`registering-image` section. +To speed up provisioning, the HDP packages can be pre-installed on the image +used. The packages' versions depend on the HDP version required. + High Availability for HDFS and YARN ----------------------------------- High Availability (Using the Quorum Journal Manager) can be diff --git a/doc/source/user/building-guest-images.rst b/doc/source/user/building-guest-images.rst new file mode 100644 index 0000000000..da42ce5a65 --- /dev/null +++ b/doc/source/user/building-guest-images.rst @@ -0,0 +1,49 @@ +.. _building-guest-images-label: + +Building guest images +===================== + +Sahara plugins represent different Hadoop or other Big Data platforms +and requires specific guest images. + +While it is possible to use cloud images which only contain the basic +software requirements (also called *plain images*), their usage slows down +the cluster provisioning process and was not throughly tested recently. + +It is strongly advised to build images which contain +the software required to create the clusters for the various plugins +and use them instead of *plain images*. + +Sahara currently provides two different tools for building +guest images: +- ``sahara-image-pack`` is newer and support more recent images; +- ``sahara-image-create`` is the older tool. + +Both tools are described in the details in the next sections. + +The documentation of each plugin describes which method is supported +for the various versions. If both are supported, ``sahara-image-pack`` +is recommended. + +General requirements for guest images +------------------------------------- + +There are few common requirements for all guest images, +which must be based on GNU/Linux distributions. + +* cloud-init must be installed +* the ssh server must be installed +* the firewall, if enabled, must allow connections on port 22 (ssh) + +The cloud images provided by the GNU/Linux distributions respect +those requirements. + +Each plugin specifies additional requirements. +The image building tools provided by Sahara take care of preparing the images +with those additional requirements. + +.. toctree:: + + building-guest-images/sahara-image-pack + building-guest-images/sahara-image-create + building-guest-images/baremetal diff --git a/doc/source/user/building-guest-images/baremetal.rst b/doc/source/user/building-guest-images/baremetal.rst new file mode 100644 index 0000000000..bceb260ff3 --- /dev/null +++ b/doc/source/user/building-guest-images/baremetal.rst @@ -0,0 +1,14 @@ +.. _building-baremetal-images-label: + +Bare metal images +----------------- + +Images that can be used for bare metal deployment through Ironic +can be generated using both image building tools: + +sahara-image-create: + pass the -b parameters to the command + +sahara-image-pack: + use `virt-get-kernel` on the generated image to extract the kernel and + the initramfs file diff --git a/doc/source/user/building-guest-images/sahara-image-create.rst b/doc/source/user/building-guest-images/sahara-image-create.rst new file mode 100644 index 0000000000..e2f2533f46 --- /dev/null +++ b/doc/source/user/building-guest-images/sahara-image-create.rst @@ -0,0 +1,80 @@ +sahara-image-create +------------------- + +The historical tool for building images, ``sahara-image-create``, is based on +`Disk Image Builder `_. + +`Disk Image Builder` builds disk images using elements. An element is a +particular set of code that alters how the image is built, or runs within the +chroot to prepare the image. + +The additional elements required by Sahara images and the ``sahara-image-create`` +command itself are stored in the +`Sahara image elements repository `_ + +To create images for a specific plugin follow these steps: + +1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements" + locally. + +2. Use tox to build images. + + You can run the command below in sahara-image-elements + directory to build images. By default this script will attempt to create + cloud images for all versions of supported plugins and all operating systems + (subset of Ubuntu, Fedora, and CentOS depending on plugin). + + .. sourcecode:: console + + tox -e venv -- sahara-image-create -u + + If you want to build a image for ```` with ```` on a specific + ```` just execute: + + .. sourcecode:: console + + tox -e venv -- sahara-image-create -p -v -i + + Tox will create a virtualenv and install required python packages in it, + clone the repositories "https://git.openstack.org/cgit/openstack/diskimage-builder" and + "https://git.openstack.org/cgit/openstack/sahara-image-elements" and export necessary + parameters. + + The valid values for the ```` argument are: + + - Ubuntu (all versions): ``ubuntu`` + - CentOS 7: ``centos7`` + - Fedora: ``fedora`` + + ``sahara-image-create`` will then create the required cloud images + using image elements that install all the necessary packages + and configure them. + You will find created images in the parent directory. + +Variables +~~~~~~~~~ + +The following environment variables can be used to change the behavior of the +image building: + +* ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin) +* ``DIB_IMAGE_SIZE`` - parameter that specifies a volume of hard disk + of instance. You need to specify it only for Fedora because Fedora + doesn't use all available volume + +The following variables can be used to change the name of the output +image: + +* ``centos7_image_name`` +* ``ubuntu_image_name`` +* ``fedora_image_name`` + +.. note:: + + Disk Image Builder will generate QCOW2 images, used with the default + OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different + hypervisor, the generated image should be converted to an appropriate + format. + +For finer control of ``sahara-image-create`` see the `official documentation +`_ diff --git a/doc/source/user/building-guest-images/sahara-image-pack.rst b/doc/source/user/building-guest-images/sahara-image-pack.rst new file mode 100644 index 0000000000..26e3e0ce12 --- /dev/null +++ b/doc/source/user/building-guest-images/sahara-image-pack.rst @@ -0,0 +1,90 @@ +.. _sahara-image-pack-label: + +sahara-image-pack +----------------- + +The CLI command ``sahara-image-pack`` operates in-place on an existing image +and installs and configures the software required for the plugin. + +The script ``sahara-image-pack`` takes the following primary arguments: + +:: + + --config-file PATH Path to a config file to use. Multiple config files + can be specified, with values in later files taking + precedence. Defaults to None. + --image IMAGE The path to an image to modify. This image will be + modified in-place: be sure to target a copy if you + wish to maintain a clean master image. + --root-filesystem ROOT_FS + The filesystem to mount as the root volume on the + image. Novalue is required if only one filesystem is + detected. + --test-only If this flag is set, no changes will be made to the + image; instead, the script will fail if discrepancies + are found between the image and the intended state. + +After these arguments, the script takes ``PLUGIN`` and ``VERSION`` arguments. +These arguments will allow any plugin and version combination which supports +the image packing feature. Plugins may require their own arguments at specific +versions; use the ``--help`` feature with ``PLUGIN`` and ``VERSION`` to see +the appropriate argument structure. + + +a plausible command-line invocation would be: + +:: + + sahara-image-pack --image CentOS.qcow2 \ + --config-file etc/sahara/sahara.conf \ + cdh 5.7.0 [cdh 5.7.0 specific arguments, if any] + +This script will modify the target image in-place. Please copy your image +if you want a backup or if you wish to create multiple images from a single +base image. + +This CLI will automatically populate the set of available plugins and +versions from the plugin set loaded in Sahara, and will show any plugin for +which the image packing feature is available. The next sections of this guide +will first describe how to modify an image packing specification for one +of the plugins, and second, how to enable the image packing feature for new +or existing plugins. + +Note: In case of a RHEL 7 images, it is necessary to register the image before +starting to pack it, also enable some required repos. + +:: + + virt-customize -v -a $SAHARA_RHEL_IMAGE --sm-register \ + --sm-credentials ${REG_USER}:password:${REG_PASSWORD} --sm-attach \ + pool:${REG_POOL_ID} --run-command 'subscription-manager repos \ + --disable=* --enable=$REPO_A \ --enable=$REPO_B \ --enable=$REPO_C' + +Installation and developer notes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The script is part of of the Sahara repository, but it does not depend +on the Sahara services. In order to use its development version, +clone the `Sahara repository `_, +check out the branch which matches the Sahara version used, and +install the repository in a virtualenv. + +The script is also provided by binary distributions of OpenStack. +For example, RDO ships it in the ``openstack-sahara-image-pack`` package. + +The script depends on a python library which is not packaged +in pip, but is available through yum, dnf, and apt. If you have installed +Sahara through yum, dnf, or apt, you should have appropriate dependencies, +but if you wish to use the script but are working with Sahara from source, +run whichever of the following is appropriate to your OS: + +:: + + sudo yum install libguestfs python-libguestfs libguestfs-tools + sudo dnf install libguestfs python-libguestfs libguestfs-tools + sudo apt-get install libguestfs python-guestfs libguestfs-tools + +If you are using tox to create virtual environments for your Sahara work, +please use the ``images`` environment to run sahara-image-pack. This +environment is configured to use system site packages, and will thus +be able to find its dependency on python-libguestfs. diff --git a/doc/source/user/cdh-imagebuilder.rst b/doc/source/user/cdh-imagebuilder.rst deleted file mode 100644 index 64e609cb8e..0000000000 --- a/doc/source/user/cdh-imagebuilder.rst +++ /dev/null @@ -1,89 +0,0 @@ -.. _cdh-diskimage-builder-label: - -Building Images for Cloudera Plugin -=================================== - -In this document you will find instructions on how to build Ubuntu and CentOS -images with Cloudera Express (now only versions {5.0.0, 5.3.0 5.4.0, 5.5.0, -5.7.x, 5.9.x} are supported). - -To simplify the task of building such images we use -`Disk Image Builder `_. - -`Disk Image Builder` builds disk images using elements. An element is a -particular set of code that alters how the image is built, or runs within the -chroot to prepare the image. - -Elements for building Cloudera images are stored in -`Sahara extra repository `_ - -.. note:: - - Sahara requires images with cloud-init package installed: - - * `For CentOS 7 `_ - * `For Ubuntu 14.04 `_ - * `For Ubuntu 16.04 `_ - -To create cloudera images follow these steps: - -1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements" locally. - -2. Use tox to build images. - - You can run "tox -e venv -- sahara-image-create" command in - sahara-image-elements directory to build images. By default this script will - attempt to create cloud images for all versions of supported plugins and all - operating systems (subset of Ubuntu, Fedora, and CentOS depending on - plugin). To only create Cloudera images, you should use the "-p cloudera" - parameter in the command line. If you want to create the image only for a - specific operating system, you should use the "-i ubuntu|centos|centos7" - parameter to assign the operating system (the cloudera plugin only supports - Ubuntu and Centos). If you want to create the image only for a specific - Cloudera version, you should use the "-v 5.0|5.3|5.4|5.5|5.7|5.9" parameter - to assign the version. Note that Centos 7 can only be used with CDH 5.5 and - higher. Below is an example to create Cloudera images for both Ubuntu and - CentOS with Cloudera Express 5.5.0 version. - - .. sourcecode:: console - - tox -e venv -- sahara-image-create -p cloudera -v 5.5 - - If you want to create only an Ubuntu image, you may use following example - for that: - - .. sourcecode:: console - - tox -e venv -- sahara-image-create -p cloudera -i ubuntu -v 5.5 - - For CDH 5.7 and higher we support minor versions. If you want to build a minor - version just export DIB_CDH_MINOR_VERSION before sahara-image-create launch, e.g.: - - .. sourcecode:: console - - export DIB_CDH_MINOR_VERSION=5.7.1 - - NOTE: If you don't want to use default values, you should explicitly set the - values of your required parameters. - - The script will create required cloud images using image elements that - install all the necessary packages and configure them. You will find the - created images in the parent directory. - -.. note:: - - Disk Image Builder will generate QCOW2 images, used with the default - OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different - hypervisor, the generated image should be converted to an appropriate - format. - - The VMware Nova backend requires the VMDK image format. You may use - qemu-img utility to convert a QCOW2 image to VMDK. - - .. sourcecode:: console - - qemu-img convert -O vmdk .qcow2 .vmdk - - -For finer control of diskimage-create.sh see the `official documentation -`_ diff --git a/doc/source/user/cdh-plugin.rst b/doc/source/user/cdh-plugin.rst index 8d0841716f..6a6431c7cb 100644 --- a/doc/source/user/cdh-plugin.rst +++ b/doc/source/user/cdh-plugin.rst @@ -8,17 +8,70 @@ The Cloudera plugin is enabled in Sahara by default. You can manually modify the Sahara configuration file (default /etc/sahara/sahara.conf) to explicitly enable or disable it in "plugins" line. -You need to build images using :doc:`cdh-imagebuilder` to produce images used -to provision cluster. +Images +------ + +For cluster provisioning, prepared images should be used. + +.. list-table:: Support matrix for the `vanilla` plugin + :widths: 15 15 20 15 35 + :header-rows: 1 + + * - Version + (image tag) + - Distribution + - Build method + - Version + (build parameter) + - Notes + + * - 5.13.0 + - Ubuntu 16.04, CentOS 7 + - sahara-image-pack + - 5.13.0 + - + + * - 5.11.0 + - Ubuntu 16.04, CentOS 7 + - sahara-image-pack, sahara-image-create + - 5.11.0 + - + + * - 5.9.0 + - Ubuntu 14.04, CentOS 7 + - sahara-image-pack, sahara-image-create + - 5.9.0 + - + + * - 5.7.0 + - Ubuntu 14.04, CentOS 7 + - sahara-image-pack, sahara-image-create + - 5.7.0 + - + +For more information about building image, refer to +:doc:`building-guest-images`. The cloudera plugin requires an image to be tagged in Sahara Image Registry -with two tags: 'cdh' and '' (e.g. '5', '5.3.0', '5.4.0', -'5.5.0', '5.7.0', '5.9.0' or '5.9.1', here '5' stands for '5.0.0'). +with two tags: 'cdh' and '' (e.g. '5.13.0', '5.11.0', +'5.9.0', etc). The default username specified for these images is different for each distribution. For more information, refer to the :doc:`registering-image` section. +Build settings +~~~~~~~~~~~~~~ + +It is possible to specify minor versions of CDH when ``sahara-image-create`` +is used. +If you want to use a minor versions, export ``DIB_CDH_MINOR_VERSION`` +before starting the build command, e.g.: + + .. sourcecode:: console + + export DIB_CDH_MINOR_VERSION=5.7.1 + Services Supported ------------------ diff --git a/doc/source/user/guest-requirements.rst b/doc/source/user/guest-requirements.rst deleted file mode 100644 index a2e3cfd1cd..0000000000 --- a/doc/source/user/guest-requirements.rst +++ /dev/null @@ -1,57 +0,0 @@ -Requirements for Guests -======================= - -Sahara manages guests of various platforms (for example Ubuntu, Fedora, RHEL, -and CentOS) with various versions of the Hadoop ecosystem projects installed. -There are common requirements for all guests, and additional requirements based -on the plugin that is used for cluster deployment. - -Common Requirements -------------------- - -* The operating system must be Linux -* cloud-init must be installed -* ssh-server must be installed - - + if a firewall is active it must allow connections on port 22 to enable ssh - -Vanilla Plugin Requirements ---------------------------- - -If the Vanilla Plugin is used for cluster deployment the guest is required to -have - -* ssh-client installed -* Java (version >= 6) -* Apache Hadoop installed -* 'hadoop' user created - -See :doc:`hadoop-swift` for information on using Swift with your sahara cluster -(for EDP support Swift integration is currently required). - -To support EDP, the following components must also be installed on the guest: - -* Oozie version 4 or higher -* mysql -* hive - -See :doc:`vanilla-imagebuilder` for instructions on building images for this -plugin. - -Hortonworks Plugin Requirements -------------------------------- - -This plugin does not have any additional requirements. Currently, only the -CentOS Linux and Ubuntu distributions are supported but other distributions -will be supported in the future. -To speed up provisioning, the HDP packages can be pre-installed on the image -used. The packages' versions depend on the HDP version being used. - -Cloudera Plugin Requirements ----------------------------- - -Cloudera Plugin does not have any additional requirements, just build a CDH -image to deploy the cluster. - -See :doc:`cdh-imagebuilder` for instructions on building images for this -plugin. diff --git a/doc/source/user/index.rst b/doc/source/user/index.rst index 2c15fb0600..1f8b45b59d 100644 --- a/doc/source/user/index.rst +++ b/doc/source/user/index.rst @@ -48,7 +48,5 @@ Guest Images .. toctree:: :maxdepth: 2 - guest-requirements + building-guest-images hadoop-swift - vanilla-imagebuilder - cdh-imagebuilder diff --git a/doc/source/user/mapr-plugin.rst b/doc/source/user/mapr-plugin.rst index 681e25eb04..d1015fa314 100644 --- a/doc/source/user/mapr-plugin.rst +++ b/doc/source/user/mapr-plugin.rst @@ -30,12 +30,33 @@ spent in the provisioning cycle. In addition, provisioning large clusters will put a burden on the network as packages for all nodes need to be downloaded from the package repository. -For more information about MapR images, refer to -``_. -There are VM images provided for use with the MapR Plugin, that can also be -built using the tools available in sahara-image-elements: -https://s3-us-west-2.amazonaws.com/sahara-images/index.html +.. list-table:: Support matrix for the `mapr` plugin + :widths: 15 15 20 15 35 + :header-rows: 1 + + * - Version + (image tag) + - Distribution + - Build method + - Version + (build parameter) + - Notes + + * - 5.2.0.mrv2 + - Ubuntu 14.04, CentOS 7 + - sahara-image-pack + - 5.2.0.mrv2 + - + + * - 5.2.0.mrv2 + - Ubuntu 14.04, CentOS 7 + - sahara-image-create + - 5.2.0 + - + +For more information about building image, refer to +:doc:`building-guest-images`. MapR plugin needs an image to be tagged in Sahara Image Registry with two tags: 'mapr' and '' (e.g. '5.2.0.mrv2'). diff --git a/doc/source/user/quickstart.rst b/doc/source/user/quickstart.rst index f8cef3536c..fe84f06975 100644 --- a/doc/source/user/quickstart.rst +++ b/doc/source/user/quickstart.rst @@ -44,11 +44,9 @@ generated Ubuntu vanilla image, referred to as ``sahara-vanilla-latest-ubuntu.qcow2``, and the latest version of vanilla plugin as an example. -* Download a pre-built image - -* Build the image using `sahara-image-elements - `_ - +Build an image which works for the specific plugin. +Please refer to :ref:`building-guest-images-label` and +to the plugin-specific documentation. Upload the generated image into the OpenStack Image service: diff --git a/doc/source/user/sahara-on-ironic.rst b/doc/source/user/sahara-on-ironic.rst index b8f2e2ec38..d8a98ff07a 100644 --- a/doc/source/user/sahara-on-ironic.rst +++ b/doc/source/user/sahara-on-ironic.rst @@ -19,8 +19,8 @@ from the bare metal performance with self-service resource provisioning. 4. Build the Sahara image and prepare it for uploading to Glance: - - Build an image for Sahara plugin with the ``-b`` flag. Use sahara image elements - when building the image. See ``_ + - Build an image for Sahara plugin which supports baremetal deployment. + Refer to the :ref:`building-baremetal-images-label` section. - Convert the qcow2 image format to the raw format. For example: diff --git a/doc/source/user/spark-plugin.rst b/doc/source/user/spark-plugin.rst index 5d906b3d02..021511dab7 100644 --- a/doc/source/user/spark-plugin.rst +++ b/doc/source/user/spark-plugin.rst @@ -10,14 +10,34 @@ support. Images ------ -For cluster provisioning, prepared images should be used. The Spark plugin -has been developed and tested with the images generated by -sahara-image-elements: +For cluster provisioning, prepared images should be used. -* https://git.openstack.org/cgit/openstack/sahara-image-elements +.. list-table:: Support matrix for the `spark` plugin + :widths: 15 15 20 15 35 + :header-rows: 1 -The latest Ubuntu images generated by sahara-image-elements have Cloudera -CDH 5.4.0 HDFS and Apache Spark installed. + * - Version + (image tag) + - Distribution + - Build method + - Version + (build parameter) + - Notes + + * - 2.3 + - Ubuntu 16.04 + - sahara-image-create + - 2.3.0 + - based on CDH 5.11 + + * - 2.2 + - Ubuntu 16.04 + - sahara-image-create + - 2.2.0 + - based on CDH 5.11 + +For more information about building image, refer to +:doc:`building-guest-images`. The Spark plugin requires an image to be tagged in the sahara image registry with two tags: 'spark' and '' (e.g. '1.6.0'). @@ -31,6 +51,15 @@ slaves), stop all services and so on. As such (and as opposed to CDH HDFS daemons), Spark is not deployed as a standard Ubuntu service and if the virtual machines are rebooted, Spark will not be restarted. +Build settings +~~~~~~~~~~~~~~ + +When ``sahara-image-create`` is used, you can override few settings +by exporting the corresponding environment variables +before starting the build command: + +* ``SPARK_DOWNLOAD_URL`` - download link for Spark + Spark configuration ------------------- diff --git a/doc/source/user/storm-plugin.rst b/doc/source/user/storm-plugin.rst index 8778bae5f3..6f5d7f04be 100644 --- a/doc/source/user/storm-plugin.rst +++ b/doc/source/user/storm-plugin.rst @@ -9,11 +9,34 @@ Currently Storm is installed in standalone mode, with no YARN support. Images ------ -For cluster provisioning, prepared images should be used. The Storm plugin -has been developed and tested with the images generated by -sahara-image-elements: +For cluster provisioning, prepared images should be used. -* https://git.openstack.org/cgit/openstack/sahara-image-elements +.. list-table:: Support matrix for the `storm` plugin + :widths: 15 15 20 15 35 + :header-rows: 1 + + * - Version + (image tag) + - Distribution + - Build method + - Version + (build parameter) + - Notes + + * - 1.2 + - Ubuntu 16.04 + - sahara-image-create + - 1.2.1, 1.2.0 + - both versions are supported by the same image tag + + * - 1.1.0 + - Ubuntu 16.04 + - sahara-image-create + - 1.1.1, 1.1.0 + - both versions are supported by the same image tag + +For more information about building image, refer to +:doc:`building-guest-images`. The Storm plugin requires an image to be tagged in the sahara image registry with two tags: 'storm' and '' (e.g. '1.1.0'). diff --git a/doc/source/user/vanilla-imagebuilder.rst b/doc/source/user/vanilla-imagebuilder.rst deleted file mode 100644 index cf609fdd92..0000000000 --- a/doc/source/user/vanilla-imagebuilder.rst +++ /dev/null @@ -1,94 +0,0 @@ -.. _diskimage-builder-label: - -Building Images for Vanilla Plugin -================================== - -In this document you will find instruction on how to build Ubuntu, Fedora, and -CentOS images with Apache Hadoop version 2.x.x. - -As of now the vanilla plugin works with images with pre-installed versions of -Apache Hadoop. To simplify the task of building such images we use -`Disk Image Builder `_. - -`Disk Image Builder` builds disk images using elements. An element is a -particular set of code that alters how the image is built, or runs within the -chroot to prepare the image. - -Elements for building vanilla images are stored in the -`Sahara image elements repository `_ - -.. note:: - - Sahara requires images with cloud-init package installed: - - * `For Ubuntu 16.04 `_ - * `For CentOS 7 `_ - * `For Fedora `_ - -To create vanilla images follow these steps: - -1. Clone repository "https://git.openstack.org/cgit/openstack/sahara-image-elements" - locally. - -2. Use tox to build images. - - You can run the command below in sahara-image-elements - directory to build images. By default this script will attempt to create - cloud images for all versions of supported plugins and all operating systems - (subset of Ubuntu, Fedora, and CentOS depending on plugin). - - .. sourcecode:: console - - tox -e venv -- sahara-image-create -u - - If you want to build Vanilla 2.7.1 image with centos 7 just execute: - - .. sourcecode:: console - - tox -e venv -- sahara-image-create -p vanilla -v 2.7.1 -i centos7 - - Tox will create a virtualenv and install required python packages in it, - clone the repositories "https://git.openstack.org/cgit/openstack/diskimage-builder" and - "https://git.openstack.org/cgit/openstack/sahara-image-elements" and export necessary - parameters. - -* ``DIB_HADOOP_VERSION`` - version of Hadoop to install -* ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin) -* ``OOZIE_DOWNLOAD_URL`` - download link for OOZIE (we have built - Oozie libs here: ``https://tarballs.openstack.org/sahara-extra/dist/oozie/oozie-4.2.0-hadoop-2.7.1.tar.gz``) -* ``SPARK_DOWNLOAD_URL`` - download link for Spark -* ``HIVE_VERSION`` - version of Hive to install - (currently supports only 0.11.0) -* ``ubuntu_image_name`` -* ``fedora_image_name`` -* ``DIB_IMAGE_SIZE`` - parameter that specifies a volume of hard disk - of instance. You need to specify it only for Fedora because Fedora - doesn't use all available volume -* ``DIB_COMMIT_ID`` - latest commit id of diskimage-builder project -* ``SAHARA_ELEMENTS_COMMIT_ID`` - latest commit id of - sahara-image-elements project - - NOTE: If you don't want to use default values, you should set your values - of parameters. - - Then it will create required cloud images using image elements that install - all the necessary packages and configure them. You will find created images - in the parent directory. - -.. note:: - - Disk Image Builder will generate QCOW2 images, used with the default - OpenStack Qemu/KVM hypervisors. If your OpenStack uses a different - hypervisor, the generated image should be converted to an appropriate - format. - - VMware Nova backend requires VMDK image format. You may use qemu-img - utility to convert a QCOW2 image to VMDK. - - .. sourcecode:: console - - qemu-img convert -O vmdk .qcow2 .vmdk - - -For finer control of diskimage-create.sh see the `official documentation -`_ diff --git a/doc/source/user/vanilla-plugin.rst b/doc/source/user/vanilla-plugin.rst index 967b500969..def1e42d0d 100644 --- a/doc/source/user/vanilla-plugin.rst +++ b/doc/source/user/vanilla-plugin.rst @@ -7,10 +7,43 @@ a cluster with Apache Hadoop. Since the Newton release Spark is integrated into the Vanilla plugin so you can launch Spark jobs on a Vanilla cluster. -For cluster provisioning prepared images should be used. They already have -Apache Hadoop 2.7.1 installed. +Images +------ -You may build images by yourself using :doc:`vanilla-imagebuilder`. +For cluster provisioning, prepared images should be used. + +.. list-table:: Support matrix for the `vanilla` plugin + :widths: 15 15 20 15 35 + :header-rows: 1 + + * - Version + (image tag) + - Distribution + - Build method + - Version + (build parameter) + - Notes + + * - 2.8.2 + - Ubuntu 16.04, CentOS 7 + - sahara-image-create + - 2.8.2 + - Hive 2.3.2, Oozie 4.3.0 + + * - 2.7.5 + - Ubuntu 16.04, CentOS 7 + - sahara-image-create + - 2.7.5 + - Hive 2.3.2, Oozie 4.3.0 + + * - 2.7.1 + - Ubuntu 16.04, CentOS 7 + - sahara-image-create + - 2.7.1 + - Hive 0.11.0, Oozie 4.2.0 + +For more information about building image, refer to +:doc:`building-guest-images`. Vanilla plugin requires an image to be tagged in Sahara Image Registry with two tags: 'vanilla' and '' (e.g. '2.7.1'). @@ -18,6 +51,41 @@ two tags: 'vanilla' and '' (e.g. '2.7.1'). The image requires a username. For more information, refer to the :doc:`registering-image` section. +Build settings +~~~~~~~~~~~~~~ + +When ``sahara-image-create`` is used, you can override few settings +by exporting the corresponding environment variables +before starting the build command: + +* ``DIB_HADOOP_VERSION`` - version of Hadoop to install +* ``HIVE_VERSION`` - version of Hive to install +* ``OOZIE_DOWNLOAD_URL`` - download link for Oozie (we have built + Oozie libs here: https://tarballs.openstack.org/sahara-extra/dist/oozie/) +* ``SPARK_DOWNLOAD_URL`` - download link for Spark + +Vanilla Plugin Requirements +--------------------------- + +The image building tools described in :ref:`building-guest-images-label` +add the required software to the image and their usage is strongly suggested. +Nevertheless, here are listed the software that should be pre-loaded +on the guest image so that it can be used to create Vanilla clusters: + +* ssh-client installed +* Java (version >= 7) +* Apache Hadoop installed +* 'hadoop' user created + +See :doc:`hadoop-swift` for information on using Swift with your sahara cluster +(for EDP support Swift integration is currently required). + +To support EDP, the following components must also be installed on the guest: + +* Oozie version 4 or higher +* mysql/mariadb +* hive + Cluster Validation ------------------ diff --git a/doc/test/redirect-tests.txt b/doc/test/redirect-tests.txt index a9e6a37db8..267d8780e8 100644 --- a/doc/test/redirect-tests.txt +++ b/doc/test/redirect-tests.txt @@ -1,3 +1,7 @@ /sahara/pike/contributor/launchpad.html 301 /sahara/pike/contributor/project.html /sahara/queens/contributor/launchpad.html 301 /sahara/queens/contributor/project.html /sahara/latest/contributor/launchpad.html 301 /sahara/latest/contributor/project.html +/sahara/latest/user/vanilla-imagebuilder.html 301 /sahara/latest/user/vanilla-plugin.html +/sahara/latest/user/cdh-imagebuilder.html 301 /sahara/latest/user/cdh-plugin.html +/sahara/latest/user/guest-requirements.html 301 /sahara/latest/user/building-guest-images.html +/sahara/rocky/user/guest-requirements.html 301 /sahara/rocky/user/building-guest-images.html