openstack
/
sahara
Code Issues Proposed changes

Fix doc8 check failures 

Resolves D000 and D001 checks in sahara docs.

Change-Id: If6d577859170ae55d578d70885a9ed5a6b0db651
Closes-Bug: #1514887
This commit is contained in:
Alok Jani 2015-11-13 18:36:08 +05:30
parent f0d11daf6d
commit 7ba05c5e30
30 changed files with 577 additions and 449 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
doc/source/devref/adding_database_migrations.rst
View File

@ -1,16 +1,18 @@
Adding Database Migrations
==========================
The migrations in ``sahara/db/migration/alembic_migrations/versions`` contain the changes needed to migrate
between Sahara database revisions. A migration occurs by executing a script that details the changes needed to
upgrade or downgrade the database. The migration scripts are ordered so that multiple scripts
can run sequentially. The scripts are executed by Sahara's migration wrapper
which uses the Alembic library to manage the migration. Sahara supports
The migrations in ``sahara/db/migration/alembic_migrations/versions`` contain
the changes needed to migrate between Sahara database revisions. A migration
occurs by executing a script that details the changes needed to upgrade or
downgrade the database. The migration scripts are ordered so that multiple
scripts can run sequentially. The scripts are executed by Sahara's migration
wrapper which uses the Alembic library to manage the migration. Sahara supports
migration from Icehouse or later.
Any code modifications that change the structure of the database require a migration script so that
previously existing databases will continue to function when the new code is released. This page gives a
brief overview of how to add the migration.
Any code modifications that change the structure of the database require a
migration script so that previously existing databases will continue to
function when the new code is released. This page gives a brief overview of how
to add the migration.
Generate a New Migration Script
+++++++++++++++++++++++++++++++
@ -21,22 +23,26 @@ To generate a migration stub to be filled in by the developer::
$ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision"
To autogenerate a migration script that reflects the current structure of the database::
To autogenerate a migration script that reflects the current structure of the
database::
$ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" --autogenerate
Each of these commands will create a file of the form ``revision_description`` where ``revision`` is a string
generated by Alembic and ``description`` is based on the text passed with the ``-m`` option.
Each of these commands will create a file of the form ``revision_description``
where ``revision`` is a string generated by Alembic and ``description`` is
based on the text passed with the ``-m`` option.
Follow the Sahara Naming Convention
+++++++++++++++++++++++++++++++++++
By convention Sahara uses 3-digit revision numbers, and this scheme differs from the strings generated by Alembic.
Consequently, it's necessary to rename the generated script and modify the revision identifiers in the script.
By convention Sahara uses 3-digit revision numbers, and this scheme differs
from the strings generated by Alembic. Consequently, it's necessary to rename
the generated script and modify the revision identifiers in the script.
Open the new script and look for the variable ``down_revision``. The value should be a 3-digit numeric string, and it
identifies the current revision number of the database. Set the ``revision`` value to the ``down_revision`` value + 1.
For example, the lines::
Open the new script and look for the variable ``down_revision``. The value
should be a 3-digit numeric string, and it identifies the current revision
number of the database. Set the ``revision`` value to the ``down_revision``
value + 1. For example, the lines::
# revision identifiers, used by Alembic.
revision = '507eb70202af'
@ -48,7 +54,8 @@ will become::
revision = '007'
down_revision = '006'
Modify any comments in the file to match the changes and rename the file to match the new revision number::
Modify any comments in the file to match the changes and rename the file to
match the new revision number::
$ mv 507eb70202af_my_new_revision.py 007_my_new_revision.py
@ -56,7 +63,10 @@ $ mv 507eb70202af_my_new_revision.py 007_my_new_revision.py
Add Alembic Operations to the Script
++++++++++++++++++++++++++++++++++++
The migration script contains method ``upgrade()``. Since Kilo release Sahara doesn't support downgrades. Fill in this method with the appropriate Alembic operations to perform upgrades. In the above example, an upgrade will move from revision '006' to revision '007'.
The migration script contains method ``upgrade()``. Since Kilo release Sahara
doesn't support downgrades. Fill in this method with the appropriate Alembic
operations to perform upgrades. In the above example, an upgrade will move from
revision '006' to revision '007'.
Command Summary for sahara-db-manage
++++++++++++++++++++++++++++++++++++

3
doc/source/devref/development.environment.rst
View File

@ -119,7 +119,8 @@ Just add the following lines to .git/hooks/pre-commit and do chmod +x for it.
# Run fast checks (PEP8 style check and PyFlakes fast static analysis)
tools/run_fast_checks
You can added the same check for pre-push, for example, run_tests and run_pylint.
You can added the same check for pre-push, for example, run_tests and
run_pylint.
3. Running static analysis (PyLint)

28
doc/source/devref/development.guidelines.rst
View File

@ -20,9 +20,10 @@ To check your code against PEP 8 and bashate run:
Static analysis
---------------
The static analysis checks are optional in Sahara. but they are still very useful.
The gate job will inform you if the number of static analysis warnings
has increased after your change. We recommend to always check the static warnings.
The static analysis checks are optional in Sahara. but they are still very
useful. The gate job will inform you if the number of static analysis warnings
has increased after your change. We recommend to always check the static
warnings.
To run the check commit yor change first and execute the following command:
@ -33,15 +34,16 @@ To run the check commit yor change first and execute the following command:
Modification of Upstream Files
------------------------------
We never modify upstream files in Sahara. Any changes in upstream files should be made
in the upstream project and then merged back in to Sahara. This includes whitespace
changes, comments, and typos. Any change requests containing upstream file modifications
are almost certain to receive lots of negative reviews. Be warned.
We never modify upstream files in Sahara. Any changes in upstream files should
be made in the upstream project and then merged back in to Sahara. This
includes whitespace changes, comments, and typos. Any change requests
containing upstream file modifications are almost certain to receive lots of
negative reviews. Be warned.
Examples of upstream files are default xml configuration files used to configure Hadoop, or
code imported from the OpenStack Oslo project. The xml files will usually be found in
``resource`` directories with an accompanying ``README`` file that identifies where the
files came from. For example:
Examples of upstream files are default xml configuration files used to
configure Hadoop, or code imported from the OpenStack Oslo project. The xml
files will usually be found in ``resource`` directories with an accompanying
``README`` file that identifies where the files came from. For example:
.. sourcecode:: console
@ -101,8 +103,8 @@ Run the following command to build docs locally.
$ tox -e docs
After it you can access generated docs in ``doc/build/`` directory, for example,
main page - ``doc/build/html/index.html``.
After it you can access generated docs in ``doc/build/`` directory, for
example, main page - ``doc/build/html/index.html``.
To make docs generation process faster you can use:

12
doc/source/devref/devstack.rst
View File

@ -119,7 +119,8 @@ appended after the git repo URL as follows:
5. Once previous step is finished Devstack will print Horizon URL. Navigate to
this URL and login with login "admin" and password from ``local.conf``.
6. Congratulations! You have OpenStack running in your VM and ready to launch VMs inside that VM :)
6. Congratulations! You have OpenStack running in your VM and ready to launch
VMs inside that VM :)
Managing sahara in DevStack
@ -137,6 +138,7 @@ Setting fixed IP address for VMware Fusion VM
---------------------------------------------
1. Open file ``/Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf``
2. There is a block named "subnet". It might look like this:
.. sourcecode:: text
@ -144,9 +146,11 @@ Setting fixed IP address for VMware Fusion VM
subnet 192.168.55.0 netmask 255.255.255.0 {
range 192.168.55.128 192.168.55.254;
3. You need to pick an IP address outside of that range. For example - ``192.168.55.20``
4. Copy VM MAC address from VM settings->Network->Advanced
5. Append the following block to file ``dhcpd.conf`` (don't forget to replace ``VM_HOSTNAME`` and ``VM_MAC_ADDRESS`` with actual values):
3. You need to pick an IP address outside of that range. For example -
``192.168.55.20`` 4. Copy VM MAC address from VM settings->Network->Advanced
5. Append the following block to file ``dhcpd.conf`` (don't forget to replace
``VM_HOSTNAME`` and ``VM_MAC_ADDRESS`` with actual values):
.. sourcecode:: text

39
doc/source/devref/edp.spi.rst
View File

@ -1,13 +1,13 @@
Elastic Data Processing (EDP) SPI
=================================
EDP job engine objects provide methods for creating, monitoring, and terminating
jobs on Sahara clusters. Provisioning plugins that support EDP must return an
EDP job engine object from the :ref:`get_edp_engine` method described in
:doc:`plugin.spi`.
EDP job engine objects provide methods for creating, monitoring, and
terminating jobs on Sahara clusters. Provisioning plugins that support EDP must
return an EDP job engine object from the :ref:`get_edp_engine` method described
in :doc:`plugin.spi`.
Sahara provides subclasses of the base job engine interface that support EDP on
clusters running Oozie or on Spark standalone clusters. These are described
Sahara provides subclasses of the base job engine interface that support EDP
on clusters running Oozie or on Spark standalone clusters. These are described
below.
.. _edp_spi_job_types:
@ -60,8 +60,8 @@ cancel_job(job_execution)
Stops the running job whose id is stored in the job_execution object.
*Returns*: None if the operation was unsuccessful or an updated job status value
*Returns*: None if the operation was unsuccessful or an updated job status
value
get_job_status(job_execution)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -133,10 +133,10 @@ JobEngine. It provides implementations for all of the methods in the base
interface but adds a few more abstract methods.
Note, the *validate_job_execution(cluster, job, data)* method does basic checks
on the job configuration but probably should be overloaded to include additional
checks on the cluster configuration. For example, the job engines for plugins
that support Oozie add checks to make sure that the Oozie service is up and
running.
on the job configuration but probably should be overloaded to include
additional checks on the cluster configuration. For example, the job engines
for plugins that support Oozie add checks to make sure that the Oozie service
is up and running.
get_hdfs_user()
@ -160,8 +160,8 @@ creates the HDFS directory *dir_name* under the user specified by
if the HDFS user is 'hadoop' and the dir_name is 'test' this method would
create '/user/hadoop/test'.
The reason that this method is broken out in the interface as an abstract method
is that different versions of Hadoop treat path creation differently.
The reason that this method is broken out in the interface as an abstract
method is that different versions of Hadoop treat path creation differently.
*Returns*: None
@ -207,14 +207,15 @@ The sahara.service.edp.spark.engine.SparkJobEngine class provides a full EDP
implementation for Spark standalone clusters.
Note, the *validate_job_execution(cluster, job, data)* method does basic checks
on the job configuration but probably should be overloaded to include additional
checks on the cluster configuration. For example, the job engine returned by the
Spark plugin checks that the Spark version is >= 1.0.0 to ensure that
*spark-submit* is available.
on the job configuration but probably should be overloaded to include
additional checks on the cluster configuration. For example, the job engine
returned by the Spark plugin checks that the Spark version is >= 1.0.0 to
ensure that *spark-submit* is available.
get_driver_classpath(self)
~~~~~~~~~~~~~~~~~~~~~~~~~~
Returns driver class path.
*Returns*: a string of the following format ' --driver-class-path *class_path_value*'
*Returns*: a string of the following format ' --driver-class-path
*class_path_value*'

14
doc/source/devref/how_to_build_oozie.rst
View File

@ -40,10 +40,14 @@ JDK Versions
By default, the build configuration enforces that JDK 1.6.* is being used.
There are 2 build properties that can be used to change the JDK version requirements:
There are 2 build properties that can be used to change the JDK version
requirements:
* ``javaVersion`` specifies the version of the JDK used to compile (default 1.6)
* ``targetJavaVersion`` specifies the version of the generated bytecode (default 1.6)
* ``javaVersion`` specifies the version of the JDK used to compile (default
1.6)
* ``targetJavaVersion`` specifies the version of the generated bytecode
(default 1.6)
For example, to specify 1.7 JDK version, build command should contain
``-D javaVersion=1.7 -D tagetJavaVersion=1.7`` flags.
@ -63,4 +67,6 @@ used:
Also, pig version can be passed as maven property with ``-D pig.version=x.x.x``
flag.
Similar instruction to build oozie.tar.gz you may find there: http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Building_Oozie
Similar instruction to build oozie.tar.gz you may find there:
http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Building_Oozie

3
doc/source/devref/how_to_participate.rst
View File

@ -43,7 +43,8 @@ How to stay in touch with the community?
* Weekly on Thursdays at 1400 UTC and 1800 UTC (on alternate weeks)
* IRC channel: ``#openstack-meeting-alt`` (1800UTC) and ``#openstack-meeting-3`` (1400UTC)
* IRC channel: ``#openstack-meeting-alt`` (1800UTC) and
``#openstack-meeting-3`` (1400UTC)
How to send your first patch on review?

44
doc/source/devref/jenkins.rst
View File

@ -1,23 +1,41 @@
Continuous Integration with Jenkins
===================================
Each change made to Sahara core code is tested with unit and integration tests and style checks flake8.
Each change made to Sahara core code is tested with unit and integration tests
and style checks flake8.
Unit tests and style checks are performed on public `OpenStack Jenkins
<https://jenkins.openstack.org/>`_ managed by `Zuul
<http://status.openstack.org/zuul/>`_.
Unit tests and style checks are performed on public `OpenStack Jenkins <https://jenkins.openstack.org/>`_ managed by `Zuul <http://status.openstack.org/zuul/>`_.
Unit tests are checked using python 2.7.
The result of those checks and Unit tests are +1 or -1 to *Verify* column in a code review from *Jenkins* user.
The result of those checks and Unit tests are +1 or -1 to *Verify* column in a
code review from *Jenkins* user.
Integration tests check CRUD operations for Image Registry, Templates and Clusters.
Also a test job is launched on a created Cluster to verify Hadoop work.
Integration tests check CRUD operations for Image Registry, Templates and
Clusters. Also a test job is launched on a created Cluster to verify Hadoop
work.
All integration tests are launched by `Jenkins <https://sahara.mirantis.com/jenkins/>`_ on the internal Mirantis OpenStack Lab.
Jenkins keeps a pool of VMs to run tests in parallel. Even with the pool of VMs integration testing may take a while.
Jenkins is controlled for the most part by Zuul which determines what jobs are run when.
Zuul status is available by address: `Zuul Status <https://sahara.mirantis.com/zuul>`_.
For more information see: `Sahara Hadoop Cluster CI <https://wiki.openstack.org/wiki/Sahara/SaharaCI>`_.
All integration tests are launched by `Jenkins
<https://sahara.mirantis.com/jenkins/>`_ on the internal Mirantis OpenStack
Lab.
The integration tests result is +1 or -1 to *Verify* column in a code review from *Sahara Hadoop Cluster CI* user.
Jenkins keeps a pool of VMs to run tests in parallel. Even with the pool of VMs
integration testing may take a while.
You can put *sahara-ci-recheck* in comment, if you want to recheck sahara-ci jobs.
Also, you can put *recheck* in comment, if you want to recheck both jenkins and sahara-ci jobs.
Jenkins is controlled for the most part by Zuul which determines what jobs are
run when.
Zuul status is available by address: `Zuul Status
<https://sahara.mirantis.com/zuul>`_.
For more information see: `Sahara Hadoop Cluster CI
<https://wiki.openstack.org/wiki/Sahara/SaharaCI>`_.
The integration tests result is +1 or -1 to *Verify* column in a code review
from *Sahara Hadoop Cluster CI* user.
You can put *sahara-ci-recheck* in comment, if you want to recheck sahara-ci
jobs. Also, you can put *recheck* in comment, if you want to recheck both
jenkins and sahara-ci jobs.

10
doc/source/devref/launchpad.rst
View File

@ -1,8 +1,8 @@
Project hosting
===============
`Launchpad`_ hosts the Sahara project. The Sahara project homepage on Launchpad is
http://launchpad.net/sahara.
`Launchpad`_ hosts the Sahara project. The Sahara project homepage on Launchpad
is http://launchpad.net/sahara.
Launchpad credentials
---------------------
@ -18,9 +18,9 @@ OpenStack-related sites. These sites include:
Mailing list
------------
The mailing list email is ``openstack-dev@lists.openstack.org`` with subject prefix
``[sahara]``. To participate in the mailing list subscribe to the list at
http://lists.openstack.org/cgi-bin/mailman/listinfo
The mailing list email is ``openstack-dev@lists.openstack.org`` with subject
prefix ``[sahara]``. To participate in the mailing list subscribe to the list
at http://lists.openstack.org/cgi-bin/mailman/listinfo
Bug tracking
------------

8
doc/source/devref/log.guidelines.rst
View File

@ -5,8 +5,8 @@ Log Guidelines
Levels Guidelines
-----------------
During the Kilo release cycle the sahara community defined the following log levels:
During the Kilo release cycle the sahara community defined the following
log levels:
* Debug: Shows everything and is likely not suitable for normal production
operation due to the sheer size of logs generated (e.g. scripts executions,
@ -41,8 +41,8 @@ Now sahara uses string formatting defined in `PEP 3101`_ for logs.
Translation Guidelines
----------------------
All log levels except Debug requires translation. None of the separate cli tools packaged
with sahara contain log translations.
All log levels except Debug requires translation. None of the separate
cli tools packaged with sahara contain log translations.
* Debug: no translation
* Info: _LI

322
doc/source/devref/plugin.spi.rst
View File

@ -7,9 +7,10 @@ Plugin interface
get_versions()
~~~~~~~~~~~~~~
Returns all versions of Hadoop that could be used with the plugin.
It is responsibility of the plugin to make sure that all required images for each hadoop version are available,
as well as configs and whatever else that plugin needs to create the Hadoop cluster.
Returns all versions of Hadoop that could be used with the plugin. It is
responsibility of the plugin to make sure that all required images for each
hadoop version are available, as well as configs and whatever else that plugin
needs to create the Hadoop cluster.
*Returns*: list of strings - Hadoop versions
@ -18,27 +19,31 @@ as well as configs and whatever else that plugin needs to create the Hadoop clus
get_configs( hadoop_version)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lists all configs supported by plugin with descriptions, defaults and targets for which this config is applicable.
Lists all configs supported by plugin with descriptions, defaults and targets
for which this config is applicable.
*Returns*: list of configs
*Example return value*: ((“JobTracker heap size”, "JobTracker heap size, in MB", "int", “512”, `“mapreduce”`, "node", True, 1))
*Example return value*: ((“JobTracker heap size”, "JobTracker heap size, in
MB", "int", “512”, `“mapreduce”`, "node", True, 1))
get_node_processes( hadoop_version)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Returns all supported services and node processes for a given Hadoop version.
Each node process belongs to a single service and that relationship is reflected in the returned dict object.
See example for details.
Each node process belongs to a single service and that relationship is
reflected in the returned dict object. See example for details.
*Returns*: dictionary having entries (service -> list of processes)
*Example return value*: {"mapreduce": ["tasktracker", "jobtracker"], "hdfs": ["datanode", "namenode"]}
*Example return value*: {"mapreduce": ["tasktracker", "jobtracker"], "hdfs":
["datanode", "namenode"]}
get_required_image_tags( hadoop_version)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lists tags, that should be added to OpenStack Image via Image Registry. Tags are used to filter Images by plugin and hadoop version.
Lists tags, that should be added to OpenStack Image via Image Registry. Tags
are used to filter Images by plugin and hadoop version.
*Returns*: list of tags
@ -47,11 +52,14 @@ Lists tags, that should be added to OpenStack Image via Image Registry. Tags are
validate(cluster)
~~~~~~~~~~~~~~~~~
Validates a given cluster object. Raises *SaharaException* with meaningful message.
Validates a given cluster object. Raises *SaharaException* with meaningful
message.
*Returns*: None
*Example exception*: <NotSingleNameNodeException {code='NOT_SINGLE_NAME_NODE', message='Hadoop cluster should contain only 1 NameNode instance. Actual NN count is 2' }>
*Example exception*: <NotSingleNameNodeException {code='NOT_SINGLE_NAME_NODE',
message='Hadoop cluster should contain only 1 NameNode instance. Actual NN
count is 2' }>
validate_scaling(cluster, existing, additional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -75,24 +83,26 @@ For instance, plugin can ask for additional VMs for the management tool.
configure_cluster(cluster)
~~~~~~~~~~~~~~~~~~~~~~~~~~
Configures cluster on provisioned by sahara VMs.
In this function plugin should perform all actions like adjusting OS, installing required packages (including Hadoop, if needed), configuring Hadoop, etc.
Configures cluster on provisioned by sahara VMs. In this function plugin
should perform all actions like adjusting OS, installing required packages
(including Hadoop, if needed), configuring Hadoop, etc.
*Returns*: None
start_cluster(cluster)
~~~~~~~~~~~~~~~~~~~~~~
Start already configured cluster. This method is guaranteed to be called only on cluster which was already prepared with configure_cluster(...) call.
Start already configured cluster. This method is guaranteed to be called only
on cluster which was already prepared with configure_cluster(...) call.
*Returns*: None
scale_cluster(cluster, instances)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scale an existing cluster with additional instances. Instances argument is a list
of ready-to-configure instances. Plugin should do all configuration operations in this
method and start all services on those instances.
Scale an existing cluster with additional instances. Instances argument is a
list of ready-to-configure instances. Plugin should do all configuration
operations in this method and start all services on those instances.
*Returns*: None
@ -103,17 +113,19 @@ get_edp_engine(cluster, job_type)
Returns an EDP job engine object that supports the specified job_type on the
given cluster, or None if there is no support. The EDP job engine object
returned must implement the interface described in :doc:`edp.spi`. The job_type
is a String matching one of the job types listed in :ref:`edp_spi_job_types`.
returned must implement the interface described in :doc:`edp.spi`. The
job_type is a String matching one of the job types listed in
:ref:`edp_spi_job_types`.
*Returns*: an EDP job engine object or None
decommission_nodes(cluster, instances)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scale cluster down by removing a list of instances. Plugin should stop services on a provided list
of instances. Plugin also may want to update some configurations on other instances, so this
method is the right place to do that.
Scale cluster down by removing a list of instances. Plugin should stop services
on a provided list of instances. Plugin also may want to update some
configurations on other instances, so this method is the right place to do
that.
*Returns*: None
@ -129,7 +141,9 @@ See “Cluster Lifecycle for Config File Mode” section below for clarification
on_terminate_cluster(cluster)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When user terminates cluster, sahara simply shuts down all the cluster VMs. This method is guaranteed to be invoked before that, allowing plugin to do some clean-up.
When user terminates cluster, sahara simply shuts down all the cluster VMs.
This method is guaranteed to be invoked before that, allowing plugin to do some
clean-up.
*Returns*: None
@ -140,19 +154,22 @@ When user requests sahara to automatically create security group for the node
group (``auto_security_group`` property set to True), sahara will call this
plugin method to get list of ports that need to be opened.
*Returns*: list of ports to be open in auto security group for the given node group
*Returns*: list of ports to be open in auto security group for the given node
group
def get_edp_job_types(versions)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Optional method, which provides ability to see all supported job types for specified plugin versions
Optional method, which provides ability to see all supported job types for
specified plugin versions
*Returns*: dict with supported job types for specified versions of plugin
def recommend_configs(self, cluster, scaling=False)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Optional method, which provides recommendations for cluster configuration before creating/scaling operation.
Optional method, which provides recommendations for cluster configuration
before creating/scaling operation.
*Returns*: None
@ -163,8 +180,9 @@ Here is a description of all the objects involved in the API.
Notes:
- cluster and node_group have ‘extra’ field allowing plugin to persist any complementary info about the cluster.
- node_process is just a process that runs at some node in cluster.
- cluster and node_group have ‘extra’ field allowing plugin to persist any
complementary info about the cluster.
- node_process is just a process that runs at some node in cluster.
Example list of node processes:
@ -180,30 +198,36 @@ Config
An object, describing one configuration entry
+-------------------+--------+--------------------------------------------------------------------------------+
| Property | Type | Description |
+===================+========+================================================================================+
| name | string | Config name. |
+-------------------+--------+--------------------------------------------------------------------------------+
| description | string | A hint for user, what this config is used for. |
+-------------------+--------+--------------------------------------------------------------------------------+
| config_type | enum | possible values are: 'string', 'integer', 'boolean', 'enum'. |
+-------------------+--------+--------------------------------------------------------------------------------+
| config_values | list | List of possible values, if config_type is enum. |
+-------------------+--------+--------------------------------------------------------------------------------+
| default_value | string | Default value for config. |
+-------------------+--------+--------------------------------------------------------------------------------+
| applicable_target | string | The target could be either a service returned by get_node_processes(...) call |
| | | in form of 'service:<service name>', or 'general'. |
+-------------------+--------+--------------------------------------------------------------------------------+
| scope | enum | Could be either 'node' or 'cluster'. |
+-------------------+--------+--------------------------------------------------------------------------------+
| is_optional | bool | If is_optional is False and no default_value is specified, user should |
| | | provide a value. |
+-------------------+--------+--------------------------------------------------------------------------------+
| priority | int | 1 or 2. A Hint for UI. Configs with priority *1* are always displayed. |
| | | Priority *2* means user should click a button to see the config. |
+-------------------+--------+--------------------------------------------------------------------------------+
+-------------------+--------+------------------------------------------------+
| Property | Type | Description |
+===================+========+================================================+
| name | string | Config name. |
+-------------------+--------+------------------------------------------------+
| description | string | A hint for user, what this config is used for. |
+-------------------+--------+------------------------------------------------+
| config_type | enum | possible values are: 'string', 'integer', |
| | | 'boolean', 'enum'. |
+-------------------+--------+------------------------------------------------+
| config_values | list | List of possible values, if config_type is |
| | | enum. |
+-------------------+--------+------------------------------------------------+
| default_value | string | Default value for config. |
+-------------------+--------+------------------------------------------------+
| applicable_target | string | The target could be either a service returned |
| | | by get_node_processes(...) call |
| | | in form of 'service:<service name>', or |
| | | 'general'. |
+-------------------+--------+------------------------------------------------+
| scope | enum | Could be either 'node' or 'cluster'. |
+-------------------+--------+------------------------------------------------+
| is_optional | bool | If is_optional is False and no default_value |
| | | is specified, user should provide a value |
+-------------------+--------+------------------------------------------------+
| priority | int | 1 or 2. A Hint for UI. Configs with priority |
| | | *1* are always displayed. |
| | | Priority *2* means user should click a button |
| | | to see the config. |
+-------------------+--------+------------------------------------------------+
User Input
@ -225,31 +249,33 @@ Instance
An instance created for cluster.
+---------------+---------+---------------------------------------------------------+
| Property | Type | Description |
+===============+=========+=========================================================+
| instance_id | string | Unique instance identifier. |
+---------------+---------+---------------------------------------------------------+
| instance_name | string | OpenStack Instance name. |
+---------------+---------+---------------------------------------------------------+
| internal_ip | string | IP to communicate with other instances. |
+---------------+---------+---------------------------------------------------------+
| management_ip | string | IP of instance, accessible outside of internal network. |
+---------------+---------+---------------------------------------------------------+
| volumes | list | List of volumes attached to instance. Empty if |
| | | ephemeral drive is used. |
+---------------+---------+---------------------------------------------------------+
| nova_info | object | Nova Instance object. |
+---------------+---------+---------------------------------------------------------+
| username | string | Username, that sahara uses for establishing remote |
| | | connections to instance. |
+---------------+---------+---------------------------------------------------------+
| hostname | string | Same as instance_name. |
+---------------+---------+---------------------------------------------------------+
| fqdn | string | Fully qualified domain name for this instance. |
+---------------+---------+---------------------------------------------------------+
| remote | helpers | Object with helpers for performing remote operations |
+---------------+---------+---------------------------------------------------------+
+---------------+---------+---------------------------------------------------+
| Property | Type | Description |
+===============+=========+===================================================+
| instance_id | string | Unique instance identifier. |
+---------------+---------+---------------------------------------------------+
| instance_name | string | OpenStack Instance name. |
+---------------+---------+---------------------------------------------------+
| internal_ip | string | IP to communicate with other instances. |
+---------------+---------+---------------------------------------------------+
| management_ip | string | IP of instance, accessible outside of internal |
| | | network. |
+---------------+---------+---------------------------------------------------+
| volumes | list | List of volumes attached to instance. Empty if |
| | | ephemeral drive is used. |
+---------------+---------+---------------------------------------------------+
| nova_info | object | Nova Instance object. |
+---------------+---------+---------------------------------------------------+
| username | string | Username, that sahara uses for establishing |
| | | remote connections to instance. |
+---------------+---------+---------------------------------------------------+
| hostname | string | Same as instance_name. |
+---------------+---------+---------------------------------------------------+
| fqdn | string | Fully qualified domain name for this instance. |
+---------------+---------+---------------------------------------------------+
| remote | helpers | Object with helpers for performing remote |
| | | operations |
+---------------+---------+---------------------------------------------------+
Node Group
@ -257,78 +283,86 @@ Node Group
Group of instances.
+----------------------+--------+--------------------------------------------------------+
| Property | Type | Description |
+======================+========+========================================================+
| name | string | Name of this Node Group in Cluster. |
+----------------------+--------+--------------------------------------------------------+
| flavor_id | string | OpenStack Flavor used to boot instances. |
+----------------------+--------+--------------------------------------------------------+
| image_id | string | Image id used to boot instances. |
+----------------------+--------+--------------------------------------------------------+
| node_processes | list | List of processes running on each instance. |
+----------------------+--------+--------------------------------------------------------+
| node_configs | dict | Configs dictionary, applied to instances. |
+----------------------+--------+--------------------------------------------------------+
| volumes_per_node | int | Number of volumes mounted to each instance. 0 means |
| | | use ephemeral drive. |
+----------------------+--------+--------------------------------------------------------+
| volumes_size | int | Size of each volume (GB). |
+----------------------+--------+--------------------------------------------------------+
| volumes_mount_prefix | string | Prefix added to mount path of each volume. |
+----------------------+--------+--------------------------------------------------------+
| floating_ip_pool | string | Floating IP Pool name. All instances in the Node Group |
| | | will have Floating IPs assigned from this pool. |
+----------------------+--------+--------------------------------------------------------+
| count | int | Number of instances in this Node Group. |
+----------------------+--------+--------------------------------------------------------+
| username | string | Username used by sahara to establish remote |
| | | connections to instances. |
+----------------------+--------+--------------------------------------------------------+
| configuration | dict | Merged dictionary of node configurations and cluster |
| | | configurations. |
+----------------------+--------+--------------------------------------------------------+
| storage_paths | list | List of directories where storage should be placed. |
+----------------------+--------+--------------------------------------------------------+
+----------------------+--------+---------------------------------------------+
| Property | Type | Description |
+======================+========+=============================================+
| name | string | Name of this Node Group in Cluster. |
+----------------------+--------+---------------------------------------------+
| flavor_id | string | OpenStack Flavor used to boot instances. |
+----------------------+--------+---------------------------------------------+
| image_id | string | Image id used to boot instances. |
+----------------------+--------+---------------------------------------------+
| node_processes | list | List of processes running on each instance. |
+----------------------+--------+---------------------------------------------+
| node_configs | dict | Configs dictionary, applied to instances. |
+----------------------+--------+---------------------------------------------+
| volumes_per_node | int | Number of volumes mounted to each instance. |
| | | 0 means use ephemeral drive. |
+----------------------+--------+---------------------------------------------+
| volumes_size | int | Size of each volume (GB). |
+----------------------+--------+---------------------------------------------+
| volumes_mount_prefix | string | Prefix added to mount path of each volume. |
+----------------------+--------+---------------------------------------------+
| floating_ip_pool | string | Floating IP Pool name. All instances in the |
| | | Node Group will have Floating IPs assigned |
| | | from this pool. |
+----------------------+--------+---------------------------------------------+
| count | int | Number of instances in this Node Group. |
+----------------------+--------+---------------------------------------------+
| username | string | Username used by sahara to establish remote |
| | | connections to instances. |
+----------------------+--------+---------------------------------------------+
| configuration | dict | Merged dictionary of node configurations |
| | | and cluster configurations. |
+----------------------+--------+---------------------------------------------+
| storage_paths | list | List of directories where storage should be |
| | | placed. |
+----------------------+--------+---------------------------------------------+
Cluster
-------
Contains all relevant info about cluster.
This object is is provided to the plugin for both cluster creation and scaling.
The “Cluster Lifecycle” section below further specifies which fields are filled at which moment.
Contains all relevant info about cluster. This object is is provided to the
plugin for both cluster creation and scaling. The “Cluster Lifecycle” section
below further specifies which fields are filled at which moment.
+----------------------------+--------+-----------------------------------------------------------+
| Property | Type | Description |
+============================+========+===========================================================+
| name | string | Cluster name. |
+----------------------------+--------+-----------------------------------------------------------+
| tenant_id | string | OpenStack Tenant id where this Cluster is available. |
+----------------------------+--------+-----------------------------------------------------------+
| plugin_name | string | Plugin name. |
+----------------------------+--------+-----------------------------------------------------------+
| hadoop_version | string | Hadoop version running on instances. |
+----------------------------+--------+-----------------------------------------------------------+
| default_image_id | string | OpenStack image used to boot instances. |
+----------------------------+--------+-----------------------------------------------------------+
| node_groups | list | List of Node Groups. |
+----------------------------+--------+-----------------------------------------------------------+
| cluster_configs | dict | Dictionary of Cluster scoped configurations. |
+----------------------------+--------+-----------------------------------------------------------+
| cluster_template_id | string | Cluster Template used for Node Groups and Configurations. |
+----------------------------+--------+-----------------------------------------------------------+
| user_keypair_id | string | OpenStack keypair added to instances to make them |
| | | accessible for user. |
+----------------------------+--------+-----------------------------------------------------------+
| neutron_management_network | string | Neutron network ID. Instances will get fixed IPs in this |
| | | network if 'use_neutron' config is set to True. |
+----------------------------+--------+-----------------------------------------------------------+
| anti_affinity | list | List of processes that will be run on different hosts. |
+----------------------------+--------+-----------------------------------------------------------+
| description | string | Cluster Description. |
+----------------------------+--------+-----------------------------------------------------------+
| info | dict | Dictionary for additional information. |
+----------------------------+--------+-----------------------------------------------------------+
+----------------------------+--------+---------------------------------------+
| Property | Type | Description |
+============================+========+=======================================+
| name | string | Cluster name. |
+----------------------------+--------+---------------------------------------+
| tenant_id | string | OpenStack Tenant id where this |
| | | Cluster is available. |
+----------------------------+--------+---------------------------------------+
| plugin_name | string | Plugin name. |
+----------------------------+--------+---------------------------------------+
| hadoop_version | string | Hadoop version running on instances. |
+----------------------------+--------+---------------------------------------+
| default_image_id | string | OpenStack image used to boot |
| | | instances. |
+----------------------------+--------+---------------------------------------+
| node_groups | list | List of Node Groups. |
+----------------------------+--------+---------------------------------------+
| cluster_configs | dict | Dictionary of Cluster scoped |
| | | configurations. |
+----------------------------+--------+---------------------------------------+
| cluster_template_id | string | Cluster Template used for Node Groups |
| | | and Configurations. |
+----------------------------+--------+---------------------------------------+
| user_keypair_id | string | OpenStack keypair added to instances |
| | | to make them accessible for user. |
+----------------------------+--------+---------------------------------------+
| neutron_management_network | string | Neutron network ID. Instances will |
| | | get fixed IPs in this network if |
| | | 'use_neutron' config is set to True. |
+----------------------------+--------+---------------------------------------+
| anti_affinity | list | List of processes that will be run on |
| | | different hosts. |
+----------------------------+--------+---------------------------------------+
| description | string | Cluster Description. |
+----------------------------+--------+---------------------------------------+
| info | dict | Dictionary for additional information.|
+----------------------------+--------+---------------------------------------+
Validation Error

39
doc/source/horizon/dashboard.user.guide.rst
View File

@ -2,8 +2,8 @@ Sahara (Data Processing) UI User Guide
======================================
This guide assumes that you already have the Sahara service and Horizon
dashboard up and running. Don't forget to make sure that Sahara is registered in
Keystone. If you require assistance with that, please see the
dashboard up and running. Don't forget to make sure that Sahara is
registered in Keystone. If you require assistance with that, please see the
`installation guide <../installation.guide.html>`_.
The sections below give a panel by panel overview of setting up clusters
@ -25,8 +25,8 @@ Registering an Image
4) Enter the username of the cloud-init user on the image
5) Click on the tags that you want to add to the image. (A version ie: 1.2.1 and
a type ie: vanilla are required for cluster functionality)
5) Click on the tags that you want to add to the image. (A version ie: 1.2.1
and a type ie: vanilla are required for cluster functionality)
6) Click the "Done" button to finish the registration
@ -46,8 +46,8 @@ Create Node Group Templates
5) Choose a flavor for this template (based on your CPU/memory/disk needs)
6) Choose the storage location for your instance, this can be either "Ephemeral
Drive" or "Cinder Volume". If you choose "Cinder Volume", you will need to add
additional configuration
Drive" or "Cinder Volume". If you choose "Cinder Volume", you will need to
add additional configuration
7) Choose which processes should be run for any instances that are spawned from
this Node Group Template
@ -131,8 +131,8 @@ Data Sources
------------
Data Sources are where the input and output from your jobs are housed.
1) From the Data Processing/Data Sources page, click on the "Create Data Source"
button at the top right
1) From the Data Processing/Data Sources page, click on the "Create Data
Source" button at the top right
2) Give your Data Source a name
@ -223,8 +223,8 @@ status of your job to see when it has completed its run
- Additional configuration properties can be defined by clicking on the "Add"
button
- An example configuration entry might be mapred.mapper.class for the Name and
org.apache.oozie.example.SampleMapper for the Value
- An example configuration entry might be mapred.mapper.class for the Name
and org.apache.oozie.example.SampleMapper for the Value
5) Click on "Launch". To monitor the status of your job, you can navigate to
the Data Processing/Jobs panel
@ -241,8 +241,9 @@ status of your job to see when it has completed its run
Example Jobs
------------
There are sample jobs located in the Sahara repository. In this section, we
will give a walkthrough on how to run those jobs via the Horizon UI. These steps
assume that you already have a cluster up and running (in the "Active" state).
will give a walkthrough on how to run those jobs via the Horizon UI. These
steps assume that you already have a cluster up and running (in the "Active"
state).
1) Sample Pig job -
https://github.com/openstack/sahara/tree/master/etc/edp-examples/edp-pig/trim-spaces
@ -287,8 +288,8 @@ assume that you already have a cluster up and running (in the "Active" state).
- Name = pigsample, Job Type = Pig, Choose "example.pig" as the main binary
- Click on the "Libs" tab and choose "udf.jar", then hit the "Choose" button
beneath the dropdown, then click on "Create"
- Click on the "Libs" tab and choose "udf.jar", then hit the "Choose"
button beneath the dropdown, then click on "Create"
- Launch your job
@ -364,11 +365,11 @@ Additional Notes
existing Job. In order to be able to delete that job, you would
first need to delete any Job Templates that relate to that job.
2) In the examples above, we mention adding your username/password for the Swift
Data Sources. It should be noted that it is possible to configure Sahara such
that the username/password credentials are *not* required. For more
information on that, please refer to:
:doc:`Sahara Advanced Configuration Guide <../userdoc/advanced.configuration.guide>`
2) In the examples above, we mention adding your username/password for the
Swift Data Sources. It should be noted that it is possible to configure
Sahara such that the username/password credentials are *not* required. For
more information on that, please refer to: :doc:`Sahara Advanced
Configuration Guide <../userdoc/advanced.configuration.guide>`
Launching a cluster via the Cluster Creation Guide
--------------------------------------------------

37
doc/source/horizon/dev.environment.guide.rst
View File

@ -2,8 +2,8 @@ Sahara UI Dev Environment Setup
===============================
This page describes how to setup Horizon for developing Sahara by either
installing it as part of DevStack with Sahara or installing it in an isolated environment
and running from the command line.
installing it as part of DevStack with Sahara or installing it in an
isolated environment and running from the command line.
Install as a part of DevStack
-----------------------------
@ -11,8 +11,8 @@ Install as a part of DevStack
See the `DevStack guide <../devref/devstack.html>`_ for more information
on installing and configuring DevStack with Sahara.
After Horizon installation, it will contain a Data Processing tab under Projects tab.
Sahara UI source code will be located at
After Horizon installation, it will contain a Data Processing tab under
Projects tab. Sahara UI source code will be located at
``$DEST/horizon/openstack_dashboard/contrib/sahara/content/data_processing``
where ``$DEST/`` is usually ``/opt/stack/``.
@ -42,7 +42,8 @@ You can list the registered services with this command:
.. sourcecode:: console
$ sudo apt-get update
$ sudo apt-get install git-core python-dev gcc python-setuptools python-virtualenv node-less libssl-dev libffi-dev libxslt-dev
$ sudo apt-get install git-core python-dev gcc python-setuptools \
python-virtualenv node-less libssl-dev libffi-dev libxslt-dev
..
On Ubuntu 12.10 and higher you have to install the following lib as well:
@ -72,7 +73,8 @@ You can list the registered services with this command:
.. sourcecode:: console
$ cp openstack_dashboard/local/local_settings.py.example openstack_dashboard/local/local_settings.py
$ cp openstack_dashboard/local/local_settings.py.example
openstack_dashboard/local/local_settings.py
..
4. Modify ``openstack_dashboard/local/local_settings.py``
@ -84,16 +86,18 @@ You can list the registered services with this command:
OPENSTACK_HOST = "ip of your controller"
..
If you are using Nova-Network with ``auto_assign_floating_ip=True`` add the following parameter:
If you are using Nova-Network with ``auto_assign_floating_ip=True`` add the
following parameter:
.. sourcecode:: python
SAHARA_AUTO_IP_ALLOCATION_ENABLED = True
..
5. If Sahara is not registered with the keystone service catalog, it may be added
with the following commands. To use Sahara from Horizon without keystone
registration, see `Using the Data Processing Dashboard without Keystone Registration`_.
5. If Sahara is not registered with the keystone service catalog, it may be
added with the following commands. To use Sahara from Horizon without keystone
registration, see `Using the Data Processing Dashboard without Keystone
Registration`_.
.. sourcecode:: console
@ -109,8 +113,9 @@ You can list the registered services with this command:
$ tools/with_venv.sh python manage.py runserver 0.0.0.0:8080
..
This will start Horizon in debug mode. That means the logs will be written to console
and if any exceptions happen, you will see the stack-trace rendered as a web-page.
This will start Horizon in debug mode. That means the logs will be written to
console and if any exceptions happen, you will see the stack-trace rendered
as a web-page.
Debug mode can be disabled by changing ``DEBUG=True`` to ``False`` in
``local_settings.py``. In that case Horizon should be started slightly
@ -126,10 +131,10 @@ You can list the registered services with this command:
7. Applying changes
If you have changed any ``*.py`` files in
``horizon/openstack_dashboard/contrib/sahara/content/data_processing`` directory,
Horizon will notice that and reload automatically. However changes made to
non-python files may not be noticed, so you have to restart Horizon again
manually, as described in step 6.
``horizon/openstack_dashboard/contrib/sahara/content/data_processing``
directory, Horizon will notice that and reload automatically. However
changes made to non-python files may not be noticed, so you have to restart
Horizon again manually, as described in step 6.
Using the Data Processing Dashboard without Keystone Registration
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

6
doc/source/restapi.rst
View File

@ -81,6 +81,8 @@ error description.
2 API
=====
- `Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing)`_
- `Sahara REST API Reference (OpenStack API Complete Reference -
DataProcessing)`_
.. _`Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing)`: http://api.openstack.org/api-ref-data-processing-v1.1.html
.. _`Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing)`: http://api.openstack.org/api-ref-data-processing-v1.1.html

25
doc/source/userdoc/advanced.configuration.guide.rst
View File

@ -446,18 +446,19 @@ NTP package is included in the image (the sahara disk image builder will
include NTP in all images it generates). The default NTP server will be
``pool.ntp.org``; this can be overridden using the ``default_ntp_server``
setting in the ``DEFAULT`` section of the sahara configuration file.
If you are creating cluster templates using the sahara UI and would like to specify
a different NTP server for a particular cluster template, use the
``URL of NTP server`` setting in the ``General Parameters``
section when you create the template. If you would like to disable NTP for a
particular cluster template, deselect the ``Enable NTP service`` checkbox in
the ``General Parameters`` section when you create the template.
If you are creating clusters using the sahara CLI, you can specify another NTP server
or disable NTP service using the examples below.
If you are creating cluster templates using the sahara UI and would like to
specify a different NTP server for a particular cluster template, use the ``URL
of NTP server`` setting in the ``General Parameters`` section when you create
the template. If you would like to disable NTP for a particular cluster
template, deselect the ``Enable NTP service`` checkbox in the ``General
Parameters`` section when you create the template.
If you want to disable configuring the NTP service, you should specify the following
configs for the cluster:
If you are creating clusters using the sahara CLI, you can specify another NTP
server or disable NTP service using the examples below.
If you want to disable configuring the NTP service, you should specify the
following configs for the cluster:
.. sourcecode:: json
@ -467,8 +468,8 @@ configs for the cluster:
}
}
If you want to disable configuring NTP service, you should specify following configs
for the cluster:
If you want to disable configuring NTP service, you should specify following
configs for the cluster:
.. sourcecode:: json

30
doc/source/userdoc/cdh_imagebuilder.rst
View File

@ -30,18 +30,18 @@ To create cloudera images follow these steps:
2. Use tox to build images.
You can run "tox -e venv -- sahara-image-create" command in sahara-image-elements
directory to build images. By default this script will attempt to create cloud
images for all versions of supported plugins and all operating systems
(subset of Ubuntu, Fedora, and CentOS depending on plugin). To only create
Cloudera images, you should use the "-p cloudera" parameter in the command
line. If you want to create the image only for a specific operating system,
you should use the "-i ubuntu|centos" parameter to assign the operating
system (the cloudera plugin only supports Ubuntu and Centos). If you want
to create the image only for a specific Cloudera version, you should use the
"-v 5.0|5.3|5.4" parameter to assign the version. Below is an example to create
Cloudera images for both Ubuntu and CentOS with Cloudera Express 5.4.0
version.
You can run "tox -e venv -- sahara-image-create" command in
sahara-image-elements directory to build images. By default this script will
attempt to create cloud images for all versions of supported plugins and all
operating systems (subset of Ubuntu, Fedora, and CentOS depending on
plugin). To only create Cloudera images, you should use the "-p cloudera"
parameter in the command line. If you want to create the image only for a
specific operating system, you should use the "-i ubuntu|centos" parameter
to assign the operating system (the cloudera plugin only supports Ubuntu and
Centos). If you want to create the image only for a specific Cloudera
version, you should use the "-v 5.0|5.3|5.4" parameter to assign the
version. Below is an example to create Cloudera images for both Ubuntu and
CentOS with Cloudera Express 5.4.0 version.
.. sourcecode:: console
@ -57,9 +57,9 @@ To create cloudera images follow these steps:
NOTE: If you don't want to use default values, you should explicitly set the
values of your required parameters.
The script will create required cloud images using image elements that install
all the necessary packages and configure them. You will find the created
images in the parent directory.
The script will create required cloud images using image elements that
install all the necessary packages and configure them. You will find the
created images in the parent directory.
.. note::

18
doc/source/userdoc/cdh_plugin.rst
View File

@ -13,9 +13,9 @@ to provision cluster or you could download prepared images from
http://sahara-files.mirantis.com/images/upstream/kilo/
They already have Cloudera Express installed (5.0.0, 5.3.0, or 5.4.0 version).
The cloudera plugin requires an image to be tagged in Sahara Image Registry with
two tags: 'cdh' and '<cloudera version>' (e.g. '5', '5.3.0' or '5.4.0', here '5'
stands for '5.0.0').
The cloudera plugin requires an image to be tagged in Sahara Image Registry
with two tags: 'cdh' and '<cloudera version>' (e.g. '5', '5.3.0' or '5.4.0',
here '5' stands for '5.0.0').
The default username specified for these images is different for each
distribution:
@ -32,8 +32,8 @@ Services Supported
------------------
Currently below services are supported in both versions of Cloudera plugin:
HDFS, Oozie, YARN, Spark, Zookeeper, Hive, Hue, HBase. 5.3.0 version
of Cloudera Plugin also supported following services: Impala, Flume, Solr, Sqoop,
HDFS, Oozie, YARN, Spark, Zookeeper, Hive, Hue, HBase. 5.3.0 version of
Cloudera Plugin also supported following services: Impala, Flume, Solr, Sqoop,
and Key-value Store Indexer. 5.4.0 version added KMS service support based on
5.3.0 version.
@ -47,8 +47,8 @@ and Key-value Store Indexer. 5.4.0 version added KMS service support based on
High Availablity Support
------------------------
Currently HDFS NameNode High Availability is supported in Cloudera 5.4.0 version.
You can refer to :doc:`features` for the detail info.
Currently HDFS NameNode High Availability is supported in Cloudera 5.4.0
version. You can refer to :doc:`features` for the detail info.
YARN ResourceManager High Availability is supported in Cloudera 5.4.0 version.
This feature adds redundancy in the form of an Active/Standby ResourceManager
@ -87,8 +87,8 @@ cloudera plugin versions:
and at least one hbase regionserver.
+ Cluster can't contain hbase regionserver without at least one hbase maser.
In case of 5.3.0 or 5.4.0 version of Cloudera Plugin there are few extra limitations
in the cluster topology:
In case of 5.3.0 or 5.4.0 version of Cloudera Plugin there are few extra
limitations in the cluster topology:
+ Cluster can't contain flume without at least one datanode.
+ Cluster can contain at most one sentry server service.

36
doc/source/userdoc/configs_recommendations.rst
View File

@ -1,31 +1,35 @@
Autoconfiguring templates
=========================
During the Liberty development cycle sahara implemented a tool that recommends and
applies configuration values for cluster templates and node group templates.
These recommendations are based on the number of specific instances and on
flavors of the cluster node groups. Currently the following plugins support
this feature:
During the Liberty development cycle sahara implemented a tool that recommends
and applies configuration values for cluster templates and node group
templates. These recommendations are based on the number of specific instances
and on flavors of the cluster node groups. Currently the following plugins
support this feature:
* Cloudera;
* Spark;
* the Vanilla Apache Hadoop plugin.
By default this feature is enabled for all cluster templates and node group templates.
If you want to disable this feature for a particular cluster or node group template
you should set the ``use_autoconfig`` field to ``false``.
By default this feature is enabled for all cluster templates and node group
templates. If you want to disable this feature for a particular cluster or
node group template you should set the ``use_autoconfig`` field to ``false``.
.. NOTE
Also, if you manually set configs from the list below, the recommended configs
will not be applied.
Also, if you manually set configs from the list below, the recommended
configs will not be applied.
The following describes the settings for which sahara can recommend autoconfiguration:
The following describes the settings for which sahara can recommend
autoconfiguration:
The Cloudera, Spark and Vanilla Apache Hadoop plugin support configuring
``dfs.replication`` (``dfs_replication`` for Cloudera plugin) which is calculated as
a minimun from the amount of ``datanode`` (``HDFS_DATANODE`` for Cloudera plugin) instances in
the cluster and the default value for ``dfs.replication``.
``dfs.replication`` (``dfs_replication`` for Cloudera plugin) which is
calculated as a minimun from the amount of ``datanode`` (``HDFS_DATANODE`` for
Cloudera plugin) instances in the cluster and the default value for
``dfs.replication``.
The Vanilla Apache Hadoop plugin and Cloudera plugin support autoconfiguration
of basic YARN and MapReduce configs. These autoconfigurations are based on the following
documentation: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html
of basic YARN and MapReduce configs. These autoconfigurations are based on the
following documentation:
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html

4
doc/source/userdoc/dashboard.guide.rst
View File

@ -12,8 +12,8 @@ The sample configuration file is available `here. <https://github.com/openstack/
1. Networking
-------------
Depending on the Networking backend (Nova Network or Neutron) used in the cloud,
Sahara panels will determine automatically which input fields should be
Depending on the Networking backend (Nova Network or Neutron) used in the
cloud, Sahara panels will determine automatically which input fields should be
displayed.
While using Nova Network backend the cloud may be configured to automatically

34
doc/source/userdoc/edp.rst
View File

@ -281,17 +281,21 @@ in the future), sahara allows the addition of an interface (or method
signature) to your job template. A sample interface for the Teragen Hadoop
example might be:
+---------------+--------------+------------------+-------------+----------+---------------------+
| Name | Mapping Type | Location | Value Type | Required | Default |
+===============+==============+==================+=============+==========+=====================+
| Example Class | args | 0 | string | false | teragen |
+---------------+--------------+------------------+-------------+----------+---------------------+
| Rows | args | 1 | number | true | ``unset`` |
+---------------+--------------+------------------+-------------+----------+---------------------+
| Output Path | args | 2 | data_source | false | hdfs://ip:port/path |
+---------------+--------------+------------------+-------------+----------+---------------------+
| Mapper Count | configs | mapred.map.tasks | number | false | ``unset`` |
+---------------+--------------+------------------+-------------+----------+---------------------+
+---------+---------+------------+-------------+----------+-------------------+
| Name | Mapping | Location | Value | Required | Default |
| | Type | | Type | | |
+=========+=========+============+=============+=======+======================+
| Example | args | 0 | string | false | teragen |
| Class | | | | | |
+---------+---------+------------+-------------+-------+----------------------+
| Rows | args | 1 | number | true | ``unset`` |
+---------+---------+------------+-------------+-------+----------------------+
| Output | args | 2 | data_source | false | hdfs://ip:port/path |
| Path | | | | | |
+---------+---------+------------+-------------+-------+----------------------+
| Mapper | configs | mapred. | number | false | ``unset`` |
| Count | | map.tasks | | | |
+---------+---------+------------+-------------+-------+----------------------+
A "Description" field may also be added to each interface argument.
@ -496,10 +500,10 @@ running jobs on a cluster generated by the Cloudera plugin with the
generated by sahara in HDFS be added to the **oozie.libpath**. This for use
when an Hbase application is driven from a Java job. Default is **False**.
The **edp-wordcount** example bundled with sahara shows how to use configuration
values, arguments, and swift data paths in a Java job type. Note that the
example does not use the **edp.java.adapt_for_oozie** option but includes the
code to load the configuration values explicitly.
The **edp-wordcount** example bundled with sahara shows how to use
configuration values, arguments, and swift data paths in a Java job type. Note
that the example does not use the **edp.java.adapt_for_oozie** option but
includes the code to load the configuration values explicitly.
Additional Details for Shell jobs
+++++++++++++++++++++++++++++++++

9
doc/source/userdoc/features.rst
View File

@ -190,10 +190,11 @@ Security group management
-------------------------
Security groups are sets of IP filter rules that are applied to an instance's
networking. They are project specified, and project members can edit the default
rules for their group and add new rules sets. All projects have a "default"
security group, which is applied to instances that have no other security
group defined. Unless changed, this security group denies all incoming traffic.
networking. They are project specified, and project members can edit the
default rules for their group and add new rules sets. All projects have a
"default" security group, which is applied to instances that have no other
security group defined. Unless changed, this security group denies all incoming
traffic.
Sahara allows you to control which security groups will be used for created
instances. This can be done by providing the ``security_groups`` parameter for

27
doc/source/userdoc/guest-requirements.rst
View File

@ -1,7 +1,10 @@
Requirements for Guests
=======================
Sahara manages guests of various platforms (for example Ubuntu, Fedora, RHEL, and CentOS) with various versions of the Hadoop ecosystem projects installed. There are common requirements for all guests, and additional requirements based on the plugin that is used for cluster deployment.
Sahara manages guests of various platforms (for example Ubuntu, Fedora, RHEL,
and CentOS) with various versions of the Hadoop ecosystem projects installed.
There are common requirements for all guests, and additional requirements based
on the plugin that is used for cluster deployment.
Common Requirements
-------------------
@ -15,14 +18,16 @@ Common Requirements
Vanilla Plugin Requirements
---------------------------
If the Vanilla Plugin is used for cluster deployment the guest is required to have
If the Vanilla Plugin is used for cluster deployment the guest is required to
have
* ssh-client installed
* Java (version >= 6)
* Apache Hadoop installed
* 'hadoop' user created
See :doc:`hadoop-swift` for information on using Swift with your sahara cluster (for EDP support Swift integration is currently required).
See :doc:`hadoop-swift` for information on using Swift with your sahara cluster
(for EDP support Swift integration is currently required).
To support EDP, the following components must also be installed on the guest:
@ -30,17 +35,23 @@ To support EDP, the following components must also be installed on the guest:
* mysql
* hive
See :doc:`vanilla_imagebuilder` for instructions on building images for this plugin.
See :doc:`vanilla_imagebuilder` for instructions on building images for this
plugin.
Hortonworks Plugin Requirements
-------------------------------
This plugin does not have any additional requirements. Currently, only the CentOS Linux distribution is supported but other distributions will be supported in the future.
To speed up provisioning, the HDP packages can be pre-installed on the image used. The packages' versions depend on the HDP version being used.
This plugin does not have any additional requirements. Currently, only the
CentOS Linux distribution is supported but other distributions will be
supported in the future.
To speed up provisioning, the HDP packages can be pre-installed on the image
used. The packages' versions depend on the HDP version being used.
Cloudera Plugin Requirements
----------------------------
Cloudera Plugin does not have any additional requirements, just build a CDH image to deploy the cluster.
Cloudera Plugin does not have any additional requirements, just build a CDH
image to deploy the cluster.
See :doc:`cdh_imagebuilder` for instructions on building images for this plugin.
See :doc:`cdh_imagebuilder` for instructions on building images for this
plugin.

20
doc/source/userdoc/hadoop-swift.rst
View File

@ -47,13 +47,18 @@ instances using this template:
There are two types of configs here:
1. General. The ``${name}`` in this case equals to ``fs.swift``. Here is the list of ``${config}``:
1. General. The ``${name}`` in this case equals to ``fs.swift``. Here is the
list of ``${config}``:
* ``.impl`` - Swift FileSystem implementation. The ${value} is ``org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem``
* ``.impl`` - Swift FileSystem implementation. The ${value} is
``org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem``
* ``.connect.timeout`` - timeout for all connections by default: 15000
* ``.socket.timeout`` - how long the connection waits for responses from servers. by default: 60000
* ``.connect.retry.count`` - connection retry count for all connections. by default: 3
* ``.connect.throttle.delay`` - delay in millis between bulk (delete, rename, copy operations). by default: 0
* ``.socket.timeout`` - how long the connection waits for responses from
servers. by default: 60000
* ``.connect.retry.count`` - connection retry count for all connections. by
default: 3
* ``.connect.throttle.delay`` - delay in millis between bulk (delete,
rename, copy operations). by default: 0
* ``.blocksize`` - blocksize for filesystem. By default: 32Mb
* ``.partsize`` - the partition size for uploads. By default: 4608*1024Kb
* ``.requestsize`` - request size for reads in KB. By default: 64Kb
@ -76,8 +81,9 @@ There are two types of configs here:
* ``.http.port``
* ``.https.port``
* ``.region`` - Swift region is used when cloud has more than one Swift
installation. If region param is not set first region from Keystone endpoint
list will be chosen. If region param not found exception will be thrown.
installation. If region param is not set first region from Keystone
endpoint list will be chosen. If region param not found exception will be
thrown.
* ``.location-aware`` - turn On location awareness. Is false by default
* ``.apikey``
* ``.public``

24
doc/source/userdoc/hdp_plugin.rst
View File

@ -18,11 +18,11 @@ The HDP plugin can make use of Ambari Blueprints for cluster provisioning.
Apache Ambari Blueprints
------------------------
Apache Ambari Blueprints is a portable document definition, which provides
a complete definition for an Apache Hadoop cluster, including cluster topology,
Apache Ambari Blueprints is a portable document definition, which provides a
complete definition for an Apache Hadoop cluster, including cluster topology,
components, services and their configurations. Ambari Blueprints can be
consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack.
The benefits of this approach is that it allows for Hadoop clusters to be
consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack. The
benefits of this approach is that it allows for Hadoop clusters to be
configured and deployed using an Ambari native format that can be used with as
well as outside of OpenStack allowing for clusters to be re-instantiated in a
variety of environments.
@ -49,8 +49,8 @@ Images
------
The sahara HDP plugin can make use of either minimal (operating system only)
images or pre-populated HDP images. The base requirement for both is that the
image is cloud-init enabled and contains a supported operating system
(see http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html).
image is cloud-init enabled and contains a supported operating system (see
http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html).
The advantage of a pre-populated image is that provisioning time is reduced,
as packages do not need to be downloaded and installed which make up the
@ -72,10 +72,10 @@ The username specified should be 'cloud-user'.
HDFS NameNode High Availability
-------------------------------
HDFS NameNode High Availability (Using the Quorum Journal Manager) can be deployed
automatically with HDP 2.0.6. Currently the only way to deploy it is through the
command line client (python-saharaclient) or sahara REST API by simply adding the
following cluster_configs parameter in the cluster's JSON :
HDFS NameNode High Availability (Using the Quorum Journal Manager) can be
deployed automatically with HDP 2.0.6. Currently the only way to deploy it is
through the command line client (python-saharaclient) or sahara REST API by
simply adding the following cluster_configs parameter in the cluster's JSON :
.. sourcecode:: cfg
@ -89,8 +89,8 @@ The NameNode High Availability is deployed using 2 NameNodes, one active and
one standby. The NameNodes use a set of JOURNALNODES and ZOOKEEPER_SERVERS to
ensure the necessary synchronization.
A typical Highly available HDP 2.0.6 cluster uses 2 separate NameNodes, at least 3
JOURNALNODES and at least 3 ZOOKEEPER_SERVERS.
A typical Highly available HDP 2.0.6 cluster uses 2 separate NameNodes, at
least 3 JOURNALNODES and at least 3 ZOOKEEPER_SERVERS.
When HDFS NameNode High Availability is enabled, the plugin will perform the
following additional validations:

32
doc/source/userdoc/mapr_plugin.rst
View File

@ -5,22 +5,27 @@ OpenStack in an easy way and do it, quickly, conveniently and simply.
Operation
---------
The MapR Plugin performs the following four primary functions during cluster creation:
The MapR Plugin performs the following four primary functions during cluster
creation:
1. MapR components deployment - the plugin manages the deployment of the required software to the target VMs
2. Services Installation - MapR services are installed according to provided roles list
3. Services Configuration - the plugin combines default settings with user provided settings
4. Services Start - the plugin starts appropriate services according to specified roles
1. MapR components deployment - the plugin manages the deployment of the
required software to the target VMs
2. Services Installation - MapR services are installed according to provided
roles list
3. Services Configuration - the plugin combines default settings with user
provided settings
4. Services Start - the plugin starts appropriate services according to
specified roles
Images
------
The Sahara MapR plugin can make use of either minimal (operating system only)
images or pre-populated MapR images. The base requirement for both is that the
image is cloud-init enabled and contains a supported operating system
(see http://doc.mapr.com/display/MapR/OS+Support+Matrix).
image is cloud-init enabled and contains a supported operating system (see
http://doc.mapr.com/display/MapR/OS+Support+Matrix).
The advantage of a pre-populated image is that provisioning time is reduced,
as packages do not need to be downloaded which make up the majority of the time
The advantage of a pre-populated image is that provisioning time is reduced, as
packages do not need to be downloaded which make up the majority of the time
spent in the provisioning cycle. In addition, provisioning large clusters will
put a burden on the network as packages for all nodes need to be downloaded
from the package repository.
@ -45,7 +50,8 @@ two tags: 'mapr' and '<MapR version>' (e.g. '4.0.1.mrv2').
Note that Spark should be run on plain or 4.0.1 images.
The default username specified for these images is different for each distribution:
The default username specified for these images is different for each
distribution:
+--------------+------------+
| OS | username |
@ -58,7 +64,8 @@ The default username specified for these images is different for each distributi
Hadoop Version Support
----------------------
The MapR plugin currently supports Hadoop 0.20.2 (3.1.1, 4.0.1.mrv1, 4.0.2.mrv1),
The MapR plugin currently supports Hadoop 0.20.2 (3.1.1, 4.0.1.mrv1,
4.0.2.mrv1),
Hadoop 2.4.1 (4.0.2.mrv2) and Hadoop 2.5.1 (4.0.2.mrv2).
Cluster Validation
@ -110,7 +117,8 @@ Hue service is considered valid if:
HttpFS service is considered valid if cluster has exactly 1 *HttpFS* process
Sqoop service is considered valid if cluster has exactly 1 *Sqoop2-Server* process
Sqoop service is considered valid if cluster has exactly 1 *Sqoop2-Server*
process
The MapR Plugin
---------------

9
doc/source/userdoc/registering_image.rst
View File

@ -2,9 +2,10 @@ Registering an Image
====================
Sahara deploys a cluster of machines using images stored in Glance.
Each plugin has its own requirements on the image contents (see specific plugin
documentation for details). Two general requirements for an image are to have the
cloud-init and the ssh-server packages installed.
documentation for details). Two general requirements for an image are to have
the cloud-init and the ssh-server packages installed.
Sahara requires the images to be registered in the Sahara Image Registry.
A registered image must have two properties set:
@ -12,5 +13,5 @@ A registered image must have two properties set:
* username - a name of the default cloud-init user.
* tags - certain tags mark image to be suitable for certain plugins.
The username depends on the image that is used and the tags depend on the plugin used.
You can find both in the respective plugin's documentation.
The username depends on the image that is used and the tags depend on the
plugin used. You can find both in the respective plugin's documentation.

80
doc/source/userdoc/statuses.rst
View File

@ -3,8 +3,8 @@ Sahara Cluster Statuses Overview
All Sahara Cluster operations are performed in multiple steps. A Cluster object
has a ``Status`` attribute which changes when Sahara finishes one step of
operations and starts another one. Also a Cluster object has a ``Status description``
attribute which changes whenever Cluster errors occur.
operations and starts another one. Also a Cluster object has a ``Status
description`` attribute which changes whenever Cluster errors occur.
Sahara supports three types of Cluster operations:
* Create a new Cluster
@ -21,19 +21,22 @@ Before performing any operations with OpenStack environment, Sahara validates
user input.
There are two types of validations, that are done:
* Check that a request contains all necessary fields and that the request does not violate
any constraints like unique naming, etc.
* Plugin check (optional). The provisioning Plugin may also perform any specific checks like a Cluster topology validation check.
* Check that a request contains all necessary fields and that the request does
not violate any constraints like unique naming, etc.
* Plugin check (optional). The provisioning Plugin may also perform any
specific checks like a Cluster topology validation check.
If any of the validations fails during creating, the Cluster object will
still be kept in the database with an ``Error`` status. If any validations fails during scaling the ``Active`` Cluster, it will be
kept with an ``Active`` status.
In both cases status description will contain error messages about the reasons of failure.
If any of the validations fails during creating, the Cluster object will still
be kept in the database with an ``Error`` status. If any validations fails
during scaling the ``Active`` Cluster, it will be kept with an ``Active``
status. In both cases status description will contain error messages about the
reasons of failure.
2. InfraUpdating
~~~~~~~~~~~~~~~~
This status means that the Provisioning plugin is performing some infrastructure updates.
This status means that the Provisioning plugin is performing some
infrastructure updates.
3. Spawning
~~~~~~~~~~~
@ -49,22 +52,23 @@ so sahara will wait until all of the VMs are in an ``Active`` state.
4. Waiting
~~~~~~~~~~
Sahara waits while VMs' operating systems boot up and all internal infrastructure
components like networks and volumes are attached and ready to use.
Sahara waits while VMs' operating systems boot up and all internal
infrastructure components like networks and volumes are attached and ready to
use.
5. Preparing
~~~~~~~~~~~~
Sahara prepares a Cluster for starting. This step includes generating the ``/etc/hosts``
file, so that all instances can access each other by a hostname. Also Sahara
updates the ``authorized_keys`` file on each VM, so that VMs can communicate without
passwords.
Sahara prepares a Cluster for starting. This step includes generating the
``/etc/hosts`` file, so that all instances can access each other by a hostname.
Also Sahara updates the ``authorized_keys`` file on each VM, so that VMs can
communicate without passwords.
6. Configuring
~~~~~~~~~~~~~~
Sahara pushes service configurations to VMs. Both XML and JSON based configurations and
environmental variables are set on this step.
Sahara pushes service configurations to VMs. Both XML and JSON based
configurations and environmental variables are set on this step.
7. Starting
~~~~~~~~~~~
@ -74,7 +78,8 @@ Sahara is starting Hadoop services on Cluster's VMs.
8. Active
~~~~~~~~~
Active status means that a Cluster has started successfully and is ready to run EDP Jobs.
Active status means that a Cluster has started successfully and is ready to run
EDP Jobs.
Scaling/Shrinking an existing Cluster
@ -84,7 +89,8 @@ Scaling/Shrinking an existing Cluster
~~~~~~~~~~~~~
Sahara checks the scale/shrink request for validity. The Plugin method called
for performing Plugin specific checks is different from the validation method in creation.
for performing Plugin specific checks is different from the validation method
in creation.
2. Scaling
~~~~~~~~~~
@ -95,22 +101,23 @@ and creating new ones to join the existing Node Groups.
3. Adding Instances
~~~~~~~~~~~~~~~~~~~
Status is similar to ``Spawning`` in Custer creation. Sahara adds required amount
of VMs to the existing Node Groups and creates new Node Groups.
Status is similar to ``Spawning`` in Custer creation. Sahara adds required
amount of VMs to the existing Node Groups and creates new Node Groups.
4. Configuring
~~~~~~~~~~~~~~
Status is similar to ``Configuring`` in Cluster creation. New instances are being configured
in the same manner as already existing ones. The VMs in the existing Cluster are also updated
with a new ``/etc/hosts`` file.
Status is similar to ``Configuring`` in Cluster creation. New instances are
being configured in the same manner as already existing ones. The VMs in the
existing Cluster are also updated with a new ``/etc/hosts`` file.
5. Decommissioning
~~~~~~~~~~~~~~~~~~
Sahara stops Hadoop services on VMs that will be deleted from a Cluster.
Decommissioning a Data Node may take some time because Hadoop rearranges data replicas
around the Cluster, so that no data will be lost after that Data Node is deleted.
Decommissioning a Data Node may take some time because Hadoop rearranges data
replicas around the Cluster, so that no data will be lost after that Data Node
is deleted.
6. Deleting Instances
~~~~~~~~~~~~~~~~~~~~~
@ -132,19 +139,22 @@ Deleting an existing Cluster
1. Deleting
~~~~~~~~~~~
The only step, that releases all Cluster's resources and removes it from the database.
The only step, that releases all Cluster's resources and removes it from the
database.
Error State
-----------
If the Cluster creation fails, the Cluster will enter the ``Error`` state.
This status means the Cluster may not be able to perform any operations normally.
This cluster will stay in the database until it is manually deleted. The reason for
failure may be found in the sahara logs. Also, the status description will contain
information about the error.
This status means the Cluster may not be able to perform any operations
normally. This cluster will stay in the database until it is manually deleted.
The reason for failure may be found in the sahara logs. Also, the status
description will contain information about the error.
If an error occurs during the ``Adding Instances`` operation, Sahara will first
try to rollback this operation. If a rollback is impossible or fails itself, then
the Cluster will also go into an ``Error`` state. If a rollback was successful, Cluster will get into an ``Active`` state
and status description will contain a short message about the reason of ``Adding Instances`` failure.
try to rollback this operation. If a rollback is impossible or fails itself,
then the Cluster will also go into an ``Error`` state. If a rollback was
successful, Cluster will get into an ``Active`` state and status description
will contain a short message about the reason of ``Adding Instances`` failure.

48
doc/source/userdoc/upgrade.guide.rst
View File

@ -1,9 +1,8 @@
Sahara Upgrade Guide
====================
This page contains details about upgrading sahara between releases such
as configuration file updates, database migrations, and architectural
changes.
This page contains details about upgrading sahara between releases such as
configuration file updates, database migrations, and architectural changes.
Icehouse -> Juno
----------------
@ -18,11 +17,11 @@ All-In-One sahara is desired.
Authentication middleware changes
+++++++++++++++++++++++++++++++++
The custom auth_token middleware has been deprecated in favor of the
keystone middleware. This change requires an update to the sahara
configuration file. To update your configuration file you should replace
the following parameters from the ``[DEFAULT]`` section with the new
parameters in the ``[keystone_authtoken]`` section:
The custom auth_token middleware has been deprecated in favor of the keystone
middleware. This change requires an update to the sahara configuration file. To
update your configuration file you should replace the following parameters from
the ``[DEFAULT]`` section with the new parameters in the
``[keystone_authtoken]`` section:
+----------------------+--------------------+
| Old parameter name | New parameter name |
@ -49,12 +48,11 @@ The oslo based code from sahara.openstack.common.db has been replaced by
the usage of the oslo.db package. This change does not require any
update to sahara's configuration file.
Additionally, the usage of SQLite databases has been deprecated. Please
use MySQL or PostgreSQL databases for sahara. SQLite has been
deprecated because it does not, and is not going to, support the
``ALTER COLUMN`` and ``DROP COLUMN`` commands required for migrations
between versions. For more information please see
http://www.sqlite.org/omitted.html
Additionally, the usage of SQLite databases has been deprecated. Please use
MySQL or PostgreSQL databases for sahara. SQLite has been deprecated because it
does not, and is not going to, support the ``ALTER COLUMN`` and ``DROP COLUMN``
commands required for migrations between versions. For more information please
see http://www.sqlite.org/omitted.html
Sahara integration into OpenStack Dashboard
+++++++++++++++++++++++++++++++++++++++++++
@ -82,10 +80,9 @@ name for instances created by sahara using HEAT was always 'ec2-user'. As
of Juno, the user name is taken from the image registry as described in
the :doc:`registering_image` document.
This change breaks backward compatibility for clusters created using the
HEAT infrastructure engine prior to the Juno release. Clusters will
continue to operate, but we do not recommended using the scaling operations
with them.
This change breaks backward compatibility for clusters created using the HEAT
infrastructure engine prior to the Juno release. Clusters will continue to
operate, but we do not recommended using the scaling operations with them.
Anti affinity implementation changed
++++++++++++++++++++++++++++++++++++
@ -126,13 +123,14 @@ Kilo -> Liberty
Direct engine deprecation
+++++++++++++++++++++++++
In the Liberty release the direct infrastructure engine has been deprecated
and the heat infrastructure engine is now default. This means, that it is preferable
to use heat engine instead now. In the Liberty release you can continue to
operate clusters with the direct engine (create, delete, scale). Using heat engine only
the delete operation is available on clusters that were created by the direct engine.
After the Liberty release the direct engine will be removed, this means that you will
only be able to delete clusters created with the direct engine.
In the Liberty release the direct infrastructure engine has been deprecated and
the heat infrastructure engine is now default. This means, that it is
preferable to use heat engine instead now. In the Liberty release you can
continue to operate clusters with the direct engine (create, delete, scale).
Using heat engine only the delete operation is available on clusters that were
created by the direct engine. After the Liberty release the direct engine will
be removed, this means that you will only be able to delete clusters created
with the direct engine.
Policy namespace changed (policy.json)
++++++++++++++++++++++++++++++++++++++

12
doc/source/userdoc/vanilla_imagebuilder.rst
View File

@ -26,13 +26,14 @@ Elements for building vanilla images are stored in the
To create vanilla images follow these steps:
1. Clone repository "https://github.com/openstack/sahara-image-elements" locally.
1. Clone repository "https://github.com/openstack/sahara-image-elements"
locally.
2. Use tox to build images.
You can run the command below in sahara-image-elements
directory to build images. By default this script will attempt to create cloud
images for all versions of supported plugins and all operating systems
directory to build images. By default this script will attempt to create
cloud images for all versions of supported plugins and all operating systems
(subset of Ubuntu, Fedora, and CentOS depending on plugin).
.. sourcecode:: console
@ -40,7 +41,10 @@ To create vanilla images follow these steps:
tox -e venv -- sahara-image-create -u
Tox will create a virtualenv and install required python packages in it,
clone the repositories "https://github.com/openstack/diskimage-builder" and "https://github.com/openstack/sahara-image-elements" and export necessary parameters.
clone the repositories "https://github.com/openstack/diskimage-builder" and
"https://github.com/openstack/sahara-image-elements" and export necessary
parameters.
* ``DIB_HADOOP_VERSION`` - version of Hadoop to install
* ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin)
* ``OOZIE_DOWNLOAD_URL`` - download link for OOZIE (we have built

7
tox.ini
View File

@ -41,18 +41,13 @@ downloadcache = ~/cache/pip
[testenv:pep8]
commands =
flake8 {posargs}
doc8 doc/source
# Check that .po and .pot files are valid:
bash -c "find sahara -type f -regex '.*\.pot?' -print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null"
# Run bashate checks
bash -c "find sahara -iname '*.sh' -print0 | xargs -0 bashate -v"
bash -c "find devstack -not -name README.rst -and -not -name \*.json -type f -print0 | xargs -0 bashate -v"
[testenv:doc8]
deps =
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = doc8 doc/source
[testenv:venv]
commands = {posargs}