From 7ba05c5e302952252ebd3101d5a1875c484cdd9a Mon Sep 17 00:00:00 2001 From: Alok Jani Date: Fri, 13 Nov 2015 18:36:08 +0530 Subject: [PATCH] Fix doc8 check failures Resolves D000 and D001 checks in sahara docs. Change-Id: If6d577859170ae55d578d70885a9ed5a6b0db651 Closes-Bug: #1514887 --- .../devref/adding_database_migrations.rst | 46 ++- doc/source/devref/development.environment.rst | 3 +- doc/source/devref/development.guidelines.rst | 28 +- doc/source/devref/devstack.rst | 12 +- doc/source/devref/edp.spi.rst | 39 +-- doc/source/devref/how_to_build_oozie.rst | 14 +- doc/source/devref/how_to_participate.rst | 3 +- doc/source/devref/jenkins.rst | 44 ++- doc/source/devref/launchpad.rst | 10 +- doc/source/devref/log.guidelines.rst | 8 +- doc/source/devref/plugin.spi.rst | 322 ++++++++++-------- doc/source/horizon/dashboard.user.guide.rst | 39 +-- doc/source/horizon/dev.environment.guide.rst | 37 +- doc/source/restapi.rst | 6 +- .../userdoc/advanced.configuration.guide.rst | 25 +- doc/source/userdoc/cdh_imagebuilder.rst | 30 +- doc/source/userdoc/cdh_plugin.rst | 18 +- .../userdoc/configs_recommendations.rst | 36 +- doc/source/userdoc/dashboard.guide.rst | 4 +- doc/source/userdoc/edp.rst | 34 +- doc/source/userdoc/features.rst | 9 +- doc/source/userdoc/guest-requirements.rst | 27 +- doc/source/userdoc/hadoop-swift.rst | 20 +- doc/source/userdoc/hdp_plugin.rst | 24 +- doc/source/userdoc/mapr_plugin.rst | 32 +- doc/source/userdoc/registering_image.rst | 9 +- doc/source/userdoc/statuses.rst | 80 +++-- doc/source/userdoc/upgrade.guide.rst | 48 ++- doc/source/userdoc/vanilla_imagebuilder.rst | 12 +- tox.ini | 7 +- 30 files changed, 577 insertions(+), 449 deletions(-) diff --git a/doc/source/devref/adding_database_migrations.rst b/doc/source/devref/adding_database_migrations.rst index 0a451b2807..594357b442 100644 --- a/doc/source/devref/adding_database_migrations.rst +++ b/doc/source/devref/adding_database_migrations.rst @@ -1,16 +1,18 @@ Adding Database Migrations ========================== -The migrations in ``sahara/db/migration/alembic_migrations/versions`` contain the changes needed to migrate -between Sahara database revisions. A migration occurs by executing a script that details the changes needed to -upgrade or downgrade the database. The migration scripts are ordered so that multiple scripts -can run sequentially. The scripts are executed by Sahara's migration wrapper -which uses the Alembic library to manage the migration. Sahara supports +The migrations in ``sahara/db/migration/alembic_migrations/versions`` contain +the changes needed to migrate between Sahara database revisions. A migration +occurs by executing a script that details the changes needed to upgrade or +downgrade the database. The migration scripts are ordered so that multiple +scripts can run sequentially. The scripts are executed by Sahara's migration +wrapper which uses the Alembic library to manage the migration. Sahara supports migration from Icehouse or later. -Any code modifications that change the structure of the database require a migration script so that -previously existing databases will continue to function when the new code is released. This page gives a -brief overview of how to add the migration. +Any code modifications that change the structure of the database require a +migration script so that previously existing databases will continue to +function when the new code is released. This page gives a brief overview of how +to add the migration. Generate a New Migration Script +++++++++++++++++++++++++++++++ @@ -21,22 +23,26 @@ To generate a migration stub to be filled in by the developer:: $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" -To autogenerate a migration script that reflects the current structure of the database:: +To autogenerate a migration script that reflects the current structure of the +database:: $ sahara-db-manage --config-file /path/to/sahara.conf revision -m "description of revision" --autogenerate -Each of these commands will create a file of the form ``revision_description`` where ``revision`` is a string -generated by Alembic and ``description`` is based on the text passed with the ``-m`` option. +Each of these commands will create a file of the form ``revision_description`` +where ``revision`` is a string generated by Alembic and ``description`` is +based on the text passed with the ``-m`` option. Follow the Sahara Naming Convention +++++++++++++++++++++++++++++++++++ -By convention Sahara uses 3-digit revision numbers, and this scheme differs from the strings generated by Alembic. -Consequently, it's necessary to rename the generated script and modify the revision identifiers in the script. +By convention Sahara uses 3-digit revision numbers, and this scheme differs +from the strings generated by Alembic. Consequently, it's necessary to rename +the generated script and modify the revision identifiers in the script. -Open the new script and look for the variable ``down_revision``. The value should be a 3-digit numeric string, and it -identifies the current revision number of the database. Set the ``revision`` value to the ``down_revision`` value + 1. -For example, the lines:: +Open the new script and look for the variable ``down_revision``. The value +should be a 3-digit numeric string, and it identifies the current revision +number of the database. Set the ``revision`` value to the ``down_revision`` +value + 1. For example, the lines:: # revision identifiers, used by Alembic. revision = '507eb70202af' @@ -48,7 +54,8 @@ will become:: revision = '007' down_revision = '006' -Modify any comments in the file to match the changes and rename the file to match the new revision number:: +Modify any comments in the file to match the changes and rename the file to +match the new revision number:: $ mv 507eb70202af_my_new_revision.py 007_my_new_revision.py @@ -56,7 +63,10 @@ $ mv 507eb70202af_my_new_revision.py 007_my_new_revision.py Add Alembic Operations to the Script ++++++++++++++++++++++++++++++++++++ -The migration script contains method ``upgrade()``. Since Kilo release Sahara doesn't support downgrades. Fill in this method with the appropriate Alembic operations to perform upgrades. In the above example, an upgrade will move from revision '006' to revision '007'. +The migration script contains method ``upgrade()``. Since Kilo release Sahara +doesn't support downgrades. Fill in this method with the appropriate Alembic +operations to perform upgrades. In the above example, an upgrade will move from +revision '006' to revision '007'. Command Summary for sahara-db-manage ++++++++++++++++++++++++++++++++++++ diff --git a/doc/source/devref/development.environment.rst b/doc/source/devref/development.environment.rst index 3f24d44b16..2ac23b9461 100644 --- a/doc/source/devref/development.environment.rst +++ b/doc/source/devref/development.environment.rst @@ -119,7 +119,8 @@ Just add the following lines to .git/hooks/pre-commit and do chmod +x for it. # Run fast checks (PEP8 style check and PyFlakes fast static analysis) tools/run_fast_checks -You can added the same check for pre-push, for example, run_tests and run_pylint. +You can added the same check for pre-push, for example, run_tests and +run_pylint. 3. Running static analysis (PyLint) diff --git a/doc/source/devref/development.guidelines.rst b/doc/source/devref/development.guidelines.rst index 64233167f2..724552cc5f 100644 --- a/doc/source/devref/development.guidelines.rst +++ b/doc/source/devref/development.guidelines.rst @@ -20,9 +20,10 @@ To check your code against PEP 8 and bashate run: Static analysis --------------- -The static analysis checks are optional in Sahara. but they are still very useful. -The gate job will inform you if the number of static analysis warnings -has increased after your change. We recommend to always check the static warnings. +The static analysis checks are optional in Sahara. but they are still very +useful. The gate job will inform you if the number of static analysis warnings +has increased after your change. We recommend to always check the static +warnings. To run the check commit yor change first and execute the following command: @@ -33,15 +34,16 @@ To run the check commit yor change first and execute the following command: Modification of Upstream Files ------------------------------ -We never modify upstream files in Sahara. Any changes in upstream files should be made -in the upstream project and then merged back in to Sahara. This includes whitespace -changes, comments, and typos. Any change requests containing upstream file modifications -are almost certain to receive lots of negative reviews. Be warned. +We never modify upstream files in Sahara. Any changes in upstream files should +be made in the upstream project and then merged back in to Sahara. This +includes whitespace changes, comments, and typos. Any change requests +containing upstream file modifications are almost certain to receive lots of +negative reviews. Be warned. -Examples of upstream files are default xml configuration files used to configure Hadoop, or -code imported from the OpenStack Oslo project. The xml files will usually be found in -``resource`` directories with an accompanying ``README`` file that identifies where the -files came from. For example: +Examples of upstream files are default xml configuration files used to +configure Hadoop, or code imported from the OpenStack Oslo project. The xml +files will usually be found in ``resource`` directories with an accompanying +``README`` file that identifies where the files came from. For example: .. sourcecode:: console @@ -101,8 +103,8 @@ Run the following command to build docs locally. $ tox -e docs -After it you can access generated docs in ``doc/build/`` directory, for example, -main page - ``doc/build/html/index.html``. +After it you can access generated docs in ``doc/build/`` directory, for +example, main page - ``doc/build/html/index.html``. To make docs generation process faster you can use: diff --git a/doc/source/devref/devstack.rst b/doc/source/devref/devstack.rst index 6aa42b95d7..80dba19dc9 100644 --- a/doc/source/devref/devstack.rst +++ b/doc/source/devref/devstack.rst @@ -119,7 +119,8 @@ appended after the git repo URL as follows: 5. Once previous step is finished Devstack will print Horizon URL. Navigate to this URL and login with login "admin" and password from ``local.conf``. -6. Congratulations! You have OpenStack running in your VM and ready to launch VMs inside that VM :) +6. Congratulations! You have OpenStack running in your VM and ready to launch + VMs inside that VM :) Managing sahara in DevStack @@ -137,6 +138,7 @@ Setting fixed IP address for VMware Fusion VM --------------------------------------------- 1. Open file ``/Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf`` + 2. There is a block named "subnet". It might look like this: .. sourcecode:: text @@ -144,9 +146,11 @@ Setting fixed IP address for VMware Fusion VM subnet 192.168.55.0 netmask 255.255.255.0 { range 192.168.55.128 192.168.55.254; -3. You need to pick an IP address outside of that range. For example - ``192.168.55.20`` -4. Copy VM MAC address from VM settings->Network->Advanced -5. Append the following block to file ``dhcpd.conf`` (don't forget to replace ``VM_HOSTNAME`` and ``VM_MAC_ADDRESS`` with actual values): +3. You need to pick an IP address outside of that range. For example - + ``192.168.55.20`` 4. Copy VM MAC address from VM settings->Network->Advanced + +5. Append the following block to file ``dhcpd.conf`` (don't forget to replace + ``VM_HOSTNAME`` and ``VM_MAC_ADDRESS`` with actual values): .. sourcecode:: text diff --git a/doc/source/devref/edp.spi.rst b/doc/source/devref/edp.spi.rst index e2c6072821..04bdac3a97 100644 --- a/doc/source/devref/edp.spi.rst +++ b/doc/source/devref/edp.spi.rst @@ -1,13 +1,13 @@ Elastic Data Processing (EDP) SPI ================================= -EDP job engine objects provide methods for creating, monitoring, and terminating -jobs on Sahara clusters. Provisioning plugins that support EDP must return an -EDP job engine object from the :ref:`get_edp_engine` method described in -:doc:`plugin.spi`. +EDP job engine objects provide methods for creating, monitoring, and +terminating jobs on Sahara clusters. Provisioning plugins that support EDP must +return an EDP job engine object from the :ref:`get_edp_engine` method described +in :doc:`plugin.spi`. -Sahara provides subclasses of the base job engine interface that support EDP on -clusters running Oozie or on Spark standalone clusters. These are described +Sahara provides subclasses of the base job engine interface that support EDP +on clusters running Oozie or on Spark standalone clusters. These are described below. .. _edp_spi_job_types: @@ -60,8 +60,8 @@ cancel_job(job_execution) Stops the running job whose id is stored in the job_execution object. -*Returns*: None if the operation was unsuccessful or an updated job status value - +*Returns*: None if the operation was unsuccessful or an updated job status +value get_job_status(job_execution) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -133,10 +133,10 @@ JobEngine. It provides implementations for all of the methods in the base interface but adds a few more abstract methods. Note, the *validate_job_execution(cluster, job, data)* method does basic checks -on the job configuration but probably should be overloaded to include additional -checks on the cluster configuration. For example, the job engines for plugins -that support Oozie add checks to make sure that the Oozie service is up and -running. +on the job configuration but probably should be overloaded to include +additional checks on the cluster configuration. For example, the job engines +for plugins that support Oozie add checks to make sure that the Oozie service +is up and running. get_hdfs_user() @@ -160,8 +160,8 @@ creates the HDFS directory *dir_name* under the user specified by if the HDFS user is 'hadoop' and the dir_name is 'test' this method would create '/user/hadoop/test'. -The reason that this method is broken out in the interface as an abstract method -is that different versions of Hadoop treat path creation differently. +The reason that this method is broken out in the interface as an abstract +method is that different versions of Hadoop treat path creation differently. *Returns*: None @@ -207,14 +207,15 @@ The sahara.service.edp.spark.engine.SparkJobEngine class provides a full EDP implementation for Spark standalone clusters. Note, the *validate_job_execution(cluster, job, data)* method does basic checks -on the job configuration but probably should be overloaded to include additional -checks on the cluster configuration. For example, the job engine returned by the -Spark plugin checks that the Spark version is >= 1.0.0 to ensure that -*spark-submit* is available. +on the job configuration but probably should be overloaded to include +additional checks on the cluster configuration. For example, the job engine +returned by the Spark plugin checks that the Spark version is >= 1.0.0 to +ensure that *spark-submit* is available. get_driver_classpath(self) ~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns driver class path. -*Returns*: a string of the following format ' --driver-class-path *class_path_value*' +*Returns*: a string of the following format ' --driver-class-path +*class_path_value*' diff --git a/doc/source/devref/how_to_build_oozie.rst b/doc/source/devref/how_to_build_oozie.rst index f1918ddf0e..1f720559a3 100644 --- a/doc/source/devref/how_to_build_oozie.rst +++ b/doc/source/devref/how_to_build_oozie.rst @@ -40,10 +40,14 @@ JDK Versions By default, the build configuration enforces that JDK 1.6.* is being used. -There are 2 build properties that can be used to change the JDK version requirements: +There are 2 build properties that can be used to change the JDK version +requirements: - * ``javaVersion`` specifies the version of the JDK used to compile (default 1.6) - * ``targetJavaVersion`` specifies the version of the generated bytecode (default 1.6) + * ``javaVersion`` specifies the version of the JDK used to compile (default + 1.6) + + * ``targetJavaVersion`` specifies the version of the generated bytecode + (default 1.6) For example, to specify 1.7 JDK version, build command should contain ``-D javaVersion=1.7 -D tagetJavaVersion=1.7`` flags. @@ -63,4 +67,6 @@ used: Also, pig version can be passed as maven property with ``-D pig.version=x.x.x`` flag. -Similar instruction to build oozie.tar.gz you may find there: http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Building_Oozie \ No newline at end of file +Similar instruction to build oozie.tar.gz you may find there: +http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Building_Oozie + diff --git a/doc/source/devref/how_to_participate.rst b/doc/source/devref/how_to_participate.rst index 786465e0ae..5d9cd6a3fc 100644 --- a/doc/source/devref/how_to_participate.rst +++ b/doc/source/devref/how_to_participate.rst @@ -43,7 +43,8 @@ How to stay in touch with the community? * Weekly on Thursdays at 1400 UTC and 1800 UTC (on alternate weeks) -* IRC channel: ``#openstack-meeting-alt`` (1800UTC) and ``#openstack-meeting-3`` (1400UTC) +* IRC channel: ``#openstack-meeting-alt`` (1800UTC) and + ``#openstack-meeting-3`` (1400UTC) How to send your first patch on review? diff --git a/doc/source/devref/jenkins.rst b/doc/source/devref/jenkins.rst index 596e9c993e..46edc2daee 100644 --- a/doc/source/devref/jenkins.rst +++ b/doc/source/devref/jenkins.rst @@ -1,23 +1,41 @@ Continuous Integration with Jenkins =================================== -Each change made to Sahara core code is tested with unit and integration tests and style checks flake8. +Each change made to Sahara core code is tested with unit and integration tests +and style checks flake8. + +Unit tests and style checks are performed on public `OpenStack Jenkins +`_ managed by `Zuul +`_. -Unit tests and style checks are performed on public `OpenStack Jenkins `_ managed by `Zuul `_. Unit tests are checked using python 2.7. -The result of those checks and Unit tests are +1 or -1 to *Verify* column in a code review from *Jenkins* user. +The result of those checks and Unit tests are +1 or -1 to *Verify* column in a +code review from *Jenkins* user. -Integration tests check CRUD operations for Image Registry, Templates and Clusters. -Also a test job is launched on a created Cluster to verify Hadoop work. +Integration tests check CRUD operations for Image Registry, Templates and +Clusters. Also a test job is launched on a created Cluster to verify Hadoop +work. -All integration tests are launched by `Jenkins `_ on the internal Mirantis OpenStack Lab. -Jenkins keeps a pool of VMs to run tests in parallel. Even with the pool of VMs integration testing may take a while. -Jenkins is controlled for the most part by Zuul which determines what jobs are run when. -Zuul status is available by address: `Zuul Status `_. -For more information see: `Sahara Hadoop Cluster CI `_. +All integration tests are launched by `Jenkins +`_ on the internal Mirantis OpenStack +Lab. -The integration tests result is +1 or -1 to *Verify* column in a code review from *Sahara Hadoop Cluster CI* user. +Jenkins keeps a pool of VMs to run tests in parallel. Even with the pool of VMs +integration testing may take a while. -You can put *sahara-ci-recheck* in comment, if you want to recheck sahara-ci jobs. -Also, you can put *recheck* in comment, if you want to recheck both jenkins and sahara-ci jobs. +Jenkins is controlled for the most part by Zuul which determines what jobs are +run when. + +Zuul status is available by address: `Zuul Status +`_. + +For more information see: `Sahara Hadoop Cluster CI +`_. + +The integration tests result is +1 or -1 to *Verify* column in a code review +from *Sahara Hadoop Cluster CI* user. + +You can put *sahara-ci-recheck* in comment, if you want to recheck sahara-ci +jobs. Also, you can put *recheck* in comment, if you want to recheck both +jenkins and sahara-ci jobs. diff --git a/doc/source/devref/launchpad.rst b/doc/source/devref/launchpad.rst index da222d4377..bc6e9ffaa2 100644 --- a/doc/source/devref/launchpad.rst +++ b/doc/source/devref/launchpad.rst @@ -1,8 +1,8 @@ Project hosting =============== -`Launchpad`_ hosts the Sahara project. The Sahara project homepage on Launchpad is -http://launchpad.net/sahara. +`Launchpad`_ hosts the Sahara project. The Sahara project homepage on Launchpad +is http://launchpad.net/sahara. Launchpad credentials --------------------- @@ -18,9 +18,9 @@ OpenStack-related sites. These sites include: Mailing list ------------ -The mailing list email is ``openstack-dev@lists.openstack.org`` with subject prefix -``[sahara]``. To participate in the mailing list subscribe to the list at -http://lists.openstack.org/cgi-bin/mailman/listinfo +The mailing list email is ``openstack-dev@lists.openstack.org`` with subject +prefix ``[sahara]``. To participate in the mailing list subscribe to the list +at http://lists.openstack.org/cgi-bin/mailman/listinfo Bug tracking ------------ diff --git a/doc/source/devref/log.guidelines.rst b/doc/source/devref/log.guidelines.rst index 7b312dd2b3..1bd934c481 100644 --- a/doc/source/devref/log.guidelines.rst +++ b/doc/source/devref/log.guidelines.rst @@ -5,8 +5,8 @@ Log Guidelines Levels Guidelines ----------------- -During the Kilo release cycle the sahara community defined the following log levels: - +During the Kilo release cycle the sahara community defined the following +log levels: * Debug: Shows everything and is likely not suitable for normal production operation due to the sheer size of logs generated (e.g. scripts executions, @@ -41,8 +41,8 @@ Now sahara uses string formatting defined in `PEP 3101`_ for logs. Translation Guidelines ---------------------- -All log levels except Debug requires translation. None of the separate cli tools packaged -with sahara contain log translations. +All log levels except Debug requires translation. None of the separate +cli tools packaged with sahara contain log translations. * Debug: no translation * Info: _LI diff --git a/doc/source/devref/plugin.spi.rst b/doc/source/devref/plugin.spi.rst index 8f16f07934..912ef2eee1 100644 --- a/doc/source/devref/plugin.spi.rst +++ b/doc/source/devref/plugin.spi.rst @@ -7,9 +7,10 @@ Plugin interface get_versions() ~~~~~~~~~~~~~~ -Returns all versions of Hadoop that could be used with the plugin. -It is responsibility of the plugin to make sure that all required images for each hadoop version are available, -as well as configs and whatever else that plugin needs to create the Hadoop cluster. +Returns all versions of Hadoop that could be used with the plugin. It is +responsibility of the plugin to make sure that all required images for each +hadoop version are available, as well as configs and whatever else that plugin +needs to create the Hadoop cluster. *Returns*: list of strings - Hadoop versions @@ -18,27 +19,31 @@ as well as configs and whatever else that plugin needs to create the Hadoop clus get_configs( hadoop_version) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Lists all configs supported by plugin with descriptions, defaults and targets for which this config is applicable. +Lists all configs supported by plugin with descriptions, defaults and targets +for which this config is applicable. *Returns*: list of configs -*Example return value*: ((“JobTracker heap size”, "JobTracker heap size, in MB", "int", “512”, `“mapreduce”`, "node", True, 1)) +*Example return value*: ((“JobTracker heap size”, "JobTracker heap size, in +MB", "int", “512”, `“mapreduce”`, "node", True, 1)) get_node_processes( hadoop_version) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Returns all supported services and node processes for a given Hadoop version. -Each node process belongs to a single service and that relationship is reflected in the returned dict object. -See example for details. +Each node process belongs to a single service and that relationship is +reflected in the returned dict object. See example for details. *Returns*: dictionary having entries (service -> list of processes) -*Example return value*: {"mapreduce": ["tasktracker", "jobtracker"], "hdfs": ["datanode", "namenode"]} +*Example return value*: {"mapreduce": ["tasktracker", "jobtracker"], "hdfs": +["datanode", "namenode"]} get_required_image_tags( hadoop_version) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Lists tags, that should be added to OpenStack Image via Image Registry. Tags are used to filter Images by plugin and hadoop version. +Lists tags, that should be added to OpenStack Image via Image Registry. Tags +are used to filter Images by plugin and hadoop version. *Returns*: list of tags @@ -47,11 +52,14 @@ Lists tags, that should be added to OpenStack Image via Image Registry. Tags are validate(cluster) ~~~~~~~~~~~~~~~~~ -Validates a given cluster object. Raises *SaharaException* with meaningful message. +Validates a given cluster object. Raises *SaharaException* with meaningful +message. *Returns*: None -*Example exception*: +*Example exception*: validate_scaling(cluster, existing, additional) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -75,24 +83,26 @@ For instance, plugin can ask for additional VMs for the management tool. configure_cluster(cluster) ~~~~~~~~~~~~~~~~~~~~~~~~~~ -Configures cluster on provisioned by sahara VMs. -In this function plugin should perform all actions like adjusting OS, installing required packages (including Hadoop, if needed), configuring Hadoop, etc. +Configures cluster on provisioned by sahara VMs. In this function plugin +should perform all actions like adjusting OS, installing required packages +(including Hadoop, if needed), configuring Hadoop, etc. *Returns*: None start_cluster(cluster) ~~~~~~~~~~~~~~~~~~~~~~ -Start already configured cluster. This method is guaranteed to be called only on cluster which was already prepared with configure_cluster(...) call. +Start already configured cluster. This method is guaranteed to be called only +on cluster which was already prepared with configure_cluster(...) call. *Returns*: None scale_cluster(cluster, instances) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Scale an existing cluster with additional instances. Instances argument is a list -of ready-to-configure instances. Plugin should do all configuration operations in this -method and start all services on those instances. +Scale an existing cluster with additional instances. Instances argument is a +list of ready-to-configure instances. Plugin should do all configuration +operations in this method and start all services on those instances. *Returns*: None @@ -103,17 +113,19 @@ get_edp_engine(cluster, job_type) Returns an EDP job engine object that supports the specified job_type on the given cluster, or None if there is no support. The EDP job engine object -returned must implement the interface described in :doc:`edp.spi`. The job_type -is a String matching one of the job types listed in :ref:`edp_spi_job_types`. +returned must implement the interface described in :doc:`edp.spi`. The +job_type is a String matching one of the job types listed in +:ref:`edp_spi_job_types`. *Returns*: an EDP job engine object or None decommission_nodes(cluster, instances) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Scale cluster down by removing a list of instances. Plugin should stop services on a provided list -of instances. Plugin also may want to update some configurations on other instances, so this -method is the right place to do that. +Scale cluster down by removing a list of instances. Plugin should stop services +on a provided list of instances. Plugin also may want to update some +configurations on other instances, so this method is the right place to do +that. *Returns*: None @@ -129,7 +141,9 @@ See “Cluster Lifecycle for Config File Mode” section below for clarification on_terminate_cluster(cluster) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When user terminates cluster, sahara simply shuts down all the cluster VMs. This method is guaranteed to be invoked before that, allowing plugin to do some clean-up. +When user terminates cluster, sahara simply shuts down all the cluster VMs. +This method is guaranteed to be invoked before that, allowing plugin to do some +clean-up. *Returns*: None @@ -140,19 +154,22 @@ When user requests sahara to automatically create security group for the node group (``auto_security_group`` property set to True), sahara will call this plugin method to get list of ports that need to be opened. -*Returns*: list of ports to be open in auto security group for the given node group +*Returns*: list of ports to be open in auto security group for the given node +group def get_edp_job_types(versions) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Optional method, which provides ability to see all supported job types for specified plugin versions +Optional method, which provides ability to see all supported job types for +specified plugin versions *Returns*: dict with supported job types for specified versions of plugin def recommend_configs(self, cluster, scaling=False) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Optional method, which provides recommendations for cluster configuration before creating/scaling operation. +Optional method, which provides recommendations for cluster configuration +before creating/scaling operation. *Returns*: None @@ -163,8 +180,9 @@ Here is a description of all the objects involved in the API. Notes: -- cluster and node_group have ‘extra’ field allowing plugin to persist any complementary info about the cluster. -- node_process is just a process that runs at some node in cluster. + - cluster and node_group have ‘extra’ field allowing plugin to persist any + complementary info about the cluster. + - node_process is just a process that runs at some node in cluster. Example list of node processes: @@ -180,30 +198,36 @@ Config An object, describing one configuration entry -+-------------------+--------+--------------------------------------------------------------------------------+ -| Property | Type | Description | -+===================+========+================================================================================+ -| name | string | Config name. | -+-------------------+--------+--------------------------------------------------------------------------------+ -| description | string | A hint for user, what this config is used for. | -+-------------------+--------+--------------------------------------------------------------------------------+ -| config_type | enum | possible values are: 'string', 'integer', 'boolean', 'enum'. | -+-------------------+--------+--------------------------------------------------------------------------------+ -| config_values | list | List of possible values, if config_type is enum. | -+-------------------+--------+--------------------------------------------------------------------------------+ -| default_value | string | Default value for config. | -+-------------------+--------+--------------------------------------------------------------------------------+ -| applicable_target | string | The target could be either a service returned by get_node_processes(...) call | -| | | in form of 'service:', or 'general'. | -+-------------------+--------+--------------------------------------------------------------------------------+ -| scope | enum | Could be either 'node' or 'cluster'. | -+-------------------+--------+--------------------------------------------------------------------------------+ -| is_optional | bool | If is_optional is False and no default_value is specified, user should | -| | | provide a value. | -+-------------------+--------+--------------------------------------------------------------------------------+ -| priority | int | 1 or 2. A Hint for UI. Configs with priority *1* are always displayed. | -| | | Priority *2* means user should click a button to see the config. | -+-------------------+--------+--------------------------------------------------------------------------------+ ++-------------------+--------+------------------------------------------------+ +| Property | Type | Description | ++===================+========+================================================+ +| name | string | Config name. | ++-------------------+--------+------------------------------------------------+ +| description | string | A hint for user, what this config is used for. | ++-------------------+--------+------------------------------------------------+ +| config_type | enum | possible values are: 'string', 'integer', | +| | | 'boolean', 'enum'. | ++-------------------+--------+------------------------------------------------+ +| config_values | list | List of possible values, if config_type is | +| | | enum. | ++-------------------+--------+------------------------------------------------+ +| default_value | string | Default value for config. | ++-------------------+--------+------------------------------------------------+ +| applicable_target | string | The target could be either a service returned | +| | | by get_node_processes(...) call | +| | | in form of 'service:', or | +| | | 'general'. | ++-------------------+--------+------------------------------------------------+ +| scope | enum | Could be either 'node' or 'cluster'. | ++-------------------+--------+------------------------------------------------+ +| is_optional | bool | If is_optional is False and no default_value | +| | | is specified, user should provide a value | ++-------------------+--------+------------------------------------------------+ +| priority | int | 1 or 2. A Hint for UI. Configs with priority | +| | | *1* are always displayed. | +| | | Priority *2* means user should click a button | +| | | to see the config. | ++-------------------+--------+------------------------------------------------+ User Input @@ -225,31 +249,33 @@ Instance An instance created for cluster. -+---------------+---------+---------------------------------------------------------+ -| Property | Type | Description | -+===============+=========+=========================================================+ -| instance_id | string | Unique instance identifier. | -+---------------+---------+---------------------------------------------------------+ -| instance_name | string | OpenStack Instance name. | -+---------------+---------+---------------------------------------------------------+ -| internal_ip | string | IP to communicate with other instances. | -+---------------+---------+---------------------------------------------------------+ -| management_ip | string | IP of instance, accessible outside of internal network. | -+---------------+---------+---------------------------------------------------------+ -| volumes | list | List of volumes attached to instance. Empty if | -| | | ephemeral drive is used. | -+---------------+---------+---------------------------------------------------------+ -| nova_info | object | Nova Instance object. | -+---------------+---------+---------------------------------------------------------+ -| username | string | Username, that sahara uses for establishing remote | -| | | connections to instance. | -+---------------+---------+---------------------------------------------------------+ -| hostname | string | Same as instance_name. | -+---------------+---------+---------------------------------------------------------+ -| fqdn | string | Fully qualified domain name for this instance. | -+---------------+---------+---------------------------------------------------------+ -| remote | helpers | Object with helpers for performing remote operations | -+---------------+---------+---------------------------------------------------------+ ++---------------+---------+---------------------------------------------------+ +| Property | Type | Description | ++===============+=========+===================================================+ +| instance_id | string | Unique instance identifier. | ++---------------+---------+---------------------------------------------------+ +| instance_name | string | OpenStack Instance name. | ++---------------+---------+---------------------------------------------------+ +| internal_ip | string | IP to communicate with other instances. | ++---------------+---------+---------------------------------------------------+ +| management_ip | string | IP of instance, accessible outside of internal | +| | | network. | ++---------------+---------+---------------------------------------------------+ +| volumes | list | List of volumes attached to instance. Empty if | +| | | ephemeral drive is used. | ++---------------+---------+---------------------------------------------------+ +| nova_info | object | Nova Instance object. | ++---------------+---------+---------------------------------------------------+ +| username | string | Username, that sahara uses for establishing | +| | | remote connections to instance. | ++---------------+---------+---------------------------------------------------+ +| hostname | string | Same as instance_name. | ++---------------+---------+---------------------------------------------------+ +| fqdn | string | Fully qualified domain name for this instance. | ++---------------+---------+---------------------------------------------------+ +| remote | helpers | Object with helpers for performing remote | +| | | operations | ++---------------+---------+---------------------------------------------------+ Node Group @@ -257,78 +283,86 @@ Node Group Group of instances. -+----------------------+--------+--------------------------------------------------------+ -| Property | Type | Description | -+======================+========+========================================================+ -| name | string | Name of this Node Group in Cluster. | -+----------------------+--------+--------------------------------------------------------+ -| flavor_id | string | OpenStack Flavor used to boot instances. | -+----------------------+--------+--------------------------------------------------------+ -| image_id | string | Image id used to boot instances. | -+----------------------+--------+--------------------------------------------------------+ -| node_processes | list | List of processes running on each instance. | -+----------------------+--------+--------------------------------------------------------+ -| node_configs | dict | Configs dictionary, applied to instances. | -+----------------------+--------+--------------------------------------------------------+ -| volumes_per_node | int | Number of volumes mounted to each instance. 0 means | -| | | use ephemeral drive. | -+----------------------+--------+--------------------------------------------------------+ -| volumes_size | int | Size of each volume (GB). | -+----------------------+--------+--------------------------------------------------------+ -| volumes_mount_prefix | string | Prefix added to mount path of each volume. | -+----------------------+--------+--------------------------------------------------------+ -| floating_ip_pool | string | Floating IP Pool name. All instances in the Node Group | -| | | will have Floating IPs assigned from this pool. | -+----------------------+--------+--------------------------------------------------------+ -| count | int | Number of instances in this Node Group. | -+----------------------+--------+--------------------------------------------------------+ -| username | string | Username used by sahara to establish remote | -| | | connections to instances. | -+----------------------+--------+--------------------------------------------------------+ -| configuration | dict | Merged dictionary of node configurations and cluster | -| | | configurations. | -+----------------------+--------+--------------------------------------------------------+ -| storage_paths | list | List of directories where storage should be placed. | -+----------------------+--------+--------------------------------------------------------+ ++----------------------+--------+---------------------------------------------+ +| Property | Type | Description | ++======================+========+=============================================+ +| name | string | Name of this Node Group in Cluster. | ++----------------------+--------+---------------------------------------------+ +| flavor_id | string | OpenStack Flavor used to boot instances. | ++----------------------+--------+---------------------------------------------+ +| image_id | string | Image id used to boot instances. | ++----------------------+--------+---------------------------------------------+ +| node_processes | list | List of processes running on each instance. | ++----------------------+--------+---------------------------------------------+ +| node_configs | dict | Configs dictionary, applied to instances. | ++----------------------+--------+---------------------------------------------+ +| volumes_per_node | int | Number of volumes mounted to each instance. | +| | | 0 means use ephemeral drive. | ++----------------------+--------+---------------------------------------------+ +| volumes_size | int | Size of each volume (GB). | ++----------------------+--------+---------------------------------------------+ +| volumes_mount_prefix | string | Prefix added to mount path of each volume. | ++----------------------+--------+---------------------------------------------+ +| floating_ip_pool | string | Floating IP Pool name. All instances in the | +| | | Node Group will have Floating IPs assigned | +| | | from this pool. | ++----------------------+--------+---------------------------------------------+ +| count | int | Number of instances in this Node Group. | ++----------------------+--------+---------------------------------------------+ +| username | string | Username used by sahara to establish remote | +| | | connections to instances. | ++----------------------+--------+---------------------------------------------+ +| configuration | dict | Merged dictionary of node configurations | +| | | and cluster configurations. | ++----------------------+--------+---------------------------------------------+ +| storage_paths | list | List of directories where storage should be | +| | | placed. | ++----------------------+--------+---------------------------------------------+ Cluster ------- -Contains all relevant info about cluster. -This object is is provided to the plugin for both cluster creation and scaling. -The “Cluster Lifecycle” section below further specifies which fields are filled at which moment. +Contains all relevant info about cluster. This object is is provided to the +plugin for both cluster creation and scaling. The “Cluster Lifecycle” section +below further specifies which fields are filled at which moment. -+----------------------------+--------+-----------------------------------------------------------+ -| Property | Type | Description | -+============================+========+===========================================================+ -| name | string | Cluster name. | -+----------------------------+--------+-----------------------------------------------------------+ -| tenant_id | string | OpenStack Tenant id where this Cluster is available. | -+----------------------------+--------+-----------------------------------------------------------+ -| plugin_name | string | Plugin name. | -+----------------------------+--------+-----------------------------------------------------------+ -| hadoop_version | string | Hadoop version running on instances. | -+----------------------------+--------+-----------------------------------------------------------+ -| default_image_id | string | OpenStack image used to boot instances. | -+----------------------------+--------+-----------------------------------------------------------+ -| node_groups | list | List of Node Groups. | -+----------------------------+--------+-----------------------------------------------------------+ -| cluster_configs | dict | Dictionary of Cluster scoped configurations. | -+----------------------------+--------+-----------------------------------------------------------+ -| cluster_template_id | string | Cluster Template used for Node Groups and Configurations. | -+----------------------------+--------+-----------------------------------------------------------+ -| user_keypair_id | string | OpenStack keypair added to instances to make them | -| | | accessible for user. | -+----------------------------+--------+-----------------------------------------------------------+ -| neutron_management_network | string | Neutron network ID. Instances will get fixed IPs in this | -| | | network if 'use_neutron' config is set to True. | -+----------------------------+--------+-----------------------------------------------------------+ -| anti_affinity | list | List of processes that will be run on different hosts. | -+----------------------------+--------+-----------------------------------------------------------+ -| description | string | Cluster Description. | -+----------------------------+--------+-----------------------------------------------------------+ -| info | dict | Dictionary for additional information. | -+----------------------------+--------+-----------------------------------------------------------+ ++----------------------------+--------+---------------------------------------+ +| Property | Type | Description | ++============================+========+=======================================+ +| name | string | Cluster name. | ++----------------------------+--------+---------------------------------------+ +| tenant_id | string | OpenStack Tenant id where this | +| | | Cluster is available. | ++----------------------------+--------+---------------------------------------+ +| plugin_name | string | Plugin name. | ++----------------------------+--------+---------------------------------------+ +| hadoop_version | string | Hadoop version running on instances. | ++----------------------------+--------+---------------------------------------+ +| default_image_id | string | OpenStack image used to boot | +| | | instances. | ++----------------------------+--------+---------------------------------------+ +| node_groups | list | List of Node Groups. | ++----------------------------+--------+---------------------------------------+ +| cluster_configs | dict | Dictionary of Cluster scoped | +| | | configurations. | ++----------------------------+--------+---------------------------------------+ +| cluster_template_id | string | Cluster Template used for Node Groups | +| | | and Configurations. | ++----------------------------+--------+---------------------------------------+ +| user_keypair_id | string | OpenStack keypair added to instances | +| | | to make them accessible for user. | ++----------------------------+--------+---------------------------------------+ +| neutron_management_network | string | Neutron network ID. Instances will | +| | | get fixed IPs in this network if | +| | | 'use_neutron' config is set to True. | ++----------------------------+--------+---------------------------------------+ +| anti_affinity | list | List of processes that will be run on | +| | | different hosts. | ++----------------------------+--------+---------------------------------------+ +| description | string | Cluster Description. | ++----------------------------+--------+---------------------------------------+ +| info | dict | Dictionary for additional information.| ++----------------------------+--------+---------------------------------------+ Validation Error diff --git a/doc/source/horizon/dashboard.user.guide.rst b/doc/source/horizon/dashboard.user.guide.rst index 1e6d12cba4..68a3719c13 100644 --- a/doc/source/horizon/dashboard.user.guide.rst +++ b/doc/source/horizon/dashboard.user.guide.rst @@ -2,8 +2,8 @@ Sahara (Data Processing) UI User Guide ====================================== This guide assumes that you already have the Sahara service and Horizon -dashboard up and running. Don't forget to make sure that Sahara is registered in -Keystone. If you require assistance with that, please see the +dashboard up and running. Don't forget to make sure that Sahara is +registered in Keystone. If you require assistance with that, please see the `installation guide <../installation.guide.html>`_. The sections below give a panel by panel overview of setting up clusters @@ -25,8 +25,8 @@ Registering an Image 4) Enter the username of the cloud-init user on the image -5) Click on the tags that you want to add to the image. (A version ie: 1.2.1 and - a type ie: vanilla are required for cluster functionality) +5) Click on the tags that you want to add to the image. (A version ie: 1.2.1 + and a type ie: vanilla are required for cluster functionality) 6) Click the "Done" button to finish the registration @@ -46,8 +46,8 @@ Create Node Group Templates 5) Choose a flavor for this template (based on your CPU/memory/disk needs) 6) Choose the storage location for your instance, this can be either "Ephemeral - Drive" or "Cinder Volume". If you choose "Cinder Volume", you will need to add - additional configuration + Drive" or "Cinder Volume". If you choose "Cinder Volume", you will need to + add additional configuration 7) Choose which processes should be run for any instances that are spawned from this Node Group Template @@ -131,8 +131,8 @@ Data Sources ------------ Data Sources are where the input and output from your jobs are housed. -1) From the Data Processing/Data Sources page, click on the "Create Data Source" - button at the top right +1) From the Data Processing/Data Sources page, click on the "Create Data + Source" button at the top right 2) Give your Data Source a name @@ -223,8 +223,8 @@ status of your job to see when it has completed its run - Additional configuration properties can be defined by clicking on the "Add" button - - An example configuration entry might be mapred.mapper.class for the Name and - org.apache.oozie.example.SampleMapper for the Value + - An example configuration entry might be mapred.mapper.class for the Name + and org.apache.oozie.example.SampleMapper for the Value 5) Click on "Launch". To monitor the status of your job, you can navigate to the Data Processing/Jobs panel @@ -241,8 +241,9 @@ status of your job to see when it has completed its run Example Jobs ------------ There are sample jobs located in the Sahara repository. In this section, we -will give a walkthrough on how to run those jobs via the Horizon UI. These steps -assume that you already have a cluster up and running (in the "Active" state). +will give a walkthrough on how to run those jobs via the Horizon UI. These +steps assume that you already have a cluster up and running (in the "Active" +state). 1) Sample Pig job - https://github.com/openstack/sahara/tree/master/etc/edp-examples/edp-pig/trim-spaces @@ -287,8 +288,8 @@ assume that you already have a cluster up and running (in the "Active" state). - Name = pigsample, Job Type = Pig, Choose "example.pig" as the main binary - - Click on the "Libs" tab and choose "udf.jar", then hit the "Choose" button - beneath the dropdown, then click on "Create" + - Click on the "Libs" tab and choose "udf.jar", then hit the "Choose" + button beneath the dropdown, then click on "Create" - Launch your job @@ -364,11 +365,11 @@ Additional Notes existing Job. In order to be able to delete that job, you would first need to delete any Job Templates that relate to that job. -2) In the examples above, we mention adding your username/password for the Swift - Data Sources. It should be noted that it is possible to configure Sahara such - that the username/password credentials are *not* required. For more - information on that, please refer to: - :doc:`Sahara Advanced Configuration Guide <../userdoc/advanced.configuration.guide>` +2) In the examples above, we mention adding your username/password for the + Swift Data Sources. It should be noted that it is possible to configure + Sahara such that the username/password credentials are *not* required. For + more information on that, please refer to: :doc:`Sahara Advanced + Configuration Guide <../userdoc/advanced.configuration.guide>` Launching a cluster via the Cluster Creation Guide -------------------------------------------------- diff --git a/doc/source/horizon/dev.environment.guide.rst b/doc/source/horizon/dev.environment.guide.rst index 1028a3909c..67c2803dc6 100644 --- a/doc/source/horizon/dev.environment.guide.rst +++ b/doc/source/horizon/dev.environment.guide.rst @@ -2,8 +2,8 @@ Sahara UI Dev Environment Setup =============================== This page describes how to setup Horizon for developing Sahara by either -installing it as part of DevStack with Sahara or installing it in an isolated environment -and running from the command line. +installing it as part of DevStack with Sahara or installing it in an +isolated environment and running from the command line. Install as a part of DevStack ----------------------------- @@ -11,8 +11,8 @@ Install as a part of DevStack See the `DevStack guide <../devref/devstack.html>`_ for more information on installing and configuring DevStack with Sahara. -After Horizon installation, it will contain a Data Processing tab under Projects tab. -Sahara UI source code will be located at +After Horizon installation, it will contain a Data Processing tab under +Projects tab. Sahara UI source code will be located at ``$DEST/horizon/openstack_dashboard/contrib/sahara/content/data_processing`` where ``$DEST/`` is usually ``/opt/stack/``. @@ -42,7 +42,8 @@ You can list the registered services with this command: .. sourcecode:: console $ sudo apt-get update - $ sudo apt-get install git-core python-dev gcc python-setuptools python-virtualenv node-less libssl-dev libffi-dev libxslt-dev + $ sudo apt-get install git-core python-dev gcc python-setuptools \ + python-virtualenv node-less libssl-dev libffi-dev libxslt-dev .. On Ubuntu 12.10 and higher you have to install the following lib as well: @@ -72,7 +73,8 @@ You can list the registered services with this command: .. sourcecode:: console - $ cp openstack_dashboard/local/local_settings.py.example openstack_dashboard/local/local_settings.py + $ cp openstack_dashboard/local/local_settings.py.example + openstack_dashboard/local/local_settings.py .. 4. Modify ``openstack_dashboard/local/local_settings.py`` @@ -84,16 +86,18 @@ You can list the registered services with this command: OPENSTACK_HOST = "ip of your controller" .. - If you are using Nova-Network with ``auto_assign_floating_ip=True`` add the following parameter: + If you are using Nova-Network with ``auto_assign_floating_ip=True`` add the + following parameter: .. sourcecode:: python SAHARA_AUTO_IP_ALLOCATION_ENABLED = True .. -5. If Sahara is not registered with the keystone service catalog, it may be added - with the following commands. To use Sahara from Horizon without keystone - registration, see `Using the Data Processing Dashboard without Keystone Registration`_. +5. If Sahara is not registered with the keystone service catalog, it may be + added with the following commands. To use Sahara from Horizon without keystone + registration, see `Using the Data Processing Dashboard without Keystone + Registration`_. .. sourcecode:: console @@ -109,8 +113,9 @@ You can list the registered services with this command: $ tools/with_venv.sh python manage.py runserver 0.0.0.0:8080 .. - This will start Horizon in debug mode. That means the logs will be written to console - and if any exceptions happen, you will see the stack-trace rendered as a web-page. + This will start Horizon in debug mode. That means the logs will be written to + console and if any exceptions happen, you will see the stack-trace rendered + as a web-page. Debug mode can be disabled by changing ``DEBUG=True`` to ``False`` in ``local_settings.py``. In that case Horizon should be started slightly @@ -126,10 +131,10 @@ You can list the registered services with this command: 7. Applying changes If you have changed any ``*.py`` files in - ``horizon/openstack_dashboard/contrib/sahara/content/data_processing`` directory, - Horizon will notice that and reload automatically. However changes made to - non-python files may not be noticed, so you have to restart Horizon again - manually, as described in step 6. + ``horizon/openstack_dashboard/contrib/sahara/content/data_processing`` + directory, Horizon will notice that and reload automatically. However + changes made to non-python files may not be noticed, so you have to restart + Horizon again manually, as described in step 6. Using the Data Processing Dashboard without Keystone Registration +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ diff --git a/doc/source/restapi.rst b/doc/source/restapi.rst index fe4009d582..be04466353 100644 --- a/doc/source/restapi.rst +++ b/doc/source/restapi.rst @@ -81,6 +81,8 @@ error description. 2 API ===== -- `Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing)`_ +- `Sahara REST API Reference (OpenStack API Complete Reference - + DataProcessing)`_ + .. _`Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing)`: http://api.openstack.org/api-ref-data-processing-v1.1.html + - .. _`Sahara REST API Reference (OpenStack API Complete Reference - DataProcessing)`: http://api.openstack.org/api-ref-data-processing-v1.1.html diff --git a/doc/source/userdoc/advanced.configuration.guide.rst b/doc/source/userdoc/advanced.configuration.guide.rst index 47422227b5..32130e933e 100644 --- a/doc/source/userdoc/advanced.configuration.guide.rst +++ b/doc/source/userdoc/advanced.configuration.guide.rst @@ -446,18 +446,19 @@ NTP package is included in the image (the sahara disk image builder will include NTP in all images it generates). The default NTP server will be ``pool.ntp.org``; this can be overridden using the ``default_ntp_server`` setting in the ``DEFAULT`` section of the sahara configuration file. -If you are creating cluster templates using the sahara UI and would like to specify -a different NTP server for a particular cluster template, use the -``URL of NTP server`` setting in the ``General Parameters`` -section when you create the template. If you would like to disable NTP for a -particular cluster template, deselect the ``Enable NTP service`` checkbox in -the ``General Parameters`` section when you create the template. -If you are creating clusters using the sahara CLI, you can specify another NTP server -or disable NTP service using the examples below. +If you are creating cluster templates using the sahara UI and would like to +specify a different NTP server for a particular cluster template, use the ``URL +of NTP server`` setting in the ``General Parameters`` section when you create +the template. If you would like to disable NTP for a particular cluster +template, deselect the ``Enable NTP service`` checkbox in the ``General +Parameters`` section when you create the template. -If you want to disable configuring the NTP service, you should specify the following -configs for the cluster: +If you are creating clusters using the sahara CLI, you can specify another NTP +server or disable NTP service using the examples below. + +If you want to disable configuring the NTP service, you should specify the +following configs for the cluster: .. sourcecode:: json @@ -467,8 +468,8 @@ configs for the cluster: } } -If you want to disable configuring NTP service, you should specify following configs -for the cluster: +If you want to disable configuring NTP service, you should specify following +configs for the cluster: .. sourcecode:: json diff --git a/doc/source/userdoc/cdh_imagebuilder.rst b/doc/source/userdoc/cdh_imagebuilder.rst index 7d3dc78a41..3ae395c35f 100644 --- a/doc/source/userdoc/cdh_imagebuilder.rst +++ b/doc/source/userdoc/cdh_imagebuilder.rst @@ -30,18 +30,18 @@ To create cloudera images follow these steps: 2. Use tox to build images. - You can run "tox -e venv -- sahara-image-create" command in sahara-image-elements - directory to build images. By default this script will attempt to create cloud - images for all versions of supported plugins and all operating systems - (subset of Ubuntu, Fedora, and CentOS depending on plugin). To only create - Cloudera images, you should use the "-p cloudera" parameter in the command - line. If you want to create the image only for a specific operating system, - you should use the "-i ubuntu|centos" parameter to assign the operating - system (the cloudera plugin only supports Ubuntu and Centos). If you want - to create the image only for a specific Cloudera version, you should use the - "-v 5.0|5.3|5.4" parameter to assign the version. Below is an example to create - Cloudera images for both Ubuntu and CentOS with Cloudera Express 5.4.0 - version. + You can run "tox -e venv -- sahara-image-create" command in + sahara-image-elements directory to build images. By default this script will + attempt to create cloud images for all versions of supported plugins and all + operating systems (subset of Ubuntu, Fedora, and CentOS depending on + plugin). To only create Cloudera images, you should use the "-p cloudera" + parameter in the command line. If you want to create the image only for a + specific operating system, you should use the "-i ubuntu|centos" parameter + to assign the operating system (the cloudera plugin only supports Ubuntu and + Centos). If you want to create the image only for a specific Cloudera + version, you should use the "-v 5.0|5.3|5.4" parameter to assign the + version. Below is an example to create Cloudera images for both Ubuntu and + CentOS with Cloudera Express 5.4.0 version. .. sourcecode:: console @@ -57,9 +57,9 @@ To create cloudera images follow these steps: NOTE: If you don't want to use default values, you should explicitly set the values of your required parameters. - The script will create required cloud images using image elements that install - all the necessary packages and configure them. You will find the created - images in the parent directory. + The script will create required cloud images using image elements that + install all the necessary packages and configure them. You will find the + created images in the parent directory. .. note:: diff --git a/doc/source/userdoc/cdh_plugin.rst b/doc/source/userdoc/cdh_plugin.rst index 9e0e296be8..2dccda4481 100644 --- a/doc/source/userdoc/cdh_plugin.rst +++ b/doc/source/userdoc/cdh_plugin.rst @@ -13,9 +13,9 @@ to provision cluster or you could download prepared images from http://sahara-files.mirantis.com/images/upstream/kilo/ They already have Cloudera Express installed (5.0.0, 5.3.0, or 5.4.0 version). -The cloudera plugin requires an image to be tagged in Sahara Image Registry with -two tags: 'cdh' and '' (e.g. '5', '5.3.0' or '5.4.0', here '5' -stands for '5.0.0'). +The cloudera plugin requires an image to be tagged in Sahara Image Registry +with two tags: 'cdh' and '' (e.g. '5', '5.3.0' or '5.4.0', +here '5' stands for '5.0.0'). The default username specified for these images is different for each distribution: @@ -32,8 +32,8 @@ Services Supported ------------------ Currently below services are supported in both versions of Cloudera plugin: -HDFS, Oozie, YARN, Spark, Zookeeper, Hive, Hue, HBase. 5.3.0 version -of Cloudera Plugin also supported following services: Impala, Flume, Solr, Sqoop, +HDFS, Oozie, YARN, Spark, Zookeeper, Hive, Hue, HBase. 5.3.0 version of +Cloudera Plugin also supported following services: Impala, Flume, Solr, Sqoop, and Key-value Store Indexer. 5.4.0 version added KMS service support based on 5.3.0 version. @@ -47,8 +47,8 @@ and Key-value Store Indexer. 5.4.0 version added KMS service support based on High Availablity Support ------------------------ -Currently HDFS NameNode High Availability is supported in Cloudera 5.4.0 version. -You can refer to :doc:`features` for the detail info. +Currently HDFS NameNode High Availability is supported in Cloudera 5.4.0 +version. You can refer to :doc:`features` for the detail info. YARN ResourceManager High Availability is supported in Cloudera 5.4.0 version. This feature adds redundancy in the form of an Active/Standby ResourceManager @@ -87,8 +87,8 @@ cloudera plugin versions: and at least one hbase regionserver. + Cluster can't contain hbase regionserver without at least one hbase maser. -In case of 5.3.0 or 5.4.0 version of Cloudera Plugin there are few extra limitations -in the cluster topology: +In case of 5.3.0 or 5.4.0 version of Cloudera Plugin there are few extra +limitations in the cluster topology: + Cluster can't contain flume without at least one datanode. + Cluster can contain at most one sentry server service. diff --git a/doc/source/userdoc/configs_recommendations.rst b/doc/source/userdoc/configs_recommendations.rst index bb4bc6859c..5bcd236fd8 100644 --- a/doc/source/userdoc/configs_recommendations.rst +++ b/doc/source/userdoc/configs_recommendations.rst @@ -1,31 +1,35 @@ Autoconfiguring templates ========================= -During the Liberty development cycle sahara implemented a tool that recommends and -applies configuration values for cluster templates and node group templates. -These recommendations are based on the number of specific instances and on -flavors of the cluster node groups. Currently the following plugins support -this feature: +During the Liberty development cycle sahara implemented a tool that recommends +and applies configuration values for cluster templates and node group +templates. These recommendations are based on the number of specific instances +and on flavors of the cluster node groups. Currently the following plugins +support this feature: * Cloudera; * Spark; * the Vanilla Apache Hadoop plugin. -By default this feature is enabled for all cluster templates and node group templates. -If you want to disable this feature for a particular cluster or node group template -you should set the ``use_autoconfig`` field to ``false``. +By default this feature is enabled for all cluster templates and node group +templates. If you want to disable this feature for a particular cluster or +node group template you should set the ``use_autoconfig`` field to ``false``. .. NOTE - Also, if you manually set configs from the list below, the recommended configs - will not be applied. + Also, if you manually set configs from the list below, the recommended + configs will not be applied. -The following describes the settings for which sahara can recommend autoconfiguration: +The following describes the settings for which sahara can recommend +autoconfiguration: The Cloudera, Spark and Vanilla Apache Hadoop plugin support configuring -``dfs.replication`` (``dfs_replication`` for Cloudera plugin) which is calculated as -a minimun from the amount of ``datanode`` (``HDFS_DATANODE`` for Cloudera plugin) instances in -the cluster and the default value for ``dfs.replication``. +``dfs.replication`` (``dfs_replication`` for Cloudera plugin) which is +calculated as a minimun from the amount of ``datanode`` (``HDFS_DATANODE`` for +Cloudera plugin) instances in the cluster and the default value for +``dfs.replication``. The Vanilla Apache Hadoop plugin and Cloudera plugin support autoconfiguration -of basic YARN and MapReduce configs. These autoconfigurations are based on the following -documentation: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html +of basic YARN and MapReduce configs. These autoconfigurations are based on the +following documentation: +http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html + diff --git a/doc/source/userdoc/dashboard.guide.rst b/doc/source/userdoc/dashboard.guide.rst index db7a38f3a7..41a2361690 100644 --- a/doc/source/userdoc/dashboard.guide.rst +++ b/doc/source/userdoc/dashboard.guide.rst @@ -12,8 +12,8 @@ The sample configuration file is available `here. = 6) * Apache Hadoop installed * 'hadoop' user created -See :doc:`hadoop-swift` for information on using Swift with your sahara cluster (for EDP support Swift integration is currently required). +See :doc:`hadoop-swift` for information on using Swift with your sahara cluster +(for EDP support Swift integration is currently required). To support EDP, the following components must also be installed on the guest: @@ -30,17 +35,23 @@ To support EDP, the following components must also be installed on the guest: * mysql * hive -See :doc:`vanilla_imagebuilder` for instructions on building images for this plugin. +See :doc:`vanilla_imagebuilder` for instructions on building images for this +plugin. Hortonworks Plugin Requirements ------------------------------- -This plugin does not have any additional requirements. Currently, only the CentOS Linux distribution is supported but other distributions will be supported in the future. -To speed up provisioning, the HDP packages can be pre-installed on the image used. The packages' versions depend on the HDP version being used. +This plugin does not have any additional requirements. Currently, only the +CentOS Linux distribution is supported but other distributions will be +supported in the future. +To speed up provisioning, the HDP packages can be pre-installed on the image +used. The packages' versions depend on the HDP version being used. Cloudera Plugin Requirements ---------------------------- -Cloudera Plugin does not have any additional requirements, just build a CDH image to deploy the cluster. +Cloudera Plugin does not have any additional requirements, just build a CDH +image to deploy the cluster. -See :doc:`cdh_imagebuilder` for instructions on building images for this plugin. +See :doc:`cdh_imagebuilder` for instructions on building images for this +plugin. diff --git a/doc/source/userdoc/hadoop-swift.rst b/doc/source/userdoc/hadoop-swift.rst index 18c3a90a69..3ca9238eed 100644 --- a/doc/source/userdoc/hadoop-swift.rst +++ b/doc/source/userdoc/hadoop-swift.rst @@ -47,13 +47,18 @@ instances using this template: There are two types of configs here: -1. General. The ``${name}`` in this case equals to ``fs.swift``. Here is the list of ``${config}``: +1. General. The ``${name}`` in this case equals to ``fs.swift``. Here is the + list of ``${config}``: - * ``.impl`` - Swift FileSystem implementation. The ${value} is ``org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem`` + * ``.impl`` - Swift FileSystem implementation. The ${value} is + ``org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem`` * ``.connect.timeout`` - timeout for all connections by default: 15000 - * ``.socket.timeout`` - how long the connection waits for responses from servers. by default: 60000 - * ``.connect.retry.count`` - connection retry count for all connections. by default: 3 - * ``.connect.throttle.delay`` - delay in millis between bulk (delete, rename, copy operations). by default: 0 + * ``.socket.timeout`` - how long the connection waits for responses from + servers. by default: 60000 + * ``.connect.retry.count`` - connection retry count for all connections. by + default: 3 + * ``.connect.throttle.delay`` - delay in millis between bulk (delete, + rename, copy operations). by default: 0 * ``.blocksize`` - blocksize for filesystem. By default: 32Mb * ``.partsize`` - the partition size for uploads. By default: 4608*1024Kb * ``.requestsize`` - request size for reads in KB. By default: 64Kb @@ -76,8 +81,9 @@ There are two types of configs here: * ``.http.port`` * ``.https.port`` * ``.region`` - Swift region is used when cloud has more than one Swift - installation. If region param is not set first region from Keystone endpoint - list will be chosen. If region param not found exception will be thrown. + installation. If region param is not set first region from Keystone + endpoint list will be chosen. If region param not found exception will be + thrown. * ``.location-aware`` - turn On location awareness. Is false by default * ``.apikey`` * ``.public`` diff --git a/doc/source/userdoc/hdp_plugin.rst b/doc/source/userdoc/hdp_plugin.rst index 0604440bf3..5fe935bebf 100644 --- a/doc/source/userdoc/hdp_plugin.rst +++ b/doc/source/userdoc/hdp_plugin.rst @@ -18,11 +18,11 @@ The HDP plugin can make use of Ambari Blueprints for cluster provisioning. Apache Ambari Blueprints ------------------------ -Apache Ambari Blueprints is a portable document definition, which provides -a complete definition for an Apache Hadoop cluster, including cluster topology, +Apache Ambari Blueprints is a portable document definition, which provides a +complete definition for an Apache Hadoop cluster, including cluster topology, components, services and their configurations. Ambari Blueprints can be -consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack. -The benefits of this approach is that it allows for Hadoop clusters to be +consumed by the HDP plugin to instantiate a Hadoop cluster on OpenStack. The +benefits of this approach is that it allows for Hadoop clusters to be configured and deployed using an Ambari native format that can be used with as well as outside of OpenStack allowing for clusters to be re-instantiated in a variety of environments. @@ -49,8 +49,8 @@ Images ------ The sahara HDP plugin can make use of either minimal (operating system only) images or pre-populated HDP images. The base requirement for both is that the -image is cloud-init enabled and contains a supported operating system -(see http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html). +image is cloud-init enabled and contains a supported operating system (see +http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.2.4/bk_hdp1-system-admin-guide/content/sysadminguides_ha_chap2_3.html). The advantage of a pre-populated image is that provisioning time is reduced, as packages do not need to be downloaded and installed which make up the @@ -72,10 +72,10 @@ The username specified should be 'cloud-user'. HDFS NameNode High Availability ------------------------------- -HDFS NameNode High Availability (Using the Quorum Journal Manager) can be deployed -automatically with HDP 2.0.6. Currently the only way to deploy it is through the -command line client (python-saharaclient) or sahara REST API by simply adding the -following cluster_configs parameter in the cluster's JSON : +HDFS NameNode High Availability (Using the Quorum Journal Manager) can be +deployed automatically with HDP 2.0.6. Currently the only way to deploy it is +through the command line client (python-saharaclient) or sahara REST API by +simply adding the following cluster_configs parameter in the cluster's JSON : .. sourcecode:: cfg @@ -89,8 +89,8 @@ The NameNode High Availability is deployed using 2 NameNodes, one active and one standby. The NameNodes use a set of JOURNALNODES and ZOOKEEPER_SERVERS to ensure the necessary synchronization. -A typical Highly available HDP 2.0.6 cluster uses 2 separate NameNodes, at least 3 -JOURNALNODES and at least 3 ZOOKEEPER_SERVERS. +A typical Highly available HDP 2.0.6 cluster uses 2 separate NameNodes, at +least 3 JOURNALNODES and at least 3 ZOOKEEPER_SERVERS. When HDFS NameNode High Availability is enabled, the plugin will perform the following additional validations: diff --git a/doc/source/userdoc/mapr_plugin.rst b/doc/source/userdoc/mapr_plugin.rst index 71a2fe65c2..0b6986b7c8 100644 --- a/doc/source/userdoc/mapr_plugin.rst +++ b/doc/source/userdoc/mapr_plugin.rst @@ -5,22 +5,27 @@ OpenStack in an easy way and do it, quickly, conveniently and simply. Operation --------- -The MapR Plugin performs the following four primary functions during cluster creation: +The MapR Plugin performs the following four primary functions during cluster +creation: -1. MapR components deployment - the plugin manages the deployment of the required software to the target VMs -2. Services Installation - MapR services are installed according to provided roles list -3. Services Configuration - the plugin combines default settings with user provided settings -4. Services Start - the plugin starts appropriate services according to specified roles +1. MapR components deployment - the plugin manages the deployment of the + required software to the target VMs +2. Services Installation - MapR services are installed according to provided + roles list +3. Services Configuration - the plugin combines default settings with user + provided settings +4. Services Start - the plugin starts appropriate services according to + specified roles Images ------ The Sahara MapR plugin can make use of either minimal (operating system only) images or pre-populated MapR images. The base requirement for both is that the -image is cloud-init enabled and contains a supported operating system -(see http://doc.mapr.com/display/MapR/OS+Support+Matrix). +image is cloud-init enabled and contains a supported operating system (see +http://doc.mapr.com/display/MapR/OS+Support+Matrix). -The advantage of a pre-populated image is that provisioning time is reduced, -as packages do not need to be downloaded which make up the majority of the time +The advantage of a pre-populated image is that provisioning time is reduced, as +packages do not need to be downloaded which make up the majority of the time spent in the provisioning cycle. In addition, provisioning large clusters will put a burden on the network as packages for all nodes need to be downloaded from the package repository. @@ -45,7 +50,8 @@ two tags: 'mapr' and '' (e.g. '4.0.1.mrv2'). Note that Spark should be run on plain or 4.0.1 images. -The default username specified for these images is different for each distribution: +The default username specified for these images is different for each +distribution: +--------------+------------+ | OS | username | @@ -58,7 +64,8 @@ The default username specified for these images is different for each distributi Hadoop Version Support ---------------------- -The MapR plugin currently supports Hadoop 0.20.2 (3.1.1, 4.0.1.mrv1, 4.0.2.mrv1), +The MapR plugin currently supports Hadoop 0.20.2 (3.1.1, 4.0.1.mrv1, +4.0.2.mrv1), Hadoop 2.4.1 (4.0.2.mrv2) and Hadoop 2.5.1 (4.0.2.mrv2). Cluster Validation @@ -110,7 +117,8 @@ Hue service is considered valid if: HttpFS service is considered valid if cluster has exactly 1 *HttpFS* process -Sqoop service is considered valid if cluster has exactly 1 *Sqoop2-Server* process +Sqoop service is considered valid if cluster has exactly 1 *Sqoop2-Server* +process The MapR Plugin --------------- diff --git a/doc/source/userdoc/registering_image.rst b/doc/source/userdoc/registering_image.rst index c49af714ed..cf97bbfa1a 100644 --- a/doc/source/userdoc/registering_image.rst +++ b/doc/source/userdoc/registering_image.rst @@ -2,9 +2,10 @@ Registering an Image ==================== Sahara deploys a cluster of machines using images stored in Glance. + Each plugin has its own requirements on the image contents (see specific plugin -documentation for details). Two general requirements for an image are to have the -cloud-init and the ssh-server packages installed. +documentation for details). Two general requirements for an image are to have +the cloud-init and the ssh-server packages installed. Sahara requires the images to be registered in the Sahara Image Registry. A registered image must have two properties set: @@ -12,5 +13,5 @@ A registered image must have two properties set: * username - a name of the default cloud-init user. * tags - certain tags mark image to be suitable for certain plugins. -The username depends on the image that is used and the tags depend on the plugin used. -You can find both in the respective plugin's documentation. +The username depends on the image that is used and the tags depend on the +plugin used. You can find both in the respective plugin's documentation. diff --git a/doc/source/userdoc/statuses.rst b/doc/source/userdoc/statuses.rst index c723073a5f..5be47c176e 100644 --- a/doc/source/userdoc/statuses.rst +++ b/doc/source/userdoc/statuses.rst @@ -3,8 +3,8 @@ Sahara Cluster Statuses Overview All Sahara Cluster operations are performed in multiple steps. A Cluster object has a ``Status`` attribute which changes when Sahara finishes one step of -operations and starts another one. Also a Cluster object has a ``Status description`` -attribute which changes whenever Cluster errors occur. +operations and starts another one. Also a Cluster object has a ``Status +description`` attribute which changes whenever Cluster errors occur. Sahara supports three types of Cluster operations: * Create a new Cluster @@ -21,19 +21,22 @@ Before performing any operations with OpenStack environment, Sahara validates user input. There are two types of validations, that are done: - * Check that a request contains all necessary fields and that the request does not violate -any constraints like unique naming, etc. - * Plugin check (optional). The provisioning Plugin may also perform any specific checks like a Cluster topology validation check. + * Check that a request contains all necessary fields and that the request does + not violate any constraints like unique naming, etc. + * Plugin check (optional). The provisioning Plugin may also perform any + specific checks like a Cluster topology validation check. -If any of the validations fails during creating, the Cluster object will -still be kept in the database with an ``Error`` status. If any validations fails during scaling the ``Active`` Cluster, it will be -kept with an ``Active`` status. -In both cases status description will contain error messages about the reasons of failure. +If any of the validations fails during creating, the Cluster object will still +be kept in the database with an ``Error`` status. If any validations fails +during scaling the ``Active`` Cluster, it will be kept with an ``Active`` +status. In both cases status description will contain error messages about the +reasons of failure. 2. InfraUpdating ~~~~~~~~~~~~~~~~ -This status means that the Provisioning plugin is performing some infrastructure updates. +This status means that the Provisioning plugin is performing some +infrastructure updates. 3. Spawning ~~~~~~~~~~~ @@ -49,22 +52,23 @@ so sahara will wait until all of the VMs are in an ``Active`` state. 4. Waiting ~~~~~~~~~~ -Sahara waits while VMs' operating systems boot up and all internal infrastructure -components like networks and volumes are attached and ready to use. +Sahara waits while VMs' operating systems boot up and all internal +infrastructure components like networks and volumes are attached and ready to +use. 5. Preparing ~~~~~~~~~~~~ -Sahara prepares a Cluster for starting. This step includes generating the ``/etc/hosts`` -file, so that all instances can access each other by a hostname. Also Sahara -updates the ``authorized_keys`` file on each VM, so that VMs can communicate without -passwords. +Sahara prepares a Cluster for starting. This step includes generating the +``/etc/hosts`` file, so that all instances can access each other by a hostname. +Also Sahara updates the ``authorized_keys`` file on each VM, so that VMs can +communicate without passwords. 6. Configuring ~~~~~~~~~~~~~~ -Sahara pushes service configurations to VMs. Both XML and JSON based configurations and -environmental variables are set on this step. +Sahara pushes service configurations to VMs. Both XML and JSON based +configurations and environmental variables are set on this step. 7. Starting ~~~~~~~~~~~ @@ -74,7 +78,8 @@ Sahara is starting Hadoop services on Cluster's VMs. 8. Active ~~~~~~~~~ -Active status means that a Cluster has started successfully and is ready to run EDP Jobs. +Active status means that a Cluster has started successfully and is ready to run +EDP Jobs. Scaling/Shrinking an existing Cluster @@ -84,7 +89,8 @@ Scaling/Shrinking an existing Cluster ~~~~~~~~~~~~~ Sahara checks the scale/shrink request for validity. The Plugin method called -for performing Plugin specific checks is different from the validation method in creation. +for performing Plugin specific checks is different from the validation method +in creation. 2. Scaling ~~~~~~~~~~ @@ -95,22 +101,23 @@ and creating new ones to join the existing Node Groups. 3. Adding Instances ~~~~~~~~~~~~~~~~~~~ -Status is similar to ``Spawning`` in Custer creation. Sahara adds required amount -of VMs to the existing Node Groups and creates new Node Groups. +Status is similar to ``Spawning`` in Custer creation. Sahara adds required +amount of VMs to the existing Node Groups and creates new Node Groups. 4. Configuring ~~~~~~~~~~~~~~ -Status is similar to ``Configuring`` in Cluster creation. New instances are being configured -in the same manner as already existing ones. The VMs in the existing Cluster are also updated -with a new ``/etc/hosts`` file. +Status is similar to ``Configuring`` in Cluster creation. New instances are +being configured in the same manner as already existing ones. The VMs in the +existing Cluster are also updated with a new ``/etc/hosts`` file. 5. Decommissioning ~~~~~~~~~~~~~~~~~~ Sahara stops Hadoop services on VMs that will be deleted from a Cluster. -Decommissioning a Data Node may take some time because Hadoop rearranges data replicas -around the Cluster, so that no data will be lost after that Data Node is deleted. +Decommissioning a Data Node may take some time because Hadoop rearranges data +replicas around the Cluster, so that no data will be lost after that Data Node +is deleted. 6. Deleting Instances ~~~~~~~~~~~~~~~~~~~~~ @@ -132,19 +139,22 @@ Deleting an existing Cluster 1. Deleting ~~~~~~~~~~~ -The only step, that releases all Cluster's resources and removes it from the database. +The only step, that releases all Cluster's resources and removes it from the +database. Error State ----------- If the Cluster creation fails, the Cluster will enter the ``Error`` state. -This status means the Cluster may not be able to perform any operations normally. -This cluster will stay in the database until it is manually deleted. The reason for -failure may be found in the sahara logs. Also, the status description will contain -information about the error. +This status means the Cluster may not be able to perform any operations +normally. This cluster will stay in the database until it is manually deleted. +The reason for failure may be found in the sahara logs. Also, the status +description will contain information about the error. If an error occurs during the ``Adding Instances`` operation, Sahara will first -try to rollback this operation. If a rollback is impossible or fails itself, then -the Cluster will also go into an ``Error`` state. If a rollback was successful, Cluster will get into an ``Active`` state -and status description will contain a short message about the reason of ``Adding Instances`` failure. +try to rollback this operation. If a rollback is impossible or fails itself, +then the Cluster will also go into an ``Error`` state. If a rollback was +successful, Cluster will get into an ``Active`` state and status description +will contain a short message about the reason of ``Adding Instances`` failure. + diff --git a/doc/source/userdoc/upgrade.guide.rst b/doc/source/userdoc/upgrade.guide.rst index 6a4705c771..10611cd9ac 100644 --- a/doc/source/userdoc/upgrade.guide.rst +++ b/doc/source/userdoc/upgrade.guide.rst @@ -1,9 +1,8 @@ Sahara Upgrade Guide ==================== -This page contains details about upgrading sahara between releases such -as configuration file updates, database migrations, and architectural -changes. +This page contains details about upgrading sahara between releases such as +configuration file updates, database migrations, and architectural changes. Icehouse -> Juno ---------------- @@ -18,11 +17,11 @@ All-In-One sahara is desired. Authentication middleware changes +++++++++++++++++++++++++++++++++ -The custom auth_token middleware has been deprecated in favor of the -keystone middleware. This change requires an update to the sahara -configuration file. To update your configuration file you should replace -the following parameters from the ``[DEFAULT]`` section with the new -parameters in the ``[keystone_authtoken]`` section: +The custom auth_token middleware has been deprecated in favor of the keystone +middleware. This change requires an update to the sahara configuration file. To +update your configuration file you should replace the following parameters from +the ``[DEFAULT]`` section with the new parameters in the +``[keystone_authtoken]`` section: +----------------------+--------------------+ | Old parameter name | New parameter name | @@ -49,12 +48,11 @@ The oslo based code from sahara.openstack.common.db has been replaced by the usage of the oslo.db package. This change does not require any update to sahara's configuration file. -Additionally, the usage of SQLite databases has been deprecated. Please -use MySQL or PostgreSQL databases for sahara. SQLite has been -deprecated because it does not, and is not going to, support the -``ALTER COLUMN`` and ``DROP COLUMN`` commands required for migrations -between versions. For more information please see -http://www.sqlite.org/omitted.html +Additionally, the usage of SQLite databases has been deprecated. Please use +MySQL or PostgreSQL databases for sahara. SQLite has been deprecated because it +does not, and is not going to, support the ``ALTER COLUMN`` and ``DROP COLUMN`` +commands required for migrations between versions. For more information please +see http://www.sqlite.org/omitted.html Sahara integration into OpenStack Dashboard +++++++++++++++++++++++++++++++++++++++++++ @@ -82,10 +80,9 @@ name for instances created by sahara using HEAT was always 'ec2-user'. As of Juno, the user name is taken from the image registry as described in the :doc:`registering_image` document. -This change breaks backward compatibility for clusters created using the -HEAT infrastructure engine prior to the Juno release. Clusters will -continue to operate, but we do not recommended using the scaling operations -with them. +This change breaks backward compatibility for clusters created using the HEAT +infrastructure engine prior to the Juno release. Clusters will continue to +operate, but we do not recommended using the scaling operations with them. Anti affinity implementation changed ++++++++++++++++++++++++++++++++++++ @@ -126,13 +123,14 @@ Kilo -> Liberty Direct engine deprecation +++++++++++++++++++++++++ -In the Liberty release the direct infrastructure engine has been deprecated -and the heat infrastructure engine is now default. This means, that it is preferable -to use heat engine instead now. In the Liberty release you can continue to -operate clusters with the direct engine (create, delete, scale). Using heat engine only -the delete operation is available on clusters that were created by the direct engine. -After the Liberty release the direct engine will be removed, this means that you will -only be able to delete clusters created with the direct engine. +In the Liberty release the direct infrastructure engine has been deprecated and +the heat infrastructure engine is now default. This means, that it is +preferable to use heat engine instead now. In the Liberty release you can +continue to operate clusters with the direct engine (create, delete, scale). +Using heat engine only the delete operation is available on clusters that were +created by the direct engine. After the Liberty release the direct engine will +be removed, this means that you will only be able to delete clusters created +with the direct engine. Policy namespace changed (policy.json) ++++++++++++++++++++++++++++++++++++++ diff --git a/doc/source/userdoc/vanilla_imagebuilder.rst b/doc/source/userdoc/vanilla_imagebuilder.rst index 1fa8902889..8bc3ec7ec8 100644 --- a/doc/source/userdoc/vanilla_imagebuilder.rst +++ b/doc/source/userdoc/vanilla_imagebuilder.rst @@ -26,13 +26,14 @@ Elements for building vanilla images are stored in the To create vanilla images follow these steps: -1. Clone repository "https://github.com/openstack/sahara-image-elements" locally. +1. Clone repository "https://github.com/openstack/sahara-image-elements" + locally. 2. Use tox to build images. You can run the command below in sahara-image-elements - directory to build images. By default this script will attempt to create cloud - images for all versions of supported plugins and all operating systems + directory to build images. By default this script will attempt to create + cloud images for all versions of supported plugins and all operating systems (subset of Ubuntu, Fedora, and CentOS depending on plugin). .. sourcecode:: console @@ -40,7 +41,10 @@ To create vanilla images follow these steps: tox -e venv -- sahara-image-create -u Tox will create a virtualenv and install required python packages in it, - clone the repositories "https://github.com/openstack/diskimage-builder" and "https://github.com/openstack/sahara-image-elements" and export necessary parameters. + clone the repositories "https://github.com/openstack/diskimage-builder" and + "https://github.com/openstack/sahara-image-elements" and export necessary + parameters. + * ``DIB_HADOOP_VERSION`` - version of Hadoop to install * ``JAVA_DOWNLOAD_URL`` - download link for JDK (tarball or bin) * ``OOZIE_DOWNLOAD_URL`` - download link for OOZIE (we have built diff --git a/tox.ini b/tox.ini index 0e3b4a9fca..9ffdaebca0 100644 --- a/tox.ini +++ b/tox.ini @@ -41,18 +41,13 @@ downloadcache = ~/cache/pip [testenv:pep8] commands = flake8 {posargs} + doc8 doc/source # Check that .po and .pot files are valid: bash -c "find sahara -type f -regex '.*\.pot?' -print0|xargs -0 -n 1 msgfmt --check-format -o /dev/null" # Run bashate checks bash -c "find sahara -iname '*.sh' -print0 | xargs -0 bashate -v" bash -c "find devstack -not -name README.rst -and -not -name \*.json -type f -print0 | xargs -0 bashate -v" -[testenv:doc8] -deps = - -r{toxinidir}/requirements.txt - -r{toxinidir}/test-requirements.txt -commands = doc8 doc/source - [testenv:venv] commands = {posargs}