Add a CLI tool for managing default templates
This change adds a CLI called sahara-templates to manage default templates. Partial-implements: blueprint default-templates Change-Id: I4f30bd6bc378d90fda41b512c04afe8023d2b4b2
This commit is contained in:
parent
99e2044a7a
commit
e857447879
|
@ -8,9 +8,12 @@ include sahara/db/migration/alembic_migrations/env.py
|
||||||
include sahara/db/migration/alembic_migrations/script.py.mako
|
include sahara/db/migration/alembic_migrations/script.py.mako
|
||||||
include sahara/db/migration/alembic_migrations/versions/*.py
|
include sahara/db/migration/alembic_migrations/versions/*.py
|
||||||
include sahara/db/migration/alembic_migrations/versions/README
|
include sahara/db/migration/alembic_migrations/versions/README
|
||||||
|
include sahara/db/templates/README.rst
|
||||||
|
|
||||||
recursive-include sahara/locale *
|
recursive-include sahara/locale *
|
||||||
|
|
||||||
|
recursive-include sahara/plugins/default_templates *.json
|
||||||
|
include sahara/plugins/default_templates/template.conf
|
||||||
include sahara/plugins/cdh/v5/resources/cdh_config.py
|
include sahara/plugins/cdh/v5/resources/cdh_config.py
|
||||||
include sahara/plugins/cdh/v5/resources/*.sh
|
include sahara/plugins/cdh/v5/resources/*.sh
|
||||||
include sahara/plugins/cdh/v5/resources/*.json
|
include sahara/plugins/cdh/v5/resources/*.json
|
||||||
|
|
|
@ -0,0 +1,278 @@
|
||||||
|
Sahara Default Template CLI
|
||||||
|
===========================
|
||||||
|
|
||||||
|
The *sahara-templates* application is a simple CLI for managing default
|
||||||
|
templates in Sahara. This document gives an overview of default templates
|
||||||
|
and explains how to use the CLI.
|
||||||
|
|
||||||
|
Default Templates Overview
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
The goal of the default template facility in Sahara is to make cluster
|
||||||
|
launching quick and easy by providing users with a stable set of pre-generated
|
||||||
|
node group and cluster templates for each of the Sahara provisioning plugins.
|
||||||
|
|
||||||
|
Template sets are defined in .json files grouped into directories. The CLI
|
||||||
|
reads these template sets and writes directly to the Sahara database.
|
||||||
|
|
||||||
|
Default templates may only be created, modified, or deleted via the CLI --
|
||||||
|
operations through the python-saharaclient or REST API are restricted.
|
||||||
|
|
||||||
|
JSON Files
|
||||||
|
----------
|
||||||
|
|
||||||
|
Cluster and node group templates are defined in .json files.
|
||||||
|
|
||||||
|
A very simple cluster template JSON file might look like this:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
{
|
||||||
|
"plugin_name": "vanilla",
|
||||||
|
"hadoop_version": "2.6.0",
|
||||||
|
"node_groups": [
|
||||||
|
{
|
||||||
|
"name": "master",
|
||||||
|
"count": 1,
|
||||||
|
"node_group_template_id": "{master}"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "worker",
|
||||||
|
"count": 3,
|
||||||
|
"node_group_template_id": "{worker}"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"name": "cluster-template"
|
||||||
|
}
|
||||||
|
|
||||||
|
The values of the *node_group_template_id* fields are the
|
||||||
|
names of node group templates in set braces. In this example,
|
||||||
|
*master* and *worker* are the names of node group templates defined in
|
||||||
|
.json files in the same directory. When the CLI processes the
|
||||||
|
directory, it will create the node group templates first and
|
||||||
|
then substitute the appropriate id values for the name references
|
||||||
|
when it creates the cluster template.
|
||||||
|
|
||||||
|
Configuration Files and Value Substitutions
|
||||||
|
-------------------------------------------
|
||||||
|
|
||||||
|
The CLI supports value substitution for a limited set of fields.
|
||||||
|
For cluster templates, the following fields may use substitution:
|
||||||
|
|
||||||
|
* default_image_id
|
||||||
|
* neutron_management_network
|
||||||
|
|
||||||
|
For node group templates, the following fields may use substitution:
|
||||||
|
|
||||||
|
* image_id
|
||||||
|
* flavor_id
|
||||||
|
* floating_ip_pool
|
||||||
|
|
||||||
|
Substitution is indicated for one of these fields in a .json file
|
||||||
|
when the value is the name of the field in set braces. Here is an example
|
||||||
|
of a node group template file that uses substitution for *flavor_id*:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
{
|
||||||
|
"plugin_name": "vanilla",
|
||||||
|
"hadoop_version": "2.6.0",
|
||||||
|
"node_processes": [
|
||||||
|
"namenode",
|
||||||
|
"resourcemanager",
|
||||||
|
"oozie",
|
||||||
|
"historyserver"
|
||||||
|
],
|
||||||
|
"name": "master",
|
||||||
|
"flavor_id": "{flavor_id}",
|
||||||
|
"floating_ip_pool": "{floating_ip_pool}"
|
||||||
|
}
|
||||||
|
|
||||||
|
The values for *flavor_id* and *floating_ip_pool* in this template
|
||||||
|
will come from a configuration file.
|
||||||
|
|
||||||
|
If a configuration value is found for the substitution, the value will
|
||||||
|
be replaced. If a configuration value is not found, the field will be
|
||||||
|
omitted from the template. (In this example, *flavor_id* is a required
|
||||||
|
field of node group templates and the template will fail validation
|
||||||
|
if there is no substitution value specifed. However, *floating_ip_pool*
|
||||||
|
is not required and so the template will still pass validation if it
|
||||||
|
is omitted).
|
||||||
|
|
||||||
|
The CLI will look for configuration sections with names based on
|
||||||
|
the *plugin_name*, *hadoop_version*, and *name* fields in the
|
||||||
|
template. It will look for sections in the following order:
|
||||||
|
|
||||||
|
* **[<name>]**
|
||||||
|
|
||||||
|
May contain fields only for the type of the named template
|
||||||
|
|
||||||
|
If templates are named in an **unambiguous** way, the template
|
||||||
|
name alone can be a used as the name of the config section.
|
||||||
|
This produces shorter names and aids readability when there
|
||||||
|
is a one-to-one mapping between template names and config
|
||||||
|
sections.
|
||||||
|
|
||||||
|
* **[<plugin_name>_<hadoop_version>_<name>]**
|
||||||
|
|
||||||
|
May contain fields only for the type of the named template
|
||||||
|
|
||||||
|
This form unambiguously applies to a specific template for
|
||||||
|
a specific plugin.
|
||||||
|
|
||||||
|
* **[<plugin_name>_<hadoop_version>]**
|
||||||
|
|
||||||
|
May contain node group or cluster template fields
|
||||||
|
|
||||||
|
* **[<plugin_name>]**
|
||||||
|
|
||||||
|
May contain node group or cluster template fields
|
||||||
|
|
||||||
|
* **[DEFAULT]**
|
||||||
|
|
||||||
|
May contain node group or cluster template fields
|
||||||
|
|
||||||
|
If we have the following configuration file in our example
|
||||||
|
the CLI will find the value of *flavor_id* for the *master* template
|
||||||
|
in the first configuration section and the value for *floating_ip_pool*
|
||||||
|
in the third section:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
[vanilla_2.6.0_master]
|
||||||
|
# This is named for the plugin, version, and template.
|
||||||
|
# It may contain only node group template fields.
|
||||||
|
flavor_id = 5
|
||||||
|
image_id = b7883f8a-9a7f-42cc-89a2-d3c8b1cc7b28
|
||||||
|
|
||||||
|
[vanilla_2.6.0]
|
||||||
|
# This is named for the plugin and version.
|
||||||
|
# It may contain fields for both node group and cluster templates.
|
||||||
|
flavor_id = 4
|
||||||
|
neutron_mangement_network = 9973da0b-68eb-497d-bd48-d85aca37f088
|
||||||
|
|
||||||
|
[vanilla]
|
||||||
|
# This is named for the plugin.
|
||||||
|
# It may contain fields for both node group and cluster templates.
|
||||||
|
flavor_id = 3
|
||||||
|
default_image_id = 89de8d21-9743-4d20-873e-7677973416dd
|
||||||
|
floating_ip_pool = my_pool
|
||||||
|
|
||||||
|
[DEFAULT]
|
||||||
|
# This is the normal default section.
|
||||||
|
# It may contain fields for both node group and cluster templates.
|
||||||
|
flavor_id = 2
|
||||||
|
|
||||||
|
Sample Configuration File
|
||||||
|
-------------------------
|
||||||
|
|
||||||
|
A sample configuration file is provided in
|
||||||
|
*sahara/plugins/default_templates/template.conf*. This
|
||||||
|
file sets the *flavor_id* for most of the node group templates
|
||||||
|
supplied with Sahara to 2 which indicates the *m1.small*
|
||||||
|
flavor in a default OpenStack deployment.
|
||||||
|
|
||||||
|
The master node templates for the CDH plugin have the
|
||||||
|
*flavor_id* set to 4 which indicates the *m1.large* flavor,
|
||||||
|
since these nodes require more resources.
|
||||||
|
|
||||||
|
This configuration file may be used with the CLI as is, or
|
||||||
|
it may be copied and modified. Note that multiple configuration
|
||||||
|
files may be passed to the CLI by repeating the *--config-file*
|
||||||
|
option.
|
||||||
|
|
||||||
|
Other Special Configuration Parameters
|
||||||
|
--------------------------------------
|
||||||
|
|
||||||
|
The only configuration parameter that is strictly required is
|
||||||
|
the *connection* parameter in the *database* section. Without this
|
||||||
|
value the CLI will not be able to connect to the Sahara database.
|
||||||
|
|
||||||
|
By default, the CLI will use the value of the *plugins* parameter
|
||||||
|
in the [DEFAULT] section on *update* to filter the templates that
|
||||||
|
will be created or updated. This parameter in Sahara defaults to
|
||||||
|
the set of fully supported plugins. To restrict the set of plugins
|
||||||
|
for the *update* operation set this parameter or use the
|
||||||
|
*--plugin-name* option.
|
||||||
|
|
||||||
|
Directory Structure
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
The structure of the directory holding .json files for the CLI is
|
||||||
|
very flexible. The CLI will begin processing at the designated
|
||||||
|
starting directory and recurse through subdirectories.
|
||||||
|
|
||||||
|
At each directory level, the CLI will look for .json files to
|
||||||
|
define a set of default templates. Cluster templates may reference
|
||||||
|
node group templates in the same set by name. Templates at different
|
||||||
|
levels in the directory structure are not in the same set.
|
||||||
|
|
||||||
|
Plugin name and version are determined from the values in the .json
|
||||||
|
files, not by the file names or the directory structure.
|
||||||
|
|
||||||
|
Recursion may be turned off with the "-n" option (see below).
|
||||||
|
|
||||||
|
The default starting directory is *sahara/plugins/default_templates*
|
||||||
|
|
||||||
|
Example CLI Commands
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
For ``update``, ``delete``, ``node-group-template-delete``, and
|
||||||
|
``cluster-template-delete`` operations, the tenant must always be specified.
|
||||||
|
For ``node-group-template-delete-id`` and ``cluster-template-delete-id``
|
||||||
|
tenant is not required.
|
||||||
|
All useful information about activity by the CLI is logged
|
||||||
|
|
||||||
|
Create/update all of the default templates bundled with Sahara. Use the standard
|
||||||
|
Sahara configuration file in */etc/sahara/sahara.conf* to specify the plugin list
|
||||||
|
and the database connection string and another configuration file to supply
|
||||||
|
the *flavor_id* values::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file /etc/sahara/sahara.conf --config-file myconfig update -t $TENANT_ID
|
||||||
|
|
||||||
|
Create/update default templates from the directory *mypath*::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig update -t $TENANT_ID -d mypath
|
||||||
|
|
||||||
|
Create/update default templates from the directory *mypath* but do not descend
|
||||||
|
into subirectories::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig update -t $TENANT_ID -d mypath -n
|
||||||
|
|
||||||
|
Create/update default templates bundled with Sahara for just the vanilla plugin::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig update -t $TENANT_ID -p vanilla
|
||||||
|
|
||||||
|
Create/update default templates bundled with Sahara for just version 2.6.0
|
||||||
|
of the vanilla plugin::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig update -t $TENANT_ID -p vanilla -pv 2.6.0
|
||||||
|
|
||||||
|
Create/update default templates bundled with Sahara for just version 2.6.0
|
||||||
|
of the vanilla plugin and version 2.0.6 of the hdp plugin::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig update -t $TENANT_ID -p vanilla -pv vanilla.2.6.0 -p hdp -pv hdp.2.0.6
|
||||||
|
|
||||||
|
Delete default templates for the vanilla plugin::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig delete -t $TENANT_ID -p vanilla
|
||||||
|
|
||||||
|
Delete default templates for version 2.6.0 of the vanilla plugin::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig delete -t $TENANT_ID -p vanilla -pv 2.6.0
|
||||||
|
|
||||||
|
Delete a specific node group template by ID::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig node-group-template-delete-id --id ID
|
||||||
|
|
||||||
|
Delete a specific cluster template by ID::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig cluster-template-delete-id --id ID
|
||||||
|
|
||||||
|
Delete a specific node group template by name::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig node-group-template-delete --name NAME -t $TENANT_ID
|
||||||
|
|
||||||
|
Delete a specific cluster template by name::
|
||||||
|
|
||||||
|
$ sahara-templates --config-file myconfig cluster-template-delete-id --name NAME -t $TENANT_ID
|
|
@ -0,0 +1,803 @@
|
||||||
|
# Copyright 2015 Red Hat, Inc.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
# implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
import copy
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
import jsonschema
|
||||||
|
from oslo_config import cfg
|
||||||
|
import six
|
||||||
|
|
||||||
|
from sahara import conductor
|
||||||
|
from sahara.db.templates import utils as u
|
||||||
|
from sahara.service.validations import cluster_template_schema as clt
|
||||||
|
from sahara.service.validations import node_group_template_schema as ngt
|
||||||
|
from sahara.utils import api_validator
|
||||||
|
|
||||||
|
|
||||||
|
LOG = None
|
||||||
|
CONF = None
|
||||||
|
|
||||||
|
|
||||||
|
# This is broken out to support testability
|
||||||
|
def set_logger(log):
|
||||||
|
global LOG
|
||||||
|
LOG = log
|
||||||
|
|
||||||
|
|
||||||
|
# This is broken out to support testability
|
||||||
|
def set_conf(conf):
|
||||||
|
global CONF
|
||||||
|
CONF = conf
|
||||||
|
|
||||||
|
ng_validator = api_validator.ApiValidator(ngt.NODE_GROUP_TEMPLATE_SCHEMA)
|
||||||
|
ct_validator = api_validator.ApiValidator(clt.CLUSTER_TEMPLATE_SCHEMA)
|
||||||
|
|
||||||
|
# Options that we allow to be replaced in a node group template
|
||||||
|
node_group_template_opts = [
|
||||||
|
cfg.StrOpt('image_id',
|
||||||
|
help='Image id field for a node group template.'),
|
||||||
|
|
||||||
|
cfg.StrOpt('flavor_id',
|
||||||
|
help='Flavor id field for a node group template.'),
|
||||||
|
|
||||||
|
cfg.StrOpt('floating_ip_pool',
|
||||||
|
help='Floating ip pool field for a node group template.')
|
||||||
|
]
|
||||||
|
|
||||||
|
# Options that we allow to be replaced in a cluster template
|
||||||
|
cluster_template_opts = [
|
||||||
|
cfg.StrOpt('default_image_id',
|
||||||
|
help='Default image id field for a cluster template.'),
|
||||||
|
|
||||||
|
cfg.StrOpt('neutron_management_network',
|
||||||
|
help='Neutron management network '
|
||||||
|
'field for a cluster template.')]
|
||||||
|
|
||||||
|
all_template_opts = node_group_template_opts + cluster_template_opts
|
||||||
|
|
||||||
|
node_group_template_opt_names = [o.name for o in node_group_template_opts]
|
||||||
|
cluster_template_opt_names = [o.name for o in cluster_template_opts]
|
||||||
|
|
||||||
|
|
||||||
|
# This is a local exception class that is used to exit routines
|
||||||
|
# in cases where error information has already been logged.
|
||||||
|
# It is caught and suppressed everywhere it is used.
|
||||||
|
class Handled(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class Context(object):
|
||||||
|
'''Create a pseudo Context object
|
||||||
|
|
||||||
|
Since this tool does not use the REST interface, we
|
||||||
|
do not have a request from which to build a Context.
|
||||||
|
'''
|
||||||
|
def __init__(self, is_admin=False, tenant_id=None):
|
||||||
|
self.is_admin = is_admin
|
||||||
|
self.tenant_id = tenant_id
|
||||||
|
|
||||||
|
|
||||||
|
def check_usage_of_existing(ctx, ng_templates, cl_templates):
|
||||||
|
'''Determine if any of the specified templates are in use
|
||||||
|
|
||||||
|
This method searches for the specified templates by name and
|
||||||
|
determines whether or not any existing templates are in use
|
||||||
|
by a cluster or cluster template. Returns True if any of
|
||||||
|
the templates are in use.
|
||||||
|
|
||||||
|
:param ng_templates: A list of dictionaries. Each dictionary
|
||||||
|
has a "template" entry that represents
|
||||||
|
a node group template.
|
||||||
|
:param cl_templates: A list of dictionaries. Each dictionary
|
||||||
|
has a "template" entry that represents
|
||||||
|
a cluster template
|
||||||
|
:returns: True if any of the templates are in use, False otherwise
|
||||||
|
'''
|
||||||
|
error = False
|
||||||
|
clusters = conductor.API.cluster_get_all(ctx)
|
||||||
|
|
||||||
|
for ng_info in ng_templates:
|
||||||
|
ng = u.find_node_group_template_by_name(ctx,
|
||||||
|
ng_info["template"]["name"])
|
||||||
|
if ng:
|
||||||
|
cluster_users, template_users = u.check_node_group_template_usage(
|
||||||
|
ng["id"], clusters)
|
||||||
|
|
||||||
|
if cluster_users:
|
||||||
|
LOG.warning("Node group template {name} "
|
||||||
|
"in use by clusters {clusters}".format(
|
||||||
|
name=ng["name"], clusters=cluster_users))
|
||||||
|
if template_users:
|
||||||
|
LOG.warning("Node group template {name} "
|
||||||
|
"in use by cluster templates {templates}".format(
|
||||||
|
name=ng["name"], templates=template_users))
|
||||||
|
|
||||||
|
if cluster_users or template_users:
|
||||||
|
LOG.warning("Update of node group template "
|
||||||
|
"{name} is not allowed".format(name=ng["name"]))
|
||||||
|
error = True
|
||||||
|
|
||||||
|
for cl_info in cl_templates:
|
||||||
|
cl = u.find_cluster_template_by_name(ctx, cl_info["template"]["name"])
|
||||||
|
if cl:
|
||||||
|
cluster_users = u.check_cluster_template_usage(cl["id"], clusters)
|
||||||
|
|
||||||
|
if cluster_users:
|
||||||
|
LOG.warning("Cluster template {name} "
|
||||||
|
"in use by clusters {clusters}".format(
|
||||||
|
name=cl["name"], clusters=cluster_users))
|
||||||
|
|
||||||
|
LOG.warning("Update of cluster template "
|
||||||
|
"{name} is not allowed".format(name=cl["name"]))
|
||||||
|
error = True
|
||||||
|
|
||||||
|
return error
|
||||||
|
|
||||||
|
|
||||||
|
def log_skipping_dir(path, reason=""):
|
||||||
|
if reason:
|
||||||
|
reason = ", " + reason
|
||||||
|
LOG.warning("Skipping processing for {dir}{reason}".format(
|
||||||
|
dir=path, reason=reason))
|
||||||
|
|
||||||
|
|
||||||
|
def check_cluster_templates_valid(ng_templates, cl_templates):
|
||||||
|
# Check that if name references to node group templates
|
||||||
|
# are replaced with a uuid value that the cluster template
|
||||||
|
# passes JSON validation. We don't have the real uuid yet,
|
||||||
|
# but this will allow the validation test.
|
||||||
|
if ng_templates:
|
||||||
|
dummy_uuid = uuid.uuid4()
|
||||||
|
ng_ids = {ng["template"]["name"]: dummy_uuid for ng in ng_templates}
|
||||||
|
else:
|
||||||
|
ng_ids = {}
|
||||||
|
|
||||||
|
for cl in cl_templates:
|
||||||
|
template = copy.deepcopy(cl["template"])
|
||||||
|
u.substitute_ng_ids(template, ng_ids)
|
||||||
|
try:
|
||||||
|
ct_validator.validate(template)
|
||||||
|
except jsonschema.ValidationError as e:
|
||||||
|
LOG.warning("Validation for {path} failed, {reason}".format(
|
||||||
|
path=cl["path"], reason=e))
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def add_config_section(section_name, options):
|
||||||
|
if section_name and hasattr(CONF, section_name):
|
||||||
|
# It's already been added
|
||||||
|
return
|
||||||
|
|
||||||
|
if section_name:
|
||||||
|
group = cfg.OptGroup(name=section_name)
|
||||||
|
CONF.register_group(group)
|
||||||
|
CONF.register_opts(options, group)
|
||||||
|
else:
|
||||||
|
# Add options to the default section
|
||||||
|
CONF.register_opts(options)
|
||||||
|
|
||||||
|
|
||||||
|
def add_config_section_for_template(template):
|
||||||
|
'''Register a config section based on the template values
|
||||||
|
|
||||||
|
Check to see if the configuration files contain a section
|
||||||
|
that corresponds to the template. If an appropriate section
|
||||||
|
can be found, register options for the template so that the
|
||||||
|
config values can be read and applied to the template via
|
||||||
|
substitution (oslo supports registering groups and options
|
||||||
|
at any time, before or after the config files are parsed).
|
||||||
|
|
||||||
|
Corresponding section names may be of the following forms:
|
||||||
|
|
||||||
|
<template_name>, example "hdp-2.0.6-master"
|
||||||
|
This is useful when a template naming convention is being used,
|
||||||
|
so that the template name is already unambiguous
|
||||||
|
|
||||||
|
<plugin_name>_<hadoop_version>_<template_name>, example "hdp_2.0.6_master"
|
||||||
|
This can be used if there is a name collision between templates
|
||||||
|
|
||||||
|
<plugin_name>_<hadoop_version>, example "hdp_2.0.6"
|
||||||
|
<plugin_name>, example "hdp"
|
||||||
|
DEFAULT
|
||||||
|
|
||||||
|
Sections are tried in the order given above.
|
||||||
|
|
||||||
|
Since the first two section naming forms refer to a specific
|
||||||
|
template by name, options are added based on template type.
|
||||||
|
|
||||||
|
However, the other section naming forms may map to node group templates
|
||||||
|
or cluster templates, so options for both are added.
|
||||||
|
'''
|
||||||
|
sections = list(CONF.list_all_sections())
|
||||||
|
|
||||||
|
unique_name = "{name}".format(**template)
|
||||||
|
fullname = "{plugin_name}_{hadoop_version}_{name}".format(**template)
|
||||||
|
plugin_version = "{plugin_name}_{hadoop_version}".format(**template)
|
||||||
|
plugin = "{plugin_name}".format(**template)
|
||||||
|
|
||||||
|
section_name = None
|
||||||
|
if unique_name in sections:
|
||||||
|
section_name = unique_name
|
||||||
|
elif fullname in sections:
|
||||||
|
section_name = fullname
|
||||||
|
|
||||||
|
if section_name:
|
||||||
|
if u.is_node_group(template):
|
||||||
|
opts = node_group_template_opts
|
||||||
|
else:
|
||||||
|
opts = cluster_template_opts
|
||||||
|
else:
|
||||||
|
if plugin_version in sections:
|
||||||
|
section_name = plugin_version
|
||||||
|
elif plugin in sections:
|
||||||
|
section_name = plugin
|
||||||
|
opts = all_template_opts
|
||||||
|
|
||||||
|
add_config_section(section_name, opts)
|
||||||
|
return section_name
|
||||||
|
|
||||||
|
|
||||||
|
def substitute_config_values(configs, template, path):
|
||||||
|
if u.is_node_group(template):
|
||||||
|
opt_names = node_group_template_opt_names
|
||||||
|
else:
|
||||||
|
opt_names = cluster_template_opt_names
|
||||||
|
|
||||||
|
for opt, value in six.iteritems(configs):
|
||||||
|
if opt in opt_names and opt in template:
|
||||||
|
if value is None:
|
||||||
|
# TODO(tmckay): someday if we support 'null' in JSON
|
||||||
|
# we should replace this value with None. json.load
|
||||||
|
# will replace 'null' with None, and sqlalchemy will
|
||||||
|
# accept None as a value for a nullable field.
|
||||||
|
del template[opt]
|
||||||
|
LOG.debug("No replacement value specified for {opt} in "
|
||||||
|
"{path}, removing".format(opt=opt, path=path))
|
||||||
|
else:
|
||||||
|
# Use args to allow for keyword arguments to format
|
||||||
|
args = {opt: value}
|
||||||
|
template[opt] = template[opt].format(**args)
|
||||||
|
|
||||||
|
|
||||||
|
def get_configs(section):
|
||||||
|
if section is None:
|
||||||
|
return dict(CONF)
|
||||||
|
return dict(CONF[section])
|
||||||
|
|
||||||
|
|
||||||
|
def get_plugin_name():
|
||||||
|
if CONF.command.name == "update" and (
|
||||||
|
not CONF.command.plugin_name and (
|
||||||
|
hasattr(CONF, "plugins") and CONF.plugins)):
|
||||||
|
return CONF.plugins
|
||||||
|
return CONF.command.plugin_name
|
||||||
|
|
||||||
|
|
||||||
|
def process_files(dirname, files):
|
||||||
|
|
||||||
|
node_groups = []
|
||||||
|
clusters = []
|
||||||
|
plugin_name = get_plugin_name()
|
||||||
|
|
||||||
|
try:
|
||||||
|
for fname in files:
|
||||||
|
if os.path.splitext(fname)[1] == ".json":
|
||||||
|
fpath = os.path.join(dirname, fname)
|
||||||
|
with open(fpath, 'r') as fp:
|
||||||
|
try:
|
||||||
|
template = json.load(fp)
|
||||||
|
except ValueError as e:
|
||||||
|
LOG.warning("Error processing {path}, {reason}".format(
|
||||||
|
path=fpath, reason=e))
|
||||||
|
raise Handled("error processing files")
|
||||||
|
|
||||||
|
# If this file doesn't contain basic fields, skip it.
|
||||||
|
# If we are filtering on plugin and version make
|
||||||
|
# sure the file is one that we want
|
||||||
|
if not u.check_basic_fields(template) or (
|
||||||
|
not u.check_plugin_name_and_version(
|
||||||
|
template,
|
||||||
|
plugin_name,
|
||||||
|
CONF.command.plugin_version)):
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Look through the sections in CONF and register
|
||||||
|
# options for this template if we find a section
|
||||||
|
# related to the template (ie, plugin, version, name)
|
||||||
|
section = add_config_section_for_template(template)
|
||||||
|
LOG.debug("Using config section {section} "
|
||||||
|
"for {path}".format(section=section, path=fpath))
|
||||||
|
|
||||||
|
# Attempt to resolve substitutions using the config section
|
||||||
|
substitute_config_values(get_configs(section),
|
||||||
|
template, fpath)
|
||||||
|
|
||||||
|
file_entry = {'template': template,
|
||||||
|
'path': fpath}
|
||||||
|
|
||||||
|
if u.is_node_group(template):
|
||||||
|
# JSON validator
|
||||||
|
try:
|
||||||
|
ng_validator.validate(template)
|
||||||
|
except jsonschema.ValidationError as e:
|
||||||
|
LOG.warning("Validation for {path} failed, "
|
||||||
|
"{reason}".format(path=fpath,
|
||||||
|
reason=e))
|
||||||
|
raise Handled(
|
||||||
|
"node group template validation failed")
|
||||||
|
node_groups.append(file_entry)
|
||||||
|
LOG.debug("Added {path} to node group "
|
||||||
|
"template files".format(path=fpath))
|
||||||
|
else:
|
||||||
|
clusters.append(file_entry)
|
||||||
|
LOG.debug("Added {path} to cluster template "
|
||||||
|
"files".format(path=fpath))
|
||||||
|
|
||||||
|
except Handled as e:
|
||||||
|
log_skipping_dir(dirname, e.message)
|
||||||
|
node_groups = []
|
||||||
|
clusters = []
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
log_skipping_dir(dirname,
|
||||||
|
"unhandled exception, {reason}".format(reason=e))
|
||||||
|
node_groups = []
|
||||||
|
clusters = []
|
||||||
|
|
||||||
|
return node_groups, clusters
|
||||||
|
|
||||||
|
|
||||||
|
def delete_node_group_template(ctx, template, rollback=False):
|
||||||
|
rollback_msg = " on rollback" if rollback else ""
|
||||||
|
|
||||||
|
# If we are not deleting something that we just created,
|
||||||
|
# do usage checks to ensure that the template is not in
|
||||||
|
# use by a cluster or a cluster template
|
||||||
|
if not rollback:
|
||||||
|
clusters = conductor.API.cluster_get_all(ctx)
|
||||||
|
cluster_templates = conductor.API.cluster_template_get_all(ctx)
|
||||||
|
cluster_users, template_users = u.check_node_group_template_usage(
|
||||||
|
template["id"], clusters, cluster_templates)
|
||||||
|
|
||||||
|
if cluster_users:
|
||||||
|
LOG.warning("Node group template {info} "
|
||||||
|
"in use by clusters {clusters}".format(
|
||||||
|
info=u.name_and_id(template),
|
||||||
|
clusters=cluster_users))
|
||||||
|
if template_users:
|
||||||
|
LOG.warning("Node group template {info} "
|
||||||
|
"in use by cluster templates {templates}".format(
|
||||||
|
info=u.name_and_id(template),
|
||||||
|
templates=template_users))
|
||||||
|
|
||||||
|
if cluster_users or template_users:
|
||||||
|
LOG.warning("Deletion of node group template "
|
||||||
|
"{info} failed".format(info=u.name_and_id(template)))
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
conductor.API.node_group_template_destroy(ctx, template["id"],
|
||||||
|
ignore_default=True)
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Deletion of node group template {info} failed{rollback}"
|
||||||
|
", {reason}".format(info=u.name_and_id(template),
|
||||||
|
reason=e,
|
||||||
|
rollback=rollback_msg))
|
||||||
|
else:
|
||||||
|
LOG.info("Deleted node group template {info}{rollback}".format(
|
||||||
|
info=u.name_and_id(template), rollback=rollback_msg))
|
||||||
|
|
||||||
|
|
||||||
|
def reverse_node_group_template_creates(ctx, templates):
|
||||||
|
for template in templates:
|
||||||
|
delete_node_group_template(ctx, template, rollback=True)
|
||||||
|
|
||||||
|
|
||||||
|
def reverse_node_group_template_updates(ctx, update_info):
|
||||||
|
for template, values in update_info:
|
||||||
|
# values are the original values that we overwrote in the update
|
||||||
|
try:
|
||||||
|
conductor.API.node_group_template_update(ctx,
|
||||||
|
template["id"], values,
|
||||||
|
ignore_default=True)
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Rollback of update for node group "
|
||||||
|
"template {info} failed, {reason}".format(
|
||||||
|
info=u.name_and_id(template),
|
||||||
|
reason=e))
|
||||||
|
else:
|
||||||
|
LOG.info("Rolled back update for "
|
||||||
|
"node group template {info}".format(
|
||||||
|
info=u.name_and_id(template)))
|
||||||
|
|
||||||
|
|
||||||
|
def add_node_group_templates(ctx, node_groups):
|
||||||
|
|
||||||
|
error = False
|
||||||
|
ng_info = {"ids": {},
|
||||||
|
"created": [],
|
||||||
|
"updated": []}
|
||||||
|
|
||||||
|
def do_reversals(ng_info):
|
||||||
|
reverse_node_group_template_updates(ctx, ng_info["updated"])
|
||||||
|
reverse_node_group_template_creates(ctx, ng_info["created"])
|
||||||
|
return {}, True
|
||||||
|
|
||||||
|
try:
|
||||||
|
for ng in node_groups:
|
||||||
|
template = ng['template']
|
||||||
|
current = u.find_node_group_template_by_name(ctx, template['name'])
|
||||||
|
if current:
|
||||||
|
|
||||||
|
# Track what we see in the current template that is different
|
||||||
|
# from our update values. Save it for possible rollback.
|
||||||
|
# Note, this is not perfect because it does not recurse through
|
||||||
|
# nested structures to get an exact diff, but it ensures that
|
||||||
|
# we track only fields that are valid in the JSON schema
|
||||||
|
updated_fields = u.value_diff(current.to_dict(), template)
|
||||||
|
|
||||||
|
# Always attempt to update. Since the template value is a
|
||||||
|
# combination of JSON and config values, there is no useful
|
||||||
|
# timestamp we can use to skip an update.
|
||||||
|
# If sqlalchemy determines no change in fields, it will not
|
||||||
|
# mark it as updated.
|
||||||
|
try:
|
||||||
|
template = conductor.API.node_group_template_update(
|
||||||
|
ctx, current['id'], template, ignore_default=True)
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Update of node group template {info} "
|
||||||
|
"failed, {reason}".format(
|
||||||
|
info=u.name_and_id(current),
|
||||||
|
reason=e))
|
||||||
|
raise Handled()
|
||||||
|
|
||||||
|
if template['updated_at'] != current['updated_at']:
|
||||||
|
ng_info["updated"].append((template, updated_fields))
|
||||||
|
LOG.info("Updated node group template {info} "
|
||||||
|
"from {path}".format(
|
||||||
|
info=u.name_and_id(template),
|
||||||
|
path=ng["path"]))
|
||||||
|
else:
|
||||||
|
LOG.debug("No change to node group template {info} "
|
||||||
|
"from {path}".format(
|
||||||
|
info=u.name_and_id(current),
|
||||||
|
path=ng['path']))
|
||||||
|
else:
|
||||||
|
template['is_default'] = True
|
||||||
|
try:
|
||||||
|
template = conductor.API.node_group_template_create(
|
||||||
|
ctx, template)
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Creation of node group template "
|
||||||
|
"from {path} failed, {reason}".format(
|
||||||
|
path=ng['path'], reason=e))
|
||||||
|
raise Handled()
|
||||||
|
|
||||||
|
ng_info["created"].append(template)
|
||||||
|
LOG.info("Created node group template {info} "
|
||||||
|
"from {path}".format(info=u.name_and_id(template),
|
||||||
|
path=ng["path"]))
|
||||||
|
|
||||||
|
# For the purposes of substituion we need a dict of id by name
|
||||||
|
ng_info["ids"][template['name']] = template['id']
|
||||||
|
|
||||||
|
except Handled:
|
||||||
|
ng_info, error = do_reversals(ng_info)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Unhandled exception while processing "
|
||||||
|
"node group templates, {reason}".format(reason=e))
|
||||||
|
ng_info, error = do_reversals(ng_info)
|
||||||
|
|
||||||
|
return ng_info, error
|
||||||
|
|
||||||
|
|
||||||
|
def delete_cluster_template(ctx, template, rollback=False):
|
||||||
|
rollback_msg = " on rollback" if rollback else ""
|
||||||
|
|
||||||
|
# If we are not deleting something that we just created,
|
||||||
|
# do usage checks to ensure that the template is not in
|
||||||
|
# use by a cluster
|
||||||
|
if not rollback:
|
||||||
|
clusters = conductor.API.cluster_get_all(ctx)
|
||||||
|
cluster_users = u.check_cluster_template_usage(template["id"],
|
||||||
|
clusters)
|
||||||
|
|
||||||
|
if cluster_users:
|
||||||
|
LOG.warning("Cluster template {info} "
|
||||||
|
"in use by clusters {clusters}".format(
|
||||||
|
info=u.name_and_id(template),
|
||||||
|
clusters=cluster_users))
|
||||||
|
|
||||||
|
LOG.warning("Deletion of cluster template "
|
||||||
|
"{info} failed".format(info=u.name_and_id(template)))
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
conductor.API.cluster_template_destroy(ctx, template["id"],
|
||||||
|
ignore_default=True)
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Deletion of cluster template {info} failed{rollback}"
|
||||||
|
", {reason}".format(info=u.name_and_id(template),
|
||||||
|
reason=e,
|
||||||
|
rollback=rollback_msg))
|
||||||
|
else:
|
||||||
|
LOG.info("Deleted cluster template {info}{rollback}".format(
|
||||||
|
info=u.name_and_id(template), rollback=rollback_msg))
|
||||||
|
|
||||||
|
|
||||||
|
def reverse_cluster_template_creates(ctx, templates):
|
||||||
|
for template in templates:
|
||||||
|
delete_cluster_template(ctx, template, rollback=True)
|
||||||
|
|
||||||
|
|
||||||
|
def reverse_cluster_template_updates(ctx, update_info):
|
||||||
|
for template, values in update_info:
|
||||||
|
# values are the original values that we overwrote in the update
|
||||||
|
try:
|
||||||
|
conductor.API.cluster_template_update(ctx,
|
||||||
|
template["id"], values,
|
||||||
|
ignore_default=True)
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Rollback of update for cluster "
|
||||||
|
"template {info} failed, {reason}".format(
|
||||||
|
info=u.name_and_id(template),
|
||||||
|
reason=e))
|
||||||
|
else:
|
||||||
|
LOG.info("Rolled back update for "
|
||||||
|
"cluster template {info}".format(
|
||||||
|
info=u.name_and_id(template)))
|
||||||
|
|
||||||
|
|
||||||
|
def add_cluster_templates(ctx, clusters, ng_dict):
|
||||||
|
'''Add cluster templates to the database.
|
||||||
|
|
||||||
|
The value of any node_group_template_id fields in cluster
|
||||||
|
templates which reference a node group template in ng_dict by name
|
||||||
|
will be changed to the id of the node group template.
|
||||||
|
|
||||||
|
If there is an error in creating or updating a template, any templates
|
||||||
|
that have already been created will be delete and any updates will
|
||||||
|
be reversed.
|
||||||
|
|
||||||
|
:param clusters: a list of dictionaries. Each dictionary
|
||||||
|
has a "template" entry holding the cluster template
|
||||||
|
and a "path" entry holding the path of the file
|
||||||
|
from which the template was read.
|
||||||
|
:param ng_dict: a dictionary of node group template ids keyed
|
||||||
|
by node group template names
|
||||||
|
'''
|
||||||
|
|
||||||
|
error = False
|
||||||
|
created = []
|
||||||
|
updated = []
|
||||||
|
|
||||||
|
def do_reversals(created, updated):
|
||||||
|
reverse_cluster_template_updates(ctx, updated)
|
||||||
|
reverse_cluster_template_creates(ctx, created)
|
||||||
|
return True
|
||||||
|
|
||||||
|
try:
|
||||||
|
for cl in clusters:
|
||||||
|
template = cl['template']
|
||||||
|
|
||||||
|
# Fix up node_group_template_id fields
|
||||||
|
u.substitute_ng_ids(template, ng_dict)
|
||||||
|
|
||||||
|
# Name + tenant_id is unique, so search by name
|
||||||
|
current = u.find_cluster_template_by_name(ctx, template['name'])
|
||||||
|
if current:
|
||||||
|
|
||||||
|
# Track what we see in the current template that is different
|
||||||
|
# from our update values. Save it for possible rollback.
|
||||||
|
# Note, this is not perfect because it does not recurse through
|
||||||
|
# nested structures to get an exact diff, but it ensures that
|
||||||
|
# we track only fields that are valid in the JSON schema
|
||||||
|
updated_fields = u.value_diff(current.to_dict(), template)
|
||||||
|
|
||||||
|
# Always attempt to update. Since the template value is a
|
||||||
|
# combination of JSON and config values, there is no useful
|
||||||
|
# timestamp we can use to skip an update.
|
||||||
|
# If sqlalchemy determines no change in fields, it will not
|
||||||
|
# mark it as updated.
|
||||||
|
|
||||||
|
# TODO(tmckay): why when I change the count in an
|
||||||
|
# entry in node_groups does it not count as an update?
|
||||||
|
# Probably a bug
|
||||||
|
try:
|
||||||
|
template = conductor.API.cluster_template_update(
|
||||||
|
ctx, current['id'], template, ignore_default=True)
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Update of cluster template {info} "
|
||||||
|
"failed, {reason}".format(
|
||||||
|
info=u.name_and_id(current), reason=e))
|
||||||
|
raise Handled()
|
||||||
|
|
||||||
|
if template['updated_at'] != current['updated_at']:
|
||||||
|
updated.append((template, updated_fields))
|
||||||
|
LOG.info("Updated cluster template {info} "
|
||||||
|
"from {path}".format(
|
||||||
|
info=u.name_and_id(template),
|
||||||
|
path=cl['path']))
|
||||||
|
else:
|
||||||
|
LOG.debug("No change to cluster template {info} "
|
||||||
|
"from {path}".format(info=u.name_and_id(current),
|
||||||
|
path=cl["path"]))
|
||||||
|
else:
|
||||||
|
template["is_default"] = True
|
||||||
|
try:
|
||||||
|
template = conductor.API.cluster_template_create(ctx,
|
||||||
|
template)
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Creation of cluster template "
|
||||||
|
"from {path} failed, {reason}".format(
|
||||||
|
path=cl['path'],
|
||||||
|
reason=e))
|
||||||
|
raise Handled()
|
||||||
|
|
||||||
|
created.append(template)
|
||||||
|
LOG.info("Created cluster template {info} from {path}".format(
|
||||||
|
info=u.name_and_id(template), path=cl['path']))
|
||||||
|
|
||||||
|
except Handled:
|
||||||
|
error = do_reversals(created, updated)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
LOG.warning("Unhandled exception while processing "
|
||||||
|
"cluster templates, {reason}".format(reason=e))
|
||||||
|
error = do_reversals(created, updated)
|
||||||
|
|
||||||
|
return error
|
||||||
|
|
||||||
|
|
||||||
|
def do_update():
|
||||||
|
'''Create or update default templates for the specified tenant.
|
||||||
|
|
||||||
|
Looks for '.json' files beginning at the specified starting
|
||||||
|
directory (--directory CLI option) and descending
|
||||||
|
through subdirectories by default.
|
||||||
|
|
||||||
|
The .json files represent cluster templates or node group
|
||||||
|
templates. All '.json' files at the same directory level are treated
|
||||||
|
as a set. Cluster templates may reference node group templates
|
||||||
|
in the same set.
|
||||||
|
|
||||||
|
If an error occurs in processing a set, skip it and continue.
|
||||||
|
|
||||||
|
If creation of cluster templates fails, any node group templates
|
||||||
|
in the set that were already created will be deleted.
|
||||||
|
'''
|
||||||
|
|
||||||
|
ctx = Context(tenant_id=CONF.command.tenant_id)
|
||||||
|
start_dir = os.path.abspath(CONF.command.directory)
|
||||||
|
|
||||||
|
for root, dirs, files in os.walk(start_dir):
|
||||||
|
|
||||||
|
# Find all the template files and identify them as node_group
|
||||||
|
# or cluster templates. If there is an exception in
|
||||||
|
# processing the set, return empty lists.
|
||||||
|
node_groups, clusters = process_files(root, files)
|
||||||
|
|
||||||
|
# Now that we know what the valid node group templates are,
|
||||||
|
# we can validate the cluster templates as well.
|
||||||
|
if check_cluster_templates_valid(node_groups, clusters):
|
||||||
|
log_skipping_dir(root, "error processing cluster templates")
|
||||||
|
|
||||||
|
# If there are existing default templates that match the names
|
||||||
|
# in the template files, do usage checks here to detect update
|
||||||
|
# failures early (we can't update a template in use)
|
||||||
|
elif check_usage_of_existing(ctx, node_groups, clusters):
|
||||||
|
log_skipping_dir(root, "templates in use")
|
||||||
|
else:
|
||||||
|
ng_info, error = add_node_group_templates(ctx, node_groups)
|
||||||
|
if error:
|
||||||
|
log_skipping_dir(root, "error processing node group templates")
|
||||||
|
|
||||||
|
elif add_cluster_templates(ctx, clusters, ng_info["ids"]):
|
||||||
|
log_skipping_dir(root, "error processing cluster templates")
|
||||||
|
|
||||||
|
# Cluster templates failed so remove the node group templates
|
||||||
|
reverse_node_group_template_updates(ctx, ng_info["updated"])
|
||||||
|
reverse_node_group_template_creates(ctx, ng_info["created"])
|
||||||
|
|
||||||
|
if CONF.command.norecurse:
|
||||||
|
break
|
||||||
|
|
||||||
|
|
||||||
|
def do_delete():
|
||||||
|
'''Delete default templates in the specified tenant
|
||||||
|
|
||||||
|
Deletion uses the --plugin-name and --plugin-version options
|
||||||
|
as filters.
|
||||||
|
|
||||||
|
Only templates with 'is_default=True' will be deleted.
|
||||||
|
'''
|
||||||
|
|
||||||
|
ctx = Context(tenant_id=CONF.command.tenant_id)
|
||||||
|
|
||||||
|
for plugin in get_plugin_name():
|
||||||
|
|
||||||
|
kwargs = {'is_default': True}
|
||||||
|
kwargs['plugin_name'] = plugin
|
||||||
|
|
||||||
|
# Delete cluster templates first for the sake of usage checks
|
||||||
|
lst = conductor.API.cluster_template_get_all(ctx, **kwargs)
|
||||||
|
for l in lst:
|
||||||
|
if not u.check_plugin_version(l, CONF.command.plugin_version):
|
||||||
|
continue
|
||||||
|
delete_cluster_template(ctx, l)
|
||||||
|
|
||||||
|
lst = conductor.API.node_group_template_get_all(ctx, **kwargs)
|
||||||
|
for l in lst:
|
||||||
|
if not u.check_plugin_version(l, CONF.command.plugin_version):
|
||||||
|
continue
|
||||||
|
delete_node_group_template(ctx, l)
|
||||||
|
|
||||||
|
|
||||||
|
def do_node_group_template_delete():
|
||||||
|
ctx = Context(tenant_id=CONF.command.tenant_id)
|
||||||
|
|
||||||
|
t = u.find_node_group_template_by_name(ctx, CONF.command.template_name)
|
||||||
|
if t:
|
||||||
|
delete_node_group_template(ctx, t)
|
||||||
|
else:
|
||||||
|
LOG.warning("Deletion of node group template {name} failed, "
|
||||||
|
"no such template".format(name=CONF.command.template_name))
|
||||||
|
|
||||||
|
|
||||||
|
def do_node_group_template_delete_by_id():
|
||||||
|
ctx = Context(is_admin=True)
|
||||||
|
|
||||||
|
# Make sure it's a default
|
||||||
|
t = conductor.API.node_group_template_get(ctx, CONF.command.id)
|
||||||
|
if t:
|
||||||
|
if t["is_default"]:
|
||||||
|
delete_node_group_template(ctx, t)
|
||||||
|
else:
|
||||||
|
LOG.warning("Deletion of node group template {info} skipped, "
|
||||||
|
"not a default template".format(info=u.name_and_id(t)))
|
||||||
|
else:
|
||||||
|
LOG.warning("Deletion of node group template {id} failed, "
|
||||||
|
"no such template".format(id=CONF.command.id))
|
||||||
|
|
||||||
|
|
||||||
|
def do_cluster_template_delete():
|
||||||
|
ctx = Context(tenant_id=CONF.command.tenant_id)
|
||||||
|
|
||||||
|
t = u.find_cluster_template_by_name(ctx, CONF.command.template_name)
|
||||||
|
if t:
|
||||||
|
delete_cluster_template(ctx, t)
|
||||||
|
else:
|
||||||
|
LOG.warning("Deletion of cluster template {name} failed, "
|
||||||
|
"no such template".format(name=CONF.command.template_name))
|
||||||
|
|
||||||
|
|
||||||
|
def do_cluster_template_delete_by_id():
|
||||||
|
ctx = Context(is_admin=True)
|
||||||
|
|
||||||
|
# Make sure it's a default
|
||||||
|
t = conductor.API.cluster_template_get(ctx, CONF.command.id)
|
||||||
|
if t:
|
||||||
|
if t["is_default"]:
|
||||||
|
delete_cluster_template(ctx, t)
|
||||||
|
else:
|
||||||
|
LOG.warning("Deletion of cluster template {info} skipped, "
|
||||||
|
"not a default template".format(info=u.name_and_id(t)))
|
||||||
|
else:
|
||||||
|
LOG.warning("Deletion of cluster template {id} failed, "
|
||||||
|
"no such template".format(id=CONF.command.id))
|
|
@ -0,0 +1,208 @@
|
||||||
|
# Copyright 2015 Red Hat, Inc.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
# implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
from __future__ import print_function
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from oslo_config import cfg
|
||||||
|
from oslo_log import log
|
||||||
|
import pkg_resources as pkg
|
||||||
|
|
||||||
|
from sahara.db.templates import api
|
||||||
|
from sahara import version
|
||||||
|
|
||||||
|
LOG = log.getLogger(__name__)
|
||||||
|
|
||||||
|
CONF = cfg.CONF
|
||||||
|
|
||||||
|
|
||||||
|
def extra_option_checks():
|
||||||
|
|
||||||
|
if not CONF.database.connection:
|
||||||
|
print("No database connection string was specified in configuration",
|
||||||
|
file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
if CONF.command.name in ['update', 'delete']:
|
||||||
|
if CONF.command.plugin_version and not CONF.command.plugin_name:
|
||||||
|
print("The --plugin-version option is not valid "
|
||||||
|
"without --plugin-name", file=sys.stderr)
|
||||||
|
sys.exit(-1)
|
||||||
|
|
||||||
|
if CONF.command.name == "update":
|
||||||
|
# Options handling probably needs some refactoring in the future.
|
||||||
|
# For now, though, we touch the conductor which ultimately touches
|
||||||
|
# the plugins.base. Use the "plugins" option there as a default
|
||||||
|
# list of plugins to process, since those are the plugins that
|
||||||
|
# will be loaded by Sahara
|
||||||
|
if not CONF.command.plugin_name:
|
||||||
|
if "plugins" in CONF and CONF.plugins:
|
||||||
|
LOG.info("Using plugin list {plugins} from config".format(
|
||||||
|
plugins=CONF.plugins))
|
||||||
|
else:
|
||||||
|
print("No plugins specified with --plugin-name "
|
||||||
|
"or config", file=sys.stderr)
|
||||||
|
sys.exit(-1)
|
||||||
|
|
||||||
|
|
||||||
|
def add_command_parsers(subparsers):
|
||||||
|
# Note, there is no 'list' command here because the client
|
||||||
|
# or REST can be used for list operations. Default templates
|
||||||
|
# will display, and templates will show the 'is_default' field.
|
||||||
|
|
||||||
|
def add_id(parser):
|
||||||
|
parser.add_argument('--id', required=True,
|
||||||
|
help='The id of the default '
|
||||||
|
'template to delete')
|
||||||
|
|
||||||
|
def add_tenant_id(parser):
|
||||||
|
parser.add_argument('-t', '--tenant_id', required=True,
|
||||||
|
help='Tenant ID for database operations.')
|
||||||
|
|
||||||
|
def add_name_and_tenant_id(parser):
|
||||||
|
parser.add_argument('--name', dest='template_name', required=True,
|
||||||
|
help='Name of the default template')
|
||||||
|
add_tenant_id(parser)
|
||||||
|
|
||||||
|
def add_plugin_name_and_version(parser, require_plugin_name=False):
|
||||||
|
|
||||||
|
plugin_name_help = ('Only process templates containing '
|
||||||
|
'a "plugin_name" field matching '
|
||||||
|
'one of these values.')
|
||||||
|
|
||||||
|
if not require_plugin_name:
|
||||||
|
extra = (' The default list of plugin names '
|
||||||
|
'is taken from the "plugins" parameter in '
|
||||||
|
'the [DEFAULT] config section.')
|
||||||
|
plugin_name_help += extra
|
||||||
|
|
||||||
|
parser.add_argument('-p', '--plugin-name', nargs="*",
|
||||||
|
required=require_plugin_name,
|
||||||
|
help=plugin_name_help)
|
||||||
|
|
||||||
|
parser.add_argument('-pv', '--plugin-version', nargs="*",
|
||||||
|
help='Only process templates containing a '
|
||||||
|
'"hadoop_version" field matching one of '
|
||||||
|
'these values. This option is '
|
||||||
|
'only valid if --plugin-name is specified '
|
||||||
|
'as well. A version specified '
|
||||||
|
'here may optionally be prefixed with a '
|
||||||
|
'plugin name and a dot, for exmaple '
|
||||||
|
'"vanilla.1.2.1". Dotted versions only '
|
||||||
|
'apply to the plugin named in the '
|
||||||
|
'prefix. Versions without a prefix apply to '
|
||||||
|
'all plugins.')
|
||||||
|
|
||||||
|
fname = pkg.resource_filename(version.version_info.package,
|
||||||
|
"plugins/default_templates")
|
||||||
|
# update command
|
||||||
|
parser = subparsers.add_parser('update',
|
||||||
|
help='Update the default template set')
|
||||||
|
parser.add_argument('-d', '--directory',
|
||||||
|
default=fname,
|
||||||
|
help='Template directory. Default is %s' % fname)
|
||||||
|
parser.add_argument('-n', '--norecurse', action='store_true',
|
||||||
|
help='Do not descend into subdirectories')
|
||||||
|
|
||||||
|
add_plugin_name_and_version(parser)
|
||||||
|
add_tenant_id(parser)
|
||||||
|
parser.set_defaults(func=api.do_update)
|
||||||
|
|
||||||
|
# delete command
|
||||||
|
parser = subparsers.add_parser('delete',
|
||||||
|
help='Delete default templates '
|
||||||
|
'by plugin and version')
|
||||||
|
add_plugin_name_and_version(parser, require_plugin_name=True)
|
||||||
|
add_tenant_id(parser)
|
||||||
|
parser.set_defaults(func=api.do_delete)
|
||||||
|
|
||||||
|
# node-group-template-delete command
|
||||||
|
parser = subparsers.add_parser('node-group-template-delete',
|
||||||
|
help='Delete a default '
|
||||||
|
'node group template by name')
|
||||||
|
add_name_and_tenant_id(parser)
|
||||||
|
parser.set_defaults(func=api.do_node_group_template_delete)
|
||||||
|
|
||||||
|
# cluster-template-delete command
|
||||||
|
parser = subparsers.add_parser('cluster-template-delete',
|
||||||
|
help='Delete a default '
|
||||||
|
'cluster template by name')
|
||||||
|
add_name_and_tenant_id(parser)
|
||||||
|
parser.set_defaults(func=api.do_cluster_template_delete)
|
||||||
|
|
||||||
|
# node-group-template-delete-id command
|
||||||
|
parser = subparsers.add_parser('node-group-template-delete-id',
|
||||||
|
help='Delete a default '
|
||||||
|
'node group template by id')
|
||||||
|
add_id(parser)
|
||||||
|
parser.set_defaults(func=api.do_node_group_template_delete_by_id)
|
||||||
|
|
||||||
|
# cluster-template-delete-id command
|
||||||
|
parser = subparsers.add_parser('cluster-template-delete-id',
|
||||||
|
help='Delete a default '
|
||||||
|
'cluster template by id')
|
||||||
|
add_id(parser)
|
||||||
|
parser.set_defaults(func=api.do_cluster_template_delete_by_id)
|
||||||
|
|
||||||
|
|
||||||
|
command_opt = cfg.SubCommandOpt('command',
|
||||||
|
title='Command',
|
||||||
|
help='Available commands',
|
||||||
|
handler=add_command_parsers)
|
||||||
|
CONF.register_cli_opt(command_opt)
|
||||||
|
|
||||||
|
|
||||||
|
def unregister_extra_cli_opt(name):
|
||||||
|
try:
|
||||||
|
for cli in CONF._cli_opts:
|
||||||
|
if cli['opt'].name == name:
|
||||||
|
CONF.unregister_opt(cli['opt'])
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# Remove a few extra CLI opts that we picked up via imports
|
||||||
|
# Do this early so that they do not appear in the help
|
||||||
|
for extra_opt in ["log-exchange", "host", "port"]:
|
||||||
|
unregister_extra_cli_opt(extra_opt)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# TODO(tmckay): Work on restricting the options
|
||||||
|
# pulled in by imports which show up in the help.
|
||||||
|
# If we find a nice way to do this the calls to
|
||||||
|
# unregister_extra_cli_opt() can be removed
|
||||||
|
CONF(project='sahara')
|
||||||
|
|
||||||
|
# For some reason, this is necessary to clear cached values
|
||||||
|
# and re-read configs. For instance, if this is not done
|
||||||
|
# here the 'plugins' value will not reflect the value from
|
||||||
|
# the config file on the command line
|
||||||
|
CONF.reload_config_files()
|
||||||
|
log.setup(CONF, "sahara")
|
||||||
|
|
||||||
|
# If we have to enforce extra option checks, like one option
|
||||||
|
# requires another, do it here
|
||||||
|
extra_option_checks()
|
||||||
|
|
||||||
|
# Since this may be scripted, record the command in the log
|
||||||
|
# so a user can know exactly what was done
|
||||||
|
LOG.info("Command: {command}".format(command=' '.join(sys.argv)))
|
||||||
|
|
||||||
|
api.set_logger(LOG)
|
||||||
|
api.set_conf(CONF)
|
||||||
|
|
||||||
|
CONF.command.func()
|
||||||
|
|
||||||
|
LOG.info("Finished {command}".format(command=CONF.command.name))
|
|
@ -0,0 +1,190 @@
|
||||||
|
# Copyright 2015 Red Hat, Inc.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
# implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
import copy
|
||||||
|
|
||||||
|
import six
|
||||||
|
|
||||||
|
from sahara import conductor
|
||||||
|
|
||||||
|
|
||||||
|
def name_and_id(template):
|
||||||
|
return "{name} ({id})".format(name=template["name"],
|
||||||
|
id=template["id"])
|
||||||
|
|
||||||
|
|
||||||
|
def is_node_group(template):
|
||||||
|
# Node group templates and cluster templates have
|
||||||
|
# different required fields in validation and neither
|
||||||
|
# allows additional fields. So, the presence of
|
||||||
|
# node_processes or flavor_id should suffice to
|
||||||
|
# identify a node group template. Check for both
|
||||||
|
# to be nice, in case someone made a typo.
|
||||||
|
return 'node_processes' in template or 'flavor_id' in template
|
||||||
|
|
||||||
|
|
||||||
|
def substitute_ng_ids(cl, ng_dict):
|
||||||
|
'''Substitute node group template ids for node group template names
|
||||||
|
|
||||||
|
If the cluster template contains node group elements with
|
||||||
|
node_group_template_id fields that reference node group templates
|
||||||
|
by name, substitute the node group template id for the name.
|
||||||
|
The name reference is expected to be a string containing a format
|
||||||
|
specifier of the form "{name}", for example "{master}"
|
||||||
|
|
||||||
|
:param cl: a cluster template
|
||||||
|
:param ng_dict: a dictionary of node group template ids keyed by
|
||||||
|
node group template names
|
||||||
|
'''
|
||||||
|
for ng in cl["node_groups"]:
|
||||||
|
if "node_group_template_id" in ng:
|
||||||
|
val = ng["node_group_template_id"].format(**ng_dict)
|
||||||
|
ng["node_group_template_id"] = val
|
||||||
|
|
||||||
|
|
||||||
|
def check_basic_fields(template):
|
||||||
|
return "plugin_name" in template and (
|
||||||
|
"hadoop_version" in template and (
|
||||||
|
"name" in template))
|
||||||
|
|
||||||
|
|
||||||
|
def check_plugin_version(template, plugin_versions):
|
||||||
|
'''Check that the template matches the plugin versions list
|
||||||
|
|
||||||
|
Tests whether or not the plugin version indicated by the template
|
||||||
|
matches one of the versions specified in plugin_versions
|
||||||
|
|
||||||
|
:param template: A node group or cluster template
|
||||||
|
:param plugin_versions: A list of plugin version strings. These
|
||||||
|
values may be regular version strings or may be
|
||||||
|
the name of the plugin followed by a
|
||||||
|
"." followed by a version string.
|
||||||
|
:returns: True if the plugin version specified in the template
|
||||||
|
matches a version in plugin_versions or plugin_versions
|
||||||
|
is an empty list. Otherwise False
|
||||||
|
'''
|
||||||
|
def dotted_name(template):
|
||||||
|
return template['plugin_name'] + "." + template['hadoop_version']
|
||||||
|
|
||||||
|
version_matches = plugin_versions is None or (
|
||||||
|
template['hadoop_version'] in plugin_versions) or (
|
||||||
|
dotted_name(template) in plugin_versions)
|
||||||
|
|
||||||
|
return version_matches
|
||||||
|
|
||||||
|
|
||||||
|
def check_plugin_name_and_version(template, plugin_names, plugin_versions):
|
||||||
|
'''Check that the template is for one of the specified plugins
|
||||||
|
|
||||||
|
Tests whether or not the plugin name and version indicated by the template
|
||||||
|
matches one of the names and one of the versions specified in
|
||||||
|
plugin_names and plugin_versions
|
||||||
|
|
||||||
|
:param template: A node group or cluster template
|
||||||
|
:param plugin_names: A list of plugin names
|
||||||
|
:param plugin_versions: A list of plugin version strings. These
|
||||||
|
values may be regular version strings or may be
|
||||||
|
the name of the plugin followed by a
|
||||||
|
"." followed by a version string.
|
||||||
|
:returns: True if the plugin name specified in the template matches
|
||||||
|
a name in plugin_names or plugin_names is an empty list, and if
|
||||||
|
the plugin version specified in the template matches a version
|
||||||
|
in plugin_versions or plugin_versions is an empty list.
|
||||||
|
Otherwise False
|
||||||
|
'''
|
||||||
|
name_and_version_matches = (plugin_names is None or (
|
||||||
|
template['plugin_name'] in plugin_names)) and (
|
||||||
|
check_plugin_version(template, plugin_versions))
|
||||||
|
|
||||||
|
return name_and_version_matches
|
||||||
|
|
||||||
|
|
||||||
|
# TODO(tmckay): refactor the service validation code so
|
||||||
|
# that the node group template usage checks there can be reused
|
||||||
|
# without incurring unnecessary dependencies
|
||||||
|
def check_node_group_template_usage(node_group_template_id,
|
||||||
|
cluster_list, cluster_template_list=[]):
|
||||||
|
cluster_users = []
|
||||||
|
template_users = []
|
||||||
|
|
||||||
|
for cluster in cluster_list:
|
||||||
|
if (node_group_template_id in
|
||||||
|
[node_group.node_group_template_id
|
||||||
|
for node_group in cluster.node_groups]):
|
||||||
|
cluster_users += [cluster.name]
|
||||||
|
|
||||||
|
for cluster_template in cluster_template_list:
|
||||||
|
if (node_group_template_id in
|
||||||
|
[node_group.node_group_template_id
|
||||||
|
for node_group in cluster_template.node_groups]):
|
||||||
|
template_users += [cluster_template.name]
|
||||||
|
|
||||||
|
return cluster_users, template_users
|
||||||
|
|
||||||
|
|
||||||
|
# TODO(tmckay): refactor the service validation code so
|
||||||
|
# that the cluster template usage checks there can be reused
|
||||||
|
# without incurring unnecessary dependencies
|
||||||
|
def check_cluster_template_usage(cluster_template_id, cluster_list):
|
||||||
|
cluster_users = []
|
||||||
|
for cluster in cluster_list:
|
||||||
|
if cluster_template_id == cluster.cluster_template_id:
|
||||||
|
cluster_users.append(cluster.name)
|
||||||
|
|
||||||
|
return cluster_users
|
||||||
|
|
||||||
|
|
||||||
|
def find_node_group_template_by_name(ctx, name):
|
||||||
|
t = conductor.API.node_group_template_get_all(ctx,
|
||||||
|
name=name,
|
||||||
|
is_default=True)
|
||||||
|
if t:
|
||||||
|
return t[0]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def find_cluster_template_by_name(ctx, name):
|
||||||
|
t = conductor.API.cluster_template_get_all(ctx,
|
||||||
|
name=name,
|
||||||
|
is_default=True)
|
||||||
|
if t:
|
||||||
|
return t[0]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def value_diff(current, new_values):
|
||||||
|
'''Return the entries in current that would be overwritten by new_values
|
||||||
|
|
||||||
|
Returns the set of entries in current that would be overwritten
|
||||||
|
if current.update(new_values) was called.
|
||||||
|
|
||||||
|
:param current: A dictionary whose key values are a superset
|
||||||
|
of the key values in new_values
|
||||||
|
:param new_values: A dictionary
|
||||||
|
'''
|
||||||
|
# Current is an existing template from the db and
|
||||||
|
# template is a set of values that has been validated
|
||||||
|
# against the JSON schema for the template.
|
||||||
|
# Copy items from current if they are present in template.
|
||||||
|
|
||||||
|
# In the case of "node_groups" the conductor does magic
|
||||||
|
# to set up template relations and insures that appropriate
|
||||||
|
# fields are cleaned (like "updated_at" and "id") so we
|
||||||
|
# trust the conductor in that case.
|
||||||
|
|
||||||
|
diff_values = {}
|
||||||
|
for k, v in six.iteritems(new_values):
|
||||||
|
if k in current and current[k] != v:
|
||||||
|
diff_values[k] = copy.deepcopy(current[k])
|
||||||
|
return diff_values
|
|
@ -0,0 +1,16 @@
|
||||||
|
[DEFAULT]
|
||||||
|
# Set the flavor_id to 2 which is m1.small in the
|
||||||
|
# default flavor set
|
||||||
|
flavor_id = 2
|
||||||
|
|
||||||
|
[cdh-5-default-namenode]
|
||||||
|
# For the CDH plugin, version 5, set the flavor_id
|
||||||
|
# of the master node to 4 which is m1.large in the
|
||||||
|
# default flavor set
|
||||||
|
flavor_id = 4
|
||||||
|
|
||||||
|
[cdh-530-default-namenode]
|
||||||
|
# For the CDH plugin, version 5.3.0, set the flavor_id
|
||||||
|
# of the master node to 4 which is m1.large in the
|
||||||
|
# default flavor set
|
||||||
|
flavor_id = 4
|
|
@ -35,6 +35,7 @@ console_scripts =
|
||||||
sahara-db-manage = sahara.db.migration.cli:main
|
sahara-db-manage = sahara.db.migration.cli:main
|
||||||
sahara-rootwrap = oslo_rootwrap.cmd:main
|
sahara-rootwrap = oslo_rootwrap.cmd:main
|
||||||
_sahara-subprocess = sahara.cli.sahara_subprocess:main
|
_sahara-subprocess = sahara.cli.sahara_subprocess:main
|
||||||
|
sahara-templates = sahara.db.templates.cli:main
|
||||||
|
|
||||||
sahara.cluster.plugins =
|
sahara.cluster.plugins =
|
||||||
vanilla = sahara.plugins.vanilla.plugin:VanillaProvider
|
vanilla = sahara.plugins.vanilla.plugin:VanillaProvider
|
||||||
|
|
Loading…
Reference in New Issue