openstack
/
charm-cinder
Code Issues Proposed changes

Add defaults for openstack-origin-git config option 

openstack-origin-git currently only supports YAML that specifies
the git repositories to deploy from.

This adds support for default openstack-origin-git values. The
default values supported are: icehouse, kilo, liberty, mitaka,
and master.  For example: openstack-origin-git=master.

Change-Id: Iacd1103095985307394e472411e314c337044f4d
This commit is contained in:
Corey Bryant 2016-05-27 13:31:07 +00:00
parent 67f6175ddd
commit fb5efc409d
8 changed files with 285 additions and 205 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

98
README.md
View File

@ -107,6 +107,9 @@ Users should be aware of three options, in particular:
openstack-origin: Allows Cinder to be installed from a specific apt repository.
See config.yaml for a list of supported sources.
openstack-origin-git: Allows Cinder to be installed from source.
See config.yaml for a list of supported sources.
block-device: When using local storage, a block device should be specified to
back a LVM volume group. It's important this device exists on
all nodes that the service may be deployed to.
@ -118,101 +121,6 @@ overwrite: Whether or not to wipe local storage that of data that may prevent
enabled-services: Can be used to separate cinder services between service
service units (see previous section)
Deploying from source
---------------------
The minimum openstack-origin-git config required to deploy from source is:
openstack-origin-git: include-file://cinder-juno.yaml
cinder-juno.yaml
repositories:
- {name: requirements,
repository: 'git://github.com/openstack/requirements',
branch: stable/juno}
- {name: cinder,
repository: 'git://github.com/openstack/cinder',
branch: stable/juno}
Note that there are only two 'name' values the charm knows about: 'requirements'
and 'cinder'. These repositories must correspond to these 'name' values.
Additionally, the requirements repository must be specified first and the
cinder repository must be specified last. All other repostories are installed
in the order in which they are specified.
The following is a full list of current tip repos (may not be up-to-date):
openstack-origin-git: include-file://cinder-master.yaml
cinder-master.yaml
repositories:
- {name: requirements,
repository: 'git://github.com/openstack/requirements',
branch: master}
- {name: oslo-concurrency,
repository: 'git://github.com/openstack/oslo.concurrency',
branch: master}
- {name: oslo-config,
repository: 'git://github.com/openstack/oslo.config',
branch: master}
- {name: oslo-context,
repository': 'git://github.com/openstack/oslo.context',
branch: master}
- {name: oslo-db,
repository: 'git://github.com/openstack/oslo.db',
branch: master}
- {name: oslo-i18n,
repository: 'git://github.com/openstack/oslo.i18n',
branch: master}
- {name: oslo-messaging,
repository: 'git://github.com/openstack/oslo.messaging',
branch: master}
- {name: oslo-serialization,
repository: 'git://github.com/openstack/oslo.serialization',
branch: master}
- {name: oslo-utils,
repository: 'git://github.com/openstack/oslo.utils',
branch: master}
- {name: oslo-rootwrap,
repository: 'git://github.com/openstack/oslo.rootwrap',
branch: master}
- {name: oslo-vmware,
repository: 'git://github.com/openstack/oslo.vmware',
branch: master}
- {name: osprofiler,
repository: 'git://github.com/stackforge/osprofiler',
branch: master}
- {name: pbr,
repository: 'git://github.com/openstack-dev/pbr',
branch: master}
- {name: python-barbicanclient,
repository: 'git://github.com/openstack/python-barbicanclient',
branch: master}
- {name: python-glanceclient,
repository: 'git://github.com/openstack/python-glanceclient',
branch: master}
- {name: python-novaclient,
repository: 'git://github.com/openstack/python-novaclient',
branch: master}
- {name: python-swiftclient:
repository: 'git://github.com/openstack/python-swiftclient',
branch: master}
- {name: sqlalchemy-migrate,
repository: 'git://github.com/stackforge/sqlalchemy-migrate',
branch: master}
- {name: stevedore,
repository: 'git://github.com/openstack/stevedore',
branch: master}
- {name: taskflow,
repository: 'git://github.com/openstack/taskflow',
branch: master}
- {name: keystonemiddleware,
repository: 'git://github.com/openstack/keystonemiddleware',
branch: master}
- {name: cinder,
repository: 'git://github.com/openstack/cinder',
branch: master}
Network Space support
---------------------

25
config.yaml
View File

@ -45,14 +45,29 @@ options:
default:
type: string
description: |
Specifies a YAML-formatted dictionary listing the git
repositories and branches from which to install OpenStack and
its dependencies.
Specifies a default OpenStack release name, or a YAML dictionary
listing the git repositories to install from.
The default Openstack release name may be one of the following, where
the corresponding OpenStack github branch will be used:
* icehouse
* kilo
* liberty
* mitaka
* master
The YAML must minimally include requirements and cinder repositories,
and may also include repositories for other dependencies:
repositories:
- {name: requirements,
repository: 'git://github.com/openstack/requirements',
branch: master}
- {name: cinder,
repository: 'git://github.com/openstack/cinder',
branch: master}
Note that the installed config files will be determined based on
the OpenStack release of the openstack-origin option.
For more details see README.md.
enabled-services:
default: all
type: string

88
hooks/charmhelpers/contrib/openstack/context.py
View File

@ -23,7 +23,6 @@ from base64 import b64decode
from subprocess import check_call, CalledProcessError
import six
import yaml
from charmhelpers.fetch import (
apt_install,
@ -50,6 +49,7 @@ from charmhelpers.core.hookenv import (
from charmhelpers.core.sysctl import create as sysctl_create
from charmhelpers.core.strutils import bool_from_string
from charmhelpers.contrib.openstack.exceptions import OSContextError
from charmhelpers.core.host import (
get_bond_master,
@ -88,7 +88,10 @@ from charmhelpers.contrib.network.ip import (
is_address_in_network,
is_bridge_member,
)
from charmhelpers.contrib.openstack.utils import get_host_ip
from charmhelpers.contrib.openstack.utils import (
config_flags_parser,
get_host_ip,
)
from charmhelpers.core.unitdata import kv
try:
@ -101,10 +104,6 @@ CA_CERT_PATH = '/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt'
ADDRESS_TYPES = ['admin', 'internal', 'public']
class OSContextError(Exception):
pass
def ensure_packages(packages):
"""Install but do not upgrade required plugin packages."""
required = filter_installed_packages(packages)
@ -125,83 +124,6 @@ def context_complete(ctxt):
return True
def config_flags_parser(config_flags):
"""Parses config flags string into dict.
This parsing method supports a few different formats for the config
flag values to be parsed:
1. A string in the simple format of key=value pairs, with the possibility
of specifying multiple key value pairs within the same string. For
example, a string in the format of 'key1=value1, key2=value2' will
return a dict of:
{'key1': 'value1',
'key2': 'value2'}.
2. A string in the above format, but supporting a comma-delimited list
of values for the same key. For example, a string in the format of
'key1=value1, key2=value3,value4,value5' will return a dict of:
{'key1', 'value1',
'key2', 'value2,value3,value4'}
3. A string containing a colon character (:) prior to an equal
character (=) will be treated as yaml and parsed as such. This can be
used to specify more complex key value pairs. For example,
a string in the format of 'key1: subkey1=value1, subkey2=value2' will
return a dict of:
{'key1', 'subkey1=value1, subkey2=value2'}
The provided config_flags string may be a list of comma-separated values
which themselves may be comma-separated list of values.
"""
# If we find a colon before an equals sign then treat it as yaml.
# Note: limit it to finding the colon first since this indicates assignment
# for inline yaml.
colon = config_flags.find(':')
equals = config_flags.find('=')
if colon > 0:
if colon < equals or equals < 0:
return yaml.safe_load(config_flags)
if config_flags.find('==') >= 0:
log("config_flags is not in expected format (key=value)", level=ERROR)
raise OSContextError
# strip the following from each value.
post_strippers = ' ,'
# we strip any leading/trailing '=' or ' ' from the string then
# split on '='.
split = config_flags.strip(' =').split('=')
limit = len(split)
flags = {}
for i in range(0, limit - 1):
current = split[i]
next = split[i + 1]
vindex = next.rfind(',')
if (i == limit - 2) or (vindex < 0):
value = next
else:
value = next[:vindex]
if i == 0:
key = current
else:
# if this not the first entry, expect an embedded key.
index = current.rfind(',')
if index < 0:
log("Invalid config value(s) at index %s" % (i), level=ERROR)
raise OSContextError
key = current[index + 1:]
# Add to collection.
flags[key.strip(post_strippers)] = value.rstrip(post_strippers)
return flags
class OSContextGenerator(object):
"""Base class for all context generators."""
interfaces = []

6
hooks/charmhelpers/contrib/openstack/exceptions.py Normal file
View File

@ -0,0 +1,6 @@
class OSContextError(Exception):
"""Raised when an error occurs during context generation.
This exception is principally used in contrib.openstack.context
"""
pass

222
hooks/charmhelpers/contrib/openstack/utils.py
View File

@ -47,9 +47,11 @@ from charmhelpers.core.hookenv import (
charm_dir,
DEBUG,
INFO,
ERROR,
related_units,
relation_ids,
relation_set,
service_name,
status_set,
hook_name
)
@ -83,6 +85,7 @@ from charmhelpers.core.host import (
from charmhelpers.fetch import apt_install, apt_cache, install_remote
from charmhelpers.contrib.storage.linux.utils import is_block_device, zap_disk
from charmhelpers.contrib.storage.linux.loopback import ensure_loopback_device
from charmhelpers.contrib.openstack.exceptions import OSContextError
CLOUD_ARCHIVE_URL = "http://ubuntu-cloud.archive.canonical.com/ubuntu"
CLOUD_ARCHIVE_KEY_ID = '5EDB1B62EC4926EA'
@ -101,6 +104,8 @@ UBUNTU_OPENSTACK_RELEASE = OrderedDict([
('vivid', 'kilo'),
('wily', 'liberty'),
('xenial', 'mitaka'),
('yakkety', 'newton'),
('zebra', 'ocata'), # TODO: upload with real Z name
])
@ -115,6 +120,8 @@ OPENSTACK_CODENAMES = OrderedDict([
('2015.1', 'kilo'),
('2015.2', 'liberty'),
('2016.1', 'mitaka'),
('2016.2', 'newton'),
('2017.1', 'ocata'),
])
# The ugly duckling - must list releases oldest to newest
@ -139,50 +146,89 @@ SWIFT_CODENAMES = OrderedDict([
['2.3.0', '2.4.0', '2.5.0']),
('mitaka',
['2.5.0', '2.6.0', '2.7.0']),
('newton',
['2.8.0']),
])
# >= Liberty version->codename mapping
PACKAGE_CODENAMES = {
'nova-common': OrderedDict([
('12.0', 'liberty'),
('13.0', 'mitaka'),
('12', 'liberty'),
('13', 'mitaka'),
('14', 'newton'),
('15', 'ocata'),
]),
'neutron-common': OrderedDict([
('7.0', 'liberty'),
('8.0', 'mitaka'),
('8.1', 'mitaka'),
('7', 'liberty'),
('8', 'mitaka'),
('9', 'newton'),
('10', 'ocata'),
]),
'cinder-common': OrderedDict([
('7.0', 'liberty'),
('8.0', 'mitaka'),
('7', 'liberty'),
('8', 'mitaka'),
('9', 'newton'),
('10', 'ocata'),
]),
'keystone': OrderedDict([
('8.0', 'liberty'),
('8.1', 'liberty'),
('9.0', 'mitaka'),
('8', 'liberty'),
('9', 'mitaka'),
('10', 'newton'),
('11', 'ocata'),
]),
'horizon-common': OrderedDict([
('8.0', 'liberty'),
('9.0', 'mitaka'),
('8', 'liberty'),
('9', 'mitaka'),
('10', 'newton'),
('11', 'ocata'),
]),
'ceilometer-common': OrderedDict([
('5.0', 'liberty'),
('6.0', 'mitaka'),
('5', 'liberty'),
('6', 'mitaka'),
('7', 'newton'),
('8', 'ocata'),
]),
'heat-common': OrderedDict([
('5.0', 'liberty'),
('6.0', 'mitaka'),
('5', 'liberty'),
('6', 'mitaka'),
('7', 'newton'),
('8', 'ocata'),
]),
'glance-common': OrderedDict([
('11.0', 'liberty'),
('12.0', 'mitaka'),
('11', 'liberty'),
('12', 'mitaka'),
('13', 'newton'),
('14', 'ocata'),
]),
'openstack-dashboard': OrderedDict([
('8.0', 'liberty'),
('9.0', 'mitaka'),
('8', 'liberty'),
('9', 'mitaka'),
('10', 'newton'),
('11', 'ocata'),
]),
}
GIT_DEFAULT_REPOS = {
'requirements': 'git://github.com/openstack/requirements',
'cinder': 'git://github.com/openstack/cinder',
'glance': 'git://github.com/openstack/glance',
'horizon': 'git://github.com/openstack/horizon',
'keystone': 'git://github.com/openstack/keystone',
'neutron': 'git://github.com/openstack/neutron',
'neutron-fwaas': 'git://github.com/openstack/neutron-fwaas',
'neutron-lbaas': 'git://github.com/openstack/neutron-lbaas',
'neutron-vpnaas': 'git://github.com/openstack/neutron-vpnaas',
'nova': 'git://github.com/openstack/nova',
}
GIT_DEFAULT_BRANCHES = {
'icehouse': 'icehouse-eol',
'kilo': 'stable/kilo',
'liberty': 'stable/liberty',
'mitaka': 'stable/mitaka',
'master': 'master',
}
DEFAULT_LOOPBACK_SIZE = '5G'
@ -304,10 +350,13 @@ def get_os_codename_package(package, fatal=True):
if match:
vers = match.group(0)
# Generate a major version number for newer semantic
# versions of openstack projects
major_vers = vers.split('.')[0]
# >= Liberty independent project versions
if (package in PACKAGE_CODENAMES and
vers in PACKAGE_CODENAMES[package]):
return PACKAGE_CODENAMES[package][vers]
major_vers in PACKAGE_CODENAMES[package]):
return PACKAGE_CODENAMES[package][major_vers]
else:
# < Liberty co-ordinated project versions
try:
@ -467,6 +516,9 @@ def configure_installation_source(rel):
'mitaka': 'trusty-updates/mitaka',
'mitaka/updates': 'trusty-updates/mitaka',
'mitaka/proposed': 'trusty-proposed/mitaka',
'newton': 'xenial-updates/newton',
'newton/updates': 'xenial-updates/newton',
'newton/proposed': 'xenial-proposed/newton',
}
try:
@ -662,6 +714,53 @@ def git_install_requested():
requirements_dir = None
def git_default_repos(projects):
"""
Returns default repos if a default openstack-origin-git value is specified.
"""
service = service_name()
for default, branch in GIT_DEFAULT_BRANCHES.iteritems():
if projects == default:
# add the requirements repo first
repo = {
'name': 'requirements',
'repository': GIT_DEFAULT_REPOS['requirements'],
'branch': branch,
}
repos = [repo]
# neutron and nova charms require some additional repos
if service == 'neutron':
for svc in ['neutron-fwaas', 'neutron-lbaas', 'neutron-vpnaas']:
repo = {
'name': svc,
'repository': GIT_DEFAULT_REPOS[svc],
'branch': branch,
}
repos.append(repo)
elif service == 'nova':
repo = {
'name': 'neutron',
'repository': GIT_DEFAULT_REPOS['neutron'],
'branch': branch,
}
repos.append(repo)
# finally add the current service's repo
repo = {
'name': service,
'repository': GIT_DEFAULT_REPOS[service],
'branch': branch,
}
repos.append(repo)
return yaml.dump(dict(repositories=repos))
return projects
def _git_yaml_load(projects_yaml):
"""
Load the specified yaml into a dictionary.
@ -1616,3 +1715,82 @@ def pausable_restart_on_change(restart_map, stopstart=False,
restart_functions)
return wrapped_f
return wrap
def config_flags_parser(config_flags):
"""Parses config flags string into dict.
This parsing method supports a few different formats for the config
flag values to be parsed:
1. A string in the simple format of key=value pairs, with the possibility
of specifying multiple key value pairs within the same string. For
example, a string in the format of 'key1=value1, key2=value2' will
return a dict of:
{'key1': 'value1',
'key2': 'value2'}.
2. A string in the above format, but supporting a comma-delimited list
of values for the same key. For example, a string in the format of
'key1=value1, key2=value3,value4,value5' will return a dict of:
{'key1', 'value1',
'key2', 'value2,value3,value4'}
3. A string containing a colon character (:) prior to an equal
character (=) will be treated as yaml and parsed as such. This can be
used to specify more complex key value pairs. For example,
a string in the format of 'key1: subkey1=value1, subkey2=value2' will
return a dict of:
{'key1', 'subkey1=value1, subkey2=value2'}
The provided config_flags string may be a list of comma-separated values
which themselves may be comma-separated list of values.
"""
# If we find a colon before an equals sign then treat it as yaml.
# Note: limit it to finding the colon first since this indicates assignment
# for inline yaml.
colon = config_flags.find(':')
equals = config_flags.find('=')
if colon > 0:
if colon < equals or equals < 0:
return yaml.safe_load(config_flags)
if config_flags.find('==') >= 0:
juju_log("config_flags is not in expected format (key=value)",
level=ERROR)
raise OSContextError
# strip the following from each value.
post_strippers = ' ,'
# we strip any leading/trailing '=' or ' ' from the string then
# split on '='.
split = config_flags.strip(' =').split('=')
limit = len(split)
flags = {}
for i in range(0, limit - 1):
current = split[i]
next = split[i + 1]
vindex = next.rfind(',')
if (i == limit - 2) or (vindex < 0):
value = next
else:
value = next[:vindex]
if i == 0:
key = current
else:
# if this not the first entry, expect an embedded key.
index = current.rfind(',')
if index < 0:
juju_log("Invalid config value(s) at index %s" % (i),
level=ERROR)
raise OSContextError
key = current[index + 1:]
# Add to collection.
flags[key.strip(post_strippers)] = value.rstrip(post_strippers)
return flags

41
hooks/charmhelpers/contrib/storage/linux/ceph.py
View File

@ -40,6 +40,7 @@ from subprocess import (
CalledProcessError,
)
from charmhelpers.core.hookenv import (
config,
local_unit,
relation_get,
relation_ids,
@ -64,6 +65,7 @@ from charmhelpers.fetch import (
)
from charmhelpers.core.kernel import modprobe
from charmhelpers.contrib.openstack.utils import config_flags_parser
KEYRING = '/etc/ceph/ceph.client.{}.keyring'
KEYFILE = '/etc/ceph/ceph.client.{}.key'
@ -1204,3 +1206,42 @@ def send_request_if_needed(request, relation='ceph'):
for rid in relation_ids(relation):
log('Sending request {}'.format(request.request_id), level=DEBUG)
relation_set(relation_id=rid, broker_req=request.request)
class CephConfContext(object):
"""Ceph config (ceph.conf) context.
Supports user-provided Ceph configuration settings. Use can provide a
dictionary as the value for the config-flags charm option containing
Ceph configuration settings keyede by their section in ceph.conf.
"""
def __init__(self, permitted_sections=None):
self.permitted_sections = permitted_sections or []
def __call__(self):
conf = config('config-flags')
if not conf:
return {}
conf = config_flags_parser(conf)
if type(conf) != dict:
log("Provided config-flags is not a dictionary - ignoring",
level=WARNING)
return {}
permitted = self.permitted_sections
if permitted:
diff = set(conf.keys()).symmetric_difference(set(permitted))
if diff:
log("Config-flags contains invalid keys '%s' - they will be "
"ignored" % (', '.join(diff)), level=WARNING)
ceph_conf = {}
for key in conf:
if permitted and key not in permitted:
log("Ignoring key '%s'" % key, level=WARNING)
continue
ceph_conf[key] = conf[key]
return ceph_conf

8
hooks/charmhelpers/fetch/__init__.py
View File

@ -106,6 +106,14 @@ CLOUD_ARCHIVE_POCKETS = {
'mitaka/proposed': 'trusty-proposed/mitaka',
'trusty-mitaka/proposed': 'trusty-proposed/mitaka',
'trusty-proposed/mitaka': 'trusty-proposed/mitaka',
# Newton
'newton': 'xenial-updates/newton',
'xenial-newton': 'xenial-updates/newton',
'xenial-newton/updates': 'xenial-updates/newton',
'xenial-updates/newton': 'xenial-updates/newton',
'newton/proposed': 'xenial-proposed/newton',
'xenial-newton/proposed': 'xenial-proposed/newton',
'xenial-proposed/newton': 'xenial-proposed/newton',
}
# The order of this list is very important. Handlers should be listed in from

2
hooks/cinder_utils.py
View File

@ -79,6 +79,7 @@ from charmhelpers.contrib.openstack.utils import (
configure_installation_source,
get_os_codename_install_source,
git_clone_and_install,
git_default_repos,
git_generate_systemd_init_files,
git_install_requested,
git_pip_venv_dir,
@ -677,6 +678,7 @@ def git_install(projects_yaml):
"""Perform setup, and install git repos specified in yaml parameter."""
if git_install_requested():
git_pre_install()
projects_yaml = git_default_repos(projects_yaml)
git_clone_and_install(projects_yaml, core_project='cinder')
git_post_install(projects_yaml)