openstack
/
tripleo-heat-templates
Code Issues Proposed changes

docker: new hybrid deployment architecture and configuration 

This patch implements a new docker deployment architecture that
should us to install docker services in a stepwise manner alongside
of baremetal puppet services. This works by using Yaql to select
docker specific services (docker/services/*.yaml) vs the puppet
specific ones and then applying the selected Json to relevant Heat
software deployments for docker and baremetal puppet in a stepwise
fashion.

Additionally the new architecture
leverages new composable services interfaces from Newton to
allow configuration of per-service container configuration
sets (directories that are bind mounted into kolla containers) by
using the Kolla containers themselves. It does this by spinning up
a throw away "configuration only" version of the container being
configured itself, then running the puppet apply in that container and
copying the generated config files into /var/lib/config-data. This
avoids having to install all of the OpenStack dependency packages
in the heat-agent-container itself (our previous approach) and should
allow us to configure a much wider variety of container config files
that would otherwise be impossible with the previous shared approach.

The new approach (combined) should allow us to configure containers in
both the undercloud and overcloud and incrementally add CI coverage to
services as we containerize them.

Co-Authored-By: Martin André <m.andre@redhat.com>
Co-Authored-By: Ian Main <imain@redhat.com>
Co-Authored-By: Flavio Percoco <flavio@redhat.com>

Change-Id: Ibcff99f03e6751fbf3197adefd5d344178b71fc2
This commit is contained in:
Dan Prince 2017-01-03 22:21:44 -05:00
parent b06e49302d
commit ad2ea290be
16 changed files with 584 additions and 327 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
docker/copy-etc.sh
View File

@ -1,3 +0,0 @@
#!/bin/bash
echo "Copying agent container /etc to /var/lib/etc-data"
cp -a /etc/* /var/lib/etc-data/

6
docker/create-config-dir.sh Normal file
View File

@ -0,0 +1,6 @@
#!/bin/bash
# This is where we stack puppet configuration (for now)...
mkdir -p /var/lib/config-data
# This is the docker-puppet configs end in
mkdir -p /var/lib/docker-puppet

210
docker/docker-puppet.py Executable file
View File

@ -0,0 +1,210 @@
#!/usr/bin/env python
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# Shell script tool to run puppet inside of the given docker container image.
# Uses the config file at /var/lib/docker-puppet/docker-puppet.json as a source for a JSON
# array of [config_volume, puppet_tags, manifest, config_image, [volumes]] settings
# that can be used to generate config files or run ad-hoc puppet modules
# inside of a container.
import json
import os
import subprocess
import sys
import tempfile
# this is to match what we do in deployed-server
def short_hostname():
subproc = subprocess.Popen(['hostname', '-s'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
cmd_stdout, cmd_stderr = subproc.communicate()
return cmd_stdout.rstrip()
def pull_image(name):
print('Pulling image: %s' % name)
subproc = subprocess.Popen(['/usr/bin/docker', 'pull', name],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
cmd_stdout, cmd_stderr = subproc.communicate()
print(cmd_stdout)
print(cmd_stderr)
def rm_container(name):
print('Removing container: %s' % name)
subproc = subprocess.Popen(['/usr/bin/docker', 'rm', name],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
cmd_stdout, cmd_stderr = subproc.communicate()
print(cmd_stdout)
print(cmd_stderr)
config_file = os.environ.get('CONFIG', '/var/lib/docker-puppet/docker-puppet.json')
print('docker-puppet')
print('CONFIG: %s' % config_file)
with open(config_file) as f:
json_data = json.load(f)
# To save time we support configuring 'shared' services at the same
# time. For example configuring all of the heat services
# in a single container pass makes sense and will save some time.
# To support this we merge shared settings together here.
#
# We key off of config_volume as this should be the same for a
# given group of services. We are also now specifying the container
# in which the services should be configured. This should match
# in all instances where the volume name is also the same.
configs = {}
for service in json_data:
config_volume = service[0] or ''
puppet_tags = service[1] or ''
manifest = service[2] or ''
config_image = service[3] or ''
volumes = service[4] if len(service) > 4 else []
print('---------')
print('config_volume %s' % config_volume)
print('puppet_tags %s' % puppet_tags)
print('manifest %s' % manifest)
print('config_image %s' % config_image)
print('volumes %s' % volumes)
# We key off of config volume for all configs.
if config_volume in configs:
# Append puppet tags and manifest.
print("Existing service, appending puppet tags and manifest\n")
if puppet_tags:
configs[config_volume][1] = '%s,%s' % (configs[config_volume][1],
puppet_tags)
if manifest:
configs[config_volume][2] = '%s\n%s' % (configs[config_volume][2],
manifest)
if configs[config_volume][3] != config_image:
print("WARNING: Config containers do not match even though"
" shared volumes are the same!\n")
else:
print("Adding new service\n")
configs[config_volume] = service
print('Service compilation completed.\n')
for config_volume in configs:
service = configs[config_volume]
puppet_tags = service[1] or ''
manifest = service[2] or ''
config_image = service[3] or ''
volumes = service[4] if len(service) > 4 else []
if puppet_tags:
puppet_tags = "file,file_line,concat,%s" % puppet_tags
else:
puppet_tags = "file,file_line,concat"
print('---------')
print('config_volume %s' % config_volume)
print('puppet_tags %s' % puppet_tags)
print('manifest %s' % manifest)
print('config_image %s' % config_image)
hostname = short_hostname()
with open('/var/lib/docker-puppet/docker-puppet.sh', 'w') as script_file:
os.chmod(script_file.name, 0755)
script_file.write("""#!/bin/bash
set -ex
mkdir -p /etc/puppet
cp -a /tmp/puppet-etc/* /etc/puppet
rm -Rf /etc/puppet/ssl # not in use and causes permission errors
echo '{"step": 6}' > /etc/puppet/hieradata/docker.json
TAGS=""
if [ -n "%(puppet_tags)s" ]; then
TAGS='--tags "%(puppet_tags)s"'
fi
FACTER_hostname=%(hostname)s FACTER_uuid=docker /usr/bin/puppet apply --verbose $TAGS /etc/config.pp
# Disables archiving
if [ -z "%(no_archive)s" ]; then
rm -Rf /var/lib/config-data/%(name)s
# copying etc should be enough for most services
mkdir -p /var/lib/config-data/%(name)s/etc
cp -a /etc/* /var/lib/config-data/%(name)s/etc/
if [ -d /root/ ]; then
cp -a /root/ /var/lib/config-data/%(name)s/root/
fi
if [ -d /var/lib/ironic/tftpboot/ ]; then
mkdir -p /var/lib/config-data/%(name)s/var/lib/ironic/
cp -a /var/lib/ironic/tftpboot/ /var/lib/config-data/%(name)s/var/lib/ironic/tftpboot/
fi
if [ -d /var/lib/ironic/httpboot/ ]; then
mkdir -p /var/lib/config-data/%(name)s/var/lib/ironic/
cp -a /var/lib/ironic/httpboot/ /var/lib/config-data/%(name)s/var/lib/ironic/httpboot/
fi
# apache services may files placed in /var/www/
if [ -d /var/www/ ]; then
mkdir -p /var/lib/config-data/%(name)s/var/www
cp -a /var/www/* /var/lib/config-data/%(name)s/var/www/
fi
fi
""" % {'puppet_tags': puppet_tags, 'name': config_volume,
'hostname': hostname,
'no_archive': os.environ.get('NO_ARCHIVE', '')})
with tempfile.NamedTemporaryFile() as tmp_man:
with open(tmp_man.name, 'w') as man_file:
man_file.write('include ::tripleo::packages\n')
man_file.write(manifest)
rm_container('docker-puppet-%s' % config_volume)
pull_image(config_image)
dcmd = ['/usr/bin/docker', 'run',
'--user', 'root',
'--name', 'docker-puppet-%s' % config_volume,
'--volume', '%s:/etc/config.pp:ro' % tmp_man.name,
'--volume', '/etc/puppet/:/tmp/puppet-etc/:ro',
'--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro',
'--volume', '/var/lib/config-data/:/var/lib/config-data/:rw',
'--volume', 'tripleo_logs:/var/log/tripleo/',
'--volume', '/var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:ro']
for volume in volumes:
dcmd.extend(['--volume', volume])
dcmd.extend(['--entrypoint', '/var/lib/docker-puppet/docker-puppet.sh'])
env = {}
if os.environ.get('NET_HOST', 'false') == 'true':
print('NET_HOST enabled')
dcmd.extend(['--net', 'host', '--volume',
'/etc/hosts:/etc/hosts:ro'])
dcmd.append(config_image)
subproc = subprocess.Popen(dcmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, env=env)
cmd_stdout, cmd_stderr = subproc.communicate()
print(cmd_stdout)
print(cmd_stderr)
if subproc.returncode != 0:
print('Failed running docker-puppet.py for %s' % config_volume)
sys.exit(subproc.returncode)
else:
rm_container('docker-puppet-%s' % config_volume)

26
docker/firstboot/setup_docker_host.sh Executable file
View File

@ -0,0 +1,26 @@
#!/bin/bash
set -eux
# TODO This would be better in puppet
# TODO remove this when built image includes docker
if [ ! -f "/usr/bin/docker" ]; then
yum -y install docker
fi
# NOTE(mandre) $docker_namespace_is_registry is not a bash variable but is
# a place holder for text replacement done via heat
if [ "$docker_namespace_is_registry" = "True" ]; then
/usr/bin/systemctl stop docker.service
# if namespace is used with local registry, trim all namespacing
trim_var=$docker_registry
registry_host="${trim_var%%/*}"
/bin/sed -i -r "s/^[# ]*INSECURE_REGISTRY *=.+$/INSECURE_REGISTRY='--insecure-registry $registry_host'/" /etc/sysconfig/docker
fi
# enable and start docker
/usr/bin/systemctl enable docker.service
/usr/bin/systemctl start docker.service
# Disable libvirtd
/usr/bin/systemctl disable libvirtd.service
/usr/bin/systemctl stop libvirtd.service

13
docker/firstboot/install_docker_agents.yaml → docker/firstboot/setup_docker_host.yaml
View File

@ -1,9 +1,6 @@
heat_template_version: ocata
parameters:
DockerAgentImage:
type: string
default: heat-docker-agents
DockerNamespace:
type: string
default: tripleoupstream
@ -17,22 +14,18 @@ resources:
type: OS::Heat::MultipartMime
properties:
parts:
- config: {get_resource: install_docker_agents}
- config: {get_resource: setup_docker_host}
install_docker_agents:
setup_docker_host:
type: OS::Heat::SoftwareConfig
properties:
group: script
config:
str_replace:
params:
$agent_image:
list_join:
- '/'
- [ {get_param: DockerNamespace}, {get_param: DockerAgentImage} ]
$docker_registry: {get_param: DockerNamespace}
$docker_namespace_is_registry: {get_param: DockerNamespaceIsRegistry}
template: {get_file: ./start_docker_agents.sh}
template: {get_file: ./setup_docker_host.sh}
outputs:
OS::stack_id:

69
docker/firstboot/start_docker_agents.sh
View File

@ -1,69 +0,0 @@
#!/bin/bash
set -eux
# TODO remove this when built image includes docker
if [ ! -f "/usr/bin/docker" ]; then
yum -y install docker
fi
# Local docker registry 1.8
# NOTE(mandre) $docker_namespace_is_registry is not a bash variable but is
# a place holder for text replacement done via heat
if [ "$docker_namespace_is_registry" = "True" ]; then
/usr/bin/systemctl stop docker.service
# if namespace is used with local registry, trim all namespacing
trim_var=$docker_registry
registry_host="${trim_var%%/*}"
/bin/sed -i -r "s/^[# ]*INSECURE_REGISTRY *=.+$/INSECURE_REGISTRY='--insecure-registry $registry_host'/" /etc/sysconfig/docker
fi
mkdir -p /var/lib/etc-data/json-config #FIXME: this should be a docker data container
# NOTE(flaper87): Heat Agent required mounts
AGENT_COMMAND_MOUNTS="\
-v /var/lib/etc-data:/var/lib/etc-data \
-v /run:/run \
-v /etc/hosts:/etc/hosts \
-v /etc:/host/etc \
-v /var/lib/dhclient:/var/lib/dhclient \
-v /var/lib/cloud:/var/lib/cloud \
-v /var/lib/heat-cfntools:/var/lib/heat-cfntools \
-v /var/lib/os-collect-config:/var/lib/os-collect-config \
-v /var/lib/os-apply-config-deployments:/var/lib/os-apply-config-deployments \
-v /var/lib/heat-config:/var/lib/heat-config \
-v /etc/sysconfig/docker:/etc/sysconfig/docker \
-v /etc/sysconfig/network-scripts:/etc/sysconfig/network-scripts \
-v /usr/lib64/libseccomp.so.2:/usr/lib64/libseccomp.so.2 \
-v /usr/bin/docker:/usr/bin/docker \
-v /usr/bin/docker-current:/usr/bin/docker-current \
-v /var/lib/os-collect-config:/var/lib/os-collect-config"
# heat-docker-agents service
cat <<EOF > /etc/systemd/system/heat-docker-agents.service
[Unit]
Description=Heat Docker Agent Container
After=docker.service
Requires=docker.service
Before=os-collect-config.service
Conflicts=os-collect-config.service
[Service]
User=root
Restart=always
ExecStartPre=-/usr/bin/docker rm -f heat-agents
ExecStart=/usr/bin/docker run --name heat-agents --privileged --net=host \
$AGENT_COMMAND_MOUNTS \
--entrypoint=/usr/bin/os-collect-config $agent_image
ExecStop=/usr/bin/docker stop heat-agents
[Install]
WantedBy=multi-user.target
EOF
# enable and start heat-docker-agents
/usr/bin/systemctl enable heat-docker-agents.service
/usr/bin/systemctl start --no-block heat-docker-agents.service
# Disable libvirtd
/usr/bin/systemctl disable libvirtd.service
/usr/bin/systemctl stop libvirtd.service

274
docker/post.j2.yaml
View File

@ -1,3 +1,7 @@
# certain initialization steps (run in a container) will occur
# on the first role listed in the roles file
{% set primary_role_name = roles[0].name -%}
heat_template_version: ocata
description: >
@ -11,7 +15,6 @@ parameters:
role_data:
type: json
description: Mapping of Role name e.g Controller to the per-role data
DeployIdentifier:
default: ''
type: string
@ -46,81 +49,171 @@ resources:
input_values:
update_identifier: {get_param: DeployIdentifier}
{{role.name}}CreateConfigDir:
type: OS::Heat::SoftwareConfig
properties:
group: script
config: {get_file: create-config-dir.sh}
{{role.name}}CreateConfigDirDeployment:
type: OS::Heat::SoftwareDeploymentGroup
properties:
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}CreateConfigDir}
# this creates a JSON config file for our docker-puppet.py script
{{role.name}}GenPuppetConfig:
type: OS::Heat::StructuredConfig
properties:
group: json-file
config:
/var/lib/docker-puppet/docker-puppet.json:
yaql:
# select only services that have a non-null config_image with
# a step_config as well
expression:
$.data.config_volume.zip($.data.puppet_tags, $.data.step_config, $.data.config_image).where($[3] != null and $[1] != null)
data:
config_volume: {get_param: [role_data, {{role.name}}, config_volume]}
step_config: {get_param: [role_data, {{role.name}}, step_config]}
puppet_tags: {get_param: [role_data, {{role.name}}, puppet_tags]}
config_image: {get_param: [role_data, {{role.name}}, config_image]}
{{role.name}}GenPuppetDeployment:
type: OS::Heat::SoftwareDeploymentGroup
properties:
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}GenPuppetConfig}
{{role.name}}GenerateConfig:
type: OS::Heat::SoftwareConfig
properties:
group: script
config: {get_file: docker-puppet.py}
{{role.name}}GenerateConfigDeployment:
type: OS::Heat::SoftwareDeploymentGroup
depends_on: [{{role.name}}GenPuppetDeployment, {{role.name}}ArtifactsDeploy, {{role.name}}CreateConfigDirDeployment]
properties:
name: {{role.name}}GenerateConfigDeployment
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}GenerateConfig}
{{role.name}}PuppetStepConfig:
type: OS::Heat::Value
properties:
type: string
value:
yaql:
expression:
# select 'step_config' only from services that do not have a docker_image
$.data.service_names.zip($.data.step_config, $.data.docker_image).where($[2] = null).where($[1] != null).select($[1]).join("\n")
data:
service_names: {get_param: [role_data, {{role.name}}, service_names]}
step_config: {get_param: [role_data, {{role.name}}, step_config]}
docker_image: {get_param: [role_data, {{role.name}}, docker_image]}
{{role.name}}DockerConfig:
type: OS::Heat::Value
properties:
type: json
value:
yaql:
expression:
# select 'docker_config' only from services that have a docker_image
$.data.service_names.zip($.data.docker_config, $.data.docker_image).where($[2] != null).select($[1]).reduce($1.mergeWith($2), {})
data:
service_names: {get_param: [role_data, {{role.name}}, service_names]}
docker_config: {get_param: [role_data, {{role.name}}, docker_config]}
docker_image: {get_param: [role_data, {{role.name}}, docker_image]}
{{role.name}}KollaJsonConfig:
type: OS::Heat::StructuredConfig
properties:
group: json-file
config:
{get_param: [role_data, {{role.name}}, kolla_config]}
{{role.name}}KollaJsonDeployment:
type: OS::Heat::SoftwareDeploymentGroup
properties:
name: {{role.name}}KollaJsonDeployment
config: {get_resource: {{role.name}}KollaJsonConfig}
servers: {get_param: [servers, {{role.name}}]}
# BEGIN BAREMETAL CONFIG STEPS
{% if role.name == 'Controller' %}
ControllerPrePuppet:
type: OS::TripleO::Tasks::ControllerPrePuppet
properties:
servers: {get_param: [servers, Controller]}
input_values:
update_identifier: {get_param: DeployIdentifier}
{% endif %}
{{role.name}}Config:
type: OS::TripleO::{{role.name}}Config
properties:
StepConfig: {get_param: [role_data, {{role.name}}, step_config]}
{% if role.name.lower() == 'compute' %}
PuppetTags: {get_param: [role_data, {{role.name}}, puppet_tags]}
{% endif %}
StepConfig: {get_attr: [{{role.name}}PuppetStepConfig, value]}
# Step through a series of configuration steps
{{role.name}}Deployment_Step1:
{% for step in range(1, 6) %}
{{role.name}}Deployment_Step{{step}}:
type: OS::Heat::StructuredDeploymentGroup
{% if step == 1 %}
depends_on: [{{role.name}}PreConfig, {{role.name}}ArtifactsDeploy]
{% else %}
depends_on:
{% for dep in roles %}
- {{dep.name}}Deployment_Step{{step -1}}
- {{dep.name}}ContainersDeployment_Step{{step -1}}
{% endfor %}
{% endif %}
properties:
name: {{role.name}}Deployment_Step1
name: {{role.name}}Deployment_Step{{step}}
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}Config}
input_values:
step: 1
step: {{step}}
update_identifier: {get_param: DeployIdentifier}
{{role.name}}Deployment_Step2:
type: OS::Heat::StructuredDeploymentGroup
depends_on:
{% for dep in roles %}
- {{dep.name}}Deployment_Step1
{% endfor %}
properties:
name: {{role.name}}Deployment_Step2
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}Config}
input_values:
step: 2
update_identifier: {get_param: DeployIdentifier}
# END BAREMETAL CONFIG STEPS
{{role.name}}Deployment_Step3:
type: OS::Heat::StructuredDeploymentGroup
depends_on:
{% for dep in roles %}
- {{dep.name}}Deployment_Step2
{% endfor %}
properties:
name: {{role.name}}Deployment_Step3
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}Config}
input_values:
step: 3
update_identifier: {get_param: DeployIdentifier}
# BEGIN CONTAINER CONFIG STEPS
{% for step in range(1, 6) %}
{{role.name}}Deployment_Step4:
type: OS::Heat::StructuredDeploymentGroup
depends_on:
{% for dep in roles %}
- {{dep.name}}Deployment_Step3
{% endfor %}
{{role.name}}ContainersConfig_Step{{step}}:
type: OS::Heat::StructuredConfig
properties:
name: {{role.name}}Deployment_Step4
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}Config}
input_values:
step: 4
update_identifier: {get_param: DeployIdentifier}
group: docker-cmd
config:
{get_attr: [{{role.name}}DockerConfig, value, step_{{step}}]}
{{role.name}}Deployment_Step5:
{{role.name}}ContainersDeployment_Step{{step}}:
type: OS::Heat::StructuredDeploymentGroup
{% if step == 1 %}
depends_on:
{% for dep in roles %}
- {{dep.name}}Deployment_Step4
{% endfor %}
- {{role.name}}PreConfig
- {{role.name}}KollaJsonDeployment
- {{role.name}}GenPuppetDeployment
- {{role.name}}GenerateConfigDeployment
{% else %}
depends_on:
{% for dep in roles %}
- {{dep.name}}ContainersDeployment_Step{{step -1}}
- {{dep.name}}Deployment_Step{{step}} # baremetal steps of the same level run first
- {{dep.name}}Deployment_Step{{step -1}}
{% endfor %}
{% endif %}
properties:
name: {{role.name}}Deployment_Step5
name: {{role.name}}ContainersDeployment_Step{{step}}
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}Config}
input_values:
step: 5
update_identifier: {get_param: DeployIdentifier}
config: {get_resource: {{role.name}}ContainersConfig_Step{{step}}}
{% endfor %}
# END CONTAINER CONFIG STEPS
{{role.name}}PostConfig:
type: OS::TripleO::Tasks::{{role.name}}PostConfig
@ -144,68 +237,15 @@ resources:
properties:
servers: {get_param: [servers, {{role.name}}]}
{% if role.name.lower() == 'compute' %}
CopyEtcConfig:
type: OS::Heat::SoftwareConfig
depends_on: {{role.name}}PostConfig
{% if role.name == 'Controller' %}
ControllerPostPuppet:
depends_on:
- ControllerExtraConfigPost
type: OS::TripleO::Tasks::ControllerPostPuppet
properties:
group: script
outputs:
- name: result
config: {get_file: ../docker/copy-etc.sh}
CopyEtcDeployment:
type: OS::Heat::SoftwareDeploymentGroup
properties:
name: CopyEtcDeployment
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: CopyEtcConfig}
{{role.name}}KollaJsonConfig:
type: OS::Heat::StructuredConfig
depends_on: CopyEtcDeployment
properties:
group: json-file
config:
{get_param: [role_data, {{role.name}}, kolla_config]}
{{role.name}}KollaJsonDeployment:
type: OS::Heat::SoftwareDeploymentGroup
properties:
name: {{role.name}}KollaJsonDeployment
config: {get_resource: {{role.name}}KollaJsonConfig}
servers: {get_param: [servers, {{role.name}}]}
{{role.name}}ContainersConfig_Step1:
type: OS::Heat::StructuredConfig
depends_on: {{role.name}}KollaJsonDeployment
properties:
group: docker-cmd
config:
{get_param: [role_data, {{role.name}}, docker_config, step_1]}
{{role.name}}ContainersConfig_Step2:
type: OS::Heat::StructuredConfig
depends_on: {{role.name}}KollaJsonDeployment
properties:
group: docker-cmd
config:
{get_param: [role_data, {{role.name}}, docker_config, step_2]}
{{role.name}}ContainersDeployment_Step1:
type: OS::Heat::StructuredDeploymentGroup
depends_on: [{{role.name}}PreConfig, {{role.name}}ArtifactsDeploy]
properties:
name: {{role.name}}ContainersDeployment_Step1
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}ContainersConfig_Step1}
{{role.name}}ContainersDeployment_Step2:
type: OS::Heat::StructuredDeploymentGroup
depends_on: {{role.name}}ContainersDeployment_Step1
properties:
name: {{role.name}}ContainersDeployment_Step2
servers: {get_param: [servers, {{role.name}}]}
config: {get_resource: {{role.name}}ContainersConfig_Step2}
servers: {get_param: [servers, Controller]}
input_values:
update_identifier: {get_param: DeployIdentifier}
{% endif %}
{% endfor %}

127
docker/services/README.rst
View File

@ -1,65 +1,104 @@
========
services
========
===============
Docker Services
===============
A TripleO nested stack Heat template that encapsulates generic configuration
data to configure a specific service. This generally includes everything
needed to configure the service excluding the local bind ports which
are still managed in the per-node role templates directly (controller.yaml,
compute.yaml, etc.). All other (global) service settings go into
the puppet/service templates.
TripleO docker services are currently built on top of the puppet services.
To do this each of the docker services includes the output of the
t-h-t puppet/service templates where appropriate.
Input Parameters
----------------
In general global docker specific service settings should reside in these
templates (templates in the docker/services directory.) The required and
optional items are specified in the docker settings section below.
Each service may define its own input parameters and defaults.
Operators will use the parameter_defaults section of any Heat
environment to set per service parameters.
If you are adding a config setting that applies to both docker and
baremetal that setting should (so long as we use puppet) go into the
puppet/services templates themselves.
Config Settings
Building Kolla Images
---------------------
TripleO currently relies on Kolla docker containers. Kolla supports container
customization and we are making use of this feature within TripleO to inject
puppet (our configuration tool of choice) into the Kolla base images. To
build Kolla images for TripleO adjust your kolla config to build your
centos base image with puppet using the example below:
.. code-block::
$ cat template-overrides.j2
{% extends parent_template %}
{% set base_centos_binary_packages_append = ['puppet'] %}
kolla-build --base centos --template-override template-overrides.j2
..
Docker settings
---------------
Each service may define a config_settings output variable which returns
Hiera settings to be configured.
Steps
-----
Each service may define an output variable which returns a puppet manifest
snippet that will run at each of the following steps. Earlier manifests
are re-asserted when applying latter ones.
* config_settings: Custom hiera settings for this service. These are
used to generate configs.
* config_settings: This setting is generally inherited from the
puppet/services templates and only need to be appended
to on accasion if docker specific config settings are required.
* step_config: This setting controls the manifest that is used to
create docker config files via puppet. The puppet tags below are
used along with this manifest to generate a config directory for
this container.
* kolla_config: Contains YAML that represents how to map config files
into the kolla container. This config file is typically mapped into
the container itself at the /var/lib/kolla/config_files/config.json
location and drives how kolla's external config mechanisms work.
* step_config: A puppet manifest that is used to step through the deployment
sequence. Each sequence is given a "step" (via hiera('step') that provides
information for when puppet classes should activate themselves.
* docker_image: The full name of the docker image that will be used.
* docker_compose:
* docker_config: Data that is passed to the docker-cmd hook to configure
a container, or step of containers at each step. See the available steps
below and the related docker-cmd hook documentation in the heat-agents
project.
* container_name:
* puppet_tags: Puppet resource tag names that are used to generate config
files with puppet. Only the named config resources are used to generate
a config file. Any service that specifies tags will have the default
tags of 'file,concat,file_line' appended to the setting.
Example: keystone_config
* volumes:
* config_volume: The name of the volume (directory) where config files
will be generated for this service. Use this as the location to
bind mount into the running Kolla container for configuration.
* config_image: The name of the docker image that will be used for
generating configuration files. This is often the same value as
'docker_image' above but some containers share a common set of
config files which are generated in a common base container.
Docker steps
------------
Similar to baremetal docker containers are brought up in a stepwise manner.
The current architecture supports bringing up baremetal services alongside
of containers. For each step the baremetal puppet manifests are executed
first and then any docker containers are brought up afterwards.
Steps correlate to the following:
1) Service configuration generation with puppet.
2) Early Openstack Service setup (database init?)
3) Early containerized networking services startup (OVS)
4) Network configuration
5) General OpenStack Services
6) Service activation (Pacemaker)
7) Fencing (Pacemaker)
Pre) Containers config files generated per hiera settings.
1) Load Balancer configuration baremetal
a) step 1 baremetal
b) step 1 containers
2) Core Services (Database/Rabbit/NTP/etc.)
a) step 2 baremetal
b) step 2 containers
3) Early Openstack Service setup (Ringbuilder, etc.)
a) step 3 baremetal
b) step 3 containers
4) General OpenStack Services
a) step 4 baremetal
b) step 4 containers
c) Keystone containers post initialization (tenant,service,endpoint creation)
5) Service activation (Pacemaker)
a) step 5 baremetal
b) step 5 containers

36
docker/services/neutron-ovs-agent.yaml
View File

@ -10,7 +10,7 @@ parameters:
type: string
DockerOpenvswitchImage:
description: image
default: 'centos-binary-neutron-openvswitch-agent'
default: 'centos-binary-neutron-openvswitch-agent:latest'
type: string
ServiceNetMap:
default: {}
@ -32,53 +32,53 @@ resources:
NeutronOvsAgentBase:
type: ../../puppet/services/neutron-ovs-agent.yaml
properties:
EndpointMap: {get_param: EndpointMap}
ServiceNetMap: {get_param: ServiceNetMap}
DefaultPasswords: {get_param: DefaultPasswords}
EndpointMap: {get_param: EndpointMap}
outputs:
role_data:
description: Role data for Neutron openvswitch service
value:
service_name: {get_attr: [NeutronOvsAgentBase, role_data, service_name]}
config_settings: {get_attr: [NeutronOvsAgentBase, role_data, config_settings]}
step_config: {get_attr: [NeutronOvsAgentBase, role_data, step_config]}
docker_image: &neutron_ovs_agent_image
list_join:
- '/'
- [ {get_param: DockerNamespace}, {get_param: DockerOpenvswitchImage} ]
puppet_tags: neutron_config,neutron_agent_ovs,neutron_plugin_ml2
config_volume: neutron
config_image: *neutron_ovs_agent_image
kolla_config:
/var/lib/etc-data/json-config/neutron-openvswitch-agent.json:
command: /usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
/var/lib/kolla/config_files/neutron-openvswitch-agent.json:
command: /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
config_files:
- dest: /etc/neutron/neutron.conf
owner: neutron
perm: '0600'
source: /var/lib/kolla/config_files/neutron.conf
source: /var/lib/kolla/config_files/src/etc/neutron/neutron.conf
- dest: /etc/neutron/plugins/ml2/openvswitch_agent.ini
owner: neutron
perm: '0600'
source: /var/lib/kolla/config_files/openvswitch_agent.ini
source: /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/openvswitch_agent.ini
- dest: /etc/neutron/plugins/ml2/ml2_conf.ini
owner: neutron
perm: '0600'
source: /var/lib/kolla/config_files/ml2_conf.ini
source: /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/ml2_conf.ini
docker_config:
step_1:
step_4:
neutronovsagent:
image:
list_join:
- '/'
- [ {get_param: DockerNamespace}, {get_param: DockerOpenvswitchImage} ]
image: *neutron_ovs_agent_image
net: host
pid: host
privileged: true
restart: always
volumes:
- /var/lib/etc-data/json-config/neutron-openvswitch-agent.json:/var/lib/kolla/config_files/config.json
- /var/lib/etc-data/neutron/neutron.conf:/var/lib/kolla/config_files/neutron.conf:ro
- /var/lib/etc-data/neutron/plugins/ml2/ml2_conf.ini:/var/lib/kolla/config_files/ml2_conf.ini:ro
- /var/lib/etc-data/neutron/plugins/ml2/openvswitch_agent.ini:/var/lib/kolla/config_files/openvswitch_agent.ini:ro
- /var/lib/kolla/config_files/neutron-openvswitch-agent.json:/var/lib/kolla/config_files/config.json:ro
- /var/lib/config-data/neutron:/var/lib/kolla/config_files/src:ro
- /etc/localtime:/etc/localtime:ro
- /lib/modules:/lib/modules:ro
- /run:/run
- logs:/var/log/kolla/
environment:
- KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
step_2: {}

38
docker/services/nova-compute.yaml
View File

@ -10,7 +10,7 @@ parameters:
type: string
DockerNovaComputeImage:
description: image
default: 'centos-binary-nova-compute'
default: 'centos-binary-nova-compute:latest'
type: string
ServiceNetMap:
default: {}
@ -29,53 +29,57 @@ parameters:
resources:
NovaComputeBase:
type: ../../puppet/services/nova-compute.yaml
properties:
EndpointMap: {get_param: EndpointMap}
ServiceNetMap: {get_param: ServiceNetMap}
DefaultPasswords: {get_param: DefaultPasswords}
outputs:
role_data:
description: Role data for the Nova Compute service.
value:
service_name: {get_attr: [NovaComputeBase, role_data, service_name]}
config_settings: {get_attr: [NovaComputeBase, role_data, config_settings]}
step_config: {get_attr: [NovaComputeBase, role_data, step_config]}
puppet_tags: nova_config,nova_paste_api_ini
docker_image: &nova_compute_image
list_join:
- '/'
- [ {get_param: DockerNamespace}, {get_param: DockerNovaComputeImage} ]
config_volume: nova_libvirt
config_image: *nova_compute_image
kolla_config:
/var/lib/etc-data/json-config/nova-compute.json:
/var/lib/kolla/config_files/nova-compute.json:
command: /usr/bin/nova-compute --config-file /etc/nova/nova.conf --config-file /etc/nova/rootwrap.conf
config_files:
- dest: /etc/nova/nova.conf
owner: nova
perm: '0600'
source: /var/lib/kolla/config_files/nova.conf
source: /var/lib/kolla/config_files/src/etc/nova/nova.conf
- dest: /etc/nova/rootwrap.conf
owner: nova
perm: '0600'
source: /var/lib/kolla/config_files/rootwrap.conf
source: /var/lib/kolla/config_files/src/etc/nova/rootwrap.conf
docker_config:
step_1:
step_4:
novacompute:
image:
list_join:
- '/'
- [ {get_param: DockerNamespace}, {get_param: DockerNovaComputeImage} ]
image: *nova_compute_image
net: host
privileged: true
user: root
restart: always
volumes:
- /var/lib/etc-data/json-config/nova-compute.json:/var/lib/kolla/config_files/config.json
- /var/lib/etc-data/nova/nova.conf:/var/lib/kolla/config_files/nova.conf:ro
- /var/lib/etc-data/nova/rootwrap.conf:/var/lib/kolla/config_files/rootwrap.conf:ro
- /var/lib/kolla/config_files/nova-compute.json:/var/lib/kolla/config_files/config.json:ro
- /var/lib/config-data/nova_libvirt:/var/lib/kolla/config_files/src:ro
- /dev:/dev
- /etc/iscsi:/etc/iscsi
- /etc/localtime:/etc/localtime:ro
- /lib/modules:/lib/modules:ro
- /run:/run
- /dev:/dev
- logs:/var/log/kolla/
- /etc/iscsi:/etc/iscsi
- /var/lib/nova:/var/lib/nova
- libvirtd:/var/lib/libvirt
- nova_compute:/var/lib/nova/
environment:
- KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
step_2: {}

48
docker/services/nova-libvirt.yaml
View File

@ -10,7 +10,13 @@ parameters:
type: string
DockerLibvirtImage:
description: image
default: 'centos-binary-libvirt'
default: 'centos-binary-nova-libvirt:latest'
type: string
# we configure libvirt via the nova-compute container due to coupling
# in the puppet modules
DockerNovaComputeImage:
description: image
default: 'centos-binary-nova-compute:latest'
type: string
ServiceNetMap:
default: {}
@ -33,50 +39,54 @@ resources:
type: ../../puppet/services/nova-libvirt.yaml
properties:
EndpointMap: {get_param: EndpointMap}
ServiceNetMap: {get_param: ServiceNetMap}
DefaultPasswords: {get_param: DefaultPasswords}
outputs:
role_data:
description: Role data for the Libvirt service.
value:
service_name: {get_attr: [NovaLibvirtBase, role_data, service_name]}
config_settings: {get_attr: [NovaLibvirtBase, role_data, config_settings]}
step_config: {get_attr: [NovaLibvirtBase, role_data, step_config]}
docker_image: &libvirt_image
list_join:
- '/'
- [ {get_param: DockerNamespace}, {get_param: DockerLibvirtImage} ]
puppet_tags: nova_config
config_volume: nova_libvirt
config_image:
list_join:
- '/'
- [ {get_param: DockerNamespace}, {get_param: DockerNovaComputeImage} ]
kolla_config:
/var/lib/etc-data/json-config/nova-libvirt.json:
/var/lib/kolla/config_files/nova-libvirt.json:
command: /usr/sbin/libvirtd --config /etc/libvirt/libvirtd.conf
config_files:
- dest: /etc/libvirt/libvirtd.conf
owner: root
perm: '0644'
source: /var/lib/kolla/config_files/libvirtd.conf
source: /var/lib/kolla/config_files/src/etc/libvirt/libvirtd.conf
docker_config:
step_1:
step_3:
nova_libvirt:
image:
list_join:
- '/'
- [ {get_param: DockerNamespace}, {get_param: DockerLibvirtImage} ]
image: *libvirt_image
net: host
pid: host
privileged: true
restart: always
volumes:
- /var/lib/etc-data/json-config/nova-libvirt.json:/var/lib/kolla/config_files/config.json
- /var/lib/etc-data/libvirt/libvirtd.conf:/var/lib/kolla/config_files/libvirtd.conf
# NOTE(mandre) Ideally the qemu.conf file is mounted in
# /var/lib/kolla/config_files and copied to the right place but
# copy-json.py doesn't allow us to do that without appending the
# file as an additional config on the CLI
- /var/lib/etc-data/libvirt/qemu.conf:/etc/libvirt/qemu.conf:ro
- /var/lib/kolla/config_files/nova-libvirt.json:/var/lib/kolla/config_files/config.json:ro
- /var/lib/config-data/nova_libvirt:/var/lib/kolla/config_files/src:ro
- /dev:/dev
- /etc/localtime:/etc/localtime:ro
- /lib/modules:/lib/modules:ro
- /run:/run
- /dev:/dev
- /sys/fs/cgroup:/sys/fs/cgroup
- logs:/var/log/kolla/
- /var/lib/nova:/var/lib/nova
# Needed to use host's virtlogd
- /var/run/libvirt:/var/run/libvirt
- libvirtd:/var/lib/libvirt
- nova_compute:/var/lib/nova/
- nova_libvirt_qemu:/etc/libvirt/qemu
environment:
- KOLLA_CONFIG_STRATEGY=COPY_ALWAYS
step_2: {}

10
docker/services/services.yaml
View File

@ -66,10 +66,12 @@ outputs:
global_config_settings:
{get_attr: [PuppetServices, role_data, global_config_settings]}
step_config:
{get_attr: [PuppetServices, role_data, step_config]}
puppet_tags: {list_join: [",", {get_attr: [ServiceChain, role_data, puppet_tags]}]}
{get_attr: [ServiceChain, role_data, step_config]}
docker_image: {get_attr: [ServiceChain, role_data, docker_image]}
puppet_tags: {get_attr: [ServiceChain, role_data, puppet_tags]}
config_volume: {get_attr: [ServiceChain, role_data, config_volume]}
config_image: {get_attr: [ServiceChain, role_data, config_image]}
kolla_config:
map_merge: {get_attr: [ServiceChain, role_data, kolla_config]}
docker_config:
step_1: {map_merge: {get_attr: [ServiceChain, role_data, docker_config, step_1]}}
step_2: {map_merge: {get_attr: [ServiceChain, role_data, docker_config, step_2]}}
{get_attr: [ServiceChain, role_data, docker_config]}

19
environments/docker.yaml
View File

@ -1,28 +1,19 @@
resource_registry:
# Docker container with heat agents for containerized compute node.
OS::TripleO::Compute::NodeUserData: ../docker/firstboot/install_docker_agents.yaml
OS::TripleO::Compute::NodeUserData: ../docker/firstboot/setup_docker_host.yaml
#NOTE (dprince) add roles to be docker enabled as we support them
OS::TripleO::Services::NovaLibvirt: ../docker/services/nova-libvirt.yaml
OS::TripleO::Services::ComputeNeutronOvsAgent: ../docker/services/neutron-ovs-agent.yaml
OS::TripleO::Services::NovaCompute: ../docker/services/nova-compute.yaml
# NOTE (dprince) here we set new roles to be docker enabled as we add support
#OS::TripleO::ComputePostDeploySteps: ../docker/post.yaml
# NOTE (mandre) Defining per role post deploy steps doesn't work yet
# Set a global PostDeploySteps that works for both containerized and
# non-containerized roles
OS::TripleO::PostDeploySteps: ../docker/post.yaml
OS::TripleO::Services: ../docker/services/services.yaml
parameter_defaults:
# Defaults to 'tripleoupstream'. Specify a local docker registry
# Example: 192.0.2.1:8787/tripleoupstream
# Example: 192.168.24.1:8787/tripleoupstream
DockerNamespace: tripleoupstream
# Enable local Docker registry
DockerNamespaceIsRegistry: false
DockerAgentImage: heat-docker-agents:newton
# Docker containers
DockerNovaComputeImage: centos-binary-nova-compute:newton
DockerLibvirtImage: centos-binary-nova-libvirt:newton
DockerOpenvswitchImage: centos-binary-neutron-openvswitch-agent:newton
ComputeServices:
- OS::TripleO::Services::NovaCompute

22
overcloud.j2.yaml
View File

@ -255,6 +255,18 @@ resources:
EndpointMap: {get_attr: [EndpointMap, endpoint_map]}
DefaultPasswords: {get_attr: [DefaultPasswords, passwords]}
# Filter any null/None service_names which may be present due to mapping
# of services to OS::Heat::None
{{role.name}}ServiceNames:
type: OS::Heat::Value
depends_on: {{role.name}}ServiceChain
properties:
type: comma_delimited_list
value:
yaql:
expression: coalesce($.data, []).where($ != null)
data: {get_attr: [{{role.name}}ServiceChain, role_data, service_names]}
{{role.name}}HostsDeployment:
type: OS::Heat::StructuredDeployments
properties:
@ -305,7 +317,7 @@ resources:
StorageMgmtIpList: {get_attr: [{{role.name}}, storage_mgmt_ip_address]}
TenantIpList: {get_attr: [{{role.name}}, tenant_ip_address]}
ManagementIpList: {get_attr: [{{role.name}}, management_ip_address]}
EnabledServices: {get_attr: [{{role.name}}ServiceChain, role_data, service_names]}
EnabledServices: {get_attr: [{{role.name}}ServiceNames, value]}
ServiceNetMap: {get_attr: [ServiceNetMap, service_net_map_lower]}
ServiceHostnameList: {get_attr: [{{role.name}}, hostname]}
NetworkHostnameMap:
@ -361,8 +373,8 @@ resources:
{% for r in roles %}
- get_attr: [{{r.name}}ServiceChain, role_data, service_config_settings]
{% endfor %}
services: {get_attr: [{{role.name}}ServiceChain, role_data, service_names]}
ServiceNames: {get_attr: [{{role.name}}ServiceChain, role_data, service_names]}
services: {get_attr: [{{role.name}}ServiceNames, value]}
ServiceNames: {get_attr: [{{role.name}}ServiceNames, value]}
MonitoringSubscriptions: {get_attr: [{{role.name}}ServiceChain, role_data, monitoring_subscriptions]}
ServiceMetadataSettings: {get_attr: [{{role.name}}ServiceChain, role_data, service_metadata_settings]}
{% endfor %}
@ -396,7 +408,7 @@ resources:
list_join:
- ','
{% for role in roles %}
- {get_attr: [{{role.name}}ServiceChain, role_data, service_names]}
- {get_attr: [{{role.name}}ServiceNames, value]}
{% endfor %}
logging_groups:
yaql:
@ -646,7 +658,7 @@ outputs:
description: The services enabled on each role
value:
{% for role in roles %}
{{role.name}}: {get_attr: [{{role.name}}ServiceChain, role_data, service_names]}
{{role.name}}: {get_attr: [{{role.name}}ServiceNames, value]}
{% endfor %}
RoleData:
description: The configuration data associated with each role

2
puppet/puppet-steps.j2
View File

@ -21,7 +21,7 @@
{{role.name}}Config:
type: OS::TripleO::{{role.name}}Config
properties:
StepConfig: {get_param: [role_data, {{role.name}}, step_config]}
StepConfig: {list_join: ["\n", {get_param: [role_data, {{role.name}}, step_config]}]}
{% if role.name == 'Controller' %}
ControllerPrePuppet:

8
puppet/services/services.yaml
View File

@ -52,11 +52,7 @@ outputs:
description: Combined Role data for this set of services.
value:
service_names:
# Filter any null/None service_names which may be present due to mapping
# of services to OS::Heat::None
yaql:
expression: list($.data.s_names.where($ != null))
data: {s_names: {get_attr: [ServiceChain, role_data, service_name]}}
{get_attr: [ServiceChain, role_data, service_name]}
monitoring_subscriptions:
yaql:
expression: list($.data.role_data.where($ != null).select($.get('monitoring_subscription')).where($ != null))
@ -112,7 +108,7 @@ outputs:
yaql:
expression: $.data.role_data.where($ != null).select($.get('service_config_settings')).where($ != null).reduce($1.mergeWith($2), {})
data: {role_data: {get_attr: [ServiceChain, role_data]}}
step_config: {list_join: ["\n", {get_attr: [ServiceChain, role_data, step_config]}]}
step_config: {get_attr: [ServiceChain, role_data, step_config]}
upgrade_tasks:
yaql:
# Note we use distinct() here to filter any identical tasks, e.g yum update for all services