Custom containerized undercloud deployments

* Parametrize network settings for undercloud deploy to align with
  the undercloud install's undercloud.conf and nova-join's DNS
  config steps.
* Move the undercloud nameservers defaults to the extras-common.
* Support extra deploy args for 'openstack undercloud deploy'.
  Defaults to the heat-native mode, may be altered to containerized
  heat-all agents, and other args like keep-running, if wanted.
* Support custom t-h-t templates for underclouds as well (shares
  the custom t-h-t script and vars with the overcloud-prep-config)
* Template prerequisite packages installed by the
  undercloud-install.sh script. These should be moved to the
  tripleoclient's 'openstack undercloud deploy', eventually.
* Allow requested dev env packages to be updated by the
  undercloud-install.sh script (defaults to update nothing,
  which is backwards compatible). Note, this has nothing to the
  prerequisite packages.
* Document missing variables for the undercloud-deploy role.
* Document hacking dev branches for customized underclouds.

Related-bug: #1691467
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
Change-Id: Id32f492369dc74428bde8c4120308279cbba53fc
This commit is contained in:
Bogdan Dobrelya 2017-05-16 13:48:39 +02:00
parent 6a66338500
commit f2114bab76
8 changed files with 308 additions and 47 deletions

View File

@ -2,7 +2,10 @@
features:
- |
EXPERIMENTAL feature to install the undercloud with openstack heat
add support for a containerized undercloud.
add support for a containerized undercloud. It allows to update the
specified packages, install custom t-h-t templates and fetch work
in progress dependency changes. This can be done deployment runtime,
by an undercloud deploy script defined from a given template.
The feature is invoked by using the config file in
config/general_config/containers_undercloud_minimal.yml
e.g. ./quickstart.sh -c $working_dir/config/general_config/containers_undercloud_minimal.yml --tags all -R master-tripleo-ci -w $working_dir $VIRTHOST

View File

@ -6,5 +6,7 @@ containerized_overcloud_upgrade: false
overcloud_templates_path: /usr/share/openstack-tripleo-heat-templates
enable_tls_everywhere: false
undercloud_network_cidr: 192.168.24.0/24
undercloud_undercloud_nameservers:
- 8.8.8.8
timestamper_cmd: >-
| awk '{ print strftime("%Y-%m-%d %H:%M:%S |"), $0; fflush(); }'

View File

@ -0,0 +1,12 @@
---
- name: Create overcloud custom tht script
template:
src: "{{ overcloud_custom_tht_script }}"
dest: "{{ working_dir }}/overcloud-custom-tht-script.sh"
mode: 0755
- name: Checkout custom tht heat templates from src
shell: >
set -o pipefail &&
{{ working_dir }}/overcloud-custom-tht-script.sh 2>&1 {{ timestamper_cmd }} >
{{ overcloud_custom_tht_log }}

View File

@ -54,17 +54,12 @@
dest: "{{ working_dir }}"
when: baremetal_nic_configs is defined
- name: Create overcloud custom tht script
template:
src: "{{ overcloud_custom_tht_script }}"
dest: "{{ working_dir }}/overcloud-custom-tht-script.sh"
mode: 0755
- name: Checkout custom tht heat templates from src
shell: >
set -o pipefail &&
{{ working_dir }}/overcloud-custom-tht-script.sh 2>&1 {{ timestamper_cmd }} >
{{ overcloud_custom_tht_log }}
- name: Prepare custom t-h-t for overcloud deployment
include: custom_tht.yml
when:
- overcloud_templates_refspec is defined or overcloud_templates_branch is defined
- overcloud_templates_repo is defined
static: no
- name: Copy extra THT config files on the undercloud
copy:

View File

@ -14,7 +14,9 @@ Role Variables
- `undercloud_config_file`: <'undercloud.conf.j2'> -- the name of the jinja template
used as the base for the undercloud.conf
- `undercloud_install_script`: <'undercloud-install.j2'> -- the name of the jinja template
used as the base for the undercloud-install bash script
used as the base for the undercloud-install bash script and dev envs hacking. Note, the
defaults imply the 'openstack undercloud install' command will be invoked. See the undercloud
deployment methods section below for the alternative modes.
- `undercloud_install_log`: <'{{ working_dir }}/undercloud_install.log'> -- the full path
to the undercloud install log file.
- `network_environment_file`: <'network-environment.yaml.j2'> -- the name of the jinja template
@ -30,13 +32,12 @@ properly with the option --old-deploy-image
- `libvirt_uri`: <'qemu:///session'> -- the URI used by libvirt, by default tripleo-quickstart uses
user sessions to provide greater flexixiblity to our users. ** additional documentation ** is at
http://docs.openstack.org/developer/tripleo-quickstart/accessing-libvirt.html
- `undercloud_conf_extra`: "" -- extra options to be added to ~/undercloud.conf
- undercloud_undercloud_public_host: Sets up the 'undercloud_public_host'
parameter from undercloud.conf.
- undercloud_undercloud_admin_host: Sets up the 'undercloud_admin_host' from
undercloud.conf.
- `undercloud_undercloud_hostname`: Sets up the 'undercloud_hostname' value from
undercloud.conf.
- `undercloud_conf_extra`: <''> -- extra options to be added to ~/undercloud.conf
- `undercloud_extra_args`: <''> -- extra options for undercloud deploy command.
- `undercloud_update_packages`: <null> -- a string with a list of packages to update as dependencies for
your hacking setup. By defaults it updates nothing, which is backwards compatible.
- `undercloud_required_packages`: -- a string with a list of packages required for containerized
undercloud install (as a temporary solution, unless it is done by tripleo CLI)
- `undercloud_enable_ui`: Sets up the 'enable_ui' option in undercloud.conf.
It's undefined by default, however, the default value for this option in the
undercloud is true.
@ -69,22 +70,58 @@ Role Network Variables
----------------------
- `undercloud_network_cidr`: <'192.168.24.0/24'> -- the network cidr for the undercloud, note this
also currently the default cidr used in other CI environments for tripleo.
- `undercloud_network_gateway`: <a 1st host of the `undercloud_network_cidr`> -- Sets up the
`undercloud_network_gateway` parameter from undercloud.conf.
- `undercloud_local_ip`: <hostvars['undercloud'].undercloud_ip> -- Sets up the `local_ip`
parameter from an inventory. Must be belonging to the `undercloud_network_cidr`. It is used
by overcloud nodes to access the undercloud node via a routable
[ctlplane network]((https://docs.openstack.org/developer/tripleo-docs/advanced_deployment/network_isolation).
Note that the undercloud.conf takes the default value based on the `undercloud_network_cidr`
instead.
- `undercloud_undercloud_public_host`: <a 2nd host of the `undercloud_network_cidr`> -- Sets
up the 'undercloud_public_host' parameter from undercloud.conf. It is also used by overcloud
nodes to access the undercloud node via a VIP/hostname that resolves to a routable IP address.
- `undercloud_undercloud_admin_host`: <a 3rd host of the `undercloud_network_cidr`> -- Sets
up the 'undercloud_admin_host' from undercloud.conf. Note, use the `undercloud_admin_vip`
instead for Mitaka/Newton releases.
- `undercloud_dhcp_start`: <a 5th host of the `undercloud_network_cidr`> -- Sets
up the 'undercloud_dhcp_start' from undercloud.conf.
- `undercloud_dhcp_end`: <a 30th host of the `undercloud_network_cidr`> -- Sets
up the 'undercloud_dhcp_end' from undercloud.conf.
- `undercloud_undercloud_nameservers`: <['8.8.8.8']> -- Sets up the 'undercloud_undercloud_nameservers'
from undercloud.conf. May be a string or a sequence. Only the last item goes for
the undercloud deploy command.
- `undercloud_undercloud_hostname`: Sets up the 'undercloud_hostname' value from undercloud.conf.
Note, use the `undercloud_public_vip` instead for Mitaka/Newton releases.
- `undercloud_heat_public_endpoints`: <False> -- when the ctlplane network is not routable
from overcloud nodes, for example pre-provisioned
[deployed servers](https://docs.openstack.org/developer/tripleo-docs/advanced_deployment/deployed_server.html#undercloud),
the ``undercloud deploy --local_ip`` (and `local_ip` in the undercloud.conf)
may not be used. Enable this variable instead. Doing so changes the heat endpoint
type from the default internal to public and changes the signaling method to use
TempURLs from OpenStack Object Storage (swift).
Undercloud deployment methods
-----------------------------
By default, the undercloud uses the file coming from the
hieradata_override_file_classic_undercloud variable to set the hieradata
By default, the undercloud uses ``openstack undercloud install`` command,
`{{working_dir}}/undercloud.conf` config and the file coming from the
``hieradata_override_file_classic_undercloud`` variable to set the hieradata
overrides that will be used for the puppet deployment.
However, there exists another method for setting up the undercloud based on
containers. This method can be used by setting the "containerized_undercloud"
variable to 'true' and it's currently experimental. Instead of using the
"classic" way to set up hieradata, it uses the file coming from the
"hieradata_override_file_t_h_t_undercloud" variable. As the name hints, this
However, there exists another method, explained below, for setting up the
undercloud based on containers with the ``openstack undercloud deploy`` command.
This method can be used by setting the ``containerized_undercloud`` variable to
'true' and it's currently experimental. Instead of using the "classic" way to set
up hieradata, it uses the file coming from the
``hieradata_override_file_t_h_t_undercloud`` variable. As the name hints, this
deployment method is based on tripleo-heat-templates, and thus, the way of
overriding hieradata for it is similar as we do for the overcloud.
Note, the containerized undercloud is a hacking mode for developers, that allows
to test containerized components undercloud, with custom, like very minimal,
setup layouts. Please do not expect more than that when using it.
Example Playbook
----------------
@ -98,3 +135,164 @@ Sample playbook to call the role
roles:
- undercloud-deploy
```
Containerized dev environments (experimental feature)
-----------------------------------------------------
Firstly, update or create a custom undercloud install script template. It can be
found under the locally checked out tripleo-quickstart-extra repo (or at a remote
host, if you prefer to apply ansible playbooks from a non local virthost). The
template is located by the
`roles/undercloud-deploy/templates/undercloud-deploy-my-dev.sh.j2` path.
Once changes are done, update the ``undercloud_install_script`` var, like:
```yaml
undercloud_install_script: undercloud-deploy-dev.sh.j2
```
or use the updated template:
```
undercloud_install_script: undercloud-deploy.sh.j2
```
Note, it is important to change the default value for the ``undercloud_install_script``
to switch from the classic 'install' to the experimental 'deploy' method.
Secondly, you may want to refer an unpackaged t-h-t change refspec or a dev
branch. To do so, override the custom t-h-t repo and branch/refspec
for t-h-t templates to be fetched, for example:
```yaml
overcloud_templates_repo: https://github.com/johndoe/tripleo-heat-templates
overcloud_templates_branch: dev
```
Note, these vars are shared with the overcloud role vars and point to
the same templates path, branch/refspec and repo. The templates path should be
R/W accessible by the given `non_root_user` without sudoing. So it is better
off using the home dir or tmp dirs with sticky bits.
Also note that you should normaly rely on the openstack-tripleo-heat-templates
package. The custom t-h-t repo vars may break that package and should be used with
caution.
You can as well specify a list of the packages to be updated to the most
recent versions. For example:
```yaml
undercloud_update_packages: >-
openstack-tripleo-common
openstack-tripleo-heat-templates
puppet-tripleo
python-tripleoclient
python-heat-agent*
```
Or use the ``undercloud_update_packages: "'*'"`` to update all packages
(may take a long).
Note, checkout/install steps for the remaining yet unpackaged custom changes
like dev branches for puppet modules, tripleo client, heat agent hooks, need
to be covered in the custom ``undercloud-install.sh`` script body (rendered
from a given in the ``undercloud_install_script`` template file name).
The last step is to define the ``undercloud_extra_args`` as needed. For
example you may want to deploy:
* an optional non containerized Octavia,
* containerized Keystone,
* optional containerized Etcd,
with disabled telemetry, docker images from `docker.io/tripleoupstream`,
debug logs for services and puppet, given a decent timeout, and keeping
the ephemeral undercloud heat agent running for debug purposes:
```yaml
undercloud_extra_args: >-
-e {{overcloud_templates_path}}/environments/disable-telemetry.yaml
-e {{overcloud_templates_path}}/environments/docker-minimal.yaml
-e {{overcloud_templates_path}}/environments/services-docker/etcd.yaml
-e {{overcloud_templates_path}}/environments/services/octavia.yaml
-e {{overcloud_templates_path}}/environments/debug.yaml
-e {{overcloud_templates_path}}/environments/config-debug.yaml
--timeout 60
--keep-running
```
Where the t-h-t's `environments/docker-minimal.yaml` is like:
```
resource_registry:
OS::TripleO::Services::Docker: ../puppet/services/docker.yaml
OS::TripleO::Services::Keystone: ../docker/services/keystone.yaml
OS::TripleO::PostDeploySteps: ../docker/post.yaml
OS::TripleO::PostUpgradeSteps: ../docker/post-upgrade.yaml
OS::TripleO::Services: ../docker/services/services.yaml
parameter_defaults:
DockerNamespace: tripleoupstream
DockerNamespaceIsRegistry: false
ComputeServices: {}
SwiftCeilometerPipelineEnabled: false
```
Note, this template complements the default t-h-t's `environments/docker.yaml`
setup that deploys everything in containers. The default template for
the ``undercloud-install.sh`` also provides an additional set of required
services used for underlcoud to deploy overclouds (Ironic, Zaqar, MongoDB,
Mistral). If you need a lightweight undercloud, make sure your custom
undercloud installation script template omits those services and the
`environments/docker.yaml` defaults.
Also note, if you want to override the undercloud roles data, you should
provide a custom roles file and put it in place of the default
`./roles_data_undercloud.yaml` by the given ``overcloud_templates_path``.
This can be done with a ``cp`` command in the custom undercloud deploy
script. For the example above, custom undercloud roles may look like:
```
- name: Undercloud
CountDefault: 1
disable_constraints: True
tags:
- primary
- controller
ServicesDefault:
- OS::TripleO::Services::Etcd
- OS::TripleO::Services::Keystone
- OS::TripleO::Services::OctaviaApi
- OS::TripleO::Services::OctaviaHealthManager
- OS::TripleO::Services::OctaviaHousekeeping
- OS::TripleO::Services::OctaviaWorker
```
And an example playbook to call the role is:
```yaml
# Deploy the undercloud
- name: Deploy undercloud (experimental)
hosts: undercloud
gather_facts: no
vars:
containerized_undercloud: true
undercloud_install_script: undercloud-deploy.sh.j2
overcloud_templates_repo: https://github.com/johndoe/tripleo-heat-templates
overcloud_templates_branch: dev
undercloud_update_packages: >-
openstack-tripleo-common
openstack-tripleo-heat-templates
puppet-tripleo
python-tripleoclient
python-heat-agent*
undercloud_extra_args: >-
-e {{overcloud_templates_path}}/environments/disable-telemetry.yaml
-e {{overcloud_templates_path}}/environments/docker-minimal.yaml
-e {{overcloud_templates_path}}/environments/services-docker/etcd.yaml
-e {{overcloud_templates_path}}/environments/services/octavia.yaml
-e {{overcloud_templates_path}}/environments/debug.yaml
-e {{overcloud_templates_path}}/environments/config-debug.yaml
--timeout 60
--keep-running
roles:
- undercloud-deploy
```

View File

@ -9,6 +9,31 @@ network_environment_file: network-environment.yaml.j2
# containerized_undercloud is an EXPERIMENTAL FEATURE
containerized_undercloud: false
undercloud_install_script: undercloud-install.sh.j2
# TODO(bogdando) move these to the client's 'openstack undercloud deploy'
undercloud_required_packages: >-
openstack-heat-api
openstack-heat-engine
openstack-heat-monolith
python-heat-agent
python-heat-agent-apply-config
python-heat-agent-hiera
python-heat-agent-puppet
python-heat-agent-docker-cmd
python-heat-agent-json-file
python-ipaddr
python-tripleoclient
docker
openvswitch
openstack-puppet-modules
openstack-kolla
yum-plugin-priorities
openstack-tripleo-common
openstack-tripleo-heat-templates
deltarpm
# defines extra args for the undercloud deploy command
undercloud_extra_args: >-
--heat-native
# The undercloud can also be deployed with heat
# thus requiring a t-h-t driven undercloud *and* the classic undercloud

View File

@ -45,3 +45,13 @@
src: "{{ undercloud_install_script }}"
dest: "{{ working_dir }}/undercloud-install.sh"
mode: 0755
- name: Prepare custom t-h-t branch/refspec
include_role:
name: overcloud-prep-config
tasks_from: custom_tht
when:
- overcloud_templates_branch is defined or overcloud_templates_refspec is defined
- overcloud_templates_repo is defined
- overcloud_templates_path is defined
static: no

View File

@ -4,7 +4,7 @@ set -eux
### --start_docs
## Deploying the undercloud
## =========================
## ========================
## ::
## * Export the required environment variables.
@ -28,28 +28,32 @@ resource_registry:
parameter_defaults:
UndercloudNetworkCidr: {{undercloud_network_cidr}}
UndercloudDhcpRangeStart: {{undercloud_dhcp_start|default(
undercloud_network_cidr|nthhost(5))}}
UndercloudDhcpRangeEnd: {{undercloud_dhcp_end|default(
undercloud_network_cidr|nthhost(30))}}
UndercloudNetworkGateway: {{undercloud_network_gateway|default(
undercloud_network_cidr|nthhost(1))}}
{% if undercloud_undercloud_nameservers is string %}
UndercloudNameserver: {{split(undercloud_undercloud_nameservers, ',')[0]}}
{% elif undercloud_undercloud_nameservers is sequence %}
UndercloudNameserver: {{undercloud_undercloud_nameservers[-1]}}
{% endif %}
NeutronServicePlugins: ""
EOF_CAT
## * Install the prerequisite rpms.
## ::
sudo yum install -y \
openstack-heat-api \
openstack-heat-engine \
openstack-heat-monolith \
python-heat-agent \
python-heat-agent-apply-config \
python-heat-agent-hiera \
python-heat-agent-puppet \
python-heat-agent-docker-cmd \
python-heat-agent-json-file \
python-ipaddr \
python-tripleoclient \
docker \
openvswitch \
openstack-puppet-modules \
openstack-kolla
# TODO(bogdando) move this step to the tripleo client
sudo yum install -y {{ undercloud_required_packages }}
## * Update requested packages.
## ::
{% if undercloud_update_packages | default(false) %}
sudo yum update -y {{ undercloud_update_packages }}
{% endif %}
## * Start the required services.
## ::
@ -57,9 +61,21 @@ sudo yum install -y \
sudo systemctl start openvswitch
sudo systemctl start docker #FIXME: do these in the installer?
## * Update the puppet modules.
## * Update the puppet modules or install custom modules.
## NOTE: breaks installed packages, is a pure hack in for dev envs only!
## ::
# Amend "foo/bar" and uncomment as needed:
#puppet module install foo/bar || puppet module upgrade foo/bar
#MODULE_PATH="{{ working_dir }}/.puppet/code/modules"
#rm -rf "${MODULE_PATH}/bar"
#mkdir -p "${MODULE_PATH}"
#cd "${MODULE_PATH}"
#git clone -b dev https://github.com/foo/puppet-bar bar
# Copy installed/cloned modules and make a symlink
#sudo rsync -avxH "${MODULE_PATH}" /usr/share/openstack-puppet/
sudo mkdir -p /etc/puppet/modules/
sudo ln -f -s /usr/share/openstack-puppet/modules/* /etc/puppet/modules/
@ -74,8 +90,7 @@ undercloud_local_ip={{ hostvars['undercloud'].undercloud_ip }}
sudo openstack undercloud deploy \
--templates={{overcloud_templates_path}} \
--local-ip=$undercloud_local_ip \
--heat-native \
-e /home/stack/custom_undercloud_deploy.yaml \
-e {{ working_dir }}/custom_undercloud_deploy.yaml \
{% if containerized_undercloud|bool %}
-e {{overcloud_templates_path}}/environments/services-docker/ironic.yaml \
-e {{overcloud_templates_path}}/environments/services-docker/mistral.yaml \
@ -83,6 +98,7 @@ sudo openstack undercloud deploy \
-e {{overcloud_templates_path}}/environments/docker.yaml \
-e {{overcloud_templates_path}}/environments/mongodb-nojournal.yaml \
{% endif %}
{{ undercloud_extra_args }} \
-e {{ working_dir }}/hieradata-overrides-t-h-t-undercloud.yaml
## * Copy the Openstack credentials file to the non root user.