diff --git a/ansible/group_vars/all/globals b/ansible/group_vars/all/globals index b8c830d0c..274c123e0 100644 --- a/ansible/group_vars/all/globals +++ b/ansible/group_vars/all/globals @@ -28,6 +28,6 @@ virtualenv_path: "{{ base_path ~ '/venvs' }}" ############################################################################### # User configuration. -# User with which to access seed and controller nodes. This user will be -# created if it does not exist. +# User with which to access remote hosts. This user will be created if it does +# not exist. kayobe_ansible_user: "stack" diff --git a/doc/source/configuration/hosts.rst b/doc/source/configuration/hosts.rst new file mode 100644 index 000000000..771c88fa8 --- /dev/null +++ b/doc/source/configuration/hosts.rst @@ -0,0 +1,565 @@ +================== +Host Configuration +================== + +This section covers configuration of hosts. It does not cover configuration or +deployment of containers. Hosts that are configured by Kayobe include: + +* Seed hypervisor (``kayobe seed hypervisor host configure``) +* Seed (``kayobe seed host configure``) +* Overcloud (``kayobe overcloud host configure``) + +Unless otherwise stated, all host configuration described here is applied to +each of these types of host. + +Configuration Location +====================== + +Some host configuration options are set via global variables, and others have a +variable for each type of host. The latter variables are included in the +following files under ``${KAYOBE_CONFIG_PATH}``: + +* ``seed-hypervisor.yml`` +* ``seed.yml`` +* ``compute.yml`` +* ``controller.yml`` +* ``monitoring.yml`` +* ``storage.yml`` + +Note that any variable may be set on a per-host or per-group basis, by using +inventory host or group variables - these delineations are for convenience. + +Paths +===== + +Several directories are used by Kayobe on the remote hosts. There is a +hierarchy of variables in ``${KAYOBE_CONFIG_PATH}/globals.yml`` that can be +used to control where these are located. + +* ``base_path`` (default ``/opt/kayobe/``) sets the default base path for + various directories. +* ``config_path`` (default ``{{ base_path }}/etc``) is a path in which to store + configuration files. +* ``image_cache_path`` (default ``{{ base_path }}/images``) is a path in which + to cache downloaded or built images. +* ``source_checkout_path`` (default ``{{ base_path }}/src``) is a path into + which to store clones of source code repositories. +* ``virtualenv_path`` (default ``{{ base_path }}/venvs``) is a path in which to + create Python virtual environments. + +SSH Known Hosts +=============== + +While strictly this configuration is applied to the Ansible control host +(``localhost``), it is applied during the ``host configure`` commands. +The ``ansible_host`` of each host is added as an SSH known host. This is +typically the host's IP address on the admin network (``admin_oc_net_name``), +as defined in ``${KAYOBE_CONFIG_PATH}/network-allocation.yml`` (see +:ref:`configuration-network-ip-allocation`). + +Kayobe User Bootstrapping +========================= + +Kayobe uses a user account defined by the ``kayobe_ansible_user`` variable (in +``${KAYOBE_CONFIG_PATH}/globals.yml``) for remote SSH access. By default, this +is ``stack``. + +Typically, the image used to provision these hosts will not include this user +account, so Kayobe performs a bootstrapping step to create it, as a different +user. In cloud images, there is often a user named after the OS distro, e.g. +``centos`` or ``ubuntu``. This user defaults to the name of the user running +Kayobe, but may be set via the following variables: + +* ``seed_hypervisor_bootstrap_user`` +* ``seed_bootstrap_user`` +* ``compute_bootstrap_user`` +* ``controller_bootstrap_user`` +* ``monitoring_bootstrap_user`` +* ``storage_bootstrap_user`` + +For example, to set the bootstrap user for controllers to ``centos``: + +.. code-block:: yaml + :caption: ``controllers.yml`` + + controller_bootstrap_user: centos + +PyPI Mirror +=========== + +Kayobe supports configuration of a PyPI mirror, via variables in +``${KAYOBE_CONFIG_PATH}/pip.yml``. This functionality is enabled by setting the +``pip_local_mirror`` variable to ``true``. + +Kayobe will generate configuration for +``pip`` and ``easy_install`` to use the mirror, for the list of users defined +by ``pip_applicable_users`` (default ``kayobe_ansible_user`` and ``root``), in +addition to the user used for Kolla Ansible (``kolla_ansible_user``). The +mirror URL is configured via ``pip_index_url``, and ``pip_trusted_hosts`` is a +list of 'trusted' hosts, for which SSL verification will be disabled. + +For example, to configure use of the test PyPI mirror at +https://test.pypi.org/simple/: + +.. code-block:: yaml + :caption: ``pip.yml`` + + pip_local_mirror: true + pip_index_url: https://test.pypi.org/simple/ + +Kayobe Remote Virtual Environment +================================= + +By default, Ansible executes modules remotely using the system python +interpreter, even if the Ansible control process is executed from within a +virtual environment (unless the ``local`` connection plugin is used). +This is not ideal if there are python dependencies that must be installed +with isolation from the system python packages. Ansible can be configured to +use a virtualenv by setting the host variable ``ansible_python_interpreter`` +to a path to a python interpreter in an existing virtual environment. + +If kayobe detects that ``ansible_python_interpreter`` is set and references a +virtual environment, it will create the virtual environment if it does not +exist. Typically this variable should be set via a group variable in the +inventory for hosts in the ``seed``, ``seed-hypervisor``, and/or ``overcloud`` +groups. + +The default Kayobe configuration in the ``kayobe-config`` repository sets +``ansible_python_interpreter`` to ``{{ virtualenv_path }}/kayobe/bin/python`` +for the ``seed``, ``seed-hypervisor``, and ``overcloud`` groups. + +Disk Wiping +=========== + +Using hosts that may have stale data on their disks could affect the deployment +of the cloud. This is not a configuration option, since it should only be +performed once to avoid losing useful data. It is triggered by passing the +``--wipe-disks`` argument to the ``host configure`` commands. + +Users and Groups +================ + +Linux user accounts and groups can be configured using the ``users_default`` +variable in ``${KAYOBE_CONFIG_PATH}/users.yml``. The format of the list is +that used by the ``users`` variable of the `singleplatform-eng.users +`__ role. The following +variables can be used to set the users for specific types of hosts: + +* ``seed_hypervisor_users`` +* ``seed_users`` +* ``compute_users`` +* ``controller_users`` +* ``monitoring_users`` +* ``storage_users`` + +In the following example, a single user named ``bob`` is created. A password +hash has been generated via ``mkpasswd --method=sha-512``. The user is added to +the ``wheel`` group, and an SSH key is authorised. The SSH public key should be +added to the Kayobe configuration. + +.. code-block:: yaml + :caption: ``users.yml`` + + users_default: + - username: bob + name: Bob + password: "$6$wJt9MLWrHlWN8$oXJHbdaslm9guD5EC3Dry1mphuqF9NPeQ43OXk3cXZa2ze/F9FOTxm2KvvDkbdxBDs7ouwdiLTUJ1Ff40.cFU." + groups: + - wheel + append: True + ssh_key: + - "{{ lookup('file', kayobe_config_path ~ '/ssh-keys/id_rsa_bob.pub') }}" + +Package Repositories +==================== + +Kayobe supports configuration of package repositories via Yum, via variables in +``${KAYOBE_CONFIG_PATH}/yum.yml``. + +CentOS and EPEL Mirrors +----------------------- + +CentOS and EPEL mirrors can be enabled by setting ``yum_use_local_mirror`` to +``true``. CentOS repository mirrors are configured via the following +variables: + +* ``yum_centos_mirror_host`` (default ``mirror.centos.org``) is the mirror + hostname. +* ``yum_centos_mirror_directory`` (default ``centos``) is a directory on the + mirror in which repositories may be accessed. + +EPEL repository mirrors are configured via the following variables: + +* ``yum_epel_mirror_host`` (default ``download.fedoraproject.org``) is the + mirror hostname. +* ``yum_epel_mirror_directory`` (default ``pub/epel``) is a directory on the + mirror in which repositories may be accessed. + +For example, to configure CentOS and EPEL mirrors at mirror.example.com: + +.. code-block:: yaml + :caption: ``yum.yml`` + + yum_use_local_mirror: true + yum_centos_mirror_host: mirror.example.com + yum_epel_mirror_host: mirror.example.com + +Custom Yum Repositories +----------------------- + +It is also possible to configure a list of custom Yum repositories via the +``yum_custom_repos`` variable. The format is a dict/map, with repository names +mapping to a dict/map of arguments to pass to the Ansible ``yum`` module. + +For example, the following configuration defines a single Yum repository called +``widgets``. + +.. code-block:: yaml + :caption: ``yum.yml`` + + yum_custom_repos: + widgets: + baseurl: http://example.com/repo + file: widgets + gpgkey: http://example.com/gpgkey + gpgcheck: yes + +Disabling EPEL +-------------- + +It is possible to disable the EPEL Yum repository by setting +``yum_install_epel`` to ``false``. + +SELinux +======= + +SELinux is not supported by Kolla Ansible currently, so it is disabled by +Kayobe. If necessary, Kayobe will reboot systems in order to apply a change to +the SELinux configuration. The timeout for waiting for systems to reboot is +``disable_selinux_reboot_timeout``. Alternatively, the reboot may be avoided by +setting ``disable_selinux_do_reboot`` to ``false``. + +Network Configuration +===================== + +Configuration of host networking is covered in depth in +:ref:`configuration-network`. + +Sysctls +======= + +Arbitrary ``sysctl`` configuration can be applied to hosts. The variable format +is a dict/map, mapping parameter names to their required values. The following +variables can be used to set ``sysctl`` configuration specific types of hosts: + +* ``seed_hypervisor_sysctl_parameters`` +* ``seed_sysctl_parameters`` +* ``compute_sysctl_parameters`` +* ``controller_sysctl_parameters`` +* ``monitoring_sysctl_parameters`` +* ``storage_sysctl_parameters`` + +For example, to set the ``net.ipv4.ip_forward`` parameter to ``1`` on controllers: + +.. code-block:: yaml + :caption: ``controllers.yml`` + + controller_sysctl_parameters: + net.ipv4.ip_forward: 1 + +Disable cloud-init +================== + +cloud-init is a popular service for performing system bootstrapping. If you are +not using cloud-init, this section can be skipped. + +If using the seed's Bifrost service to provision the control plane hosts, the +use of cloud-init may be configured via the ``kolla_bifrost_dib_init_element`` +variable. + +cloud-init searches for network configuration in order of increasing +precedence; each item overriding the previous. In some cases, on subsequent +boots cloud-init can automatically reconfigure network interfaces and cause +some issues in network configuration. To disable cloud-init from running after +the initial server bootstrapping, set ``disable_cloud_init`` to ``true`` in +``${KAYOBE_CONFIG_PATH}/overcloud.yml``. + +Disable Glean +============= + +The ``glean`` service can be used to perform system bootstrapping, serving a +similar role to ``cloud-init``. If you are not using ``glean``, this section +can be skipped. + +If using the seed's Bifrost service to provision the control plane hosts, the +use of ``glean`` may be configured via the ``kolla_bifrost_dib_init_element`` +variable. + +After the initial server bootstrapping, the glean service can cause problems as +it attempts to enable all network interfaces, which can lead to timeouts while +booting. To avoid this, the ``glean`` service is disabled. Additionally, any +network interface configuration files generated by ``glean`` and not +overwritten by Kayobe are removed. + +Timezone +======== + +The timezone can be configured via the ``timezone`` variable in +``${KAYOBE_CONFIG_PATH}/ntp.yml``. The value must be a valid Linux +timezone. For example: + +.. code-block:: yaml + :caption: ``ntp.yml`` + + timezone: Europe/London + +NTP +=== + +Network Time Protocol (NTP) may be configured via variables in +``${KAYOBE_CONFIG_PATH}/ntp.yml``. The list of NTP servers is +configured via ``ntp_config_server``, and by default the ``pool.ntp.org`` +servers are used. A list of restrictions may be added via +``ntp_config_restrict``, and a list of interfaces to listen on via +``ntp_config_listen``. Other options and their default values may be found in +the `resmo.ntp `__ Ansible role. + +.. code-block:: yaml + :caption: ``ntp.yml`` + + ntp_config_server: + - 1.ubuntu.pool.ntp.org + - 2.ubuntu.pool.ntp.org + + ntp_config_restrict: + - '-4 default kod notrap nomodify nopeer noquery' + + ntp_config_listen: + - eth0 + +The NTP service may be disabled as follows: + +.. code-block:: yaml + :caption: ``ntp.yml`` + + ntp_package_state: absent + ntp_service_state: stopped + ntp_service_enabled: false + +.. _configuration-hosts-lvm: + +LVM +=== + +Logical Volume Manager (LVM) physical volumes, volume groups, and logical +volumes may be configured via the ``lvm_groups`` variable. For convenience, +this is mapped to the following variables: + +* ``seed_lvm_groups`` +* ``compute_lvm_groups`` +* ``controller_lvm_groups`` +* ``monitoring_lvm_groups`` +* ``storage_lvm_groups`` + +Note that LVM is not configured on the seed hypervisor. + +The format of these variables is as defined by the ``lvm_groups`` variable of +the `mrlesmithjr.manage-lvm +`__ Ansible role. + +LVM for Docker +-------------- + +The default LVM configuration is optimised for the ``devicemapper`` Docker +storage driver, which requires a thin provisioned LVM volume. A second logical +volume is used for storing Docker volume data, mounted at +``/var/lib/docker/volumes``. Both logical volumes are created from a single +``data`` volume group. + +To use this configuration, a list of disks must be configured via the following +variables: + +* ``seed_lvm_group_data_disks`` +* ``compute_lvm_group_data_disks`` +* ``controller_lvm_group_data_disks`` +* ``monitoring_lvm_group_data_disks`` +* ``storage_lvm_group_data_disks`` + +For example, to configure two of the seed's disks for use by LVM: + +.. code-block:: yaml + :caption: ``seed.yml`` + + seed_lvm_group_data_disks: + - /dev/sdb + - /dev/sdc + +The Docker volumes LVM volume is assigned a size given by the following +variables, with a default value of 75% (of the volume group's capacity): + +* ``seed_lvm_group_data_lv_docker_volumes_size`` +* ``compute_lvm_group_data_lv_docker_volumes_size`` +* ``controller_lvm_group_data_lv_docker_volumes_size`` +* ``monitoring_lvm_group_data_lv_docker_volumes_size`` +* ``storage_lvm_group_data_lv_docker_volumes_size`` + +If using a Docker storage driver other than ``devicemapper``, the remaining 25% +of the volume group can be used for Docker volume data. In this case, the LVM +volume's size can be increased to 100%: + +.. code-block:: yaml + :caption: ``controllers.yml`` + + controller_lvm_group_data_lv_docker_volumes_size: 100% + +If using a Docker storage driver other than ``devicemapper``, it is possible to +avoid using LVM entirely, thus avoiding the requirement for multiple disks. In +this case, set the appropriate ``_lvm_groups`` variable to an empty list: + +.. code-block:: yaml + :caption: ``storage.yml`` + + storage_lvm_groups: [] + +Custom LVM +---------- + +To define additional logical logical volumes in the default ``data`` volume +group, modify one of the following variables: + +* ``seed_lvm_group_data_lvs`` +* ``compute_lvm_group_data_lvs`` +* ``controller_lvm_group_data_lvs`` +* ``monitoring_lvm_group_data_lvs`` +* ``storage_lvm_group_data_lvs`` + +Include the variable ``_lvm_group_data_lv_docker_volumes`` in the list to +include the LVM volume for Docker volume data: + +.. code-block:: yaml + :caption: ``monitoring.yml`` + + monitoring_lvm_group_data_lvs: + - "{{ monitoring_lvm_group_data_lv_docker_volumes }}" + - lvname: other-vol + size: 1% + create: true + filesystem: ext4 + mount: true + mntp: /path/to/mount + +It is possible to define additional LVM volume groups via the following +variables: + +* ``seed_lvm_groups_extra`` +* ``compute_lvm_groups_extra`` +* ``controller_lvm_groups_extra`` +* ``monitoring_lvm_groups_extra`` +* ``storage_lvm_groups_extra`` + +For example: + +.. code-block:: yaml + :caption: ``compute.yml`` + + compute_lvm_groups_extra: + - vgname: other-vg + disks: /dev/sdb + create: true + lvnames: + - lvname: other-vol + size: 100% + create: true + mount: false + +Alternatively, replace the entire volume group list via one of the +``_lvm_groups`` variables to replace the default configuration with a +custom one. + +.. code-block:: yaml + :caption: ``controllers.yml`` + + controller_lvm_groups: + - vgname: only-vg + disks: /dev/sdb + create: true + lvnames: + - lvname: only-vol + size: 100% + create: true + mount: false + +Kolla-ansible bootstrap-servers +=============================== + +Kolla Ansible provides some host configuration functionality via the +``bootstrap-servers`` command, which may be leveraged by Kayobe. Due to the +bootstrapping nature of the command, Kayobe uses ``kayobe_ansible_user`` to +execute it, and uses the Kayobe remote Python virtual environment (or the +system Python interpreter if no virtual environment is in use). + +See the `Kolla Ansible documentation +`__ +for more information on the functions performed by this command, and how to +configure it. + +Kolla-ansible Remote Virtual Environment +======================================== + +See :ref:`configuration-kolla-ansible-venv` for information about remote Python +virtual environments for Kolla Ansible. + +Docker Engine +============= + +Docker engine configuration is applied by both Kayobe and Kolla Ansible (during +bootstrap-servers). + +The ``docker_storage_driver`` variable sets the Docker storage driver, and by +default the ``devicemapper`` driver is used. If using this driver, see +:ref:`configuration-hosts-lvm` for information about configuring LVM for +Docker. + +Various options are defined in ``${KAYOBE_CONFIG_PATH}/docker.yml`` +for configuring the ``devicemapper`` storage. + +A private Docker registry may be configured via ``docker_registry``, with a +Certificate Authority (CA) file configured via ``docker_registry_ca``. + +To use one or more Docker Registry mirrors, use the ``docker_registry_mirrors`` +variable. + +If using an MTU other than 1500, ``docker_daemon_mtu`` can be used to configure +this. This setting does not apply to containers using ``net=host`` (as Kolla +Ansible's containers do), but may be necessary when building images. + +Docker's live restore feature can be configured via +``docker_daemon_live_restore``, although it is disabled by default due to +issues observed. + +Ceph Block Devices +================== + +If using Kolla Ansible to deploy Ceph, some preparation of block devices is +required. The list of disks to configure for use by Ceph is specified via +``ceph_disks``. This is mapped to the following variables: + +* ``compute_ceph_disks`` +* ``controller_ceph_disks`` +* ``storage_ceph_disks`` + +The format of the variable is a list of dict/mapping objects. Each mapping +should contain an ``osd`` item that defines the full path to a block device to +use for data. Optionally, each mapping may contain a ``journal`` item that +specifies the full path to a block device to use for journal data. + +The following example defines two OSDs for use by controllers, one of which has +a journal: + +.. code-block:: yaml + :caption: ``controller.yml`` + + controller_ceph_disks: + - osd: /dev/sdb + - osd: /dev/sdc + journal: /dev/sdd diff --git a/doc/source/configuration/index.rst b/doc/source/configuration/index.rst index fa8102f29..99cf36dcc 100644 --- a/doc/source/configuration/index.rst +++ b/doc/source/configuration/index.rst @@ -8,4 +8,5 @@ Configuration Guide kayobe physical-network network + hosts kolla-ansible diff --git a/doc/source/configuration/kayobe.rst b/doc/source/configuration/kayobe.rst index d3159231c..d18ce534f 100644 --- a/doc/source/configuration/kayobe.rst +++ b/doc/source/configuration/kayobe.rst @@ -116,23 +116,3 @@ configuration files may be encrypted. Since encryption can make working with Kayobe difficult, it is recommended to follow `best practice `_, adding a layer of indirection and using encryption only where necessary. - -Remote Execution Environment ----------------------------- - -By default, ansible executes modules remotely using the system python -interpreter, even if the ansible control process is executed from within a -virtual environment (unless the ``local`` connection plugin is used). -This is not ideal if there are python dependencies that must be installed -without isolation from the system python packages. Ansible can be configured to -use a virtualenv by setting the host variable ``ansible_python_interpreter`` -to a path to a python interpreter in an existing virtual environment. - -If kayobe detects that ``ansible_python_interpreter`` is set and references a -virtual environment, it will create the virtual environment if it does not -exist. Typically this variable should be set via a group variable for hosts in -the ``seed``, ``seed-hypervisor``, and/or ``overcloud`` groups. - -The default Kayobe configuration in the ``kayobe-config`` repository sets -``ansible_python_interpreter`` to ``{{ virtualenv_path }}/kayobe/bin/python`` -for the ``seed``, ``seed-hypervisor``, and ``overcloud`` groups. diff --git a/doc/source/configuration/kolla-ansible.rst b/doc/source/configuration/kolla-ansible.rst index 056863f90..b80b8e253 100644 --- a/doc/source/configuration/kolla-ansible.rst +++ b/doc/source/configuration/kolla-ansible.rst @@ -43,6 +43,8 @@ lookup plugin kolla_ansible_venv_extra_requirements: - "hvac" +.. _configuration-kolla-ansible-venv: + Remote Execution Environment ============================ @@ -50,7 +52,7 @@ By default, ansible executes modules remotely using the system python interpreter, even if the ansible control process is executed from within a virtual environment (unless the ``local`` connection plugin is used). This is not ideal if there are python dependencies that must be installed -without isolation from the system python packages. Ansible can be configured to +with isolation from the system python packages. Ansible can be configured to use a virtualenv by setting the host variable ``ansible_python_interpreter`` to a path to a python interpreter in an existing virtual environment. diff --git a/doc/source/configuration/network.rst b/doc/source/configuration/network.rst index ebcc24269..a674d8513 100644 --- a/doc/source/configuration/network.rst +++ b/doc/source/configuration/network.rst @@ -1,3 +1,5 @@ +.. _configuration-network: + ===================== Network Configuration ===================== @@ -39,6 +41,10 @@ supported: VLAN ID. ``mtu`` Maximum Transmission Unit (MTU). +``vip_address`` + Virtual IP address (VIP) used by API services on this network. +``fqdn`` + Fully Qualified Domain Name (FQDN) used by API services on this network. ``routes`` List of static IP routes. Each item should be a dict containing the item ``cidr``, and optionally ``gateway`` and ``table``. ``cidr`` is the CIDR @@ -85,6 +91,36 @@ This gateway will be configured on all hosts to which the network is mapped. Note that configuring multiple IP gateways on a single host will lead to unpredictable results. +Configuring an API Virtual IP Address +------------------------------------- + +A virtual IP (VIP) address may be configured for use by Kolla Ansible on the +internal and external networks, on which the API services will be exposed. +The variable will be passed through to the ``kolla_internal_vip_address`` or +``kolla_external_vip_address`` Kolla Ansible variable. + +To configure a network called ``example`` with a VIP at ``10.0.0.2``: + +.. code-block:: yaml + :caption: ``networks.yml`` + + example_vip_address: 10.0.0.2 + +Configuring an API Fully Qualified Domain Name +---------------------------------------------- + +A Fully Qualified Domain Name (FQDN) may be configured for use by Kolla Ansible +on the internal and external networks, on which the API services will be +exposed. The variable will be passed through to the ``kolla_internal_fqdn`` or +``kolla_external_fqdn`` Kolla Ansible variable. + +To configure a network called ``example`` with an FQDN at ``api.example.com``: + +.. code-block:: yaml + :caption: ``networks.yml`` + + example_fqdn: api.example.com + Configuring Static IP Routes ---------------------------- @@ -116,6 +152,8 @@ To configure a network called ``example`` with VLAN ID ``123``: example_vlan: 123 +.. _configuration-network-ip-allocation: + IP Address Allocation ===================== diff --git a/etc/kayobe/globals.yml b/etc/kayobe/globals.yml index f3bbb772a..592f5744b 100644 --- a/etc/kayobe/globals.yml +++ b/etc/kayobe/globals.yml @@ -13,6 +13,9 @@ # Base path for kayobe state on remote hosts. #base_path: +# Path in which to store configuration on remote hosts. +#config_path: + # Path in which to cache downloaded images on remote hosts. #image_cache_path: @@ -25,8 +28,8 @@ ############################################################################### # User configuration. -# User with which to access seed and controller nodes. This user will be -# created if it does not exist. +# User with which to access remote hosts. This user will be created if it does +# not exist. #kayobe_ansible_user: ###############################################################################