Merge "Add password rotation docs page"
This commit is contained in:
Zuul
Gerrit Code Review
commit
d42cdef420
|
|
@ -12,3 +12,4 @@ Admin Guides
|
|||
etcd
|
||||
production-architecture-guide
|
||||
deployment-philosophy
|
||||
password-rotation
|
||||
|
|
|
|||
|
|
@ -0,0 +1,295 @@
|
|||
=================
|
||||
Password Rotation
|
||||
=================
|
||||
|
||||
This guide describes how to change the internal secrets from ``passwords.yml``
|
||||
used by Kolla-Ansible. It does not cover every possible ``passwords.yml``
|
||||
variable, only the most common ones.
|
||||
|
||||
.. warning::
|
||||
|
||||
Always back up your ``passwords.yml`` file before making any changes.
|
||||
Otherwise, it is easy to make unrecoverable mistakes.
|
||||
|
||||
.. warning::
|
||||
|
||||
This guide relies on recent changes to Kolla and Kolla-Ansible. You may
|
||||
encounter errors if applying this guide to older deployments. It is
|
||||
recommended that you update your containers and kolla-ansible to the latest
|
||||
available versions before proceeding.
|
||||
|
||||
Regenerating secrets
|
||||
--------------------
|
||||
|
||||
Passwords can be quickly re-generated using ``kolla-genpwd``.
|
||||
|
||||
Assuming an existing ``/etc/kolla/passwords.yml`` file, make a backup:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cp /etc/kolla/passwords.yml ./passwords.yml.bak
|
||||
|
||||
Edit the ``passwords.yml`` file to remove the password strings for any secrets
|
||||
that need to be regenerated i.e. change ``foo: "bar"`` to ``foo:``.
|
||||
|
||||
Regenerate the removed passwords:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
kolla-genpwd -p /etc/kolla/passwords.yml
|
||||
|
||||
Applying regenerated secrets
|
||||
----------------------------
|
||||
|
||||
The majority of the secrets can be applied by simply reconfiguring services
|
||||
with ``kolla-ansible reconfigure``. Below is a list of secrets that can be
|
||||
applied this way.
|
||||
|
||||
|
||||
* ``*_keystone_password``
|
||||
* ``*_database_password`` (excluding ``nova_database_password``)
|
||||
* ``*_ssh_key`` (excluding ``kolla_ssh_key``)
|
||||
* ``keystone_admin_password``
|
||||
* ``designate_rndc_key``
|
||||
* ``keepalived_password``
|
||||
* ``libvirt_sasl_password``
|
||||
* ``metadata_secret``
|
||||
* ``opensearch_dashboards_password``
|
||||
* ``osprofiler_secret``
|
||||
* ``prometheus_alertmanager_password``
|
||||
* ``qdrouterd_password``
|
||||
* ``redis_master_password``
|
||||
|
||||
It is possible to change more secrets however some require manual steps. The
|
||||
manual steps vary depending on the secret. They are listed below in the order
|
||||
they should be applied if they are to be changed at the same time. Once all
|
||||
manual steps are complete, reconfigure services (``kolla-ansible
|
||||
reconfigure``).
|
||||
|
||||
For simplicity, this guide assumes Docker is being used. The same commands
|
||||
should also work for Podman deployments by replacing instances of ``docker``
|
||||
with ``podman`` in all relevant commands.
|
||||
|
||||
Kolla SSH key
|
||||
^^^^^^^^^^^^^
|
||||
There is currently no mechanism within Kolla-Ansible to rotate
|
||||
``kolla_ssh_key``. It is however a relatively simple task to perform using a
|
||||
standard Ansible playbook, or can be performed by hand on smaller deployments.
|
||||
|
||||
Horizon Secret Key
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
The Horizon secret key (``horizon_secret_key``) is unique because it explicitly
|
||||
supports rotation. In reality, it is a Django secret key, and is used for
|
||||
cryptographic signing e.g. generating password recovery links. To minimise user
|
||||
impact, it is possible to set two secret keys at once. The new one will be used
|
||||
for generating new artifacts, while the old one will still be accepted for
|
||||
existing artifacts.
|
||||
|
||||
Take note of the old password, generate a new one, and take note of it as well.
|
||||
|
||||
Add it to the ``passwords.yml`` file, along with the old secret, in this
|
||||
exact format (including quotes in the middle):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
horizon_secret_key: newsecret' 'oldsecret
|
||||
|
||||
It is important to remember to remove the old key and reconfigure services
|
||||
again, after all old artifacts have expired e.g. after approximately one to two
|
||||
weeks.
|
||||
|
||||
Grafana Admin Password
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
The Grafana admin password (``grafana_admin_password``) must be rotated
|
||||
manually.
|
||||
|
||||
#. Generate a new Grafana Admin password.
|
||||
|
||||
#. Replace the old password in ``passwords.yml``.
|
||||
|
||||
#. Exec into any Grafana container:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
docker exec -it grafana bash
|
||||
|
||||
#. Run the password reset command, then enter the new password:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
grafana-cli admin reset-admin-password --password-from-stdin
|
||||
|
||||
Database Password
|
||||
^^^^^^^^^^^^^^^^^
|
||||
The database administrator password (``database_password``) must be rotated
|
||||
manually.
|
||||
|
||||
#. Generate a new database password.
|
||||
|
||||
#. Replace the old password in ``passwords.yml``, take note of both the old and
|
||||
new passwords.
|
||||
|
||||
#. SSH to a host running a MariaDB container.
|
||||
|
||||
#. Exec into the MariaDB container:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker exec -it mariadb bash
|
||||
|
||||
#. Log in to the database. You will be prompted for the password. Use the
|
||||
old value of ``database_password``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
mysql --batch -uroot -p
|
||||
|
||||
#. Check the current state of the ``root`` user:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
SELECT Host,User,Password FROM mysql.user WHERE User='root';
|
||||
|
||||
#. Update the password for the ``root`` user:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
SET PASSWORD FOR 'root'@'%' = PASSWORD('newpassword');
|
||||
|
||||
#. Check that the password hash has changed in the user list:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
SELECT Host,User,Password FROM mysql.user WHERE User='root';
|
||||
|
||||
#. If there are any remaining root users with the old password e.g.
|
||||
``root@localhost``, change the password for them too.
|
||||
|
||||
Nova Database Password
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
The nova database admin user password (``nova_database_password``) must be
|
||||
rotated manually.
|
||||
|
||||
.. warning::
|
||||
|
||||
From this point onward, API service may be disrupted.
|
||||
|
||||
#. Generate a new Nova database password.
|
||||
|
||||
#. Replace the old password in ``passwords.yml``.
|
||||
|
||||
#. Exec into the ``nova_conductor`` container:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
docker exec -it nova_conductor bash
|
||||
|
||||
#. List the cells:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
nova-manage cell_v2 list_cells --verbose
|
||||
|
||||
#. Find the entry for ``cell0``, copy the Database Connection value,
|
||||
replace the password in the string with the new value, and update it
|
||||
with the following command:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
nova-manage cell_v2 update_cell --cell_uuid 00000000-0000-0000-0000-000000000000 --database_connection "CONNECTION WITH NEW PASSWORD HERE" --transport-url "none:///"
|
||||
|
||||
(If the ``cell_uuid`` for ``cell0`` is not
|
||||
``00000000-0000-0000-0000-000000000000``, change the above command
|
||||
accordingly)
|
||||
|
||||
Heat Domain Admin Password
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
The keystone password for the heat domain admin service user
|
||||
(``heat_domain_admin_password``) must be rotated manually.
|
||||
|
||||
It can be changed by an administrator just like any other standard OpenStack
|
||||
user password. Generate a new password, replace the old password in
|
||||
``passwords.yml``, then apply the change manually:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
openstack user set --password <password> heat_domain_admin --domain heat_user_domain
|
||||
|
||||
RabbitMQ Secrets
|
||||
^^^^^^^^^^^^^^^^
|
||||
RabbitMQ uses two main secrets. An Erlang cookie for cluster membership
|
||||
(``rabbitmq_cluster_cookie``), and a RabbitMQ management user password
|
||||
(``rabbitmq_password``). There is currently no documented process for
|
||||
seamlessly rotating these secrets. Many OpenStack services use RabbitMQ for
|
||||
communication and reconfiguring them with the new credentials can take some
|
||||
time, resulting in a relatively long API outage.
|
||||
|
||||
It is recommended that you stop all services, then stop and destroy the
|
||||
RabbitMQ containers and volumes. Because the RabbitMQ containers are destroyed,
|
||||
``kolla-ansible deploy`` should be used to restart services rather than
|
||||
``kolla-ansible reconfigure``. Detailed steps are listed below:
|
||||
|
||||
#. Generate a new ``rabbitmq_cluster_cookie`` and ``rabbitmq_password``.
|
||||
|
||||
#. Replace the old values in ``passwords.yml``.
|
||||
|
||||
#. Stop OpenStack services:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
kolla-ansible -i inventory stop
|
||||
|
||||
#. On each node running RabbitMQ, destroy its containers and volumes:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker stop rabbitmq
|
||||
docker rm rabbitmq
|
||||
docker volume rm rabbitmq
|
||||
|
||||
#. Redeploy services:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
kolla-ansible -i inventory deploy
|
||||
|
||||
Post-redeploy changes
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
Once services have been redeployed, the existing Memcached data should be
|
||||
flushed. The old Memcached password will no longer be used so any data stored
|
||||
using it will be inaccessible.
|
||||
|
||||
The instructions below must be run from a host that has access to the network
|
||||
the Memcached containers are using. If you are not sure, run them from a host
|
||||
that is running Memcached.
|
||||
|
||||
#. Install a telnet client:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
apt/dnf install telnet
|
||||
|
||||
#. Check the config for the IP and port used by Memcached (on every host
|
||||
running Memcached):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo grep command /etc/kolla/memcached/config.json
|
||||
|
||||
The IP and port will be printed after ``-l`` and ``-p`` respectively
|
||||
|
||||
#. For each container start a Telnet session, clear all data, then
|
||||
exit:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
telnet <ip> <port>
|
||||
flush_all
|
||||
quit
|
||||
|
||||
Known out-of-scope secrets
|
||||
--------------------------
|
||||
Below is a list of passwords that are known to be outside the scope of this
|
||||
guide.
|
||||
|
||||
* ``docker_registry_password`` - kolla-ansible cannot manage docker registries.
|
||||
Loading…
Reference in New Issue