Document retrieving logs from the deploy ramdisk

This patch is documenting how operators can configure Ironic to be able
to retrieve the logs from the deploy ramdisk (or disable it).

Closes-Bug: #1587143
Change-Id: I233e925f4dd9a1aa04a722eb852a6f95c74603f2

doc/source/deploy/troubleshooting.rst

@@ -106,3 +106,139 @@ API Errors
 
 The `debug_tracebacks_in_api` config option may be set to return tracebacks
 in the API response for all 4xx and 5xx errors.
+
+Retrieving logs from the deploy ramdisk
+=======================================
+
+When troubleshooting deployments (specially in case of a deploy failure)
+it's important to have access to the deploy ramdisk logs to be able to
+identify the source of the problem. By default, Ironic will retrieve the
+logs from the deploy ramdisk when the deployment fails and save it on the
+local filesystem at ``/var/log/ironic/deploy``.
+
+To change this behavior, operators can make the following changes to
+``/etc/ironic/ironic.conf`` under the ``[agent]`` group:
+
+* ``deploy_logs_collect``:  Whether Ironic should collect the deployment
+  logs on deployment. Valid values for this option are:
+
+  * ``on_failure`` (**default**): Retrieve the deployment logs upon a
+    deployment failure.
+
+  * ``always``: Always retrieve the deployment logs, even if the
+    deployment succeed.
+
+  * ``never``: Disable retrieving the deployment logs.
+
+* ``deploy_logs_storage_backend``: The name of the storage backend where
+  the logs will be stored. Valid values for this option are:
+
+  * ``local`` (**default**): Store the logs in the local filesystem.
+
+  * ``swift``: Store the logs in Swift.
+
+* ``deploy_logs_local_path``: The path to the directory where the
+  logs should be stored, used when the ``deploy_logs_storage_backend``
+  is configured to ``local``. By default logs will be stored at
+  **/var/log/ironic/deploy**.
+
+* ``deploy_logs_swift_container``: The name of the Swift container to
+  store the logs, used when the deploy_logs_storage_backend is configured to
+  "swift". By default **ironic_deploy_logs_container**.
+
+* ``deploy_logs_swift_days_to_expire``: Number of days before a log object
+  is marked as expired in Swift. If None, the logs will be kept forever
+  or until manually deleted. Used when the deploy_logs_storage_backend is
+  configured to "swift". By default **30** days.
+
+When the logs are collected, Ironic will store a *tar.gz* file containing
+all the logs according to the ``deploy_logs_storage_backend``
+configuration option. All log objects will be named with the following
+pattern::
+
+  <node-uuid>[_<instance-uuid>]_<timestamp yyyy-mm-dd-hh:mm:ss>.tar.gz
+
+.. note::
+   The *instance_uuid* field is not required for deploying a node when
+   Ironic is configured to be used in standalone mode. If present it
+   will be appended to the name.
+
+
+Accessing the log data
+----------------------
+
+When storing in the local filesystem
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When storing the logs in the local filesystem, the log files can
+be found at the path configured in the ``deploy_logs_local_path``
+configuration option. For example, to find the logs from the node
+``5e9258c4-cfda-40b6-86e2-e192f523d668``:
+
+.. code-block:: bash
+
+   $ ls /var/log/ironic/deploy | grep 5e9258c4-cfda-40b6-86e2-e192f523d668
+   5e9258c4-cfda-40b6-86e2-e192f523d668_88595d8a-6725-4471-8cd5-c0f3106b6898_2016-08-08-13:52:12.tar.gz
+   5e9258c4-cfda-40b6-86e2-e192f523d668_db87f2c5-7a9a-48c2-9a76-604287257c1b_2016-08-08-14:07:25.tar.gz
+
+.. note::
+   When saving the logs to the filesystem, operators may want to enable
+   some form of rotation for the logs to avoid disk space problems.
+
+
+When storing in Swift
+~~~~~~~~~~~~~~~~~~~~~
+
+When using Swift, operators can associate the objects in the
+container with the nodes in Ironic and search for the logs for the node
+``5e9258c4-cfda-40b6-86e2-e192f523d668`` using the **prefix** parameter.
+For example:
+
+.. code-block:: bash
+
+  $ swift list ironic_deploy_logs_container -p 5e9258c4-cfda-40b6-86e2-e192f523d668
+  5e9258c4-cfda-40b6-86e2-e192f523d668_88595d8a-6725-4471-8cd5-c0f3106b6898_2016-08-08-13:52:12.tar.gz
+  5e9258c4-cfda-40b6-86e2-e192f523d668_db87f2c5-7a9a-48c2-9a76-604287257c1b_2016-08-08-14:07:25.tar.gz
+
+To download a specific log from Swift, do:
+
+.. code-block:: bash
+
+   $ swift download ironic_deploy_logs_container "5e9258c4-cfda-40b6-86e2-e192f523d668_db87f2c5-7a9a-48c2-9a76-604287257c1b_2016-08-08-14:07:25.tar.gz"
+   5e9258c4-cfda-40b6-86e2-e192f523d668_db87f2c5-7a9a-48c2-9a76-604287257c1b_2016-08-08-14:07:25.tar.gz [auth 0.341s, headers 0.391s, total 0.391s, 0.531 MB/s]
+
+The contents of the log file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The log is just a ``.tar.gz`` file that can be extracted as:
+
+.. code-block:: bash
+
+   $ tar xvf <file path>
+
+
+The contents of the file may differ slightly depending on the distribution
+that the deploy ramdisk is using:
+
+* For distributions using ``systemd`` there will be a file called
+  **journal** which contains all the system logs collected via the
+  ``journalctl`` command.
+
+* For other distributions, the ramdisk will collect all the contents of
+  the ``/var/log`` directory.
+
+For all distributions, the log file will also contain the output of
+the following commands (if present): ``ps``, ``df``, ``ip addr`` and
+``iptables``.
+
+Here's one example when extracting the content of a log file for a
+distribution that uses ``systemd``:
+
+.. code-block:: bash
+
+   $ tar xvf 5e9258c4-cfda-40b6-86e2-e192f523d668_88595d8a-6725-4471-8cd5-c0f3106b6898_2016-08-08-13:52:12.tar.gz
+   df
+   ps
+   journal
+   ip_addr
+   iptables