diff --git a/doc/source/deploy/troubleshooting.rst b/doc/source/deploy/troubleshooting.rst index 378ce443d8..f2fb2646ef 100644 --- a/doc/source/deploy/troubleshooting.rst +++ b/doc/source/deploy/troubleshooting.rst @@ -106,3 +106,139 @@ API Errors The `debug_tracebacks_in_api` config option may be set to return tracebacks in the API response for all 4xx and 5xx errors. + +Retrieving logs from the deploy ramdisk +======================================= + +When troubleshooting deployments (specially in case of a deploy failure) +it's important to have access to the deploy ramdisk logs to be able to +identify the source of the problem. By default, Ironic will retrieve the +logs from the deploy ramdisk when the deployment fails and save it on the +local filesystem at ``/var/log/ironic/deploy``. + +To change this behavior, operators can make the following changes to +``/etc/ironic/ironic.conf`` under the ``[agent]`` group: + +* ``deploy_logs_collect``: Whether Ironic should collect the deployment + logs on deployment. Valid values for this option are: + + * ``on_failure`` (**default**): Retrieve the deployment logs upon a + deployment failure. + + * ``always``: Always retrieve the deployment logs, even if the + deployment succeed. + + * ``never``: Disable retrieving the deployment logs. + +* ``deploy_logs_storage_backend``: The name of the storage backend where + the logs will be stored. Valid values for this option are: + + * ``local`` (**default**): Store the logs in the local filesystem. + + * ``swift``: Store the logs in Swift. + +* ``deploy_logs_local_path``: The path to the directory where the + logs should be stored, used when the ``deploy_logs_storage_backend`` + is configured to ``local``. By default logs will be stored at + **/var/log/ironic/deploy**. + +* ``deploy_logs_swift_container``: The name of the Swift container to + store the logs, used when the deploy_logs_storage_backend is configured to + "swift". By default **ironic_deploy_logs_container**. + +* ``deploy_logs_swift_days_to_expire``: Number of days before a log object + is marked as expired in Swift. If None, the logs will be kept forever + or until manually deleted. Used when the deploy_logs_storage_backend is + configured to "swift". By default **30** days. + +When the logs are collected, Ironic will store a *tar.gz* file containing +all the logs according to the ``deploy_logs_storage_backend`` +configuration option. All log objects will be named with the following +pattern:: + + [_]_.tar.gz + +.. note:: + The *instance_uuid* field is not required for deploying a node when + Ironic is configured to be used in standalone mode. If present it + will be appended to the name. + + +Accessing the log data +---------------------- + +When storing in the local filesystem +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When storing the logs in the local filesystem, the log files can +be found at the path configured in the ``deploy_logs_local_path`` +configuration option. For example, to find the logs from the node +``5e9258c4-cfda-40b6-86e2-e192f523d668``: + +.. code-block:: bash + + $ ls /var/log/ironic/deploy | grep 5e9258c4-cfda-40b6-86e2-e192f523d668 + 5e9258c4-cfda-40b6-86e2-e192f523d668_88595d8a-6725-4471-8cd5-c0f3106b6898_2016-08-08-13:52:12.tar.gz + 5e9258c4-cfda-40b6-86e2-e192f523d668_db87f2c5-7a9a-48c2-9a76-604287257c1b_2016-08-08-14:07:25.tar.gz + +.. note:: + When saving the logs to the filesystem, operators may want to enable + some form of rotation for the logs to avoid disk space problems. + + +When storing in Swift +~~~~~~~~~~~~~~~~~~~~~ + +When using Swift, operators can associate the objects in the +container with the nodes in Ironic and search for the logs for the node +``5e9258c4-cfda-40b6-86e2-e192f523d668`` using the **prefix** parameter. +For example: + +.. code-block:: bash + + $ swift list ironic_deploy_logs_container -p 5e9258c4-cfda-40b6-86e2-e192f523d668 + 5e9258c4-cfda-40b6-86e2-e192f523d668_88595d8a-6725-4471-8cd5-c0f3106b6898_2016-08-08-13:52:12.tar.gz + 5e9258c4-cfda-40b6-86e2-e192f523d668_db87f2c5-7a9a-48c2-9a76-604287257c1b_2016-08-08-14:07:25.tar.gz + +To download a specific log from Swift, do: + +.. code-block:: bash + + $ swift download ironic_deploy_logs_container "5e9258c4-cfda-40b6-86e2-e192f523d668_db87f2c5-7a9a-48c2-9a76-604287257c1b_2016-08-08-14:07:25.tar.gz" + 5e9258c4-cfda-40b6-86e2-e192f523d668_db87f2c5-7a9a-48c2-9a76-604287257c1b_2016-08-08-14:07:25.tar.gz [auth 0.341s, headers 0.391s, total 0.391s, 0.531 MB/s] + +The contents of the log file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The log is just a ``.tar.gz`` file that can be extracted as: + +.. code-block:: bash + + $ tar xvf + + +The contents of the file may differ slightly depending on the distribution +that the deploy ramdisk is using: + +* For distributions using ``systemd`` there will be a file called + **journal** which contains all the system logs collected via the + ``journalctl`` command. + +* For other distributions, the ramdisk will collect all the contents of + the ``/var/log`` directory. + +For all distributions, the log file will also contain the output of +the following commands (if present): ``ps``, ``df``, ``ip addr`` and +``iptables``. + +Here's one example when extracting the content of a log file for a +distribution that uses ``systemd``: + +.. code-block:: bash + + $ tar xvf 5e9258c4-cfda-40b6-86e2-e192f523d668_88595d8a-6725-4471-8cd5-c0f3106b6898_2016-08-08-13:52:12.tar.gz + df + ps + journal + ip_addr + iptables