265 lines
14 KiB
XML
265 lines
14 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<section xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink"
|
|
version="5.0"
|
|
xml:id="section_compute-troubleshooting">
|
|
<title>Troubleshoot Compute</title>
|
|
<para>Common problems for Compute typically involve misconfigured
|
|
networking or credentials that are not sourced properly in the
|
|
environment. Also, most flat networking configurations do not
|
|
enable <command>ping</command> or <command>ssh</command> from
|
|
a compute node to the instances that run on that node. Another
|
|
common problem is trying to run 32-bit images on a 64-bit
|
|
compute node. This section shows you how to troubleshoot
|
|
Compute.</para>
|
|
<section xml:id="log-files-for-openstack-compute">
|
|
<title>Compute service logging</title>
|
|
<para>Compute stores a log file for each service in
|
|
<filename>/var/log/nova</filename>. For example,
|
|
<filename>nova-compute.log</filename> is the log for
|
|
the <systemitem class="service">nova-compute</systemitem>
|
|
service. You can set the following options to format log
|
|
strings for the nova.log module in the
|
|
<filename>nova.conf</filename> file:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><literal>logging_context_format_string</literal></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><literal>logging_default_format_string</literal></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>If the log level is set to <literal>debug</literal>, you
|
|
can also specify
|
|
<literal>logging_debug_format_suffix</literal> to
|
|
append extra formatting. For information about what
|
|
variables are available for the formatter see: <link
|
|
xlink:href="http://docs.python.org/library/logging.html#formatter-objects"
|
|
>http://docs.python.org/library/logging.html#formatter-objects</link>.</para>
|
|
<para>You have two options for logging for OpenStack Compute
|
|
based on configuration settings. In
|
|
<filename>nova.conf</filename>, include the
|
|
<literal>logfile</literal> option to enable logging.
|
|
Alternatively you can set <literal>use_syslog = 1</literal>
|
|
so that the nova daemon logs to syslog.</para>
|
|
</section>
|
|
<section xml:id="section_compute-GuruMed-reports">
|
|
<title>Guru Meditation reports</title>
|
|
<para>A Guru Meditation report is sent by the Compute Service upon receipt of the
|
|
<systemitem>SIGUSR1</systemitem> signal. This report is a general-purpose error
|
|
report, including a complete report of the service's current state, and is sent to
|
|
<systemitem>stderr</systemitem>.</para>
|
|
<para>For example, if you redirect error output to <filename>nova-api-err.log</filename>
|
|
using <command>nova-api 2>/var/log/nova/nova-api-err.log</command>, resulting in the
|
|
process ID 8675, you can then run:</para>
|
|
<screen><prompt>#</prompt> <userinput>kill -USR1 8675</userinput></screen>
|
|
<para>This command triggers the Guru Meditation report to be printed to
|
|
<filename>/var/log/nova/nova-api-err.log</filename>.</para>
|
|
<para>The report has the following sections:
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
Package: Displays information about the package to
|
|
which the process belongs, including version
|
|
information.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Threads: Displays stack traces and thread IDs for
|
|
each of the threads within the process.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Green Threads: Displays stack traces for each of
|
|
the green threads within the process (green
|
|
threads do not have thread IDs).
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Configuration: Lists all configuration options
|
|
currently accessible through the CONF object for
|
|
the current process.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist></para>
|
|
<para>For more information, see <link
|
|
xlink:href="http://docs.openstack.org/developer/nova/devref/gmr.html"
|
|
>Guru Meditation Reports</link>.</para>
|
|
</section>
|
|
<section xml:id="section_compute-common-errors-and-fixes">
|
|
<title>Common errors and fixes for Compute</title>
|
|
<para>The <link xlink:href="http://ask.openstack.org"
|
|
>ask.openstack.org</link> site offers a place to ask
|
|
and answer questions, and you can also mark questions as
|
|
frequently asked questions. This section describes some
|
|
errors people have posted previously. Bugs are constantly
|
|
being fixed, so online resources are a great way to get
|
|
the most up-to-date errors and fixes.</para>
|
|
<section xml:id="section_credential-errors">
|
|
<title>Credential errors, 401, and 403 forbidden
|
|
errors</title>
|
|
<para>Missing credentials cause a
|
|
<errorcode>403</errorcode>
|
|
<errortext>forbidden</errortext> error. To resolve
|
|
this issue, use one of these methods:<orderedlist>
|
|
<listitem>
|
|
<para><emphasis role="bold">Manual
|
|
method</emphasis>. Gets the
|
|
<filename>novarc</filename> file from
|
|
the project ZIP file, saves existing
|
|
credentials in case of override, and
|
|
manually sources the
|
|
<filename>novarc</filename>
|
|
file.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><emphasis role="bold">Script
|
|
method</emphasis>. Generates
|
|
<filename>novarc</filename> from the
|
|
project ZIP file and sources it for
|
|
you.</para>
|
|
</listitem>
|
|
</orderedlist></para>
|
|
<para>When you run <systemitem class="service"
|
|
>nova-api</systemitem> the first time, it
|
|
generates the certificate authority information,
|
|
including <filename>openssl.cnf</filename>. If you
|
|
start the CA services before this, you might not be
|
|
able to create your ZIP file. Restart the services.
|
|
When your CA information is available, create your ZIP
|
|
file.</para>
|
|
<para>Also, check your HTTP proxy settings to see whether
|
|
they cause problems with <filename>novarc</filename>
|
|
creation.</para>
|
|
</section>
|
|
<section xml:id="section_instance-errors">
|
|
<title>Instance errors</title>
|
|
<para>Sometimes a particular instance shows
|
|
<literal>pending</literal> or you cannot SSH to
|
|
it. Sometimes the image itself is the problem. For
|
|
example, when you use flat manager networking, you do
|
|
not have a DHCP server and certain images do not
|
|
support interface injection; you cannot connect to
|
|
them. The fix for this problem is to use an image that
|
|
does support this method, such as Ubuntu, which
|
|
obtains an IP address correctly with FlatManager
|
|
network settings.</para>
|
|
<para>To troubleshoot other possible problems with an
|
|
instance, such as an instance that stays in a spawning
|
|
state, check the directory for the particular instance
|
|
under <filename>/var/lib/nova/instances</filename> on
|
|
the <systemitem class="service"
|
|
>nova-compute</systemitem> host and make sure that
|
|
these files are present:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><filename>libvirt.xml</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename>disk</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename>disk-raw</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename>kernel</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename>ramdisk</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>After the instance starts,
|
|
<filename>console.log</filename></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>If any files are missing, empty, or very small, the
|
|
<systemitem class="service"
|
|
>nova-compute</systemitem> service did not
|
|
successfully download the images from the Image
|
|
Service.</para>
|
|
<para>Also check <filename>nova-compute.log</filename> for
|
|
exceptions. Sometimes they do not appear in the
|
|
console output.</para>
|
|
<para>Next, check the log file for the instance in the
|
|
<filename>/var/log/libvirt/qemu</filename>
|
|
directory to see if it exists and has any useful error
|
|
messages in it.</para>
|
|
<para>Finally, from the
|
|
<filename>/var/lib/nova/instances</filename>
|
|
directory for the instance, see if this command
|
|
returns an error:</para>
|
|
<screen><prompt>#</prompt> <userinput>virsh create libvirt.xml</userinput></screen>
|
|
</section>
|
|
<section xml:id="section_compute-empty-log-output">
|
|
<title>Empty log output for Linux instances</title>
|
|
<para>You can view the log output of running instances
|
|
from either the <guilabel>Log</guilabel> tab of the
|
|
dashboard or the output of
|
|
<command>nova console-log</command>. In some cases, the
|
|
log output of a running Linux instance will be empty or
|
|
only display a single character (for example,
|
|
the <keycap>?</keycap> character).</para>
|
|
<para>This occurs when the Compute service attempts to
|
|
retrieve the log output of the instance via a serial
|
|
console while the instance itself is not configured to
|
|
send output to the console. To rectify this, append the
|
|
following parameters to kernel arguments specified in the
|
|
instance's boot loader:</para>
|
|
<screen>console=tty0 console=ttyS0,115200n8</screen>
|
|
<para>Upon rebooting, the instance will be configured to
|
|
send output to the Compute service.</para>
|
|
</section>
|
|
</section>
|
|
<section xml:id="reset-state">
|
|
<title>Reset the state of an instance</title>
|
|
<para>If an instance remains in an intermediate state, such as
|
|
<literal>deleting</literal>, you can use the
|
|
<command>nova reset-state</command> command to
|
|
manually reset the state of an instance to an error state.
|
|
You can then delete the instance. For example:</para>
|
|
<screen><prompt>$</prompt> <userinput>nova reset-state c6bbbf26-b40a-47e7-8d5c-eb17bf65c485</userinput>
|
|
<prompt>$</prompt> <userinput>nova delete c6bbbf26-b40a-47e7-8d5c-eb17bf65c485</userinput></screen>
|
|
<para>You can also use the <parameter>--active</parameter>
|
|
parameter to force the instance back to an active state
|
|
instead of an error state. For example:</para>
|
|
<screen><prompt>$</prompt> <userinput>nova reset-state --active c6bbbf26-b40a-47e7-8d5c-eb17bf65c485</userinput></screen>
|
|
</section>
|
|
<section xml:id="problems-with-injection">
|
|
<title>Injection problems</title>
|
|
<para>If instances do not boot or boot slowly, investigate
|
|
file injection as a cause.</para>
|
|
<para>To disable injection in libvirt, set the following in
|
|
<filename>nova.conf</filename>:</para>
|
|
<programlisting language="ini">[libvirt]
|
|
inject_partition = -2</programlisting>
|
|
<note>
|
|
<para>If you have not enabled the configuration drive and
|
|
you want to make user-specified files available from
|
|
the metadata server for to improve performance and
|
|
avoid boot failure if injection fails, you must
|
|
disable injection.</para>
|
|
</note>
|
|
</section>
|
|
<section xml:id="live-snapshotting-fail">
|
|
<title>Disable live snapshotting</title>
|
|
<para>If you use libvirt version <literal>1.2.2</literal>,
|
|
you may experience problems with live snapshots creation.
|
|
Occasionally, libvirt of the specified version
|
|
fails to perform the live snapshotting under load
|
|
that presupposes a concurrent creation of
|
|
multiple snapshots.</para>
|
|
|
|
<para>To effectively disable the libvirt live snapshotting, until the problem
|
|
is resolved, configure the <option>disable_libvirt_livesnapshot</option>
|
|
option. You can turn off the live snapshotting mechanism by setting
|
|
up its value to <literal>True</literal> in the
|
|
<literal>[workarounds]</literal> section
|
|
of the <filename>nova.conf</filename> configuration file:</para>
|
|
<programlisting language="ini">[workarounds]
|
|
disable_libvirt_livesnapshot = True</programlisting>
|
|
</section>
|
|
</section>
|