624 lines
23 KiB
XML
624 lines
23 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<chapter version="5.0" xml:id="compute_nodes"
|
|
xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
xmlns:ns5="http://www.w3.org/1999/xhtml"
|
|
xmlns:ns4="http://www.w3.org/1998/Math/MathML"
|
|
xmlns:ns3="http://www.w3.org/2000/svg"
|
|
xmlns:ns="http://docbook.org/ns/docbook">
|
|
<?dbhtml stop-chunking?>
|
|
|
|
<title>Compute Nodes</title>
|
|
|
|
<para>In this chapter, we discuss some of the choices you need to consider
|
|
when building out your compute nodes. Compute nodes form the resource core
|
|
of the OpenStack Compute cloud, providing the processing, memory, network
|
|
and storage resources to run instances.</para>
|
|
|
|
<section xml:id="cpu_choice">
|
|
<title>Choosing a CPU</title>
|
|
|
|
<para>The type of CPU in your compute node is a very important choice.
|
|
First, ensure that the CPU supports virtualization by way of
|
|
<emphasis>VT-x</emphasis> for Intel chips and <emphasis>AMD-v</emphasis>
|
|
for AMD chips.<indexterm class="singular">
|
|
<primary>CPUs (central processing units)</primary>
|
|
|
|
<secondary>choosing</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>compute nodes</primary>
|
|
|
|
<secondary>CPU choice</secondary>
|
|
</indexterm></para>
|
|
|
|
<tip>
|
|
<para>Consult the vendor documentation to check for virtualization
|
|
support. For Intel, read <link xlink:href="http://www.intel.com/support/processors/sb/cs-030729.htm"
|
|
xlink:title="Intel VT-x"> “Does my processor support Intel®
|
|
Virtualization Technology?”</link>. For AMD, read <link
|
|
xlink:href=" http://www.amd.com/en-us/innovations/software-technologies/server-solution/virtualization" xlink:title="AMD-v"> AMD
|
|
Virtualization</link>. Note that your CPU may support virtualization but
|
|
it may be disabled. Consult your BIOS documentation for how to enable
|
|
CPU features.<indexterm class="singular">
|
|
<primary>virtualization technology</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>AMD Virtualization</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>Intel Virtualization Technology</primary>
|
|
</indexterm></para>
|
|
</tip>
|
|
|
|
<para>The number of cores that the CPU has also affects the decision. It's
|
|
common for current CPUs to have up to 12 cores. Additionally, if an Intel
|
|
CPU supports hyperthreading, those 12 cores are doubled to 24 cores. If
|
|
you purchase a server that supports multiple CPUs, the number of cores is
|
|
further multiplied.<indexterm class="singular">
|
|
<primary>cores</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>hyperthreading</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>multithreading</primary>
|
|
</indexterm></para>
|
|
|
|
<?hard-pagebreak ?>
|
|
|
|
<sidebar>
|
|
<title>Multithread Considerations</title>
|
|
|
|
<para>Hyper-Threading is Intel's proprietary simultaneous multithreading
|
|
implementation used to improve parallelization on their CPUs. You might
|
|
consider enabling Hyper-Threading to improve the performance of
|
|
multithreaded applications.</para>
|
|
|
|
<para>Whether you should enable Hyper-Threading on your CPUs depends
|
|
upon your use case. For example, disabling Hyper-Threading can be
|
|
beneficial in intense computing environments. We recommend that you do
|
|
performance testing with your local workload with both Hyper-Threading
|
|
on and off to determine what is more appropriate in your case.<indexterm
|
|
class="singular">
|
|
<primary>CPUs (central processing units)</primary>
|
|
|
|
<secondary>enabling hyperthreading on</secondary>
|
|
</indexterm></para>
|
|
</sidebar>
|
|
</section>
|
|
|
|
<section xml:id="hypervisor_choice">
|
|
<title>Choosing a Hypervisor</title>
|
|
|
|
<para>A hypervisor provides software to manage virtual machine access to
|
|
the underlying hardware. The hypervisor creates, manages, and monitors
|
|
virtual machines.<indexterm class="singular">
|
|
<primary>Docker</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>Hyper-V</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>ESXi hypervisor</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>ESX hypervisor</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>VMware API</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>Quick EMUlator (QEMU)</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>Linux containers (LXC)</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>kernel-based VM (KVM) hypervisor</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>Xen API</primary>
|
|
|
|
<secondary>XenServer hypervisor</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>hypervisors</primary>
|
|
|
|
<secondary>choosing</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>compute nodes</primary>
|
|
|
|
<secondary>hypervisor choice</secondary>
|
|
</indexterm> OpenStack Compute supports many hypervisors to various
|
|
degrees, including: <itemizedlist role="compact">
|
|
<listitem>
|
|
<para><link xlink:href="http://www.linux-kvm.org/page/Main_Page"
|
|
xlink:title="reference manual">KVM</link></para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para><link xlink:href="https://linuxcontainers.org/"
|
|
xlink:title="reference manual">LXC</link></para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para><link xlink:href="http://wiki.qemu.org/Main_Page"
|
|
xlink:title="reference manual">QEMU</link></para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para><link xlink:href="https://www.vmware.com/support/vsphere-hypervisor"
|
|
xlink:title="reference manual">VMware ESX/ESXi</link></para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para><link xlink:href="http://www.xenproject.org/"
|
|
xlink:title="reference manual">Xen</link></para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para><link xlink:href="http://technet.microsoft.com/en-us/library/hh831531.aspx"
|
|
xlink:title="reference manual">Hyper-V</link></para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para><link xlink:href="https://www.docker.com/"
|
|
xlink:title="reference manual">Docker</link></para>
|
|
</listitem>
|
|
</itemizedlist></para>
|
|
|
|
<para>Probably the most important factor in your choice of hypervisor is
|
|
your current usage or experience. Aside from that, there are practical
|
|
concerns to do with feature parity, documentation, and the level of
|
|
community experience.</para>
|
|
|
|
<para>For example, KVM is the most widely adopted hypervisor in the
|
|
OpenStack community. Besides KVM, more deployments run Xen, LXC, VMware,
|
|
and Hyper-V than the others listed. However, each of these are
|
|
lacking some feature support or the documentation on how to use them with
|
|
OpenStack is out of date.</para>
|
|
|
|
<para>The best information available to support your choice is found on
|
|
the <link xlink:href="http://docs.openstack.org/developer/nova/support-matrix.html"
|
|
xlink:title="reference manual">Hypervisor Support Matrix</link> and in the
|
|
<link xlink:href="http://docs.openstack.org/liberty/config-reference/content/section_compute-hypervisors.html"
|
|
xlink:title="configuration reference">configuration
|
|
reference</link>.</para>
|
|
|
|
<note>
|
|
<para>It is also possible to run multiple hypervisors in a single
|
|
deployment using host aggregates or cells. However, an individual
|
|
compute node can run only a single hypervisor at a time.<indexterm
|
|
class="singular">
|
|
<primary>hypervisors</primary>
|
|
|
|
<secondary>running multiple</secondary>
|
|
</indexterm></para>
|
|
</note>
|
|
</section>
|
|
|
|
<section xml:id="instance_storage">
|
|
<title>Instance Storage Solutions</title>
|
|
|
|
<para>As part of the procurement for a compute cluster, you must specify
|
|
some storage for the disk on which the instantiated instance runs. There
|
|
are three main approaches to providing this temporary-style storage, and
|
|
it is important to understand the implications of the choice.<indexterm
|
|
class="singular">
|
|
<primary>storage</primary>
|
|
|
|
<secondary>instance storage solutions</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>instances</primary>
|
|
|
|
<secondary>storage solutions</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>compute nodes</primary>
|
|
|
|
<secondary>instance storage solutions</secondary>
|
|
</indexterm></para>
|
|
|
|
<para>They are:</para>
|
|
|
|
<itemizedlist role="compact">
|
|
<listitem>
|
|
<para>Off compute node storage—shared file system</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>On compute node storage—shared file system</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>On compute node storage—nonshared file system</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>In general, the questions you should ask when selecting storage are
|
|
as follows:</para>
|
|
|
|
<itemizedlist role="compact">
|
|
<listitem>
|
|
<para>What is the platter count you can achieve?</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Do more spindles result in better I/O despite network
|
|
access?</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Which one results in the best cost-performance scenario you're
|
|
aiming for?</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>How do you manage the storage operationally?</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>Many operators use separate compute and storage hosts. Compute
|
|
services and storage services have different requirements, and compute
|
|
hosts typically require more CPU and RAM than storage hosts. Therefore,
|
|
for a fixed budget, it makes sense to have different configurations for
|
|
your compute nodes and your storage nodes. Compute nodes will be invested
|
|
in CPU and RAM, and storage nodes will be invested in block
|
|
storage.</para>
|
|
|
|
<para>However, if you are more restricted in the number of physical hosts
|
|
you have available for creating your cloud and you want to be able to
|
|
dedicate as many of your hosts as possible to running instances, it makes
|
|
sense to run compute and storage on the same machines.</para>
|
|
|
|
<para>We'll discuss the three main approaches to instance storage in the
|
|
next few sections.</para>
|
|
|
|
<?hard-pagebreak ?>
|
|
|
|
<section xml:id="off_compute_node_storage">
|
|
<title>Off Compute Node Storage—Shared File System</title>
|
|
|
|
<para>In this option, the disks storing the running instances are hosted
|
|
in servers outside of the compute nodes.<indexterm class="singular">
|
|
<primary>shared storage</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>file systems</primary>
|
|
|
|
<secondary>shared</secondary>
|
|
</indexterm></para>
|
|
|
|
<para>If you use separate compute and storage hosts, you can treat your
|
|
compute hosts as "stateless." As long as you don't have any instances
|
|
currently running on a compute host, you can take it offline or wipe it
|
|
completely without having any effect on the rest of your cloud. This
|
|
simplifies maintenance for the compute hosts.</para>
|
|
|
|
<para>There are several advantages to this approach:</para>
|
|
|
|
<itemizedlist role="compact">
|
|
<listitem>
|
|
<para>If a compute node fails, instances are usually easily
|
|
recoverable.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Running a dedicated storage system can be operationally
|
|
simpler.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>You can scale to any number of spindles.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>It may be possible to share the external storage for other
|
|
purposes.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>The main downsides to this approach are:</para>
|
|
|
|
<itemizedlist role="compact">
|
|
<listitem>
|
|
<para>Depending on design, heavy I/O usage from some instances can
|
|
affect unrelated instances.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Use of the network can decrease performance.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
|
|
<section xml:id="on_compute_node_storage">
|
|
<title>On Compute Node Storage—Shared File System</title>
|
|
|
|
<para>In this option, each compute node is specified with a significant
|
|
amount of disk space, but a distributed file system ties the disks from
|
|
each compute node into a single mount.</para>
|
|
|
|
<para>The main advantage of this option is that it scales to external
|
|
storage when you require additional storage.</para>
|
|
|
|
<para>However, this option has several downsides:</para>
|
|
|
|
<itemizedlist role="compact">
|
|
<listitem>
|
|
<para>Running a distributed file system can make you lose your data
|
|
locality compared with nonshared storage.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Recovery of instances is complicated by depending on multiple
|
|
hosts.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>The chassis size of the compute node can limit the number of
|
|
spindles able to be used in a compute node.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Use of the network can decrease performance.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
|
|
<section xml:id="on_compute_node_storage_nonshared">
|
|
<title>On Compute Node Storage—Nonshared File System</title>
|
|
|
|
<para>In this option, each compute node is specified with enough disks
|
|
to store the instances it hosts.<indexterm class="singular">
|
|
<primary>file systems</primary>
|
|
|
|
<secondary>nonshared</secondary>
|
|
</indexterm></para>
|
|
|
|
<para>There are two main reasons why this is a good idea:</para>
|
|
|
|
<itemizedlist role="compact">
|
|
<listitem>
|
|
<para>Heavy I/O usage on one compute node does not affect instances
|
|
on other compute nodes.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Direct I/O access can increase performance.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>This has several downsides:</para>
|
|
|
|
<itemizedlist role="compact">
|
|
<listitem>
|
|
<para>If a compute node fails, the instances running on that node
|
|
are lost.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>The chassis size of the compute node can limit the number of
|
|
spindles able to be used in a compute node.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Migrations of instances from one node to another are more
|
|
complicated and rely on features that may not continue to be
|
|
developed.</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>If additional storage is required, this option does not
|
|
scale.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>Running a shared file system on a storage system apart from the
|
|
computes nodes is ideal for clouds where reliability and scalability are
|
|
the most important factors. Running a shared file system on the compute
|
|
nodes themselves may be best in a scenario where you have to deploy to
|
|
preexisting servers for which you have little to no control over their
|
|
specifications. Running a nonshared file system on the compute nodes
|
|
themselves is a good option for clouds with high I/O requirements and
|
|
low concern for reliability.<indexterm class="singular">
|
|
<primary>scaling</primary>
|
|
|
|
<secondary>file system choice</secondary>
|
|
</indexterm></para>
|
|
</section>
|
|
|
|
<section xml:id="live_migration">
|
|
<title>Issues with Live Migration</title>
|
|
|
|
<para>We consider live migration an integral part of the operations of
|
|
the cloud. This feature provides the ability to seamlessly move
|
|
instances from one physical host to another, a necessity for performing
|
|
upgrades that require reboots of the compute hosts, but only works well
|
|
with shared storage.<indexterm class="singular">
|
|
<primary>storage</primary>
|
|
|
|
<secondary>live migration</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>migration</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>live migration</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>compute nodes</primary>
|
|
|
|
<secondary>live migration</secondary>
|
|
</indexterm></para>
|
|
|
|
<para>Live migration can also be done with nonshared storage, using a
|
|
feature known as <emphasis>KVM live block migration</emphasis>. While an
|
|
earlier implementation of block-based migration in KVM and QEMU was
|
|
considered unreliable, there is a newer, more reliable implementation of
|
|
block-based live migration as of QEMU 1.4 and libvirt 1.0.2 that is also
|
|
compatible with OpenStack. However, none of the authors of this guide
|
|
have first-hand experience using live block migration.<indexterm
|
|
class="singular">
|
|
<primary>block migration</primary>
|
|
</indexterm></para>
|
|
</section>
|
|
|
|
<section xml:id="file_systems">
|
|
<title>Choice of File System</title>
|
|
|
|
<para>If you want to support shared-storage live migration, you need to
|
|
configure a distributed file system.<indexterm class="singular">
|
|
<primary>compute nodes</primary>
|
|
|
|
<secondary>file system choice</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>file systems</primary>
|
|
|
|
<secondary>choice of</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>storage</primary>
|
|
|
|
<secondary>file system choice</secondary>
|
|
</indexterm></para>
|
|
|
|
<para>Possible options include:</para>
|
|
|
|
<itemizedlist role="compact">
|
|
<listitem>
|
|
<para>NFS (default for Linux)</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>GlusterFS</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>MooseFS</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>Lustre</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>We've seen deployments with all, and recommend that you choose the
|
|
one you are most familiar with operating. If you are not familiar with
|
|
any of these, choose NFS, as it is the easiest to set up and there is
|
|
extensive community knowledge about it.</para>
|
|
</section>
|
|
</section>
|
|
|
|
<section xml:id="overcommit">
|
|
<title>Overcommitting</title>
|
|
|
|
<para>OpenStack allows you to overcommit CPU and RAM on compute nodes.
|
|
This allows you to increase the number of instances you can have running
|
|
on your cloud, at the cost of reducing the performance of the
|
|
instances.<indexterm class="singular">
|
|
<primary>RAM overcommit</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>CPUs (central processing units)</primary>
|
|
|
|
<secondary>overcommitting</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>overcommitting</primary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>compute nodes</primary>
|
|
|
|
<secondary>overcommitting</secondary>
|
|
</indexterm> OpenStack Compute uses the following ratios by
|
|
default:</para>
|
|
|
|
<itemizedlist role="compact">
|
|
<listitem>
|
|
<para>CPU allocation ratio: 16:1</para>
|
|
</listitem>
|
|
|
|
<listitem>
|
|
<para>RAM allocation ratio: 1.5:1</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>The default CPU allocation ratio of 16:1 means that the scheduler
|
|
allocates up to 16 virtual cores per physical core. For example, if a
|
|
physical node has 12 cores, the scheduler sees 192 available virtual
|
|
cores. With typical flavor definitions of 4 virtual cores per instance,
|
|
this ratio would provide 48 instances on a physical node.</para>
|
|
|
|
<para>The formula for the number of virtual instances on a compute node is
|
|
<emphasis>(OR*PC)/VC</emphasis>, where:</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term><emphasis>OR</emphasis></term>
|
|
|
|
<listitem>
|
|
<para>CPU overcommit ratio (virtual cores per physical core)</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><emphasis>PC</emphasis></term>
|
|
|
|
<listitem>
|
|
<para>Number of physical cores</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><emphasis>VC</emphasis></term>
|
|
|
|
<listitem>
|
|
<para>Number of virtual cores per instance</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
<para>Similarly, the default RAM allocation ratio of 1.5:1 means that the
|
|
scheduler allocates instances to a physical node as long as the total
|
|
amount of RAM associated with the instances is less than 1.5 times the
|
|
amount of RAM available on the physical node.</para>
|
|
|
|
<para>For example, if a physical node has 48 GB of RAM, the scheduler
|
|
allocates instances to that node until the sum of the RAM associated with
|
|
the instances reaches 72 GB (such as nine instances, in the case where
|
|
each instance has 8 GB of RAM).</para>
|
|
|
|
<note>
|
|
<para>Regardless of the overcommit ratio, an instance can not be placed
|
|
on any physical node with fewer raw (pre-overcommit) resources than
|
|
the instance flavor requires.</para>
|
|
</note>
|
|
|
|
<para>You must select the appropriate CPU and RAM allocation ratio for
|
|
your particular use case.</para>
|
|
</section>
|
|
|
|
<section xml:id="logging">
|
|
<title>Logging</title>
|
|
|
|
<para>Logging is detailed more fully in <xref
|
|
linkend="logging_monitoring" />. However, it is an important design
|
|
consideration to take into account before commencing operations of your
|
|
cloud.<indexterm class="singular">
|
|
<primary>logging/monitoring</primary>
|
|
|
|
<secondary>compute nodes and</secondary>
|
|
</indexterm><indexterm class="singular">
|
|
<primary>compute nodes</primary>
|
|
|
|
<secondary>logging</secondary>
|
|
</indexterm></para>
|
|
|
|
<para>OpenStack produces a great deal of useful logging information,
|
|
however; but for the information to be useful for operations purposes, you
|
|
should consider having a central logging server to send logs to, and a log
|
|
parsing/analysis system (such as <phrase
|
|
role="keep-together">logstash</phrase>).</para>
|
|
</section>
|
|
|
|
<section xml:id="networking">
|
|
<title>Networking</title>
|
|
|
|
<para>Networking in OpenStack is a complex, multifaceted challenge. See
|
|
<xref linkend="network_design" />.<indexterm class="singular">
|
|
<primary>compute nodes</primary>
|
|
|
|
<secondary>networking</secondary>
|
|
</indexterm></para>
|
|
</section>
|
|
|
|
<section xml:id="conclusion">
|
|
<title>Conclusion</title>
|
|
|
|
<para>Compute nodes are the workhorse of your cloud and the place where
|
|
your users' applications will run. They are likely to be affected by your
|
|
decisions on what to deploy and how you deploy it. Their requirements
|
|
should be reflected in the choices you make.</para>
|
|
</section>
|
|
</chapter>
|