221 lines
7.1 KiB
ReStructuredText
221 lines
7.1 KiB
ReStructuredText
..
|
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
|
License.
|
|
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
|
|
|
==========================================
|
|
Libvirt file backed memory
|
|
==========================================
|
|
|
|
Spec for blueprint libvirt-file-backed-memory
|
|
https://blueprints.launchpad.net/nova/+spec/libvirt-file-backed-memory
|
|
|
|
With the advent of large capacity memory devices, it's now reasonable to run
|
|
virtual machines with file backed memory. This enables a much larger total
|
|
memory area per compute node
|
|
|
|
Problem description
|
|
===================
|
|
|
|
New memory technology and modern NVMe SSDs have progressed to the point where
|
|
it is reasonable to utilize these devices as a backing store for VM memory to
|
|
expand the memory capacity of a compute host.
|
|
|
|
Use Cases
|
|
---------
|
|
|
|
As an operator, I want to be able to leverage libvirt's support for
|
|
file-backed memory technologies to expand my compute node memory capacity.
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
The proposed change is to add a new option in ``nova.conf`` under the
|
|
``libvirt`` section to configure file backed memory:
|
|
|
|
* ``file_backed_memory``
|
|
|
|
``file_backed_memory`` will default to ``0``, indicating file backed memory is
|
|
disabled. When set to non-zero, the libvirt driver will include elements
|
|
to enable file backed memory within instances.
|
|
|
|
When ``file_backed_memory`` is set to non-zero, the libvirt driver will
|
|
include a ``MemoryBacking`` element for all instances on the compute node,
|
|
with a ``source`` subelement with type ``file``, and an ``access`` subelement
|
|
with mode ``shared``.
|
|
|
|
The value configured in ``file_backed_memory`` will be reported by the libvirt
|
|
driver as the total memory capacity of the compute node in MiB. As the memory
|
|
capacity is dependent on the backing store(s) in use, libvirt must report a
|
|
value other than the real system memory capacity. Available capacity will then
|
|
be calculated from the value in ``file_backed_memory`` minus the currently
|
|
used memory for instances. System memory will be used as a cache for the file
|
|
backed memory through the kernel pagecache.
|
|
|
|
As overcommit is not expected to work with file backed memory, enabling this
|
|
option requires the value of ``ram_allocation_ratio`` in ``nova.conf``, to be
|
|
set to the value ``1.0``, and will block Nova startup if this is not
|
|
configured properly.
|
|
|
|
.. note:: ``shared`` access mode is required, as ``private`` access will not
|
|
utilize an underlying backing store for pages in-use by the
|
|
instance, but will keep those pages within main system memory.
|
|
|
|
.. note:: During migration, the destination compute node must be checked for
|
|
the ``file_backed_memory`` option, and add or remove the
|
|
MemoryBacking element and subelements as appropriate, to
|
|
ensure memory is appropriately allocated during the migration
|
|
process. This can be in nova/virt/libvirt/migration.py, within the
|
|
get_updated_guest_xml method, similar to how graphics, serial,
|
|
volume, and perf events are handled today.
|
|
|
|
.. note:: Migration from compute nodes running versions of Nova without this
|
|
feature will not include the appropriate libvirt XML for
|
|
file backed memory. Nova will block these migrations, to ensure
|
|
all instances migrated to a node with ``file_backed_memory`` enabled
|
|
are actually using file backed memory. Migrations will be blocked
|
|
within ``check_can_live_migrate_destination``
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
In Nova, the available memory capacity could be detected dynamically. Due to
|
|
the variety of memory and SSD technologies and ways the memory backing
|
|
directory could be configured within libvirt and on the system, this would
|
|
require implementing an external call point to determine the available
|
|
capacity, or require vendor-specific code included in Nova. Either option
|
|
would significantly increase the scope of this spec.
|
|
|
|
In Nova, file backed memory could be enabled via a flavor extra-spec, allowing
|
|
for control per individual instance. This results in an inconsistent use of RAM
|
|
and backing devices, leading to a confusing / conflicting memory capacity
|
|
calculation on the compute node.
|
|
|
|
Not implement this spec at all. In this case, a compute node is limited to
|
|
the capacity of the standard DRAM in the compute node.
|
|
|
|
Data model impact
|
|
-----------------
|
|
|
|
None
|
|
|
|
REST API impact
|
|
---------------
|
|
|
|
None
|
|
|
|
Security impact
|
|
---------------
|
|
|
|
As the instance memory will now be contained within a file on a filesystem,
|
|
instance memory will be accessible to any process owned by a user with
|
|
permissions necessary to access the files. Currently, the root user on a
|
|
compute node already has capability to read system memory and VM memory.
|
|
|
|
Notifications impact
|
|
--------------------
|
|
|
|
None
|
|
|
|
Other end user impact
|
|
---------------------
|
|
|
|
None
|
|
|
|
Performance Impact
|
|
------------------
|
|
|
|
When the ``file_backed_memory`` option is enabled, instance memory performance
|
|
will be dependent on the backing store, as configured by the operator.
|
|
|
|
Other deployer impact
|
|
---------------------
|
|
|
|
New config options would need to be explicitly enabled to take effect.
|
|
|
|
Prior to enabling new config options, an operator should configure the libvirt
|
|
``memory_backing_dir`` configuration setting to point to their selected backing
|
|
store, such as a filesystem on an NVMe device.
|
|
|
|
Enabling ``file_backed_memory`` will reject migrations from compute nodes
|
|
running versions of Nova that do not support file backed memory.
|
|
|
|
Developer impact
|
|
----------------
|
|
|
|
None
|
|
|
|
Upgrade impact
|
|
--------------
|
|
|
|
Enabling ``file_backed_memory`` will reject migrations from compute nodes
|
|
running versions of Nova that do not support file backed memory.
|
|
|
|
It's recommended to only enable ``file_backed_memory`` after all compute nodes
|
|
are upgraded.
|
|
|
|
Implementation
|
|
==============
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignee:
|
|
Zack Cornelius <zcornelius>
|
|
|
|
Other contributors:
|
|
None
|
|
|
|
Work Items
|
|
----------
|
|
|
|
* Add new configuration options to nova.conf
|
|
* Check for configuration options within libvirt driver and report the
|
|
file-backed capacity value instead of system memory
|
|
* Generate additional libvirt domain XML as needed
|
|
* Validate / correct libvirt domain XML during migration process
|
|
|
|
Dependencies
|
|
============
|
|
|
|
- Qemu >= 2.6.0
|
|
- Libvirt >= 4.0.0
|
|
|
|
Testing
|
|
=======
|
|
|
|
Unit tests will be added to validate the instances booted on host have files
|
|
in the libvirt ``memory_backing_dir`` on the host.
|
|
|
|
A test will be needed for the edge cases of migrating between a host with the
|
|
new options enabled, and a host without the new options enabled, to ensure the
|
|
memory is allocated from the correct source.
|
|
|
|
We will investigate adding an integration test for this by creating a large
|
|
ramdisk, enabling these settings, and validating that an instance is utilizing
|
|
memory within files on that ramdisk. We believe this test layout should be
|
|
possible, but if not, will fall back to relying on unit tests.
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
The documentation for ``nova.conf`` should be updated with the new
|
|
configuration options.
|
|
|
|
References
|
|
==========
|
|
|
|
* https://libvirt.org/formatdomain.html#elementsMemoryBacking
|
|
|
|
History
|
|
=======
|
|
|
|
.. list-table:: Revisions
|
|
:header-rows: 1
|
|
|
|
* - Release Name
|
|
- Description
|
|
* - Rocky
|
|
- Introduced
|