.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode ========================================== Enhanced KVM Storage QoS ========================================== https://blueprints.launchpad.net/nova/+spec/enhanced-kvm-storage-qos QEMU 1.7 [1]_ and Libvirt 1.2.11 [2]_ [3]_ provides options to specify maximum burst IOPS and maximum burst bandwidth per disk. Additionally disk IO size can be specified as part of this version of QEMU. This blueprint proposes to add support for these additonal settings to QoS specs for connected volumes. Problem description =================== At the moment, the Nova libvirt driver does not support setting storage IOPS limits. For this reason, some instances might exhaust storage resources, impacting other tenants. Use Cases ---------- * Associate burst IOPs and bandwidth front-end QoS specs for volumes exported through Cinder, which will be handled on the hypervisor side. This is in addition to the existing IOPs and bandwidth caps. * Set block IO size for IOPs to volumes exported through Cinder. * Set IOPs and bandwidth burst limits and block IO sizes for instance attached disks by using Cinder extra specs Proposed change =============== Cinder attached volumes can have additional QoS specs assigned. Front-end QoS specs should be applied by Nova when the volume is attached. These are applied per volume. This blueprint proposes additional per volume QoS specs that will be specified using Cinder volume extra specs. The libvirt driver will apply those IOPS and bandwidth caps to the instance disks on a per volume basis. Additionally, this blueprint proposes adding the block IO size control using cinder volume extra specs to cinder attached volumes on a per volume basis. Front-end volume specs will be supported only in case of volumes exported through Cinder. No QoS specs are provided for local drives provided directly by Nova. Use case examples: * Admin sets front-end QoS specs on a specific volume type cinder qos-create my-qos consumer=front-end \ total_bytes_sec_max=300000000 \ cinder qos-associate my-qos cinder create --volume-type # Those QoS specs are applied when the volume is # attached to a KVM instance nova volume-attach Available additional QoS specs, where each will add an extra line into the libvirt XML definition, specifically in the section for each device, are: * read_bytes_sec_max add value * write_bytes_sec_max add value * total_bytes_sec_max - includes read/writes add value * read_iops_sec_max add value * write_iops_sec_max add value * total_iops_sec_max - includes read/writes add value * size_iops_sec add value Alternatives ------------ None Data model impact ----------------- None REST API impact --------------- None Security impact --------------- None Notifications impact -------------------- None Other end user impact --------------------- As Nova supports N-1 version computes, if these new qoS specs are applied to a compute node running Queens, these new specs will be ignored. No error message will be provided from the Queens node. Existing Cinder QoS specs are documented in the Cinder Administration documentation set. [8]_ Performance Impact ------------------ Allowing burst IOPs and bandwidth for certain volumes will allow some applications to better perform when required. Other deployer impact --------------------- None Developer impact ---------------- None Upgrade impact -------------- None Implementation ============== Assignee(s) ----------- Primary assignee: simon-dodsley Work Items ---------- * Add burst and IO size QoS specs support in the libvirt volume driver by extending the `nova/virt/libvirt/volumes/` block volume drivers to pass the new properties of a volume to `LibvirtConfigGuestDisk` via the `qos_specs' in the `connection_info` dict. * Extend the `LibvirtConfigGuestDisk` class to add the disk burst limits of a disk device [2]_ Dependencies ============ * QEMU 1.7 [1]_ * Libvirt 1.2.11 [2]_ [3]_ QEMU included in Ubuntu 16.04 [4]_ [5]_ and libvirt at a higher version in Ubuntu 16.04 [4]_ [5]_. Also already included in Fedora 24 [6]_ [7]_ Testing ======= * Unit tests * Existing tests will ensure that the quest XML is formatted correctly assuming the required versions of libvirt and QEMU are present. Documentation Impact ==================== The additional QoS features are described in the libvirt driver documentation [1]_. Will update the Cinder Administrators Guide to add these new front-end QoS storage parameters. References ========== .. [1] https://libvirt.org/formatdomain.html .. [2] https://libvirt.org/news-2014.html .. [3] https://www.redhat.com/archives/libvir-list/2014-August/msg01354.html .. [4] https://launchpad.net/ubuntu/+source/libvirt .. [5] https://launchpad.net/ubuntu/+source/qemu .. [6] https://apps.fedoraproject.org/packages/qemu .. [7] https://apps.fedoraproject.org/packages/libvirt .. [8] https://docs.openstack.org/cinder/latest/admin/blockstorage-capacity-based-qos.html History ======= .. list-table:: Revisions :header-rows: 1 * - Release Name - Description * - Rocky - Introduced