diff --git a/specs/queens/cyborg-spdk-driver-proposal.rst b/specs/queens/cyborg-spdk-driver-proposal.rst new file mode 100755 index 00000000..6b26fbaf --- /dev/null +++ b/specs/queens/cyborg-spdk-driver-proposal.rst @@ -0,0 +1,221 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +=========================== +Cyborg SPDK Driver Proposal +=========================== + +https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-spdk-driver + +This spec proposes to provide the initial design for Cyborg's SPDK driver. + +Problem description +=================== + +SPDK is a high performance kit and provides a user space, polled-mode, +asynchronous, lockless NVMe driver for storage acceleration on the +backend. Our goal is to add a SPDK driver for Cyborg to manage SPDK, +and further improve storage performance. + +Use Cases +--------- + +* When Cinder uses Ceph as its backend, the user should be able to +use the Cyborg SPDK driver to discover the SPDK accelerator backend, +enumerate the list of the Ceph nodes that have installed the SPDK. +* When Cinder directly uses SPDK's BlobStore as its backend, the user +should be able to accomplish the same life cycle management operations +for SPDK as mentioned above. After enumerating the SPDK, the user can +attach (install) SPDK on that node. When the task completes, the user +can also detach the SPDK from the node. Last but not least the user +should be able to update the latest and available SPDK. + +Proposed change +=============== + +In general, the goal is to develop the Cyborg SPDK driver that supports +discover/list/update/attach/detach operations for SPDK framework. + +SPDK framework +-------------- + +The SPDK framework comprises of the following components: + + +-----------userspace--------+ +--------------+ + | +------+ +------+ +------+ | | +-----------+ | ++---+ | |DPDK | |NVMe | |NVMe | | | | Ceph | | +| N +-+-+NIC | |Target| |Driver+-+-+ |NVMe Device| | +| I | | |Driver| | | | | | | +-----------+ | +| C | | +------+ +------+ +------+ | | +-----------+ | ++---+ | +------------------------+ | | | Blobstore | | + | | DPDK Libraries | | | |NVMe Device| | + | +------------------------+ | | +-----------+ | + +----------------------------+ +---------------+ + +BlobStore NVMe Device Format +---------------------------- + +BlobStore owns the entire NVMe device including metadata management +and data management, which defines three basic units of disk space (like +logical block, page, cluster). The NVMe device is divided into clusters +starting from the first logical block. + +LBA 0 LBA N ++-----------+-----------+-----+-----------+ +| Cluster 0 | Cluster 1 | ... | Cluster N | ++-----------+-----------+-----+-----------+ + +Cluster0 has special format which consists of pages. Page0 is the +first page of Cluster0. Super Block contains the basic information of +BlobStore. + ++--------+-------------------+ +| Page 0 | Page 1 ... Page N | ++--------+-------------------+ +| Super | Metadata Region | +| Block | | ++--------+-------------------+ + +Each blob is allocated a non-contiguous set of pages. These pages form +a linked list. +In general, the BlobStore adopts direct operation of bare metal device and +avoids the filesystem, which improves efficiency. + +Life Cycle Management Phases +---------------------------- +* We should be able to add a judgement whether the backend node has SPDK kit +in generic driver module. If true, initialize the DPDK environment (such as +hugepage). +* Import the generic driver module, and then we should be able to +discover (probe) the system for SPDK. +* Determined by the backend storage scenario, enumerate (list) the optimal +SPDK node, returning a boolean value to judge whether the SPDK should be +attached. +* After the node where SPDK will be running is attached, we can now send a +request about the information of namespaces, and then create an I/O queue +pair to submit read/write requests to a namespace. +* When Ceph is used as the backend, as the latest Ceph (such as Luminous) +uses the BlueStore to be the storage engine, BlueStore and BlobStore are +very similar things. We will not be able to use BlobStore to accelerate +Ceph, but we can use Ioat and poller to boost speed for storage. +* When SPDK is used as the backend, we should be able to use BlobStore to +improve performance. +* Whenever user requests, we should be able to detach the SPDK device. +* Whenever user requests, we should be able to update SPDK to the latest and +stable release. + +Alternatives +------------ + +None + +Data model impact +----------------- + +* The Cyborg SPDK driver will notify Cyborg Agent to update the database +when discover/list/update/attach/detach operations take place. + +REST API impact +--------------- + +This blueprint proposes to add the following APIs: +*cyborg discover-driver（driver_type） +*cyborg driver-list(driver_type) +*cyborg install-driver(driver_id, driver_type) +*cyborg attach-instance +*cyborg detach-instance +*cyborg uninstall-driver(driver_id, driver_type) +*cyborg update-driver + +Security impact +--------------- + +None + +Notifications impact +-------------------- + +None + +Other end user impact +--------------------- + +None + +Performance Impact +------------------ + +The SPDK can provide a user space, polled-mode, asynchronous, +lockless NVMe driver for storage acceleration on the backend. + +Other deployer impact +--------------------- + +Deployers can call SPDK from the nodes which have installed SPDK +after the driver has been implemented. + +Developer impact +---------------- + +None + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + luwei he + +Work Items +---------- + +* Implement the cyborg-spdk-driver in this spec. +* Propose SPDK to py-spdk. The py-spdk is designed as a SPDK client +which provides the python binding. + + +Dependencies +============ + +* Cyborg API Spec +* Cyborg Agent Spec +* Cyborg Driver Spec +* Cyborg Conductor Spec + +Testing +======== + +* Unit tests will be added to test Cyborg SPDK driver. +* Functional tests will be added to test Cyborg SPDK driver. For example: +discover-->list-->attach，whether the workflow can be passed successfully. + +Documentation Impact +=================== + +Document SPDK driver in the Cyborg project + +References +========== + +* Cyborg API Spec +* Cyborg Agent Spec +* Cyborg Driver Spec +* Cyborg Conductor Spec + + +History +======= +c + +.. list-table:: Revisions + :header-rows: 1 + + * - Release + - Description + * - Pike + - Introduced