From b02052900d129703fc3cd95ba7fd92eb9e53344d Mon Sep 17 00:00:00 2001 From: Dmitrii Shcherbakov Date: Fri, 9 Nov 2018 15:47:14 +0300 Subject: [PATCH] Add spec for ceph-rbd-mirror charm Add specification for new ceph-rbd-mirror charm to support replication of ceph pools containing RBD images between ceph clusters. Change-Id: Ic3ba2b9bbf1ef68c9d8fed1ba885f44c1afaf2eb --- specs/stein/approved/ceph-rbd-mirror.rst | 188 +++++++++++++++++++++++ 1 file changed, 188 insertions(+) create mode 100644 specs/stein/approved/ceph-rbd-mirror.rst diff --git a/specs/stein/approved/ceph-rbd-mirror.rst b/specs/stein/approved/ceph-rbd-mirror.rst new file mode 100644 index 0000000..b62ecf7 --- /dev/null +++ b/specs/stein/approved/ceph-rbd-mirror.rst @@ -0,0 +1,188 @@ +.. + Copyright 2018 Canonical Ltd. + + This work is licensed under a Creative Commons Attribution 3.0 + Unported License. + http://creativecommons.org/licenses/by/3.0/legalcode + +.. + This template should be in ReSTructured text. Please do not delete + any of the sections in this template. If you have nothing to say + for a whole section, just write: "None". For help with syntax, see + http://sphinx-doc.org/rest.html To test out your formatting, see + http://www.tele3.cz/jbar/rest/rest.html + +===================== +Ceph RBD Mirror Charm +===================== + +Problem Description +=================== + +RBD image mirroring can be used to provide a solution for Ceph cluster disaster +recovery. Ceph has a daemon called +`rbd-mirror `__ which can +be placed on a primary and a backup cluster and provide asynchronous +replication of RBD images in a given pool. + +rbd-mirror can work in two modes: + +* pool (all images in a given pool are synchronized); +* image (per-image synchronization). + +The scenario targeted by this spec involves an operator when it comes to +performing promote/demote actions and the DR procedure is operator-driven +as opposed to a full automatic failover in the event of a outage at the +primary site. + +.. note:: + + Promotion/demotion of pools will be operator driven + +.. note:: + + RADOS objects are not mirrored so for mirroring radosgw objects which + use RADOS objects or gnocchi metrics stored in RADOS objects different + backup mechanisms are required - this spec covers RBD images only. + +.. note:: + + RBD mirroring relies on the use of the exclusive-lock and journaling + features of RBD; these are only supported in the userspace integration + libraries as used by libvirt and qemu for native KVM virtualization. + This requirement excludes the use of this feature with LXD based clouds + which disable the majority of RBD features for compatibility with the + Linux kernel RBD driver. + +.. note:: + + The initial RBD mirror charm will only support mirroring of whole + pools. + +Proposed Change +=============== + +High Level Design +----------------- + +As rbd-mirror is a separate package and the service itself acts as an RBD +client it makes sense to implement the target functionality in a separate +principle charm (ceph-rbd-mirror). The charm will accept parameters on which +pools to replicate and be able to be related to multiple ceph-mon applications +in separate clusters. + +The charm will relate to a local ceph cluster and a remote ceph cluster +typically using a cross model relation. + +A new interface type ('rbd-mirror') will be created to support this +integration; this will be provided by the ceph-mon charm, and consumed by +the new ceph-rbd-mirror charm for both local and remote cluster connections. + +Each rbd-mirror daemon requires a key for connectivity to the local cluster +(named uniquely for the daemon) and a key for connectivity to the remote +cluster (named globally for all rbd-mirror daemons). Multiple ceph +configurations will also be maintained on the ceph-rbd-mirror units - +'ceph' to reference the local cluster and 'remote' to reference the +remote cluster. Configuration files and keys will be prefixed inline +with this naming - for example: + +.. code:: + + $ ls /etc/ceph + ceph.conf + ceph.client.rbd-mirror..keyring + remote.conf + remote.client.rbd-mirror.keyring + +In order to support resilience and scale-out of rbd mirroring, multiple +units of the charm may be deployed; as a result this feature will only +be supported with Ceph Luminous or later (which support multiple instances +of the rbd-mirror service). + +Deployment and Scalability Considerations +----------------------------------------- + +From the deployment perspective the charm units should have high-bandwidth +and low-latency L3 connectivity to access and replication networks of both +clusters to be able to keep up with the changes to Ceph pools it tries to +replicate. At minimum, static routes will need to be configured on the node +running rbd-mirror daemon but that is outside of the scope of this spec. + +Multiple units of the ceph-rbd-mirror charm may be used to scale out +replication traffic. + +Alternatives +------------ + +No alternative solutions have been considered. + +Dependencies +============ + +This feature relies on use of a Juju version which supports cross model +relations. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + + + +Gerrit Topic +------------ + +Use Gerrit topic "rbd-mirror" for all patches related to this spec. + +.. code-block:: bash + + git-review -t rbd-mirror + +Work Items +---------- + +* Implement a new reactive charm called ceph-rbd-mirror. +* Implement the following relation: + * rbd-mirror - ceph-mon (local and cross model). +* Add "cluster" endpoint to extra-bindings in metadata.yaml to allow binding + the "cluster" endpoint to a ceph replication space. +* ceph-mon relations should retrieve cluster details and cephx keys via the + broker protocol implemented in Ceph charms (code reuse). +* Add config options to specify pool names for replication. +* Automate creation of pools on a backup cluster if they are not present. +* Add actions to promote and demote pools. +* Enable RBD journaling feature as documented in rbd-mirror docs. +* Write unit tests. +* Write functional tests via zaza framework. + +Repositories +------------ + +A new git repository will be required for the ``ceph-rbd-mirror`` charm: + +.. code-block:: bash + + git://git.openstack.org/openstack/charm-ceph-rbd-mirror + +Documentation +------------- + +The ``ceph-rbd-mirror`` charm should contain a README with instructions on +deploying the charm and on limitations related to scalability and networking. + +Security +-------- + +- Users created for replication must not have admin privileges - they only + need to be able to write to the pools they require on the target cluster. + This is supported through the existing group based permissions system + in the ceph-mon broker using the 'rbd' profile for mon and osd permissions. + +Testing +------- + +Code written or changed will be covered by unit tests; functional testing will +be done using ``Zaza``.