diff --git a/specs/ocata/extended-filesystems.rst b/specs/ocata/extended-filesystems.rst new file mode 100644 index 0000000..2eab03c --- /dev/null +++ b/specs/ocata/extended-filesystems.rst @@ -0,0 +1,649 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + + Sections of this template were taken directly from the Nova spec + template at: + https://github.com/openstack/nova-specs/blob/master/specs/juno-template.rst + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/trove/+spec/awesome-thing should be named + awesome-thing.rst. + + Please do not delete any of the sections in this template. If you + have nothing to say for a whole section, just write: None + + Note: This comment may be removed if desired, however the license notice + above should remain. + + +================================= +Trove Extended Filesystem Support +================================= + +.. If section numbers are desired, unindent this + .. sectnum:: + +.. If a TOC is desired, unindent this + .. contents:: + +Trove currently only supports a single volume on a guest instance, and +that volume must be a block device. This spec outlines a proposal for +supporting multiple volumes, both block devices and filesystems, on +the guest. + +Launchpad Blueprint: +https://blueprints.launchpad.net/trove/+spec/extended-filesystems + + +Problem Description +=================== + +In the current implementation, during instance creation the +taskmanager allocates a single cinder volume and informs the guest +during the prepare phase. The guest then formats the volume and +mounts it as a volume to hold the data for the database. + +Additional functionality is required: + +- Some datastores require more than a single volume +- Guests should support both block devices and preformatted filesystems + + +Proposed Change +=============== + +To support this new functionality, the taskmanager will pass to the +guest a collection of volumes, each tagged as to their type and +intended use. The taskmanager will be responsible for configuring the +types of volumes to be attached to a guest and how those volumes will +be provisioned with Cinder or Nova; the guest will be responsible +for preparing the block device, if required, and connecting the device +to the filesystem on the guest operating system. + + +Configuration +------------- + +Three configuration values will be used to assist in attaching volumes +to instances. + +cg_volumes +////////// + +The global *cg_volumes* configuration setting in the DEFAULT section +of the taskmanager.conf file will contain a list of volume_types which +support consistency groups. In future feature development, only +volume types which support consistency groups will be available for +snapshots and backups. + +.. code-block:: bash + + cg_volumes = NETAPP,SCALEIO + +For Openstack allocated volumes, the values of the *cg_volumes* +setting should correspond to the *provider_type* parameters in the +sections that follow. + +guest_device_type +///////////////// + +The global *guest_device_type* configuration setting describes how the +guest instance should deal with devices provided to it by the taskmanager. + +.. code-block:: python + + cfg.DictOpt('guest_device_type', + help="Dictionary of device types with help", + default={ + 'block_ext4': "Block storage to be formatted and managed by guest", + 'shared': "Shared storage, typically NFS, to be mounted as is" + } + ) + +available_volume_kind +///////////////////// + +The per-datastore *available_volume_kind* configuration setting configures +the volume_kinds available to a specific datastore: + +.. code-block:: python + + cfg.DictOpt('available_volume_kind', + help="Dictionary of available volume types with help", + default={ + 'data': "Storage volume for the database", + 'binlog': "Volume to hold binary logs for the database", + 'user_log': "Volume to hold user and error logs" + } + ) + +In the initial implementation, this setting will be used to prevent +spinning up a collection of instances in a cluster, only to find that +an inappropriate volume configuration was selected. + +/etc/trove/guest.conf +///////////////////// + +A new olso config file will be introduced on the guest. This file is +to be installed on the guest image by the DIB elements and will +contain configuration options specific to the operating system and +database software installed on the guest. + +The *device_type* configuration setting describes how the guest +instance should prepare a volume before mounting, plus any other +device specific configuration. + +The *volume_mapping* configuration setting configures for each +operating system how the guest should handle the volumes provided to +it. The most common setting here would be where in the filesystem the +device should be mounted. The *mount_priority* option, if supplied, +is used to configure mount order in the event that devices need to be +mounted beneath each other. + +This description outlines the set of options defined specifically for +extended filesystem support: + +.. code-block:: python + + from oslo_config import cfg + + device_type = [ + cfg.DictOpt('block_ext4', help="Block device to be formatted by guest", + default={ + 'raw_block_device': True, + 'format_options': 'ext4', + 'mount_type': 'ext4', + 'mount_options': '-o rw' + }, + ), + cfg.DictOpt('shared', help="Shared filesystem (NFS) to be used as is", + default={ + 'raw_block_device': False + 'mount_type': 'nfs', + 'mount_options': '-o rw' + }, + ) + ] + + volume_mapping = [ + cfg.DictOpt('data', help="Storage volume for the database", + default={ + 'mount_point': '/var/lib/mysql', + 'mount_priority': 1 + }, + ), + cfg.DictOpt('binlog', + help="Volume to hold binary logs for the database", + default={ + 'mount_point': '/var/lib/mysql/binlog', + 'mount_priority': 2 + }, + ), + cfg.DictOpt('user_log', help="Volume to hold user and error logs", + default={ + 'mount_point': '/var/lib/mysql/userlog' + 'mount_priority': 2 + } + ) + ] + +Database +-------- + +A new table *volume_config* will be created to describe the volumes to +be configured. Each volume configuration will detail each volume to +be attached to an instance. The *volume_config* table will have the +following columns: + +================= ====================================================== +Column Name Description +================= ====================================================== +id uuid identifier +name Name of volume configuration +volume_kind intended use - defined by datastore (data, binlog, userlog) +size size of volume (unless overridden by user) +provider Openstack component providing volume ("cinder", "nova") +provider_type Volume type passed to Cinder/Nova +device_type Indicates to guest how to prepare volume +================= ====================================================== + +For example, an instance which requires its data to be stored on a +block device and its logs to be stored on a different volume would be +configured as follows: + +======= ======= =========== ==== ======== ============= =========== +id name volume_kind size provider provider_type device_type +======= ======= =========== ==== ======== ============= =========== + vc1 root 3 cinder vtype1 root + vc1 data 10 cinder vtype1 block_ext4 + vc1 binlog 2 cinder vtype1 block_ext4 +======= ======= =========== ==== ======== ============= =========== + +Instances which do not specify a volume_config to use in allocating +the instance will use the volume_config named "default". If no such +volume_config exists, the instance will default to the current single +Cinder volume. + +Each volume defined in the volume_config will have a size specified +for it. This size can be overridden by the user, either by the +"--sysvol:,size=" option, or by the "--size" option +for the data volume. This eliminates the current requirement that the +"--size" option be specified for "trove create" or "trove +cluster-create"; note that this means that should the user specify +neither volume_config nor size when no "default" volume config is +defined, the create call will fail in the taskmanager rather than the +current behaviour of catching the discrepancy in the shell. + +Two volume providers will initially be supported. If the "cinder" +provider is selected, the volume will be allocated through the Cinder +API. If the "nova" provider is specified, root volumes will be +alloced on a root disk on the compute host and volumes with a +volume_kind other than "root" will be allocated on Ephemeral storage. +An error will be raised by Nova should the allocated volume sizes +exceed the amount of storage configured in the instance flavor +selected. Future volume provide support may include support for +Manila volumes and/or multi-attach Cinder volumes. + +A new table *instance_volumes* will be added to track the volumes +associated with an instance. This will include both volumes created +by the taskmanager based on the datastore volume configuration and +volumes specified by the user. + +================= ====================================================== +Column Name Description +================= ====================================================== +instance_id Id of instance to which volumes are associated +volume_kind intended use - defined by datastore (data, binlog, userlog) +volume_id Id of volume in Cinder or Manila (Null for ephemeral) +provider Component providing volume ("cinder", "nova", "manila") +device_type Indicates to guest how to prepare volume +================= ====================================================== + +An instance configured as above would be represented by the following +entries in the *instance_volumes* table: + +========= =========== =========== ======== =========== +volume_id instance_id volume_kind provider device_type +========= =========== =========== ======== =========== + root cinder root + data cinder block_ext4 + binlog cinder block_ext4 +========= =========== =========== ======== =========== + +For backward compatibility, instances which have no entries in the +*instance_volumes* table will be assumed to have their root volume +allocated on Nova local storage. + + +Public API +---------- + +*volume_config* and *volume_list* parameters will be added to the + payload of the following REST APIs: + +- create +- cluster create +- cluster grow + +The *volume_config* parameter, if specified, will select the volume +configuration to select from the volume_config table. If not +specified, a root volume on Nova storage and a single Cinder data +volume will be configured. + +The *volume_list* will encapsulate the information from the *--sysvol* +options. For each of these options, for backward compatibility, if +the *volume_list* parameter is not included in the API payload, it +will be computed from the *volume* parameter. It will be an error if +both *volume* and *volume_list* (or neither) are specified. + +The *resize_volume* API will be updated to include the volume_kind of +the volume to be resized. For backwards compatibility, if no +volume_kind is specified, the "data" volume will be resized. An error +will be returned if the provider does not support the resize operation +on the specified volume. + +There will additionally be a new "GET volume_configs" API which will +return a list of available volume configurations. These volume +configurations will be the valid values to be specified for the above +*volume_config* parameter. + +Request:: + + GET v1/{tenant_id}/volume_configs + { + } + +Response:: + + { + [ + "default", + "ora_prod" + ] + } + +Plus a new "GET volume_configs/" API which will return details +about a given volume_config. + +Request:: + + GET v1/{tenant_id}/volume_configs/ + { + } + +Response:: + + { + "instance_id": , + "volume_kind": 'data', + "volume_id": , + "provider": "cinder", + "device_type": "block_ext4" + } + +Plus a new "GET instance//volumes" API which will return +details about the volumes attached to a given instance. + +Request:: + + GET v1/{tenant_id}/instance//volumes + { + } + +Response:: + + { + [ + { + 'volume_id': , + 'size': 8, + 'volume_kind': 'root', + 'provider': 'cinder', + 'device_type': 'root' + }, + { + 'volume_id': , + 'size': 50, + 'volume_kind': 'data', + 'provider': 'cinder', + 'device_type': 'block_ext4' + }, + { + 'volume_id': , + 'size': 2, + 'volume_kind': 'binlog', + 'provider': 'cinder', + 'device_type': 'block_ext4' + } + ] + } + +Public API Security +------------------- + +No impact. + +Python API +---------- + +*volume_config* and *volume_list* parameters will be added to the + following python APIs: + +- Instance.create +- Cluster.create +- Cluster.grow + +See above for descriptions of the *volume_config* and *volume_list* +parameters. + +The *volume* parameter will be deprecated in the above APIs. + +The Instance.resize_volume python API will be updated to include the +volume_kind of the volume to be resized. + +A new *volume_configs* object will be added to the Trove python API. +This client API will implement "list" and "show" methods. + +CLI (python-troveclient) +------------------------ + +Three new sub-commands will be added to the trove-manage command to +support associating volumes with datastore versions: + +.. code-block:: bash + + $ trove-manage volume-config-add \ + + + $ trove-manage volume-config-delete + + $ trove-manage volume-config-list + +For example, to configure a volume configuration "ora_prod" to have a data +volume and an optional binlog volume, both Cinder block devices, the +following commands would be executed: + +.. code-block:: bash + + $ trove-manage volume-config-add ora_prod \ + data cinder ram_backed block_ext4 True + $ trove-manage volume-config-volume-add ora_prod \ + binlog cinder ram_backed block_ext4 False + +A new *--sysvol* option will be added to several trove CLI commands to +support the configuration of volumes created by the taskmanager. The +*--sysvol* option may be specified multiple times to configure +multiple volumes. Each specification will include the *volume_kind* +for which a specification is being made followed by a colon separated +list of configuration options - for the initial implementation, only +size will be supported. + +.. code-block:: bash + + $ trove create mydb myflavor --datastore=foo --datastore_version=1.0 \ + --volume_config=ora_prod \ + --sysvol=data:size=5 --sysvol=binlog:size=2 + +For backwards compatibility, an alternative to this command would be +one which specifies the size of the data volume with the "--size" +option, though this may be deprecated in future releases: + + $ trove create mydb myflavor --datastore=foo --datastore_version=1.0 \ + --volume_config=ora_prod --size=5 --sysvol=binlog:size=2 + +A new option will be added to the resize-volume command to support +specifying which of the volumes attached to the instance are to be +resized: + +.. code-block:: bash + + $ trove resize-volume my_inst 3 --volume_kind binlog + +For backwards compatibility, if the --volume_kind option is not +specified, the volume of type "data" will be resized. + +A new "trove volume-config-list" CLI command will be added to return a +list of the names of the available volume_configs. + + +.. code-block:: bash + + $ trove volume-config-list + +==============+ + | VolumeConfig | + +==============+ + | default | + | ora_prod | + +==============+ + +And a corresponding new "trove volume-config-show". + +.. code-block:: bash + + $ trove volume-config-show ora_prod + =========== ==== ======== ============= =========== + volume_kind size provider provider_type device_type + =========== ==== ======== ============= =========== + root 3 cinder vtype1 root + data 10 cinder vtype1 block_ext4 + binlog 2 cinder vtype1 block_ext4 + =========== ==== ======== ============= =========== + +And of course a command to show which volumes are attached to a +specific instance. + + $ trove volumes-list + =========== =========== ======== =========== ========= + volume_kind size provider device_type volume_id + =========== =========== ======== =========== ========= + root 8 cinder root + data 50 cinder block_ext4 + binlog 2 cinder block_ext4 + =========== =========== ======== =========== ========= + + +Internal API +------------ + +The *create_instance* API will be updated to include the *volume_info* +parameter. + +The *resize_volume* API will be updated to include the id and +volume_kind of the volume to be resized. + + +Guest Agent +----------- + +A new prepare call will be implemented which replaces the device_path +and mount_point parameters with a new *device_config* list. The +*device_config* will supply information about the number and types of +volumes to be configured on the guest: + +.. code-block:: json + + [ + { + 'volume_kind': 'data', + 'device_type': 'block_ext4', + 'device_path': '/dev/vdb1' + }, + { + 'volume_kind': 'binlog', + 'device_type': 'block_ext4', + 'device_path': '/dev/vdb2' + }, + ] + +To support backwards compatibility when the guest is upgraded before +the guestagent API, the guest agent's prepare method will take both +old and new parameters. The correct behaviour will be deduced based +on the values of the paramaters. + +It is up to each datastore to define what volumes are supported, the +meaning of each value of the volume_kind, and where each volume +will be mounted on the local filesystem. + +The guestagent resize_fs API will be updated to include the +*volume_kind* parameter. The *mount_point* parameter will be ignored +when the *volume_kind* parameter is specified. + +As the parameter to the *post_upgrade* RPC endpoint is a dict, it will +be updated to remain compatible with both old and new guests, +providing both the mount point of the data volume plus the full +*device_config* context. + + +Alternatives +------------ + +Not available. + + +Dashboard Impact (UX) +===================== + +The dashboard would be changed to allow the user to specify the +volume_config on the instance.create, cluster.create, and cluster.grow +functions. A volume.configs.list method will be added to the Python +API to facilite this functionality. + +A new panel/action will be added to the instance panel to show the +volumes configured for a particular instance. + +The volume_resize functionality will be updated to allow the user to +specify the volume_kind that they wish to resize. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + 6-morgan + +Dashboard assignee: + + +Milestones +---------- + +Target Milestone for completion: + post ocata + +Work Items +---------- + +- implement taskmanager changes +- implement base guestagent and guestagent.api changes +- adapt datastores to new APIs +- develop unit tests + + +Upgrade Implications +==================== + +Backward compatibility for this feature will be dependent on the new +RPC versioning feature. + + +Dependencies +============ + +na + + +Testing +======= + +An int-test should be developed to test adding multiple volumes for at +least one datastore. + + +Documentation Impact +==================== + +This feature will affect the following documents: + +- Installation +- API +- CLI +- Building Guest Images for Openstack Trove + + +References +========== + +na + +Appendix +======== + +Any additional technical information and data.