Extended Filesytem Support
Outlines a proposal for supporting the creation of Trove instances with multiple volumes attached. Included in this spec is a new guest configuration file. Change-Id: If14438396a2864428cf8e0b9a70368e889e41906
This commit is contained in:
Morgan Jones
parent
397d00016a
commit
fe25e27ae4
|
|
@ -0,0 +1,649 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
Sections of this template were taken directly from the Nova spec
|
||||
template at:
|
||||
https://github.com/openstack/nova-specs/blob/master/specs/juno-template.rst
|
||||
|
||||
..
|
||||
This template should be in ReSTructured text. The filename in the git
|
||||
repository should match the launchpad URL, for example a URL of
|
||||
https://blueprints.launchpad.net/trove/+spec/awesome-thing should be named
|
||||
awesome-thing.rst.
|
||||
|
||||
Please do not delete any of the sections in this template. If you
|
||||
have nothing to say for a whole section, just write: None
|
||||
|
||||
Note: This comment may be removed if desired, however the license notice
|
||||
above should remain.
|
||||
|
||||
|
||||
=================================
|
||||
Trove Extended Filesystem Support
|
||||
=================================
|
||||
|
||||
.. If section numbers are desired, unindent this
|
||||
.. sectnum::
|
||||
|
||||
.. If a TOC is desired, unindent this
|
||||
.. contents::
|
||||
|
||||
Trove currently only supports a single volume on a guest instance, and
|
||||
that volume must be a block device. This spec outlines a proposal for
|
||||
supporting multiple volumes, both block devices and filesystems, on
|
||||
the guest.
|
||||
|
||||
Launchpad Blueprint:
|
||||
https://blueprints.launchpad.net/trove/+spec/extended-filesystems
|
||||
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
|
||||
In the current implementation, during instance creation the
|
||||
taskmanager allocates a single cinder volume and informs the guest
|
||||
during the prepare phase. The guest then formats the volume and
|
||||
mounts it as a volume to hold the data for the database.
|
||||
|
||||
Additional functionality is required:
|
||||
|
||||
- Some datastores require more than a single volume
|
||||
- Guests should support both block devices and preformatted filesystems
|
||||
|
||||
|
||||
Proposed Change
|
||||
===============
|
||||
|
||||
To support this new functionality, the taskmanager will pass to the
|
||||
guest a collection of volumes, each tagged as to their type and
|
||||
intended use. The taskmanager will be responsible for configuring the
|
||||
types of volumes to be attached to a guest and how those volumes will
|
||||
be provisioned with Cinder or Nova; the guest will be responsible
|
||||
for preparing the block device, if required, and connecting the device
|
||||
to the filesystem on the guest operating system.
|
||||
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
Three configuration values will be used to assist in attaching volumes
|
||||
to instances.
|
||||
|
||||
cg_volumes
|
||||
//////////
|
||||
|
||||
The global *cg_volumes* configuration setting in the DEFAULT section
|
||||
of the taskmanager.conf file will contain a list of volume_types which
|
||||
support consistency groups. In future feature development, only
|
||||
volume types which support consistency groups will be available for
|
||||
snapshots and backups.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cg_volumes = NETAPP,SCALEIO
|
||||
|
||||
For Openstack allocated volumes, the values of the *cg_volumes*
|
||||
setting should correspond to the *provider_type* parameters in the
|
||||
sections that follow.
|
||||
|
||||
guest_device_type
|
||||
/////////////////
|
||||
|
||||
The global *guest_device_type* configuration setting describes how the
|
||||
guest instance should deal with devices provided to it by the taskmanager.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
cfg.DictOpt('guest_device_type',
|
||||
help="Dictionary of device types with help",
|
||||
default={
|
||||
'block_ext4': "Block storage to be formatted and managed by guest",
|
||||
'shared': "Shared storage, typically NFS, to be mounted as is"
|
||||
}
|
||||
)
|
||||
|
||||
available_volume_kind
|
||||
/////////////////////
|
||||
|
||||
The per-datastore *available_volume_kind* configuration setting configures
|
||||
the volume_kinds available to a specific datastore:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
cfg.DictOpt('available_volume_kind',
|
||||
help="Dictionary of available volume types with help",
|
||||
default={
|
||||
'data': "Storage volume for the database",
|
||||
'binlog': "Volume to hold binary logs for the database",
|
||||
'user_log': "Volume to hold user and error logs"
|
||||
}
|
||||
)
|
||||
|
||||
In the initial implementation, this setting will be used to prevent
|
||||
spinning up a collection of instances in a cluster, only to find that
|
||||
an inappropriate volume configuration was selected.
|
||||
|
||||
/etc/trove/guest.conf
|
||||
/////////////////////
|
||||
|
||||
A new olso config file will be introduced on the guest. This file is
|
||||
to be installed on the guest image by the DIB elements and will
|
||||
contain configuration options specific to the operating system and
|
||||
database software installed on the guest.
|
||||
|
||||
The *device_type* configuration setting describes how the guest
|
||||
instance should prepare a volume before mounting, plus any other
|
||||
device specific configuration.
|
||||
|
||||
The *volume_mapping* configuration setting configures for each
|
||||
operating system how the guest should handle the volumes provided to
|
||||
it. The most common setting here would be where in the filesystem the
|
||||
device should be mounted. The *mount_priority* option, if supplied,
|
||||
is used to configure mount order in the event that devices need to be
|
||||
mounted beneath each other.
|
||||
|
||||
This description outlines the set of options defined specifically for
|
||||
extended filesystem support:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from oslo_config import cfg
|
||||
|
||||
device_type = [
|
||||
cfg.DictOpt('block_ext4', help="Block device to be formatted by guest",
|
||||
default={
|
||||
'raw_block_device': True,
|
||||
'format_options': 'ext4',
|
||||
'mount_type': 'ext4',
|
||||
'mount_options': '-o rw'
|
||||
},
|
||||
),
|
||||
cfg.DictOpt('shared', help="Shared filesystem (NFS) to be used as is",
|
||||
default={
|
||||
'raw_block_device': False
|
||||
'mount_type': 'nfs',
|
||||
'mount_options': '-o rw'
|
||||
},
|
||||
)
|
||||
]
|
||||
|
||||
volume_mapping = [
|
||||
cfg.DictOpt('data', help="Storage volume for the database",
|
||||
default={
|
||||
'mount_point': '/var/lib/mysql',
|
||||
'mount_priority': 1
|
||||
},
|
||||
),
|
||||
cfg.DictOpt('binlog',
|
||||
help="Volume to hold binary logs for the database",
|
||||
default={
|
||||
'mount_point': '/var/lib/mysql/binlog',
|
||||
'mount_priority': 2
|
||||
},
|
||||
),
|
||||
cfg.DictOpt('user_log', help="Volume to hold user and error logs",
|
||||
default={
|
||||
'mount_point': '/var/lib/mysql/userlog'
|
||||
'mount_priority': 2
|
||||
}
|
||||
)
|
||||
]
|
||||
|
||||
Database
|
||||
--------
|
||||
|
||||
A new table *volume_config* will be created to describe the volumes to
|
||||
be configured. Each volume configuration will detail each volume to
|
||||
be attached to an instance. The *volume_config* table will have the
|
||||
following columns:
|
||||
|
||||
================= ======================================================
|
||||
Column Name Description
|
||||
================= ======================================================
|
||||
id uuid identifier
|
||||
name Name of volume configuration
|
||||
volume_kind intended use - defined by datastore (data, binlog, userlog)
|
||||
size size of volume (unless overridden by user)
|
||||
provider Openstack component providing volume ("cinder", "nova")
|
||||
provider_type Volume type passed to Cinder/Nova
|
||||
device_type Indicates to guest how to prepare volume
|
||||
================= ======================================================
|
||||
|
||||
For example, an instance which requires its data to be stored on a
|
||||
block device and its logs to be stored on a different volume would be
|
||||
configured as follows:
|
||||
|
||||
======= ======= =========== ==== ======== ============= ===========
|
||||
id name volume_kind size provider provider_type device_type
|
||||
======= ======= =========== ==== ======== ============= ===========
|
||||
<uuid0> vc1 root 3 cinder vtype1 root
|
||||
<uuid1> vc1 data 10 cinder vtype1 block_ext4
|
||||
<uuid2> vc1 binlog 2 cinder vtype1 block_ext4
|
||||
======= ======= =========== ==== ======== ============= ===========
|
||||
|
||||
Instances which do not specify a volume_config to use in allocating
|
||||
the instance will use the volume_config named "default". If no such
|
||||
volume_config exists, the instance will default to the current single
|
||||
Cinder volume.
|
||||
|
||||
Each volume defined in the volume_config will have a size specified
|
||||
for it. This size can be overridden by the user, either by the
|
||||
"--sysvol:<volume_kind>,size=<n>" option, or by the "--size" option
|
||||
for the data volume. This eliminates the current requirement that the
|
||||
"--size" option be specified for "trove create" or "trove
|
||||
cluster-create"; note that this means that should the user specify
|
||||
neither volume_config nor size when no "default" volume config is
|
||||
defined, the create call will fail in the taskmanager rather than the
|
||||
current behaviour of catching the discrepancy in the shell.
|
||||
|
||||
Two volume providers will initially be supported. If the "cinder"
|
||||
provider is selected, the volume will be allocated through the Cinder
|
||||
API. If the "nova" provider is specified, root volumes will be
|
||||
alloced on a root disk on the compute host and volumes with a
|
||||
volume_kind other than "root" will be allocated on Ephemeral storage.
|
||||
An error will be raised by Nova should the allocated volume sizes
|
||||
exceed the amount of storage configured in the instance flavor
|
||||
selected. Future volume provide support may include support for
|
||||
Manila volumes and/or multi-attach Cinder volumes.
|
||||
|
||||
A new table *instance_volumes* will be added to track the volumes
|
||||
associated with an instance. This will include both volumes created
|
||||
by the taskmanager based on the datastore volume configuration and
|
||||
volumes specified by the user.
|
||||
|
||||
================= ======================================================
|
||||
Column Name Description
|
||||
================= ======================================================
|
||||
instance_id Id of instance to which volumes are associated
|
||||
volume_kind intended use - defined by datastore (data, binlog, userlog)
|
||||
volume_id Id of volume in Cinder or Manila (Null for ephemeral)
|
||||
provider Component providing volume ("cinder", "nova", "manila")
|
||||
device_type Indicates to guest how to prepare volume
|
||||
================= ======================================================
|
||||
|
||||
An instance configured as above would be represented by the following
|
||||
entries in the *instance_volumes* table:
|
||||
|
||||
========= =========== =========== ======== ===========
|
||||
volume_id instance_id volume_kind provider device_type
|
||||
========= =========== =========== ======== ===========
|
||||
<volid0> <inst1> root cinder root
|
||||
<volid1> <inst1> data cinder block_ext4
|
||||
<volid2> <inst1> binlog cinder block_ext4
|
||||
========= =========== =========== ======== ===========
|
||||
|
||||
For backward compatibility, instances which have no entries in the
|
||||
*instance_volumes* table will be assumed to have their root volume
|
||||
allocated on Nova local storage.
|
||||
|
||||
|
||||
Public API
|
||||
----------
|
||||
|
||||
*volume_config* and *volume_list* parameters will be added to the
|
||||
payload of the following REST APIs:
|
||||
|
||||
- create
|
||||
- cluster create
|
||||
- cluster grow
|
||||
|
||||
The *volume_config* parameter, if specified, will select the volume
|
||||
configuration to select from the volume_config table. If not
|
||||
specified, a root volume on Nova storage and a single Cinder data
|
||||
volume will be configured.
|
||||
|
||||
The *volume_list* will encapsulate the information from the *--sysvol*
|
||||
options. For each of these options, for backward compatibility, if
|
||||
the *volume_list* parameter is not included in the API payload, it
|
||||
will be computed from the *volume* parameter. It will be an error if
|
||||
both *volume* and *volume_list* (or neither) are specified.
|
||||
|
||||
The *resize_volume* API will be updated to include the volume_kind of
|
||||
the volume to be resized. For backwards compatibility, if no
|
||||
volume_kind is specified, the "data" volume will be resized. An error
|
||||
will be returned if the provider does not support the resize operation
|
||||
on the specified volume.
|
||||
|
||||
There will additionally be a new "GET volume_configs" API which will
|
||||
return a list of available volume configurations. These volume
|
||||
configurations will be the valid values to be specified for the above
|
||||
*volume_config* parameter.
|
||||
|
||||
Request::
|
||||
|
||||
GET v1/{tenant_id}/volume_configs
|
||||
{
|
||||
}
|
||||
|
||||
Response::
|
||||
|
||||
{
|
||||
[
|
||||
"default",
|
||||
"ora_prod"
|
||||
]
|
||||
}
|
||||
|
||||
Plus a new "GET volume_configs/<config>" API which will return details
|
||||
about a given volume_config.
|
||||
|
||||
Request::
|
||||
|
||||
GET v1/{tenant_id}/volume_configs/<config>
|
||||
{
|
||||
}
|
||||
|
||||
Response::
|
||||
|
||||
{
|
||||
"instance_id": <uuid>,
|
||||
"volume_kind": 'data',
|
||||
"volume_id": <uuid>,
|
||||
"provider": "cinder",
|
||||
"device_type": "block_ext4"
|
||||
}
|
||||
|
||||
Plus a new "GET instance/<instance_id>/volumes" API which will return
|
||||
details about the volumes attached to a given instance.
|
||||
|
||||
Request::
|
||||
|
||||
GET v1/{tenant_id}/instance/<instance_uuid>/volumes
|
||||
{
|
||||
}
|
||||
|
||||
Response::
|
||||
|
||||
{
|
||||
[
|
||||
{
|
||||
'volume_id': <volid0>,
|
||||
'size': 8,
|
||||
'volume_kind': 'root',
|
||||
'provider': 'cinder',
|
||||
'device_type': 'root'
|
||||
},
|
||||
{
|
||||
'volume_id': <volid1>,
|
||||
'size': 50,
|
||||
'volume_kind': 'data',
|
||||
'provider': 'cinder',
|
||||
'device_type': 'block_ext4'
|
||||
},
|
||||
{
|
||||
'volume_id': <volid2>,
|
||||
'size': 2,
|
||||
'volume_kind': 'binlog',
|
||||
'provider': 'cinder',
|
||||
'device_type': 'block_ext4'
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Public API Security
|
||||
-------------------
|
||||
|
||||
No impact.
|
||||
|
||||
Python API
|
||||
----------
|
||||
|
||||
*volume_config* and *volume_list* parameters will be added to the
|
||||
following python APIs:
|
||||
|
||||
- Instance.create
|
||||
- Cluster.create
|
||||
- Cluster.grow
|
||||
|
||||
See above for descriptions of the *volume_config* and *volume_list*
|
||||
parameters.
|
||||
|
||||
The *volume* parameter will be deprecated in the above APIs.
|
||||
|
||||
The Instance.resize_volume python API will be updated to include the
|
||||
volume_kind of the volume to be resized.
|
||||
|
||||
A new *volume_configs* object will be added to the Trove python API.
|
||||
This client API will implement "list" and "show" methods.
|
||||
|
||||
CLI (python-troveclient)
|
||||
------------------------
|
||||
|
||||
Three new sub-commands will be added to the trove-manage command to
|
||||
support associating volumes with datastore versions:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ trove-manage volume-config-add <name> <volume_kind> \
|
||||
<provider> <provider_type> <device_type> <required>
|
||||
|
||||
$ trove-manage volume-config-delete <name> <volume_kind>
|
||||
|
||||
$ trove-manage volume-config-list <name>
|
||||
|
||||
For example, to configure a volume configuration "ora_prod" to have a data
|
||||
volume and an optional binlog volume, both Cinder block devices, the
|
||||
following commands would be executed:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ trove-manage volume-config-add ora_prod \
|
||||
data cinder ram_backed block_ext4 True
|
||||
$ trove-manage volume-config-volume-add ora_prod \
|
||||
binlog cinder ram_backed block_ext4 False
|
||||
|
||||
A new *--sysvol* option will be added to several trove CLI commands to
|
||||
support the configuration of volumes created by the taskmanager. The
|
||||
*--sysvol* option may be specified multiple times to configure
|
||||
multiple volumes. Each specification will include the *volume_kind*
|
||||
for which a specification is being made followed by a colon separated
|
||||
list of configuration options - for the initial implementation, only
|
||||
size will be supported.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ trove create mydb myflavor --datastore=foo --datastore_version=1.0 \
|
||||
--volume_config=ora_prod \
|
||||
--sysvol=data:size=5 --sysvol=binlog:size=2
|
||||
|
||||
For backwards compatibility, an alternative to this command would be
|
||||
one which specifies the size of the data volume with the "--size"
|
||||
option, though this may be deprecated in future releases:
|
||||
|
||||
$ trove create mydb myflavor --datastore=foo --datastore_version=1.0 \
|
||||
--volume_config=ora_prod --size=5 --sysvol=binlog:size=2
|
||||
|
||||
A new option will be added to the resize-volume command to support
|
||||
specifying which of the volumes attached to the instance are to be
|
||||
resized:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ trove resize-volume my_inst 3 --volume_kind binlog
|
||||
|
||||
For backwards compatibility, if the --volume_kind option is not
|
||||
specified, the volume of type "data" will be resized.
|
||||
|
||||
A new "trove volume-config-list" CLI command will be added to return a
|
||||
list of the names of the available volume_configs.
|
||||
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ trove volume-config-list
|
||||
+==============+
|
||||
| VolumeConfig |
|
||||
+==============+
|
||||
| default |
|
||||
| ora_prod |
|
||||
+==============+
|
||||
|
||||
And a corresponding new "trove volume-config-show".
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ trove volume-config-show ora_prod
|
||||
=========== ==== ======== ============= ===========
|
||||
volume_kind size provider provider_type device_type
|
||||
=========== ==== ======== ============= ===========
|
||||
root 3 cinder vtype1 root
|
||||
data 10 cinder vtype1 block_ext4
|
||||
binlog 2 cinder vtype1 block_ext4
|
||||
=========== ==== ======== ============= ===========
|
||||
|
||||
And of course a command to show which volumes are attached to a
|
||||
specific instance.
|
||||
|
||||
$ trove volumes-list <instance>
|
||||
=========== =========== ======== =========== =========
|
||||
volume_kind size provider device_type volume_id
|
||||
=========== =========== ======== =========== =========
|
||||
root 8 cinder root <volid0>
|
||||
data 50 cinder block_ext4 <volid1>
|
||||
binlog 2 cinder block_ext4 <volid2>
|
||||
=========== =========== ======== =========== =========
|
||||
|
||||
|
||||
Internal API
|
||||
------------
|
||||
|
||||
The *create_instance* API will be updated to include the *volume_info*
|
||||
parameter.
|
||||
|
||||
The *resize_volume* API will be updated to include the id and
|
||||
volume_kind of the volume to be resized.
|
||||
|
||||
|
||||
Guest Agent
|
||||
-----------
|
||||
|
||||
A new prepare call will be implemented which replaces the device_path
|
||||
and mount_point parameters with a new *device_config* list. The
|
||||
*device_config* will supply information about the number and types of
|
||||
volumes to be configured on the guest:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
[
|
||||
{
|
||||
'volume_kind': 'data',
|
||||
'device_type': 'block_ext4',
|
||||
'device_path': '/dev/vdb1'
|
||||
},
|
||||
{
|
||||
'volume_kind': 'binlog',
|
||||
'device_type': 'block_ext4',
|
||||
'device_path': '/dev/vdb2'
|
||||
},
|
||||
]
|
||||
|
||||
To support backwards compatibility when the guest is upgraded before
|
||||
the guestagent API, the guest agent's prepare method will take both
|
||||
old and new parameters. The correct behaviour will be deduced based
|
||||
on the values of the paramaters.
|
||||
|
||||
It is up to each datastore to define what volumes are supported, the
|
||||
meaning of each value of the volume_kind, and where each volume
|
||||
will be mounted on the local filesystem.
|
||||
|
||||
The guestagent resize_fs API will be updated to include the
|
||||
*volume_kind* parameter. The *mount_point* parameter will be ignored
|
||||
when the *volume_kind* parameter is specified.
|
||||
|
||||
As the parameter to the *post_upgrade* RPC endpoint is a dict, it will
|
||||
be updated to remain compatible with both old and new guests,
|
||||
providing both the mount point of the data volume plus the full
|
||||
*device_config* context.
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Not available.
|
||||
|
||||
|
||||
Dashboard Impact (UX)
|
||||
=====================
|
||||
|
||||
The dashboard would be changed to allow the user to specify the
|
||||
volume_config on the instance.create, cluster.create, and cluster.grow
|
||||
functions. A volume.configs.list method will be added to the Python
|
||||
API to facilite this functionality.
|
||||
|
||||
A new panel/action will be added to the instance panel to show the
|
||||
volumes configured for a particular instance.
|
||||
|
||||
The volume_resize functionality will be updated to allow the user to
|
||||
specify the volume_kind that they wish to resize.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
6-morgan
|
||||
|
||||
Dashboard assignee:
|
||||
<launchpad-id or None>
|
||||
|
||||
Milestones
|
||||
----------
|
||||
|
||||
Target Milestone for completion:
|
||||
post ocata
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
- implement taskmanager changes
|
||||
- implement base guestagent and guestagent.api changes
|
||||
- adapt datastores to new APIs
|
||||
- develop unit tests
|
||||
|
||||
|
||||
Upgrade Implications
|
||||
====================
|
||||
|
||||
Backward compatibility for this feature will be dependent on the new
|
||||
RPC versioning feature.
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
na
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
An int-test should be developed to test adding multiple volumes for at
|
||||
least one datastore.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
This feature will affect the following documents:
|
||||
|
||||
- Installation
|
||||
- API
|
||||
- CLI
|
||||
- Building Guest Images for Openstack Trove
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
na
|
||||
|
||||
Appendix
|
||||
========
|
||||
|
||||
Any additional technical information and data.
|
||||
Loading…
Reference in New Issue