Expose BlueStore

Explain how BlueStore vs traditional filesystems
are selected.

Improve config.yaml correspondingly. Move the
'bluestore' option's location in that file. Remove
blank lines for consistency.

Change-Id: Iebc21bdcac742a437719afb53f26729abbf8e87f

README.md

@@ -18,6 +18,17 @@ default values. A YAML file (e.g. `ceph-osd.yaml`) is often used to store
 configuration options. See the [Juju documentation][juju-docs-config-apps] for
 details on configuring applications.
 
+#### `bluestore`
+
+The `bluestore` option specifies whether the
+[BlueStore][upstream-ceph-bluestore] storage backend is used for all OSD
+devices. The feature is enabled by default (value 'True'). If set to 'True',
+this option overrides the `osd-format` option as BlueStore does not use a
+traditional filesystem.
+
+> **Important**: This option has no effect unless Ceph Luminous (or greater) is
+  in use.
+
 #### `customize-failure-domain`
 
 The `customize-failure-domain` option determines how a Ceph CRUSH map is
@@ -38,6 +49,12 @@ The `osd-devices` option lists what block devices can be used for OSDs across
 the cluster. See section 'Storage devices' for an elaboration on this
 fundamental topic.
 
+#### `osd-format`
+
+The `osd-format` option specifies what filesystem to use for all OSD devices
+('xfs' or 'ext4'). The default value is 'xfs'. This option only applies when
+Ceph Luminous (or greater) is in use and option `bluestore` is set to 'False'.
+
 #### `source`
 
 The `source` option states the software sources. A common value is an OpenStack
@@ -418,4 +435,5 @@ For general charm questions refer to the OpenStack [Charm Guide][cg].
 [lp-bugs-charm-ceph-osd]: https://bugs.launchpad.net/charm-ceph-osd/+filebug
 [cdg-install-openstack]: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-openstack.html
 [upstream-ceph-buckets]: https://docs.ceph.com/docs/master/rados/operations/crush-map/#types-and-buckets
+[upstream-ceph-bluestore]: https://docs.ceph.com/en/latest/rados/configuration/storage-devices/#bluestore
 [cloud-archive-ceph]: https://wiki.ubuntu.com/OpenStack/CloudArchive#Ceph_and_the_UCA

config.yaml

@@ -83,6 +83,15 @@ options:
       where the specified journal device does not exist on a node.
       .
       Only supported with ceph >= 0.48.3.
+  bluestore:
+    type: boolean
+    default: True
+    description: |
+      Enable BlueStore storage backend for OSD devices.
+      .
+      Only supported with ceph >= 12.2.0.
+      .
+      Setting to 'False' will use FileStore as the storage format.
   bluestore-wal:
     type: string
     default:
@@ -128,7 +137,6 @@ options:
       .
       A default value is not set as it is calculated by ceph-disk (before Luminous)
       or the charm itself, when ceph-volume is used (Luminous and above).
-
   bluestore-block-db-size:
     type: int
     default: 0
@@ -141,24 +149,22 @@ options:
       .
       A default value is not set as it is calculated by ceph-disk (before Luminous)
       or the charm itself, when ceph-volume is used (Luminous and above).
-
   osd-format:
     type: string
     default: xfs
     description: |
-      Format of filesystem to use for OSD devices; supported formats include:
+      Format of filesystem to use for OSD devices. Supported formats include:
       .
-        xfs (Default >= 0.48.3)
-        ext4 (Only option < 0.48.3)
+        xfs (Default with >= ceph 0.48.3)
+        ext4 (Only option < ceph 0.48.3)
         btrfs (experimental and not recommended)
       .
-      Only supported with ceph >= 0.48.3.
-  bluestore:
-    type: boolean
-    default: True
-    description: |
-      Enable bluestore storage format for OSD devices; Only applies for Ceph
-      Luminous or later.
+      Only supported with >= ceph 0.48.3.
+      .
+      Used with FileStore storage backend.
+      .
+      Always applies prior to ceph 12.2.0. Otherwise, only applies when the
+      "bluestore" option is False.
   osd-encrypt:
     type: boolean
     default: False