commit f8f9ba1def9d9964113f5748e30fc4ac31a95381 Author: Funs Kessen Date: Mon Mar 21 18:38:06 2016 +0100 Initial check in of reworked version diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..af068bd --- /dev/null +++ b/.gitignore @@ -0,0 +1,11 @@ +.tox +.build +*.pyc +*.bak +repositories/centos/* +repositories/ubuntu/* +deployment_scripts/puppet/modules/inifile +deployment_scripts/puppet/modules/stdlib +build.sh +*.rpm +.project diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..e06d208 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + diff --git a/README.md b/README.md new file mode 100644 index 0000000..c738f1b --- /dev/null +++ b/README.md @@ -0,0 +1,44 @@ +fuel-plugin-datera-cinder +============ + +Plugin description +-------------- + +Datera plugin for Fuel extends Mirantis OpenStack functionality by +adding support for Datera EBS. + +The Datera cluster is an iSCSI block storage device used as a +Cinder backend. + +Requirements +------------ + +| Requirement | Version/Comment | +|------------------------------------------------------|-----------------| +| Mirantis OpenStack compatibility | >= 7.1 | +| Access to Datera via ccinder-volume node | | +| iSCSI initiator on all compute/cinder-volume nodes | | + +Limitations +----------- + +Datera configuration +--------------------- + +Before deployment the following needs to be verified: +1. Your Datera Cluster is reachable by all compute nodes, as well as the + Cinder Control/Manager node. +2. Create an Openstack account on the Datera cluster that can create + volumes. + (san_login/password). +3. Use the Management VIP address for the Datera cluster. + (as the san_ip)` + +Datera Cinder plugin installation +--------------------------- + +All of the code required for using Datera in an OpenStack deployment is +included in the upstream OpenStack distribution. + +Datera plugin configuration +---------------------------- diff --git a/deployment_scripts/puppet/deploy.sh b/deployment_scripts/puppet/deploy.sh new file mode 100644 index 0000000..0605dd5 --- /dev/null +++ b/deployment_scripts/puppet/deploy.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo datera > /tmp/datera diff --git a/deployment_scripts/puppet/manifests/cinder_datera_config.pp b/deployment_scripts/puppet/manifests/cinder_datera_config.pp new file mode 100644 index 0000000..f50bec3 --- /dev/null +++ b/deployment_scripts/puppet/manifests/cinder_datera_config.pp @@ -0,0 +1 @@ +include cinder_datera_config::cinder diff --git a/deployment_scripts/puppet/manifests/cinder_datera_driver.pp b/deployment_scripts/puppet/manifests/cinder_datera_driver.pp new file mode 100644 index 0000000..1d67d6e --- /dev/null +++ b/deployment_scripts/puppet/manifests/cinder_datera_driver.pp @@ -0,0 +1 @@ +include cinder_datera_driver::cinder diff --git a/deployment_scripts/puppet/modules/cinder_datera_config/manifests/backend/datera.pp b/deployment_scripts/puppet/modules/cinder_datera_config/manifests/backend/datera.pp new file mode 100644 index 0000000..df4e8e4 --- /dev/null +++ b/deployment_scripts/puppet/modules/cinder_datera_config/manifests/backend/datera.pp @@ -0,0 +1,56 @@ +# == Class: cinder_datera_config::backend::datera +# +# Configures Cinder volume Datera driver. +# Parameters are particular to each volume driver. +# +# === Parameters +# +# [*volume_backend_name*] +# (optional) Allows for the volume_backend_name to be separate of $name. +# Defaults to: $name +# +# [*volume_driver*] +# (optional) Setup cinder-volume to use Datera volume driver. +# Defaults to 'cinder.volume.drivers.datera.DateraDriver' +# +# [*san_ip*] +# (required) IP address of Datera clusters MVIP. +# +# [*san_login*] +# (required) Username for Datera tenant admin account. +# +# [*san_password*] +# (required) Password for Datera tenant admin account. +# +# [*datera_num_replicas*] +# (optional) The number of replicas to keep. +# Defaults to 2 +# +# [*extra_options*] +# (optional) Hash of extra options to pass to the cinder.conf +# Defaults to: {} +# Example : +# { 'datera_backend/param1' => { 'value' => value1 } } +# +define cinder_datera_config::backend::datera( + $san_ip, + $san_login, + $san_password, + $datera_num_replicas, + $volume_backend_name = $name, + $volume_driver = 'cinder.volume.drivers.datera.DateraDriver', + $extra_options = {}, +) { + + cinder_config { + "${name}/volume_backend_name": value => $volume_backend_name; + "${name}/volume_driver": value => $volume_driver; + "${name}/san_ip": value => $san_ip; + "${name}/san_login": value => $san_login; + "${name}/san_password": value => $san_password, secret => true; + "${name}/datera_num_replicas": value => $datera_num_replicas; + } + + create_resources('cinder_config', $extra_options) + +} diff --git a/deployment_scripts/puppet/modules/cinder_datera_config/manifests/cinder.pp b/deployment_scripts/puppet/modules/cinder_datera_config/manifests/cinder.pp new file mode 100755 index 0000000..277eb91 --- /dev/null +++ b/deployment_scripts/puppet/modules/cinder_datera_config/manifests/cinder.pp @@ -0,0 +1,62 @@ +# Copyright 2016 Datera, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +# + +class cinder_datera_config::cinder ( + $backend_name = 'datera', + $backends = '' +) { + include cinder::params + include cinder::client + + $plugin_settings = hiera('fuel-plugin-datera-cinder') + + if $::cinder::params::volume_package { + package { $::cinder::params::volume_package: + ensure => present, + } + Package[$::cinder::params::volume_package] -> Cinder_config<||> + } + + if $plugin_settings['multibackend'] { + $section = $backend_name + cinder_config { + "DEFAULT/enabled_backends": value => "${backend_name},${backends}"; + } + } else { + $section = 'DEFAULT' + } + + cinder_datera_config::backend::datera{ $section : + san_ip => $plugin_settings['datera_mvip'], + san_login => $plugin_settings['datera_admin_login'], + san_password => $plugin_settings['datera_admin_password'], + datera_num_replicas => $plugin_settings['datera_num_replicas'], + extra_options => {} + } + + Cinder_config<||>~> Service[cinder_volume] + + service { 'cinder_volume': + ensure => running, + name => $::cinder::params::volume_service, + enable => true, + hasstatus => true, + hasrestart => true, + } + package { 'open-iscsi' : + ensure => 'installed', + } + +} diff --git a/deployment_scripts/puppet/modules/cinder_datera_config/manifests/volume/datera.pp b/deployment_scripts/puppet/modules/cinder_datera_config/manifests/volume/datera.pp new file mode 100644 index 0000000..1c602a2 --- /dev/null +++ b/deployment_scripts/puppet/modules/cinder_datera_config/manifests/volume/datera.pp @@ -0,0 +1,48 @@ +# == Class: cinder_datera_config::volume::datera +# +# Configures Cinder volume Datera driver. +# Parameters are particular to each volume driver. +# +# === Parameters +# +# [*volume_driver*] +# (optional) Setup cinder-volume to use Datera volume driver. +# Defaults to 'cinder.volume.drivers.datera.DateraDriver' +# +# [*san_ip*] +# (required) IP address of Datera clusters MVIP. +# +# [*san_login*] +# (required) Username for Datera admin account. +# +# [*san_password*] +# (required) Password for Datera admin account. +# +# [*datera_num_replicas*] +# (optional) Number of replicas to keep. +# Defaults to 2 +# +# [*extra_options*] +# (optional) Hash of extra options to pass to the cinder.conf +# Defaults to: {} +# Example : +# { 'datera_backend/param1' => { 'value' => value1 } } +# +class cinder_datera_config::volume::datera( + $san_ip, + $san_login, + $san_password, + $volume_driver = 'cinder.volume.drivers.datera.DateraDriver', + $datera_num_replicas= '2', + $extra_options = {}, +) { + + cinder::backend::datera { 'DEFAULT': + san_ip => $san_ip, + san_login => $san_login, + san_password => $san_password, + volume_driver => $volume_driver, + datera_num_replicas => $datera_num_replicas, + extra_options => $extra_options, + } +} diff --git a/deployment_scripts/puppet/modules/cinder_datera_config/spec/classes/cinder_volume_datera_spec.rb b/deployment_scripts/puppet/modules/cinder_datera_config/spec/classes/cinder_volume_datera_spec.rb new file mode 100644 index 0000000..596ae55 --- /dev/null +++ b/deployment_scripts/puppet/modules/cinder_datera_config/spec/classes/cinder_volume_datera_spec.rb @@ -0,0 +1,44 @@ +require 'spec_helper' + +describe 'cinder::volume::datera' do + let :req_params do + { + :san_ip => '192.168.1.81', + :san_login => 'openstack_tenant0', + :san_password => 'password', + :datera_num_replicas => '2', + } + end + + let :params do + req_params + end + + describe 'datera volume driver' do + it 'configure datera volume driver' do + is_expected.to contain_cinder_config('DEFAULT/volume_driver').with_value('cinder.volume.drivers.datera.DateraDriver') + is_expected.to contain_cinder_config('DEFAULT/san_ip').with_value('192.168.1.81') + is_expected.to contain_cinder_config('DEFAULT/san_login').with_value('openstack_tenant0') + is_expected.to contain_cinder_config('DEFAULT/san_password').with_value('password') + is_expected.to contain_cinder_config('DEFAULT/datera_num_replicas').with_value('2') + end + + it 'marks san_password as secret' do + is_expected.to contain_cinder_config('DEFAULT/san_password').with_secret( true ) + end + + end + + describe 'datera volume driver with additional configuration' do + before :each do + params.merge!({:extra_options => {'datera_backend/param1' => {'value' => 'value1'}}}) + end + + it 'configure datera volume with additional configuration' do + should contain_cinder__backend__datera('DEFAULT').with({ + :extra_options => {'datera_backend/param1' => {'value' => 'value1'}} + }) + end + end + +end diff --git a/deployment_scripts/puppet/modules/cinder_datera_config/spec/defines/cinder_backend_datera_spec.rb b/deployment_scripts/puppet/modules/cinder_datera_config/spec/defines/cinder_backend_datera_spec.rb new file mode 100644 index 0000000..c76ad49 --- /dev/null +++ b/deployment_scripts/puppet/modules/cinder_datera_config/spec/defines/cinder_backend_datera_spec.rb @@ -0,0 +1,46 @@ +require 'spec_helper' + +describe 'cinder::backend::datera' do + let (:title) { 'datera' } + + let :req_params do + { + :san_ip => '192.168.1.81', + :san_login => 'openstack_tenant0', + :san_password => 'password', + :datera_num_replicas => '2', + } + end + + let :params do + req_params + end + + describe 'datera volume driver' do + it 'configure datera volume driver' do + is_expected.to contain_cinder_config('datera/volume_driver').with_value( + 'cinder.volume.drivers.datera.DateraDriver') + is_expected.to contain_cinder_config('datera/san_ip').with_value( + '192.168.1.81') + is_expected.to contain_cinder_config('datera/san_login').with_value( + 'openstack_tenant0') + is_expected.to contain_cinder_config('datera/san_password').with_value( + 'password') + is_expected.to contain_cinder_config('datera/datera_num_replicas').with_value( + '2') + end + end + + describe 'datera backend with additional configuration' do + before :each do + params.merge!({:extra_options => {'datera/param1' => {'value' => 'value1'}}}) + end + + it 'configure datera backend with additional configuration' do + should contain_cinder_config('datera/param1').with({ + :value => 'value1', + }) + end + end + +end diff --git a/deployment_scripts/puppet/modules/cinder_datera_driver/files/7.0/datera.py b/deployment_scripts/puppet/modules/cinder_datera_driver/files/7.0/datera.py new file mode 100644 index 0000000..8c271ed --- /dev/null +++ b/deployment_scripts/puppet/modules/cinder_datera_driver/files/7.0/datera.py @@ -0,0 +1,470 @@ +# Copyright 2015 Datera +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import json +from functools import wraps + +from oslo_config import cfg +from oslo_log import log as logging +from oslo_utils import excutils +from oslo_utils import units +from cinder import utils +import requests + +from cinder import exception +from cinder.i18n import _, _LI, _LE, _LW +from cinder.openstack.common import versionutils +from cinder.volume.drivers.san import san + +LOG = logging.getLogger(__name__) + +d_opts = [ + cfg.StrOpt('datera_api_token', + default=None, + help='DEPRECATED: This will be removed in the Liberty release. ' + 'Use san_login and san_password instead. This directly ' + 'sets the Datera API token.'), + cfg.StrOpt('datera_api_port', + default='7717', + help='Datera API port.'), + cfg.StrOpt('datera_api_version', + default='2', + help='Datera API version.'), + cfg.StrOpt('datera_num_replicas', + default='3', + help='Number of replicas to create of an inode.') +] + + +CONF = cfg.CONF +CONF.import_opt('driver_client_cert_key', 'cinder.volume.driver') +CONF.import_opt('driver_client_cert', 'cinder.volume.driver') +CONF.import_opt('driver_use_ssl', 'cinder.volume.driver') +CONF.register_opts(d_opts) + +DEFAULT_STORAGE_NAME = 'storage-1' +DEFAULT_VOLUME_NAME = 'volume-1' + + +def _authenticated(func): + """Ensure the driver is authenticated to make a request. + + In do_setup() we fetch an auth token and store it. If that expires when + we do API request, we'll fetch a new one. + """ + @wraps(func) + def func_wrapper(self, *args, **kwargs): + try: + return func(self, *args, **kwargs) + except exception.NotAuthorized: + # Prevent recursion loop. After the self arg is the + # resource_type arg from _issue_api_request(). If attempt to + # login failed, we should just give up. + if args[0] == 'login': + raise + + # Token might've expired, get a new one, try again. + self._login() + return func(self, *args, **kwargs) + return func_wrapper + + +class DateraDriver(san.SanISCSIDriver): + + """The OpenStack Datera Driver + + Version history: + 1.0 - Initial driver + 1.1 - Look for lun-0 instead of lun-1. + 2.0 - Update For Datera API v2 + """ + VERSION = '2.0' + + def __init__(self, *args, **kwargs): + super(DateraDriver, self).__init__(*args, **kwargs) + self.configuration.append_config_values(d_opts) + self.num_replicas = self.configuration.datera_num_replicas + self.username = self.configuration.san_login + self.password = self.configuration.san_password + self.auth_token = None + self.cluster_stats = {} + + def _get_lunid(self): + return 0 + + def do_setup(self, context): + # If any of the deprecated options are set, we'll warn the operator to + # use the new authentication method. + DEPRECATED_OPTS = [ + self.configuration.driver_client_cert_key, + self.configuration.driver_client_cert, + self.configuration.datera_api_token + ] + + if any(DEPRECATED_OPTS): + msg = _LW("Client cert verification and datera_api_token are " + "deprecated in the Datera driver, and will be removed " + "in the Liberty release. Please set the san_login and " + "san_password in your cinder.conf instead.") + versionutils.report_deprecated_feature(LOG, msg) + return + + # If we can't authenticate through the old and new method, just fail + # now. + if not all([self.username, self.password]): + msg = _("san_login and/or san_password is not set for Datera " + "driver in the cinder.conf. Set this information and " + "start the cinder-volume service again.") + LOG.error(msg) + raise exception.InvalidInput(msg) + + self._login() + + @utils.retry(exception.VolumeDriverException, retries=3) + def _wait_for_resource(self, id, resource_type): + result = self._issue_api_request(resource_type, 'get', id) + + if result['storage_instances'][DEFAULT_STORAGE_NAME]['volumes'][ + DEFAULT_VOLUME_NAME]['op_state'] == 'available': + return + else: + raise exception.VolumeDriverException(msg=_('Resource not ready.')) + + def _create_resource(self, resource, resource_type, body): + result = self._issue_api_request(resource_type, 'post', body=body) + + if result['storage_instances'][DEFAULT_STORAGE_NAME]['volumes'][ + DEFAULT_VOLUME_NAME]['op_state'] == 'available': + return + self._wait_for_resource(resource['id'], resource_type) + + def create_volume(self, volume): + """Create a logical volume.""" + # Generate App Instance, Storage Instance and Volume + # Volume ID will be used as the App Instance Name + # Storage Instance and Volumes will have standard names + + app_params = \ + { + 'create_mode': "openstack", + 'uuid': str(volume['id']), + 'name': str(volume['id']), + 'access_control_mode': 'allow_all', + 'storage_instances': { + DEFAULT_STORAGE_NAME: { + 'name': DEFAULT_STORAGE_NAME, + 'volumes': { + DEFAULT_VOLUME_NAME: { + 'name': DEFAULT_VOLUME_NAME, + 'size': volume['size'], + 'replica_count': int(self.num_replicas), + 'snapshot_policies': { + } + } + } + } + } + } + self._create_resource(volume, 'app_instances', body=app_params) + + def extend_volume(self, volume, new_size): + # Offline App Instance, if necessary + reonline = False + app_inst = self._issue_api_request( + "app_instances/{}".format(volume['id'])) + if app_inst['admin_state'] == 'online': + reonline = True + self.detach_volume(None, volume) + # Change Volume Size + app_inst = volume['id'] + storage_inst = DEFAULT_STORAGE_NAME + data = { + 'size': new_size + } + self._issue_api_request( + 'app_instances/{}/storage_instances/{}/volumes/{}'.format( + app_inst, storage_inst, DEFAULT_VOLUME_NAME), + method='put', body=data) + # Online Volume, if it was online before + if reonline: + self.create_export(None, volume) + + def create_cloned_volume(self, volume, src_vref): + clone_src_template = "/app_instances/{}/storage_instances/{" + \ + "}/volumes/{}" + src = clone_src_template.format(src_vref['id'], DEFAULT_STORAGE_NAME, + DEFAULT_VOLUME_NAME) + data = { + 'create_mode': 'openstack', + 'name': str(volume['id']), + 'uuid': str(volume['id']), + 'clone_src': src, + 'access_control_mode': 'allow_all' + } + self._issue_api_request('app_instances', 'post', body=data) + + def delete_volume(self, volume): + self.detach_volume(None, volume) + app_inst = volume['id'] + try: + self._issue_api_request('app_instances/{}'.format(app_inst), + method='delete') + except exception.NotFound: + msg = _("Tried to delete volume %s, but it was not found in the " + "Datera cluster. Continuing with delete.") + LOG.info(msg, volume['id']) + + def ensure_export(self, context, volume): + """Gets the associated account, retrieves CHAP info and updates.""" + return self.create_export(context, volume) + + def create_export(self, context, volume): + url = "app_instances/{}".format(volume['id']) + data = { + 'admin_state': 'online' + } + app_inst = self._issue_api_request(url, method='put', body=data) + storage_instance = app_inst['storage_instances'][ + DEFAULT_STORAGE_NAME] + + # Portal, IQN, LUNID + portal = storage_instance['access']['ips'][0] + ':3260' + iqn = storage_instance['access']['iqn'] + + provider_location = '%s %s %s' % (portal, iqn, self._get_lunid()) + return {'provider_location': provider_location} + + def detach_volume(self, context, volume, attachment=None): + url = "app_instances/{}".format(volume['id']) + data = { + 'admin_state': 'offline', + 'force': True + } + try: + self._issue_api_request(url, method='put', body=data) + except exception.NotFound: + msg = _("Tried to detach volume %s, but it was not found in the " + "Datera cluster. Continuing with detach.") + LOG.info(msg, volume['id']) + + def create_snapshot(self, snapshot): + url_template = 'app_instances/{}/storage_instances/{}/volumes/{' \ + '}/snapshots' + url = url_template.format(snapshot['volume_id'], + DEFAULT_STORAGE_NAME, + DEFAULT_VOLUME_NAME) + + snap_params = { + 'uuid': snapshot['id'], + } + self._issue_api_request(url, method='post', body=snap_params) + + def delete_snapshot(self, snapshot): + snap_temp = 'app_instances/{}/storage_instances/{}/volumes/{' \ + '}/snapshots' + snapu = snap_temp.format(snapshot['volume_id'], + DEFAULT_STORAGE_NAME, + DEFAULT_VOLUME_NAME) + + snapshots = self._issue_api_request(snapu, method='get') + + try: + for ts, snap in snapshots.viewitems(): + if snap['uuid'] == snapshot['id']: + url_template = snapu + '/{}' + url = url_template.format(ts) + self._issue_api_request(url, method='delete') + break + else: + raise exception.NotFound + except exception.NotFound: + msg = _LI("Tried to delete snapshot %s, but was not found in " + "Datera cluster. Continuing with delete.") + LOG.info(msg, snapshot['id']) + + def create_volume_from_snapshot(self, volume, snapshot): + snap_temp = 'app_instances/{}/storage_instances/{}/volumes/{' \ + '}/snapshots' + snapu = snap_temp.format(snapshot['volume_id'], + DEFAULT_STORAGE_NAME, + DEFAULT_VOLUME_NAME) + + snapshots = self._issue_api_request(snapu, method='get') + for ts, snap in snapshots.viewitems(): + if snap['uuid'] == snapshot['id']: + found_ts = ts + break + else: + raise exception.NotFound + + src = '/app_instances/{}/storage_instances/{}/volumes/{' \ + '}/snapshots/{}'.format( + snapshot['volume_id'], + DEFAULT_STORAGE_NAME, + DEFAULT_VOLUME_NAME, + found_ts) + app_params = \ + { + 'create_mode': 'openstack', + 'uuid': str(volume['id']), + 'name': str(volume['id']), + 'clone_src': src, + 'access_control_mode': 'allow_all' + } + self._issue_api_request('app_instances', method='post', body=app_params) + + def get_volume_stats(self, refresh=False): + """Get volume stats. + + If 'refresh' is True, run update first. + The name is a bit misleading as + the majority of the data here is cluster + data. + """ + if refresh or not self.cluster_stats: + try: + self._update_cluster_stats() + except exception.DateraAPIException: + LOG.error('Failed to get updated stats from Datera cluster.') + + return self.cluster_stats + + def _update_cluster_stats(self): + LOG.debug("Updating cluster stats info.") + + results = self._issue_api_request('system') + + if 'uuid' not in results: + LOG.error(_LE('Failed to get updated stats from Datera Cluster.')) + + backend_name = self.configuration.safe_get('volume_backend_name') + stats = { + 'volume_backend_name': backend_name or 'Datera', + 'vendor_name': 'Datera', + 'driver_version': self.VERSION, + 'storage_protocol': 'iSCSI', + 'total_capacity_gb': int(results['total_capacity']) / units.Gi, + 'free_capacity_gb': int(results['available_capacity']) / units.Gi, + 'reserved_percentage': 0, + } + + self.cluster_stats = stats + + def _login(self): + """Use the san_login and san_password to set self.auth_token.""" + body = { + 'name': self.username, + 'password': self.password + } + + # Unset token now, otherwise potential expired token will be sent + # along to be used for authorization when trying to login. + self.auth_token = None + + try: + LOG.debug('Getting Datera auth token.') + results = self._issue_api_request('login', 'put', body=body, + sensitive=True) + self.auth_token = results['key'] + self.configuration.datera_api_token = results['key'] + except exception.NotAuthorized: + with excutils.save_and_reraise_exception(): + LOG.error(_LE('Logging into the Datera cluster failed. Please ' + 'check your username and password set in the ' + 'cinder.conf and start the cinder-volume' + 'service again.')) + + @_authenticated + def _issue_api_request(self, resource_type, method='get', resource=None, + body=None, action=None, sensitive=False): + """All API requests to Datera cluster go through this method. + + :param resource_type: the type of the resource + :param method: the request verb + :param resource: the identifier of the resource + :param body: a dict with options for the action_type + :param action: the action to perform + :returns: a dict of the response from the Datera cluster + """ + host = self.configuration.san_ip + port = self.configuration.datera_api_port + api_token = self.configuration.datera_api_token + api_version = self.configuration.datera_api_version + + payload = json.dumps(body, ensure_ascii=False) + payload.encode('utf-8') + + if not sensitive: + LOG.debug("Payload for Datera API call: %s", payload) + + header = {'Content-Type': 'application/json; charset=utf-8'} + + protocol = 'http' + if self.configuration.driver_use_ssl: + protocol = 'https' + + # TODO(thingee): Auth method through Auth-Token is deprecated. Remove + # this and client cert verification stuff in the Liberty release. + if api_token: + header['Auth-Token'] = api_token + + client_cert = self.configuration.driver_client_cert + client_cert_key = self.configuration.driver_client_cert_key + cert_data = None + + if client_cert: + protocol = 'https' + cert_data = (client_cert, client_cert_key) + + connection_string = '%s://%s:%s/v%s/%s' % (protocol, host, port, + api_version, resource_type) + + if resource is not None: + connection_string += '/%s' % resource + if action is not None: + connection_string += '/%s' % action + + LOG.debug("Endpoint for Datera API call: %s", connection_string) + try: + response = getattr(requests, method)(connection_string, + data=payload, headers=header, + verify=False, cert=cert_data) + except requests.exceptions.RequestException as ex: + msg = _('Failed to make a request to Datera cluster endpoint due ' + 'to the following reason: %s') % ex.message + LOG.error(msg) + raise exception.DateraAPIException(msg) + + data = response.json() + if not sensitive: + LOG.debug("Results of Datera API call: %s", data) + + if not response.ok: + LOG.debug(_(response.url)) + LOG.debug(_(payload)) + LOG.debug(_(vars(response))) + if response.status_code == 404: + raise exception.NotFound(data['message']) + elif response.status_code in [403, 401]: + raise exception.NotAuthorized() + else: + msg = _('Request to Datera cluster returned bad status:' + ' %(status)s | %(reason)s') % { + 'status': response.status_code, + 'reason': response.reason} + LOG.error(msg) + raise exception.DateraAPIException(msg) + + return data diff --git a/deployment_scripts/puppet/modules/cinder_datera_driver/manifests/cinder.pp b/deployment_scripts/puppet/modules/cinder_datera_driver/manifests/cinder.pp new file mode 100644 index 0000000..e254d2c --- /dev/null +++ b/deployment_scripts/puppet/modules/cinder_datera_driver/manifests/cinder.pp @@ -0,0 +1,26 @@ +notice('PLUGIN: cinder_datera_driver::cinder: cinder.pp') + +class cinder_datera_driver::cinder { + $version = hiera('fuel_version') + + # install the driver, only required on cinder nodes + notice("PLUGIN: cinder_datera_driver::cinder: trying to install Fuel $version plugin.") + if($version == '7.0') { + file { "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/datera.py": + mode => "0644", + owner => 'root', + group => 'root', + source => 'puppet:///modules/cinder_datera_driver/7.0/datera.py', + } + } elsif ($version == '8.0') { + file { "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/datera.py": + mode => "0644", + owner => 'root', + group => 'root', + source => 'puppet:///modules/cinder_datera_driver/8.0/datera.py', + } + } else { + notice("PLUGIN: cinder_datera_driver::cinder: $version is not supported by us.") + } +} +class { 'cinder_datera_driver::cinder': } diff --git a/deployment_tasks.yaml b/deployment_tasks.yaml new file mode 100644 index 0000000..f4bf540 --- /dev/null +++ b/deployment_tasks.yaml @@ -0,0 +1,19 @@ +- id: cinder-datera-driver + type: puppet + role: [cinder] + required_for: [post_deployment_end] + requires: [post_deployment_start] + parameters: + puppet_manifest: puppet/manifests/cinder_datera_driver.pp + puppet_modules: "puppet/modules/:/etc/puppet/modules/" + timeout: 360 + +- id: cinder-datera-config + type: puppet + role: [cinder] + required_for: [post_deployment_end] + requires: [post_deployment_start, cinder-datera-driver] + parameters: + puppet_manifest: puppet/manifests/cinder_datera_config.pp + puppet_modules: "puppet/modules/:/etc/puppet/modules/" + timeout: 360 diff --git a/environment_config.yaml b/environment_config.yaml new file mode 100644 index 0000000..008f318 --- /dev/null +++ b/environment_config.yaml @@ -0,0 +1,34 @@ +attributes: + multibackend: + value: false + label: 'Multibackend enabled' + description: 'Datera driver will be used with Cinder Multibackend feature' + weight: 15 + type: "checkbox" + datera_mvip: + value: '' + label: 'Cluster Management VIP (san_ip)' + description: 'The hostname (or IP address) for the Datera management API endpoint.' + weight: 20 + type: "text" + datera_admin_login: + value: '' + label: 'Login for Admin account (san_login)' + description: 'account used by Cinder service.' + weight: 30 + type: "text" + regex: + source: '\S' + error: "Username field cannot be empty" + datera_admin_password: + value: '' + label: 'Password for Admin account (san_password)' + description: 'account used by Cinder service.' + weight: 40 + type: "password" + datera_num_replicas: + value: '2' + label: 'Data replication factor' + description: 'Repliacte data X times over the cluster' + weight: 50 + type: "text" diff --git a/metadata.yaml b/metadata.yaml new file mode 100644 index 0000000..63b6196 --- /dev/null +++ b/metadata.yaml @@ -0,0 +1,24 @@ +name: fuel-plugin-datera-cinder +title: Fuel Datera driver for Cinder +version: '0.1.43' +description: Installs and enables the Datera driver in Cinder +fuel_version: ['7.0'] +licenses: ['Apache License Version 2.0'] +authors: [ 'Funs Kessen ' ] +homepage: 'https://github.com/stackforge/fuel-plugin-datera-cinder' +groups: ['storage::cinder'] + +releases: + - os: ubuntu + version: 2015.1-7.0 + mode: ['ha', 'multinode'] + deployment_scripts_path: deployment_scripts/ + repository_path: repositories/ubuntu + - os: centos + version: 2015.1.0-7.0 + mode: ['ha', 'multinode'] + deployment_scripts_path: deployment_scripts/ + repository_path: repositories/centos + +# Version of plugin package +package_version: '3.0.0' diff --git a/pre_build_hook b/pre_build_hook new file mode 100755 index 0000000..dc05e98 --- /dev/null +++ b/pre_build_hook @@ -0,0 +1,5 @@ +#!/bin/bash + +# Add here any the actions which are required before plugin build +# like packages building, packages downloading from mirrors and so on. +# The script should return 0 if there were no errors. diff --git a/specs/datera-plugin-specs.rst b/specs/datera-plugin-specs.rst new file mode 100644 index 0000000..be6398b --- /dev/null +++ b/specs/datera-plugin-specs.rst @@ -0,0 +1,131 @@ + + This work is licensed under the Apache License, Version 2.0. + + http://www.apache.org/licenses/LICENSE-2.0 + +================================================== +Fuel plugin for Datera as a Cinder backend +================================================== + +The Datera plugin for Fuel extends Mirantis OpenStack functionality by adding +support for Datera EBS in Cinder using the iSCSI protocol. + +Problem description +=================== + +Currently, Fuel has no support for Datera EBS as block storage for +OpenStack environments. The Datera plugin aims to provide support for it. + +Proposed change +=============== + +Implement a Fuel plugin that will configure the Datera driver for +Cinder on all Controller nodes. + +Alternatives +------------ + +None + +Data model impact +----------------- + +None + +REST API impact +--------------- + +None + +Upgrade impact +-------------- + +None + +Security impact +--------------- + +None + +Notifications impact +-------------------- + +None + +Other end user impact +--------------------- + +None + +Performance Impact +------------------ + +The Datera EBS provides high performance block storage for OpenStack +environments, and therefore enabling the Datera driver in OpenStack +will greatly improve the peformance of OpenStack. + +Other deployer impact +--------------------- + +The deployer should make sure the IP of the management VIP is correct, prior +to deploying the Fuel plugin to the controllers. If this is not done the VIP +needs to altered and the cinder service will have to be restarted. + +Developer impact +---------------- + +None + +Implementation +============== + +The plugin generates the approriate cinder.conf stanzas to enable the Datera +within OpenStack. There are NO other packages required, the Datera driver +which is included in the OpenStack distribution is all that is necessary. + +Plugin has two tasks. Each task per role. They are run in the following order: + +* The first task installs and configures cinder-volume on Primary Controller. +* The second task installs and configures cinder-volume on Controller nodes. + +Cinder-volume service is installed on all Controller nodes and is managed by +Pacemaker. It runs in active/passive mode where only one instance is active. +All instances of cinder-volume have the same “host” parameter in cinder.conf +file. This is required to achieve ability to manage all volumes in the +environment by any cinder-volume instance. + +Assignee(s) +----------- + +| Funs + +Work Items +---------- + +* Implement the Fuel plugin. +* Implement the Puppet manifests. +* Testing. +* Write the documentation. + +Dependencies +============ + +* Fuel 7.0 and higher. + +Testing +======= + +* Prepare a test plan. +* Test the plugin by deploying environments with all Fuel deployment modes. + +Documentation Impact +==================== + +* Deployment Guide (how to install the storage backends, how to prepare an + environment for installation, how to install the plugin, how to deploy an + OpenStack environment with the plugin). +* User Guide (which features the plugin provides, how to use them in the + deployed OpenStack environment). +* Test Plan. +* Test Report. + diff --git a/volumes.yaml b/volumes.yaml new file mode 100644 index 0000000..5c91d84 --- /dev/null +++ b/volumes.yaml @@ -0,0 +1,8 @@ +volumes_roles_mapping: + # Default role mapping + cinder_datera: + - {allocate_size: "min", id: "os"} + +# Set here new volumes for your role +volumes: [] +