Conditionally Import Parakimo
Separate SSH functions into ssh_utils.py for safe conditional import.
Change-Id: Ia1a3ee69bef76b52e4e6df1e73488c018ac0f3c9
"self" needs to be the first parameter; or pylint
complains.
Change-Id: I23b5c2c3f53c66e120fc852f50ad13e9c889c98c
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
We are replacing all usages of the 'retrying' package with
'tenacity' as the author of retrying is not actively maintaining
the project. Tenacity is a fork of retrying, but has improved the
interface and extensibility (see [1] for more details). Our end
goal here is removing the retrying package from our requirements.
Tenacity provides the same functionality as retrying, but has the
following major differences to account for:
- Tenacity uses seconds rather than ms as retrying did
(the retry interface in manila exposed time in seconds as well)
- Tenacity has different kwargs for the decorator and
Retrying class itself.
- Tenacity has a different approach for retrying args by
using classes for its stop/wait/retry kwargs.
- By default tenacity raises a RetryError if a retried callable
times out; retrying raises the last exception from the callable.
Tenacity provides backwards compatibility here by offering
the 'reraise' kwarg - we are going to set this in the retry interface
by default.
- For retries that check a result, tenacity will raise if the
retried function raises, whereas retrying retried on all
exceptions - we haven't exposed this in the retry interface.
This patch updates all usages of retrying with tenacity.
Unit tests are added where applicable.
[1] https://github.com/jd/tenacity
Co-Authored-By: boden <bodenvmw@gmail.com>
Co-Authored-By: Goutham Pacha Ravi <gouthampravi@gmail.com>
Closes-Bug: #1635393
Change-Id: Ia0c3fa5cd82356a33becbf57444f3db5ffbb0dd0
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
collections.deque does not have a 'pop'
method, and the sshpool.remove method
currently leaks SSH connections that
it creates.
This bugfix was ported from cinder [1]
[1] https://review.openstack.org/#/c/285687/
Change-Id: I2dc9ffc13f11884b3069e6b4a453c933edf16d89
Co-Authored-By: Surya Ghatty <ghatty@us.ibm.com>
Closes-Bug: #1463557
Generic driver jobs are failing because of timeouts when
establishing the initial ssh connection from manila-share
to the service VM.
Bump up the default value of the connection timeout for paramiko
client and also set the banner timeout since the failure occurred
during banner exchange. Set the two timeouts to the same value
for now. This ensures that the connection timeout is at least as
long as the banner timeout and there is no current need in manila
to control these independently.
This is more of a workaround than a real fix since a real fix
would remove the delay during banner exchange. I suspect that
the real fix will need to be in neutron/ovs though.
Change-Id: Ib5e59faaf9667b9cb5e7d4072531b7d6c3d4da39
Partial-bug: #1807216
add manila.utils.is_all_tenants to check all_tenants value.
share_networks and security_service api will check all_tenants value
in the following patches.
Change-Id: I1aa9903276038dc839df45b852b329a75be5657f
Partial-Bug: #1777551
Co-Authored-By: Jiao Pengju <jiaopengju@cmss.chinamobile.com>
Previously, the Data Service allowed only 1 IP to be specified
for the 'data_node_access_ip' config option, which prevented
Host-assisted Share Migration to be successful on a combination
of backends that include IPv6 and IPv4 export locations.
To fix this, we are adding a new config option and deprecating
the old one. The new one is named 'data_node_access_ips' and
accepts a list of IPs. All IPs included in this list should
correspond to interfaces on the node running the Data Service
and will be allowed access during Host-assisted Share Migration.
Additionally, we are improving the responsiveness of the
Host-assisted Share Migration by using exponential waiters
instead of polynomial. Those waiters are used when managing
resources created by Share Migration.
Closes-bug: #1745436
Partial-bug: #1708491
Change-Id: I1c0b7621ae8192f75f691987b185a4fc6a7228ce
Major changes:
* Support to create/delete/extend/access NFS and CIFS share/snapshot
in the IPv6 network which created by Neutron
* Support to connect VNX management interface using IPv6 address
Change-Id: Ibe7620f9548d5f57780e49c08214dc627b91a945
Implements: blueprint vnx-manila-ipv6-support
An issue was discovered in the ZFSonLinux driver wherein on a system
set to have a locale where the decimal separator is not a period but
rather a comma the method translate_string_size_to_float() would fail
because it's regex to find sizes in strings did not account for any
other decimal separators other than periods.
This fix updates the regular expression to accept either a period or
a comma as a decimal separator in a size string.
Many thanks to Dr. Clemens Hardewig for reporting the original bug
and providing an updated regular expression.
Change-Id: I15da4aaff90814eed03816db013d6acc89ba2ee8
Closes-Bug: #1714691
Basically, if you decide to override a method, that is because you want
to specify(or modify) the behavior of a method. If you just call the
inherited method from the overridden method, that is unnecessary.
Furthermore, every time when self.setUp() is called, now the super's
setUp() is called, which is basically exact the same behavior as
we have now.
TrivialFix
Change-Id: I9f4e03b93ff075c66b4e52342711c37f226d2f81
As part of the manila-telemetry integration work
we are now triggering notifications for several
events. We should provide a way to disable those.
In this patch-set we define a decorator that can be
used to change the behavior of those classes used
to emit notification.
Partially-Implements: bp ceilometer-integration
Change-Id: I806e0133e9fe3ad5cc35ad1e000e5ca61ff3b8ca
This can be used if an endless loop is needed.
Also add a new parameter to allow a maximum backoff sleep time.
Partial-Bug: #1690159
Change-Id: Ib544b5bd4781d116dd3dffc8f35f43323cc9e2db
One of the goals for Pike [1] is to make each API service be able to
run under web servers that support WSGI applications,
such as Apache (+mod-wsgi) and Nginx (+uWSGI).
Do following to address governance requirements:
- Split existing manila/wsgi.py module into 3 modules:
First (manila/wsgi/eventlet_server.py) is used by
eventlet-based WSGI application approach.
Second (manila/wsgi/wsgi.py) is used for WSGI web servers.
And third (manila/wsgi/common.py) is common code for both.
All three are made in cinder-like way to have alike-approach.
- Reuse common code from "oslo_service/wsgi.py" module that
allows us to remove code duplication.
- Delete config opts that are defined by newly reused common code.
- Register new entry point that will be manila wsgi app: "manila-wsgi".
- Fix "manila/api/openstack/wsgi.py" module to be compatible
with str/bytes handling approach used by Apache mod-wsgi plugin using
different python versions (2/3).
- Add web server config template "devstack/apache-manila.template"
- Add devstack support where usage of this feature can be
enabled or disabled using "MANILA_USE_MOD_WSGI" env var.
It is set to "True" by default, because it is requirement for Pike
release - to have it running in all CI jobs.
Disable it only for one CI job that uses dummy driver and tests
various manila core features that are not covered by other CI jobs.
[1] https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html
Partially-Implements BluePrint wsgi-web-servers-support
DocImpact
Change-Id: Ibdef3c6810b65a5d6f3611e2d0079c635ee523ab
A "bugfix" was commited which allows users to configure backends
to use SSL in an insecure way. This change removes the bad bugfix.
This reverts commit 8b6868aaae.
Change-Id: I342b3639cd42aa4ac62a85d3dad29cab0a421e31
From python 2.7.9 on, the ssl verification is enabled by default,
This commit adds 2 options for Manila, so user is able to control the
ssl verification.
The 2 options are ported from cinder/driver.py, these can be used by any
vendor driver which needs to handle ssl verification.
Closes-bug: 1669202
Implements: blueprint add-ssl-verification-options
Change-Id: Ia9a488cab9d4d2d25c5ab534dbf4d61e930cfd7f
- Pull up policy check to beginning of the APIs.
- Avoid making access rules changes when one or
more instances of the share are in an invalid state.
- Add back the per rule share instance access status.
This restoration provides better visibility for which
rules were applied successfully.
- Remove 'updating' and 'updating_multiple' as valid
states for the share instance access rules status.
- Deprecate the access rule state 'new' in favor of
'queued_to_apply' and the share instance access rules
status 'out_of_sync' in favor of 'syncing'.
In a new API micro-version:
- Allow access rule changes irrespective of the share's
access_rules_status.
- Expose new access rule states and share's
access_rules_status values.
Access rules for each share instance now transition
from 'queued_to_apply' to 'applying' to 'active' or 'error';
and from 'active', 'queued_to_apply', 'applying' or 'error'
to 'queued_to_deny' to 'denying' to 'deleted'.
APIImpact
DocImpact
Partially-implements: bp fix-and-improve-access-rules
Co-Authored-By: Mike Rooney <rooneym@netapp.com>
Change-Id: Ic25e63215b5ba723cbc8cab7c51789c698e76f28
Openstack common has a wrapper for generating uuids.
We should only use that function when generating uuids for consistency.
Change-Id: Ie68850e529eee51c66e8d29a9025e1390576bc1a
Closes-bug: #1082248
We parse the boolean value from string in dict with identical
logic in different areas, it's reasonable to add an util function
to cover this.
Change-Id: Id54029e2e4a0dd2f2093e6ef2583a9cada84c540
oslo.utils provides similar function[1] to check if ipv6 is supported.
Though Manila and oslo.utils implemented in different way, we can
improve the version of oslo.utils to make it suitable to other projects.
[1]https://github.com/openstack/oslo.utils/blob\
/d1e08f533d4351f10b8103e24c254004b6627a29/oslo_utils/netutils.py#L170
TrivialFix
Change-Id: I4ff99189943f4ca56f0532e58b1aedc63516074d
In module netaddr valid_ipv4/valid_ipv6 raise exception
AddrFormatError if address is specified as empty string.
Methods is_valid_ipv4/is_valid_ipv6 catch the exception
and return boolean value. So we'd better use them from
oslo.utils. This commit also adds empty string case in
unit test.
Change-Id: If58889bb322686c252b4fe89a4ddf9469121028c
test_wait_until_true_not_ok waits for an exception to be raised.
This timeout was configured to 1 second which caused the test to
take a whole second to run. Lowered the value to 0.01 seconds to
speed it up.
TrivialFix
Change-Id: Ie7a4aca3ac58329ee466e832cad8b7d95012c3cd
Removed functionality of Share Migration relying on Manila Share
Service node, moved code to Data Service node for copy phase.
Added parameter 'notify' and share/api methods for future
implementation (see dependent patches).
Added new copy operation statuses, in order to implement future
API calls to obtain progress and cancel migration.
Added possibility of 2-phase migration for driver migration and
generic (fallback) migration.
Added admin export location support and removed approach of
replacing IP with config parameter.
Added Admin-only API entry points to:
- Migration Cancel (only during copying)
- Reset Task State field
- Migration Get Progress (only during copying)
- Migration Complete (2nd phase migration)
- Notify parameter on Migrate Share
APIImpact
DocImpact
Implements: blueprint data-service-migration
Change-Id: I1d65aac2f36942cd70eb214be561d59a15a4ba26
This driver exploits a container as a share server instead
of a virtual machine. The container acts as a NFS/CIFS server
and shares logical volume attached to it. The volume gets created
simultaneously to the container on a host in a volume group
specified by the user. Currently NFS is limited to v3 since the
only userspace NFS server that works in a container out of the box
seems to be unfs3. Later it will be replaced with nfs-ganesha.
Co-Authored-By: Alexey Ovchinnikov <aovchinnikov@mirantis.com>
Implements: blueprint lxc-driver
Depends-On: Ib1896997f2e7a505b5bf8ec0c9e5ee35942f79a6
Change-Id: Iea73bd34b94919d77ebe17cf054dde1f580384be
Add support for ZFS (on Linux) filesystem.
Following features are supported:
- Create/delete share
- Create/delete snapshot
- Create share from snapshot
- Extend/shrink share
- Update NFS IP-based access rules using new interface.
(SMB support planned for future).
- create/delete/update/promote share replica
ZFS-related notes:
- Any amount of ZFS zpools can be used by share driver.
- Allowed to configure default options for ZFS datasets that are used
for share creation.
- Any amount of nested datasets is allowed to be used.
- All share replicas are read-only.
- All share replicas are synched periodically, not continuously.
So, status 'in_sync' means latest sync was successful.
Time range between syncs equals to value of
config global opt 'replica_state_update_interval'.
Driver-related notes:
- Able to use remote ZFSonLinux storage as well as local.
Other made changes:
- updated driver private data DB methods removing filtering by host
as redundant operation. Replication requires some common metadata
storage and filtering by host breaks it. It is safe to do so, because
if driver gets some ID of entity then it is allowed to read its info
too.
Implements bp zfsonlinux-driver
DocImpact
Change-Id: I3ddd3767184e4843037de0ac75ff18dce709b6dc
Fix params order to correspond to real signature:
assertEqual(expected, actual)
Change-Id: I5887e9c4fbd8953b3be9e89ce86758f8d1d842b2
Closes-Bug: #1277104
Recent change [1] made share manager ignoring all driver setup errors
and continue run process. So, if driver init failed and we do not have
errors in "stats reporting" then manager/driver will report that
it is OK and scheduler will consider it alive when it is not indeed.
So, add attr 'initialized' to driver base class that will be used by
share manager to get to know is it allowed to call driver methods or
not.
Manila-share service will still run, but will be marked as 'down'.
[1] I5c4c551da9d3576ea49118ef09655d2939990cb2
Change-Id: I15d1f2aaf1d7cec6e8afb93fc32e46a877261d4e
Closes-Bug: #1502809
Multiple requests sent to HNAS can cause concurrency problems
and that ends up with 'SSC failed connection' errors. This
patch adds a retry decorator to _execute in HDS HNAS Manila
driver to fix this problem.
The current retry functionality in Manila always uses fixed
numbers to define the time to wait before performing the next
attempts. This behavior can make the retries of multiple
requests to collide in each attempt as they will wait the
same amount of time and try to use the same resource together
again. So additionally, this patch changes the behavior of
manila.utils.retry() to receive a parameter that allows the
function to implement randomly generated wait intervals.
Change-Id: Ib862f62517fcc8816781204b902119e9b20121e0
Closes-bug: 1491550
This patch adds new Manila HDS HNAS Driver,
according to blueprint.
Change-Id: I0f9ae2a940df5415b93f6a6c5c2b0fac1cb062fd
Implements: blueprint hds-hnas
The OpenStack API working group recommends all projects adopt
Nova-style microversions for versioning their REST APIs. This
commit ports the Nova microversion code and docs to Manila.
With this patch, the API version is bumped to 1.1, and the
versions API (which has always returned horribly outdated
values) is the first API that is versioned. The 1.1 version
of the versions API includes the minimum and current API
version values.
Implements bp: manila-rest-api-microversions
Change-Id: Ifa8e394335a4eb3ad21f53a873530aee241c00e8
Retrying library from OpenStack Requirement is already used in Manila
project. Add 'retry' wrapper around it which enables logging and
checking of input param 'retries' which may be useful.
Related-Bug: #1469068
Change-Id: I159b6e44fec9455634d76c093036f934cf400366
Remove the broken XML support presently available in Manila
in order to comply with the general direction among other
OpenStack projects to support JSON only.
Change-Id: Ibb542fa223f7f7d1bf95f3d1f568987ef839cd12
Closes-Bug: #1440782
This network plugin can be used with any network platform.
Features:
- Serves flat networks as well as segmented.
- Does not require some specific network services in OpenStack like Neutron or
Nova.
- Can handle IPv6 as well as IPv4
- The only thing that plugin does is reservation and release of IP addresses
from some network.
Can be configured using following options:
- standalone_network_plugin_gateway [required]
- standalone_network_plugin_mask [required]
- standalone_network_plugin_segmentation_id [optional] [default: None]
- standalone_network_plugin_allowed_ip_ranges [optional] [default: None]
- standalone_network_plugin_ip_version [optional] [default: 4]
Implements blueprint standalone-network-plugin
Change-Id: Ic9947dead1af2114ae0834b644ab19c7020232c3
The Manila test module uses a class called StubOutForTesting, which
is an awkward holdover from when the unit tests were using Mox.
Alex wrote a more elegant and capable mocker method,
manila.test.TestCase.mock_object, in the Cinder project which we
have now added to Manila. So to remove the duplicate functionality
and legacy code, all references to self.stubs.Set() in the Manila
tests should be replaced with self.mock_object() and any related
dead code should be removed.
Resolves-bug: #1415602
Change-Id: Ie5f203c0b2cb57ad00d9ec8db0529075b6fef261
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
oslo.messaging is the only exception because this package doesn't
currently support non-namespaced imports.
Change-Id: I3987e651bc880c8ffa7c0105df0298679dcd3a43
A SSHPool class object, used to hold ssh connections, fails to
authenticate to the server in its create () method if the path to
SSH private key or user password is not passed during object creation.
The create () method does not allow private key to be discovered in
the default ~/.ssh folder when trying to authenticate or connect
to the SSH server. Instead allow auto discovery of the key in the
paramiko SSHClient's connect method (called in create () method) if the
path to private key or password is not provided.
Also make the following minor cleanups in the
create() method of SSHPool class:
- pass the path to the key file input directly to the SSHClient's
connect () method using the appropriate parameter, keyfile, instead
of picking key from key file and passing that as 'pkey' parameter.
- restrict the try except block to only the steps involved in
connecting to the SSH server.
- reraise the exception that would be raised by the Paramiko library
instead of raising own exception.
initialization method of SSHPool class:
- rename the attribute privatekey as path_to_private_key to accurately
reflect what it refers to, the path of the private key file.
Change-Id: I590702d97086d33245894fd686250e75e8e359f2
Closes-Bug: #1412782
'mock' objects don't have method "assert_called_once", but this is called in
following unit test suites:
manila/tests/network/linux/test_interface.py
manila/tests/network/neutron/test_neutron_api.py
manila/tests/test_share_generic.py
manila/tests/test_service_instance.py
manila/tests/test_share_glusterfs.py
manila/tests/test_utils.py
Make unit tests call proper assertion methods.
Change-Id: I96923bd345fc5129963c77dfce3555e7284eb836
Closes-Bug: #1330391
Change usage of modules that are deprecated in oslo-incubator.
Use the corresponding module from oslo.utils.
A followup patch will sync with oslo-incubator and remove the now
obsolete modules.
Change-Id: I4f949de57e333832dcc7c1e256ce82e2db0144cb
Partial-Bug: #1382189
Zuul