The is_valid_ipvN methods from oslo.utils raises an exception for
non-string values, because of the change in netaddr 1.0.0. This adds
a wrapper logic to retain the previous behavior, which is preferred by
manila.
Change-Id: If1297e4d54fd645969ca79690a1c558c9efdbc8d
Conditionally Import Parakimo
Separate SSH functions into ssh_utils.py for safe conditional import.
Change-Id: Ia1a3ee69bef76b52e4e6df1e73488c018ac0f3c9
usernames and passwords must never be logged
by the service, even in debug mode.
Change-Id: I8eda1c849d0d1916345959178f32756e8a1e9c0e
Closes-Bug: #1976370
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
Currently we ensures the root_helper key when the run_as_root key is
included, but the logic does not check whether run_as_root has a True
value.
So even if the function is called with run_as_root=False, the logic
ensures root_helper, and this is just redundant.
Because root_helper has no effect unless root_as_root is defined and
has a True value, we can always pass root_helper.
Change-Id: Ic78d11d36dc2d0b52176a5ac2c49cb06d59476db
We currently do not use the write_data_to_file method
anymore, so there's not need for keeping it.
Partially-Implements: bp privsep-migration
Change-Id: Ibb626aea69083d77037730346bd517311e1638d4
paramiko has some calls to md5() which will break under fips.
We're going to have to replace paramiko eventually because it won't
be FIPS compliant, but until then we might be able to achieve
compatibility by patching paramiko.
Change-Id: Ia7d9b4b46b4ce9bede6fc03308c765a6521f5999
The manila/utils.py is still using six library and some of its methods, which are useless as Python 2 is no more used.
The import of six library and encodeutils from oslo_utils are removed as well as their usage inside the code.
Closes-Bug: #1948029
Change-Id: I855022b9bd038449c0eb87868d6cf39413341db0
We are replacing all usages of the 'retrying' package with
'tenacity' as the author of retrying is not actively maintaining
the project. Tenacity is a fork of retrying, but has improved the
interface and extensibility (see [1] for more details). Our end
goal here is removing the retrying package from our requirements.
Tenacity provides the same functionality as retrying, but has the
following major differences to account for:
- Tenacity uses seconds rather than ms as retrying did
(the retry interface in manila exposed time in seconds as well)
- Tenacity has different kwargs for the decorator and
Retrying class itself.
- Tenacity has a different approach for retrying args by
using classes for its stop/wait/retry kwargs.
- By default tenacity raises a RetryError if a retried callable
times out; retrying raises the last exception from the callable.
Tenacity provides backwards compatibility here by offering
the 'reraise' kwarg - we are going to set this in the retry interface
by default.
- For retries that check a result, tenacity will raise if the
retried function raises, whereas retrying retried on all
exceptions - we haven't exposed this in the retry interface.
This patch updates all usages of retrying with tenacity.
Unit tests are added where applicable.
[1] https://github.com/jd/tenacity
Co-Authored-By: boden <bodenvmw@gmail.com>
Co-Authored-By: Goutham Pacha Ravi <gouthampravi@gmail.com>
Closes-Bug: #1635393
Change-Id: Ia0c3fa5cd82356a33becbf57444f3db5ffbb0dd0
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
This bugfix [1] modified the totalcount returned by pagination
query when the argument 'limit' was specified. It caused
manila to do not return precise count of shares in a query that
satisfied the conditions.
This bug has been fixed and now manila is returning the precise
values of shares matched in a given query. Also, manila is now
performing filtering actions in the database to have more
performatic results.
[1] https://review.opendev.org/#/c/688542/
Closes-Bug: #1860061
Co-Authored-By: Carlos Eduardo <ces.eduardo98@gmail.com>
Change-Id: I6ddd919bbd5180593cc52bf986912f65a2dab3a7
Pylint does not play very well with dynamic object
manipulation in python This creates a lot of
false-positives in the code-base which affects
contributors looking for genuine failures.
So, this change
- adds pylint ignore statements where appropriate
to disable testing these lines of code and failing.
- replaces all the pylint error codes (they are
hard to remember/relate to) with error names
which are easier to understand when reading the
code.
- initializes sqlalchemy model objects as dictionaries
which is a valid representation over None.
- removes ignore directives on six.moves which
is globally ignored in our pylintrc.
- adds alembic.op to the ignored
modules list since they are not supported by
pylint and have known issues.
This patch is the beginning of a series of
commits to use pylint in a sane way on manila code.
Change-Id: I44616821c5311d6f14986697efbbe5624de364a5
collections.deque does not have a 'pop'
method, and the sshpool.remove method
currently leaks SSH connections that
it creates.
This bugfix was ported from cinder [1]
[1] https://review.openstack.org/#/c/285687/
Change-Id: I2dc9ffc13f11884b3069e6b4a453c933edf16d89
Co-Authored-By: Surya Ghatty <ghatty@us.ibm.com>
Closes-Bug: #1463557
Connections to backends via paramiko often fail for obscure
reasons.
If the service log level is DEBUG, increase paramiko logging
to DEBUG level as well so that we have more information when issues
of this sort occur.
Change-Id: I44d8902d58c2ad48b8a37dfbff96c5b9471651e4
The current implementation of ssh_utils never sends keepalive packets.
In ssh_utils.SSHPool, the socket timeout parameter is set to None
intending to keep ssh connections open.
However, when the parameter is set to None, ssh_utils does not run the
code to compare idle duration and keepalive interval.
This patch reverts the socket timeout parameter to default (0.1 sec).
The ssh_utils compares them every 0.1 seconds, and sends a keepalive
packet if idle duration > keepalive interval (= self.conn_timeout).
See cinder change: I8234083107207b9ebc0849947e8de92b5cf3e36e
Change-Id: Ib13e5c6246412d667554cb0bd9c419b513af70c7
Related-Bug: #1673662
Generic driver jobs are failing because of timeouts when
establishing the initial ssh connection from manila-share
to the service VM.
Bump up the default value of the connection timeout for paramiko
client and also set the banner timeout since the failure occurred
during banner exchange. Set the two timeouts to the same value
for now. This ensures that the connection timeout is at least as
long as the banner timeout and there is no current need in manila
to control these independently.
This is more of a workaround than a real fix since a real fix
would remove the delay during banner exchange. I suspect that
the real fix will need to be in neutron/ovs though.
Change-Id: Ib5e59faaf9667b9cb5e7d4072531b7d6c3d4da39
Partial-bug: #1807216
add manila.utils.is_all_tenants to check all_tenants value.
share_networks and security_service api will check all_tenants value
in the following patches.
Change-Id: I1aa9903276038dc839df45b852b329a75be5657f
Partial-Bug: #1777551
Co-Authored-By: Jiao Pengju <jiaopengju@cmss.chinamobile.com>
Previously, the Data Service allowed only 1 IP to be specified
for the 'data_node_access_ip' config option, which prevented
Host-assisted Share Migration to be successful on a combination
of backends that include IPv6 and IPv4 export locations.
To fix this, we are adding a new config option and deprecating
the old one. The new one is named 'data_node_access_ips' and
accepts a list of IPs. All IPs included in this list should
correspond to interfaces on the node running the Data Service
and will be allowed access during Host-assisted Share Migration.
Additionally, we are improving the responsiveness of the
Host-assisted Share Migration by using exponential waiters
instead of polynomial. Those waiters are used when managing
resources created by Share Migration.
Closes-bug: #1745436
Partial-bug: #1708491
Change-Id: I1c0b7621ae8192f75f691987b185a4fc6a7228ce
The iso8601 lib introduced a change such that if running on
python 3.2 or later it internally uses the python timezone
information instead of its own implementation.
This does not change direct date handling, but when converting
this value there is a slight difference where now python 2.x
will show UTC times as "UTC", but on python 3 they will end up
with "UTC+00:00".
The to_primitive call for DateTime fields was doing an exact match
on "UTC" to determine whether to include "Z" in the resulting string.
This updates that handling to recognize either of the new values.
Closes-bug: #1744160
Change-Id: I9fcc55b36178ff88795e1d8e14da349e338a7392
Major changes:
* Support to create/delete/extend/access NFS and CIFS share/snapshot
in the IPv6 network which created by Neutron
* Support to connect VNX management interface using IPv6 address
Change-Id: Ibe7620f9548d5f57780e49c08214dc627b91a945
Implements: blueprint vnx-manila-ipv6-support
An issue was discovered in the ZFSonLinux driver wherein on a system
set to have a locale where the decimal separator is not a period but
rather a comma the method translate_string_size_to_float() would fail
because it's regex to find sizes in strings did not account for any
other decimal separators other than periods.
This fix updates the regular expression to accept either a period or
a comma as a decimal separator in a size string.
Many thanks to Dr. Clemens Hardewig for reporting the original bug
and providing an updated regular expression.
Change-Id: I15da4aaff90814eed03816db013d6acc89ba2ee8
Closes-Bug: #1714691
Please read spec for design detail [1].
Support IPv6 in IP drivers, networks and share
type extra specs.
Co-Authored-By: TommyLikeHu(tommylikehu@gmail.com)
Co-Authored-By: Ben Swartzlander <ben@swartzlander.org>
[1] f7202a6cfe32a057f752a4e393f848f8a0211c36
DocImpact
Partial-Implements: blueprint support-ipv6-access
Change-Id: I96d3389262e9829b8b4344870cdf5c76abd22828
As part of the manila-telemetry integration work
we are now triggering notifications for several
events. We should provide a way to disable those.
In this patch-set we define a decorator that can be
used to change the behavior of those classes used
to emit notification.
Partially-Implements: bp ceilometer-integration
Change-Id: I806e0133e9fe3ad5cc35ad1e000e5ca61ff3b8ca
This can be used if an endless loop is needed.
Also add a new parameter to allow a maximum backoff sleep time.
Partial-Bug: #1690159
Change-Id: Ib544b5bd4781d116dd3dffc8f35f43323cc9e2db
Function 'oslo_utils.timeutils.isotime()' is deprecated in version '1.6'
and will be removed in a future version. We can't use
datetime.datetime.isoformat() instead. Because the format of the string
generated by isoformat isn't the same as the format of the string
generated by isotime. The string is used in tokens and other public
APIs and we can't change it without potentially breaking clients.
So the workaround is to copy the current implementation from
oslo_utils.timeutils.isotime() to utils.py
For more informations:
http://docs.openstack.org/developer/oslo.utils/api/timeutils.html#oslo_utils.timeutils.isotime
APIImpact
Closes-Bug: #1694352
Change-Id: I2feb85b7f698ae456493d02a118bb3fb969835da
One of the goals for Pike [1] is to make each API service be able to
run under web servers that support WSGI applications,
such as Apache (+mod-wsgi) and Nginx (+uWSGI).
Do following to address governance requirements:
- Split existing manila/wsgi.py module into 3 modules:
First (manila/wsgi/eventlet_server.py) is used by
eventlet-based WSGI application approach.
Second (manila/wsgi/wsgi.py) is used for WSGI web servers.
And third (manila/wsgi/common.py) is common code for both.
All three are made in cinder-like way to have alike-approach.
- Reuse common code from "oslo_service/wsgi.py" module that
allows us to remove code duplication.
- Delete config opts that are defined by newly reused common code.
- Register new entry point that will be manila wsgi app: "manila-wsgi".
- Fix "manila/api/openstack/wsgi.py" module to be compatible
with str/bytes handling approach used by Apache mod-wsgi plugin using
different python versions (2/3).
- Add web server config template "devstack/apache-manila.template"
- Add devstack support where usage of this feature can be
enabled or disabled using "MANILA_USE_MOD_WSGI" env var.
It is set to "True" by default, because it is requirement for Pike
release - to have it running in all CI jobs.
Disable it only for one CI job that uses dummy driver and tests
various manila core features that are not covered by other CI jobs.
[1] https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html
Partially-Implements BluePrint wsgi-web-servers-support
DocImpact
Change-Id: Ibdef3c6810b65a5d6f3611e2d0079c635ee523ab
A "bugfix" was commited which allows users to configure backends
to use SSL in an insecure way. This change removes the bad bugfix.
This reverts commit 8b6868aaae.
Change-Id: I342b3639cd42aa4ac62a85d3dad29cab0a421e31
From python 2.7.9 on, the ssl verification is enabled by default,
This commit adds 2 options for Manila, so user is able to control the
ssl verification.
The 2 options are ported from cinder/driver.py, these can be used by any
vendor driver which needs to handle ssl verification.
Closes-bug: 1669202
Implements: blueprint add-ssl-verification-options
Change-Id: Ia9a488cab9d4d2d25c5ab534dbf4d61e930cfd7f
Gate still runs pep8 with python2.7 but will eventually run with
python3, and current desktop linux distributions typically ship
a version of tox that uses python3 to run pep8.
Remove windows style line endings and use builtin functions that
are python3 compatible.
TrivialFix
Change-Id: I542475fe25358cc127c3e0de00fcc9f7dc084005
Implemented several improvements to share migration
according to spec [1].
Summary of changes:
- Snapshot restriction in API has been changed to return error only
when parameter force-host-assisted-migration is True
- Added preserve_snapshot to API and migration_check_compatibility
driver interface
- Changed all driver-assisted API parameters to be mandatory
- Added validation to prevent 'force_host_assisted_migration' to be
used alongside driver-assisted parameters
- Changed "same host" validation to reject only if the combination
of "host", "new_share_network" and "new_share_type" is the same as
the source
- Updated migration driver interfaces to support snapshots
- Updated zfsonlinux driver, defaulting preserve_snapshots to False
- Updated dummy driver to support preserve_snapshots
Spec update with latest changes since [1] merged
can be found in [2].
APIImpact
DocImpact
[1] I5717e902373d79ed0d55372afdedfaa98134c24e
[2] If02180ec3b5ae05c9ff18c9f5a054c33f13edcdf
Change-Id: I764b389816319ed0ac5178cadbf809cb632035b4
Partially-implements: blueprint ocata-migration-improvements
- Pull up policy check to beginning of the APIs.
- Avoid making access rules changes when one or
more instances of the share are in an invalid state.
- Add back the per rule share instance access status.
This restoration provides better visibility for which
rules were applied successfully.
- Remove 'updating' and 'updating_multiple' as valid
states for the share instance access rules status.
- Deprecate the access rule state 'new' in favor of
'queued_to_apply' and the share instance access rules
status 'out_of_sync' in favor of 'syncing'.
In a new API micro-version:
- Allow access rule changes irrespective of the share's
access_rules_status.
- Expose new access rule states and share's
access_rules_status values.
Access rules for each share instance now transition
from 'queued_to_apply' to 'applying' to 'active' or 'error';
and from 'active', 'queued_to_apply', 'applying' or 'error'
to 'queued_to_deny' to 'denying' to 'deleted'.
APIImpact
DocImpact
Partially-implements: bp fix-and-improve-access-rules
Co-Authored-By: Mike Rooney <rooneym@netapp.com>
Change-Id: Ic25e63215b5ba723cbc8cab7c51789c698e76f28
We parse the boolean value from string in dict with identical
logic in different areas, it's reasonable to add an util function
to cover this.
Change-Id: Id54029e2e4a0dd2f2093e6ef2583a9cada84c540
oslo.utils provides similar function[1] to check if ipv6 is supported.
Though Manila and oslo.utils implemented in different way, we can
improve the version of oslo.utils to make it suitable to other projects.
[1]https://github.com/openstack/oslo.utils/blob\
/d1e08f533d4351f10b8103e24c254004b6627a29/oslo_utils/netutils.py#L170
TrivialFix
Change-Id: I4ff99189943f4ca56f0532e58b1aedc63516074d
Manila doesn't use methods ensure_tree and delete_if_exists anywhere,
and oslo.utils provides same methods in module fileutils. We can use
them from oslo.utils if we need in the futrue.
TrivialFix
Change-Id: I280b674dca3738d26c2bed2cf71e94775b370cd5
In module netaddr valid_ipv4/valid_ipv6 raise exception
AddrFormatError if address is specified as empty string.
Methods is_valid_ipv4/is_valid_ipv6 catch the exception
and return boolean value. So we'd better use them from
oslo.utils. This commit also adds empty string case in
unit test.
Change-Id: If58889bb322686c252b4fe89a4ddf9469121028c
Removed functionality of Share Migration relying on Manila Share
Service node, moved code to Data Service node for copy phase.
Added parameter 'notify' and share/api methods for future
implementation (see dependent patches).
Added new copy operation statuses, in order to implement future
API calls to obtain progress and cancel migration.
Added possibility of 2-phase migration for driver migration and
generic (fallback) migration.
Added admin export location support and removed approach of
replacing IP with config parameter.
Added Admin-only API entry points to:
- Migration Cancel (only during copying)
- Reset Task State field
- Migration Get Progress (only during copying)
- Migration Complete (2nd phase migration)
- Notify parameter on Migrate Share
APIImpact
DocImpact
Implements: blueprint data-service-migration
Change-Id: I1d65aac2f36942cd70eb214be561d59a15a4ba26
Zuul