Added new column named 'backup_type' in 'share_backups' table
and changes the share common api libs to support the dhss_true
configuration for share backup creation
Partially-implements: bp/share-backup
Change-Id: Ifb88ec096674ea8bc010c1c3f6dea1b51be3beaa
Export locations are usually too difficult to memo
rize.Currently, there is no way to determine the
export location before the share is created, so
users wait until the share creation request gets
completed, and then they check the export
locations to mount the share. The generated
export locations are often not human readable
and it is hard to memorize and control them.
Implements: bp/human-readable-export-locations
Change-Id: I72ac7e24ddd4330d76cafd5e7f78bac2b0174883
- A new config option named ``admin_metadata_keys`` was introduced
and we expect it to be set in the DEFAULT section of the manila
configuration file. It is expected that administrators will provide
a list of metadata keys that can only be updated by administrators
through this configuration option.
- Drivers will be able to set metadata while creating shares
through the `get_optional_share_creation_data` driver interface.
Closes-Bug: #2050010
Change-Id: I6412710c7db89747d23033e1a5a6be9de5886b0b
The protocol access mapping options should accept dictionaries whose
values are lists, not strings.
Closes-Bug: 2038607
Change-Id: Id75082507cee27bef03318dc3af5ae5c1b233505
This is an internal change in the share manager's
"ensure_driver_resources" method, and isn't invoked
by any code path yet. The idea is to allow drivers
to opt into running "update_access" on shares that
they are ensuring.
Only drivers that implement "ensure_shares" can
take advantage of this feature.
Change-Id: Ieb7b0dedd98dc02b593078d08d4c0bdf4a1af2bf
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
Add share backup feature in Data Copy Service and
Share Service. It will allow the user to create, restore
and delete backups as well as listing backups and showing
the details of a specific backup.
APIImpact
DOCImpact
Change-Id: I7d10cf47864cd21932315375d84dc728ff738f23
Implement: blueprint share-backup
user can create a transfer for a share. will return transfer id
and auth_key. another user can use transfer_id and auth_key to
accept this share. salt of transfer and auth_key compute crypt_hash
by sha1. then compare with crypt_hash in db.
APIImpact
DocImpact
Partially-Implements: blueprint transfer-share-between-project
Change-Id: I8facf9112a6b09e6b7aed9956c0a87fb5f1fc31f
Currently netapp_snapmirror_quiesce_timeout is an option of replica
promote that can only be set by the operator. There are scenarios where
high timeout value set by operator does not fit well e.g. disaster
stroke and user want to do an unplanned failover fast. Added new
option 'quiesce_wait_time' to share replica promote API which allows
to use specified wait_time(in seconds) instead of config option.
Closes-bug: #2000171
Change-Id: Ib02063ee8b82f7374cd89f90e7f24a845c6c7cd7
'reserved_share_extend_percentage' backend config option allows Manila
to consider different reservation percentage for share extend
operation. With this option, under existing limit of
'reserved_share_percentage', we do not want user to create new share if
limit is hit, but allow user to extend existing share.
DocImpact
Closes-Bug: #1961087
Change-Id: I000a7f530569ff80495b1df62a91981dc5865023
Manila can now configure a share network with multiple subnets
in an availability zone. Also, it can add a new subnet for an
availability that has share servers, which will triger an update
share server allocations.
Changes:
- API:
- Bump version to 2.70.
- setup share network with multiple subents per az.
- Block manage server with multiple subnets.
- Allow add subnet for in-use share servers.
- `share_network_subnet_id` is dropped from ShareServer view
- `share_network_subnet_ids` is added in ShareServer view
- `network_allocation_update_support` is added to ShareServer and
ShareNetwork views.
- Add a check operation for share network subnet create.
- DB:
- Remove `share_network_subnet_id` from share_servers.
- Create mapping table `share_server_share_network_subnet_mappings`.
- Fix queries with new db design.
- Add migration downgrade and upgrade alembic.
- Add `share_network_subnet_id` to the NetworkAllocations.
- Scheduler:
- Change `AvailabilityZoneFilter` to take in account if the
host supports the allocation required by the setup request.
- Manager:
- Bump RPC API version.
- `_setup_server` allocating multiple subnets.
- Modify signature of driver `_setup_server` interface, passing a
list of `network_info` for each subnet.
- Share server DB creation to inform a list of subnets and
create with `network_allocation_update_support`.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations`.
- Drivers:
- For legacy compatibility, all drivers implementing `_setup_server`
consume the first element of the `network_ino`.
- Dummy Driver:
- Implement `_setup_server` with new signature as multiple subnet.
- Modify the `backend_details` to save allocations for all subnets.
- Report update allocation and share server multiple subnet support.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations` interfaces.
Signed-off-by: Felipe Rodrigues <felipefuty01@gmail.com>
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Co-Authored-By: Fábio Oliveira <fabioaurelio1269@gmail.com>
Co-Authored-By: Nahim Alves de Souza <nahimsouza@outlook.com>
Co-Authored-By: Caique Mello <caique_mellosbo@hotmail.com>
DocImpact
APIImpact
Partially-Implements: blueprint multiple-share-network-subnets
Change-Id: I7de9de4ae509182e9494bba604979cce03acceec
The NetApp driver has been working with FlexVol ONTAP volumes.
The driver does not support scaling FlexVol volumes higher than
100 TiB, which was a theoretical limit for the large namespace that
these containers were meant to handle. ONTAP's Flexgroup volumes
eliminate such limitations. So, added the support for provisioning
share as FlexGroup in the NetApp driver.
The FlexGroup provision is enabled by new option
`netapp_enable_flexgroup`, which will make the driver report a single
pool represeting all aggregates. The selection on which aggregates the
FlexGroup share will reside is up to ONTAP. If the administrator desires
to control that selection through Manila scheduler, it must inform the set
of aggregates that formss FlexGroup pool in the new option
`netapp_flexgroup_pool`.
Each NetApp pool will report now the capability: `netapp_flexgroup`
informing which type the pool is.
The following operations are allowed with FlexGroup shares (DHSS
True/False and NFS/CIFS):
- Create/Delete share;
- Shrink/Extend share;
- Create/Delete snapshot;
- Revert to snapshot;
- Manage/Unmanage snapshots;
- Create from snapshot;
- Replication[1]
- Manage/Unmanage shares;
The backend with one FlexGroup pool configured will drop the consistent
snapshot support for all pools.
The driver FlexGroup support requires ONTAP version 9.8 or greater.
[1] FlexGroup is limited to one single replica for ONTAP version
lower than 9.9.1.
DocImpact
Depends-On: If525e97a5d456d6ddebb4bf9bc8ff6190c95a555
Depends-On: I646f782c3e2be5ac799254f08a248a22cb9e0358
Implements: bp netapp-flexgroup-support
Change-Id: I4f68a9bb33be85f9a22e0be4ccf673647e713459
Signed-off-by: Felipe Rodrigues <felipefuty01@gmail.com>
A series of enhancements can be performed in the share server
migration operation, and share backends might support
nondisruptive share server migrations, which was not covered
in the previous approach.
- Skip the destination share server network allocation if the
driver is capable of reusing the source share server allocation.
Manila will switch the allocations in the migration complete phase.
- Allow share backends to reuse the share servers in case they
are able to do so in a share server migration scenario.
- Nondisruptive migration is now feasible depending on whether the
share driver supports it and if the share network parameters have
not changed.
- Share servers will be always deleted whe the share server
migration complete operation is finished.
Depends-On: I43bd3fdafea02eb8e853114a57bfb863a441f3ed
Co-Authored-By: Fabio Oliveira <fabioaurelio1269@gmail.com>
Change-Id: I48bafd92fe7a4d4ae0bafd5bf1961dace56b6005
This config option allows different value for reservation percentage,
mostly useful on the platforms, where shares can only be created from
the snapshot on the host where snapshot was taken. The lower value of
this config option against existing (reserved_share_percentage) allows
to create shares from the snapshot on the same host up to a higher
threshold even though non-snapshot/regular share create fails.
In case this config option is not set, the shares created from snapshot
will use reservation percentage value set in 'reserved_share_percentage'.
This will be useful for users who want to keep same reservation percentage
for both non-snapshot/regular and snapshot shares.
DocImpact
Closes-Bug: #1938060
Change-Id: I390da933fe92875e3c7ee40709eacacc030278dc
To create a share with protocol CIFS it is mandatory to add a security
service to the share network beforehand. If this is forgotten the share
ends up in error. Validate the security service association based on
share protocol from given driver. Raise invalid request exception if
needed association between share_network and security_service is
missing for given driver e.g. ONTAP.
DocImpact
Closes-Bug: #1900752
Change-Id: Ib7e9850e6439ee5d04f826d129afb1ab06950ce7
Remove manila configuration options
and auth classes that were deprecated
before the Ussuri release.
Change-Id: I148225926cd249a0dd8d1f8c02b22ed06487f405
This patch implements the update of security service's association
with in-use share networks. The following changes were added:
- New share network APIs: `share_network_security_service_update`
and `share_network_reset_state`.
- A new `status` attribute was added to share network model to
identify when it's in a modification state, called 'network_change'.
Other supported status that were added: 'active' and 'error'.
- New 'security_service_update_support' property was added to both
share server and share network models, to identify when this resources
are able to process security service update for in-use share networks.
- New driver interface was added to support update of security service's
configuration of a given share server.
DocImpact
APIImpact
Partially Implements: bp add-security-service-in-use-share-networks
Co-Authored-By: Carlos Eduardo <ces.eduardo98@gmail.com>
Co-Authored-By: Douglas Viroel <viroel@gmail.com>
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Change-Id: I129a794dfd2d179fa2b9a2fed050459d6f00b0de
Add two new capabilities to manila. It is possible to set limits
to share server size and share instances in a share server by
setting `max_share_server_size` and `max_shares_per_share_server`
in a backend stanza.
Change-Id: I3170478d3aa2d09cb2adc32233dc57bc59029a56
Partially-Implements: bp new-share-server-limits
DocImpact
This patch adds support for migration of share servers. This
migration is performed using a two-phase approach. Administrators
are now able to request the migration of a share server within and
across backends, with the possibility of chooosing a different share
network for the destination share server.
- A new field called `task_state` was added to the share server
model in order to help the administrator to track the share server
migration steps. A new field called `source_share_server_id` was
added to link destination and source share servers.
- A new periodic task was added to track migration of share servers
and its resources.
- Two new states were added: `server_migrating` and
`server_migrating_to` to represent that share migration is in
progress.
- When performing the server migration, manila will not go to the
scheduler, instead it will provide a request spec to drivers
during migration check driver call. It'll be up to the driver
validate if there is free space to handle the share server.
- A new API called `share-server-migration-check' was added to
check the feasibility of a migration, before actually triggering
the start operation.
APIImpact
DocImpact
Partially Implements: bp share-server-migration
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Co-Authored-By: Carlos Eduardo <ces.eduardo98@gmail.com>
Co-Authored-By: Felipe Rodrigues <felipefuty01@gmail.com>
Change-Id: Ic0751027d2c3f1ef7ab0f7836baff3070a230cfd
Signed-off-by: Douglas Viroel <viroel@gmail.com>
When a share driver does not provide the capability
to create share group snapshots, the share manager
iterates over the shares in the group and invokes the
driver method to snapshot the shares. However, the
create and delete routines in this fallback path
did not include some attributes necessary for the
operation in many drivers.
Change-Id: Id5a9467247df1d8f0ec6dee3fae842ba673c34ed
Closes-Bug: #1888905
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
This patch enables the creation of a share from snapshot
specifying another pool or backend. In the scheduler, a
new filter and weigher were implemented in order to consider
this operation if the backend supports it. Also, a new
field called 'progress' was added in the share and share
instance. The 'progress' field indicates the status
of the operation create share from snapshot (in percentage).
Finally, a new periodic task was added in order to constantly
check the share status.
Partially-implements: bp create-share-from-snapshot-in-another-pool-or-backend
DOCImpact
Change-Id: Iab13a0961eb4a387a502246e5d4b79bc9046e04b
Co-authored-by: carloss <ces.eduardo98@gmail.com>
Co-authored-by: dviroel <viroel@gmail.com>
This patch adds the possibility to create share networks with
multiple subnets in Manila. It also updates the share server api
to receive "share_network_subnet_id" instead of "share_network_id".
Each share network subnet must be associated with only one
availability zone. Each share network must have a single default
share network subnet.
DocImpact
APIImpact
Depends-On: I13bb48e7c03e16c26946ccf9d48e80592391a3d1
Partially-implements: bp share-network-multiple-subnets
Change-Id: Id8814a8b26c9b9dcb1fe71d0d7e9b79e8b8a9210
Closes-Bug: #1588144
Co-Authored-By: lseki <luciomitsuru.seki@fit-tecnologia.org.br>
Co-Authored-By: dviroel <viroel@gmail.com>
This patch adds Manage/Unmanage of share servers in
Manila. It also updates the Manage Share API to accept
a "share_server_id" parameter, and updates Unmanage
of Share and Snapshots API to allow unmanaging of
shares and snapshots in DHSS=True.
Managed share servers are not deleted automatically
by manila, and if a single share is unmanaged in
DHSS=True, the respective share server will not be
deleted automatically as well.
Managing share servers require that the driver
implements 2 functions:
- get_share_server_network_info: obtain IPs from
share server.
- manage_server: perform required operations to
manage and return dict of backend_details.
Unmanaging share servers require that the driver
overrides unmanage_server function.
The IPs obtained from the backend are validated
by the Network plugin, so ports with the exact
IPs must exist in the subnet and admin subnet
associated with the share network specified
when managing the share server (unless
StandaloneNetworkPlugin is used).
It is recommended to rename the backend resource
if possible when managing the share server, to
prevent issues with re-managing a share server
that has already been managed.
This patch bumps the API microversion to 2.49.
APIImpact
DocImpact
Depends-On: I17c74b2aa242918188eeff368232c762a4b31093
Partially-implements: bp manage-unmanage-with-share-servers
Change-Id: I108961e7436ba13550ef2b8f02079c6e599a6166
Pylint does not play very well with dynamic object
manipulation in python This creates a lot of
false-positives in the code-base which affects
contributors looking for genuine failures.
So, this change
- adds pylint ignore statements where appropriate
to disable testing these lines of code and failing.
- replaces all the pylint error codes (they are
hard to remember/relate to) with error names
which are easier to understand when reading the
code.
- initializes sqlalchemy model objects as dictionaries
which is a valid representation over None.
- removes ignore directives on six.moves which
is globally ignored in our pylintrc.
- adds alembic.op to the ignored
modules list since they are not supported by
pylint and have known issues.
This patch is the beginning of a series of
commits to use pylint in a sane way on manila code.
Change-Id: I44616821c5311d6f14986697efbbe5624de364a5
"storage_availability_zone" in the [DEFAULT] section of
manila's configuration file has allowed deployers to configure
and manage both service (scheduler, share manager) and storage
system availability. However, quite often manila's services
(api, scheduler, share and data managers) are run on a dedicated
control plane that is a different failure domain from that of
the storage that manila manages.
Also, when using share replication, deployers would need to
run multiple manila share manager services with different
configuration files, each with their own
"storage_availability_zone".
To allow flexibility of separating service and storage
availability zones, we introduce a new configuration option
"backend_availability_zone" within the share driver/backend
section. When this option is used, it will override the value
of the "storage_availability_zone" from the [DEFAULT] section.
Change-Id: Ice99a880dd7be7af94dea86b31a6db88be3d7d9b
Implements: bp per-backend-availability-zones
Fixed bugs that caused ensure_shares to not work properly. Mainly:
- Comparison between db object and hash string was causing
ensure_shares to run every time. Fixed by reading property
'info_hash' of db object.
- Driver implementation of ensure_shares would raise an
exception in case of returning None. Fixed by adding
'or {}' to method invocation.
- Missing share_servers parameter for each share passed
to ensure_shares. Fixed by converting share_instance
entities to dictionaries before passing to drivers.
Closes-bug: #1771866
Closes-bug: #1772644
Closes-bug: #1772647
Change-Id: Iac969e9cac6cea46deb12f5d8423be592cfeb72a
Previously, the Data Service allowed only 1 IP to be specified
for the 'data_node_access_ip' config option, which prevented
Host-assisted Share Migration to be successful on a combination
of backends that include IPv6 and IPv4 export locations.
To fix this, we are adding a new config option and deprecating
the old one. The new one is named 'data_node_access_ips' and
accepts a list of IPs. All IPs included in this list should
correspond to interfaces on the node running the Data Service
and will be allowed access during Host-assisted Share Migration.
Additionally, we are improving the responsiveness of the
Host-assisted Share Migration by using exponential waiters
instead of polynomial. Those waiters are used when managing
resources created by Share Migration.
Closes-bug: #1745436
Partial-bug: #1708491
Change-Id: I1c0b7621ae8192f75f691987b185a4fc6a7228ce
This patch makes necessary changes to devstack manila
plugin in order to be able to run IPv6 scenario tests.
Part of the changes included the dependency of
neutron-dynamic-routing plugin and the installation
of quagga, so routes in devstack host can be created
automatically for each router and private network
created by tempest during the tests.
Also, added a new config option "override_ip_for_nfs_access"
for manila tempest plugin that overrides the access rules used
for NFS scenario tests. This option can be set by exporting the
environment variable "OVERRIDE_IP_FOR_NFS_ACCESS" with the
intended value before running devstack.
This change is enabled by a following change and
tested alongside a change on manila-tempest-plugin
project.
Please note that we are temporarily disabling IPv6
functionality in Host-assisted Share Migration, as
the Data Service is not able to handle IPv4 + IPv6
scenarios.
Change-Id: I4ca727f92618998242af18908bbbda6bb5f86303
Needed-By: Id8b005cdd429d53a75624885fe7ca795746c3ede
This driver interface is currently being used wrong and needs a rewrite.
This patch adds the ability to solve the potential problem of slow start
up, and deal with non-user-initiated state changes to shares.
Implement BP ensure-share
Change-Id: I704591446cd5f6ad2293c5a18eccf5412d488baf
Allow the ganesha driver to store ganesha exports
and export counter as Ceph RADOS objects. This enables
highly available(HA) Ganesha servers in manila deployments
to store their config in a HA storage.
Implements: blueprint ganesha-ha-rados
Change-Id: Ia51156055fa10d0661e662c9c998829864f1a204
Driver shouldn't have to report support for ipv6 in two
places. Drivers that assert ipv6_implemented=True just
need to implement get_configured_ip_versions. Likewise
ipv4 support is computed by the same method.
Closes-bug: #1734127
Change-Id: I382767918a65b91e99ac1e604304ad01fac332e6
They were deprecated in pike [1].
Remove them in Queens.
TrivialFix
[1] If39f1622583dd797152de0c93ba68ab0b28cb632
Change-Id: Icd89f1d3bddbecc48d4af0f2e9ead9f61cc6f6fe
Some of the available checks are disabled by default, like:
[H106] Don't put vim configuration in source files
[H203] Use assertIs(Not)None to check for None
[H904] Use ',' instead of '%', String interpolation should be
delayed to be handled by the logging code, rather than
being done at the point of the logging call.
Change-Id: Ie985fcf78997a86d41e40eacbb4a5ace8592a348
Reverts with the LVM driver were happening asynchronously
which resulted in unreliable behavior. This change makes
the reverts synchronous.
Also, the driver now unexports and unmounts both the share and
the snapshot during the revert (required to make the operation
synchronous) which requires changing the driver interface to
include the snapshot access rules in addition to the share access
rules.
Closes-bug: #1707943
Change-Id: Ifea1799e1d2e3963fec7e90ce3f9cb47b9f02f4f
Please read spec for design detail [1].
Support IPv6 in IP drivers, networks and share
type extra specs.
Co-Authored-By: TommyLikeHu(tommylikehu@gmail.com)
Co-Authored-By: Ben Swartzlander <ben@swartzlander.org>
[1] f7202a6cfe32a057f752a4e393f848f8a0211c36
DocImpact
Partial-Implements: blueprint support-ipv6-access
Change-Id: I96d3389262e9829b8b4344870cdf5c76abd22828
We need to have a way to gather information about actual share storage
usages, so cloud operators could use this information for billing,
health checks and/or other purposes.
Change-Id: Iaca1bb541a34af862b938e17e4a56d53de7a9cc4
Implement-Blueprint: share-usage-size
That will be used for scheduling share groups based on their possibility
to create consistent snapshots.
Also apply following tempest plugin changes:
- Add new 'capability_sg_consistent_snapshot_support' tempest config
option, that will be used for creation of new share group types and used
to prove that scheduling works as expected.
- Fix some share group test attributes from 'only API involved' to
'API and Backend are involved', because it is so indeed.
Change-Id: I05553c308ae40c4ddc2c6469ff1c1a3da36a87da
Partially-Implements BP manila-share-groups
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: I2d1eb1acd7bfe297f33b2af810b3a52b5607d1ad
A "bugfix" was commited which allows users to configure backends
to use SSL in an insecure way. This change removes the bad bugfix.
This reverts commit 8b6868aaae.
Change-Id: I342b3639cd42aa4ac62a85d3dad29cab0a421e31
From python 2.7.9 on, the ssl verification is enabled by default,
This commit adds 2 options for Manila, so user is able to control the
ssl verification.
The 2 options are ported from cinder/driver.py, these can be used by any
vendor driver which needs to handle ssl verification.
Closes-bug: 1669202
Implements: blueprint add-ssl-verification-options
Change-Id: Ia9a488cab9d4d2d25c5ab534dbf4d61e930cfd7f
in the native cephfs driver to align with the overall deprecation
in manila of Consistency Groups in favor of Share Groups.
Also update a docstring reference to Consistency groups in the
base driver class.
TrivialFix
Change-Id: I69d7d390ae4d4cc7419e4e20c81f44349c533dc8
agireesh