Export locations are usually too difficult to memo
rize.Currently, there is no way to determine the
export location before the share is created, so
users wait until the share creation request gets
completed, and then they check the export
locations to mount the share. The generated
export locations are often not human readable
and it is hard to memorize and control them.
Implements: bp/human-readable-export-locations
Change-Id: I72ac7e24ddd4330d76cafd5e7f78bac2b0174883
'reserved_share_extend_percentage' backend config option allows Manila
to consider different reservation percentage for share extend
operation. With this option, under existing limit of
'reserved_share_percentage', we do not want user to create new share if
limit is hit, but allow user to extend existing share.
DocImpact
Closes-Bug: #1961087
Change-Id: I000a7f530569ff80495b1df62a91981dc5865023
Manila can now configure a share network with multiple subnets
in an availability zone. Also, it can add a new subnet for an
availability that has share servers, which will triger an update
share server allocations.
Changes:
- API:
- Bump version to 2.70.
- setup share network with multiple subents per az.
- Block manage server with multiple subnets.
- Allow add subnet for in-use share servers.
- `share_network_subnet_id` is dropped from ShareServer view
- `share_network_subnet_ids` is added in ShareServer view
- `network_allocation_update_support` is added to ShareServer and
ShareNetwork views.
- Add a check operation for share network subnet create.
- DB:
- Remove `share_network_subnet_id` from share_servers.
- Create mapping table `share_server_share_network_subnet_mappings`.
- Fix queries with new db design.
- Add migration downgrade and upgrade alembic.
- Add `share_network_subnet_id` to the NetworkAllocations.
- Scheduler:
- Change `AvailabilityZoneFilter` to take in account if the
host supports the allocation required by the setup request.
- Manager:
- Bump RPC API version.
- `_setup_server` allocating multiple subnets.
- Modify signature of driver `_setup_server` interface, passing a
list of `network_info` for each subnet.
- Share server DB creation to inform a list of subnets and
create with `network_allocation_update_support`.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations`.
- Drivers:
- For legacy compatibility, all drivers implementing `_setup_server`
consume the first element of the `network_ino`.
- Dummy Driver:
- Implement `_setup_server` with new signature as multiple subnet.
- Modify the `backend_details` to save allocations for all subnets.
- Report update allocation and share server multiple subnet support.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations` interfaces.
Signed-off-by: Felipe Rodrigues <felipefuty01@gmail.com>
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Co-Authored-By: Fábio Oliveira <fabioaurelio1269@gmail.com>
Co-Authored-By: Nahim Alves de Souza <nahimsouza@outlook.com>
Co-Authored-By: Caique Mello <caique_mellosbo@hotmail.com>
DocImpact
APIImpact
Partially-Implements: blueprint multiple-share-network-subnets
Change-Id: I7de9de4ae509182e9494bba604979cce03acceec
Python2 is no longer supported, so in this patch
set we remove the usage of the six (py2 and py3
compatibility library) in favor of py3 syntax.
Change-Id: I3ddfad568a1b578bee23a6d1a96de9551e336bb4
This config option allows different value for reservation percentage,
mostly useful on the platforms, where shares can only be created from
the snapshot on the host where snapshot was taken. The lower value of
this config option against existing (reserved_share_percentage) allows
to create shares from the snapshot on the same host up to a higher
threshold even though non-snapshot/regular share create fails.
In case this config option is not set, the shares created from snapshot
will use reservation percentage value set in 'reserved_share_percentage'.
This will be useful for users who want to keep same reservation percentage
for both non-snapshot/regular and snapshot shares.
DocImpact
Closes-Bug: #1938060
Change-Id: I390da933fe92875e3c7ee40709eacacc030278dc
This patch implements the update of security service's association
with in-use share networks. The following changes were added:
- New share network APIs: `share_network_security_service_update`
and `share_network_reset_state`.
- A new `status` attribute was added to share network model to
identify when it's in a modification state, called 'network_change'.
Other supported status that were added: 'active' and 'error'.
- New 'security_service_update_support' property was added to both
share server and share network models, to identify when this resources
are able to process security service update for in-use share networks.
- New driver interface was added to support update of security service's
configuration of a given share server.
DocImpact
APIImpact
Partially Implements: bp add-security-service-in-use-share-networks
Co-Authored-By: Carlos Eduardo <ces.eduardo98@gmail.com>
Co-Authored-By: Douglas Viroel <viroel@gmail.com>
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Change-Id: I129a794dfd2d179fa2b9a2fed050459d6f00b0de
Add two new capabilities to manila. It is possible to set limits
to share server size and share instances in a share server by
setting `max_share_server_size` and `max_shares_per_share_server`
in a backend stanza.
Change-Id: I3170478d3aa2d09cb2adc32233dc57bc59029a56
Partially-Implements: bp new-share-server-limits
DocImpact
When a share is mounted on the same host as the manila-share
process, zfs prevents us from destroying the underlying
dataset until the share has been cleanly unmounted from
the host. Kernel mounts can take a few seconds to get
unmounted fully especially when there are a lot of
linux namespaces that the mountpoint has been shared to.
Add a retry on these operations to harden the deletion
process and prevent spurious failures.
Change-Id: I4aba76b72df274d0a8cb90fe0ab8799523c260ef
Closes-Bug: #1903773
Related-Bug: #1896672
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
This patch improves the operation of create share from snapshot
to support different backends/hosts for the zfsonlinux driver
Partially-implements: bp create-share-from-snapshot-in-another-pool-or-backend
Depends-On: Iab13a0961eb4a387a502246e5d4b79bc9046e04b
Change-Id: I124803734c81d3630c5147f5f3bb75724489c929
- Add ignores to pylint false positives in the
test modules.
- Remove unnecessary fake data
- Fix wrong mock methods used in tests
Change-Id: I64ffff15cc546c67e7e545b1da7ec0efa002bdc5
Add a retry loop to ensure the dataset being renamed
is cleanly unmounted before the rename operation.
Change-Id: Ie506f237010c415ee9f0d64abbefd5854f776a5f
Closes-Bug: #1785180
Fix pylint E103 warning raised by usage of
self.__class__ to refer to the derived class
in super() methods.
self.__class__ is a reasonable first argument
to super() in any method of a class, as long
as the method is not going to be invoked in
derived classes.
Python3 removes this ambiguity by not requiring
arguments for the super() method.
[1] https://docs.pylint.org/en/1.6.0/features.html#id33
Change-Id: I6071b6cfd8cff2be3853d739f71b94da990cda97
Please read spec for design detail [1].
Support IPv6 in IP drivers, networks and share
type extra specs.
Co-Authored-By: TommyLikeHu(tommylikehu@gmail.com)
Co-Authored-By: Ben Swartzlander <ben@swartzlander.org>
[1] f7202a6cfe32a057f752a4e393f848f8a0211c36
DocImpact
Partial-Implements: blueprint support-ipv6-access
Change-Id: I96d3389262e9829b8b4344870cdf5c76abd22828
That will be used for scheduling share groups based on their possibility
to create consistent snapshots.
Also apply following tempest plugin changes:
- Add new 'capability_sg_consistent_snapshot_support' tempest config
option, that will be used for creation of new share group types and used
to prove that scheduling works as expected.
- Fix some share group test attributes from 'only API involved' to
'API and Backend are involved', because it is so indeed.
Change-Id: I05553c308ae40c4ddc2c6469ff1c1a3da36a87da
Partially-Implements BP manila-share-groups
This new feature gives the user the ability to allow and
deny access to the snapshots, so that they could be mounted in
read-only mode to retrieve files.
APIImpact
DocImpact
Co-Authored-By: Rodrigo Barbieri <rodrigo.barbieri@fit-tecnologia.org.br>
Co-Authored-By: Alyson Rosa <alyson.rosa@fit-tecnologia.org.br>
Co-Authored-By: Miriam Yumi <miriam.peixoto@fit-tecnologia.org.br>
Partially-implements: blueprint manila-mountable-snapshots
Change-Id: I65f398a05f82eef31ec317d70dfa101483b44b30
Remove the experimental consistency group APIs and
replace them with the experimental Share Group APIs.
DocImpact
APIImpact
Partially-implements-blueprint: manila-share-groups
Change-Id: I79a80a62ae4e0015d6161edc2b93fd1f9ba69537
Implemented several improvements to share migration
according to spec [1].
Summary of changes:
- Snapshot restriction in API has been changed to return error only
when parameter force-host-assisted-migration is True
- Added preserve_snapshot to API and migration_check_compatibility
driver interface
- Changed all driver-assisted API parameters to be mandatory
- Added validation to prevent 'force_host_assisted_migration' to be
used alongside driver-assisted parameters
- Changed "same host" validation to reject only if the combination
of "host", "new_share_network" and "new_share_type" is the same as
the source
- Updated migration driver interfaces to support snapshots
- Updated zfsonlinux driver, defaulting preserve_snapshots to False
- Updated dummy driver to support preserve_snapshots
Spec update with latest changes since [1] merged
can be found in [2].
APIImpact
DocImpact
[1] I5717e902373d79ed0d55372afdedfaa98134c24e
[2] If02180ec3b5ae05c9ff18c9f5a054c33f13edcdf
Change-Id: I764b389816319ed0ac5178cadbf809cb632035b4
Partially-implements: blueprint ocata-migration-improvements
- Pull up policy check to beginning of the APIs.
- Avoid making access rules changes when one or
more instances of the share are in an invalid state.
- Add back the per rule share instance access status.
This restoration provides better visibility for which
rules were applied successfully.
- Remove 'updating' and 'updating_multiple' as valid
states for the share instance access rules status.
- Deprecate the access rule state 'new' in favor of
'queued_to_apply' and the share instance access rules
status 'out_of_sync' in favor of 'syncing'.
In a new API micro-version:
- Allow access rule changes irrespective of the share's
access_rules_status.
- Expose new access rule states and share's
access_rules_status values.
Access rules for each share instance now transition
from 'queued_to_apply' to 'applying' to 'active' or 'error';
and from 'active', 'queued_to_apply', 'applying' or 'error'
to 'queued_to_deny' to 'denying' to 'deleted'.
APIImpact
DocImpact
Partially-implements: bp fix-and-improve-access-rules
Co-Authored-By: Mike Rooney <rooneym@netapp.com>
Change-Id: Ic25e63215b5ba723cbc8cab7c51789c698e76f28
This commit adds the ability for Manila to revert a
share to the latest available snapshot.
The feature is implemented in the LVM driver, for
testing purposes.
APIImpact
DocImpact
Co-Authored-By: Ben Swartzlander <ben@swartzlander.org>
Co-Authored-By: Andrew Kerr <andrew.kerr@netapp.com>
Implements: blueprint manila-share-revert-to-snapshot
Change-Id: Id497e13070e0003db2db951526a52de6c2182cca
The snapshot_support extra spec has always meant two
things: a driver can take snapshots and create shares
from snapshots. As we add alternate snapshot semantics,
it is likely that some drivers will want to support
snapshots and some of the new semantics while being
unable to create new shares from snapshots.
This work adds a new extra spec,
create_share_from_snapshot_support, that removes the
overloading on snapshot_support. It also makes the
existing snapshot_support extra spec optional,
allowing admins to create types without setting
snapshot_support; shares created with such types
will not support snapshots.
APIImpact
DocImpact
Co-Authored-By: Goutham Pacha Ravi <gouthamr@netapp.com>
Implements: blueprint add-create-share-from-snapshot-extra-spec
Change-Id: Ib0ad5fbfdf6297665c208149b08c8d21b3c232be
Because, it leads to inheritance of access rules too. That is a bug.
All other expected options are generated on-fly already.
Change-Id: Ibe744c9fd001156a8184aa8f955c424cebfb8664
Closes-Bug: #1645746
For the moment, share migration works only when manila-share and
ZFS storage are located on the same host. So, fix it adding SSH
command to migration command.
Change-Id: I25f211de6a278c6f303ef3f33ff30f504146a0fa
Closes-Bug: #1623379
Big part of share migration feature in ZFSonLinux share driver
is not covered with unit tests. So add it.
Also, add reno releasenote saying that driver now supports share
migration.
Change-Id: I558d4c6a0802b6a0156416b3c9b481181a55298c
Closes-Bug: #1621068
Current implementation of share migration works only if we use
local shell executor. So, refactor approach for it to make work
with both shell executors - local and remote (SSH).
Change-Id: Ibe31bf52006040ea26badfa7236ed67559f7aded
Closes-Bug: #1620502
Add support of share migration feature to ZFsonLinux driver.
Implements BP zfsonlinux-share-migration
Change-Id: If9e1fec9a6d4f96d2bb5c176c6249fafc7e3e891
Commit [1] implemented bug when deletion of replicated snapshots
may fail if active replica deleted not as last among all snapshot
replicas. It was caused by deletion of private data of snapshot,
only by active replica.
So, make that data be deleted only in the end of driver method
that deletes replicated snapshots.
[1] If240ecde5676c334d39faaccd5703e93544aaa06
Change-Id: Ic2b7b7daf0ba5db8eb7166969413b590b27effb5
Closes-Bug: #1607765
Implement 'manage snapshot' feature in ZFSonLinux share driver.
Prerequisites:
- snapshot being managed should exist.
Details:
- snapshot being managed will be renamed by share driver based on
taken ID.
- size for managed snapshot will be taken either from
REST API (--driver_options size=5) or
from its "used" attribute.
Implements BP zfsonlinux-manage-snapshot
DocImpact
Change-Id: If240ecde5676c334d39faaccd5703e93544aaa06
Depends-On: Ifc762a882a78159adacfe168a4edbe824178301a
This patch ports cinder's DriverFilter and GoodnessWeigher to manila.
These can use two new properties provided by backends,
'filter_function' and 'goodness_function', which can be used to filter
and weigh qualified backends, respectively.
Reference for cinder spec: I59b607a88953a346aa35e67e785a0417a7ce8cc9
Reference for cinder commit: I38408ab49b6ed869c1faae746ee64a3bae86be58
DocImpact
Change-Id: I873f4152e16efdeb30ceae26335a7974dc9b4b69
Implements: blueprint driver-filter-goodness-weigher
Implement 'manage share' feature in ZFSonLinux share driver.
Prerequisites:
- share being managed should exist.
- share being managed should not be part of any replication relations.
- share being managed should not have redefined mountpoint.
Details:
- share being managed will be renamed by share driver based on taken ID.
- share being managed will get "quota" attribute set based on provided
value in API (--driver_options size=5) or it will set quota
to nearest bigger rounded integer of already
used space (2,4 -> 3, 3.6 -> 4).
Implements blueprint zfsonlinux-manage-share
DocImpact
Change-Id: I03eab5e02835fe660dde0020601c02c0136bddeb
When taking a snapshot of a replicated share, users would
expect the snapshot to be available across the replicas.
Currently, Manila assumes that drivers will take snapshots
along with the share data.
Different backends implement replication differently. Many
of them assume that replicas are 'in_sync' with the primary
if they have been synchronized within a particular window
of time (sometimes referred to as 'recovery point objective').
Since snapshots are point in time actions on a share, drivers
must ensure that the replication of a snapshot begins as soon
as it is available on the primary/'active' instance.
To ensure that snapshots are indeed available across replicas:
1) Track snapshots across replicas as different
snapshot instances.
2) Make an 'aggregate_status' property to convey the
availability of a snapshot across all the active/in_sync
replicas of a share.
3) Provide all the information necessary for drivers to:
- Create a snapshot on the primary and send it across
to all replicas
- Delete a snapshot on the primary and ensure that it is
deleted on the replicas as well.
4) Set snapshot instance to 'error' if it was in any transitional
state when its replica was 'promoted' to become an 'active'
instance.
5) Pass down snapshot instances during create, delete and update
replica driver calls.
6) Update replica interfaces to pass in snapshot instance data. Always
include 'available' snapshots in the update_replica call to ensure
that drivers confirm their presence on the replica in order to report
that the replica is 'in_sync'.
Also, grab the share_type afresh from parent share/snapshot
to allow 'replication_key' to be copied to the share model during
share creation.
Rename SnapshotNotFound to SnapshotResourceNotFound to
signify a missing snapshot object on backends.
APIImpact
The 'status' attribute of the snapshot is now the 'aggregate_status'
of the instances of the snapshot. No key changes have been made
in the JSON response. However, API actions will be gated against
'aggregate_status' of the snapshot instances when more than one
instance exists.
Closes-Bug: #1546303
Closes-Bug: #1551064
Change-Id: I63203df00904d209e9e92eda7c79206e5ef8d611
With this fix it is now possible to
- replicate share created from snapshot.
- create share from snapshot after "promote replica" action.
Change-Id: Id318a4adc0faf64a4bef57252aa2f0d9083b82b1
Depends-On: I63203df00904d209e9e92eda7c79206e5ef8d611
Closes-Bug: #1557283
Related-Bug: #1546303
ZFSonLinux share driver supports dedupe and compression
but does not report such capabilities to scheduler.
So, make driver do following things:
- Report 'dedupe' and 'compression' capabilities
based on values of its config option 'zfs_dataset_creation_options'.
Both can be either enabled or disabled.
- Report proper value of 'thin_provisioning' as [True],
because ZFSonLinux driver does not support space reservation for
shares and uses only quotas for managing used size.
- Make driver expect type of compression as
qualified extra spec 'zfsonlinux:compression'.
Change-Id: I4028b3fae4e39d0303e7ead5eba6a49b7ca9d977
Closes-Bug: #1557031
Recently merged commit [1] removed default values for "update_access"
interface of all drivers, but didn't fix usage of that
interface for ZFSonLinux driver that led to TypeError exception.
So, provide "add_rules" and "delete_rules" args always as
they are required now.
Driver methods that were broken and being fixed are following:
- Create replica
- Update replica
- Promote replica
[1] I86f4ccc9d496ec6183bd0fa5be9a77c3451378d5
Change-Id: Ic34f8e51436b45e4129aac597fb22428d21f2bae
Closes-Bug: #1557885
Rather complex logic was added to ZFS driver to cope with
mount-namespace-related problems caused by neutron. A better
solution is to ensure mounts (and unmount) propagate to
namespaces.
Closes-Bug: 1552552
Change-Id: Id952cb110c0187021a46ed5de02308b10ddf1239
ZFSonLinux share driver uses username and service IP address
for replication actions. And if we change those values in config
and restart manila-share services driver still will use old values
in "promote" and "update/resync" operations.
So, update that info in each "ensure_share" call that is performed
on start of service.
Change-Id: I565d9e41ff788331af5f30d0edcc7f33d05b806b
Closes-Bug: #1552320
Add support for ZFS (on Linux) filesystem.
Following features are supported:
- Create/delete share
- Create/delete snapshot
- Create share from snapshot
- Extend/shrink share
- Update NFS IP-based access rules using new interface.
(SMB support planned for future).
- create/delete/update/promote share replica
ZFS-related notes:
- Any amount of ZFS zpools can be used by share driver.
- Allowed to configure default options for ZFS datasets that are used
for share creation.
- Any amount of nested datasets is allowed to be used.
- All share replicas are read-only.
- All share replicas are synched periodically, not continuously.
So, status 'in_sync' means latest sync was successful.
Time range between syncs equals to value of
config global opt 'replica_state_update_interval'.
Driver-related notes:
- Able to use remote ZFSonLinux storage as well as local.
Other made changes:
- updated driver private data DB methods removing filtering by host
as redundant operation. Replication requires some common metadata
storage and filtering by host breaks it. It is safe to do so, because
if driver gets some ID of entity then it is allowed to read its info
too.
Implements bp zfsonlinux-driver
DocImpact
Change-Id: I3ddd3767184e4843037de0ac75ff18dce709b6dc
jayaanand.borra@netapp.com