Conditionally Import Parakimo
Separate SSH functions into ssh_utils.py for safe conditional import.
Change-Id: Ia1a3ee69bef76b52e4e6df1e73488c018ac0f3c9
'reserved_share_extend_percentage' backend config option allows Manila
to consider different reservation percentage for share extend
operation. With this option, under existing limit of
'reserved_share_percentage', we do not want user to create new share if
limit is hit, but allow user to extend existing share.
DocImpact
Closes-Bug: #1961087
Change-Id: I000a7f530569ff80495b1df62a91981dc5865023
Manila can now configure a share network with multiple subnets
in an availability zone. Also, it can add a new subnet for an
availability that has share servers, which will triger an update
share server allocations.
Changes:
- API:
- Bump version to 2.70.
- setup share network with multiple subents per az.
- Block manage server with multiple subnets.
- Allow add subnet for in-use share servers.
- `share_network_subnet_id` is dropped from ShareServer view
- `share_network_subnet_ids` is added in ShareServer view
- `network_allocation_update_support` is added to ShareServer and
ShareNetwork views.
- Add a check operation for share network subnet create.
- DB:
- Remove `share_network_subnet_id` from share_servers.
- Create mapping table `share_server_share_network_subnet_mappings`.
- Fix queries with new db design.
- Add migration downgrade and upgrade alembic.
- Add `share_network_subnet_id` to the NetworkAllocations.
- Scheduler:
- Change `AvailabilityZoneFilter` to take in account if the
host supports the allocation required by the setup request.
- Manager:
- Bump RPC API version.
- `_setup_server` allocating multiple subnets.
- Modify signature of driver `_setup_server` interface, passing a
list of `network_info` for each subnet.
- Share server DB creation to inform a list of subnets and
create with `network_allocation_update_support`.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations`.
- Drivers:
- For legacy compatibility, all drivers implementing `_setup_server`
consume the first element of the `network_ino`.
- Dummy Driver:
- Implement `_setup_server` with new signature as multiple subnet.
- Modify the `backend_details` to save allocations for all subnets.
- Report update allocation and share server multiple subnet support.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations` interfaces.
Signed-off-by: Felipe Rodrigues <felipefuty01@gmail.com>
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Co-Authored-By: Fábio Oliveira <fabioaurelio1269@gmail.com>
Co-Authored-By: Nahim Alves de Souza <nahimsouza@outlook.com>
Co-Authored-By: Caique Mello <caique_mellosbo@hotmail.com>
DocImpact
APIImpact
Partially-Implements: blueprint multiple-share-network-subnets
Change-Id: I7de9de4ae509182e9494bba604979cce03acceec
Python2 is no longer supported, so in this patch
set we remove the usage of the six (py2 and py3
compatibility library) in favor of py3 syntax.
Change-Id: I3ddfad568a1b578bee23a6d1a96de9551e336bb4
This config option allows different value for reservation percentage,
mostly useful on the platforms, where shares can only be created from
the snapshot on the host where snapshot was taken. The lower value of
this config option against existing (reserved_share_percentage) allows
to create shares from the snapshot on the same host up to a higher
threshold even though non-snapshot/regular share create fails.
In case this config option is not set, the shares created from snapshot
will use reservation percentage value set in 'reserved_share_percentage'.
This will be useful for users who want to keep same reservation percentage
for both non-snapshot/regular and snapshot shares.
DocImpact
Closes-Bug: #1938060
Change-Id: I390da933fe92875e3c7ee40709eacacc030278dc
- Add ignores to pylint false positives in the
test modules.
- Remove unnecessary fake data
- Fix wrong mock methods used in tests
Change-Id: I64ffff15cc546c67e7e545b1da7ec0efa002bdc5
We assign a random UUID with tune2fs to snapshots
and shares created from snapshots so that they
don't conflict with the parent shares/snapshots
when both of them are being mounted.
tune2fs requires that a filesystem check be
performed "recently" before UUID assignments.
So, perform a filesystem check right away to
allow tune2fs to assign a random UUID.
Change-Id: I858a318f7a83e033cc3f2699859e38b6b74c8d24
Related-Bug: #1645751
Closes-Bug: #1798219
The /etc/mtab file may have mount information such as 'nfsd'
that if copied to /etc/fstab will cause the share server to
hang when rebooted.
Update /etc/fstab with exactly the newly mounted or unmounted
shares rather than simply overwriting /etc/fstab with /etc/mtab.
Change-Id: I67602bae1f928769d768008deca7bd0f2fef1ac2
Close-Bug: #1639662
Because the former was implemented just because the latter
was absent yet at that moment. Main reason is that they serve
same goals and should be consistent to each other. And it is
much easier to support one single table instead of two separate
tables syncing them each time we update one of them.
Change-Id: I81ba26437554d02a79e4b536d781617f46ce2f07
Partially-Implements bp manila-share-groups
Remove the experimental consistency group APIs and
replace them with the experimental Share Group APIs.
DocImpact
APIImpact
Partially-implements-blueprint: manila-share-groups
Change-Id: I79a80a62ae4e0015d6161edc2b93fd1f9ba69537
Reportedly [1], the kernel-nfs-server has a bug where we have overlap of
filesystem identifiers using LVM volumes and their snapshots.
We face this bug in manila LVM driver and Generic driver when we use
cinder LVM driver. So, fix it generating unique UUID for all
snapshots (only for LVM, because we cannot do it in Generic driver
while we do not have "mount-snapshot" feature support there) and
child volumes that will be exported as child shares.
[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1071733
Change-Id: Ib93bfb0d4184da1e70ecece6fde6931e44d05a30
Closes-Bug: #1645751
Recent change to devstack [1] broke our DHSS=False CI job running
generic driver that was depending on ip route from host to
private network. So, to avoid this error use floating ip address
for connection to service Nova VMs from host machine.
Also, fix generation of second export location that should be dedicated
for service needs such as mounting share doing host-assisted
migration.
[1] If45e3fc15c050cfbac11b57c1eaf137dd7ed816f
Change-Id: Ieea992293ae02898741c939da15f0dbb4609d8b0
Closes-Bug: #1644523
Closes-Bug: #1646097
If a share is shrank from 10G to 5G, the volume is still 10G.
This share`s size display is 5G, but the share extend newsize
must be > 10G in the old code. So, I think only need to perform
resize_filesystem when the share extend newsize <= volume size.
The volume extension is performed only when new size larger than
volume size.
Change-Id: I7e49b446445b8005e2eb23e1d439354eb24915e0
Close-Bug: #1639188
Generic share driver has fake support of consistency groups. It was
implemented only for testing purpose. Now, it is redundant, because
we have 'dummy' share driver that already used to test this behavior.
So, drop fake CG support from Generic share driver and disable
appropriate tempest tests.
Change-Id: I6ce07fd3a11cd62a3a01ba4ee7c424b839a62757
Closes-Bug: #1638994
It appears that remote executor (processutils.ssh_execute)
and local executor (processutils.execute) have different
expectations for command keywords containing spaces. This
difference manifests itself in helper failures either for
one executor type or for another. This change adds command
preprocessing to generic driver which seems to be the only
active user of both remote executor and failing helpers.
Change-Id: I26eaca3ca652171fbf20d7580f90eef4f935332e
Closes-Bug: 1621033
Deleting a share will bring it into error_deleting
if the volume is already deleted. And VolumeNotFound
exception is raised. The fix will ignore the exception
and print a warning.
Change-Id: I82d70b9997a689dd3fd8eefad7dc14c39f9d275a
Closes-Bug: #1563226
Implemented admin network support in generic driver
by creating extra ports in service instance.
In order to implement admin export location, Admin must
specify admin_network_id and admin_subnet_id
in driver configuration in manila.conf to create ports
between host and network of choice.
Possible scenarios are covered by this change:
*1: service network.
*2: service network and tenant network.
*3: service network and admin network.
*4: tenant network and admin network.
Admin network and tenant network configurations have
higher priority than service network. If both are
present, service network is not used.
Only admin network export locations are
is_admin_only = True.
Included additional admin_only export location
when creating shares and creating shares from
snapshots.
Removed deprecated 'service_ip' property of
server backend details.
Implements: blueprint admin-network-generic-driver
Change-Id: I8d8694ac3d83aa12a756112dfefebd7e17e32383
Add update_access() method to generic and LVM drivers,
to NFSHelper and CIFSHelper.
Remove allow_access() and deny_access() from helpers.
Partially implements bp new-share-access-driver-interface
Co-Authored-By: Rodrigo Barbieri <rodrigo.barbieri@fit-tecnologia.org.br>
Change-Id: Ie34c92f8b9dc95d6af5cb150427503bf66543ab6
Add APIs to support manage and unmanage share snapshots.
Also add support in the Generic driver.
This only supports for DHSS=False driver mode.
Add provider_location column to the share_snapshots table
to save data used to identify the snapshot on the backend.
Also need to bump microversion.
APIImpact
DocImpact
Change-Id: I87a066173c85d969607d132accd9f0e9bd49c235
Implements: blueprint manage-unmanage-snapshot
When using the generic driver, a volume is attached to the service
VM and then a mkfs is done to create a FS on that volume inside of
the VM. But it can be the case that it takes a bit of time before
the device node pops up inside of the VM. So wait some seconds
before doing a mkfs on a non-existing device node.
Change-Id: I42228d9a8ad836a95be5c59e43bd6762677dd85f
Closes-bug: #1547000
Reuse code of old LVM share driver.
LVM Driver is 1st party Manila driver with NFS and CIFS support
and no share server support.
LVM Driver doesn't rely on Neutron, Nova, or Cinder. As such, it
is ideal for testing purposes and, after a period of maturation,
it should be useful in production environments.
Move generic driver's helpers to manila/share/drivers/helpers.py
and reuse them in LVM driver.
Implement 'ro' access, user access, extend_share function for
LVM driver.
Implements bp lvm-driver
Change-Id: Ia46c51ed400dbb0f1d87a758fb165ca711ed3d7c
Some upcoming features require more than one export location and
possibility to mark them with specific labels like fast/slow or
rw/ro.
So, implement 'export locations metadata' feature:
- It allows to set any key-value pairs for each export location.
- These key-value pairs can be set only by share manager using
response from various share driver methods.
- Example of update is implemented using Generic driver
"create_instance" method.
- Metadata can be updated for any export location in any place
of share manager where db function "share_export_locations_update"
is called.
- Keys from export location metadata table will be added to 'share' and
'share instances' views as export location attributes.
Also:
- Add new attr 'is_admin_only' for existing export locations model.
If set to True, then only admins will be able to see them. Unless
policy is changed.
- Add APIs for reading export locations by share and share instance IDs.
- Remove 'export_location' and 'export_locations' attrs
from 'share' and 'share instance' views.
- Bump microversion as new APIs implemented.
APIImpact
Implements bp export-location-metadata
Change-Id: I36d1aa8d9302e097ffb08d239cf7a81101d2c1cb
QoS is a common attribute, so add the QoS
capability in common capabilities doc.
This is a simple first step towards what the
blueprints proposes.
Partially implements: bp manila-support-qos
Change-Id: I377bf0abcc62239c9a1a5ee5c28c336b2b6c410a
Set manila-server connection timeout enabled via configuration
file on generic driver to prevent ssh connections getting
stuck if manila-server is slow in responding or some network
problems occur.
Closes-Bug: #1528203
Change-Id: Icd72c561aa37fad447e3f6e1eade3b2f57d74521
The usage of assertEqual(True/False, ***) should be changed
to a meaningful format of assertTrue/False(***).
This patch fixes the same in Manila.
Change-Id: I0623c17a939e5175c15986dc7b6b64114c346c78
Closes-Bug: #1512207
If the driver does not handle share servers, during init, it
verifies that the pre-configured share server exists and is available
(behavior recently introduced by
I325d6461c4b12f481769bb01f792ef850dba789c).
If the instance does not exist when the service starts, the driver
will remain permanently in an uninitialized state, even if the
according instance is created meanwhile.
This patch changes this behavior, waiting for the common server
to become active, logging a meaningful message. This can considerably
ease the deployment of this driver as well as other drivers based
on it.
At the moment, the Windows SMB driver CI jobs fail due to this issue,
when DHSS is set to false (in which case the Nova Hyper-V nodes are
deployed after Manila).
Change-Id: Icf446ca25cf0c64396223611c6271d4c6445cb4d
Closes-Bug: #1506951
If we setup generic driver in DHSS=False mode, we need to provide
specific data for VM that will host shares and if we do not provide
such info or provide it incorrectly then we get error only trying
perform actions. But we should do validation on service start.
Change-Id: I325d6461c4b12f481769bb01f792ef850dba789c
Closes-Bug: #1502063
Fix incorrect order assertEqual(observed, expected) as below.
assertEqual(observed, expected) => assertEqual(expected, observed)
Target of this patch:
manila/tests/share/*
Note:
I also fix following asserts around above fixed parts.
assertEqual(xx, True), assertEqual(True, xx) => assertTrue(xx)
As for assertFalse,
I do not convert assertEqual(xx, False) to assertFalse(xx)
because assertFalse(None) does not raise exception.
Change-Id: I56688105ab3b77e4f71bde472763a9c02c43664b
Partial-Bug: #1259292
If we try to extend share created with generic driver and Cinder does not have
enough free space to extend its volume then we get error "Insufficient
free space". But Manila updates share status only after 3 minutes.
Add check if volume status contains 'error' to '_wait_for_available_volume'
function.
Closes-bug: #1484857
Change-Id: I1e206ec1c410c015a71f49e25ffa8e64b5870e64
In ShareManager::manage_existing(), there is a check for the DHSS mode of
the driver but there is no check for the DHSS mode in the specified share
type. This omission requires redundant checks in the drivers. This patch
adds the missing check to the share manager, removes the redundant checks
from the drivers, and updates all relevant unit tests. Also, fix a
couple minor tempest resource cleanup issues discovered while
debugging this issue.
Change-Id: Ib579fd0558e59c28777342bb9d36def12f6bf4da
Closes-Bug: #1493869
The Generic Driver in Manila has no concept of consistency
groups. This patch proposes to implement a loose
consistency group support mechanism for the Generic Driver,
using existing logic to snapshot shares and create shares
from snapshots.
The actions concerning creating and deleting a consistency
group are handled by the API and the Generic Driver does not
need to remember any relevant information persistently.
The actions concerning consistency group snapshots are not
simultaneous, but are rather let to run in succession.
DocImpact
Partially implements bp manila-consistency-groups
Change-Id: Ie5e51205a1154786f9057691aa82bffcafe04de7
Added minimal Share Migration support to generic driver,
so the Generic Migration procedure can work.
The approach to support is to add a field to Share Server
backend_details named service_ip with the port IP address that
allows the controller to access the Share Servers (same IP address
which they SSH through), this IP is used to return the access rule
required for the generic migration procedure.
For "driver_handles_share_servers=False" mode, configuration falls
back to common implementation in Core code and its parameters.
Optimized migration for Generic Driver will be included
as a separate patch.
Change-Id: Ie7d19a44c9ab05c5a6a79c30478be23b2e8a3ba2
Implements: blueprint share-migration
This patch implements the scheduler changes for
scheduling the creation of consistency groups,
shares within consistency groups, and cgsnapshots.
The consistency group scheduling filter was added
to allow the filtering of hosts that are not supported
by a specified consistency group. If no CG is specified
on share create then this filter is a no-op. CGs will be
scheduled only to a backend that supports all of its
specified share types.
Partially implements bp manila-consistency-groups
Change-Id: Ia03191085cefb47a17ce99ad3f30ba70412f5802
When deleting a share on a generic driver backend while the underlying
cinder volume isn't available, ignore the VolumeNotFound exception
and just continue to delete the share.
This prevents the situation where a share can stay in error_deleting
status forever.
Closes-Bug: #1489573
Change-Id: I8cb4400573366587ac11ed28b82bb960e3bbd566
Replace assertEqual(None, *) with assertIsNone in tests to have
more clear messages in case of failure.
Closes-Bug: #1280522
Change-Id: I2d5285add72678b5a61b4f66879c4d066afc34c7
When extending shares, the Generic share driver will extend
the underlying Cinder volume. It will consider this operation to
have finished once the volume status is "available".
In some cases, the volume status can be reported as active before
actually starting the extend operation. For this reason, attaching
the volume can fail as the volume would be in 'extending' state.
This patch fixes the issue by properly waiting for the volume
extend operation to finish, checking the volume size against the
expected size.
Change-Id: I02e4a42110e323dc295e20f9f07411a13fdcc236
Closes-Bug: #1479697
Func 'xrange' is absent in py3. So, use six.moves.range instead for py34
compatibility.
Change-Id: I76217598ae08849c2a93746392fc552dc0a51de1
Partially-Implements: bp py3-compatibility
As of liberty.2, the Manila unit tests are taking ~60 seconds to run
on my system. A dozen or so of the tests consistently take an integral
number of seconds to run (i.e. 1, 2, 3 or 4 seconds), indicating the
presence of a time.sleep() call in the code that hasn't been mocked.
By mocking time.sleep() and time.time() in just a few files, the time
needed to run the unit tests falls by a third to ~40 seconds (!).
Change-Id: I66a6d864823b10a4193eadf65a1c5d3028f2afa7
Closes-bug: 1478644
When we create a share and mount it on client VM, we cannot do
'manila extend' because of error:
Stderr: u'umount: /shares/share-xxxxxx: device is busy.
We need to 'disable_access_for_maintenance' before doing an extension
and 'restore_access_after_maintenance' after extension was done.
Closes-Bug: #1473033
Change-Id: I4b5e665f8d028d4784d02411bbe6daace30aa49e
ashrod98