Add CRUD APIs for resource locks with support
for preventing deletion of shares (applies to
soft-deletions and unmanage operations as well).
Change-Id: I146bc09e4e8a39797e22458ff6860346e11e592e
Implements: bp/allow-locking-shares-against-deletion
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
When manila services are stopped or restarted via stop(), the DB
entries are not deleted, they are destroyed only in kill() method. In
cluster deployments, where multiple instances of manila services are
deployed via PODs, unique hostname is derived from node name. However
if pods are deployed again and launched on new hosts/nodes, the old
entries of manila service remains as it is.
Fix it by adding 'state' column in 'services' table and introducing
per service cleanup function. On service stop, state is changed to
'stopped' and cleanup function will delete 'stopped' services unless
they are 'up' again before cleanup periodic interval.
Closes-bug: #1990839
Change-Id: I8b71c4c27ff8fcb25616a95a5ed8362a7f4ffc61
This change adds metadata controller for Share Network Subnets
resource. Bumps microversion to 2.78.
The subnet metadata is passed down to the driver together
with the network_info object.
APIImpact
Partially-implements: bp/metadata-for-share-resources
Change-Id: I8d5a03eb127941a84eea5e6e9bdf76b3489f17a8
Manila can now configure a share network with multiple subnets
in an availability zone. Also, it can add a new subnet for an
availability that has share servers, which will triger an update
share server allocations.
Changes:
- API:
- Bump version to 2.70.
- setup share network with multiple subents per az.
- Block manage server with multiple subnets.
- Allow add subnet for in-use share servers.
- `share_network_subnet_id` is dropped from ShareServer view
- `share_network_subnet_ids` is added in ShareServer view
- `network_allocation_update_support` is added to ShareServer and
ShareNetwork views.
- Add a check operation for share network subnet create.
- DB:
- Remove `share_network_subnet_id` from share_servers.
- Create mapping table `share_server_share_network_subnet_mappings`.
- Fix queries with new db design.
- Add migration downgrade and upgrade alembic.
- Add `share_network_subnet_id` to the NetworkAllocations.
- Scheduler:
- Change `AvailabilityZoneFilter` to take in account if the
host supports the allocation required by the setup request.
- Manager:
- Bump RPC API version.
- `_setup_server` allocating multiple subnets.
- Modify signature of driver `_setup_server` interface, passing a
list of `network_info` for each subnet.
- Share server DB creation to inform a list of subnets and
create with `network_allocation_update_support`.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations`.
- Drivers:
- For legacy compatibility, all drivers implementing `_setup_server`
consume the first element of the `network_ino`.
- Dummy Driver:
- Implement `_setup_server` with new signature as multiple subnet.
- Modify the `backend_details` to save allocations for all subnets.
- Report update allocation and share server multiple subnet support.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations` interfaces.
Signed-off-by: Felipe Rodrigues <felipefuty01@gmail.com>
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Co-Authored-By: Fábio Oliveira <fabioaurelio1269@gmail.com>
Co-Authored-By: Nahim Alves de Souza <nahimsouza@outlook.com>
Co-Authored-By: Caique Mello <caique_mellosbo@hotmail.com>
DocImpact
APIImpact
Partially-Implements: blueprint multiple-share-network-subnets
Change-Id: I7de9de4ae509182e9494bba604979cce03acceec
Add support share Recycle Bin, the end user can soft delete
share to Recycle Bin, and can restore the share within 7 days,
otherwise the share will be deleted automatically.
DocImpact
APIImpact
Partially-Implements: blueprint manila-share-support-recycle-bin
Change-Id: Ic838eec5fea890be6513514053329b1d2d86b3ba
This patch implements the update of security service's association
with in-use share networks. The following changes were added:
- New share network APIs: `share_network_security_service_update`
and `share_network_reset_state`.
- A new `status` attribute was added to share network model to
identify when it's in a modification state, called 'network_change'.
Other supported status that were added: 'active' and 'error'.
- New 'security_service_update_support' property was added to both
share server and share network models, to identify when this resources
are able to process security service update for in-use share networks.
- New driver interface was added to support update of security service's
configuration of a given share server.
DocImpact
APIImpact
Partially Implements: bp add-security-service-in-use-share-networks
Co-Authored-By: Carlos Eduardo <ces.eduardo98@gmail.com>
Co-Authored-By: Douglas Viroel <viroel@gmail.com>
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Change-Id: I129a794dfd2d179fa2b9a2fed050459d6f00b0de
This patch adds support for migration of share servers. This
migration is performed using a two-phase approach. Administrators
are now able to request the migration of a share server within and
across backends, with the possibility of chooosing a different share
network for the destination share server.
- A new field called `task_state` was added to the share server
model in order to help the administrator to track the share server
migration steps. A new field called `source_share_server_id` was
added to link destination and source share servers.
- A new periodic task was added to track migration of share servers
and its resources.
- Two new states were added: `server_migrating` and
`server_migrating_to` to represent that share migration is in
progress.
- When performing the server migration, manila will not go to the
scheduler, instead it will provide a request spec to drivers
during migration check driver call. It'll be up to the driver
validate if there is free space to handle the share server.
- A new API called `share-server-migration-check' was added to
check the feasibility of a migration, before actually triggering
the start operation.
APIImpact
DocImpact
Partially Implements: bp share-server-migration
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Co-Authored-By: Carlos Eduardo <ces.eduardo98@gmail.com>
Co-Authored-By: Felipe Rodrigues <felipefuty01@gmail.com>
Change-Id: Ic0751027d2c3f1ef7ab0f7836baff3070a230cfd
Signed-off-by: Douglas Viroel <viroel@gmail.com>
This patch enables the creation of a share from snapshot
specifying another pool or backend. In the scheduler, a
new filter and weigher were implemented in order to consider
this operation if the backend supports it. Also, a new
field called 'progress' was added in the share and share
instance. The 'progress' field indicates the status
of the operation create share from snapshot (in percentage).
Finally, a new periodic task was added in order to constantly
check the share status.
Partially-implements: bp create-share-from-snapshot-in-another-pool-or-backend
DOCImpact
Change-Id: Iab13a0961eb4a387a502246e5d4b79bc9046e04b
Co-authored-by: carloss <ces.eduardo98@gmail.com>
Co-authored-by: dviroel <viroel@gmail.com>
This patch adds the possibility to create share networks with
multiple subnets in Manila. It also updates the share server api
to receive "share_network_subnet_id" instead of "share_network_id".
Each share network subnet must be associated with only one
availability zone. Each share network must have a single default
share network subnet.
DocImpact
APIImpact
Depends-On: I13bb48e7c03e16c26946ccf9d48e80592391a3d1
Partially-implements: bp share-network-multiple-subnets
Change-Id: Id8814a8b26c9b9dcb1fe71d0d7e9b79e8b8a9210
Closes-Bug: #1588144
Co-Authored-By: lseki <luciomitsuru.seki@fit-tecnologia.org.br>
Co-Authored-By: dviroel <viroel@gmail.com>
This patch adds Manage/Unmanage of share servers in
Manila. It also updates the Manage Share API to accept
a "share_server_id" parameter, and updates Unmanage
of Share and Snapshots API to allow unmanaging of
shares and snapshots in DHSS=True.
Managed share servers are not deleted automatically
by manila, and if a single share is unmanaged in
DHSS=True, the respective share server will not be
deleted automatically as well.
Managing share servers require that the driver
implements 2 functions:
- get_share_server_network_info: obtain IPs from
share server.
- manage_server: perform required operations to
manage and return dict of backend_details.
Unmanaging share servers require that the driver
overrides unmanage_server function.
The IPs obtained from the backend are validated
by the Network plugin, so ports with the exact
IPs must exist in the subnet and admin subnet
associated with the share network specified
when managing the share server (unless
StandaloneNetworkPlugin is used).
It is recommended to rename the backend resource
if possible when managing the share server, to
prevent issues with re-managing a share server
that has already been managed.
This patch bumps the API microversion to 2.49.
APIImpact
DocImpact
Depends-On: I17c74b2aa242918188eeff368232c762a4b31093
Partially-implements: bp manage-unmanage-with-share-servers
Change-Id: I108961e7436ba13550ef2b8f02079c6e599a6166
- Add ignores to pylint false positives in the
test modules.
- Remove unnecessary fake data
- Fix wrong mock methods used in tests
Change-Id: I64ffff15cc546c67e7e545b1da7ec0efa002bdc5
The index on the foreign key is created on InnoDB mysql implicitly.
This makes it explicit.
Closes-Bug: #1776495
Change-Id: I6ff5cee86997d0d756b4852fa4d0e7bcfcc9c1cf
The downgrade incorrectly modifies deleted
share access rules.
Also add a missing unit test for this migration.
Co-Authored-By: Goutham Pacha Ravi <gouthampravi@gmail.com>
Change-Id: If3700e7a18cbd8ae496b8c503c95cf6723485679
Closes-Bug: #1717392
In gate py27 and py35 unit tests are failing on
alembic migration tests with exceptions like this [1].
Apparently sqlAlchemy 1.2.1 enforces type checks for
database fields that 1.1.15 just coerced.
Change strings to booleans and ints in the test
code so that fields in the "fakes" match the
fields in the sqlalchemy models.
Co-Authored-By: Tom Barron <tpb@dyncloud.net>
[1] http://logs.openstack.org/68/539468/1/check/openstack-tox-py27/db4a572/job-output.txt.gz#_2018-01-31_09_26_49_316004
Change-Id: Iadffb8a06b4d948d77fc00b85118e538b0a19b24
This driver interface is currently being used wrong and needs a rewrite.
This patch adds the ability to solve the potential problem of slow start
up, and deal with non-user-initiated state changes to shares.
Implement BP ensure-share
Change-Id: I704591446cd5f6ad2293c5a18eccf5412d488baf
It should be possible to use share type quotas in each project/tenant.
But we get error trying to set share types quota in second+ project
for the same share type. It happens so due to the bug in DB schema,
where we don't have 'project_id' in unique constraint.
So, add new DB migration that fixes unique constraint and cover it with
DB and tempest tests.
Change-Id: I1405ced4e12b40904b7227c9f6285e2775005f8a
Closes-Bug: #1722707
With this feature it will be possible to set quotas per share type
for all existing quota resources. It is useful for deployments with
multiple backends that are accessible via different share types.
Also, fix one of existing DB migrations that hangs on PostgreSQL.
APIImpact
DocImpact
Implements blueprint support-quotas-per-share-type
Change-Id: I8472418c2eb363cf5a76c672c7fdea72f21e4f63
For quite some time, OpenStack services have wanted to be able to send
messages to API end users (by user I do not mean the operator, but the
user that is interacting with the client).
This patch implements basic user messages with the following APIs.
GET /messages
GET /messages/<message_id>
DELETE /messages/<message_id>
Implements the basic /messages resource and tempest tests
The patch is aligned with related cinder patch where possible:
I8a635a07ed6ff93ccb71df8c404c927d1ecef005
DocImpact
APIImpact
Needed-By: I5ffb840a271c518f62ee1accfd8e20a97f45594d
Needed-By: I9ce096eebda3249687268e361b7141dea4032b57
Needed-By: Ic7d25a144905a39c56ababe8bd666b01bc0d0aef
Partially-implements: blueprint user-messages
Co-Authored-By: Jan Provaznik <jprovazn@redhat.com>
Change-Id: Ia0cc524e0bfb2ca5e495e575e17e9911c746690b
removing redundant column "share_id" and making it proxy to
"share_instance_id" as it was prior to merge of [1].
It is redundant because there is only one place, in manila, where it is
used, but used without real need.
[1] I81ba26437554d02a79e4b536d781617f46ce2f07
Change-Id: I4ee14cf89b8882941974741da311f4556e604116
Closes-Bug: #1697581
Because the former was implemented just because the latter
was absent yet at that moment. Main reason is that they serve
same goals and should be consistent to each other. And it is
much easier to support one single table instead of two separate
tables syncing them each time we update one of them.
Change-Id: I81ba26437554d02a79e4b536d781617f46ce2f07
Partially-Implements bp manila-share-groups
'Share groups' feature was not finished completely in Ocata timeframe.
There was absent possibility to specify 'availability zone' for
creation of a share group. So, implement it.
The fact of its implementation fixes mentioned bug below.
Closes-Bug: #1662944
Partially-Implements BP manila-share-groups
Change-Id: I000adeb53fe8435465cbedc3c539e6aaae6503c5
Currently we would have uncommit reservations in database if
the commit operation failed. This could obstruct creating new
shares or others which would consume quota. This patch adds
a perodic task to clean this expired reservations.
Also fix bug in db.reservation_expire method(We dropped
attribute 'usage' from Reservation, but it's still used
in this method).
Closes-Bug: #1669449
Change-Id: I3dc0973ebac5eb33832e242c72059be1eb954369
It will be required for support of 'consistent share group snapshots'
feature. It is added in Ocata to ease future possible backports.
Make it be 'Enum' with possible values 'pool', 'host' allowing to be
nullable.
Change-Id: I2e5984bc2fc4a487793ad8254c5dbfb0d6e33f26
Partial-Bug: #1661266
Partial-Bug: #1661268
Default approach uses common driver interface of single snapshot
creation. So, emulate it correctly providing all required keys.
Also add 'provider_location' attribute to share group snapshot
members DB model, because it is required for some drivers and
was absent.
Change-Id: If120d85ef3dd3ba90e2dc12a5b81b69feecb31ea
Closes-Bug: #1660321
Remove the experimental consistency group APIs and
replace them with the experimental Share Group APIs.
DocImpact
APIImpact
Partially-implements-blueprint: manila-share-groups
Change-Id: I79a80a62ae4e0015d6161edc2b93fd1f9ba69537
- Add Database migration to introduce the column on the
share instances model.
- Set the field to True if creating read-only secondary
replicas, unset while promoting them.
- Set the field to True if drivers don't support writable access
to migrating shares, or if using host assisted migration.
Unset if migration fails, or is canceled.
- Expose the field via share-instances and share-replicas
APIs to administrators.
Supporting read only-access rules is part of the minimum
driver requirements in manila.
APIImpact
DocImpact
Implements: bp fix-and-improve-access-rules
Co-Authored-By: Rodrigo Barbieri <rodrigo.barbieri@fit-tecnologia.org.br>
Change-Id: Ie8425f36f02cbcede0aaa9f3fe1f5f3cf23df8b8
- Pull up policy check to beginning of the APIs.
- Avoid making access rules changes when one or
more instances of the share are in an invalid state.
- Add back the per rule share instance access status.
This restoration provides better visibility for which
rules were applied successfully.
- Remove 'updating' and 'updating_multiple' as valid
states for the share instance access rules status.
- Deprecate the access rule state 'new' in favor of
'queued_to_apply' and the share instance access rules
status 'out_of_sync' in favor of 'syncing'.
In a new API micro-version:
- Allow access rule changes irrespective of the share's
access_rules_status.
- Expose new access rule states and share's
access_rules_status values.
Access rules for each share instance now transition
from 'queued_to_apply' to 'applying' to 'active' or 'error';
and from 'active', 'queued_to_apply', 'applying' or 'error'
to 'queued_to_deny' to 'denying' to 'deleted'.
APIImpact
DocImpact
Partially-implements: bp fix-and-improve-access-rules
Co-Authored-By: Mike Rooney <rooneym@netapp.com>
Change-Id: Ic25e63215b5ba723cbc8cab7c51789c698e76f28
This commit adds the ability for Manila to revert a
share to the latest available snapshot.
The feature is implemented in the LVM driver, for
testing purposes.
APIImpact
DocImpact
Co-Authored-By: Ben Swartzlander <ben@swartzlander.org>
Co-Authored-By: Andrew Kerr <andrew.kerr@netapp.com>
Implements: blueprint manila-share-revert-to-snapshot
Change-Id: Id497e13070e0003db2db951526a52de6c2182cca
Nova network was deprecated in Newton and is no longer supported for
regular deployments in Ocata [1].
Complete the removal of nova network support from manila begun in [2]
by removing support for ''nova_net_id`` in the share networks API,
by removing the corresponding column from the share networks DB model,
and by removing the nova network plugins themselves. Unit tests for
share network api views were refactored to remove redundancy while
extending coverage to the new microversion introduced with this patch
and maintaining full coverage of earlier microversions.
APIImpact
DocImpact
UpgradeImpact
Partially-implements: bp remove-nova-net-plugin
[1] http://docs.openstack.org/releasenotes/nova/unreleased.html
[2] I846b760fa7c01f7f86768399a2bfad9ced7e57cd
Change-Id: I8b9a559fbea61979f01737ed1dc272276c4f1269
Fix misspelling in 'ShareNetwoksFieldLengthChecks' class name.
Indent CreateFromSnapshotExtraSpecAndShareColumn so that it
is a standalone class rather than a subclass of
MoveShareTypeIdToInstancesCheck.
TrivialFix
Change-Id: I169bb22c1dc3078e187b94f9f8df69e9615daec5
The snapshot_support extra spec has always meant two
things: a driver can take snapshots and create shares
from snapshots. As we add alternate snapshot semantics,
it is likely that some drivers will want to support
snapshots and some of the new semantics while being
unable to create new shares from snapshots.
This work adds a new extra spec,
create_share_from_snapshot_support, that removes the
overloading on snapshot_support. It also makes the
existing snapshot_support extra spec optional,
allowing admins to create types without setting
snapshot_support; shares created with such types
will not support snapshots.
APIImpact
DocImpact
Co-Authored-By: Goutham Pacha Ravi <gouthamr@netapp.com>
Implements: blueprint add-create-share-from-snapshot-extra-spec
Change-Id: Ib0ad5fbfdf6297665c208149b08c8d21b3c232be
The `deleted` column in volume_type_extra_specs was as a boolean;
The same column in share_type_extra_specs is an integer. However,
the code migrating from volume_types to share_types
assumes them to be vice-versa. This breaks downgrading from that
migration.
Further, the `id` column in share_type_extra_specs is a primary key
field of Integer datatype. Such fields have auto-increment turned on
by default; we shouldn't be copying the `id` from
volume_type_extra_specs when we copy the data.
Fix these bugs and add a unit test.
Change-Id: Ic32a8a17b1b6a773de6dcf7792d9b37f6aed4a3c
Needed-by: Ib0ad5fbfdf6297665c208149b08c8d21b3c232be
Closes-Bug: 1643535
This patch adds a 'new_share_type_id' parameter to Share Migration,
where the destination share can be provisioned under a different
share type of choice.
Host-assisted migration handles it by creating a totally new share,
as before.
Driver-assisted migration handles by creating the destination
instance model with the new share type, the driver is responsible
for making the necessary changes to satisfy the provided model.
In order to accomplish this, a database change was required,
transferring the 'share_type_id' field from the 'shares' table
to the 'share_instances' table.
APIImpact
Partially implements: blueprint newton-migration-improvements
Change-Id: I3200eaaa5b66d9b8ce1cbd16c1658db8516c70fb
For backends with internal authentication system,
e.g. Ceph, that return ``access_key`` (credential) for
client identities that are granted share access:
* Retrieve ``access_key`` as return value of driver's
update_access()
* Store ``access_key`` in ShareAccessMapping model
* Expose it in access_list API
APIImpact
DocImpact
Partially implements bp auth-access-keys
Co-Authored-By: John Spray <jspray@redhat.com>
Change-Id: I486064f117cf3001dba7735ca92a7d89aee3ce5b
MTU value can be different for each neutron network.
E.g. for high-performance use-cases it's very important to also
support MTU's with jumbo frames.
This patch exposes this information to the drivers. Each driver
should setup its resources accordingly.
The tempest test actually just covers the api change. Better coverage will
be added when the container driver lands.
APIImpact
DocImpact
Change-Id: I9b4efae620ec9f6790547c8fffc58872d43277f5
Implements: bp add-network-mtu
Related-Bug: #1612528
The "host" field in driver private data related db and api is not
used and no longer needed. This patches removes it.
Change-Id: Ifd0b290b2992f6b0792250b86466d9329dfd08a7
Partially-Implements: blueprint driver-private-storage
Get gateway information from network plugin and put it into network_info.
It is required by EMC Unity storage to create a interface.
APIImpact
DocImpact
Change-Id: I8614b8686af7fa5764b49e8e3cb4a4855dc3a5f4
Implements: blueprint add-gateway-info
In case keystone is used with LDAP the user_id/project_id can be more
than 60 characters. This will cause the issue that users cannot create
any share-network or security_service.
Change-Id: I2e2ccce32bf31850c9ffd74d9612cf5237d782fe
Closes-bug: #1594824
Add APIs to support manage and unmanage share snapshots.
Also add support in the Generic driver.
This only supports for DHSS=False driver mode.
Add provider_location column to the share_snapshots table
to save data used to identify the snapshot on the backend.
Also need to bump microversion.
APIImpact
DocImpact
Change-Id: I87a066173c85d969607d132accd9f0e9bd49c235
Implements: blueprint manage-unmanage-snapshot
Mike Bayer