Added new column named 'backup_type' in 'share_backups' table
and changes the share common api libs to support the dhss_true
configuration for share backup creation
Partially-implements: bp/share-backup
Change-Id: Ifb88ec096674ea8bc010c1c3f6dea1b51be3beaa
Export locations are usually too difficult to memo
rize.Currently, there is no way to determine the
export location before the share is created, so
users wait until the share creation request gets
completed, and then they check the export
locations to mount the share. The generated
export locations are often not human readable
and it is hard to memorize and control them.
Implements: bp/human-readable-export-locations
Change-Id: I72ac7e24ddd4330d76cafd5e7f78bac2b0174883
Access rules rules allow API will now take three additional
parameters:
- lock_visibility: when True, only services, administrators and
the same user will be able to see the content of ``access_to`` and
access_key.
- lock_deletion: when True, the access rule will be locked for
deletion. Only services, administrators or the user that placed
the lock will be able to drop the access rule.
- lock_reason: a reason for the lock. This parameter should only
be provided in the presence of at least one of the former
parameters.
In order to delete an access rule that is currently locked, the
requester will need to specify ``unrestrict=True`` in the request.
In case a service placed the restrictions, only the own service or
the system administrator will be able to release it.
This change also implements filters to the access list API. It is
now possible to filter access rules based on `access_to`,
`access_type`, `access_level` and `access_key`.
DocImpact
Change-Id: Iea422c9d6bc99a81cd88c5f4b7055d6a1cf97fdc
Add CRUD APIs for resource locks with support
for preventing deletion of shares (applies to
soft-deletions and unmanage operations as well).
Change-Id: I146bc09e4e8a39797e22458ff6860346e11e592e
Implements: bp/allow-locking-shares-against-deletion
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
Add share backup feature in Data Copy Service and
Share Service. It will allow the user to create, restore
and delete backups as well as listing backups and showing
the details of a specific backup.
APIImpact
DOCImpact
Change-Id: I7d10cf47864cd21932315375d84dc728ff738f23
Implement: blueprint share-backup
Instead of
`<manila.db.sqlalchemy.models.ShareServer object at 0x7f603645bf40>`
we now get
```
ShareServer(created_at=datetime.datetime(2023, 4, 18, 15, 44, 43, 741302),
updated_at=datetime.datetime(2023, 4, 19, 10, 18, 59, 614018),
deleted_at=datetime.datetime(2023, 4, 19, 10, 18, 59, 613577),
id='3363dcd4-fcdd-4cfa-b02d-5899fba3d413',
deleted='3363dcd4-fcdd-4cfa-b02d-5899fba3d413',
share_network_subnet_id='8d4365dc-0869-4896-8049-6455f84ad08f', ...)
```
e.g. in error messages of failed purges or in interactive shell sessions.
Change-Id: I5e6513e7ca7b4a9d4d7261f873cbf3f5bf3c83ab
When the user logged is not an admin, the project find for the
policies using some information like project_id. Since the
Subnet model doesn't have project_id, the new property refering
the Share Network project_id now is used to guarantee access
Closes-bug: #2011869
Change-Id: I58331fa6906c855804abb94fad85a8370625c886
Share network subnets were not checking for the existence of share
groups while processing a delete request. Then, if a share network
had a share server that was tied to a share group, it would end up
deleting the share server, leaving the share group orphan. When we
triggered the deletion of the share group, it would fail, as the
share server was not known to Manila anymore.
Fix that issue by adding an extra validation step during the share
network subnet delete API.
Closes-Bug: #2004212
Depends-On: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/875981
Change-Id: I563bf925523fa44689c83f432ce5a460276afef7
When manila services are stopped or restarted via stop(), the DB
entries are not deleted, they are destroyed only in kill() method. In
cluster deployments, where multiple instances of manila services are
deployed via PODs, unique hostname is derived from node name. However
if pods are deployed again and launched on new hosts/nodes, the old
entries of manila service remains as it is.
Fix it by adding 'state' column in 'services' table and introducing
per service cleanup function. On service stop, state is changed to
'stopped' and cleanup function will delete 'stopped' services unless
they are 'up' again before cleanup periodic interval.
Closes-bug: #1990839
Change-Id: I8b71c4c27ff8fcb25616a95a5ed8362a7f4ffc61
This change adds metadata controller for Share Network Subnets
resource. Bumps microversion to 2.78.
The subnet metadata is passed down to the driver together
with the network_info object.
APIImpact
Partially-implements: bp/metadata-for-share-resources
Change-Id: I8d5a03eb127941a84eea5e6e9bdf76b3489f17a8
user can create a transfer for a share. will return transfer id
and auth_key. another user can use transfer_id and auth_key to
accept this share. salt of transfer and auth_key compute crypt_hash
by sha1. then compare with crypt_hash in db.
APIImpact
DocImpact
Partially-Implements: blueprint transfer-share-between-project
Change-Id: I8facf9112a6b09e6b7aed9956c0a87fb5f1fc31f
Allows to configure optional field 'default_ad_site' from version 2.76.
Restrict to make sure either server or 'default_at_site' provided, but
not both.
APIImpact
Relates-bug: #1988146
Change-Id: I8e21e9170eace134a51efed84de1ccc58eb7eaaa
Migrate service-related APIs from the legacy enginefacade to the modern
context-based enginefacade. This one is a little tricky as it highlights
two issues.
* The models in 'manila.db.sqlalchemy.models' do not match those created
by the migrations. This causes us issues since we apply the former
rather than running migrations when creating our database via the
DatabaseFixture, which means we're not actually testing against what
our customers see. One gap is closed here in order to get newly added
test working as expected but there's a larger issue that needs to be
addressed at a later date.
* We are using a pattern of "try to create and retrieve on failure"
rather than "try to get and create on failure" when determining
whether to create availability zone records. This works where each
operation results in a new transaction but does not when the
transaction is shared. We switch this.
With these two issues addressed, the rest of the migration is relatively
straightforward.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Icfe4b225e216dfda46366af1eb3820b193befa0f
Manila can now configure a share network with multiple subnets
in an availability zone. Also, it can add a new subnet for an
availability that has share servers, which will triger an update
share server allocations.
Changes:
- API:
- Bump version to 2.70.
- setup share network with multiple subents per az.
- Block manage server with multiple subnets.
- Allow add subnet for in-use share servers.
- `share_network_subnet_id` is dropped from ShareServer view
- `share_network_subnet_ids` is added in ShareServer view
- `network_allocation_update_support` is added to ShareServer and
ShareNetwork views.
- Add a check operation for share network subnet create.
- DB:
- Remove `share_network_subnet_id` from share_servers.
- Create mapping table `share_server_share_network_subnet_mappings`.
- Fix queries with new db design.
- Add migration downgrade and upgrade alembic.
- Add `share_network_subnet_id` to the NetworkAllocations.
- Scheduler:
- Change `AvailabilityZoneFilter` to take in account if the
host supports the allocation required by the setup request.
- Manager:
- Bump RPC API version.
- `_setup_server` allocating multiple subnets.
- Modify signature of driver `_setup_server` interface, passing a
list of `network_info` for each subnet.
- Share server DB creation to inform a list of subnets and
create with `network_allocation_update_support`.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations`.
- Drivers:
- For legacy compatibility, all drivers implementing `_setup_server`
consume the first element of the `network_ino`.
- Dummy Driver:
- Implement `_setup_server` with new signature as multiple subnet.
- Modify the `backend_details` to save allocations for all subnets.
- Report update allocation and share server multiple subnet support.
- Implement `check_update_share_server_network_allocations` and
`update_share_server_network_allocations` interfaces.
Signed-off-by: Felipe Rodrigues <felipefuty01@gmail.com>
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Co-Authored-By: Fábio Oliveira <fabioaurelio1269@gmail.com>
Co-Authored-By: Nahim Alves de Souza <nahimsouza@outlook.com>
Co-Authored-By: Caique Mello <caique_mellosbo@hotmail.com>
DocImpact
APIImpact
Partially-Implements: blueprint multiple-share-network-subnets
Change-Id: I7de9de4ae509182e9494bba604979cce03acceec
Add support share Recycle Bin, the end user can soft delete
share to Recycle Bin, and can restore the share within 7 days,
otherwise the share will be deleted automatically.
DocImpact
APIImpact
Partially-Implements: blueprint manila-share-support-recycle-bin
Change-Id: Ic838eec5fea890be6513514053329b1d2d86b3ba
This is a follow up patch for [1] which removes some unused model
properties and add more unit tests to increase coverage.
[1] https://review.opendev.org/c/openstack/manila/+/774728
Change-Id: I10af132203cb07cea62839014925c7e4c99499e4
Signed-off-by: Douglas Viroel <viroel@gmail.com>
This patch implements the update of security service's association
with in-use share networks. The following changes were added:
- New share network APIs: `share_network_security_service_update`
and `share_network_reset_state`.
- A new `status` attribute was added to share network model to
identify when it's in a modification state, called 'network_change'.
Other supported status that were added: 'active' and 'error'.
- New 'security_service_update_support' property was added to both
share server and share network models, to identify when this resources
are able to process security service update for in-use share networks.
- New driver interface was added to support update of security service's
configuration of a given share server.
DocImpact
APIImpact
Partially Implements: bp add-security-service-in-use-share-networks
Co-Authored-By: Carlos Eduardo <ces.eduardo98@gmail.com>
Co-Authored-By: Douglas Viroel <viroel@gmail.com>
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Change-Id: I129a794dfd2d179fa2b9a2fed050459d6f00b0de
This patch adds support for migration of share servers. This
migration is performed using a two-phase approach. Administrators
are now able to request the migration of a share server within and
across backends, with the possibility of chooosing a different share
network for the destination share server.
- A new field called `task_state` was added to the share server
model in order to help the administrator to track the share server
migration steps. A new field called `source_share_server_id` was
added to link destination and source share servers.
- A new periodic task was added to track migration of share servers
and its resources.
- Two new states were added: `server_migrating` and
`server_migrating_to` to represent that share migration is in
progress.
- When performing the server migration, manila will not go to the
scheduler, instead it will provide a request spec to drivers
during migration check driver call. It'll be up to the driver
validate if there is free space to handle the share server.
- A new API called `share-server-migration-check' was added to
check the feasibility of a migration, before actually triggering
the start operation.
APIImpact
DocImpact
Partially Implements: bp share-server-migration
Co-Authored-By: Andre Beltrami <debeltrami@gmail.com>
Co-Authored-By: Carlos Eduardo <ces.eduardo98@gmail.com>
Co-Authored-By: Felipe Rodrigues <felipefuty01@gmail.com>
Change-Id: Ic0751027d2c3f1ef7ab0f7836baff3070a230cfd
Signed-off-by: Douglas Viroel <viroel@gmail.com>
Never used cinder legacy.
It has been removed there as well in https://review.opendev.org/#/c/58736
Change-Id: Id8889c54fcfff5480e7f4654f2aff90aa818a1bb
This patch enables the creation of a share from snapshot
specifying another pool or backend. In the scheduler, a
new filter and weigher were implemented in order to consider
this operation if the backend supports it. Also, a new
field called 'progress' was added in the share and share
instance. The 'progress' field indicates the status
of the operation create share from snapshot (in percentage).
Finally, a new periodic task was added in order to constantly
check the share status.
Partially-implements: bp create-share-from-snapshot-in-another-pool-or-backend
DOCImpact
Change-Id: Iab13a0961eb4a387a502246e5d4b79bc9046e04b
Co-authored-by: carloss <ces.eduardo98@gmail.com>
Co-authored-by: dviroel <viroel@gmail.com>
Whenever we use share network subnets, we'd
always need the associated share networks; so
'joined' loading them makes sense rather than
lazy loading. Same is true for Share Group
Snapshots, since share group attributes are
relevant in contexts that snapshots are
referenced.
Fix the ORM relationship loading technique
and queries appropriately. Also convert share
network name into a property of the share
network subnet class.
Co-Authored-By: Carlos Eduardo <ces.eduardo98@gmail.com>
Closes-Bug: #1862833
Change-Id: I8c29760479f9c874d618e2c15ebab75bffeb3cbb
lazy='immediate' leads to each relationship being collected when
it is accessed. This results in at least three extra queries when
we query for all share details. lazy='subquery' collects all data
when the query is executed. In this commit we only changed code for
improving the share list with details ("manila list") speed.
Change-Id: Ia61b108ece0817069737980a614cc6c15c1a3507
Closes-Bug: #1859785
Co-authored-by: Johannes Kulik <johannes.kulik@sap.com>
Co-authored-by: Maurice Escher <maurice.escher@sap.com>
This patch adds the possibility to create share networks with
multiple subnets in Manila. It also updates the share server api
to receive "share_network_subnet_id" instead of "share_network_id".
Each share network subnet must be associated with only one
availability zone. Each share network must have a single default
share network subnet.
DocImpact
APIImpact
Depends-On: I13bb48e7c03e16c26946ccf9d48e80592391a3d1
Partially-implements: bp share-network-multiple-subnets
Change-Id: Id8814a8b26c9b9dcb1fe71d0d7e9b79e8b8a9210
Closes-Bug: #1588144
Co-Authored-By: lseki <luciomitsuru.seki@fit-tecnologia.org.br>
Co-Authored-By: dviroel <viroel@gmail.com>
This patch adds Manage/Unmanage of share servers in
Manila. It also updates the Manage Share API to accept
a "share_server_id" parameter, and updates Unmanage
of Share and Snapshots API to allow unmanaging of
shares and snapshots in DHSS=True.
Managed share servers are not deleted automatically
by manila, and if a single share is unmanaged in
DHSS=True, the respective share server will not be
deleted automatically as well.
Managing share servers require that the driver
implements 2 functions:
- get_share_server_network_info: obtain IPs from
share server.
- manage_server: perform required operations to
manage and return dict of backend_details.
Unmanaging share servers require that the driver
overrides unmanage_server function.
The IPs obtained from the backend are validated
by the Network plugin, so ports with the exact
IPs must exist in the subnet and admin subnet
associated with the share network specified
when managing the share server (unless
StandaloneNetworkPlugin is used).
It is recommended to rename the backend resource
if possible when managing the share server, to
prevent issues with re-managing a share server
that has already been managed.
This patch bumps the API microversion to 2.49.
APIImpact
DocImpact
Depends-On: I17c74b2aa242918188eeff368232c762a4b31093
Partially-implements: bp manage-unmanage-with-share-servers
Change-Id: I108961e7436ba13550ef2b8f02079c6e599a6166
This patch fixes the size value not being present in share
snapshot instances, which causes drivers to not being able to access
this property unless they read the size from the snapshot model.
Closes-bug: #1815587
Change-Id: I730629ea460c316a02f8dffb4a55eea04ad619c9
Pylint does not play very well with dynamic object
manipulation in python This creates a lot of
false-positives in the code-base which affects
contributors looking for genuine failures.
So, this change
- adds pylint ignore statements where appropriate
to disable testing these lines of code and failing.
- replaces all the pylint error codes (they are
hard to remember/relate to) with error names
which are easier to understand when reading the
code.
- initializes sqlalchemy model objects as dictionaries
which is a valid representation over None.
- removes ignore directives on six.moves which
is globally ignored in our pylintrc.
- adds alembic.op to the ignored
modules list since they are not supported by
pylint and have known issues.
This patch is the beginning of a series of
commits to use pylint in a sane way on manila code.
Change-Id: I44616821c5311d6f14986697efbbe5624de364a5
Users of replicated shares expect to see primary
export locations when viewing information regarding
the share. Because we collate exports of all replicas
within the export locations APIs, it becomes hard for
users to discern which exports belong to the primary
share. For secondary replicas, users would also need
additional information (availability zone, state of the
replication) to work with.
Introduce micro-version 2.47 from which the export locations
API (GET /v2/{tenant_id}/shares/{share_id}/export_locations)
no longer provides export locations of non-active share
replicas. A new API has been introduced to provide export
location details for share replicas, both active and non-active.
(GET /v2/{tenant_id}/share-replicas/{share_replica_id}/export-locations)
The new API provides the replica's state and availability zone
in addition to the export location information.
APIImpact
Implements: bp export-locations-az
Change-Id: I0a1d9dd00b4c13ac01988e30ca2b7d7ce4a747d1
When share instances were introduced a number of fields
were moved in the database model to the underlying share instance
and proxied back to the share in order to maintain compatability
with APIs that assume these to belong to the share. A deprecation
warning was added for these. The warning spams the API log and
confuses end users because it sounds like they or their client
should make a change to get rid of them and that cannot actually
be done. The API would have to change to expose share instances
to end users rather than just administrators and that is not
something that we plan to do.
Remove these confusing warnings.
Closes-Bug: #1776947
Change-Id: I95e99b26f416bbf25cfd1c8f7295f3ba6220efb3
Allows to configure in which organizational unit the share ends up
APIImpact
Change-Id: I70fd4e0ea7898e85dff59a927b3cb086d5e408ac
Needed-By: I77d87c8e6f98054f2f02bc4202c14314ff321964
Partial-Bug: #1696669
This driver interface is currently being used wrong and needs a rewrite.
This patch adds the ability to solve the potential problem of slow start
up, and deal with non-user-initiated state changes to shares.
Implement BP ensure-share
Change-Id: I704591446cd5f6ad2293c5a18eccf5412d488baf
With this feature it will be possible to set quotas per share type
for all existing quota resources. It is useful for deployments with
multiple backends that are accessible via different share types.
Also, fix one of existing DB migrations that hangs on PostgreSQL.
APIImpact
DocImpact
Implements blueprint support-quotas-per-share-type
Change-Id: I8472418c2eb363cf5a76c672c7fdea72f21e4f63
For quite some time, OpenStack services have wanted to be able to send
messages to API end users (by user I do not mean the operator, but the
user that is interacting with the client).
This patch implements basic user messages with the following APIs.
GET /messages
GET /messages/<message_id>
DELETE /messages/<message_id>
Implements the basic /messages resource and tempest tests
The patch is aligned with related cinder patch where possible:
I8a635a07ed6ff93ccb71df8c404c927d1ecef005
DocImpact
APIImpact
Needed-By: I5ffb840a271c518f62ee1accfd8e20a97f45594d
Needed-By: I9ce096eebda3249687268e361b7141dea4032b57
Needed-By: Ic7d25a144905a39c56ababe8bd666b01bc0d0aef
Partially-implements: blueprint user-messages
Co-Authored-By: Jan Provaznik <jprovazn@redhat.com>
Change-Id: Ia0cc524e0bfb2ca5e495e575e17e9911c746690b
removing redundant column "share_id" and making it proxy to
"share_instance_id" as it was prior to merge of [1].
It is redundant because there is only one place, in manila, where it is
used, but used without real need.
[1] I81ba26437554d02a79e4b536d781617f46ce2f07
Change-Id: I4ee14cf89b8882941974741da311f4556e604116
Closes-Bug: #1697581
Because the former was implemented just because the latter
was absent yet at that moment. Main reason is that they serve
same goals and should be consistent to each other. And it is
much easier to support one single table instead of two separate
tables syncing them each time we update one of them.
Change-Id: I81ba26437554d02a79e4b536d781617f46ce2f07
Partially-Implements bp manila-share-groups
'Share groups' feature was not finished completely in Ocata timeframe.
There was absent possibility to specify 'availability zone' for
creation of a share group. So, implement it.
The fact of its implementation fixes mentioned bug below.
Closes-Bug: #1662944
Partially-Implements BP manila-share-groups
Change-Id: I000adeb53fe8435465cbedc3c539e6aaae6503c5
Currently we would have uncommit reservations in database if
the commit operation failed. This could obstruct creating new
shares or others which would consume quota. This patch adds
a perodic task to clean this expired reservations.
Also fix bug in db.reservation_expire method(We dropped
attribute 'usage' from Reservation, but it's still used
in this method).
Closes-Bug: #1669449
Change-Id: I3dc0973ebac5eb33832e242c72059be1eb954369
It will be required for support of 'consistent share group snapshots'
feature. It is added in Ocata to ease future possible backports.
Make it be 'Enum' with possible values 'pool', 'host' allowing to be
nullable.
Change-Id: I2e5984bc2fc4a487793ad8254c5dbfb0d6e33f26
Partial-Bug: #1661266
Partial-Bug: #1661268
Default approach uses common driver interface of single snapshot
creation. So, emulate it correctly providing all required keys.
Also add 'provider_location' attribute to share group snapshot
members DB model, because it is required for some drivers and
was absent.
Change-Id: If120d85ef3dd3ba90e2dc12a5b81b69feecb31ea
Closes-Bug: #1660321
Snapshot export locations should only be created for shares
with property mount_snapshot_support=True. Otherwise, they
should not be created, even if returned by drivers.
Change-Id: Ie0ce3c4f2e7e29bf82f5c09a80a3fb132b0481a8
Closes-bug: #1660686
Zuul