Resolve the following RemovedIn20Warning warning:
Using strings to indicate relationship names in Query.join() is
deprecated and will be removed in SQLAlchemy 2.0. Please use the
class-bound attribute directly.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I155b4ce4b605720c8335d465124fd32cc973a737
Change Ie8831c04b7b4515deea27b1ceb472d07cda91ca0 replace the only users
of 'autoload' with 'autoload_with'.
Change-Id: I37bbe45d6d9ce41a3c238e33d4469f9a362491ad
after testing using an assert for DML without a transaction,
we can now turn autocommit off. enginefacade should be used
completely now
Change-Id: I2ee07c7e41ea43c2ab24a4a095550dd0b5fe47dd
Migrate service-related APIs from the legacy enginefacade to the modern
context-based enginefacade. This one is a little tricky as it highlights
two issues.
* The models in 'manila.db.sqlalchemy.models' do not match those created
by the migrations. This causes us issues since we apply the former
rather than running migrations when creating our database via the
DatabaseFixture, which means we're not actually testing against what
our customers see. One gap is closed here in order to get newly added
test working as expected but there's a larger issue that needs to be
addressed at a later date.
* We are using a pattern of "try to create and retrieve on failure"
rather than "try to get and create on failure" when determining
whether to create availability zone records. This works where each
operation results in a new transaction but does not when the
transaction is shared. We switch this.
With these two issues addressed, the rest of the migration is relatively
straightforward.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Icfe4b225e216dfda46366af1eb3820b193befa0f
Well, sort of. We enable them but immediately filter out the ones we're
actually seeing, the rationale being that we can address these in a
piecemeal fashion without the risk of introducing new issues.
There's a lot more to be done here. However, the work done in oslo.db
[1] should provide a guide for how to resolve the outstanding issues.
[1] https://review.opendev.org/q/topic:"sqlalchemy-20"+project:openstack/oslo.db
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: If3a9bed39872012f8c64333cf378c434c22a2d15
Copied almost wholesale from Nova. We're going to use this shortly as
part of the SQLAlchemy 2.0 preparation.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ia7641c84464047dd9ab281390195674febb7e2a8
Make it a little more obvious what this is.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I5297a81d319ae850f0f4d3b63ca0cba7ac2b2a34
Recently, we updated the default check strings
for all API RBAC policies in manila. These policy
changes cause a lot of deprecation warnings in
the logs. In case a deployer never modified their
defaults and accepted service defaults, these
warnings aren't really helpful. If the deployer
did modify the default, these warnings are not
emitted. So we're in a bit of a pickle whether
these are helpful.
Other services [1][2][3] have made the decision
that these deprecated default check string
warnings aren't really helpful and actionable.
Further, any kind of oslo_policy deprecation
warnings aren't helpful during unit tests.
They fill up the logs and cause unnecessary
noise.
[1] openstack/nova: Iaa6baf6877890babb6205bdb3ee2e1a2b28ebd9c
[2] openstack/neutron: Iab3966bad81b469eccf1050f0e0e48b9e2573750
[3] openstack/placement: I2853c7bd7c0afdeeed89ef412fc8830f04381d7b
Change-Id: I08de69312016389f2b4c88f2adbd749dbe4d3261
Signed-off-by: Goutham Pacha Ravi <gouthampravi@gmail.com>
*) Add osprofiler wsgi middleware
This middleware is used for 2 things:
1) It checks that person who want to trace is trusted and knows
secret HMAC key.
2) It start tracing in case of proper trace headers and add
first wsgi trace point, with info about HTTP request
*) Add initialization of osprofiler at start of serivce.
You should use python-manilaclient with this patch:
https://review.opendev.org/#/c/769731
Run any command with --profile SECRET_KEY
$ manila --profile SECRET_KEY create NFS 1 --name <share> \
--description <share_description> --share-network <network> \
--share-type default
# it will print <Trace ID>
Get pretty HTML with traces:
$ osprofiler trace show --html <Trace ID> --connection-string \
<connection_string> --out <output.html>
e.g. --connection-string can be redis://localhost:6379
Note that osprofiler should be run from admin user name & tenant.
Implements: blueprint manila-os-profiler
Change-Id: I3bce1f04d1cfebfacd78ed135a949a068c78987d
Fix:
E731 do not assign a lambda expression, use a def
I just marked the lambdas with noqa.
Fix also other problems found by hacking in files changed.
Change-Id: I4e47670f5a96e61fba617e4cb9478958f7089711
Pylint does not play very well with dynamic object
manipulation in python This creates a lot of
false-positives in the code-base which affects
contributors looking for genuine failures.
So, this change
- adds pylint ignore statements where appropriate
to disable testing these lines of code and failing.
- replaces all the pylint error codes (they are
hard to remember/relate to) with error names
which are easier to understand when reading the
code.
- initializes sqlalchemy model objects as dictionaries
which is a valid representation over None.
- removes ignore directives on six.moves which
is globally ignored in our pylintrc.
- adds alembic.op to the ignored
modules list since they are not supported by
pylint and have known issues.
This patch is the beginning of a series of
commits to use pylint in a sane way on manila code.
Change-Id: I44616821c5311d6f14986697efbbe5624de364a5
The following config option was moved to another place:
- [Default]sqlite_db : Only used for testing so move it to the test
config options. That way, it is not visible in the
manila.conf.sample file
The following config options were removed and are now handled via
oslo.db:
- [Default]sqlite_synchronous : already handled by oslo.db . The new
place is "[database]sqlite_synchronous" but the deprecated
section/names are handled
- [Default]sql_idle_timeout : already handled by oslo.db . The new
place is "[database]connection_recycle_time" but the deprecated
section/names are handled
- [Default]sql_max_retries : already handled by oslo.db . The new
place is "[database]max_retries" but the deprecated
section/names are handled
- [Default]sql_retry_interval : already handled by oslo.db . The new
place is "[database]retry_interval" but the deprecated
section/names are handled
Change-Id: I17e9b69d4ce4b8e624313d05f72d25b3fc2b458c
Lazy translation of messages is not used in OpenStack [1].
This causes problems for commands which use translatable messages.
The lazy translation effort was abandoned several years ago.
[1] http://lists.openstack.org/pipermail/openstack-dev/2018-November/136289.html
Change-Id: I25a9ec0a96b7c57f8a6928c0537bf133f6b6931c
oslo_messaging's rpc_backend setting, which is set by
ConfFixture.transport_driver has been deprecated since Newton. To allow
oslo_messaging to remove it, switch to setting transport_url instead.
Change-Id: I8ac08a2f966356f14f89c109dbfa0a66c258e939
Partial-Bug: #1712399
As part of the manila-telemetry integration work
we are now triggering notifications for several
events. We should provide a way to disable those.
In this patch-set we define a decorator that can be
used to change the behavior of those classes used
to emit notification.
Partially-Implements: bp ceilometer-integration
Change-Id: I806e0133e9fe3ad5cc35ad1e000e5ca61ff3b8ca
oslo.config has deprecated the enforce_type parameter to
set_override() and set its default value to True [1],
so remove this usage from our codebase. Also remove
explict str() cast of mock.sentinel_password in
Windows driver since it should no longer be expected.
[1] Ifa552de0a994e40388cbc9f7dbaa55700ca276b0
Change-Id: Ic02cbb37c6dbf30cacb1e6e290bfb0986f54d5f3
Manila currently uses file locks from oslo_concurrency to
coordinate operations racing with each other to perform a
particular action. In many situations, deployers may need a
distributed lock to a local file lock (or even a file lock living on
a shared file system). This need is accentuated if they were running
Manila services in HA or if they were using Share Replication across
AZs where manila-share services were running off different controllers
that would not be able to share a common oslo_concurrency
file lock or be protected against service/lock management failures.
Integrate Tooz library with helper methods to create a locking
coordinator and allow deployers to make the choice between file
and distributed locks.
Start the manila share service with Tooz backed coordination.
Replace the locks used for Share Replication work-flows in the
share manager to use Tooz based locks.
Co-Authored-By: Goutham Pacha Ravi <gouthampravi@gmail.com>
Co-Authored-By: Szymon Wroblewski <szymon.wroblewski@intel.com>
Co-Authored-By: Tom Barron <tpb@dyncloud.net>
Related-Bug: #1585241
Partially-implements: bp distributed-locking-with-tooz
Change-Id: I710e86bd42034fa3b93b87ff77fa48ada8661168
The snapshot_support extra spec has always meant two
things: a driver can take snapshots and create shares
from snapshots. As we add alternate snapshot semantics,
it is likely that some drivers will want to support
snapshots and some of the new semantics while being
unable to create new shares from snapshots.
This work adds a new extra spec,
create_share_from_snapshot_support, that removes the
overloading on snapshot_support. It also makes the
existing snapshot_support extra spec optional,
allowing admins to create types without setting
snapshot_support; shares created with such types
will not support snapshots.
APIImpact
DocImpact
Co-Authored-By: Goutham Pacha Ravi <gouthamr@netapp.com>
Implements: blueprint add-create-share-from-snapshot-extra-spec
Change-Id: Ib0ad5fbfdf6297665c208149b08c8d21b3c232be
Openstack common has a wrapper for generating uuids.
We should only use that function when generating uuids for consistency.
Change-Id: Ie68850e529eee51c66e8d29a9025e1390576bc1a
Closes-bug: #1082248
Each config option has limitation for type and value. If we use
method CONF.set_override without parameter enforce_type=True, we
may pass wrong type to config option.
This commit makes sure calling method CONF.set_override with
enforce_type=True.
Change-Id: I700dcd21941c6399c51e11bef2282e1e4565e85a
Closes-Bug: #1517839
This fix basically ports nova fix https://review.openstack.org/#/c/136931/
plus additional bug fixing to Manila. It creates a common class to get
used for all clients (neutron, nova, cinder).
Idea is to create an auth session and pass this to the client object
instead of let the client do that. The auth session will be created by
a config group which means the configuration for the clients needed to be
heavily reworked. Patch is also backward compatible with old options but
flag them as deprecated.
DocImpact
Change-Id: Ic211a11308a3295409467efd88bff413482ee58d
Closes-bug: #1555093
Replacing dict.iteritems()/.itervalues() with
six.iteritems(dict)/six.itervalues(dict) was preferred in the past,
but there was a discussion suggesting to avoid six for this[1].
The overhead of creating a temporary list on Python 2 is negligible.
[1]http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Partially-implements blueprint py3-compatibility
Change-Id: Ia2298733188b3d964d43a547504ede2ebeaba9bd
Some of tests use different method of assertTrue(isinstance(A, B)) or
assertEqual(type(A), B). The correct way is to use assertIsInstance(A,
B) provided by testtools.
Change-Id: I9053c9deaa71ee12e10db73b19250b3a32f3d93c
Closes-bug: #1268480
In some Manila source code files, oslo log is imported but not used.
Remove it.
from oslo_log import logging
LOG = logging.getLogger(__name__)
Closes-Bug: #1529254
Change-Id: I06a98b4929e2adb5c43c754799026d4637107bd7
PEP-0274 introduced dict comprehensions to replace dict constructor
with a sequence of length-2 sequences, these are benefits copied
from [1]:
The dictionary constructor approach has two distinct disadvantages
from the proposed syntax though. First, it isn't as legible as a
dict comprehension. Second, it forces the programmer to create an
in-core list object first, which could be expensive.
Manila does not support python 2.6, we can leverage this.
There is deep dive about PEP-0274[2] and basic tests about
performance[3].
Note: This commit doesn't handle dict constructor with kwagrs.
This commit also adds a hacking rule.
[1]http://legacy.python.org/dev/peps/pep-0274/
[2]http://doughellmann.com/2012/11/12/the-performance-impact-of-using
-dict-instead-of-in-cpython-2-7-2.html
[3]http://paste.openstack.org/show/480757/
Change-Id: I87d26a46ef0d494f92afb1d3bebda2797a12413c
Closes-Bug: #1524771
When the Manila share service starts, it instantiates
the Neutron network helper, which makes a call to the
Neutron service. If the service is started and the
Neutron service is unaccessible, the share service will
fail to even start.
This patch lazy loads those parameters so that they are
only loaded when needed (and presumably when the Neutron
service is up).
Change-Id: I1857c0cb6fc41afd0d01916addc1fd216070ffd7
Closes-Bug: #1481331
Manila scripts located in bin/* should be covered with unit tests.
In order to add unit tests for bin/* scripts:
- all
- api
- manage
- scheduler
- share
we move them into manila/cmd and use pbr to setup console scripts
which will call the respective main function of each script under manila/cmd.
It will allow us to import from manila.cmd and individually test each 'main' function.
Add unit tests for manila/cmd/* scripts.
Implements bp unit-tests-for-manila-scripts
Change-Id: I9be6b948a6de7ba76405411e98d7531cf225b57e
Module 'log' from oslo-incubator was removed after release of oslo_log library.
So, start using oslo_log, but keep oslo-incubator code yet other common modules
within Manila codebase use it.
Implements bp use-oslo-log-lib
Change-Id: I88224f7c2bd99adb78140dfc3fa73cea437f29cd
This change introduces pool-aware scheduler to address the need for
supporting multiple pools from one storage controller.
Derived from the Cinder Implementation of Pool-aware Scheduler -
https://review.openstack.org/#/c/98715/ and
https://review.openstack.org/#/c/119938
Implements blueprint: dynamic-storage-pools
Change-Id: I3aee5ed1f96f972f7d40fbd981393559587c1a23
The Manila test module uses a class called StubOutForTesting, which
is an awkward holdover from when the unit tests were using Mox.
Alex wrote a more elegant and capable mocker method,
manila.test.TestCase.mock_object, in the Cinder project which we
have now added to Manila. So to remove the duplicate functionality
and legacy code, all references to self.stubs.Set() in the Manila
tests should be replaced with self.mock_object() and any related
dead code should be removed.
Resolves-bug: #1415602
Change-Id: Ie5f203c0b2cb57ad00d9ec8db0529075b6fef261
It is useful for engineering to be able to see the API traffic
to and from the storage controllers, as well as to see the
progression of method calls through the driver. This work will
add a simple config option to enable various debug tracing
types to be written to the debug log at runtime.
Implements blueprint netapp-driver-tracing
Change-Id: Idf47a901abd434f256b71b2da454558878a769a2
As we have done in Cinder, NetApp is planning additional Manila
drivers and would like to have a factory to simplify the user
experience of selecting a driver in manila.conf. The
implementation will be very similar to the Cinder one. Included
is the Cinder map in the share manager that allows users to
change driver paths/names while maintaining backward
compatibility.
Implements blueprint netapp-driver-factory
Change-Id: I7dd401688e1bcccdde09b3ecf701924769ad2fe8
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
oslo.messaging is the only exception because this package doesn't
currently support non-namespaced imports.
Change-Id: I3987e651bc880c8ffa7c0105df0298679dcd3a43
Switch to BaseTestCase as base class for all Manila test classes.
This BaseTestCase class automatically uses the following fixtures:
* fixtures.FakeLogger (only if needed)
* fixtures.NestedTempfile
* fixtures.TempHomeDir
so we don't have to explicit add them to our constructor.
Change-Id: I336d024f7008f5300e1882fa960a8e32439a9a70
Module 'lockutils' was released within 'oslo concurrency' lib and removed
from oslo incubator. Use it from lib.
Changes:
- Removed local copy of 'lockutils' copied from incubator.
- Config opt 'lock path' moved to separate config group 'oslo_concurrency'.
Now it can be set for Devstack installation also by "OSLO_LOCK_PATH" env var.
- Removed explicit usage of 'lockutils.synchronized' within Manila and
reused it from manila.utils.
Change-Id: I21257bc8c0a41dc1bb342deadbcf279c9f3b670d
Remove obsoleted modules from tree.
Sync with oslo-incubator at commit
838a2a31fe009feb2501716d6656eeea426ea357.
The updated lockfile module needs setting of the lock_path option,
ensure that it is set in the testsuite.
Closes-Bug: #1382189
Change-Id: I43ca76a21bd5bdb45f23af9b99b96aefa635ccc1
Change usage of modules that are deprecated in oslo-incubator.
Use the corresponding module from oslo.utils.
A followup patch will sync with oslo-incubator and remove the now
obsolete modules.
Change-Id: I4f949de57e333832dcc7c1e256ce82e2db0144cb
Partial-Bug: #1382189
Make help strings consistent:
* Add missing spaces between words
* Capitalize first word of help
* Add "." at end of string
* Improve wording and capitalization
This follows the oslo.config style guide:
http://docs.openstack.org/developer/oslo.config/styleguide.html
Change-Id: I8243909249423b6b58ccda0d800856f46b0953c5
Stephen Finucane