Instead of getting all services, fetch topic related entries from db
during service cleanup.
Related-bug: #1990839
Depends-on: I8b71c4c27ff8fcb25616a95a5ed8362a7f4ffc61
Change-Id: I9e911021bd144b76d39193e5480d5ca88973973e
When manila services are stopped or restarted via stop(), the DB
entries are not deleted, they are destroyed only in kill() method. In
cluster deployments, where multiple instances of manila services are
deployed via PODs, unique hostname is derived from node name. However
if pods are deployed again and launched on new hosts/nodes, the old
entries of manila service remains as it is.
Fix it by adding 'state' column in 'services' table and introducing
per service cleanup function. On service stop, state is changed to
'stopped' and cleanup function will delete 'stopped' services unless
they are 'up' again before cleanup periodic interval.
Closes-bug: #1990839
Change-Id: I8b71c4c27ff8fcb25616a95a5ed8362a7f4ffc61
This change adds a new hook to Manager class called init_host_with_rpc()
to allow services like scheduler to do something once RPC is ready.
Copied from cinder 65fa80c361
and 60c563f72d
Change-Id: Iac6507a6e395c55f0fec453650009f08c2bb6563
Closes-Bug: #1271568
*) Add osprofiler wsgi middleware
This middleware is used for 2 things:
1) It checks that person who want to trace is trusted and knows
secret HMAC key.
2) It start tracing in case of proper trace headers and add
first wsgi trace point, with info about HTTP request
*) Add initialization of osprofiler at start of serivce.
You should use python-manilaclient with this patch:
https://review.opendev.org/#/c/769731
Run any command with --profile SECRET_KEY
$ manila --profile SECRET_KEY create NFS 1 --name <share> \
--description <share_description> --share-network <network> \
--share-type default
# it will print <Trace ID>
Get pretty HTML with traces:
$ osprofiler trace show --html <Trace ID> --connection-string \
<connection_string> --out <output.html>
e.g. --connection-string can be redis://localhost:6379
Note that osprofiler should be run from admin user name & tenant.
Implements: blueprint manila-os-profiler
Change-Id: I3bce1f04d1cfebfacd78ed135a949a068c78987d
This commit adds is_service_ready method to base Manager class that can be
used to indicate that service is not ready or initialzed success. This is
used to block refreshing Service heartbeats if manager will return false
from is_service_ready.
Closes-Bug:#1853940
Change-Id: Ib85468c703dfa51b03d1838bd422c9b2669bc747
"storage_availability_zone" in the [DEFAULT] section of
manila's configuration file has allowed deployers to configure
and manage both service (scheduler, share manager) and storage
system availability. However, quite often manila's services
(api, scheduler, share and data managers) are run on a dedicated
control plane that is a different failure domain from that of
the storage that manila manages.
Also, when using share replication, deployers would need to
run multiple manila share manager services with different
configuration files, each with their own
"storage_availability_zone".
To allow flexibility of separating service and storage
availability zones, we introduce a new configuration option
"backend_availability_zone" within the share driver/backend
section. When this option is used, it will override the value
of the "storage_availability_zone" from the [DEFAULT] section.
Change-Id: Ice99a880dd7be7af94dea86b31a6db88be3d7d9b
Implements: bp per-backend-availability-zones
We had SIGHUP apply the "mutate" behavior to the
ProcessLauncher with [1], however, the scheduler
process does not use ProcessLauncher.
Oslo config made "debug" a mutable config option
in [3]
Closes-Bug: #1785129
[1] I6499ae6c98d9489a123bb20febd6c1ddb9e17b86
[3] I1e7a69de169cc85f4c09954b2f46ce2da7106d90
Change-Id: Ifffe3cad0f5cd52988ce253794efc5b03ea9840b
This removes some custom code inherited from Cinder which was handling
the output of secret options in a bad way. This patch utilizes Oslo's
existing utilities to output the Manila configuration options securely.
Filtering will be done with the "secret=True" option flag.
Major thanks to Eric Harney for introducing this fix to Cinder.
Change-Id: I894e011680661c0b73b9592f70a6457e403f18c6
Related-Bug: #1750074
Manila will display the rabbitmq password if debugging is enabled.
This patch will ensure that the rabbitmq password is no longer
displayed in the connection log for Manila when debugging is
enabled by looking for the rabbitmq key and not printing it.
There should likely be an effort to utilize Oslo's secret flag for
options to truly fix this issue for this and other sensitive options.
Change-Id: I97cc88354d9b54057350c70c4742055197540d1a
Closes-Bug: 1750074
Currently, Manila does not support secure access the manila
APIs, obviously, this is a defect for manila service. This
change is to add ssl support for manila project.
Closes-bug: #1732844
Closes-bug: #1730529
Change-Id: I2dbc52ce95933e648cc065b2b2112788bf4484d0
New releases of oslo.config support a 'mutable' parameter to Opts.
This is only respected when the new method mutate_config_files is
called instead of reload_config_files. Cinder delegates making this
call to oslo.service. This was provided in patchset
Icec3e664f3fe72614e373b2938e8dee53cf8bc5e
Further patches will be needed to make select config options be
marked as mutable. This change enables support for oslo provided
config options to be updated via SIGHUP such as log level.
Refer to cinder patch: I6499ae6c98d9489a257bb20febd6c1ddb9e17b86
Change-Id: I6499ae6c98d9489a123bb20febd6c1ddb9e17b86
Some of the available checks are disabled by default, like:
[H106] Don't put vim configuration in source files
[H203] Use assertIs(Not)None to check for None
[H904] Use ',' instead of '%', String interpolation should be
delayed to be handled by the logging code, rather than
being done at the point of the logging call.
Change-Id: Ie985fcf78997a86d41e40eacbb4a5ace8592a348
One of the goals for Pike [1] is to make each API service be able to
run under web servers that support WSGI applications,
such as Apache (+mod-wsgi) and Nginx (+uWSGI).
Do following to address governance requirements:
- Split existing manila/wsgi.py module into 3 modules:
First (manila/wsgi/eventlet_server.py) is used by
eventlet-based WSGI application approach.
Second (manila/wsgi/wsgi.py) is used for WSGI web servers.
And third (manila/wsgi/common.py) is common code for both.
All three are made in cinder-like way to have alike-approach.
- Reuse common code from "oslo_service/wsgi.py" module that
allows us to remove code duplication.
- Delete config opts that are defined by newly reused common code.
- Register new entry point that will be manila wsgi app: "manila-wsgi".
- Fix "manila/api/openstack/wsgi.py" module to be compatible
with str/bytes handling approach used by Apache mod-wsgi plugin using
different python versions (2/3).
- Add web server config template "devstack/apache-manila.template"
- Add devstack support where usage of this feature can be
enabled or disabled using "MANILA_USE_MOD_WSGI" env var.
It is set to "True" by default, because it is requirement for Pike
release - to have it running in all CI jobs.
Disable it only for one CI job that uses dummy driver and tests
various manila core features that are not covered by other CI jobs.
[1] https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html
Partially-Implements BluePrint wsgi-web-servers-support
DocImpact
Change-Id: Ibdef3c6810b65a5d6f3611e2d0079c635ee523ab
Some configuration options were accepting both IP addresses
and hostnames. Since there was no specific OSLO opt type to
support this, we were using ``StrOpt``. The change [1] that
added support for ``HostAddressOpt`` type was merged in Ocata
and became available for use with oslo version 3.22.
This patch changes the opt type of configuration options to use
this more relevant opt type - HostAddressOpt.
[1] I77bdb64b7e6e56ce761d76696bc4448a9bd325eb
TrivialFix
Change-Id: I44ba478ff14a6184434dd030efd9b7fa92458c7a
Manila currently uses file locks from oslo_concurrency to
coordinate operations racing with each other to perform a
particular action. In many situations, deployers may need a
distributed lock to a local file lock (or even a file lock living on
a shared file system). This need is accentuated if they were running
Manila services in HA or if they were using Share Replication across
AZs where manila-share services were running off different controllers
that would not be able to share a common oslo_concurrency
file lock or be protected against service/lock management failures.
Integrate Tooz library with helper methods to create a locking
coordinator and allow deployers to make the choice between file
and distributed locks.
Start the manila share service with Tooz backed coordination.
Replace the locks used for Share Replication work-flows in the
share manager to use Tooz based locks.
Co-Authored-By: Goutham Pacha Ravi <gouthampravi@gmail.com>
Co-Authored-By: Szymon Wroblewski <szymon.wroblewski@intel.com>
Co-Authored-By: Tom Barron <tpb@dyncloud.net>
Related-Bug: #1585241
Partially-implements: bp distributed-locking-with-tooz
Change-Id: I710e86bd42034fa3b93b87ff77fa48ada8661168
Rework availability zones support which was inherited from Cinder:
- Add public API extension
- Preserve AZ if creating a share from a snapshot
- Always set AZ in Share API or Share Manager
- Update db schema and create db migration
- Update appropriate unit tests
APIImpact
Partially-Implements: blueprint availability-zones
Change-Id: Iea9fbc3fea5c0128772115c028989121f397e0c5
In py34 we have more strict approach to type conversions.
So, fix conversions for integers and floats.
Change-Id: I237c8972c3449f2a0b6e925d747c63dc5d479e18
Partially-Implements: bp py3-compatibility
If Manila API runs on a system with IPv6, it listens on IPv4 by default,
but clients try IPv6 first. The resulting retry is unnecessary. On Linux,
it is easy to listen on both address families simply by opening an IPv6
listening socket, so this patch makes use of that feature.
The listening address may be set explicitly as before. Only the default
is changed by this patch.
Closes-Bug: #1338825
Change-Id: I45dd3c94b332d14a49c78f480cf1f6f69749acf3
Due to the limit of Python interpreter, API service of Manila can't
really utilize underlying multi-core architecture even libraries
like eventlet has been used. To make API service much more scalable,
we'd adopt multi-process (worker) mode that has been used for long
in Glance/Swift/Nova/Cinder.
The default behavior isn't changed with this patch, Manila API
service will still run in one process (default value of
osapi_volume_workers is None).
Implementation wise, a good portion of manila/service.py has been
removed because those content has been merged in Oslo version of
service module. manila/wsgi.py is also updated to adopt the change
for multiple WSGI servers running in separate processes.
DocImpact: 'New config option osapi_share_workers is used to specify
number of API service workers (OS processes) to launch for Manila
API service. Setting this config option to a proper value (e.g.
osapi_share_workers = # of CPU cores/threads of the machine) can
greatly improve the total throughput of API service [# of API
requests can be handled per second].'
Also removed out-dated comments in manila/cmd/api.py due to the fact
that this bug [1] has been fixed in eventlet 0.9.13
This commit is port from two Cinder changes - [2] and [3].
[1] https://bitbucket.org/eventlet/eventlet/issue/92/eventletgreen-override-of-oswaitpid
[2] I8361d0dc0d43040e48634ff1aee1324e5e0af466
[3] Iab32a3fe230a11692a8cad274304214247d6c2c6
Implement bp: multi-process-api-service
Change-Id: I1cb98c938fd4e1dabe75c78a7ef392d6d7387dab
oslo.service replaces the loopingcall code from oslo-incubator so switch
to the new oslo.service module.
Change-Id: Iba8a86cc79f8bbda4b04c71568dd9a33f4ace6f6
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: I75e6e15d50ef9830d0581efd4cbcceb3e626f7b7
Module 'log' from oslo-incubator was removed after release of oslo_log library.
So, start using oslo_log, but keep oslo-incubator code yet other common modules
within Manila codebase use it.
Implements bp use-oslo-log-lib
Change-Id: I88224f7c2bd99adb78140dfc3fa73cea437f29cd
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
oslo.messaging is the only exception because this package doesn't
currently support non-namespaced imports.
Change-Id: I3987e651bc880c8ffa7c0105df0298679dcd3a43
LOG.warn etc. should be translated separately and thus messages need to
be marked with _LW for LOG.warn, _LI for LOG.info and _LE for LOG.errors
and LOG.exception.
Mark all LOG invocations with proper translation marker.
Use ',' instead of '%' when adding variables to log messages to allow
lazy evaluation.
Add new hacking checks for these.
Change-Id: I31d3ee50f30c63d7d647b1c2b1eae50bf96f0c74
Change usage of modules that are deprecated in oslo-incubator.
Use the corresponding module from oslo.utils.
A followup patch will sync with oslo-incubator and remove the now
obsolete modules.
Change-Id: I4f949de57e333832dcc7c1e256ce82e2db0144cb
Partial-Bug: #1382189
oslo.i18n provides the i18n function that were provided by
oslo-incubator's gettextutils module
Import _ where needed, oslo.i18n deprecated the builtin method.
Closes-Bug: #1382187
Change-Id: I12aa1c725aa4bb52a9aa46e9c3d2b303839de48b
LOG.debug("Hello %s" % xyz) should be LOG.debug("Hello %s", xyz).
This allows the logging package to skip creating the
formatted log message if the message is not going to be emitted because
of the current log level.
Change-Id: Ie0c91cd940017fd80d1d84b0e56780a1da980858
Make help strings consistent:
* Add missing spaces between words
* Capitalize first word of help
* Add "." at end of string
* Improve wording and capitalization
This follows the oslo.config style guide:
http://docs.openstack.org/developer/oslo.config/styleguide.html
Change-Id: I8243909249423b6b58ccda0d800856f46b0953c5
Remove own implementation of 'LoopingCall' and replace its usage
with 'FixedIntervalLoopingCall' class from common code.
Partially-implements blueprint use-common-code
Change-Id: I53f9773d67394b98d22bf548430e17561f713444
According to the latest revision of the logging guidelines message
of type AUDIT should be moved to INFO.
The latest revision is available at https://review.openstack.org/#/c/91446/
at the moment.
Change-Id: I9f8aaeec0bbd17e58ddb814eb7499dc9d6343ca6
Manila uses oslo-incubator/rpc as an RPC library.
During Havana, oslo/rpc was cleaned up and moved into
oslo.messaging with a more stable and well-defined API.
oslo-incubator/rpc considered as deprecated and should be
replaced with oslo.messaging in Manila.
Sum changes:
* New dependency oslo.messaging>=1.3.0 is required
* The new rpc module has init() and cleanup() methods which manage the
global oslo.messaging transport state. The TRANSPORT and NOTIFIER
globals are conceptually similar to the current RPCIMPL global,
except we're free to create and use alternate Transport objects.
* The rpc.get_{client,server,notifier}() methods are just helpers
which wrap the global messaging state, specifiy serializers and
specify the use of the eventlet executor.
* In oslo.messaging, a request context is expected to be a dict, so
RequestContextSerializer was added which can serialize to and from
dicts using RequestContext.{to,from}_dict()
* The allowed_rpc_exception_modules configuration option is replaced
by an allowed_remote_exmods get_transport() parameter. This is not
something that users ever need to configure, but it is something
each project using oslo.messaging needs to be able to customize.
* We maintain a global NOTIFIER object and create specializations of
it with specific publisher IDs in order to avoid notification driver
loading overhead.
* rpc.py contains transport aliases for backwards compatibility
purposes. setup.cfg also contains notification driver aliases for
backwards compat.
* messaging.ConfFixture is used in tests to override oslo.messaging
config options, rather than making assumptions about the options
registered by the library.
Partially-implements bp oslo-messaging
Change-Id: I42cd582f3e1ff96c8f6e8957122b8e9176b1771d
According to the OpenStack translation policy available at
https://wiki.openstack.org/wiki/LoggingStandards debug messages
should not be translated. Like mentioned in several changes in
Nova by garyk this is to help prioritize log translation.
Change-Id: Idd917acd97481d6b644e307adb002354b37f34a1
Moving file flags.py to manila/common/config.py,
replacing FLAGS by CONF. Rename modules fake_flags to conf_fixture,
test_flags to test_conf, declare_flags to declare_conf,
runtime_flags to runtime_conf like it was done in cinder, nova, glance etc.
Implement bp: use-oslo-conf
Change-Id: I38d869123e5e706d3b06f1844b97ead05e22668f
Kiran Pawar