Remove files from master, as development will no longer occur there.
The stable branches continue to be supported while they are in
Maintained status.
Updated the README to indicate this change.
Depends-on: Ib186ac5830e5920e264d79be946995e63e960426
Depends-on: I081cd363117671eaab6a3193094d5872f9820354
Depends-on: If2b9a82cddb20543b176ee22765049db257c89b9
Depends-on: I1143e5e5ccf8103e386fe1ce614a554e7f152d9a
Change-Id: I4722b869033ad1bd357e36c4a258b6d3ea61f5d6
With cinder change Iccfbe62406b6, the 'iscsi_secondary_ip_addresses'
opt was deprecated in favor of 'target_secondary_ip_addresses'. This
broke a cinderlib unit test that checked to see if a mocked logger
was not called (which it now was, to log a deprecation message). Use
the current option name in the test so that the test continues to
test what it was designed to test (i.e., cinderlib config loading,
not oslo.config option deprecation).
Change-Id: Ia243368a9d68aca3a717fb2be1da99bceb271e56
In previous patches (I469b4b9dd9966be8477f3cefbd6284bdaf17ca17 and
I4981ad558195895341c91fa37803713c8c0ab684) we assumed that unit tests
were failing because they needed to be updated, but that's only half of
the issue. The other half is that the DBMS plugin needs to be updated
as well.
This misunderstanding lead to a broken DBMS system that unit tests don't
detect.
This patch fixes the DBMS system and reverts changes to unit tests that
those 2 patches introduced and are not necessary as well as removed the
FIXME comments that are now fixed.
Here's a brief description of why some of the issues were happening:
- Locked database issues were happening when a unit test from
TestBasePersistence run before a DBMS unit test. It was caused by the
replacement of the main_context_manager when it had not been used. No
longer a problem since we no longer replace it and we only replace its
transaction factory when it has been used.
- "ValueError: Backend named fake_backend already exists with a
different configuration" error was caused because in TestDBPersistence
the tearDown method only called the parent's tearDown after the
cleanup of the DB, so if the cleanup of the DB failed then the global
existing backend would remain and a following test would have trouble
with it. Calling it first now to fix this.
- Splitting the running of unit tests in tox.ini was necessary because
methods in Cinder DB API were always using the context manager
transaction factory from the first DBMS Test Case class. So some DBMS
plugin methods would end up writing data to the previous database
while others would write to the new database. Now that we replace the
transaction factory everything is going to the same DB.
Change-Id: I74ce945291697bc1fcc63a35d60ef71494fdaf19
As part of the effort to simplify and properly define the driver
interface, Cinder consolidated methods do_setup and
check_for_setup_error in a single method called setup.
This patch provides support for both mechanisms in cinderlib.
Change-Id: Id037c1b8a570eea704c136f35ac0b262ca5b0dfa
This library no longer supports Python 2, thus usage of six can be
removed. This also removes workarounds for Python 2.
Change-Id: Ia3f56a2a984cc1d82b717041583a727d9cb5642c
Tests are failing because importlib_resources is limited to <3.10
in requirements.txt, which makes sense because the files module is
available in the standard library importlib.resources for python
3.10 and later. The code, however, currently tries to import
importlib_resources when running in python 3.10. Fix the code
so that it uses the standard library module for >=3.10.
Change-Id: Ie7dff06a4599f96e1983bd04fc10dbfd37dbe5a0
The persistence tests were broken when run with zed cinder.
Co-authored-by: Rajat Dhasmana <rajatdhasmana@gmail.com>
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Change-Id: I4981ad558195895341c91fa37803713c8c0ab684
Closes-bug: #1996738
When using the stable/zed cinder branch and zed upper-constraints,
we're getting a bunch of "ValueError: Backend named fake_backend
already exists with a different configuration" in unit tests. The
backend name is set in the test setUp(), but it looks like isolation
isn't being respected, maybe because the Backend instantiation uses
a singleton pattern. (Which doesn't, however, explain why this was
working previously.) Address this by giving each test backend a
unique name.
Change-Id: I469b4b9dd9966be8477f3cefbd6284bdaf17ca17
When a virtual environment is created with the "--system-site-packages"
option and privsep is installed on the system privsep will only use the
system packages and completely ignore the ones in the virtual
environment.
This results in errors such as the ones we see:
- In the Ussuri gate: ModuleNotFoundError: No module named
'os_brick.privileged.rootwrap'
- In the Wallaby gate: ModuleNotFoundError: No module named
'os_brick.privileged.nvmeof'
This happens because os-brick and cinder are starting privsep using the
"privsep-helper" mechanism, and privsep was not installed in the virtual
env because it was already present system wide, so the "privsep-helper"
that is executed is the one from "/usr/local/bin/privsep-helper".
This python script "privsep-helper" ignores the virtual environment and
forces usage of the system's python, for example in a Wallaby
installation this could be "#!/usr/bin/python3.6".
Since it ignores the virtual environment it won't use its packages and
anything that's not present on system wide will not be found, and if
found it may be executing different code.
This patch fixes this issue by replacing the helper used to start
privsep with our own command.
This command is the same as the one usually installed in /usr/local/bin
but using /usr/bin/env to select the python to use.
This new script has been included as data in the cinderlib namespace
instead of making it install as a system script (like the original
privsep command) because we don't want to polute the system wide
binaries directory just for a corner case.
We also need to preserve user site-packages for the running Python when
calling root from the virtual environment, since the packages installed
on the virtual environment with "--system-site-packages" would have
taken those into consideration during the installation and not the ones
present on the root user.
To help debug issues at the gate all functional tests are now running
with debug logs.
Change-Id: I0278b42785d14f92a521e6deff872dcba6505270
Related-Bug: #1958159
Closes-Bug: #1979534
Cinderlib's methods and property that convert objects to json end up
losing resolution on DateTime fields.
Since json doesn't have a datetime fields they are stored as strings,
but they are not storing subsecond information.
This means that datetime fields that are serialized and then
deserialized are different.
This patch changes the way we format the string to use the ISO-8601
subsecond format instead of the standard second format.
This is a backward compatible change, as this version of cinderlib will
be able to read old datetime string, and old cinderlib code will be able
to read new datetime strings as well (even if it cannot generate them).
The old code is able to read the new format because the from_primitive
code of the OVO field uses oslo_utils.timeutils.parse_isotime which in
the end uses iso8601.parse_date, and since the subsecond format is also
ISO8601 it is properly parsed.
Change-Id: I1422e833ec0ec50376d4afac91f22c7b60bf996b
Closes-Bug: #1933964
Current code in test_global_setup sets a non valid user_id and leaves it
like that for the rest of the tests, which can lead to error:
ValueError: A string is required in field user_id, not a _SentinelObject
An easy way to reproduce this error is to create a file called fail.txt
with the following content:
cinderlib.tests.unit.test_cinderlib.TestCinderlib.test_global_setup
cinderlib.tests.unit.test_serialization.TestSerialization.test_conn_to_and_from
And then run from the tox virtual environment:
stestr run --serial --load-list=fail.txt
This patch fixes this issue by restoring the values after the test is
run.
Change-Id: I121bc20c1ac4d7c07a1bcd13df19fdb868e65857
Cinderlib does not support Cinder drivers that make use of the privsep
library.
Originally privsep had a limitation that would serialize all requests,
so slow operations would create a bottleneck, thus cinderlib decided not
to use privsep and call the commands directly.
Since privsep no longer serializes requests we stop going around privsep
and use it.
Cinderlib is a library that works when run in a virtual environment, so
we must maintain backward compatibiliy and still support it, which is
problematic, because with Cinder's rootwrap+privsep we require
/etc/cinder/rootwrap.conf and /etc/cinder/rootwrap.d to exist, but under
a virtual env these are installed in the virtualenv's etc
directory instead. For example: .tox/py37/etc/cinder/rootwrap.conf
This configuration file is modified to point to the right filters
directory and add the virtual env's bin directory to exec_dirs.
We also take into account if we have installed cinder as editable in our
virtual environment, because in that case files are not installed, and
we will copy them from the source's directory into the virtual
environment so we can freely modify them.
Depends-On: https://review.opendev.org/737312
Closes-Bug: #1883720
Change-Id: I7963fbfbb0a683e3efcc5949f80b96e5daaa18f1
Instead of relying on cinder's import of sqlalchemy, import it
ourselves.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I9a163d002ad7151a0cbdb04fba137688e0316944
Importing pkg_resources scans all of the installed modules for data
that won't be used. Switch to using importlib.metdata, which more
efficiently loads the metadata for a package.
Change-Id: I1709952cf0d0329e9b56c20edef50eff1696af22
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
On the RBD driver, when doing a disconnect call inside a container, when
the "rbd_common" package is installed on the host, we can get a "No such
file or directory" error if the udev rule from the host removes the
symlink before we do.
This error is caused by the rm command returning 1 for different type of
errors, so we'll now use -f to ensure we don't fail if the file is
missing, which is what we intended before when passing the
check_exit_code parameter to the execute call.
Trivial-Fix
Closes-Bug: #1885293
Change-Id: Id1655614a8a60fba8ed225c6cea53ae1ed4b694a
With this patch we prevent us from having duplicated backend names for
different driver configurations during the same run.
If we pass the same name and configuration to Backend a second time we
get the Backend instance that was created on the first call.
If we pass the same name with a different configuration then we raise an
exception on the second call.
Change-Id: I87c14d236d7f2685315e4088db527090680c9523
Closes-Bug: #1886164
Cinder, through Oslo Config, supports references as values in its
configuration options, such as the default value used in
target_ip_address which is $my_ip.
This patch adds the same functionality to cinderlib, which is convenient
for library users that use the default values reported by the
list_supported_drivers for the options to set the values.
In other words, if for example we don't pass target_ip_address for the
LVM driver configuration, then it correctly uses the value of my_ip
field, but if we get the default value and pass it, the driver will
fail, since the literal '$my_ip' is not usable as an IP address.
Change-Id: I038da3637268c9e43c464e841f8c357b212fbc9f
Method list_supported_drivers was intended for human consumption, so all
values for the options in driver_options where strings, which makes it
harder to parse by automation tools since they have to:
- Convert values from 'None' to ``None``, 'False' to ``False``, etc.
- Parse a string with type of the option to determine the additional
options, such as choices, max and min values.
This patch adds the possibility to get an output intended for automation
tools.
To maintain backward compatibility the method accepts
``output_version`` parameter that defaults to ``1`` (old output) but can get
``2`` for the new output.
Change-Id: I2ade5d2e6b2c97d905e09fee509c81c2cfb602ed
With the security vulnerability (OSSN-0085) Cinder deprecated the
``rbd_keyring_conf`` option, but that's not a vulnerability for
cinderlib, since the user of the library already has access to the
keyring (otherwise the library couldn't read it).
Cinderlib requires this configuration option for the RBD driver to work
as expected, since cinderlib doesn't expect external user intervention
to make the keyring available on all the hosts.
In this patch we ensure that we report in the ``list_supported_drivers``
method option ``rbd_keyring_conf`` as an existing non-deprecated
configuration option, regardless if it's present (current Cinder code)
or if it has already been removed (which will happen soon).
Implements: blueprint rbd-ossn-0085-workaround
Change-Id: I7d3324e14494540f8c0d714682e1d0065c1735e3
When using the RBD driver and running the cinderlib process as a
non-root user within a container we'll get a "Command requested root,
but did not specify a root helper." error.
This error happens because the library is not passing the "root_helper"
parameter in a couple of "_execute" calls.
Trivial-Fix
Closes-Bug: #1885291
Change-Id: Idf664fdab3daf21b99215edfe3e2c053c3a9e85c
Our detection mechanism of when we are running containerized has false
positives, so we are not also checking /proc/1/mounts to see if the root
directory is mounted from /var/lib/, which is the case when running
containerized.
Change-Id: Icad0340a9f805b59a5cddd7526a1efbb24ab652e
Closes-Bug: #1885302
If we start cinderlib from the N release with the DBMS persistence
plugin on a DB where we have already run it with release N+1 then we'll
get an `oslo_db.exception.DBMigrationError` exception.
This is because cinderlib automatically does a `db_sync` on each
initialization, but on N this will fail because there will be unknown
(as in newer) migrations applied to the DB.
This patch takes this possibility and checks the exception raised by
`db_sync` and ignores it if the `migrate` package complains with a
VersionNotFoundError.
Closes-Bug: #1868145
Change-Id: I539919e01f603d19cde8750ad92e3d2b4ac2fbc7
Attaching an RBD volume fails when we run the code as root if we are
using Python 3.
We see a DeviceUnavailable exception:
conn.attach()
File "/usr/lib/python3.6/site-packages/cinderlib/objects.py", line
842, in attach reason=error_msg)
cinder.exception.DeviceUnavailable: The device in the path None is
unavailable: Unable to access the backend storage via path /dev/rbd0.
The exception message also shows None as the path, even though the
description of the exception shows it correctly.
The underlying error is a UnicodeDecodeError that happens because when
we run as root we open the RBD device directly in Python to check if it
is available, but we do it as text, not as binary.
This patch changes how we open the file, doing it as binary, and reports
the right path on the exception.
Closes-Bug: #1873312
Change-Id: I967caa2df84d870c1be4d69f32481fc75c3be010
We were relying on oslotest using the mock library and we were not
declaring it in our test-requirements.txt file, but now they are using
unittest.mock instead [1], so our unit tests are now failing because the
mock library is missing.
Switch to unittest.mock that is present in Python 3 to resolve the
problem.
[1]: https://review.opendev.org/#/c/715704/
Change-Id: If79500534aed2ab992c423f106088905edcb9d7f
When using the DBMS plugin with MySQL with default configuration (hard
DB row deletes) the deletion of a volume fails if the volume has
metadata or admin metadata.
The reason are the foreign key constraints from those 2 tables, because
we are not deleting their rows before deleting the volume row.
This doesn't fail with SQLite, but we leave rows behind.
With this patch we will delete all metadata and admin_metadata belonging
to the volume before we delete the volume row itself.
Closes-Bug: #1868148
Change-Id: I8712045752a7a150a3633451246ed992dee35972
If we locally attach a volume and we call the refresh method on it then
we cannot just call the detach method on the volume directly or we'll
get a `NotLocal` exception.
So this works:
volume.attach()
volume.detach()
But this doesn't:
volume.attach()
volume.refresh()
volume.detach()
This is because we don't carry the `local_attach` attribute from our
current instance into the refresh.
Closes-Bug: #1868153
Change-Id: Iaaac5576d443b7917a46bb11458860a9651bd366
Cinder added a new restriction to the snapshots in the DB in change
I47ff8115ae74e1a7ad41869159871ba614c388ac and now snapshot's
volume_type_id cannot be null.
This broke the cinderlib CI because one of our tests (test_set_snapshot
of the DBMS persistence plugin) didn't create the volume in the DB
(which can never happen in real use because it is saved on creation)
which is when the volume type gets assigned.
This patch saves the volume so that it gets a volume type and its ID can
be used by the Snapshot.
It also includes a workaround for the LVM job for
https://github.com/pypa/pip/issues/6264
Change-Id: I0993e2aab8511400247042c128dc3aa1a06486ac
unittest2 was needed for Python versions <= 2.6, so it has not been
needed for quite awhile (ever in this project). See unittest2 note on:
https://docs.python.org/2.7/library/unittest.html
This drops unittest2 in favor of the standard unittest module.
Change-Id: I21264290d783d582dfb886b799781380dbf5757d
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
When fixing bug #1854188 (Change-ID:
I62e992804a3ae6aa0b4aa4f883807783197d4b33) we broke configuration
options, because now we cannot pass configuration options from the
DEFAULT section, and some ListOpt don't work because cinderlib was
incorrectly expecting all these options to have bounds=True, but some
don't define it and default to False.
Most of the "crazy" code we currently have around oslo config is because
we wanted to support drivers that add configuration options on runtime
and reload the configuration.
This made sense when cinderlib was not an OpenStack official project,
but now that it is it makes more sense to force Cinder drivers to do the
right thing.
Aligned with this, this patch removes all the faking of the cinder.conf
file as a StringIO object, and simply sets things in the configuration
object that is passed to the drivers, which is the only thing they
should be using.
This will make cinderlib code more robust, even if some drivers will now
need to be modified to work with cinderlib.
We are validating known configuration options and ignoring unknown ones,
like Cinder does.
The NetApp driver was one of the drivers that would get broken, but
since I have access to it and we can make a very small and targeted
workaround to avoid breaking it, we do it as well.
Closes-Bug: #1856556
Change-Id: I67fbf8e9d7ee79f3d6617b4d0ae755dae0e1987a
For backends with support for multiple pools we need to be able to
configure the pool we want to use during the cinderlib functional tests,
since we don't have preconfigured volume types like we may have in
Cinder.
Change-Id: I4221814434d9141d42f53bd9f6e22112c3fad7a0
Brian Rosmaita