Commit Graph

47 Commits

Author SHA1 Message Date
Brian Rosmaita f165c6ff5e Stop cinderlib development
Remove files from master, as development will no longer occur there.
The stable branches continue to be supported while they are in
Maintained status.

Updated the README to indicate this change.

Depends-on: Ib186ac5830e5920e264d79be946995e63e960426
Depends-on: I081cd363117671eaab6a3193094d5872f9820354
Depends-on: If2b9a82cddb20543b176ee22765049db257c89b9
Depends-on: I1143e5e5ccf8103e386fe1ce614a554e7f152d9a
Change-Id: I4722b869033ad1bd357e36c4a258b6d3ea61f5d6
2024-01-05 16:03:46 +00:00
OpenStack Release Bot ee1d86c058 Update master for stable/2023.2
Add file to the reno documentation build to show release notes for
stable/2023.2.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.

Sem-Ver: feature
Change-Id: I90a9f25c43f9dfae97db72e73cb2f751ea32dbaf
2023-11-30 09:13:33 +00:00
OpenStack Release Bot b66c34fb55 Update master for stable/2023.1
Add file to the reno documentation build to show release notes for
stable/2023.1.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.1.

Sem-Ver: feature
Change-Id: I4b98ac85b5d5dbb5c4d0eb38909eb2dc53824f65
2023-05-04 09:32:24 +00:00
OpenStack Release Bot 09dc1f7acf Update master for stable/zed
Add file to the reno documentation build to show release notes for
stable/zed.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/zed.

Sem-Ver: feature
Change-Id: I5de46c7549117c3912faadfc26848a353e946ef3
2022-12-16 08:50:54 +00:00
Brian Rosmaita 3356e71cb9 Add Zed release note
Change-Id: I9ddf1195d180520267ab7140d24996508f029f56
2022-12-14 08:06:31 -05:00
OpenStack Release Bot 6815d80445 Update master for stable/yoga
Add file to the reno documentation build to show release notes for
stable/yoga.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/yoga.

Sem-Ver: feature
Change-Id: I3d2f8c1981aa67881dd308e5043b179525b743f6
2022-06-27 08:52:36 +00:00
Gorka Eguileor 4d784d23a9 Fix privsep issues inside virtual env
When a virtual environment is created with the "--system-site-packages"
option and privsep is installed on the system privsep will only use the
system packages and completely ignore the ones in the virtual
environment.

This results in errors such as the ones we see:

- In the Ussuri gate: ModuleNotFoundError: No module named
  'os_brick.privileged.rootwrap'

- In the Wallaby gate: ModuleNotFoundError: No module named
  'os_brick.privileged.nvmeof'

This happens because os-brick and cinder are starting privsep using the
"privsep-helper" mechanism, and privsep was not installed in the virtual
env because it was already present system wide, so the "privsep-helper"
that is executed is the one from "/usr/local/bin/privsep-helper".

This python script "privsep-helper" ignores the virtual environment and
forces usage of the system's python, for example in a Wallaby
installation this could be "#!/usr/bin/python3.6".

Since it ignores the virtual environment it won't use its packages and
anything that's not present on system wide will not be found, and if
found it may be executing different code.

This patch fixes this issue by replacing the helper used to start
privsep with our own command.

This command is the same as the one usually installed in /usr/local/bin
but using /usr/bin/env to select the python to use.

This new script has been included as data in the cinderlib namespace
instead of making it install as a system script (like the original
privsep command) because we don't want to polute the system wide
binaries directory just for a corner case.

We also need to preserve user site-packages for the running Python when
calling root from the virtual environment, since the packages installed
on the virtual environment with "--system-site-packages" would have
taken those into consideration during the installation and not the ones
present on the root user.

To help debug issues at the gate all functional tests are now running
with debug logs.

Change-Id: I0278b42785d14f92a521e6deff872dcba6505270
Related-Bug: #1958159
Closes-Bug: #1979534
2022-06-24 15:26:48 +02:00
OpenStack Release Bot 57f90f98c8 Update master for stable/xena
Add file to the reno documentation build to show release notes for
stable/xena.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/xena.

Sem-Ver: feature
Change-Id: Iea0b3268aac70e4d241b7720b3175506dc6b029c
2021-12-13 16:11:12 +00:00
OpenStack Release Bot 68519a8e9a Update master for stable/wallaby
Add file to the reno documentation build to show release notes for
stable/wallaby.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/wallaby.

Sem-Ver: feature
Change-Id: I5d69003decdd8b8855078956aa1613ec3eeef4a2
2021-06-30 13:45:45 +00:00
Gorka Eguileor 51609b61a4 Fix datetime serialization
Cinderlib's methods and property that convert objects to json end up
losing resolution on DateTime fields.

Since json doesn't have a datetime fields they are stored as strings,
but they are not storing subsecond information.

This means that datetime fields that are serialized and then
deserialized are different.

This patch changes the way we format the string to use the ISO-8601
subsecond format instead of the standard second format.

This is a backward compatible change, as this version of cinderlib will
be able to read old datetime string, and old cinderlib code will be able
to read new datetime strings as well (even if it cannot generate them).

The old code is able to read the new format because the from_primitive
code of the OVO field uses oslo_utils.timeutils.parse_isotime which in
the end uses iso8601.parse_date, and since the subsecond format is also
ISO8601 it is properly parsed.

Change-Id: I1422e833ec0ec50376d4afac91f22c7b60bf996b
Closes-Bug: #1933964
2021-06-29 17:48:07 +02:00
Zuul 4f4c8c15f5 Merge "Add privsep support" 2021-06-22 00:11:18 +00:00
Gorka Eguileor cf638c41f6 Add privsep support
Cinderlib does not support Cinder drivers that make use of the privsep
library.

Originally privsep had a limitation that would serialize all requests,
so slow operations would create a bottleneck, thus cinderlib decided not
to use privsep and call the commands directly.

Since privsep no longer serializes requests we stop going around privsep
and use it.

Cinderlib is a library that works when run in a virtual environment, so
we must maintain backward compatibiliy and still support it, which is
problematic, because with Cinder's rootwrap+privsep we require
/etc/cinder/rootwrap.conf and /etc/cinder/rootwrap.d to exist, but under
a virtual env these are installed in the virtualenv's etc
directory instead.  For example: .tox/py37/etc/cinder/rootwrap.conf

This configuration file is modified to point to the right filters
directory and add the virtual env's bin directory to exec_dirs.

We also take into account if we have installed cinder as editable in our
virtual environment, because in that case files are not installed, and
we will copy them from the source's directory into the virtual
environment so we can freely modify them.

Depends-On: https://review.opendev.org/737312
Closes-Bug: #1883720
Change-Id: I7963fbfbb0a683e3efcc5949f80b96e5daaa18f1
2021-06-18 15:12:24 +02:00
tushargite96 390f04f433 Dropping explicit unicode literal
In python 3, all strings are considered as unicode string.

This patch drops the explicit unicode literal (u'...')
or (u"..") appearances from the unicode strings.

Change-Id: I8cf2c7a1532e1df454321387a2d49e3f1e5ed1b7
2021-02-17 19:08:41 +05:30
OpenStack Release Bot 45c7b02ba5 Update master for stable/victoria
Add file to the reno documentation build to show release notes for
stable/victoria.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/victoria.

Sem-Ver: feature
Change-Id: I0cd9d7115b6796adc61638b7428f65c431cee64a
2021-01-14 13:46:20 +00:00
OpenStack Release Bot 55e78be3b6 Update master for stable/ussuri
Add file to the reno documentation build to show release notes for
stable/ussuri.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/ussuri.

Change-Id: I9dcbe2557fbacb1281a74eaba695844144585897
Sem-Ver: feature
2020-07-23 09:21:26 +00:00
Gorka Eguileor efcec1d0ee RBD: Fix error on disconnect
On the RBD driver, when doing a disconnect call inside a container, when
the "rbd_common" package is installed on the host, we can get a "No such
file or directory" error if the udev rule from the host removes the
symlink before we do.

This error is caused by the rm command returning 1 for different type of
errors, so we'll now use -f to ensure we don't fail if the file is
missing, which is what we intended before when passing the
check_exit_code parameter to the execute call.

Trivial-Fix
Closes-Bug: #1885293
Change-Id: Id1655614a8a60fba8ed225c6cea53ae1ed4b694a
2020-07-15 12:39:29 +02:00
Zuul 4347547a0d Merge "Prevent duplicated backend names" 2020-07-14 16:17:14 +00:00
Zuul 59a8db40c5 Merge "Support references in driver configuration" 2020-07-14 16:17:13 +00:00
Zuul d17e32ec0d Merge "RBD: Fix missing root_helper" 2020-07-14 16:17:12 +00:00
Zuul 5975297960 Merge "Improve cinder supported drivers output" 2020-07-14 16:17:06 +00:00
Zuul 81467bc1d1 Merge "RBD: Fix containerized detection" 2020-07-14 16:17:02 +00:00
Gorka Eguileor db573863aa Prevent duplicated backend names
With this patch we prevent us from having duplicated backend names for
different driver configurations during the same run.

If we pass the same name and configuration to Backend a second time we
get the Backend instance that was created on the first call.

If we pass the same name with a different configuration then we raise an
exception on the second call.

Change-Id: I87c14d236d7f2685315e4088db527090680c9523
Closes-Bug: #1886164
2020-07-14 11:28:01 +02:00
Gorka Eguileor b6ddbcda43 Support references in driver configuration
Cinder, through Oslo Config, supports references as values in its
configuration options, such as the default value used in
target_ip_address which is $my_ip.

This patch adds the same functionality to cinderlib, which is convenient
for library users that use the default values reported by the
list_supported_drivers for the options to set the values.

In other words, if for example we don't pass target_ip_address for the
LVM driver configuration, then it correctly uses the value of my_ip
field, but if we get the default value and pass it, the driver will
fail, since the literal '$my_ip' is not usable as an IP address.

Change-Id: I038da3637268c9e43c464e841f8c357b212fbc9f
2020-07-14 11:28:00 +02:00
Gorka Eguileor a37a93154f Improve cinder supported drivers output
Method list_supported_drivers was intended for human consumption, so all
values for the options in driver_options where strings, which makes it
harder to parse by automation tools since they have to:

- Convert values from 'None' to ``None``, 'False' to ``False``, etc.
- Parse a string with type of the option to determine the additional
  options, such as choices, max and min values.

This patch adds the possibility to get an output intended for automation
tools.

To maintain backward compatibility the method accepts
``output_version`` parameter that defaults to ``1`` (old output) but can get
``2`` for the new output.

Change-Id: I2ade5d2e6b2c97d905e09fee509c81c2cfb602ed
2020-07-14 11:25:42 +02:00
Gorka Eguileor 9e17143194 RBD: Fix missing root_helper
When using the RBD driver and running the cinderlib process as a
non-root user within a container we'll get a "Command requested root,
but did not specify a root helper." error.

This error happens because the library is not passing the "root_helper"
parameter in a couple of "_execute" calls.

Trivial-Fix
Closes-Bug: #1885291
Change-Id: Idf664fdab3daf21b99215edfe3e2c053c3a9e85c
2020-07-14 11:20:37 +02:00
Gorka Eguileor 669235b8ab RBD: Fix containerized detection
Our detection mechanism of when we are running containerized has false
positives, so we are not also checking /proc/1/mounts to see if the root
directory is mounted from /var/lib/, which is the case when running
containerized.

Change-Id: Icad0340a9f805b59a5cddd7526a1efbb24ab652e
Closes-Bug: #1885302
2020-07-14 11:19:22 +02:00
Zuul ad54e86aba Merge "DBMS: Fix db_sync between N and N+1 releases" 2020-07-13 16:38:17 +00:00
Gorka Eguileor 01dfcb172e DBMS: Fix db_sync between N and N+1 releases
If we start cinderlib from the N release with the DBMS persistence
plugin on a DB where we have already run it with release N+1 then we'll
get an `oslo_db.exception.DBMigrationError` exception.

This is because cinderlib automatically does a `db_sync` on each
initialization, but on N this will fail because there will be unknown
(as in newer) migrations applied to the DB.

This patch takes this possibility and checks the exception raised by
`db_sync` and ignores it if the `migrate` package complains with a
VersionNotFoundError.

Closes-Bug: #1868145
Change-Id: I539919e01f603d19cde8750ad92e3d2b4ac2fbc7
2020-06-09 16:31:20 +02:00
Andreas Jaeger c0641a1cb7 Switch to newer openstackdocstheme and reno versions
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems

Update Sphinx version as well.

Set openstackdocs_pdf_link to link to PDF file. Note that
the link to the published document only works on docs.openstack.org
where the PDF file is placed in the top-level html directory. The
site-preview places the PDF in a pdf directory.

Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.

Remove docs requirements from lower-constraints, they are not needed
during install or test but only for docs building.

openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.

See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html

Change-Id: I69fc009dd2ba86c47a9e113212e8c79b8afcb008
2020-05-21 12:56:25 +02:00
Zuul 602c9f71c2 Merge "NotLocal exception after refresh" 2020-04-23 17:22:35 +00:00
Gorka Eguileor cefeef4c7e DBMS: Fix volume deletion with MySQL
When using the DBMS plugin with MySQL with default configuration (hard
DB row deletes) the deletion of a volume fails if the volume has
metadata or admin metadata.

The reason are the foreign key constraints from those 2 tables, because
we are not deleting their rows before deleting the volume row.

This doesn't fail with SQLite, but we leave rows behind.

With this patch we will delete all metadata and admin_metadata belonging
to the volume before we delete the volume row itself.

Closes-Bug: #1868148
Change-Id: I8712045752a7a150a3633451246ed992dee35972
2020-03-24 13:43:22 +01:00
Gorka Eguileor 4e4247b248 NotLocal exception after refresh
If we locally attach a volume and we call the refresh method on it then
we cannot just call the detach method on the volume directly or we'll
get a `NotLocal` exception.

So this works:
   volume.attach()
   volume.detach()

But this doesn't:
   volume.attach()
   volume.refresh()
   volume.detach()

This is because we don't carry the `local_attach` attribute from our
current instance into the refresh.

Closes-Bug: #1868153
Change-Id: Iaaac5576d443b7917a46bb11458860a9651bd366
2020-03-24 13:40:06 +01:00
Zuul 04a23b488b Merge "Update master for stable/train" 2020-01-16 01:48:56 +00:00
Luigi Toscano 08108b5964 Use Python 3 by default and drop Python 2
Follow the rest of OpenStack and consistently switch to
python3 as default python interpreter and drop python2.

The "functional" tox target was failing with Python 2
as cinder/master has cut the Python 2.7 compatibility.
"functional" now points to the default python3. An explicit
"functional-py37" target has been added as well.

Fix the functional jobs as well:
- move the LVM job to centos-8 and adapt it when needed;
- move the Ceph job to ubuntu-bionic, as there are no
  CentOS 8 Ceph packages (and the CentOS 7 ones do not
  provide Python 3 bindings).

Closes-Bug: #1853372
Change-Id: Iea4f4f53df7400248848399494564910d3870f63
2020-01-10 15:03:23 +01:00
OpenStack Release Bot cafeb6edd2 Update master for stable/train
Add file to the reno documentation build to show release notes for
stable/train.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/train.

Change-Id: Ifc37f5af9ed1cf94db79402ea453aafc5bdc8741
Sem-Ver: feature
2019-12-12 10:08:09 +00:00
Zuul b3ee3447e0 Merge "Add missing release notes" 2019-12-10 17:42:43 +00:00
Zuul 5ce588a223 Merge "Fix complex configuration options" 2019-12-10 17:42:42 +00:00
Gorka Eguileor 27efeaad1e Add missing release notes
We have merged some patches without release notes. This patch adds the
appropriate release notes for all this patches, one file for each patch.

Change-Id: I271876628b56ee9af9627017393c9b9f49785905
2019-12-10 15:05:54 +01:00
Brian Rosmaita bd316fa91f Patch release note
Changed the yaml syntax to use literal style (the # before the bug
number was being treated as a comment indicator).  Also changed 'fix'
to 'bug' to be consistent with the other release notes.

Change-Id: Ib10d164ec47293fdd54a55e93395311140142c7e
2019-12-05 15:48:16 -05:00
Zuul 3c8a65c6da Merge "Fix cloning metadata storage" 2019-12-03 19:56:07 +00:00
Zuul 37f6e1fae5 Merge "Fix LVM extend volume" 2019-12-03 19:56:06 +00:00
Gorka Eguileor 085cb4e432 Fix complex configuration options
Cinderlib cannot properly load configuration options of the type
ListOpt, or MultiOpt where each item is a dictionary, or DictOpt...

For example, when we try to load a ListOpt option we will see:

  oslo_config.cfg.ConfigFileValueError: Value for option XYZ from
  LocationInfo(location=<Locations.user: (4, True)>,
  detail='/home/me/in_memory_file') is not valid: Value should start
  with "["

This is because cinderlib is treating this option as a MultiOpt.

Cinderlib used to just set the values of the options in the instance and
pass it to the driver, and back then we had no problems with complex
types.

The problem with that approach is that there are some drivers (like
NetApp's) that dynamically add configuration options and then force a
reload of the file, and they couldn't work with our old approach.

So we changed to used oslo.config's normal parsing of files, but to do
that we had to convert the driver parameters passed to cinderlib to a
config file and make the parser use a StringIO instance instead of a
real file.  We also had to prevent it from looking at the CLI options
(since those belong to the program that imports the library).

This new approach is more complex, because cinderlib receives the
parameters as Python primitives (list, dicts, integers, etc.), but the
Oslo config parser expects an INI file conforming to the non standard
types it provides.

Some of these non standar types are:

- DictOpt is in the form:
     option = key1:value1,key2:value2

- MultiOpt is in the form:
     option = value1
     option = value2

- ListOpt is in the form:
     option = [value1,value2]

So cinderlib would receive a list [value1, value2] or a tuple (value1,
value2) and needs to generate a string with the right configuration
option which can be in the form of a MultiOpt or a ListOpt, depending on
how the configuration option was defined.

This patch adds more logic to the conversion of cinder driver
configuration parameters to oslo config file and uses the configuration
options definitions to decide what conversion needs to be done.

With this change we now do the right converstion for ListOpt, DictOpt,
and we can even support MultiOpt options that have dictionaries as
items.

Closes-Bug: #1854188
Change-Id: I62e992804a3ae6aa0b4aa4f883807783197d4b33
2019-11-27 22:23:07 +01:00
Gorka Eguileor 6f989af09b Enhance extend functionality
Cinderlib's volume extend functionality would extend the volume in the
backend yet ignore any refresh of the view of the volume if it was
attached.

It also didn't provide a way to refresh the view of an extended volume
given a Connection instance.

This patch adds the funtionality to automatically refresh the view when
extending a locally attached volume as well as to do the refresh using a
Connection instance.

The refresh is done making the appropriate extend_volume call to
os-brick's connector.

The OS-Brick RBD connector doesn't have the method because there's
nothing for it to do, but to provide a consistent API we add the method
to our NOS-Brick to return the new size like other connectors do.

Change-Id: I04a29597e7515526fb8505c76c3d7e3026b2d9c8
2019-11-19 03:25:29 +01:00
Gorka Eguileor b7f27bb9c4 Fix LVM extend volume
When we try to extend an attached LVM volume after retrieving the volume
metadata from the persistent storage we get a TargetUpdateFailed
exception cause by exception AttributeError: 'DB' object has no
attribute 'volume_attachment_get_all_by_volume_id'.

This is caused by the LVM driver checking volume_attachment which is not
populated, which triggers a lazy loading on the OVO which calls a method
in the DB layer that is not available in our fake DB instance.

This patch resolves this by replacing the get_all_by_volume_id method in
the VolumeAttachmentList OVO.

Close-Bug: #1852629
Change-Id: I7eade434b3d86e72678769f8ec929f66967d74e3
2019-11-18 19:39:48 +01:00
Gorka Eguileor b76eae9ac5 Fix cloning metadata storage
When cloning volumes we are not properly storing the new volume's
metadata.

For one we don't store the source volume id because we use source_vol_id
instead of source_volid.

Then we also don't properly store the new state, so when we clone an
in-use volume our new volume will also be in-use.

This patch fixes the clone feature to work as expected.

Change-Id: If368f006b62c5133f6d671fcf4f4f8942d365ad8
Closes-Bug: #1849339
Closes-Bug: #1849828
2019-10-29 12:19:25 +01:00
pengyuesheng fd90bc6900 Bump the openstackdocstheme extension to 1.20
Some options are now automatically configured by the version 1.20:
- project
- html_last_updated_fmt
- latex_engine
- latex_elements
- version
- release.

Change-Id: Ifbb7aeb2a9e0963faefadeaaed8bc8b3738a9c78
2019-08-01 10:46:54 +08:00
Gorka Eguileor 77f399fd96 Format as a Cinder-related OpenStack project
Since we are going to be importing the project into OpenStack we need it
to follow the same structure as the other projects under the Cinder
umbrella.
2019-02-19 16:28:03 +01:00