This commit is part of a series to retire the Packaging Deb
project. Step 2 is to remove all content from the project
repos, replacing it with a README notification where to find
ongoing work, and how to recover the repo if needed at some
future point (as in
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project).
Change-Id: I6371190302f621f04138927539f3fa9a7197df02
On iSCSI connections that make use of discovery we will be obscuring the
the logs if the sendtargets command fails because when we try to do the
cleanup another exception will be raised (TargetPortalsNotFound).
The original exception is still logged since we are using
excutils.save_and_reraise_exception(), but it will be misleading if we
don't pay close attention.
This patch fixes this by ignoring the 'Unable to find target portals
information on discoverydb.' error, but logging a debug log message,
when doing the cleanup because this exception means that the sendtargets
failed and therefore we don't have anything to cleanup.
Change-Id: I7ddf827c7f2285acd72fd5a2fcd351928cb5d2df
When we refactored the iSCSI connect mechanism we inadvertently changed
the value returned for the "multipath_id" key.
This patch fixes this and return the WWN as the value again.
This value is used by the encryption mechanism and expects it to be the
WWN.
Related-Bug: #1703954
Change-Id: Ia6d96a1e3a71488b44b3ca2323610a8f0a7cf675
When connecting encrypted volumes we need to return a symbolink link or
we will break all future attachments after detaching the volume.
OS-Brick on 1.14 and 1.15 returns real paths instead of returning symbolic
links, which results in the encryption attach_volume call replacing the
real device with a link to the crypt dm.
The issue comes from the Nova flow when attaching an encrypted volume:
1- Attach volume
2- Generate libvirt configuration with path from step 1
3- Encrypt attach volume
Since step 2 has already generated the config with the path from step 1 then
step 3 must preserve this path.
When step 1 returns a symbolic link we just forcefully replace it with a link
to the crypt dm and everything is OK, but when we return a real path it
does the same thing, which means we'll be replacing for example /dev/sda
with a symlink, which will then break the detach process, and all future
attachments.
Until Nova, Cinder, and OS-Brick are changed to have a different flow
(1, 3, 2) we need a workaround to make it work.
The workaround this patch introduces is to return a symbolic link when
the volume is encrypted.
It will try to return the symlink that always exists, but if it's not
there it will just look for ANY link to the device in '/dev/disk/by-id'.
Related-Bug: #1703954
Change-Id: If4461c3dcd67e5d948be9d9a3643c1eb81aaace9
Commit b82ab0d1a0 wrongly assumed that
the "ignoreskipactivation" flag is not needed when using thin LVM
provisioning. This is wrong.
Using thin LVM leads now to an error when the content will be copied:
ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/
rootwrap.conf dd if=/dev/mapper/cinder--volumes-_snapshot--UUID
of=/dev/mapper/cinder--volumes-volume--UUID
count=1073741824 bs=1M iflag=count_bytes conv=sparse
The reason is that the the snapshot is not activated and due to that,
no device node is available under /dev/mapper .
This reverts commit b82ab0d1a0 and makes
thin provisioning for LVM working again.
Change-Id: I062fd7d7120081c79c714d8177104cdd95522d2d
Depends-On: I788b297af92e11063b18c05a415b6f4d4d4bcd2c
Closes-Bug: #1702322
Related-Bug: #1701545
On Change-Id Iada5d4fbeb07aeaf3afb953a289b6b89778c382c we tried to fix
an issue with the multipath detach of backends that used discovery on
attach, but contrary to the commit message and the docstrings it didn't
look for ip,port in the discoverydb but ip:port instead, which meant
that it would never find what it was looking for.
This patch fixes that fix to make it search for the right regex.
TrivialFix
Closes-Bug: #1699061
Change-Id: Ibfa1a78a555e984c662f668677451f5a3ed55602
When we added open-iscsi manual scan support we made it so that it
wouldn't modify the scan mode of already existing nodes, and this is a
problem for backends that use discovery, since the discovery will add
the nodes with the default automatic scan mode.
This patch makes sure that we always try to change the mode to manual
scans and only reports scans to be automatic if the initiator doesn't
support this feature.
TrivialFix
Change-Id: Id9e2f539d5273a20038399d9af6017652a4f987f
Cinder _rbd_diff_transfer() uses "import-diff" and "export-diff" cmdline
to do a incremental backup, it will fail without a ceph-conf file. Delay
to delete temporary ceph-conf file in class RBDConnector during ceph
volume backup.
Change-Id: Ib74c85266b8c812f7a40dac293847a28768eae9a
Partial-Bug: #1578036
The new iSCSI multipath code uses multithreading to do the connections,
but since the context is a thread local variable it will not be present
in those threads so we will not see the request id or other information
from the context in those thread's log entries.
This is a small inconvenience when looking at the logs if we try to
filter by the request id,.
To resolve this we introduce a new Thread class that inherits the
context from the calling thread, this way any log entries will have the
right context and present expected information.
Change-Id: I27624a59738809d30531c0c4f37308f897dd9654
With the latest iSCSI disconnect refactoring [1] we have introduced a
regression where not all devices are properly cleaned up on a multipath
connection if we are using target discovery.
On connection we do the discovery so we get the targets, but on
disconnection the code is expecting to have all the ips, iqns, and luns
information on the connection properties, so it will only flush the
multipath and disconnect volumes that were connected using the
information from the connection property.
This patch fixes this retrieving the targets information from the
discoverydb that was gathered during the connection phase.
Output from discoverydb query looks like this:
SENDTARGETS:
DiscoveryAddress: 192.168.1.33,3260
DiscoveryAddress: 192.168.1.2,3260
Target: iqn.2004-04.com.qnap:ts-831x:iscsi.cinder-20170531114245.9eff88
Portal: 192.168.1.3:3260,1
Iface Name: default
Portal: 192.168.1.2:3260,1
Iface Name: default
Target: iqn.2004-04.com.qnap:ts-831x:iscsi.cinder-20170531114447.9eff88
Portal: 192.168.1.3:3260,1
Iface Name: default
Portal: 192.168.1.2:3260,1
Iface Name: default
DiscoveryAddress: 192.168.1.38,3260
iSNS:
No targets found.
STATIC:
No targets found.
FIRMWARE:
No targets found.
[1] https://review.openstack.org/455392
Change-Id: Iada5d4fbeb07aeaf3afb953a289b6b89778c382c
Closes-Bug: #1699061
It was recently added to open-iscsi the functionality to disable
automatic LUN scans on iscsid start, on login, and on reception of
AEN/AER messages reporting LUN data has changed.
Those 3 cases were one of the causes why Nova-CPU and Cinder-Volumes
nodes would have unexpected devices. With this new feature we can
prevent them from appearing unnexpectedly.
This patch adds the mechanism required to configure our sessions for
manual scans in a backward compatible way.
Manual scans are enabled setting `node.session.scan` to `manual`.
Change-Id: I146a74f9f79c68a89677b9b26a324e06a35886f2
This patch refactors iSCSI connect code changing the approach to one
that relies primarily on sysfs, instead of CLI tools, to retrieve all
the required information: devices from the connection, multipath system
device name, multipath name, the WWN for the block devices...
By doing so, not only do we fix a good number of bugs, but we also
improve the reliability and speed of the mechanism.
A good example of improvements and benefits achieved by this patch are:
- Clean all leftovers on exceptions on a connection.
- Parallelize logins on multipath to increase speed on flaky network
connections.
- As long as there is at least one good path when working with multipath
we will return a multipath device instead of a single path device,
which helps with temporary flaky connections.
- Don't use the rescan retry parameter as log in retry on multipath
connections so both single and multipath cases are consistent.
- We no longer rely on just one device to get the wwn and look for the
multipath. This would be problematic with flaky connections.
- No more querying iSCSI devices for their WWN (page 0x83) removing
delays and issue on flaky connections.
- It's no longer a problem for the mechanism the fact that a device
exists but is not accessible.
- We use links in `/dev/disk/by-id` to get the WWID on connect, so we
make sure there are no leftovers on disconnect, but we no longer use
symlinks from `/dev/disk/by-path`, `/dev/disk/by-id`, or `/dev/mapper`
to find devices.
- We no longer need to rely on the WWN to determine the multipath, we
have the session and the LUN, so we trace the devices and from those
we get if they belong to a multipath.
- Stop polluting the logs with unnecessary exceptions from checking if
the node or session exist.
- Action retries will now only log the final exception instead of
logging all the exceptions.
- Warn when a multipath could not be formed and a single device is being
used, as performance will be degraded.
- We no longer do global rescans on single connections that could be
bringing in unrelated and unwanted devices (`iscsiadm -T iqn -p portal
--rescan`).
- Fix scan mechanism that would not request all scans when the same iqn
was shareed between portals and could send a scan request to the wrong
IP if they shared the same iqn.
- When doing single pathed connections we could end with a multipath
because we didn't clean up unfruitful connections.
It's worth mentioning that this patch does not touch the extend
operation.
Change-Id: Ia1c47bfaa7bc3544a5feba6a8a30faf2f132b8d7
This test unnecessarily takes 6 seconds to execute so this patch reduces
this time to 0.0000 seconds.
We have to mock `eventlet.greenthread.sleep` instead of `time.sleep`
because that's what is directly being used in oslo_service's
loopingcall.
TrivialFix
Change-Id: I239cfa5dbaccf5648b732d6c414792b8497acc19
Source builds using cryptography need libssl-dev (and assorted other
platform package names) to compile. Adding package declaration for
dpkg, rpm, and suse.
Change-Id: I9238cc977529c46470479656e449ea6cef4ae5fb
Closes-bug: #1696712
This patch refactors iSCSI disconnect code changing the approach to one
that just uses `iscsiadm -m session` and sysfs to get all the required
information: devices from the connection, multipath system device name,
multipath name, the WWN for the block devices...
By doing so, not only do we fix a good number of bugs, but we also
improve the reliability and speed of the mechanism.
A good example of improvements and benefits achieved by this patch are:
- Common code for multipath and single path disconnects.
- No more querying iSCSI devices for their WWN (page 0x83) removing
delays and issue on flaky connections.
- All devices are properly cleaned even if they are not part of the
multipath.
- We wait for device removal and do it in parallel if there are
multiple.
- Removed usage of `multipath -l` to find devices which is really slow
with flaky connections and didn't work when called with a device from
a path that is down.
- Prevent losing data when detaching, currently if the multipath flush
fails for any other reason than "in use" we silently continue with the
removal. That is the case when all paths are momentarily down.
- Adds a new mechanism for the caller of the disconnect to specify that
it's acceptable to lose data and that it's more important to leave a
clean system. That is the case if we are creating a volume from an
image, since the volume will just be set to error, but we don't want
leftovers. Optionally we can tell os-brick to ignore errors and don't
raise an exception if the flush fails.
- Add a warning when we could be leaving leftovers behind due to
disconnect issues.
- Action retries (like multipath flush) will now only log the final
exception instead of logging all the exceptions.
- Flushes of individual paths now use exponential backoff retries
instead of random retries between 0.2 and 2 seconds (from oslo
library).
- We no longer use symlinks from `/dev/disk/by-path`, `/dev/disk/by-id`,
or `/dev/mapper` to find devices or multipaths, as they could be
leftovers from previous runs.
- With high failure rates (above 30%) some CLI calls will enter into a
weird state where they wait forever, so we add a timeout mechanism in
our `execute` method and add it to those specific calls.
Closes-Bug: #1502534
Change-Id: I058ff0a0e5ad517507dc3cda39087c913558561d
Id2283195d2b6002175079070a0ff0a137a9d6806 enabled fiber channel
connector but broke all the other supported ones, including iscsi.
Change-Id: I3d6d12acf5fc864eda1bbc790bf0a310c8d0c60f
Fixes-bug: 1693336
This change is one in the series of changes to
align os-brick with Cinder rules.
- H904 'Wrap long lines in parentheses instead of a backslash'
See following for more information:
https://github.com/openstack-dev/hacking
Change-Id: Ia08ce1b9cdab968c0965bc04aa7d202190cb4bb3
Signed-off-by: Maciej Kucia <maciej@kucia.net>
This change is one in the series of changes to
align os-brick with Cinder rules.
- H405 'multi line docstring summary not
separated with an empty line'
See following for more information:
https://github.com/openstack-dev/hacking
Change-Id: Id67324c0ef22287d91b49ae11eda094bf73739a1
Signed-off-by: Maciej Kucia <maciej@kucia.net>
This change is one in the series of changes to
align os-brick with Cinder rules.
- E265 'block comment should start with "# " '
See following for more information:
https://github.com/PyCQA/pycodestyle
Change-Id: I392d5ed179ddf1fd644cb3b53436f2aee2f47cbe
Signed-off-by: Maciej Kucia <maciej@kucia.net>
Remove options from flake8 section in tox.ini
This change is one in the series of changes to
align os-brick with Cinder rules.
- E123 'closing bracket does not match indentation of
opening bracket’s line'
This option is ignored by default as it is not PEP8 enforced
- E125 'continuation line with same indent as next logical line'
See following for more information:
https://github.com/PyCQA/pycodestyle
Change-Id: I439e677e66d59a48d4b2ac87eb0256146ff507e0
Signed-off-by: Maciej Kucia <maciej@kucia.net>
This patch updates the return path from 'device_path' to 'path'
to be compatible with every other connector.
Change-Id: I2c1d561521fdd37626e3301148c16feff48a1fde
Minor refactor to code that is not "Pythonic".
Using a generator comprehension with "all" can improves performance
Change-Id: I4817ddb8a2af4e1e5b8a74e3950194d23df18a0f