Commit Graph

530 Commits

Author SHA1 Message Date
Tony Breeds 79452f470c Retire Packaging Deb project repos
This commit is part of a series to retire the Packaging Deb
project. Step 2 is to remove all content from the project
repos, replacing it with a README notification where to find
ongoing work, and how to recover the repo if needed at some
future point (as in
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project).

Change-Id: I6371190302f621f04138927539f3fa9a7197df02
2017-09-12 16:04:25 -06:00
OpenStack Proposal Bot 13b0affae5 Updated from global requirements
Change-Id: I09185c6b29b2cc684fe30c3f1b68ef619d33ed36
2017-07-18 01:54:17 +00:00
Gorka Eguileor 8e4adda001 Don't obscure logs on iSCSI sendtargets failure
On iSCSI connections that make use of discovery we will be obscuring the
the logs if the sendtargets command fails because when we try to do the
cleanup another exception will be raised (TargetPortalsNotFound).

The original exception is still logged since we are using
excutils.save_and_reraise_exception(), but it will be misleading if we
don't pay close attention.

This patch fixes this by ignoring the 'Unable to find target portals
information on discoverydb.' error, but logging a debug log message,
when doing the cleanup because this exception means that the sendtargets
failed and therefore we don't have anything to cleanup.

Change-Id: I7ddf827c7f2285acd72fd5a2fcd351928cb5d2df
2017-07-13 14:38:08 +02:00
Gorka Eguileor 66520dcf6c Return WWN in multipath_id
When we refactored the iSCSI connect mechanism we inadvertently changed
the value returned for the "multipath_id" key.

This patch fixes this and return the WWN as the value again.

This value is used by the encryption mechanism and expects it to be the
WWN.

Related-Bug: #1703954
Change-Id: Ia6d96a1e3a71488b44b3ca2323610a8f0a7cf675
2017-07-13 14:33:05 +02:00
Gorka Eguileor f341e9c3ed Return symlinks for encrypted volumes
When connecting encrypted volumes we need to return a symbolink link or
we will break all future attachments after detaching the volume.

OS-Brick on 1.14 and 1.15 returns real paths instead of returning symbolic
links, which results in the encryption attach_volume call replacing the
real device with a link to the crypt dm.

The issue comes from the Nova flow when attaching an encrypted volume:

1- Attach volume
2- Generate libvirt configuration with path from step 1
3- Encrypt attach volume

Since step 2 has already generated the config with the path from step 1 then
step 3 must preserve this path.

When step 1 returns a symbolic link we just forcefully replace it with a link
to the crypt dm and everything is OK, but when we return a real path it
does the same thing, which means we'll be replacing for example /dev/sda
with a symlink, which will then break the detach process, and all future
attachments.

Until Nova, Cinder, and OS-Brick are changed to have a different flow
(1, 3, 2) we need a workaround to make it work.

The workaround this patch introduces is to return a symbolic link when
the volume is encrypted.

It will try to return the symlink that always exists, but if it's not
there it will just look for ANY link to the device in '/dev/disk/by-id'.

Related-Bug: #1703954
Change-Id: If4461c3dcd67e5d948be9d9a3643c1eb81aaace9
2017-07-13 14:31:40 +02:00
Jenkins a1b0dda617 Merge "Fix ceph incremental backup fail" 2017-07-05 18:34:30 +00:00
Jenkins b71c7b9da7 Merge "Revert "Don't use ignoreskipactivation for thin LVM"" 2017-07-05 17:10:09 +00:00
Thomas Bechtold 925f435cb9 Revert "Don't use ignoreskipactivation for thin LVM"
Commit b82ab0d1a0 wrongly assumed that
the "ignoreskipactivation" flag is not needed when using thin LVM
provisioning. This is wrong.
Using thin LVM leads now to an error when the content will be copied:

ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/
      rootwrap.conf dd if=/dev/mapper/cinder--volumes-_snapshot--UUID
      of=/dev/mapper/cinder--volumes-volume--UUID
      count=1073741824 bs=1M iflag=count_bytes conv=sparse

The reason is that the the snapshot is not activated and due to that,
no device node is available under /dev/mapper .

This reverts commit b82ab0d1a0 and makes
thin provisioning for LVM working again.

Change-Id: I062fd7d7120081c79c714d8177104cdd95522d2d
Depends-On: I788b297af92e11063b18c05a415b6f4d4d4bcd2c
Closes-Bug: #1702322
Related-Bug: #1701545
2017-07-05 13:16:38 +00:00
Jenkins 69b7e1c67b Merge "Updated from global requirements" 2017-07-02 00:11:11 +00:00
Jenkins 6cdfd2a9a3 Merge "Fix manual scan for discovery type backends" 2017-07-02 00:11:05 +00:00
Jenkins cb2b61a35e Merge "Fix iSCSI cleanup fix on discovery backends" 2017-07-02 00:09:07 +00:00
Eric Harney b82ab0d1a0 Don't use ignoreskipactivation for thin LVM
This causes failures for volume clone and
is not needed for thin.

Closes-Bug: #1701545

Change-Id: Ia47ca585985917f998e0a3d4cec36912ce57f579
2017-06-30 08:57:15 -04:00
Gorka Eguileor be37c2e040 Fix iSCSI cleanup fix on discovery backends
On Change-Id Iada5d4fbeb07aeaf3afb953a289b6b89778c382c we tried to fix
an issue with the multipath detach of backends that used discovery on
attach, but contrary to the commit message and the docstrings it didn't
look for ip,port in the discoverydb but ip:port instead, which meant
that it would never find what it was looking for.

This patch fixes that fix to make it search for the right regex.

TrivialFix
Closes-Bug: #1699061

Change-Id: Ibfa1a78a555e984c662f668677451f5a3ed55602
2017-06-29 13:59:04 +02:00
Gorka Eguileor d310e06caf Fix manual scan for discovery type backends
When we added open-iscsi manual scan support we made it so that it
wouldn't modify the scan mode of already existing nodes, and this is a
problem for backends that use discovery, since the discovery will add
the nodes with the default automatic scan mode.

This patch makes sure that we always try to change the mode to manual
scans and only reports scans to be automatic if the initiator doesn't
support this feature.

TrivialFix

Change-Id: Id9e2f539d5273a20038399d9af6017652a4f987f
2017-06-29 13:41:13 +02:00
Xiaojun Liao 1fe844efaf Fix ceph incremental backup fail
Cinder _rbd_diff_transfer() uses "import-diff" and "export-diff" cmdline
to do a incremental backup, it will fail without a ceph-conf file. Delay
to delete temporary ceph-conf file in class RBDConnector during ceph
volume backup.

Change-Id: Ib74c85266b8c812f7a40dac293847a28768eae9a
Partial-Bug: #1578036
2017-06-29 10:36:36 +08:00
OpenStack Proposal Bot b2910100a7 Updated from global requirements
Change-Id: I06f8a00207ab6f986e074a9a91e814d0a9106f3c
2017-06-27 12:19:31 +00:00
Gorka Eguileor 32ab0d1670 iSCSI multipath: improve logging on connect
The new iSCSI multipath code uses multithreading to do the connections,
but since the context is a thread local variable it will not be present
in those threads so we will not see the request id or other information
from the context in those thread's log entries.

This is a small inconvenience when looking at the logs if we try to
filter by the request id,.

To resolve this we introduce a new Thread class that inherits the
context from the calling thread, this way any log entries will have the
right context and present expected information.

Change-Id: I27624a59738809d30531c0c4f37308f897dd9654
2017-06-26 20:31:59 +02:00
Jenkins 31bec05d72 Merge "Fix iSCSI cleanup issue when using discovery" 2017-06-24 17:44:26 +00:00
Gorka Eguileor 1905398f61 Fix iSCSI cleanup issue when using discovery
With the latest iSCSI disconnect refactoring [1] we have introduced a
regression where not all devices are properly cleaned up on a multipath
connection if we are using target discovery.

On connection we do the discovery so we get the targets, but on
disconnection  the code is expecting to have all the ips, iqns, and luns
information on the connection properties, so it will only flush the
multipath and disconnect volumes that were connected using the
information from the connection property.

This patch fixes this retrieving the targets information from the
discoverydb that was gathered during the connection phase.

Output from discoverydb query looks like this:

 SENDTARGETS:
 DiscoveryAddress: 192.168.1.33,3260
 DiscoveryAddress: 192.168.1.2,3260
 Target: iqn.2004-04.com.qnap:ts-831x:iscsi.cinder-20170531114245.9eff88
     Portal: 192.168.1.3:3260,1
         Iface Name: default
     Portal: 192.168.1.2:3260,1
         Iface Name: default
 Target: iqn.2004-04.com.qnap:ts-831x:iscsi.cinder-20170531114447.9eff88
     Portal: 192.168.1.3:3260,1
         Iface Name: default
     Portal: 192.168.1.2:3260,1
         Iface Name: default
 DiscoveryAddress: 192.168.1.38,3260
 iSNS:
 No targets found.
 STATIC:
 No targets found.
 FIRMWARE:
 No targets found.

[1] https://review.openstack.org/455392

Change-Id: Iada5d4fbeb07aeaf3afb953a289b6b89778c382c
Closes-Bug: #1699061
2017-06-20 15:57:22 +02:00
OpenStack Proposal Bot 455acbbcf1 Updated from global requirements
Change-Id: I51b79c64fbc61667d8a3d07bff59a50d1cf0d059
2017-06-19 19:06:27 +00:00
Jenkins 9728815feb Merge "Fix slow test_connect_volume_device_not_valid test" 2017-06-19 11:14:32 +00:00
Jenkins 84b5f71933 Merge "Prevent rbd map again if it's already mapped" 2017-06-18 07:10:38 +00:00
Gorka Eguileor f67d46c538 Add open-iscsi manual scan support
It was recently added to open-iscsi the functionality to disable
automatic LUN scans on iscsid start, on login, and on reception of
AEN/AER messages reporting LUN data has changed.

Those 3 cases were one of the causes why Nova-CPU and Cinder-Volumes
nodes would have unexpected devices.  With this new feature we can
prevent them from appearing unnexpectedly.

This patch adds the mechanism required to configure our sessions for
manual scans in a backward compatible way.

Manual scans are enabled setting `node.session.scan` to `manual`.

Change-Id: I146a74f9f79c68a89677b9b26a324e06a35886f2
2017-06-16 16:09:35 +02:00
Gorka Eguileor 56c8665d3d Refactor iSCSI connect
This patch refactors iSCSI connect code changing the approach to one
that relies primarily on sysfs, instead of CLI tools, to retrieve all
the required information: devices from the connection, multipath system
device name, multipath name, the WWN for the block devices...

By doing so, not only do we fix a good number of bugs, but we also
improve the reliability and speed of the mechanism.

A good example of improvements and benefits achieved by this patch are:

- Clean all leftovers on exceptions on a connection.

- Parallelize logins on multipath to increase speed on flaky network
  connections.

- As long as there is at least one good path when working with multipath
  we will return a multipath device instead of a single path device,
  which helps with temporary flaky connections.

- Don't use the rescan retry parameter as log in retry on multipath
  connections so both single and multipath cases are consistent.

- We no longer rely on just one device to get the wwn and look for the
  multipath.  This would be problematic with flaky connections.

- No more querying iSCSI devices for their WWN (page 0x83) removing
  delays and issue on flaky connections.

- It's no longer a problem for the mechanism the fact that a device
  exists but is not accessible.

- We use links in `/dev/disk/by-id` to get the WWID on connect, so we
  make sure there are no leftovers on disconnect, but we no longer use
  symlinks from `/dev/disk/by-path`, `/dev/disk/by-id`, or `/dev/mapper`
  to find devices.

- We no longer need to rely on the WWN to determine the multipath, we
  have the session and the LUN, so we trace the devices and from those
  we get if they belong to a multipath.

- Stop polluting the logs with unnecessary exceptions from checking if
  the node or session exist.

- Action retries will now only log the final exception instead of
  logging all the exceptions.

- Warn when a multipath could not be formed and a single device is being
  used, as performance will be degraded.

- We no longer do global rescans on single connections that could be
  bringing in unrelated and unwanted devices (`iscsiadm -T iqn -p portal
  --rescan`).

- Fix scan mechanism that would not request all scans when the same iqn
  was shareed between portals and could send a scan request to the wrong
  IP if they shared the same iqn.

- When doing single pathed connections we could end with a multipath
  because we didn't clean up unfruitful connections.

It's worth mentioning that this patch does not touch the extend
operation.

Change-Id: Ia1c47bfaa7bc3544a5feba6a8a30faf2f132b8d7
2017-06-16 16:09:35 +02:00
Jenkins 192bdfaab6 Merge "Refactor iSCSI disconnect" 2017-06-15 23:12:38 +00:00
Jenkins 0f1a14beba Merge "Fix supported connectors for Power platform" 2017-06-14 18:16:45 +00:00
Gorka Eguileor cf9eeb1a4e Fix slow test_connect_volume_device_not_valid test
This test unnecessarily takes 6 seconds to execute so this patch reduces
this time to 0.0000 seconds.

We have to mock `eventlet.greenthread.sleep` instead of `time.sleep`
because that's what is directly being used in oslo_service's
loopingcall.

TrivialFix

Change-Id: I239cfa5dbaccf5648b732d6c414792b8497acc19
2017-06-09 14:18:58 +02:00
Sean McGinnis b83838356d Add libssl to bindep
Source builds using cryptography need libssl-dev (and assorted other
platform package names) to compile. Adding package declaration for
dpkg, rpm, and suse.

Change-Id: I9238cc977529c46470479656e449ea6cef4ae5fb
Closes-bug: #1696712
2017-06-08 06:14:15 -05:00
Gorka Eguileor 400ca5d6db Refactor iSCSI disconnect
This patch refactors iSCSI disconnect code changing the approach to one
that just uses `iscsiadm -m session` and sysfs to get all the required
information: devices from the connection, multipath system device name,
multipath name, the WWN for the block devices...

By doing so, not only do we fix a good number of bugs, but we also
improve the reliability and speed of the mechanism.

A good example of improvements and benefits achieved by this patch are:

- Common code for multipath and single path disconnects.

- No more querying iSCSI devices for their WWN (page 0x83) removing
  delays and issue on flaky connections.

- All devices are properly cleaned even if they are not part of the
  multipath.

- We wait for device removal and do it in parallel if there are
  multiple.

- Removed usage of `multipath -l` to find devices which is really slow
  with flaky connections and didn't work when called with a device from
  a path that is down.

- Prevent losing data when detaching, currently if the multipath flush
  fails for any other reason than "in use" we silently continue with the
  removal.  That is the case when all paths are momentarily down.

- Adds a new mechanism for the caller of the disconnect to specify that
  it's acceptable to lose data and that it's more important to leave a
  clean system.  That is the case if we are creating a volume from an
  image, since the volume will just be set to error, but we don't want
  leftovers.  Optionally we can tell os-brick to ignore errors and don't
  raise an exception if the flush fails.

- Add a warning when we could be leaving leftovers behind due to
  disconnect issues.

- Action retries (like multipath flush) will now only log the final
  exception instead of logging all the exceptions.

- Flushes of individual paths now use exponential backoff retries
  instead of random retries between 0.2 and 2 seconds (from oslo
  library).

- We no longer use symlinks from `/dev/disk/by-path`, `/dev/disk/by-id`,
  or `/dev/mapper` to find devices or multipaths, as they could be
  leftovers from previous runs.

- With high failure rates (above 30%) some CLI calls will enter into a
  weird state where they wait forever, so we add a timeout mechanism in
  our `execute` method and add it to those specific calls.

Closes-Bug: #1502534
Change-Id: I058ff0a0e5ad517507dc3cda39087c913558561d
2017-05-31 15:31:20 +02:00
Jenkins 7fc3bf912a Merge "Updated from global requirements" 2017-05-29 00:04:42 +00:00
Jenkins 8b33e672e8 Merge "Removed invalid comments in tox.ini [flake8]" 2017-05-28 23:29:43 +00:00
Jenkins d5e76c20ed Merge "Stop ignoring H904 hacking rule in tox" 2017-05-28 23:29:38 +00:00
Jenkins 455332e3b3 Merge "Change code to be more Pythonic" 2017-05-28 23:29:32 +00:00
Jenkins 4e6a4a3ce7 Merge "encryptors: Delay removal of legacy provider names" 2017-05-28 23:06:02 +00:00
OpenStack Proposal Bot 7fb287762e Updated from global requirements
Change-Id: Ia622c7964bcee29bbaccb5993c2f3c30e766a0fb
2017-05-26 17:26:52 +00:00
Jenkins 0e692e19bc Merge "Force LUN_ID to an int" 2017-05-26 16:22:01 +00:00
Jenkins b0ed38c1f8 Merge "Stop ignoring H405 hacking rule in tox" 2017-05-26 15:51:23 +00:00
Jenkins 045ff1e3d6 Merge "Stop ignoring E265 pycodestyle rule in tox" 2017-05-26 15:46:24 +00:00
Walter A. Boring IV 1ceeabadf9 Force LUN_ID to an int
This patch just enforces that LUN_ID passed in is an int.

Change-Id: I66999e999f7d307edb25f62ef31730b4f90be0d3
Partial-Bug: #1692153
2017-05-25 19:29:34 +00:00
Jenkins d48931f639 Merge "Check host device alive before multipath id discovery" 2017-05-25 16:12:39 +00:00
Jenkins 46daeb0058 Merge "Fixed the veritas connector path" 2017-05-24 19:39:46 +00:00
Rafael Folco b66b29b7af Fix supported connectors for Power platform
Id2283195d2b6002175079070a0ff0a137a9d6806 enabled fiber channel
connector but broke all the other supported ones, including iscsi.

Change-Id: I3d6d12acf5fc864eda1bbc790bf0a310c8d0c60f
Fixes-bug: 1693336
2017-05-24 19:31:10 +00:00
Maciej Kucia 22f7500025 Removed invalid comments in tox.ini [flake8]
Change-Id: I9b12f6b1ad2612403de7c904a7752ef6960480e2
Signed-off-by: Maciej Kucia <maciej@kucia.net>
2017-05-24 00:44:57 +02:00
Maciej Kucia a14653b255 Stop ignoring H904 hacking rule in tox
This change is one in the series of changes to
align os-brick with Cinder rules.

- H904 'Wrap long lines in parentheses instead of a backslash'

See following for more information:
 https://github.com/openstack-dev/hacking

Change-Id: Ia08ce1b9cdab968c0965bc04aa7d202190cb4bb3
Signed-off-by: Maciej Kucia <maciej@kucia.net>
2017-05-24 00:44:57 +02:00
Maciej Kucia d9ac24d0d2 Stop ignoring H405 hacking rule in tox
This change is one in the series of changes to
align os-brick with Cinder rules.

- H405 'multi line docstring summary not
        separated with an empty line'

See following for more information:
 https://github.com/openstack-dev/hacking

Change-Id: Id67324c0ef22287d91b49ae11eda094bf73739a1
Signed-off-by: Maciej Kucia <maciej@kucia.net>
2017-05-24 00:44:57 +02:00
Maciej Kucia 211bda0a04 Stop ignoring E265 pycodestyle rule in tox
This change is one in the series of changes to
align os-brick with Cinder rules.

- E265 'block comment should start with "# " '

See following for more information:
 https://github.com/PyCQA/pycodestyle

Change-Id: I392d5ed179ddf1fd644cb3b53436f2aee2f47cbe
Signed-off-by: Maciej Kucia <maciej@kucia.net>
2017-05-24 00:44:30 +02:00
Maciej Kucia 32b4b24bfc Stop ignoring E123 and E125 pycodestyle rules
Remove options from flake8 section in tox.ini

This change is one in the series of changes to
align os-brick with Cinder rules.

- E123 'closing bracket does not match indentation of
        opening bracket’s line'
  This option is ignored by default as it is not PEP8 enforced

- E125 'continuation line with same indent as next logical line'

See following for more information:
 https://github.com/PyCQA/pycodestyle

Change-Id: I439e677e66d59a48d4b2ac87eb0256146ff507e0
Signed-off-by: Maciej Kucia <maciej@kucia.net>
2017-05-24 00:43:02 +02:00
Maciej Kucia 4e175333e9 Update hacking version to align with Cinder
Change-Id: I9749e222d70fd150fb6bd8df8cd747b1e3b7fcfa
Signed-off-by: Maciej Kucia <maciej@kucia.net>
2017-05-24 00:08:24 +02:00
Walter A. Boring IV 06e9d9cc97 Fixed the veritas connector path
This patch updates the return path from 'device_path' to 'path'
to be compatible with every other connector.

Change-Id: I2c1d561521fdd37626e3301148c16feff48a1fde
2017-05-23 18:32:54 +00:00
Arnon Yaari 00390448b8 Change code to be more Pythonic
Minor refactor to code that is not "Pythonic".
Using a generator comprehension with "all" can improves performance

Change-Id: I4817ddb8a2af4e1e5b8a74e3950194d23df18a0f
2017-05-22 18:17:19 +03:00