Commit Graph

77 Commits

Author SHA1 Message Date
Sean Mooney f3d48000b1 Add autopep8 to tox and pre-commit
autopep8 is a code formating tool that makes python code pep8
compliant without changing everything. Unlike black it will
not radically change all code and the primary change to the
existing codebase is adding a new line after class level doc strings.

This change adds a new tox autopep8 env to manually run it on your
code before you submit a patch, it also adds autopep8 to pre-commit
so if you use pre-commit it will do it for you automatically.

This change runs autopep8 in diff mode with --exit-code in the pep8
tox env so it will fail if autopep8 would modify your code if run
in in-place mode. This allows use to gate on autopep8 not modifying
patches that are submited. This will ensure authorship of patches is
maintianed.

The intent of this change is to save the large amount of time we spend
on ensuring style guidlines are followed automatically to make it
simpler for both new and old contibutors to work on nova and save
time and effort for all involved.

Change-Id: Idd618d634cc70ae8d58fab32f322e75bfabefb9d
2021-11-08 12:37:27 +00:00
Lee Yarwood 606d588e3e guestfs: With libguestfs >= v1.41.1 decode returned bytes to string
libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
of read_file from string to bytes.

0ee02e0117

As we don't check the version of libguestfs installed this change
handles both the original behaviour where a string is returned and the
newer behaviour by decoding any returned bytes to a string.

Closes-Bug: #1882421
Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8
2021-04-23 14:02:35 +01:00
Zuul f55f5daed8 Merge "Remove VFSLocalFS" 2021-03-16 17:33:45 +00:00
Zuul bcb78e5a02 Merge "Remove non-libguestfs file injection for libvirt" 2021-03-15 11:07:45 +00:00
Balazs Gibizer 4b32f9c9e3 Remove VFSLocalFS
The fix Iac8496065c8b6212d7edac320659444ab341b513 removed the last user
of VFSLocalFS so this patch remove the class, the related tests and all
the privsep functions that become dead code after this cleanup.

Change-Id: Ia1eb1d93d1f9699a4027b7a07107109ab9a3a29a
2021-03-03 17:55:43 +01:00
Sean Dague d06a10f096 Remove non-libguestfs file injection for libvirt
This is a security concern, as mounting filesystems on the host has
had previous CVEs around executing code on the host. libguestfs is
much safer, and is the only way we should allow this.

Some caveats came up during the discussion of the bug and this change
which are documented in the release note.

Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>

Closes-Bug: #1552042

Change-Id: Iac8496065c8b6212d7edac320659444ab341b513
2021-03-03 17:54:57 +01:00
Stephen Finucane f7e3e17991 tests: Poison os.uname
We shouldn't be looking up the architecture of the test node during
tests to ensure tests work across nodes with varying architectures.

Change-Id: I458b1db091e33c0b835e44b5a86de6c0a08f99a3
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Co-authored-by: Lee Yarwood <lyarwood@redhat.com>
2021-02-20 15:32:15 +00:00
Takashi Natsume 383e2a8bdc Remove six.text_type (1/2)
Replace six.text_type with str.
A subsequent patch will replace other six.text_type.

Change-Id: I23bb9e539d08f5c6202909054c2dd49b6c7a7a0e
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
2020-12-13 11:25:31 +00:00
Stephen Finucane 231908a7f4 hacking: Resolve W503 (line break occurred before a binary operator)
Change-Id: I6381365ff882cf23808e8dabfce41143c5e35192
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2019-06-24 14:24:06 -05:00
Matt Riedemann 2a8c0c800b Fix guestfs.set_backend_settings call
This method takes a list of strings. It was never failing
because python-libguestfs takes the input and casts it to
a list, which turns "force_tcg" into:

  ['f', 'o', 'r', 'c', 'e', '_', 't', 'c', 'g']

Which is obviously not correct.

http://libguestfs.org/guestfs.3.html#guestfs_set_backend_settings

This fixes that issue and leaves a TODO about investigating
the usage of set_backend_setting for just setting the
single force_tcg value instead of overrwriting all backend
settings.

Credit to Clark Boylan for identifying the issue.

Change-Id: I36edfaf30fd0cc234f929fd6937d07b6a1b646ae
Partial-Bug: #1735823
2019-05-18 18:53:13 +00:00
Zuul 60e6aa808d Merge "Check the return code when forcing TCG mode with libguestfs" 2018-03-12 22:59:14 +00:00
Matt Riedemann 7c30da1384 Don't launch guestfs in a thread pool if guestfs.debug is enabled
When guestfs.debug is enabled, we're handling callback events
from guestfs and logging them at debug level. When guestfs
is launched to inspect capabilities, that is currently done
in an eventlet thread pool. Because of the concurrent logging
along with the eventlet thread, we can hit an issue where eventlet
tries to switch threads and fails and then we hang the launch
call to guestfs, which hangs creating an instance.

This change simply avoids using a thread pool to launch guestfs
if guestfs.debug is True.

Change-Id: I0ffe93a031154b123c8beff96a695df5a280b935
Closes-Bug: #1737214
2017-12-08 13:15:50 -05:00
Matt Riedemann ec5c27a8af Check the return code when forcing TCG mode with libguestfs
According to the guestfs docs:

http://libguestfs.org/guestfs.3.html#guestfs_set_backend_settings

This method returns a 0 on success and -1 on failure. We suspect
we might not be calling this correctly, so log a warning if it
returns a non-zero return code.

Change-Id: I14a5969667a6f6f4ae529aaf0f6b9475fdf25496
Related-Bug: #1735823
2017-12-01 15:56:34 -05:00
Michael Still 3c7a72c213 Move blkid calls to privsep.
The same pattern as before.

Change-Id: If9aaca8dd9c9a82378807bbc5d2c157e719dab4d
blueprint: hurrah-for-privsep
2017-10-26 07:16:09 +11:00
Michael Still 8ea68a5ebe Move the dac_admin privsep code to a new location.
Having utilities that are named after the context they use that no
longer exists is going to be super confusing later. Move these to
a better place.

Change-Id: Id203aa6c02c3b486f63151b3607e928990a6ca7b
blueprint: hurrah-for-privsep
2017-09-18 23:14:11 +10:00
Michael Still f535e8bb99 First attempt at adding a privsep user to nova itself.
I don't particularly care about this use case (although the localfs
code should perhaps go away), but it was a nice contained example
of a privsep user which wasn't just calling a command line.

This patch also starts to layout what an API to the privsep'd code
might look like. For now its modelled on python's os module, because
that's where all the operations we perform are coming from.

The rootwrap configuration is cleaned up as we remove users.

Co-Authored-By: Tony Breeds <tony@bakeyournoodle.com>
Change-Id: I911cc51a226d6af29d63a7a2c69253de870073e9
2017-09-07 03:01:01 +10:00
Ngo Quoc Cuong 6c3520ac5b Remove translation of log messages
The i18n team has decided not to translate the logs because it
seems like it not very useful; operators prefer to have them in
English so that they can search for those strings on the internet.

Partially fix on nova/virt other paths will be fixed on next commits

Change-Id: Ie7821aa4a5147cdb0616741bd1a1b1fc22080440
2017-06-13 11:20:28 +07:00
Andrea Frittoli ce577442b0 Guestfs handle no passwd or group in image
When setting ownership of a file or directory, the guestfs driver
looks for the /etc/passwd and/or /etc/group files. In case they
are not found, the current driver lets the auges RuntimeError
through, which does not produce a very helpful error message.
Fixing that by handling the original exception and rasing a
Nova exception with more details in it.

Related-bug: #1646002

Change-Id: I2d15865c8be13b938e10e67c1b1b160f2a80f0c0
2016-12-09 12:48:44 +00:00
Kashyap Chamarthy e604946018 guestfs: Don't report exception if there's read access to kernel
Commit 92ae0f1 ("libvirt - Add log if libguestfs can't read host
kernel") reworks the logic of handling access to Kernel for libguestfs.
In doing that, it erroneously raises an exception when libguestfs is
_able_ to access the Kernel.

Fix it by reporting exception only when libguestfs does _not_ have
read access to the Kernel.

This was first tried by Kevin Zhao here
Ic6802650cb8f93e0d02c51e9014eb85a7e71f6fe, but is abandoned for some
reason.

This also adds a little more direction on what to do to fix this error.

Related-Bug: #1646002

Change-Id: Id6b4108e4e4af7c98b3e1bd9de3a09ad057e7b92
2016-12-01 15:00:39 +00:00
bhagyashris 046f34290d Fix invalid import order
Made corrections in import order as per OpenStack import standards [1].

[1] http://docs.openstack.org/developer/hacking/#import-order-template

TrivialFix

Change-Id: Ieb63c7fc5becc68db8d68b3e6847321e77ceeed3
2016-07-04 16:24:10 +05:30
Chung Chih, Hung 92ae0f1077 libvirt - Add log if libguestfs can't read host kernel
Host's kernel only allows a root user to have read/write permission in
ubuntu. If compute-service didn't have read permission then libguestfs
will launch image fail.

In libguestfs offical FAQ site had point out this issue, following is
the link
http://libguestfs.org/guestfs-faq.1.html#binaries
It had suggested users to change host's kernel permission. But this
action should be handled by other patch. Here only give a hint what's
going wrong.

Change-Id: I36c93610831e2935d46f7ee37f95619fe6dc1481
Related-Bug: 1413142
Closes-Bug: 1491216
2016-04-20 07:00:24 +00:00
Tracy Jones 4f79501215 config options: centralize section "guestfs"
The config options of the "nova.conf" section "guestfs"
got moved to the new central location "nova/conf/guestfs.py".

Change-Id: I9db186e7d665fa7ce05e806843592d739fdd9c7c
Implements: centralize-config-options-newton
2016-04-04 13:22:11 +01:00
Daniel P. Berrange 356e0651a6 virt: convert VFS API to use nova.virt.image.model
Convert the nova.virt.disk.vfs classes to use the
nova.virt.image.model classes instead of manually
passing a filename and image format as parameters.
This enables the libguestfs impl to now support the
configuration of RBD images.

Related-Bug: #1257674
Change-Id: I41d6af52ffed924d35ac255625488f40f7989a0a
2015-06-04 13:47:40 +01:00
Daniel P. Berrange 0ea510f556 virt: convert disk mount API to use nova.virt.image.model
Convert the nova.virt.disk.mount classes to use the
nova.virt.image.model classes instead of manually
passing a filename and image format as parameters.

Related-Bug: #1257674
Change-Id: Ibabcf48c95185ba443ef7575590eae6a63e0f8e5
2015-06-04 13:47:40 +01:00
Jenkins 2a5b1db132 Merge "Handle return code 2 from blkid calls" 2015-05-11 18:43:56 +00:00
James Page 051fced477 Handle return code 2 from blkid calls
blkid returns code 2 if the requested TYPE key is not found
for the specified device.

Ensure that this situation is handled correctly; blkid will
not return any data, so the return value will be empty.

Change-Id: I38cd7b53cdfd694c0eba1b6054d8e4c33759b0b8
Closes-Bug: #1426324
2015-05-11 14:38:36 +01:00
Qin Zhao da9671aa7f Libvirt: Use tpool to invoke guestfs api
In inspect_capabilities() of nova/virt/disk/vfs/guestfs.py, guestfs api, which
is C-extension, will hang nova-compute process when it is invoked. This problem
will result in message queue timeout error and instance booting failure. Need
to use tpool to invoke guestfs api, in order to make nova-compute switch to
other green threads before guestfs api returns.

Change-Id: I24187ba01965fb886351b78eee605bc052a94e21
Closes-Bug: 1453666
2015-05-11 15:01:38 +08:00
Davanum Srinivas 97d63d8745 Use oslo.log
Convert the use of the incubated version of the log module
to the new oslo.log library.

Sync oslo-incubator modules to update their imports as well.

Co-Authored-By: Doug Hellmann <doug@doughellmann.com>
Change-Id: Ic4932e3f58191869c30bd07a010a6e9fdcb2a12c
2015-02-22 07:56:40 -05:00
Davanum Srinivas af2d6c9576 Switch to using oslo_* instead of oslo.*
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.

This commit includes:
- using oslo_utils instead of oslo.utils
- using oslo_serialization instead of oslo.serialization
- using oslo_db instead of oslo.db
- using oslo_i18n instead of oslo.i18n
- using oslo_middleware instead of oslo.middleware
- using oslo_config instead of oslo.config
- using oslo_messaging instead of "from oslo import messaging"
- using oslo_vmware instead of oslo.vmware

Change-Id: I3e2eb147b321ce3e928817b62abcb7d023c5f13f
2015-02-06 06:03:10 -05:00
Takashi NATSUME 7cb773e247 Fix typo in nova/virt/disk/vfs/localfs.py
Fix log message typo in 'setup' method of 'VFSLocalFS' class.

Change-Id: Id1e369e4546c093edc8a94f49f00842ad65607cc
Closes-Bug: 1416970
2015-02-02 16:06:11 +09:00
Sahid Orentino Ferdjaoui c0e7cafff1 disk/vfs: introduce new option to setup
Introduces new option mount=True|False which prevents
mounting devices when setuping an instance of vfs.

Related-bug: #1358316
Change-Id: I8d2d0b44890fa5277dc9d902d12d6f9ac883f155
2015-01-21 04:18:28 -05:00
Sahid Orentino Ferdjaoui fa04e1e7df disk/vfs: introduce new method get_image_fs
Introduces a new method get_image_fs which returns
the filesystem type of an image given.

Related-bug: #1358316
Change-Id: I00e608ebcefc2db35eed23ead6e7556fe3dccb2d
2015-01-21 04:18:28 -05:00
Jenkins 62159ebe53 Merge "Replacement `_` on `_LW` in all LOG.warning part 4" 2014-12-05 19:11:17 +00:00
Matt Thompson 42b5bd686b Fix incorrectly formatted log message
Whilst troubleshooting a nova migration issue, I kept seeing this
string being logged.  This minor change fixes the log string in
nova/virt/disk/vfs/api.py to address the lack of comma and spacing.

Change-Id: I98313b600977f32edd4ec89486b1770ae54ac179
2014-12-05 12:40:26 +00:00
Mike Durnosvistov 76953c00c3 Replacement `_` on `_LW` in all LOG.warning part 4
oslo.i18n uses different marker functions to separate the
translatable messages into different catalogs, which the translation
teams can prioritize translating. For details, please refer to:
http://docs.openstack.org/developer/oslo.i18n/guidelines.html#guidelines-for-use-in-openstack

There were not marker fuctions some places in directory network.
This commit makes changes:
* Add missing marker functions
* Use ',' instead of '%' while adding variables to log messages

Change-Id: I913077d3b0fdee78e423c35b3a48137a17946a7b
2014-12-04 13:43:38 +02:00
Sahid Orentino Ferdjaoui a2f843e0c6 vfs: guestfs logging integration
Adds support to enable guestfs logging.

Change-Id: I46c5ec6606e4a2eb8277b081d0c773ceb6d36c4c
DocImpact: New group of option has been introduced "guestfs"
Closes-Bug: #1279857
2014-10-09 03:33:42 -04:00
Davanum Srinivas 323fa6fef7 Use oslo.utils
oslo.utils library now provides the functionality previously in
oslo-incubator's excutils, importutils, network_utils, strutils
timeutils, units etc. Some modules already moved to oslo.utils
will still be around since other code in nova/openstack/common/
are using it and will be removed in a subsequent commit.

Change-Id: Idc716342535fdfa680963e0e073ddb46f5f1eb34
2014-10-06 21:41:17 -04:00
Michael Still c464ed4e76 Reduce the log level for the guestfs being missing.
There are platforms where we can never reasonably expect libguestfs
to be functional (FreeBSD, libvirt / lxc, etc). Reduce the log
level of the message recording that libguestfs is unavailable to
reduce operator confusion.

As discussed on openstack-dev at:

  http://lists.openstack.org/pipermail/openstack-dev/2014-September/046910.html

Change-Id: I4bd7ed4fbffb8ac22aea4285fe74ee0d78e1581c
2014-09-24 08:44:20 +10:00
Sahid Orentino Ferdjaoui 4d6955d207 disk/vfs: make docstring conventional to python
Updates docstring to make it conventional to Python

Change-Id: I5a9383be77a639a3cc25f727f59cddfd22889d70
2014-09-15 12:46:53 +00:00
Sahid Orentino Ferdjaoui 6a0e6209ca disk/vfs: ensure guestfs capabilities
Ensures that guestfs is available and well configured. Also
when checking for capabilities this commit introduces an in-memory
flag to avoid repeating the process if succeeded.
If we are able to load guestfs but something wrong happens a fatal
error is raised.

DocImpact: We should add a not for ubuntu like systems that use
libguestfs to execute the command 'update-guestfs-appliance' for
configuring it.

Closes-Bug: #1275267
Closes-Bug: #1157922
Change-Id: I97b3a23829ea1f3aadfe08ca6448b35010d2f312
2014-09-15 12:46:47 +00:00
Jenkins de34ccfa96 Merge "Fix error in log when log exception in guestfs.py" 2014-08-30 15:40:50 +00:00
jufeng 3455825905 Fix error in log when log exception in guestfs.py
This code review: https://review.openstack.org/#/c/104262/
brings an issue when log exception in guestfs.py at
LOG.info(_LI("Unable to force TCG mode, libguestfs too old?")
To fix this issue we just need to add %s in it.

Change-Id: Ic33a6b7c2ce49e21e96a087984aba6656065d8fc
Closes-bug: #1352768
2014-08-28 10:50:13 +08:00
James Carey 298b0328d2 Remove use of str on exceptions
An exception's message can be a translatable message.  If it is, the message
can contain unicode characters which will cause str to fail.

In cases where the message is explicity needed, the use of str is replaced
with the use of six.text_type.  When an exception is used as a replacement
string in a format string, the logger correctly handles it without the
need for str, so it is removed.

In addition to the case where a translatable message contains unicode,
enabling lazy translation in the oslo.i18n library causes translatable
messages to be replaced with an object that does not support str, this
causes all calls to str on a translatable message to fail.  Thus this patch
is also needed to support blueprint: i18n-enablement.

This patch includes a hacking check for use of str() on exceptions identified
in except statements.

Change-Id: Idb426d7f710ea69b835f70d0e883e93e9b9111d2
Partially-Implements: blueprint i18n-enablement
2014-08-27 13:55:34 +01:00
ChangBo Guo(gcb) a8c8af8260 libvirt: make guestfs methods always return list of tuples
guestfs.GuestFS supports parameter python_return_dict with default
False in 1.22 (backported in 1.20). This indicates that your program
wants to receive Python dicts for methods in the API that return list
of tuples. See http://libguestfs.org/guestfs-release-notes.1.html
In a future version of libguestfs, its default value will be True,
then that will break the code, to avoid this, we'd better force
python_return_dict as False. See http://libguestfs.org/guestfs-python.3.html

This commit makes method inspect_get_mountpoints always return list
of tuples.

Change-Id: I8211cb31a7a890f86cdd818767b3d5e8cfd5bbed
2014-08-05 09:57:51 +08:00
Jon Grimm 5be2e456c0 Improve logging when python-guestfs/libguestfs isn't working
Bumped from debug to warn level, as most often refers to
missing python-guestfs package or other libguestfs misconfiguration,
and the fallback path is usually not desired.

Closes-bug: 1240339
Change-Id: I5d90a3ea109e79fe17221c001f94d49d599048ba
2014-07-30 13:59:03 +00:00
Daniel P. Berrange 8b6ea606d9 virt: force TCG with libguestfs unless KVM is enabled in libvirt
If the libvirt driver has not been configured to use KVM, then
the libguestfs module should be forced to use TCG. This is
particularly important when running Nova inside a VM, which
might claim to have VMX/SVM support when it is in fact broken.
This will avoid libguestfs hanging in such scenarios.

Resolves-bug: #1286256
Change-Id: I9316dcedd65244c60d468b270311f032b45b051f
2014-07-25 14:59:57 +01:00
Michael Still 8159e1daa2 Convert to importutils
We had some remaining users of __import__ that really deserved to
be converted to oslo's importutils.

Change-Id: I8c0c3e885151018b974025087a4abc96dd6ae77c
2014-07-23 12:18:55 +10:00
Davanum Srinivas 826aed0ec7 Use oslo.i18n
oslo.i18n provides the i18n functions that were provided by
oslo-incubator's gettextutils module. Some tests that were
using internal details of the library were removed.

Change-Id: I44cfd5552e0dd86af21073419d31622f5fdb28e0
2014-07-18 14:28:09 -04:00
Gary Kotton cfd0846498 Don't translate debug level logs in nova.virt
Our translation policy
(https://wiki.openstack.org/wiki/LoggingStandards#Log_Translation) calls
for not translating debug level logs. This is to help prioritize log
translation. Furthermore translation has a performance overhead, even if
the log isn't used (since nova doesn't support lazy translation yet).

Change-Id: I524b48f530d8afd59a067074332e3964426e4d70
2014-06-08 00:36:49 -07:00
Alex Gaynor b055e16846 Fixed many typos.
These were found using: https://github.com/intgr/topy

Change-Id: Ia4e14508c285d95ab4eaeabbde032ecc5e7c9e4b
2014-04-25 21:47:41 -07:00