autopep8 is a code formating tool that makes python code pep8
compliant without changing everything. Unlike black it will
not radically change all code and the primary change to the
existing codebase is adding a new line after class level doc strings.
This change adds a new tox autopep8 env to manually run it on your
code before you submit a patch, it also adds autopep8 to pre-commit
so if you use pre-commit it will do it for you automatically.
This change runs autopep8 in diff mode with --exit-code in the pep8
tox env so it will fail if autopep8 would modify your code if run
in in-place mode. This allows use to gate on autopep8 not modifying
patches that are submited. This will ensure authorship of patches is
maintianed.
The intent of this change is to save the large amount of time we spend
on ensuring style guidlines are followed automatically to make it
simpler for both new and old contibutors to work on nova and save
time and effort for all involved.
Change-Id: Idd618d634cc70ae8d58fab32f322e75bfabefb9d
libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
of read_file from string to bytes.
0ee02e0117
As we don't check the version of libguestfs installed this change
handles both the original behaviour where a string is returned and the
newer behaviour by decoding any returned bytes to a string.
Closes-Bug: #1882421
Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8
The fix Iac8496065c8b6212d7edac320659444ab341b513 removed the last user
of VFSLocalFS so this patch remove the class, the related tests and all
the privsep functions that become dead code after this cleanup.
Change-Id: Ia1eb1d93d1f9699a4027b7a07107109ab9a3a29a
This is a security concern, as mounting filesystems on the host has
had previous CVEs around executing code on the host. libguestfs is
much safer, and is the only way we should allow this.
Some caveats came up during the discussion of the bug and this change
which are documented in the release note.
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Closes-Bug: #1552042
Change-Id: Iac8496065c8b6212d7edac320659444ab341b513
We shouldn't be looking up the architecture of the test node during
tests to ensure tests work across nodes with varying architectures.
Change-Id: I458b1db091e33c0b835e44b5a86de6c0a08f99a3
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Co-authored-by: Lee Yarwood <lyarwood@redhat.com>
Replace six.text_type with str.
A subsequent patch will replace other six.text_type.
Change-Id: I23bb9e539d08f5c6202909054c2dd49b6c7a7a0e
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
This method takes a list of strings. It was never failing
because python-libguestfs takes the input and casts it to
a list, which turns "force_tcg" into:
['f', 'o', 'r', 'c', 'e', '_', 't', 'c', 'g']
Which is obviously not correct.
http://libguestfs.org/guestfs.3.html#guestfs_set_backend_settings
This fixes that issue and leaves a TODO about investigating
the usage of set_backend_setting for just setting the
single force_tcg value instead of overrwriting all backend
settings.
Credit to Clark Boylan for identifying the issue.
Change-Id: I36edfaf30fd0cc234f929fd6937d07b6a1b646ae
Partial-Bug: #1735823
When guestfs.debug is enabled, we're handling callback events
from guestfs and logging them at debug level. When guestfs
is launched to inspect capabilities, that is currently done
in an eventlet thread pool. Because of the concurrent logging
along with the eventlet thread, we can hit an issue where eventlet
tries to switch threads and fails and then we hang the launch
call to guestfs, which hangs creating an instance.
This change simply avoids using a thread pool to launch guestfs
if guestfs.debug is True.
Change-Id: I0ffe93a031154b123c8beff96a695df5a280b935
Closes-Bug: #1737214
According to the guestfs docs:
http://libguestfs.org/guestfs.3.html#guestfs_set_backend_settings
This method returns a 0 on success and -1 on failure. We suspect
we might not be calling this correctly, so log a warning if it
returns a non-zero return code.
Change-Id: I14a5969667a6f6f4ae529aaf0f6b9475fdf25496
Related-Bug: #1735823
Having utilities that are named after the context they use that no
longer exists is going to be super confusing later. Move these to
a better place.
Change-Id: Id203aa6c02c3b486f63151b3607e928990a6ca7b
blueprint: hurrah-for-privsep
I don't particularly care about this use case (although the localfs
code should perhaps go away), but it was a nice contained example
of a privsep user which wasn't just calling a command line.
This patch also starts to layout what an API to the privsep'd code
might look like. For now its modelled on python's os module, because
that's where all the operations we perform are coming from.
The rootwrap configuration is cleaned up as we remove users.
Co-Authored-By: Tony Breeds <tony@bakeyournoodle.com>
Change-Id: I911cc51a226d6af29d63a7a2c69253de870073e9
The i18n team has decided not to translate the logs because it
seems like it not very useful; operators prefer to have them in
English so that they can search for those strings on the internet.
Partially fix on nova/virt other paths will be fixed on next commits
Change-Id: Ie7821aa4a5147cdb0616741bd1a1b1fc22080440
When setting ownership of a file or directory, the guestfs driver
looks for the /etc/passwd and/or /etc/group files. In case they
are not found, the current driver lets the auges RuntimeError
through, which does not produce a very helpful error message.
Fixing that by handling the original exception and rasing a
Nova exception with more details in it.
Related-bug: #1646002
Change-Id: I2d15865c8be13b938e10e67c1b1b160f2a80f0c0
Commit 92ae0f1 ("libvirt - Add log if libguestfs can't read host
kernel") reworks the logic of handling access to Kernel for libguestfs.
In doing that, it erroneously raises an exception when libguestfs is
_able_ to access the Kernel.
Fix it by reporting exception only when libguestfs does _not_ have
read access to the Kernel.
This was first tried by Kevin Zhao here
Ic6802650cb8f93e0d02c51e9014eb85a7e71f6fe, but is abandoned for some
reason.
This also adds a little more direction on what to do to fix this error.
Related-Bug: #1646002
Change-Id: Id6b4108e4e4af7c98b3e1bd9de3a09ad057e7b92
Host's kernel only allows a root user to have read/write permission in
ubuntu. If compute-service didn't have read permission then libguestfs
will launch image fail.
In libguestfs offical FAQ site had point out this issue, following is
the link
http://libguestfs.org/guestfs-faq.1.html#binaries
It had suggested users to change host's kernel permission. But this
action should be handled by other patch. Here only give a hint what's
going wrong.
Change-Id: I36c93610831e2935d46f7ee37f95619fe6dc1481
Related-Bug: 1413142
Closes-Bug: 1491216
The config options of the "nova.conf" section "guestfs"
got moved to the new central location "nova/conf/guestfs.py".
Change-Id: I9db186e7d665fa7ce05e806843592d739fdd9c7c
Implements: centralize-config-options-newton
Convert the nova.virt.disk.vfs classes to use the
nova.virt.image.model classes instead of manually
passing a filename and image format as parameters.
This enables the libguestfs impl to now support the
configuration of RBD images.
Related-Bug: #1257674
Change-Id: I41d6af52ffed924d35ac255625488f40f7989a0a
Convert the nova.virt.disk.mount classes to use the
nova.virt.image.model classes instead of manually
passing a filename and image format as parameters.
Related-Bug: #1257674
Change-Id: Ibabcf48c95185ba443ef7575590eae6a63e0f8e5
blkid returns code 2 if the requested TYPE key is not found
for the specified device.
Ensure that this situation is handled correctly; blkid will
not return any data, so the return value will be empty.
Change-Id: I38cd7b53cdfd694c0eba1b6054d8e4c33759b0b8
Closes-Bug: #1426324
In inspect_capabilities() of nova/virt/disk/vfs/guestfs.py, guestfs api, which
is C-extension, will hang nova-compute process when it is invoked. This problem
will result in message queue timeout error and instance booting failure. Need
to use tpool to invoke guestfs api, in order to make nova-compute switch to
other green threads before guestfs api returns.
Change-Id: I24187ba01965fb886351b78eee605bc052a94e21
Closes-Bug: 1453666
Convert the use of the incubated version of the log module
to the new oslo.log library.
Sync oslo-incubator modules to update their imports as well.
Co-Authored-By: Doug Hellmann <doug@doughellmann.com>
Change-Id: Ic4932e3f58191869c30bd07a010a6e9fdcb2a12c
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
This commit includes:
- using oslo_utils instead of oslo.utils
- using oslo_serialization instead of oslo.serialization
- using oslo_db instead of oslo.db
- using oslo_i18n instead of oslo.i18n
- using oslo_middleware instead of oslo.middleware
- using oslo_config instead of oslo.config
- using oslo_messaging instead of "from oslo import messaging"
- using oslo_vmware instead of oslo.vmware
Change-Id: I3e2eb147b321ce3e928817b62abcb7d023c5f13f
Introduces new option mount=True|False which prevents
mounting devices when setuping an instance of vfs.
Related-bug: #1358316
Change-Id: I8d2d0b44890fa5277dc9d902d12d6f9ac883f155
Introduces a new method get_image_fs which returns
the filesystem type of an image given.
Related-bug: #1358316
Change-Id: I00e608ebcefc2db35eed23ead6e7556fe3dccb2d
Whilst troubleshooting a nova migration issue, I kept seeing this
string being logged. This minor change fixes the log string in
nova/virt/disk/vfs/api.py to address the lack of comma and spacing.
Change-Id: I98313b600977f32edd4ec89486b1770ae54ac179
oslo.i18n uses different marker functions to separate the
translatable messages into different catalogs, which the translation
teams can prioritize translating. For details, please refer to:
http://docs.openstack.org/developer/oslo.i18n/guidelines.html#guidelines-for-use-in-openstack
There were not marker fuctions some places in directory network.
This commit makes changes:
* Add missing marker functions
* Use ',' instead of '%' while adding variables to log messages
Change-Id: I913077d3b0fdee78e423c35b3a48137a17946a7b
Adds support to enable guestfs logging.
Change-Id: I46c5ec6606e4a2eb8277b081d0c773ceb6d36c4c
DocImpact: New group of option has been introduced "guestfs"
Closes-Bug: #1279857
oslo.utils library now provides the functionality previously in
oslo-incubator's excutils, importutils, network_utils, strutils
timeutils, units etc. Some modules already moved to oslo.utils
will still be around since other code in nova/openstack/common/
are using it and will be removed in a subsequent commit.
Change-Id: Idc716342535fdfa680963e0e073ddb46f5f1eb34
There are platforms where we can never reasonably expect libguestfs
to be functional (FreeBSD, libvirt / lxc, etc). Reduce the log
level of the message recording that libguestfs is unavailable to
reduce operator confusion.
As discussed on openstack-dev at:
http://lists.openstack.org/pipermail/openstack-dev/2014-September/046910.html
Change-Id: I4bd7ed4fbffb8ac22aea4285fe74ee0d78e1581c
Ensures that guestfs is available and well configured. Also
when checking for capabilities this commit introduces an in-memory
flag to avoid repeating the process if succeeded.
If we are able to load guestfs but something wrong happens a fatal
error is raised.
DocImpact: We should add a not for ubuntu like systems that use
libguestfs to execute the command 'update-guestfs-appliance' for
configuring it.
Closes-Bug: #1275267
Closes-Bug: #1157922
Change-Id: I97b3a23829ea1f3aadfe08ca6448b35010d2f312
This code review: https://review.openstack.org/#/c/104262/
brings an issue when log exception in guestfs.py at
LOG.info(_LI("Unable to force TCG mode, libguestfs too old?")
To fix this issue we just need to add %s in it.
Change-Id: Ic33a6b7c2ce49e21e96a087984aba6656065d8fc
Closes-bug: #1352768
An exception's message can be a translatable message. If it is, the message
can contain unicode characters which will cause str to fail.
In cases where the message is explicity needed, the use of str is replaced
with the use of six.text_type. When an exception is used as a replacement
string in a format string, the logger correctly handles it without the
need for str, so it is removed.
In addition to the case where a translatable message contains unicode,
enabling lazy translation in the oslo.i18n library causes translatable
messages to be replaced with an object that does not support str, this
causes all calls to str on a translatable message to fail. Thus this patch
is also needed to support blueprint: i18n-enablement.
This patch includes a hacking check for use of str() on exceptions identified
in except statements.
Change-Id: Idb426d7f710ea69b835f70d0e883e93e9b9111d2
Partially-Implements: blueprint i18n-enablement
guestfs.GuestFS supports parameter python_return_dict with default
False in 1.22 (backported in 1.20). This indicates that your program
wants to receive Python dicts for methods in the API that return list
of tuples. See http://libguestfs.org/guestfs-release-notes.1.html
In a future version of libguestfs, its default value will be True,
then that will break the code, to avoid this, we'd better force
python_return_dict as False. See http://libguestfs.org/guestfs-python.3.html
This commit makes method inspect_get_mountpoints always return list
of tuples.
Change-Id: I8211cb31a7a890f86cdd818767b3d5e8cfd5bbed
Bumped from debug to warn level, as most often refers to
missing python-guestfs package or other libguestfs misconfiguration,
and the fallback path is usually not desired.
Closes-bug: 1240339
Change-Id: I5d90a3ea109e79fe17221c001f94d49d599048ba
If the libvirt driver has not been configured to use KVM, then
the libguestfs module should be forced to use TCG. This is
particularly important when running Nova inside a VM, which
might claim to have VMX/SVM support when it is in fact broken.
This will avoid libguestfs hanging in such scenarios.
Resolves-bug: #1286256
Change-Id: I9316dcedd65244c60d468b270311f032b45b051f
We had some remaining users of __import__ that really deserved to
be converted to oslo's importutils.
Change-Id: I8c0c3e885151018b974025087a4abc96dd6ae77c
oslo.i18n provides the i18n functions that were provided by
oslo-incubator's gettextutils module. Some tests that were
using internal details of the library were removed.
Change-Id: I44cfd5552e0dd86af21073419d31622f5fdb28e0
Our translation policy
(https://wiki.openstack.org/wiki/LoggingStandards#Log_Translation) calls
for not translating debug level logs. This is to help prioritize log
translation. Furthermore translation has a performance overhead, even if
the log isn't used (since nova doesn't support lazy translation yet).
Change-Id: I524b48f530d8afd59a067074332e3964426e4d70