libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
of read_file from string to bytes.
0ee02e0117
As we don't check the version of libguestfs installed this change
handles both the original behaviour where a string is returned and the
newer behaviour by decoding any returned bytes to a string.
Closes-Bug: #1882421
Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8
We neither need nor want these translated so remove them. A couple of
logs with unnecessary brackets or weird indentation were identified in
the process, and these are also modified. Note that exceptions must
still be translated. Refer to [1] for more details.
[1] https://docs.openstack.org/oslo.i18n/latest/user/guidelines.html
Change-Id: I4573a7c5f3a7b5716a15fbd15ad9336807843a03
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This brings in a couple of new checks which must be addressed, many of
which involve a rather large amount of changes, so these are ignored for
now. A series of follow-up changes will resolved these.
'pycodestyle' is added as a dependency rather than it being pulled in
transitively. This is necessary since we're using it in tests.
Change-Id: I35c654bd39f343417e0a1124263ff31dcd0b05c9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
I464bc2b88123a012cd12213beac4b572c3c20a56 introduced a second call to
``qemu-img`` that can easily be collapsed into one with the addition of
a new call within the disk_api.
Related-Bug: #1785827
Change-Id: Ibfd0527ed79f60282b542034d7cb97b424becba3
At present the Libvirt driver can preallocate file based disks using the
fallocate command and importantly the `-n` option. This option allocates
blocks on the filesystem past the initial EOF of a given file:
```
$ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
$ ll -lah test.img
-rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
$ du -h test.img
1.0M test.img
```
This results in a miscalculation of the total disk overcommit for file
based (excluding ploop) disks as os.path.getsize is currently used to
determine the allocated size of these disks:
```
>>> import os
>>> os.path.getsize('test.img')
0
```
Using the above example the disk overcommit would be reported as 1.0M as
the disk appears empty yet will report a potential (virtual) size of 1.0M.
However as the required blocks have already been allocated on the
filesystem the host will report disk_available_least as missing an
additional 1.0M, essentially doubling the allocation for each disk.
To correct this the allocated size of file based (excluding ploop) disks
is reported using `disk_size` from the `qemu-img info` command. This
should ensure blocks allocated past the EOF of the file are taken into
account and correctly reported as allocated.
A future change should ultimately remove the use of the `-n` option with
fallocate, however as this would not help disks that have already been
allocated this has not been included in this change to simplify backports.
Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
Closes-bug: #1764489
The same pattern as the rest of the changes. This means that privsep now
needs to let you pass flags to e2fsck, which I don't love and will remove
in a later patch.
Change-Id: I6c695c04ae586fec6adc354257638116277dda88
blueprint: hurrah-for-privsep
Nova allows deployers to configure the command line which is used to create
a filesystem of a given type. This is frankly a little bit weird, but its
also historical. Move this functionality to privsep, including doing a
dance at startup to load config flags into privsep in a hopefully secure
manner.
Honestly, all of this code should be deprecated, but that's above my pay
grade and would take time to do. Oh, and maybe deployers love it the way
it is.
Change-Id: Id8eeb21e10f98a448946f178c8c5a36e48c7cac6
blueprint: hurrah-for-privsep
This patch introduces the first code in the privsep directory which
_does_not_ run with escalated premissions. This is a requirement because
the privsep code has a restricted python path when executing. This
pattern will be used for other methods which are only sometimes
escalated.
Change-Id: Ie09e40d6476dcabda2d599e96701d419e3e8bdf0
blueprint: hurrah-for-privsep
The same pattern as previous patches. Some of these unit tests
are starting to be a bit simpler as we finish the transition.
Change-Id: If0e1fe4c0466f2f88525dc575af2ef366d4bb59d
blueprint: hurrah-for-privsep
The same pattern as the others, but with an added security concern.
Co-Authored-By: Evgeny Antyshev <eantyshev@virtuozzo.com>
Closes-Bug: #1717533
Change-Id: I1ac3a0ea4756ec68884866435c3da69171bbeb13
blueprint: hurrah-for-privsep
The i18n team has decided not to translate the logs because it
seems like it not very useful; operators prefer to have them in
English so that they can search for those strings on the internet.
Partially fix on nova/virt other paths will be fixed on next commits
Change-Id: Ie7821aa4a5147cdb0616741bd1a1b1fc22080440
String interpolation should be delayed to be handled
by the logging code, rather than being done
at the point of the logging call.
So add the following hacking rule for it.
- [N354] String interpolation should be delayed at logging calls.
See the oslo i18n guideline.
* http://docs.openstack.org/developer/oslo.i18n/guidelines.html
Change-Id: Ief6d3ee3539c0857098fffdb7acfeec3e0fed6eb
Closes-Bug: #1596829
Adapt "nova resize" code to support Virtuozzo ploop disks.
As far as ploop disks are in fact directories we add '-r' argument
to all utilities that deal with instance' disks such as cp, rsync and scp.
Thus we copy disks universally whether they are folders or files.
Also using "prl_disk_tool" instead of "qemu-img" is better for ploop images
because it resizes guest filesystem as well.
We can't resize disks from guest OS in containers,
because they are not allowed to write directly to block device.
ploop tool can resize partition table and internal filesystem,
but only for container's disks. Such disks must have only one partition
with ext filesystem.
prl_disk_tool can resize disks with internal filesystems
and doesn't require any special layout so it can resize disks
for virtual machines. So it's better to use this tool instead of ploop.
Also we make compute.filters more strict
We call "ploop" only with "restore-descriptor" argument
And we set disk size in megabytes for prl_disk_tool
Co-Authored-By: Dmitry Guryanov <dguryanov@parallels.com>
Depends-On: I04c4379459c2fc1fd4801ec2aad53d0f6053b6d6
Change-Id: I38dbf73beb01fe1939ddca63fbfedbec1dc3c826
Implements: blueprint virtuozzo-instance-resize-support
Add options from 'virt.disk'. These options are part of the
'DEFAULT' group but are included in the "nova.conf.virt" file in hope
that they can eventually be moved to their own group.
Change-Id: Ie69f5bef5f65c7ffe42adeff1defc8e86649bdba
Implements: bp centralize-config-options
Add options from 'virt.driver'. These options are part of the
'DEFAULT' group but are included in the "nova.conf.virt" file in hope
that they can eventually be moved to their own group.
Change-Id: Icf6084c156c4990ac60c778c28a48ae5ab97273a
Implements: bp centralize-config-options
Block backed images, are already presented as block devices, so
LocalBlockImages can be mounted directly instead of being linked
with loopback or nbd to another block device.
Change-Id: I5aae1e293832f7cff7bc00d793bcc86ededf5eec
Closes-bug: #1287754
There are races to setup and teardown lxc-backed instances using nbd
mounts when running tempest which result in mount failing.
This change adds some debug logging to help figure out what's going on
to see where the race is happening.
Related-Bug: #1491563
Change-Id: Ief8edef0df8da1a6ee94aafa9ceaf96da5e3211b
Nova compute raises AttributeError on creating a new instance with raw
image format leading instance to go in to error state. This bug is
introduced by commit I34e2f29e1f068636036226a7745d88640f724d93 which
added image.file in disk api instead of image.path.
Changed image.file to image.path to fix this issue.
Closes-Bug: 1470356
Change-Id: I4a12263a992eaf8abf2e90089591051d5aa57585
Convert the nova.virt.disk.api methods over to use the
nova.virt.image.model classes instead of taking a file
path and use_cow boolean as parameters.
Related-Bug: #1257674
Change-Id: I34e2f29e1f068636036226a7745d88640f724d93
Convert the nova.virt.disk.vfs classes to use the
nova.virt.image.model classes instead of manually
passing a filename and image format as parameters.
This enables the libguestfs impl to now support the
configuration of RBD images.
Related-Bug: #1257674
Change-Id: I41d6af52ffed924d35ac255625488f40f7989a0a
Convert the nova.virt.disk.mount classes to use the
nova.virt.image.model classes instead of manually
passing a filename and image format as parameters.
Related-Bug: #1257674
Change-Id: Ibabcf48c95185ba443ef7575590eae6a63e0f8e5
When resizing filesystems is disabled, the extend() function currently
accesses an image of a spawning instance to determine if it is
extendable. This check is not necessary as the image is not resized
afterwards.
When there is no method available to access the image (no libguestfs
or nbd installed), the superfluous access causes delays in the spawn
procedure.
With this fix, extend() returns if the resize should not be performed
according to the configuration before trying to access the image
in is_image_extendable().
Change-Id: Icb2f58b005539fec18b49d629ecd568d1f897e51
Closes-bug: #1438245
LOG.exception will print out the full stacktrace, that's enough.
We should only record meaningful message.
Change-Id: I8b5c581717e6b78c11c46e226463927450c9f287
Convert the use of the incubated version of the log module
to the new oslo.log library.
Sync oslo-incubator modules to update their imports as well.
Co-Authored-By: Doug Hellmann <doug@doughellmann.com>
Change-Id: Ic4932e3f58191869c30bd07a010a6e9fdcb2a12c
Resolved issues in nova code like
Function 'func_name' does not have a parameter 'param_name'
TrivialFix
Change-Id: Ie83f1eb31a0ce781430dc5a969dd52ac1f00a70c
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
This commit includes:
- using oslo_utils instead of oslo.utils
- using oslo_serialization instead of oslo.serialization
- using oslo_db instead of oslo.db
- using oslo_i18n instead of oslo.i18n
- using oslo_middleware instead of oslo.middleware
- using oslo_config instead of oslo.config
- using oslo_messaging instead of "from oslo import messaging"
- using oslo_vmware instead of oslo.vmware
Change-Id: I3e2eb147b321ce3e928817b62abcb7d023c5f13f
The ext3 filesystem does orders of magnitude more I/O than ext4 in
mkfs - a brief benchmark showed 14m to make a 1TB ext3 filesystem
vs 7 seconds to make a 1TB ext4 filesystem, with the ext3 filesystem
doing upwards of 6GB I/O.
Since ext4 has been a default in all major Linux distributions for
a long time now - RHEL5 has it as default, for instance - there is
no reason to default to ext3: users that need ext3 can configure
that themselves. So the default now is ext4 for all linux with
this commit. Also note that for non-linux and non-windows
the default used to be ext3, now it is vfat.
Note that there is still support for ext3 as the default if some
deployers do choose to keep that as the default, by using either
virt_mkfs or default_ephemeral_format configuration option. We
are just trying to move the default for everyone who is starting
fresh.
Originally logged by lifeless as I833356af81df743a63d4442d389be82bc25f3914,
reverted in I8de225729bbc77b263fcac2fa72b0e9cc6c1b2bf by phil and re-proposed
by swapnil in Ibe9e9ba4265125c5bd933cde1a93cca02511bb46
DocImpact
Closes-Bug: #1266262
Change-Id: I92cb51c6118e48fd66ada0e2fee624597c4ad8f0
This commit renames the method is_image_partitionless
to is_image_extendable which reflect better the role of it
also uses new method and option introduced in a previous
commits to reduce the unecessary among of work done.
Related-bug: #1358316
Change-Id: I491a1e3be59471dc78745c1234d95b2f68ac42b3
At some point, we should switch over from ext3 to ext4. If we have
'default', we can't tell if a file was created with ext3 or ext4
as the file system. So we need a better way.
The name format is from discussion on the mailing list [1] and
was proposed by Pádraig Brady. Once we have this in place, we
can resurrect changeset to switch over from ext3 to ext4
first proposed by lifeless in
I833356af81df743a63d4442d389be82bc25f3914 and reverted in
I8de225729bbc77b263fcac2fa72b0e9cc6c1b2bf.
[1] http://markmail.org/message/xzwt2r7acyj7n33t
Related-Bug: #1266262
Change-Id: I795d4165d668018658967661f593a9eb48652b2b
oslo.concurrency-0.3.0 has moved its path to oslo_concurrency,
the old path oslo.concurrency can still work but is deprecated now.
Change-Id: I1538dad5cbacf1e6041c571d5bc55994689c08b0
Closes-Bug: #1398656
oslo.i18n uses different marker functions to separate the
translatable messages into different catalogs, which the translation
teams can prioritize translating. For details, please refer to:
http://docs.openstack.org/developer/oslo.i18n/guidelines.html#guidelines-for-use-in-openstack
There were not marker fuctions some places in directory network.
This commit makes changes:
* Add missing marker functions
* Use ',' instead of '%' while adding variables to log messages
Change-Id: I913077d3b0fdee78e423c35b3a48137a17946a7b
oslo.i18n uses different marker functions to separate the
translatable messages into different catalogs, which the translation
teams can prioritize translating. For details, please refer to:
http://docs.openstack.org/developer/oslo.i18n/guidelines.html#guidelines-for-use-in-openstack
There were not marker fuctions some places in directory network.
This commit makes changes:
* Add missing marker functions
* Use ',' instead of '%' while adding variables to log messages
Added a hacking rule for the log exception about checking
translation for it.
Change-Id: If80ea6f177bb65afcdffce71550bb38fedcc54eb
Let's switch to the newly released oslo library for the
processutils and lockutils. We use the config fixture(s) to
specify disable_process_locking and lock_path in the CONF
variable of oslo.concurrency library for correctly setting the
flags.
The previous attempt at updating nova to use oslo.concurrency
was reverted as it stopped running any tests. lockutils-wrapper
should have been used in ini/shell scripts
instead of lockutils.py module as the the module is
no longer runnable (__main__ has been removed). In this
review we use lockutils-wrapper correctly. The test harness
has since been updated to bail out if 0 tests are run as
well for additional insurance.
Change-Id: I14c75ef2196fd08c9465d648e5cd53777bd9c7d1
Co-Authored-By: Nikola Dipanov <ndipanov@redhat.com>
Co-Authored-By: Corey Wright <corey.wright@rackspace.com>
This patch made unit tests stop running (always runs 0 unit tests) we
should revert and put a check in place to prevent that from happening
so that we don't regress the rest of Nova trying to figure out the
oslo.concurrency issue.
This reverts commit 382294762b.
Change-Id: I98b04ac2a0ba67ba8af44867799317fdf95b8aff
Very simple import change in each file, just touches a
whole lot of files. jsonutils has graduated into a stand alone
library and has been removed from the oslo-incubator repository.
We should be using the library for all projects.
Change-Id: I09d780a47d377871f357654603ee4b0fa0169d66
oslo.i18n provides the i18n functions that were provided by
oslo-incubator's gettextutils module. Some tests that were
using internal details of the library were removed.
Change-Id: I44cfd5552e0dd86af21073419d31622f5fdb28e0
Lee Yarwood