Firewall support is not needed with neutron, which supports both
security groups for per-port filtering and FWaaS for per-network
filtering. Remove both the generic firewalls and the hypervisor-specific
implementations.
This change focuses on removing the firewall-related API calls from the
various virt drivers. The firewall drivers themselves are removed
separately.
Change-Id: I5a9e5532c46a5f7064441ae644125d21efe5fda1
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
I suspect there is more to be removed here, but this is all I could spot
for the moment.
Change-Id: Ifb8dc7de5d4845dd639fba71b853d38cdb851c7d
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Replace mox stubs with stub_out in
nova/tests/unit/virt/xenapi/stubs.py.
Change-Id: I721fdc8e6cc7d36caf96b6f17150dfbcaa59c209
Implements: blueprint mox-removal
This commit is to deprecate the config of image_upload_handler
which need to be configured with a class path string. And we
should make download handler be configurable to be consistent
with the upload handler.
So it contains the following changes:
1. deprecate the option of "image_upload_handler".
2. added a new option of "image_handler" which will be used to
choose a short name from the supported image handler plugins.
The proper handler will be loaded at runtime basing on the
option setting.
TODO(jianghuaw): For the legacy handler of VdiThroughDevStore
which has no download function implemented. For downloading, it
is handled in a different way (call xenapi.vm_utils._stream_disk()).
We keep it as it is now. In the future, we should consider to
move _stream_disk to VdiThroughDevStore's download().
Change-Id: I37860e63de32d988525cdc32440fe711777263e6
Blueprint: xenapi-image-handler-option-improvement
Replace mox with mock in
nova/tests/unit/virt/xenapi/test_vmops.py.
The 'stubout_session' will be modified
at the same time when nova/tests/unit/virt/xenapi/stubs.by
is modified in another patch.
Change-Id: I192a67585d81cc76980139ee8f8f046d4029ecfe
Implements: blueprint mox-removal
CI tests frequently teardown by rolling back interface
attachments and then deleting the server, both of which
are asynchronous. If we're trying to detach an interface
on a guest that is gone, we don't need to log a traceback
exception in the logs, just let the InstanceNotFound
raise up to the ComputeManager which will handle the
error and log it appropriately.
Change-Id: I9428be0e6e5b640fdda00410817925001361fd2c
Closes-Bug: #1759979
xenapi is going to support pool-based multi-hosts OpenStack
environments, this patch is used to remove dependency to the
old aggregate-based-pools and add support to xenapi pool.
Also include unit test changes.
Updated configuring migrations document:
https://docs.openstack.org/nova/latest/admin/configuring-
migrations.html#configuring-migrations-xenserver-shared-storage
Other related documents will be updated in another patch.
Implements: blueprint live-migration-in-xapi-pool
Change-Id: I2c492c46e85c1df96aa7fdc12cdee0b1c7ba775e
The previous implementation doesn't support live migration in
pool-based multi-hosts OpenStack environments. Modify code to
support it.
In this patch we will,
1. Support live migration in XAPI pool:
-Add code to support resource pre-check before live migration.
-Add code to support resource clean after live migration.
-Add code to support using local storage in xapi pool.
Depends-On: I80007bb897a1f7eda47ebda82e08fe270f97ff40
Implements: blueprint live-migration-in-xapi-pool
Change-Id: I6437a315f1400a73f7a149bed5550f6d6dd1406f
This commit aims to correct problems with the resize_up codebase that allows
the snapshot and migration of volume backed VDI/VHDs. Since these are empty
stub disks, and the XenAPI does not allow these VDIs to be snapped, this results
in an SR_OPERATION_NOT_ALLOWED or similar error on attempt.
This change adds a check into the _process_ephemeral_chain_recursive method to
run the current userdevice through volume_utils.is_booted_from_volume. To
achieve this, the method has been opened in scope to accept custom user_device
objects. In a future commit we will need to rename this method for clarity
and correct its dependancies that call it. I have added a TODO for this to be
done by myself. The check will ensure that the userdevice is not volume backed
and then continue to snapshot and migrate the disk as needed, else increment
and move on.
Closes-Bug: #1745072
Change-Id: I7cd2977c8268c1f73062b5d0b2b68ea686db99fe
Changes within XenAPI have enforced a more strict policy when checking
assert_can_migrate. In particular when checking the source_vdi:dest_sr
mapping it insists that the SR actually exist. This is not a problem for
local disks, however this assertation is called extremely early in the
live migration process (check_can_migrate_source) which is called from
conductor, which makes a problem for attached volumes.
This early in the process the host has just barely been chosen and no SR
information has been configured yet for these volumes or their initiators.
Additionally we cannot prepare this SR any earlier as BDM information is
not set up until the pre_live_migration method. With the options to either
skip this assertion completely or swallow the exception, I have chosen to
swallow the exception. My reasons for this are two-fold:
1. --block-migration can be called without regard for whether an iSCSI
volume is attached, and we still want to ensure that VIF, CPU and other
factors are checked, and not just skip all checks entirely.
2. Currently the Assert only exists within the --block-migration code
base but this needs to change. A future commit will remove this logic
to ensure that the commit runs without this flag. Once that is done we
want to be able to continue to use this Exception swallow logic rather
than continuing to skip the assert for all XCP2.1.0+ even without volumes.
This decision should help us handle less work in a future commit and does not
seem to align with the goals of that commit, where it does align properly here.
This commit still changes very little of the current codebase and puts us in
a good position to refactor the way this is handled at a later date, while
adding a TODO note to correct VM.assert_can_migrate only running during a
block migration.
Additionally there seems to be some confusion that the mapping data that is
generated during this initial trip through _call_live_migrate_command is needed
to continue along the code, however this data appears to be purely used to send
the mapping information through the assertation call, and is then discarded.
The only data returned from these methods is the original dest_data which
is carried into the live_migration method. The _call_live_migration method is
called again during the live_migration() method, and during this time it does
need that mapping to send along to XenAPI for the actual migration, but not
yet. Because this codebase is so confusing, I am providing a little bit of
context on the movement of these variables with some psuedocode:
---CONDUCTOR.TASKS.LIVE_MIGRATE---
LiveMigrationTask.Execute()
self._find_destination() <-----
Unrelated Work
compute.live_migration(self, host, instance, destination,
block_migrate, migration, migrate_data)
LiveMigrationTask._find_destination()
Scheduler Things. Gets a Dest ref.
_check_compatible_with_source_hyp
_call_livem_checks_on_host(host) <-----
_check_can_live_migrate_destination()
returns Host Node Name and Host Compute. That's all.
---COMPUTE.MANAGER---
_do / _check_live_migration_destination
dest_check_data = xenops.can_live_migrate_destination
(Checks for the Assert)
try:
migrate_data = check_can_live_migrate_source(dest_check_data)
return migrate_data
---VMOPS--
check_can_migrate_source(self, ctxt, instance_ref, dest_check_data)
if block_migration:
_call_live_migration_command(assert_can_migrate)
_generate_vdi_map()
Does NOT return
ALSO Does NOT return
return dest_check_data
The changes made to address this issue are a fairly simple oslo_utils
version check. To pull this data I created two new host related methods
within VMops as well as a new import of oslo_config.versionutils.
I believe these methods ultimately belong in the xenapi.host class, but
for two very small methods I believed it better to avoid such a large import
to get minimal data.
Finally, an adjustment to the Fake XenAPI driver had to be made as it currently
does not include the host details beyond hostname environment in the
create_host check. The change amends the stub dictionary to include this data.
Change-Id: Ie5295564a68f877fc061376c00f3fa706370a251
Closes-Bug: #1704071
To prevent internal changes in os_xenapi from breaking nova unit
tests, we must mock os_xenapi calls (particular around session
management) rather than attempting to mock the internals of os_xenapi.
This issue was introduced when os_xenapi was split out from nova,
but the UT were not similarly abstracted
Change-Id: Ifdcddb63f64d5759640a62e8ddd3c276fa4fc743
This commit parses the allocation to check if VGPU is
allocated. If yes, maps the allocation to GPU group and
vGPU type which can be understood by XenAPI. Then creates
the vGPU for the instance.
Before booting instance with vGPU, it needs to check if the
GPU group has remaining vGPUs available. So also updates the
function of _get_vgpu_stats_in_group() to report remaining
vGPU capacity.
As we can use vGPU feature since this commit, it also includes
a release note.
blueprint: add-support-for-vgpu
Change-Id: Ie24dde0f1fd4b281d598f4040097d82ad251eb06
When using neutron polling mode, booting a VM will timeout because
nova-compute cannot receive network-vif-plugged event. This is
because we set vif['id'](i.e. neutron port uuid) to two different
OVS ports. One is Xenserver VIF, the other is tap device qvo-XXXX
which neutron ovs agent will monitor and deal with.
My previous patch https://review.openstack.org/#/c/463246/ has
deprecated the setting of 'nicira-iface-id' to XenServer. This
patch is to remove the setting of 'nicira-iface-id'.
Closes-Bug: #1732624
Change-Id: I221e89686c8e9041651f5affc4e3542d75ff55bc
Change Ie1b49a206b57219083059871f326926cc4628142 in os-xenapi
0.3.0 requires that the URL passed into the XenAPISession is
an actual URL, which means we need to fix a bunch of unit tests.
Change-Id: Ida4b8c33e8b3bbd03548648f8e57d923b255f35c
Closes-Bug: #1715217
This change makes _save_device_metadata only query the database
for the instance BlockDeviceMappingList if there are block
device mappings in the block_device_info dict. If there are none,
then there is no point in making the extra useless query to the
database.
A TODO is left in the code since this could be further optimized
if the DriverVolumeBlockDevice object would proxy the 'tag' attribute
from it's underlying BlockDeviceMapping object, but that is a bigger
change saved for another time.
The existing unit test is fixed up and another unit test is added
to validate that we don't query the database if there are no
block device mappings to process.
Change-Id: Ia879b97775125406bd4134f2438d8b988eabc41d
This patch is to change nova project to use os-xenapi 0.2.0 which
has added python interfaces for calling dom0 plugins
1. use os_xenapi.client.host_network
2. use os_xenapi.client.disk_management
3. use os_xenapi.client.vm_management
4. use os_xenapi.client.host_xenstore
5. use os_xenapi.client.host_agent
6. use os_xenapi.client.host_management
7. use os_xenapi.client.host_glance
Change-Id: Id87b83ef3ca496525f63b9c7b731e8a22c693e02
When using neutron polling mode, booting a VM will timeout because cannot
receive network-vif-plugged event. After investigation we found we set
vif['id'](i.e. neutron port uuid from neutron's perspective of view) to
two different OVS ports. One is the Xenserver VIF, the other is tap device
qvo-XXXX which neutron ovs agent will monitor and deal with.
This patch is to deprecate the setting of 'nicira-iface-id' to XenServer
VIF (which will result in 'iface-id' in ovsdb) and use 'neutron-port-id'.
Note: There is no problem if we don't use neutron-ovs-agent polling mode
Change-Id: I878602b5a84106634a211735c0002580293e3c06
The newly released XenServer 7.1 has changed to check vif_map in
its api VM.assert_can_migrate(), if the vif_map isn't set, it will
raise exception with VIF_NOT_IN_MAP, but at this point destination
compute node doesn't have dest_vif_map, this patch is to make code
in xenserver driver be compatible with xenserver's changes
Closes-bug: 1669719
Change-Id: I8fb8d26fbbc12dd4f17b3541968fc16254131b6c
Remove mox from GetConsoleOutputTestCase in
nova/tests/unit/virt/xenapi/test_vmops.py
Part of blueprint remove-mox-pike
Change-Id: I3bc75d00d368a9c229d312ed6525fa8ddd5d4844
Live migration with XenServer as hypervisor failed with xapi
errors "VIF_NOT_IN_MAP". There are two reasons for this
problem:
(1) Before XS7.0, it supports VM live migration without
setting vif_ref and network_ref explicitly if the destination
host has same network, but since XS7.0, it doesn't support
this way, we must give vif_ref and network_ref mapping.
(2) In nova, XenServer has introduced interim network for
fixing ovs updating wrong port in neutron, see bug 1268955
and also interim network can assist support neutron security
group (linux bridge) as we cannot make VIF connected to
linux bridge directly via XAPI
To achieve this, we will add {src_vif_ref: dest_network_ref}
mapping information, in pre_live_migration, we first create
interim network in destination host and store
{neutron_vif_uuid: dest_network_ref} in migrate_data, then in
source host, before live_migration, we will calculate the
{src_vif_ref: dest_network_ref} and set it as parameters to
xapi when calling VM.migrate_send. Also, we will handle the
case where the destination host is running older code that
doesn't have this new src_vif_ref mapping, like live migrating
from an Ocata compute node to a Newton compute node.
Closes-bug: 1658877
Change-Id: If0fb5d764011521916fbbe15224f524a220052f3
During VM rescue tests, we found nova xenserver driver failed due
to waiting vif-plug-event from neutron timeout. when checking
nova and neutron logs, we found there are several mistakes in
nova driver:
(1) After several rounds of rescuing/unrescuing, it will wait for
vif-plug-event, but actually, it shouldn't wait for such event
(2) Checking neutron log, we found the port status sometimes will
change during rescuing/unrescuing, which also shouldn't happen
(3) Checking nova related code, we found each time when booting a
VM, it will delete and create the tap device, which is used by
neutron security group, this delete/re-create action will cause
the port status change which shouldn't be changed.
(4) When adding/deleting security groups to VM's port, it will
trigger the port status change, e.g. from ACTIVE to BUILDING, but
under rescue scenario, we also depends on VIF's status to determine
whether waiting for vif plug event is not appropriate.
This patch is to fix the above problem and there is another patch
to enable the exclude rescue tests to test this fix
https://review.openstack.org/#/c/416197/
Closes-Bug: #1651650
Change-Id: I32c66733330bc9877caea7e2a2290c02b3906708
Nova already has support for hotplugging neutorn ports in the libvirt
driver. This extends the support to the XenAPI driver, implement
attaching/detaching VIFs
I have made a patch to run releated testcase in xenserver CI
https://review.openstack.org/#/c/416797/
Change-Id: I22f3fe52d07100592015007653c7f8c47c25d22c
Implements: blueprint xenapi-vif-hotplug
XenServer has released os-xenapi lib on pypi, this patch is to
use os-xenapi in nova project. In this patch, we only change
the usage of XenAPI to os-xenapi and fix unit tests. In the next
patch, we will delete those unused files.
Partially-Implements: blueprint add-os-xenapi-library
Depends-On: Ic327135b893a77672fd42af919f47f181e932773
Change-Id: I424dfcd507c4b7fbeea5245cc1c234dec01d5781
The main issue in the xenapi unit tests was faking out network_info
to be an unrealistic string, it must be a list of dicts. There was
also a neutron-specific test which wasn't actually running against
neutron before this, so this fixes that test to behave properly.
Part of blueprint use-neutron-by-default
Change-Id: I3a817ada790b9aa827c7844a2e249dd061ed3936
With XenServer 7.0 (XCP 2.1.0), VM live migration with iSCSI volume
will fail due to no relax_xsm_sr_check in /etc/xapi.conf in Dom0.
But since XS7.0, the default value of relax_xsm_sr_check is true if
not configured in /etc/xapi.conf in Dom0.
This fix is to add checks on XCP version, if it's equal or greater
than 2.1.0, will treat relax_xsm_sr_check=true if not found in
/etc/xapi.conf
Change-Id: I88d1d384ab7587c428e517d184258bb517dfb4ab
Closes-bug: #1610932
Fixes unit tests related to incomparable types.
Fixes unit tests related to writing / reading bytes in / from files
or response bodies.
Fixes console websocketproxy unit tests.
Fixes consoleauth unit tests.
Fixes db unit tests.
Fixes image unit tests.
Fixes virt/block_devices unit tests.
Fixes virt/configdrive unit tests.
Fixes virt_drivers unit tests.
Fixes xenapi agent, vmops and volume_utils unit tests.
Fixes wsgi unit tests.
Enables some python3 unit tests.
Partially Implements: blueprint goal-python35
Change-Id: Ie98e968740d1015eae3278edeb93d4ba08155169
The driver function pre_live_migration() has only 1 caller:
ComputeManager.pre_live_migration(), which always passes a
LiveMigrateData object to the driver. This is easy to prove as
ComputeManager.pre_live_migration() explicitly checks if migrate_data
is an object, and assigns it the return value of
LiveMigrateData.detect_implementation() if not.
detect_implementation() can only return a subclass of LiveMigrateData.
It will raise an exception if it is passed None.
This change removes handling of the case where migrate_data is None or
otherwise not a LiveMigrateData object from all driver implementations
of pre_live_migration().
Change-Id: I9d46019e06dc2b59f7b2ddf7c57c333809cbc94b
This patch adds support for passing the user context through to
vendordata2 to all the other drivers. libvirt got this in the
original patch, so this is a catch up for the others.
Change-Id: I1e1f0aa1a41626970ae3291caaf8d1d1661f68e6
Blueprint: vendordata-reboot
Many places in tests, wrong fake UUID is used, which
pollutes the test log with the deprecation warnings.
In this patchset, I fixed them in the unit tests.
Change-Id: Ifd6a8bfa05a84d1d540e4382e9561c590b731044
Partial-Bug: #1606647
This makes the xenapi driver use instance.flavor.root_gb instead
of the legacy instance.root_gb field (and friends).
The actual change here is very small, but a lot of xenapi tests
around this needed to be object-ified.
Change-Id: I6a45a22207a44e57506bc2171b10ba1df76277e0
In particular, we were expecting the domid to be an integer
when comparing against -1 in vmops.py to decide if the domain
was shut down. This test always failed, so domid of -1 was
assumed to be a live domain.
Fix all places where domid is an integer to ensure only string
comparisons are done.
If the domid is '-1' then we raise an exception based on the
instance. An instance is always passed, so make it clear that
this is non-optional
Change-Id: I8c0d3274fbf68351a1b077e791e50e449ea0af99
Closes-Bug: 1592849
The py34 check jobs are failing occasionally when trying to use mox on
the vm_utils.lookup method, seemingly because it is not restored
properly at some point. Rather than chase that down this patch replaces
the usage of mox with mock for patching this method.
Change-Id: I3f36b72ab435dea224eb61d449e13ae80d9b61a8
Closes-Bug: 1590896
Currently, unit testcases of virt emit FutureWarnings
from oslo.versionedobjects about invalid uuids as documented here[1].
This commit changes these tests to use valid uuids so
that when they are run these warnings are no longer emitted.
Fake constants are defined for project_id and user_id as they
are used without '-' at keystone side.
[1] http://docs.openstack.org/developer/oslo.versionedobjects/api/fields.html#oslo_versionedobjects.fields.UUIDField
Partial-Bug: #1557378
Change-Id: If69412eddb5e64cb8ccc5dce9176d19382c0b3f1
Xenapi driver doesn't disconnect block device volumes if any of them
failed to connect. Block 'except' do nothing because of incorrect
second argument in method 'forget_sr'.
Change-Id: I53f07616df547ef357c10abd4f28394b321f5044
Currently pre_live_migration breaks the chain of passing the migrate
data between the different driver calls.
Also, the place that was using sr_uuid_map was not looking for it in the
migrate data object, where it now lives.
Related-Bug: #1557585
Change-Id: I6f210d876816dc59325f3c36c75c2df5c880c61e
AggregateList now returns [] if the host is not in an
aggregate. This was causing an IndexError in XenAPI.
This change also replaces the AggregateHostNotFound exception,
which is not caught properly further up the stack, with the
MigrationPreCheckError as this is a suitable substitute since
the aggregate check is only part of the pre migration check.
Probably introduced by Ibcb896b2eef673bce23491161ff394bcc87aa541
Related-Bug: #1557585
Change-Id: Icacd3cbe065ee8598e5e5f03bd73463b95ed2f75
When we try to call get_console_output(), we may face console is not
available error, raise it up and let API raise HTTPNotFound error.
Closes-Bug: #1536513
Change-Id: I718d2168f353fb127a6d217b50fa595a66a1084a
When booting an instance, nova and neutron has race condition because
nova don't know whether vif(port) is ready in neutron. There is a
mechenism that letting neutron notify nova when port status changed
from down to active. This fix is for xen driver to add usage of this
event notification to avoid race condition
Closes-Bug: #1512955
Change-Id: I77be3bb728db72e01701c94ee292fa0f237358ed
Convert xmlrpc-unmarshallable CoercedDict from DictOfStringsField
object field to marshallable built-in dict.
Upstream commit f710f709 changed migrate_send_data to a
DictOfStringsField in the XenapiLiveMigrateData object, which produces
a CoercedDict, and upstream commit 69e01758 changed the handling of
migrate_send_data in the xenapi virt driver, but didn't guard against
migrate_send_data from being passed to xmlrpc, by way of XenAPI, as
the unmarshallable CoercedDict, so convert to a standard dict.
Technically, migrate_send_data is a nullable object field and can
produce None, but that's invalid at this point in the code as the
XenAPI commands VM.assert_can_migrate and VM.migrate_send require it,
so throw an InvalidParameterValue exception and correct the unit tests
with incorrect test data (ie block_migration=True, but
migrate_send_data=None) that are accidentally triggering this
exception.
Change-Id: Ia28f9af8087dd45e9159264c3510637f6cf24be1
Closes-Bug: #1550795
Stephen Finucane