This expects a name and metadata argument in 2.0. Add them.
Change-Id: Id9d9ac4a2302ab769be679a39e5a339a483faa16
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This appears to be the new way to do things.
Change-Id: Ieec4d62b3ef7614fdd2ebf57aa5bc1763bdf7b5a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib. Most of this
is autogenerated, as described below, but there is one manual change
necessary:
nova/tests/functional/regressions/test_bug_1781286.py
We need to avoid using 'fixtures.MockPatch' since fixtures is using
'mock' (the library) under the hood and a call to 'mock.patch.stop'
found in that test will now "stop" mocks from the wrong library. We
have discussed making this configurable but the option proposed isn't
that pretty [1] so this is better.
The remainder was auto-generated with the following (hacky) script, with
one or two manual tweaks after the fact:
import glob
for path in glob.glob('nova/tests/**/*.py', recursive=True):
with open(path) as fh:
lines = fh.readlines()
if 'import mock\n' not in lines:
continue
import_group_found = False
create_first_party_group = False
for num, line in enumerate(lines):
line = line.strip()
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
for lib in (
'ddt', 'six', 'webob', 'fixtures', 'testtools'
'neutron', 'cinder', 'ironic', 'keystone', 'oslo',
):
if lib in tokens[1]:
create_first_party_group = True
break
if create_first_party_group:
break
import_group_found = True
if not import_group_found:
continue
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
if tokens[1] > 'unittest':
break
elif tokens[1] == 'unittest' and (
len(tokens) == 2 or tokens[4] > 'mock'
):
break
elif not line:
break
if create_first_party_group:
lines.insert(num, 'from unittest import mock\n\n')
else:
lines.insert(num, 'from unittest import mock\n')
del lines[lines.index('import mock\n')]
with open(path, 'w+') as fh:
fh.writelines(lines)
Note that we cannot remove mock from our requirements files yet due to
importing pypowervm unit test code in nova unit tests. This library
still uses the mock lib, and since we are importing test code and that
lib (correctly) only declares mock in its test-requirements.txt, mock
would not otherwise be installed and would cause errors while loading
nova unit test code.
[1] https://github.com/testing-cabal/fixtures/pull/49
Change-Id: Id5b04cf2f6ca24af8e366d23f15cf0e5cac8e1cc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This patch fix test_default_logging test.
The test validates that we have two logging handlers:
1 x to display default messages (info, error, warnings...)
1 x to redirect debug messages to null and so don't display them.
However, if OS_DEBUG=True is set in a shell session, then the test is
run and fails. Because, in debug mode, we should have only one handler
to display all messages. (look at comments for more details and
test_debug_logging test).
To fix the test, we explicitly set OS_DEBUG=0 when running
test_default_logging, so it will ensure we have two handlers whatever
OS_DEBUG value.
Co-authored-by: Rene Ribaud <rribaud@redhat.com>
Closes-Bug: #1964497
Change-Id: I7c0151d988c538dd2d083aab4b3e18ddb8151045
Resolve the following RemovedIn20Warning warning:
Passing a string to Connection.execute() is deprecated and will be
removed in version 2.0. Use the text() construct, or the
Connection.exec_driver_sql() method to invoke a driver-level SQL
string.
Change-Id: I44d6bf1ebfaf24f00a21389364456bceaae7c4d1
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This patch apply the following changes to the fixture to make the
intention clearer and remove some unnecessary complexity:
* the fixture does a lot of dynamic thing in its __init__, these are
moved to setUp() instead to facilitate proper reset functionality of
the fixture
* the caching and applying of the DB schema is made a explicit and
moved to setUp() too
* the explicit reset() function is removed as it is probably
unintentionally overwrote the Fixture.reset(). Now the Fixture can be
properly reset by calling the Fixture.reset() which is by default
implemented by calling cleanUp() and setUp()
Change-Id: Ic58e93d6aafb88be4abeb6e52089f7ee43d8db01
Introduce a new 'nova.db.api.api' module to hold API database-specific
helpers, plus a generic 'nova.db.utils' module to hold code suitable for
both main and API databases. This highlights a level of complexity
around connection management that is present for the main database but
not for the API database. This is because we need to handle the
complexity of cells for the former but not the latter.
Change-Id: Ia5304c552ce552ae3c5223a2bfb3a9cd543ec57c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Merge these, removing an unnecessary layer of abstraction, and place
them in the new 'nova.db.main' directory. The resulting change is huge,
but it's mainly the result of 's/sqlalchemy import api/main import api/'
and 's/nova.db.api/nova.db.main.api/' with some necessary cleanup. We
also need to rework how we do the blocking of API calls since we no
longer have a 'DBAPI' object that we can monkey patch as we were doing
before. This is now done via a global variable that is set by the 'main'
function of 'nova.cmd.compute'.
The main impact of this change is that it's no longer possible to set
'[database] use_db_reconnect' and have all APIs automatically wrapped in
a DB retry. Seeing as this behavior is experimental, isn't applied to
any of the API DB methods (which don't use oslo.db's 'DBAPI' helper),
and is used explicitly in what would appear to be the critical cases
(via the explicit 'oslo_db.api.wrap_db_retry' decorator), this doesn't
seem like a huge loss.
Change-Id: Iad2e4da4546b80a016e477577d23accb2606a6e4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This can go into the 'nova.tests.fixture' module also.
Change-Id: I31743a9e813e69f2342bcb148f632c8729ac7000
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Compact Mitaka database migrations into a single migration,
'319_mitaka.py'.
Users will now need to update to Mitaka before updating to Newton or
later.
Specific changes include:
- Add 'parent_addr' column to 'pci_devices' table
- Add 'resource_providers', 'inventories', and 'allocations' tables
- Add 'memory_total', 'memory_processed', 'memory_remaining', 'disk_total',
'disk_processed', and 'disk_remaining' columns to 'migrations' table
- Add 'disk_allocation_ratio' column to 'compute_nodes' table
- Add 'uuid' column and corresponding index to 'aggregates' table
- Add 'name', 'generation' and 'can_host' columns to
'resource_providers' table
- Add unique constraint on 'name' column of 'resource_providers' table
- Add index on 'name' column of 'resource_providers' table
- Add 'resource_provider_aggregates' table
Some tests that depended on some of these migrations are removed.
When testing, the previous base version was 301. It is now 318.
Change-Id: Ib0c9229899e9a83992757270503e0e1a988f2402
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Nova services only support computes that are not older than
the previous major release. This patch introduces a check in the
service startup that prevents staring the service if too old computes
are detected.
Change-Id: Ie15ec8299ae52ae8f5334d591ed3944e9585cf71
This rather beefy (but also quite simple) patch replaces the
'stub_out_image_service' call and associated cleanup in all functional
tests with a new 'GlanceFixture', based on the old 'FakeImageService'.
The use of a fixture means we don't have to worry about teardown and
allows us to stub Glance in the same manners as Cinder, Neutron,
Placement etc.
Unit test cleanup is handled in a later patch.
Change-Id: I6daea47988181dfa6dde3d9c42004c0ecf6ae87a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
DatabaseAtVersion causes the transaction context manager of its
database to be started, but doesn't set SESSION_CONFIGURED as used by
the Database fixture. Consequently, a subsequent test using the
Database fixture will attempt to call configure on a started context
manager, and fail.
We fix this by rolling the functionality of DatabaseAtVersion into
Database.
Change-Id: Ia6b63e0d26df53e611a5a36e5a2eeb09a73e4704
Closes-bug: #1804811
At some point in the past, there was only nova-network and its code
could be found in 'nova.network'. Neutron was added and eventually found
itself (mostly!) in the 'nova.network.neutronv2' submodule. With
nova-network now gone, we can remove one layer of indirection and move
the code from 'nova.network.neutronv2' back up to 'nova.network',
mirroring what we did with the old nova-volume code way back in 2012
[1]. To ensure people don't get nova-network and 'nova.network'
confused, 'neutron' is retained in filenames.
[1] https://review.opendev.org/#/c/14731/
Change-Id: I329f0fd589a4b2e0426485f09f6782f94275cc07
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Finish the job by removing all the now-unused modules. This also allows
us to - wait for it - kill mox at long last. It's a great day in the
parish.
Partial-Implements: blueprint remove-nova-network-ussuri
Partial-Implements: blueprint mox-removal-ussuri
Change-Id: Ia33ec2604b2fc2d3b6830b596cac669cc3ad6c96
This addresses a NOTE/TODO from change
I3cf6eb4654663865d9258c38f05cd05974ffcf9d where the NeutronFixture is
storing "admin-ness" from the last get_client call and using it to
control the behavior of the list_ports method.
This adds a wrapper class that represents a Neutron client object and
stores a reference to a NeutronFixture in order to track admin-ness for
separate get_client calls.
While making this change, unused methods:
fake_get_instance_security_group_bindings and _get_first_id_match were
noticed in NeutronFixture, so they are removed.
Change-Id: I7aa05e857599db820d0fc4daeb730f2a40b3291c
Nova had its own copy of oslotest's CaptureOutput fixture called
OutputStreamCapture (probably predating the oslotest one, actually). DRY
it up.
A future change should be able to remove references to this fixture from
nova.test.TestCase by inheriting from oslotest.base.BaseTestCase instead
of testtools.TestCase. But we may or may not want to do the same for
TestOSAPIFixture since that would pull in fixtures we don't care about
(timeout and tempfile/tempdir).
Change-Id: I18a5d621c9e414452852d4aeb0379be0d30af5df
Nova had its own copy of oslotest's Timeout fixture (probably predating
that one, actually). DRY it up.
A future change should be able to remove references to this fixture from
nova.test.TestCase by inheriting from oslotest.base.BaseTestCase instead
of testtools.TestCase. But we may or may not want to do the same for
NovaMigrationsCheckers subclasses (TestNovaMigrations*) since that would
pull in fixtures we don't care about (output and tempfile/tempdir).
Change-Id: I813a80fae306334abb653e6d3603eabe2d2332bf
The fixture is not mocking the Future.result() method correctly and an
attempt to access it results in the following error:
AttributeError: Mock object has no attribute 'result'
Instead of mocking the submit() method, we can just swap out the
GreenThreadPoolExecutor for the SynchronousExecutor to make the
executor synchronous for testing.
A few changes were also needed in unit tests that are running with the
thread pool executor. Because the thread pool executor is created in
the compute manager's __init__ method, we need to patch the synchronous
executor in before calling the base test class's setUp method. Next, a
couple of unit tests were asserting that an exception would be raised
from the thread pool executor's submit() method but that will not
happen because if an exception is raised during a submitted task, that
exception will not be raised until Future.result() is called. And
Future.result() is never called in the compute manager.
Closes-Bug: #1823278
Change-Id: I719c222a5fa3d654e7397f126a1338e653b7bbcc
This is to give the freedom to simulate down cells for each
individual cell targeted function calls.
Part of blueprint handling-down-cell
Change-Id: Ib5bfa1b6365fb78c7d3beb07c561c62ded5cb0e1
This adds a new testing fixture which allows controlling
when nova.context.scatter_gather_cells returns exception
results for down cells. This will be useful in functional
tests for the down-cell support API microversion to show
the API returning minimal construct results for resources
in down cells like instances and services.
Part of blueprint handling-down-cell
Change-Id: Iabc1e55550e8d9d82e20facfaf84316892c5564a
Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).
openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.
The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.
tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.
The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.
Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,
TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.
Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
The UUIDFields ovo field emits a warning if the field's value is not a
valid UUID. As previous patches removed all the violation from our test
code this patch now escalates the warning to exception raising to
prevent adding new violations in test in the future.
Change-Id: I0bf6d82b7c1a83419b8c54b085142af127b7a011
Traits sync had been tried any time a request that might involve
traits was called. If the global was set no syncing was done, but
lock handling was happening.
This change moves the syncing into the the deploy.load_app() handling.
This means that the syncing will be attempted any time a new WSGI
application is created. Most of the time this will be at the start of a
new process, but some WSGI servers have interesting threading models so
there's a (slim) possibility that it could be in a thread. Because of
this latter possibility, the locking is still in place.
Functional tests are updated to explicitly do the sync in their
setUp(). Some changes in fixtures are required to make sure that
the database is present prior to the sync.
While these changes are not strictly part of extracting placement, the
consolidation and isolation of database handling code makes where to put
this stuff a bit cleaner and more evident: an update_database() method
in deploy uses an empty DbContext class from db_api to the call the
ensure_trait_sync method in resource_provider. update_database is in
deploy because it an app deployment task and because putting it in
db_api leads to circual import problems.
blueprint placement-extract
Closes-Bug: #1756151
Change-Id: Ic87518948ed5bf4ab79f9819cd94714e350ce265
This adds a granular policy checking framework for
placement based on nova.policy but with a lot of
the legacy cruft removed, like the is_admin and
context_is_admin rules.
A new PlacementPolicyFixture is added along with
a new configuration option, [placement]/policy_file,
which is needed because the default policy file
that gets used in config is from [oslo_policy]/policy_file
which is being used as the nova policy file. As
far as I can tell, oslo.policy doesn't allow for
multiple policy files with different names unless
I'm misunderstanding how the policy_dirs option works.
With these changes, we can have something like:
/etc/nova/policy.json - for nova policy rules
/etc/nova/placement-policy.yaml - for placement rules
The docs are also updated to include the placement
policy sample along with a tox builder for the sample.
This starts by adding granular rules for CRUD operations
on the /resource_providers and /resource_providers/{uuid}
routes which use the same descriptions from the placement
API reference. Subsequent patches will add new granular
rules for the other routes.
Part of blueprint granular-placement-policy
Change-Id: I17573f5210314341c332fdcb1ce462a989c21940
We support flavor extra_specs, so it would probably make sense to
have some in one of the test fixture flavors.
In order to make it compatible with the functional tests we'll make
it as small as m1.tiny.
Change-Id: Ic98d7dd2f44796279e18083a8c423aed0066d177
Use the flake8 plugin flake8-import-order to check import ordering. It
can do it automatically and don't need reviewers to check it.
Change-Id: Ia3d81bbbb44b40804b3268c0e648276a36cb4805
Some of the functional tests are calling the scheduler for getting the destination.
Once we will accept having the scheduler be calling the Placement service in order
to know a list of valid resource providers, we will need to run a local fixture
for having those tests faking the Placement service.
Change-Id: I6883888bf5d1920ab57dbf3dcaa1a7b375591754
This patch implements the versioned notifications for flavor
create and destroy.
Implement blueprint: flavor-notifications
Change-Id: Icc2169b87ffba4e7e2079076a82029156c584c6a
This fixture makes it trivial to mock out all the cell listing stuff
for the simple case where you just want to assume a single cell that
is configured as the default database.
Change-Id: I7eecee388780677d2b342556fef0179b02707f3c
StableObjectJsonFixture has been in o.vo since the 1.9.0
release and we require >=1.13.0 in g-r now, so we should
use the fixture from the library rather than keep a
duplicate in nova.
This is also needed to use o.vo 1.17.0 which has change
39a057becc10d1cfb5a5d5024bfcbbe6db1b56be that breaks the
fixture's erroneous unit test.
Change-Id: Idd0e02a1c19300c3ab7a57cbacb78d1f07037843
Closes-Bug: #1618115
This fixture allows functional tests to be written in which code
execution flows through the API but stops before being sent to the
conductor. One example is testing the first part of an instance boot
request.
Change-Id: I080a484f9e817f691b15f92a4b88743edf67bdf2
This adds a fixture that simulates a fully up-to-date system where
all services are current. This is the vast majority of what we test
for our unit tests, of course.
Change-Id: I209cd0d986304b2d913f2cf49b5ba54dcc872c18
In a time long ago and a land far far away, someone thought it was a
good idea to populate the database with default flavors. That was
probably reasonable at the time, but it no longer makes sense and
in fact causes us some pain now.
This patch removes those default flavors from the database. That means
that new deploys will not have them, but doesn't actually rewrite
history in any way.
This will require changes to our docs, which largely assume the presence
of these default flavors from time zero.
DocImpact
Depends-On: Ic275887e97221d9ce5ce6f12cdcfb5ac94e300b0
Change-Id: I80b63ce1ebca01be61ac0f43d26a2992ecf16678
Move all wsgi configuration into [wsgi] group. Update references to
these configuration options to reflect new group.
Change-Id: I1a0a0ce1b0c1bdf60114caf6f5d4e28fd2c3cda5
In normal usage utils.spawn returns an eventlet.greenthread.GreenThread
object which has a few methods defined on it. Currently the wait() and
link() methods are used in Nova. However the SpawnIsSynchronous fixture
was returning the results of wait() directly which led to an interface
mismatch. Normally utils.spawn_n returns a greenlet object which does
not allow checking the return but in testing we have never blocked that
capability before. This could be enhanced later to be strict for spawn_n
if we choose, but it is handy to have raised exceptions pass through in
testing.
SpawnIsSynchronous now returns a GreenThread object lookalike which has
wait() and link() methods that work as expected. This allows for the
removal of the wait() method from network/model.py:NetworkInfo.
A few mocks which emulated the improper behavior were also updated.
Change-Id: I1ded81f93235e217822e385ddd983b24c6359a53
This makes it easier to do tests against a specific migration
with realistic data instead of mocking out all the important parts.
Change-Id: I3a635233c8f676b95fd4cf821123c69768e6dc37
This will ensure that our object JSON is stable during unit tests.
Specifically, the changes field, which is listified from a set, which
can vary in order.
Closes-Bug: #1545177
Change-Id: Ib40ca16c41d5422c18f4fc7d658273a8059100ab
setUp calls rpc.init() twice for each test, once in each of:
self.useFixture(conf_fixture.ConfFixture(CONF))
self.useFixture(nova_fixtures.RPCFixture('nova.test'))
Stop calling rpc.init() in ConfFixture, and defer to RPCFixture for
initialization.
The calls to rpc.init() in ConfFixture were taking 0.01 seconds each.
0.01 seconds * 14784 tests = 2.5 minutes
Change-Id: I710399b9b9286884f3ed71522d2697680951c20e
assertEqual(True/False, ***) should be changed to a meaningful
format of assertTrue/False(***) in test files.
Change-Id: I0b6d4fc088030d953bea7c69880f9aab170ca5cc
This adds service records for the API services for the purpose of
version-tracking. Unlike other services, these records are not periodically
updated, so this adds no additional runtime database burden. The API
services update the version (if necessary) of their record at startup.
This also adds filtering in three places to prevent us from exposing
API service records out of the hosts and availability-zone APIs.
Related to blueprint service-version-behavior
Change-Id: I66becb14788de42bee35064d1155dd919059bcb7
Some tests used incorrect order assertEqual(observed, expected).
The correct order expected by testtools is
assertEqual(expected, observed).
Partial-Bug: #1259292
Change-Id: I30e7c4346844ec0eba50c50ca327ca1966600e7d
In Kilo, we made a point to not allow any subtractive DB migrations. That
gave us a lot of flexibility for upgrades from Juno. Reviewers did the
awesome job of keeping to that plan. However, we need a mechanical filter
to prevent regressions. This adds that in the form of a banned-ops fixture
in test_migrations.
It excludes migrations from pre-Kilo times, as well as excludes a few
migrations that are already in place and do reaspnable things.
In general, we need to be extremely careful about letting migrations
put themselves on the exclusion list, so I added scary words around it.
Change-Id: Iefe678d0aeb1ad898fc5a0cee10fd55ccb7e06db
This just turns spawn_n into a blocking passthrough so that we actually
run the spawned thing synchronously instead of not running it (and often
thinking we did).
Change-Id: I47ce83505361084e262843505031e3327d8e68a8
Before the service code can be converted to objects, a fixture to
handle the global NovaObject indirection api is needed. Without it
some tests would fail with a messaging timeout due to requests for
a nonexistent conductor service because the global indirection_api
would mistakenly be set.
Related to blueprint kilo-objects
Change-Id: I2d417923c09df3a7923e5f15e2e8cf41b40f4f62
In order to run fuctional tests against the database we should have a
fixture to handle the setup and cleanup. This adapts the current
Database fixture to work for either database. It is left out the Nova
TestCase test class and should be set up in specific test classes that
need it.
Change-Id: I98e98cb6dea0008d36397b4e2bf7fcf9ed721721
Stephen Finucane