Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib. Most of this
is autogenerated, as described below, but there is one manual change
necessary:
nova/tests/functional/regressions/test_bug_1781286.py
We need to avoid using 'fixtures.MockPatch' since fixtures is using
'mock' (the library) under the hood and a call to 'mock.patch.stop'
found in that test will now "stop" mocks from the wrong library. We
have discussed making this configurable but the option proposed isn't
that pretty [1] so this is better.
The remainder was auto-generated with the following (hacky) script, with
one or two manual tweaks after the fact:
import glob
for path in glob.glob('nova/tests/**/*.py', recursive=True):
with open(path) as fh:
lines = fh.readlines()
if 'import mock\n' not in lines:
continue
import_group_found = False
create_first_party_group = False
for num, line in enumerate(lines):
line = line.strip()
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
for lib in (
'ddt', 'six', 'webob', 'fixtures', 'testtools'
'neutron', 'cinder', 'ironic', 'keystone', 'oslo',
):
if lib in tokens[1]:
create_first_party_group = True
break
if create_first_party_group:
break
import_group_found = True
if not import_group_found:
continue
if line.startswith('import ') or line.startswith('from '):
tokens = line.split()
if tokens[1] > 'unittest':
break
elif tokens[1] == 'unittest' and (
len(tokens) == 2 or tokens[4] > 'mock'
):
break
elif not line:
break
if create_first_party_group:
lines.insert(num, 'from unittest import mock\n\n')
else:
lines.insert(num, 'from unittest import mock\n')
del lines[lines.index('import mock\n')]
with open(path, 'w+') as fh:
fh.writelines(lines)
Note that we cannot remove mock from our requirements files yet due to
importing pypowervm unit test code in nova unit tests. This library
still uses the mock lib, and since we are importing test code and that
lib (correctly) only declares mock in its test-requirements.txt, mock
would not otherwise be installed and would cause errors while loading
nova unit test code.
[1] https://github.com/testing-cabal/fixtures/pull/49
Change-Id: Id5b04cf2f6ca24af8e366d23f15cf0e5cac8e1cc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Merge these, removing an unnecessary layer of abstraction, and place
them in the new 'nova.db.main' directory. The resulting change is huge,
but it's mainly the result of 's/sqlalchemy import api/main import api/'
and 's/nova.db.api/nova.db.main.api/' with some necessary cleanup. We
also need to rework how we do the blocking of API calls since we no
longer have a 'DBAPI' object that we can monkey patch as we were doing
before. This is now done via a global variable that is set by the 'main'
function of 'nova.cmd.compute'.
The main impact of this change is that it's no longer possible to set
'[database] use_db_reconnect' and have all APIs automatically wrapped in
a DB retry. Seeing as this behavior is experimental, isn't applied to
any of the API DB methods (which don't use oslo.db's 'DBAPI' helper),
and is used explicitly in what would appear to be the critical cases
(via the explicit 'oslo_db.api.wrap_db_retry' decorator), this doesn't
seem like a huge loss.
Change-Id: Iad2e4da4546b80a016e477577d23accb2606a6e4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Only nova network used the vpn arg of allocate_for_instance but since
that is purge from the codebase the vpn arg is unused.
This patch removes the unused parameter.
Change-Id: I8dfc3b2d8fd9f0756d19a0392443e519a0dd0c15
Current tests do not have good test coverage of existing policies.
Either tests for policies do not exist or if they exist then they
do not cover the actual negative and positive testing.
For Example, if any policy with default rule as admin only then
test should verify:
- policy check pass with context having admin or server owner
- policy check fail with context having non-admin and not server owner
As discussed in policy-defaults-refresh, to change the policies
with new default roles and scope_type, we need to have the enough
testing coverage of existing policy behavior.
When we will add the scope_type in policies or new default roles,
then these test coverage will be extended to adopt the new changes
and also make sure we do not break the existing behavior.
This commit covers the testing coverage of existing security groups policies.
Partial implement blueprint policy-defaults-refresh
Change-Id: I212a91dc4b6c2eed9c6522d56ae672618311d346
We're wrestling with multiple imports for this thing and have introduced
a cache to avoid having to load the thing repeatedly. However, Python
already has a way to ensure this doesn't happen: the use of a module.
Given that we don't have any state, we can straight up drop the class
and just call functions directly. Along the way, we drop the
'ensure_default' function, which is a no-op for neutron and switch all
the mocks over, where necessary.
Change-Id: Ia8dbe8ba61ec6d1b8498918a53a103a6eff4d488
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
At some point in the past, there was only nova-network and its code
could be found in 'nova.network'. Neutron was added and eventually found
itself (mostly!) in the 'nova.network.neutronv2' submodule. With
nova-network now gone, we can remove one layer of indirection and move
the code from 'nova.network.neutronv2' back up to 'nova.network',
mirroring what we did with the old nova-volume code way back in 2012
[1]. To ensure people don't get nova-network and 'nova.network'
confused, 'neutron' is retained in filenames.
[1] https://review.opendev.org/#/c/14731/
Change-Id: I329f0fd589a4b2e0426485f09f6782f94275cc07
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
A future change will drop the actual security group driver. For now,
just focus on merging the two disparate test cases, which is easier said
than done.
Change-Id: Idd0b741760601aaccca4adfc5101c3539ba9ec09
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
The neutron security group driver tests subclass the nova-network ones.
Since there are some tests that don't make sense for neutron, these are
stubbed out for the tests for same. Simply remove them from both as a
another step towards unifying these test classes.
In addition, remove a test that does "work" with neutron but doesn't
make any sense, since it's testing that nova is validating something
when in reality neutron is doing the validation.
Change-Id: I967ba6abf38dd0e991a070c76decacc579bb4ff7
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This patch is a follow-up for I3eac9ac0a2a698a3c72b4e646cdcaf293ec4b057.
Replace 'fake' with a real project ID in URL paths and related places
in unit tests.
Change-Id: I603212f2d3cbbd482a2a68cd835af0261a85c392
The 'project_id_regex' opt is used as a workaround for the test
like 'fake' 'openstack' etc, it's deprecated since 13.0.0
so time to remove it.
Co-Authored-By: jichenjc <jichenjc@cn.ibm.com>
Change-Id: I3eac9ac0a2a698a3c72b4e646cdcaf293ec4b057
These old unit tests were stubbing the DB API to get
instance records which is too low level for what we're
testing in the API. This changes the stubs to be on the
compute API.get() method instead.
While in here, tests are changed to re-use common stub
code and some unnecessary stubs/tests are removed. Notably
the non-running and 'invalid instance' tests are removed
since they don't test anything useful that is not already
tested (power state doesn't matter for associating/disassociating
security groups to an instance and there are already tests
for instances which don't exist).
This is part of a larger series of changes to drop some
pre-cellsv2 compatibility code from compute API.get().
Change-Id: I203dc5f2abc14c7ab7c140f6a1e35c2a433b9568
API extensions policies have been deprecated in 17.0.0
release[1]. This commit removes them.
[1] Ie05f4e84519f8a00ffb66ea5ee920d5c7722a66b
Change-Id: Ib3faf85c78bc2cdee13175560dc1458ddb6cb7a8
nova/db/__init__.py was importing * from nova.db.api. This meant that
any time any code anywhere within the nova.db package was imported
then nova.db.api was too, leading to a cascade of imports that may
not have been desired. Also, in general, code in __init__.py is a pain.
Therefore, this change adjusts code that so that either:
* nova.db.api is used directly
* nova.db.api is imported as 'db'
In either case, the functionality remains the same.
The primary goal of this change was to make it possible to import the
model files without having to import the db api. Moving the model files
to a different place in the directory hierarchy was considered, but
given that "code in __init__.py is a pain" this mode was chosen.
This looks like a very large change, but it is essentially adjusting
package names, many in mocks.
Change-Id: Ic1fd7c87ceda05eeb96735da2a415ef37060bb1a
The 'test_get_security_group_list_missing_group_id_rule' method
in nova/tests/unit/api/openstack/compute/test_security_groups.py
is executed in the TestNeutronSecurityGroupsV21 class
because the TestNeutronSecurityGroupsV21 class is a subclass of
the TestSecurityGroupsV21 class.
So replace mox with mock in the method.
The 'stub_compute_with_ips' method in nova/tests/unit/fake_network.py
is used in API sample functional tests in both nova-network and
neutron cases.
So replace mox with stub_out in the method.
Change-Id: I1807a5e513ff53c9268a02bac6b38983620fb623
Implements: blueprint mox-removal
Replace stubs.Set with stub_out or mock in
nova/tests/unit/api/openstack/compute/test_neutron_security_groups.py.
The 'fake_get_instances_security_groups_bindings' method
in nova/tests/unit/api/openstack/compute/test_security_groups.py
is only used in test_neutron_security_groups.py.
So the method is moved to test_neutron_security_groups.py.
Change-Id: Ia2857d51f65f2acb0d22bb873461a50f8c9f9ba4
Implements: blueprint mox-removal
This patch removes code from check_string_length method
in utils.py and makes call to check_string_length method
from oslo_utils.strutils.
TrivialFix
Change-Id: Ic557270e5ca73a8a6be41814150d302541334a9c
sec group APIs accept query param to filter the
quota.
This commit adds json schema to validate the valid
query parameters.
There is no change in API behaviour and additionalProperties
is kept True for backward compatibility.
Partially implements blueprint json-schema-validation-for-query-param
Change-Id: If55296e0efe6676bb931aad8b2b0133efbb910a7
The GET /servers/{server_id}/os-security-groups API code can
perform poorly if the instance has several security groups and
each security group has several rules. This is because when processing
the output, we loop over the groups, and loop over the rules per group,
and then for each rule, if it has a group_id specified, we query
the security group details (from Neutron in most cases).
If more than one rule points at the same group_id, we're doing a redundant
group lookup and sending more traffic to the security group API (aka Neutron)
than needed.
This change optimizes that single API to load the rule group details
up front so that we only do at most one lookup per group_id.
This could be extended to GET /os-security-groups but that API is
deprecated so any optimization there is lower priority.
Change-Id: Ia451429f61b15526fade6838386e562c17591d36
Closes-Bug: #1729741
This converts security_group_rules to use check_deltas() for quota
checks instead of the legacy limit_check().
Part of blueprint cells-count-resources-to-check-quota-in-api
Change-Id: Ief6d0a6284390c23c5080c6d052179c39c0f535a
This changes security groups from a ReservableResource to a
CountableResource and replaces quota reserve/commit/rollback with
check_deltas accordingly. Note that security group quota is only
relevant to nova-network and will be obsolete when nova-network
is removed.
Part of blueprint cells-count-resources-to-check-quota-in-api
Change-Id: I51b74b863fafd14ef31b1abd9fd9820deb622aeb
A number of endpoints enable pagination via 'limit' & 'offset' query
parameters, but don't document or test this functionality.
- os-cells
- os-security-groups
- os-server-groups
- os-snapshots
- os-virtual-interfaces
- os-volume-attachments
- os-volumes
Change-Id: I5b0ad25f7282f4a13bbb6f21b76e986e1008936a
As suggested in John Garbutt in I7e1986c5f11356060cc9db12605b1322c39e79c0,
move the quota config options into a config group of their own.
Change-Id: Ie06a370868f01fb9a1cc246a77d7823fac83e70e
The tests in this module are all based on using nova-network for
security groups so we need to explicitly set use_neutron=False for
these tests. The security groups REST API is tested using Neutron
in the test_neutron_security_groups module.
Because of the joy of multiple inheritance we do have to modify
the test_neutron_security_groups tests to set an attribute that is
used in the base test classes before creating the controller object.
Part of blueprint use-neutron-by-default
Change-Id: I814949630bcef296773c14bff21b921d58c25d8b
Since d8673cb256 the init_only
kwarg is no longer used in the compute API wsgi app setup.
This patch removes it along with it's usage throughout the
compute REST API unit tests where it was used to isolate
testing of extensions, which is not valid anymore as all
extensions are enabled and we removed the ability to
whitelist/blacklist them.
Change-Id: I2f58b7944d0f7d7c05683238474954a65a617ec3
In wsgi controllers tests fakes.HTTPRequest and webob.Request are used.
fakes.HTTPRequest is the descendant of webob.Request defined in nova
test codebase. To have single point of modification it's better to have
common class for controllers tests.
This change replaces webob.Request to fakes.HTTPRequest and updates
blank method signature with respect to actual webob.Request.blank
method.
Change-Id: I043b9cf4426ec1d2c9b178a6ceacb74fc113e21b
Related-Bug: #1610153
Make db.security_group_get only join rules if specified in
the columns_to_join. This works around a performance issue
with lots of instances and security groups.
Co-Authored-By: Dan Smith <dansmith@redhat.com>
Change-Id: Ie3daed133419c41ed22646f9a790570ff47f0eec
Closes-Bug: #1552971
The config options 'osapi_compute_ext_list' and 'osapi_compute_extension'
that are deprecated in mitaka has been removed in newton.
Blueprint centralize-config-options-newton
Change-Id: I899c0a7da371138ed5e448d33a10d8ec46e42fe1
This patch deprecates all the APIs which related SecurityGroup. All those
APIs will return 404.
The deprecated API endpoints are
'/os-security-group-default-rules'
'/os-security-groups'
'/os-security-group-rules'
The action 'addSecurityGroup' and 'removeSecurityGroup' will be kept. And
the attribute 'security_groups' in the servers response will be kept also.
Due to the current implementation of Microversion didn't support inheritance
very well. This patch uses object as SecurityGroupControllerBase's base class
to avoid two controller share same base controller which is subclass of
'wsgi.Controller'. The support of inheritance will be improved later to
avoid increase the complicated in this series patches.
This patch doesn't bump the max api version, due to the patch separation.
The max api version will bump in the last patch.
Partially implements blueprint deprecate-api-proxies
Change-Id: Ic834db770f68c72892a6497d5c60707b75f1beef
There are two implementation code for similar API in Nova repository.
One is newer: v2.1 API, another is legacy: v2 API. v2.1 API has been
used as the default API since Liberty and legacy v2 API has been marked
as deprecated. We have used and tested v2.1 API so well and now is nice
time to remove legacy API code based on the consensus of the design
summit of Austin. This patch removes unit tests of legacy v2 API[q-v].
Partially implements blueprint remove-legacy-v2-api-code
Change-Id: I7fd86265ffa4d5bf3d4350a98b48685258947e43
This change replaces use of mox with mock in
test_security_groups.py.
Change-Id: Iebf98d401e3076f31d3169ae2ec4cb419f2bb60b
Implements: blueprint remove-mox-newton
Because of the in tree ec2 api we used to need to easily handle both
integer and uuid ids as if they were the same thing. This led to
making compute_api.get take either and try to do the right thing. This
unfortunately meant we exposed the indexed integer ids over the
OpenStack API, which was never intended.
... and turns out to totally explode because of the way we cache
requests for extensions to post process them. The hide_address
extension shows a nice bit of exploding which has been noticed by our
users.
Remove the int id facility. Remove the test for it. Update the other
tests that were sneaking in and using id==1 like very naughty folks
(there will be coal in your stocking this year).
Closes-Bug: #1522536
Change-Id: Ibff4cebe71714ac9c470337c2ee5f57df4343829
As part of the ongoing effort to stop using mox, start replacing
the stubs.Set calls with stub_out.
Limit the scope of this patch to the ~400 db stubs.
Part of bp:remove-mox
Change-Id: I449726ede61a18d2c504cf370bebc2b3291fcb04
As part of the ongoing effort to stop using mox, start replacing
stubs.Set calls with stub_out.
Limit the scope of this patch to the following fake functions:
- fake_get_instance_nw_info
- stub_nw_info
- stub_out_instance_quota
- stub_out_networking
- stub_out_nw_api
- stub_out_nw_api_get_instance_nw_info
- set_stub_network_methods
- stub_out_network_cleanup
- unset_stub_network_methods
Part of bp:remove-mox
Change-Id: I70215fb25ef25422786b96d33c91d8f1d4760a23
Enable more tests to work by making sure we use
encodeutils.safe_encode and jsonutils.dump_as_bytes
as needed.
Change-Id: I05de0cd58ee1734df0967bb58082a7e017741c0b
The normal authorize and soft authorize has the same rule.
All actions of SecurityGroupsOutputController need soft authorize.
If soft authorize fails, no chance to do normal authorize, also if soft
authorize passes, the normal authorize is redundant.
Change-Id: Ie354b14f592738a882ca261133de4372a3e6507b
Closes-Bug: 1425849
Cleanup HTTPReuqest for security_groups test and
test_neutron_security_groups test on create function.
Change-Id: I63e5f6c2e58d60bf77f86ee82b2474283b736ea9
This patch moves the tests in contrib/ and plugins/v3/ into the
base directory.
Note that some of the tests have both v2 and v21 tests, The v2
tests will be deleted when V2 API removed.
Co-Authored-By: Ed Leafe <ed@leafe.com>
Change-Id: I6ff1d6594e7a44f2bcb6bbb04a4277b98d1cac74
Partial-Bug: #1462901
Stephen Finucane