Add support for wallaby.
Below were the extra changes needed to support
the wallaby branch:
1. Add new attribute 'remote_address_group_id'
for the security group resource.
2. Handle new standard_attr_id argument for resources.
3. Fix kwargs passed to the alembic migrations
create_foreign_key and create_primary_key.
4. Change CONTEXT_WRITER to CONTEXT_READER in the
get_subnets function.
Change-Id: I3835df151cad2f7ca52afcb701de2bc508c90014
Add support for stable victoria. Changes include:
* https://review.opendev.org/#/c/716049/ switched to unittest for mock
Change-Id: I053657f535d985205ae9d3548291ec1d1409cb74
Add support for the ussuri stable branch.
* Removed use of services in devstack (e.g. FWaaS and LBaas), which
were only used by the deprecated legacy plugin.
* https://review.opendev.org/#/c/572767/ changed the return
value of _get_security_groups_on_port from a list of security
group IDs to a list of security group OVOs. The monkey patch
of this method has been updated to be consistent with this
upstream change.
* https://review.opendev.org/#/c/703143/ removed the upstream
get_binding_levels, which is replaced by the corresponding
OVO call, get_binding_level_objs.
* https://review.opendev.org/#/c/709122/ broke the __repr__
method in the AddressScope model class. This patch works
around this by using the dictionary representation instead.
* https://review.opendev.org/#/c/679399/ made the MTU field
of networks non-nullable, and sets it to a constant if not
set explicitly. This broke GBP APIs which create networks
as part of their implementation. This patch adds a monkey
patch to pass in a value of 0, if one wasn't specified.
* Fixed alias uncovered by PEP8 checks.
Change-Id: I219bc9a5c2034499e59788ab11ef0ae310e97e1e
Import stable/stein rather than stable/rocky branches of upstream
and ACI-specific repositories.
Changes include:
* https://review.opendev.org/#/c/634790/ removed the rpc module
from neutron.common, which was rehomed to neutron-lib.
* https://review.opendev.org/#/c/634497/ removed the exceptions
module from neutron.common, which was rehomed to neutron-lib.
* https://review.opendev.org/#/c/581377/ removed exercises from the
devstack gate. The shell scripts that ran the tests from the
devstack exercises are now called directly.
* https://review.opendev.org/#/c/619087/ removed the common_db_mixin
from the FlowClassifierDbPlugin, replacing it with the use of a
method in neutron-lib.
* https://review.opendev.org/#/c/595369/ removed _setUpExtension,
replacing it with the setup_extension method.
* https://review.opendev.org/#/c/623415/ added validation to host
route CIDRs. The metadata CIDRs have been corrected to pass
this new validation.
* https://review.opendev.org/#/c/615486/ added a call to get a
nova client, and https://review.opendev.org/#/c/368631/ was
added to ensure it was a singleton. These are now used to get
a notifier for nova.
* https://review.opendev.org/#/c/628033/ removed the use of the
_resource_extend module, which has been moved to neutron-lib.
* https://review.opendev.org/#/c/585037/ converted policy.json
to policy in code. This resulted in better policy enforcement,
and flagged problems with existing UTs, mainly in the use of
shared resources (requires admin privileges). These UTs have
been fixed.
Change-Id: Ia7bd0799a814e38ff37b7ff062fa1eae7928991c
Import stable/rocky rather than stable/queens branches of upstream
and ACI-specific repositories. Changes needed for compatability
with stable/rocky that were also compatible with stable/queens were
made in previous patches, so only rocky-specific changes are included
here.
Change-Id: If533a955fb4bc23d6e4081a43df7018b1b36a0ba
Enhance compatabilty with newer Neutron branches while maintaining
compatability with stable/queens Neutron, and improve the build/test
process. Highlights include:
* Eliminate unneeded requirements and test-requirements, and update
remaining ones to match upstream stable/queens Neutron.
* Use pip directly instead of the tox_install.sh script to install
dependencies, as is done on newer upstream branches.
* Use stestr directly instead of ostestr to run UTs, as is done in newer
upstream branches.
* Specify basepython as python2.7 for pep8, cover, functional and
dsvm-functional jobs, in case a python3 version of tox is used.
* Fix pep8 issues that result in failures with the versions of hacking
and flake8 used by Neutron's stable/rocky through stable/train
branches. These changes are not necessary with the hacking and
flake8 versions used in stable/queens, but we want to minimize code
differences across our currently supported stable branches.
* Enable flake8-import-order and fix all the pep8 issues that it
uncovered, particularly with order and grouping of import
statements.
* Update pep8 configuration in tox.ini to more closely match upstream
Neutron, and fix resulting issues. Remaining ignored checks that
should be fixed but haven't been are marked with REVISIT in tox.ini.
* Update devstack scripts with proper branches and repository URLs.
Change-Id: I538b8c95c61a09d834be4b7c28a3becf2f3e6a50
* dict_items, dict_values, dict_keys returns view in Py3 instead of
list so wrapped with list.
* hashlib.md5 needs bytes data in Py3, so encoded data with 'utf-8'.
* Replaced dict.iteritems() with dict.items().
* Replaced dict.itervalues() with dict.values().
* Replaced iterator.next() with next(iterator)
* 'magicMock' and 'int' cannot be compared, so patched
'vmware_nsxlib.v3.utils._update_max_nsgroups_criteria_tags()'
with its return value.
* Added zip() as a replacement of itertools.izip() in Py3.
* Added a method to compare 2 (nested) dict or list of (nested) dict,
since Py3 (till python3.6) dict are not ordered.
* Added zuul python35 CI jobs.
* Added py35 env in 'tox.ini'.
* Added patcher.stop() in 'NsxPolicyMappingTestCase' tearDown().
Fixes bug 1877200
Change-Id: Id96faba6c2ecd29918bda000ac0ccbfdf988d17d
1) Revert "Remove tests for master branch", commit
d149f30a4b.
2) Use test-requirements.txt from stable/queens to select the
stable/queens branches of python-opflex-agent and
python-group-based-policy-client, and the noiro-lite branch of
acitoolkit.
3) Use pushd/popd in tox_install.sh to restore initial CWD after
switching requirements branch.
Change-Id: I39895732aac0bdfaee95274cbcb262d6744faeb1
This is an ugly temporary fix to the upstream master gate,
until we can work out the correct way to fix it with the
openstack-infra folks.
Change-Id: Ibc76c899074890564b4579ee6e6e66e9a4af12e7
For gate jobs run on the master, the master branch
of OpenStack global requirements is automatically checked
out. However, since GBP master currently trails in release
version, we checkout the relevant branch for the requirements
repo before running tox.
A separate commit that temporarily disables NSX UTs is squashed here.
The original commit message of that patch reads as follows:
"
Nsx policy: skip UTs temporarily
Skip UTs while driver code is being adjusted to new backend version.
We will not maintain support for old backend version APIs.
"
Neither of these two commits will pass the gate independently, hence
they are being combined in one patch here.
Change-Id: I20cb957411e734165151ad40e1c0978af31c3cca
The following changes have been made to coordinate with the changes
made in Neutron for Pike:
* Partial use of Neutron context has been completely moved to neutron_lib's
context.
* The patching of neutron.db.api.get_session() has been replaced with
patching of sqlalchemy.orm.session to add the notification_queue attribute.
This significantly reduces the earlier complexity of patching.
* Use of top-level start of transaction in GBP plugins:
with context.session.begin(subtransactions=True):
has been migrated to use of:
with db_api.context_manager.writer.using(context):
or
with db_api.context_manager.reader.using(context)
as relevant.
* Calls to _make_resource_xxx_dict() in GBP plugins have been moved
to inside the transaction.
* The use of:
neutron.callbacks.events
neutron.callbacks.exceptions
neutron.callbacks.registry
to
neutron_lib.callbacks.events
neutron_lib.callbacks.exceptions
neutron_lib.callbacks.registry
* The use of:
neutron.api.v2.attributes.resource_xxx
neutron.extensions.extension_xxx
to:
from neutron_lib.api.definitions.resource_xxx
from neutron_lib.api.definitions.extension_xxx
resp.
* The use of:
neutron.db.db_base_plugin_v2.NeutronDbPluginV2.register_dict_extend_funcs
to:
neutron.db._resource_extend.resource_extend
(the latter is a decorator)
* The use of:
neutron.db.db_base_plugin_v2.NeutronDbPluginV2.register_model_query_hook()
to:
from neutron.db import _model_query as model_query.register_hook()
* The use of:
neutron.db.segments_db.NetworkSegment
to:
neutron.db.models.segment.NetworkSegment
* In the case of Neutron ml2plus plugin (used by APIC/AIM solution),
the use of get_admin_context() has been patched to return elevated
version of the current context in use. This helps to preserve the session
and transaction semantics. Ideally, context.elevated() would have been
directly used in all these places, however the current context is not
available in these places, and hence getting the current context and elevating
it is wrapped in the get_admin_context() patched method.
* In the case of the components used by the APIC/AIM solution (including
the ml2plus and l3_plugin) the use of:
with context.session.begin(subtransactions=True):
to
with db_api.context_manager.writer.using(context):
or
with db_api.context_manager.reader.using(context):
as relevant.
* Patching of methods from Neutron which is no longer relevant have been
removed from gbpservice.neutron.extensions.patch module.
* Setting up of UTs has been fixed to load and reset configurations
appropriately. This helps to eleminate some failures when tests are
run in non-deterministic orders.
* In tree devstack plugin has been updated (aim repo commit pin needs
to be reverted).
* Gate jobs have been updated as relevant (including fixes to the exercise
scripts and job configurations).
The associated repos, namely, client, UI and automation have also been
updated (the reference to the client's gerrit patch needs to be updated
once the patch has been merged).
Change-Id: I11dd089effbf40cf104afd720dc40a9911dcf28d
Using the neutron context from neutron_lib (but
continues using db_api from neutron instead of
neutron_lib).
Note that Neutron still uses the context from the Neutron
repo in Ocata, so with this patch, both, Neutron and
neutron_lib, contexts will be in play depending on
whether the context is being instantiated by Neutron
or by GBP.
Also updating the tox hacking ignores to not require
translation hints for log messages. This should have been
removed in the previously merged patch which removed the log
message hints.
Change-Id: I8df4d00a8c0109e04e1f72576a54048332666078
Some of available checks are disabled by default, like:
[H106] Don’t put vim configuration in source files
[H203] Use assertIs(Not)None to check for None
Change-Id: I73368bc191746e9ba30d205f9f36722f8e3eb5f6
* use neutron_lib.directory for plugin retrieval
* switch to neutron_lib for neutron constants, exceptions,
extensions
* add neutron.plugins.ml2.ovo_rpc to OUT_OF_PROCESS_NOTIFICATIONS:
neutron added ovo rpc callback mechanism for ovo objects, and aim
notification manager needs to recognize those as out of process.
Since neutron moved away from get_session API to get_reader_session
and get_writer_session, override for these was added.
Few bugs were fixed in the delayed notification area as well.
* new engine facade: make use of reader and writer to grab db engine
* remove _update_fip_assoc override (didn't find a reason for the
override)
* aim driver: a fix in update_subnetpool_precommit - not to assume
address_scope_id field is returned from neutron update call if it
was not updated.
* extend_XXX_dict call was switched to receive ovo instead of db
object. As a result, foreign keys are not part of the object
anymore, and need to be retrieved from db.
* remove_router_interface - receive port dictinary rather than port
object
* fix patched neutron functions to receive correct parameter types
(like patched_get_locked_port_and_binding)
* use add_agent_status_check_worker instead of add_agent_status_check
* advertise_mtu configuration parameter was removed from neutron. It
is used in aim driver, hence added to aim driver config.
* use of project_id instead of tenant_id where required
* use segments_db module for network segments
* test_aim_mapping_driver: the test used to override uuid generation
in order to get predictable uuid results. New neutron code makes
use of python uuid module where overrides are complicated. It was
easire to remove all uuid-based values from dictionaries under test
* add filters parameter to get_address_scopes calls, otherwise the
call fails (probably should be fixed in neutron)
* in routing tests, remove the assumption that routes are returned in
specific order
Change-Id: I1943fd4196ea6199d825ae53f0e9f5b54d54a260
Since hacking check N537 has recently been enabled, translation hints
are no longer allowed on log messages in neutron-lib and in
repositories like GBP that inherit thier hacking configuration from
neutron-lib. A recent GBP patch disabled the N537 check because we
have not yet removed the translation hints from existng GBP log
messages.
This patch disables N531 as well, so that translation hints will
neither be required nor forbidden for GBP log message. This allows
translation hints to be incrementally removed from existing GBP log
messages, and new code to be merged without them.
Eventually, once all translation hints have been removed from GBP log
messages, N537 should be re-enabled, forbidding any new translation
hints for log messages.
Change-Id: Ie676b10d16c3fa32e3b72e165a1f35b72f0cc472
In http://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=48def9d14e7de95e1a44c7c89c7490a6b3ac2d9a,
neutron_lib's hacking checks were updated to replace:
- [N533] Validate that debug level logs are not translated
with:
- [N537] Don't translate logs.
Therefore, N533 is temporarily disabled until GBP code can be updated
to remove all log message translations.
Change-Id: Id18df04c2d2e4457a3fe028b9acf94c9b0d36d55
* The IP addresses handed by Neutron’s ipam are no longer sequential per the
following commit:
dcb2a931b5
Several UTs were making sequential assignment assumptions and were expecting
specific IP address allocations. These had to be refactored appropriately by
checking if the assigned IP address belongs to the expected CIDR.
* There was a bug in Neutron until stable/mitaka which prevented duplicate SG
rules being added. Since that is fixed in stable/newton overlapping SG rules
can be added within the same tenant, see:
3c1a068c7a
We actually dont want to add overlapping rules in the resource_mapping driver,
hence a check was added to prevent adding of duplicate rules.
* The unit test discovery path is being set to "gbpservice/neutron" in
.testr.conf to avoid running the tests in gbpservice/contrib path which
currently have some NFP related tests. The path can be reverted back to
“gbpservice” once the contrib code has been updated.
* There is a bug in the neutron code which always requires passing the filters
argument (even if empty) to get_sg_rules() call.
* The flavors service plugin needs to be explicitly configured in the UTs:
0e3f4b8335
* The use of unittest has been migrated to using unittest2.
* The default tenant in the Neutron UTs is no longer ‘test-tenant’. Instead the
following constant should be used:
neutron.tests.unit.db.test_db_base_plugin_v2.TEST_TENANT_ID
* The project_id is now being added to the resource by the API layer. The
extension test cases had to be updated to accommodate for this extra
argument.
* Neutron now sends DHCP and Nova notifications for operations on resources
from the ML2 plugin. See the following relevant commits in Neutron:
181bdb374fa5cd3b65d1877778ee4ca4df99ff5d
With the above changes, it is no longer needed for GBP to send DHCP and Nova
notifications (previously being sent from local_api.py). The neutron_resource
and the aim_mapping drivers, which attempt to provide transactional semantics,
still need the queueing functionality on the notification framework, so this is
being preserved. The send_or_queue_notification method from this framework is
also being preserved since the aim_mapping driver makes use of this method to
send specific notifications which are outside the scope of the notifications
that Neutron sends. When the ML2Plus plugin is used for the aim_mapping driver,
Neutron’s registry notification is patched to allow the notification to be
queued. It should be noted that at this point, some notifications cannot be
queued since for some resources the existence checks fail if they are queued
and sent at a later time. In such cases, the notifications are sent
immediately. This logic needs to be revisited.
* The _get_tenant_id_for_create() method was removed in Neutron since the
context object provides the project_id:
5d53dfb8d6
GBP should also follow this approach, however its a big change, mostly in the
UTs. So this patch temporarily adds the _get_tenant_id_for_create() method to
the GBP service plugins.
* The patch for create_floatingip in gbpservice/neutron/extensions/patch.py is
no longer needed and is being removed. Other such methods in the module
cannot be blindly removed, so for now, they have been updated to sync with
their newton version, but should be revisited to explore their removal.
* The ml2_network_segments table was renamed to network segments, and the
allowed_address_pair definition was moved in neutron:
c8fca1c96f7c0f189309
* DB objects are being detached from the session when the extension attribute
processing happens. However, ml2plus needs the session context in the
extend_dict functions. Hence, a utility function was added in
gbpservice/neutron/plugins/ml2plus/patch_neutron.py to get the currently
active session.
* The following change adds a transaction guard to some operations which
prevent then from being called from within a transaction:
https://review.openstack.org/gitweb?p=openstack/neutron.git;a=commitdiff;h=afe1a834000d33900b8646d308fa26fa807a2ca0
ml2plus however needs to support calling these operations from within a
transaction. Hence the transaction guard is disabled by use of a decorator on
the ml2plus functions.
* Neutron defines a new dns-integration extension and all the DB related DNS
handling was moved out of the DB core plugin:
64f5fc8259
* Retry decorator has been added to ml2plus methods to align with the
following:
acbabaa3db09c87425fa
* A bug in the aim_mapping log statements was fixed by using vars() to displace
aim resource dictionary attributes.
* A bug was fixed in the test_apic_aim code, where the tests set the expected
value of dns_name to None, but the implementation sets it to ‘’.
* The following changes were made to move things to neutron_lib and have been
refactored in this patch:
** neutron.db.model_base was moved to neutron_lib:
61cc14fd67
** neutron.common.exceptions was moved to neutron_lib, and has been refactored
here.
** The converter and validator functions in neutron.api.v2.attributes were
moved to neutron_lib and has been refactored here.
** Constants like ATTR_NOT_SPECIFIED have been moved from
neutron.api.v2.attributes to neutron_lib and has been refactored here.
Note that the integration tests fail in this patch since the DB schema needs to
be updated to rename the tenant_id column to project_id. This is being done in
the dependent patch, and the integration tests should be validated on that
patch.
Follow up items:
* The following test fails sporadically:
gbpservice.neutron.tests.unit.services.grouppolicy.test_aim_mapping_driver.TestNeutronPortOperation.test_gbp_details_for_allowed_address_pair
* Some hacking directives have been disabled and need to enabled but will
* require significant code refactoring.
* HasId and HasTenant are deprecated, move to HasProject and HasId in
* model_base, see commit:
61cc14fd67 (diff-b923b82d6a7b3c5cd77c32354ffc9f13)
* A couple of UTs are being skipped in:
* gbpservice/neutron/tests/unit/plugins/ml2plus/test_extension_driver_api.py
* and need to updated per the comments in the code.
Change-Id: I887ee6cfca8199710cf5c653b5f57dff86bb035a
Many deprecations are triggered early (on imports, for example)
before the warnings are enabled by the WarningsFixture in the
base test class.
To make sure all DeprecationWarning messages are emitted we enable
them via the PYTHONWARNINGS environment variable.
Change-Id: Ic47b2e964706fb38b8af71be1c510976f18f083e
This also fixes three UTs which started failing after
changing the tox environment. The issue with the tests
was that they were assuming a specific order of elements
in a JSON request body, which started breaking after the
tox environment change.
Change-Id: I3a882f8e5052c4bdf04f1b75c86ab0ebbd8c72c0
Closes-bug: 1588753
This patch implements the hooks so that functional and integration
tests can be run in the group-based-policy repo.
Currently, the configured devstack for this job is enabled to run GBP
exercise scripts, which result in complete end-to-end integration
testing of the GBP service and client components along with integration
with Neutron and Nova.
A placeholder functional test case is also added. A functional test suite
will be added in a separate commit.
Change-Id: Id705e5755b9662f6e297f29836d6983d66f845a8
(Patch series identifier: GPM-API-1)
This patch extends the initial set of Group Policy API resources with
attributes mapping those resources to traditional Neutron
resources. Subsequent patches will add the DB, plugin, and driver
support for these mapping attributes.
Author: Bob Kukura <kukura@noironetworks.com>
Co-Authored-By: Sumit Naiksatam <sumitnaiksatam@gmail.com>
Co-Authored-By: Stephen Wong <s3wong@midokura.com>
Co-Authored-By: Mohammad Banikazemi <mb@us.ibm.com>
Co-Authored-By: Mandeep Dhami <dhami@noironetworks.com>
Change-Id: I617efd632f6ca423b58f2e9ba504cf6c4bc2ac53
Nisar Khan