Commit Graph

13 Commits

Author SHA1 Message Date
Robert Kukura ca82a36cb1 Cleanup Queens (part 2)
Enhance compatabilty with newer Neutron branches while maintaining
compatability with stable/queens Neutron, and improve the build/test
process. Highlights include:

* Eliminate unneeded requirements and test-requirements, and update
  remaining ones to match upstream stable/queens Neutron.

* Use pip directly instead of the tox_install.sh script to install
  dependencies, as is done on newer upstream branches.

* Use stestr directly instead of ostestr to run UTs, as is done in newer
  upstream branches.

* Specify basepython as python2.7 for pep8, cover, functional and
  dsvm-functional jobs, in case a python3 version of tox is used.

* Fix pep8 issues that result in failures with the versions of hacking
  and flake8 used by Neutron's stable/rocky through stable/train
  branches. These changes are not necessary with the hacking and
  flake8 versions used in stable/queens, but we want to minimize code
  differences across our currently supported stable branches.

* Enable flake8-import-order and fix all the pep8 issues that it
  uncovered, particularly with order and grouping of import
  statements.

* Update pep8 configuration in tox.ini to more closely match upstream
  Neutron, and fix resulting issues. Remaining ignored checks that
  should be fixed but haven't been are marked with REVISIT in tox.ini.

* Update devstack scripts with proper branches and repository URLs.

Change-Id: I538b8c95c61a09d834be4b7c28a3becf2f3e6a50
2020-06-09 13:14:11 -04:00
Robert Kukura bae4981e85 Raise OS_TEST_TIMEOUT
Several apic_aim unit tests have been occasionally failing in the
gate, then passing when rechecked. This appears to be due to them
taking longer than 120 seconds to complete. We need to find out why
they take so long, and why the timing varies so much, but this should
help reduce the need to recheck in the mean time.

Change-Id: I6362d1cb6d7df87087282a6b78cabc741b31b9f1
2017-09-13 20:58:48 +00:00
Rajendra Machani 7db0f27231 Fix for excluding gbpservice/tests/contrib/nfp_service/ from UTs
Excluding the gbpservice/tests/contrib/nfp_service/ from python27 tests
by moving out the directory permanently to gbpservice/contrib/.
Fixed file path references in few files as part of this.

Enabled whole gbpservice tests instead of just gbpservice/neutron in
.testr.conf. Fixed FW and VPN driver UTs as part of this.

Fixed pep8 warnings in reference controller. As part of this, added
oslo.i18n installation to nfp reference configurator vm, and used
gbpservice._i18n.py for i18n in logs.

Fix to mock rmq connection in the advanced controller unit tests

Change-Id: I4743e44f86e78b7d19ccd824a7c65195f12f856d
Closes-Bug: 1684076
2017-06-17 08:06:16 +00:00
Sumit Naiksatam 725c7aa466 Newton sync
* The IP addresses handed by Neutron’s ipam are no longer sequential per the
following commit:
dcb2a931b5

Several UTs were making sequential assignment assumptions and were expecting
specific IP address allocations. These had to be refactored appropriately by
checking if the assigned IP address belongs to the expected CIDR.

* There was a bug in Neutron until stable/mitaka which prevented duplicate SG
rules being added. Since that is fixed in stable/newton overlapping SG rules
can be added within the same tenant, see:
3c1a068c7a
We actually dont want to add overlapping rules in the resource_mapping driver,
hence a check was added to prevent adding of duplicate rules.

* The unit test discovery path is being set to "gbpservice/neutron" in
.testr.conf to avoid running the tests in gbpservice/contrib path which
currently have some NFP related tests. The path can be reverted back to
“gbpservice” once the contrib code has been updated.

* There is a bug in the neutron code which always requires passing the filters
argument (even if empty) to get_sg_rules() call.

* The flavors service plugin needs to be explicitly configured in the UTs:
0e3f4b8335

* The use of unittest has been migrated to using unittest2.

* The default tenant in the Neutron UTs is no longer ‘test-tenant’. Instead the
following constant should be used:
neutron.tests.unit.db.test_db_base_plugin_v2.TEST_TENANT_ID

* The project_id is now being added to the resource by the API layer. The
extension test cases had to be updated to accommodate for this extra
argument.

* Neutron now sends DHCP and Nova notifications for operations on resources
from the ML2 plugin. See the following relevant commits in Neutron:
181bdb374f
a5cd3b65d1
877778ee4c
a4df99ff5d

With the above changes, it is no longer needed for GBP to send DHCP and Nova
notifications (previously being sent from local_api.py). The neutron_resource
and the aim_mapping drivers, which attempt to provide transactional semantics,
still need the queueing functionality on the notification framework, so this is
being preserved. The send_or_queue_notification method from this framework is
also being preserved since the aim_mapping driver makes use of this method to
send specific notifications which are outside the scope of the notifications
that Neutron sends. When the ML2Plus plugin is used for the aim_mapping driver,
Neutron’s registry notification is patched to allow the notification to be
queued. It should be noted that at this point, some notifications cannot be
queued since for some resources the existence checks fail if they are queued
and sent at a later time. In such cases, the notifications are sent
immediately. This logic needs to be revisited.

* The _get_tenant_id_for_create() method was removed in Neutron since the
context object provides the project_id:
5d53dfb8d6

GBP should also follow this approach, however its a big change, mostly in the
UTs. So this patch temporarily adds the _get_tenant_id_for_create() method to
the GBP service plugins.

* The patch for create_floatingip in gbpservice/neutron/extensions/patch.py is
no longer needed and is being removed. Other such methods in the module
cannot be blindly removed, so for now, they have been updated to sync with
their newton version, but should be revisited to explore their removal.

* The ml2_network_segments table was renamed to network segments, and the
allowed_address_pair definition was moved in neutron:
c8fca1c96f
7c0f189309

* DB objects are being detached from the session when the extension attribute
processing happens. However, ml2plus needs the session context in the
extend_dict functions. Hence, a utility function was added in
gbpservice/neutron/plugins/ml2plus/patch_neutron.py to get the currently
active session.

* The following change adds a transaction guard to some operations which
prevent then from being called from within a transaction:
https://review.openstack.org/gitweb?p=openstack/neutron.git;a=commitdiff;h=afe1a834000d33900b8646d308fa26fa807a2ca0

ml2plus however needs to support calling these operations from within a
transaction. Hence the transaction guard is disabled by use of a decorator on
the ml2plus functions.

* Neutron defines a new dns-integration extension and all the DB related DNS
handling was moved out of the DB core plugin:
64f5fc8259

* Retry decorator has been added to ml2plus methods to align with the
following:
acbabaa3db
09c87425fa

* A bug in the aim_mapping log statements was fixed by using vars() to displace
aim resource dictionary attributes.

* A bug was fixed in the test_apic_aim code, where the tests set the expected
value of dns_name to None, but the implementation sets it to ‘’.

* The following changes were made to move things to neutron_lib and have been
refactored in this patch:

** neutron.db.model_base was moved to neutron_lib:
61cc14fd67

** neutron.common.exceptions was moved to neutron_lib, and has been refactored
here.

** The converter and validator functions in neutron.api.v2.attributes were
moved to neutron_lib and has been refactored here.

** Constants like ATTR_NOT_SPECIFIED have been moved from
neutron.api.v2.attributes to neutron_lib and has been refactored here.

Note that the integration tests fail in this patch since the DB schema needs to
be updated to rename the tenant_id column to project_id. This is being done in
the dependent patch, and the integration tests should be validated on that
patch.

Follow up items:

* The following test fails sporadically:
gbpservice.neutron.tests.unit.services.grouppolicy.test_aim_mapping_driver.TestNeutronPortOperation.test_gbp_details_for_allowed_address_pair
* Some hacking directives have been disabled and need to enabled but will
* require significant code refactoring.
* HasId and HasTenant are deprecated, move to HasProject and HasId in
* model_base, see commit:
61cc14fd67 (diff-b923b82d6a7b3c5cd77c32354ffc9f13)
* A couple of UTs are being skipped in:
* gbpservice/neutron/tests/unit/plugins/ml2plus/test_extension_driver_api.py
* and need to updated per the comments in the code.

Change-Id: I887ee6cfca8199710cf5c653b5f57dff86bb035a
2017-03-17 01:35:29 -07:00
mak-454 76882d31dc NFP - Failure handling in Configurator
1) Added support to utilize NFP context supported
by nfp/core. Initializes context with default
values in all external functions which are invoked
via rpcs.

2) Fixed the issue with bulk firewall rules (>250)
and which gets multipled by number of consumers.

Change-Id: I57c39a717b1cb6bc1d9ede7a952eb919bd8f06ab
Closes-Bug: 1668198
2017-03-01 15:52:29 +05:30
mak-454 1c74296e08 NFP - Context support in Config Orchestrator
Added support to utilize NFP context supported
by nfp/core. Initializes context with default
values in all external functions which are invoked
by rpcs.
Uses the same context to store meta info used by
logger class.

Change-Id: I9edf69073f8c3073519c8e52f4871b99cc4d1b57
Partial-Bug: 1668198
2017-03-01 09:17:16 +00:00
mak-454 3a7db83c84 NFP - Context support in Proxy Agent
Added support to utilize NFP context supported
by nfp/core. Initializes context with default
values in all external functions which are invoked
by rpcs.

Change-Id: Ic42bb90c75e9e17de5250d09e49ff86411bb65d2
Partial-Bug: 1668198
2017-03-01 09:17:05 +00:00
mak-454 4662f535f0 NFP - Failure handling in Orchestrator
Added following support :
1) Context managers,
   1.1) To be used with python 'with' statement.
   1.2) support 'retry', 'ignore' and 'lock' functions.
         -> retry : retry a func for n counts
         -> ignore: Ignore certain expected exceptions.
         -> lock: Lock a db transaction
   1.3) NFP module need not handle all possible exceptions
        as try-except branches.

2) Single class Exception Handling :
   All the exceptions from module will be caught by nfp/core
   and the registered exception handler will be invoked with
   all the relevant details (event, data, context, exception..)

3) Used 'context manager' retry function with client methods,
   Neutronclient, Novaclient etc.. especially for GET methods.
   E.x, GET_TOKEN is retried 'n' times to overcome any
   temporary failures with keystone.

Change-Id: Ia821938b9f607799ebeaa1c0e2ddda74ebc96fd8
Partial-Bug: 1668198
2017-03-01 14:45:02 +05:30
mak-454 0bce1217af NFP - Failure handling in core
Added following support :
1) Watchdog support for,
   1.1) Sequenced events.
   1.2) APIs sent to controller.
   1.3) Works delivered to threads in worker process.

2) Path support,
   2.1) NFP modules can now identify their events with a Path.
        CREATE/UPDATE/DELETE
   2.2) Core offloads handling of following conditions on path,
        -> DELETE events while CREATE is going on.
        -> Discard delayed responses from controller on a path.
        -> UPDATE while an UPDATE is going on.

3) Support for Event Context,
   3.1) Inherently passed along with each event.
   3.2) Modules can choose to override.
   3.3) Maintained as a python GT context so that all methods
        executing in that thread get access without being
        explicitly passed.

Change-Id: I6526737a57271cf8d24d498d97474e8583ccc59d
Partial-Bug: 1668198
2017-02-28 19:08:46 +05:30
Robert Kukura d878c76826 Increase OS_TEST_TIMEOUT
Change-Id: Icd2be56d1ed054e4afa10067efcc96b5b83ea105
2016-12-02 13:49:25 -05:00
Suresh Dharavath 3921f16311 NFP (contrib) - Over the Cloud Network Function Controller
This changeset implements over the cloud network function controller.
The network function controller provides the API server to consume the
REST APIs from under the cloud components required to provision the
network functions rendered over the cloud. It includes a
pecan REST server which receives REST calls from under the cloud and
sends corresponding RPC calls to the configurator.

Change-Id: I74ee6d9ce7d7a362721b5d3b72ee9b4ed1993995
Implements: blueprint gbp-network-services-framework
2016-07-26 05:41:52 -07:00
Sumit Naiksatam 84939e8228 Adding GBP integration and functional gate
This patch implements the hooks so that functional and integration
tests can be run in the group-based-policy repo.

Currently, the configured devstack for this job is enabled to run GBP
exercise scripts, which result in complete end-to-end integration
testing of the GBP service and client components along with integration
with Neutron and Nova.

A placeholder functional test case is also added. A functional test suite
will be added in a separate commit.

Change-Id: Id705e5755b9662f6e297f29836d6983d66f845a8
2015-03-28 12:36:53 -07:00
Ivar Lazzaro af3cbdfc7f Project boilerplate and base modules
Change-Id: I2f7031c44afdc8faad33ba7818a90fe5cdc001cc
2014-09-26 15:16:17 -07:00