openstackdocstheme generates the toc tree in the left sidebar
based on the page hierarchy from the top page.
The previous developer guide hirarchy is a bit deep, and
this commit reorganizes the devref pages for better navigation
with openstackdocstheme.
Change-Id: I1412b047efc1c268b34ef97e78073da7bcbb6d7e
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #1 from above, shimming neutron's callbacks and
removing devref + UTs. Rather than shimming using debtcollector, this
patch leaves callback constants as-is, and simply references the lib
class/function in its respective neutron callback module. This allows
consumers to test callback types without changing code. For example,
an except block block like that below continues to work even though
the raised exception now lives in lib::
try:
neutron_cb_registry.notify(...)
except neutron_cb_exceptions.CallbackFailure:
handle_exception()
In addition this patch contains minor UT updates to support the shim
approach.
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
Change-Id: Ib6baee2aaeb044aaba42a97b35900d75dd43021f
To introduce to every neutron developer the concept of using objects,
this dev ref is describing the ecosystem of Oslo VersionedObjects
and its modification for neutron project, how to create or use the objects.
Co-Authored-By: Dariusz Smigiel <smigiel.dariusz@gmail.com>
Change-Id: If25060feb77f35873a2b6a4ecfe639a284e7f677
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
This adds a service plugin to introduce random delays
and deadlocks to DB operations to make it easier for
us to see that retry decorators are correctly applied
and race conditions are handled.
Change-Id: I8e283c1b53165faee548d26b3560a2c883dfb977
Retrying mutating operations at the API layer caused a
couple of problems. First, when components would call
the core plugin using the neutron manager, they would
have to handle the retriable failures on their own or
undo any work they had done so far and allow retriable
failures to be propagated up to the API. Second, retrying
at the API makes composite operations (e.g. auto allocate,
add_router_interface, etc) painful because they have to
consider if exceptions are retriable before raising
fatal exceptions on failures of core plugin calls.
This patch begins the process of moving them down to the
core operations with a new decorator called
'retry_if_session_inactive', which ensures that the
retry logic isn't triggered if there is an ongoing transaction
since retrying inside of a transaction is normally ineffective.
Follow-up patches apply them to various parts of the code-base.
Additionally, the args and kwargs of the method are automatically
deep copied in retries to ensure that any mangling the methods
do to their arguments don't impact their retriability.
Finally, since we are leaving the API decorators in place for now,
the retry logic will not be triggered by another decorator if an
exception has already been retried. This prevents an exponential
explosion of retries on nested retry decorators.
The ultimate goal will be to get rid of the API decorators entirely
so retries are up to each individual plugin.
Partial-Bug: #1596075
Partial-Bug: #1612798
Change-Id: I7b8a4a105aabfa1b5f5dd7a638099007b0933e66
The L3 agent extensions devref was not properly included into the
documentation tree, so it would not appear in the online devref. Also,
there were a few updates to the agent extensions documentation to
properly show the updated file names.
Change-Id: I5f55f68d2c251dfa993668024ee312085b605a8a
The db_models.rst file was not referenced by any other document. This
was generating warnings during the creation of documentation.
Change-Id: I432aabb3c22b12faa29ca88e13392e6c2d0e33d8
Closes-Bug: #1611546
This adds a check to each of the core methods that other callers
(e.g. service plugins) may use to manipulate core resources. This
check prevents them from passing in a context that is already part
of an ongoing DB session because we do not want DB rollbacks to be
allowed after the ML2 plugin calls postcommit methods on drivers.
All of the core methods are protected except for create_port and
update_port. This was left out because of a few particularily deeply
nested calls to the port methods from the L3 code that will be
addressed in change I5aa099c2264636336ab0b76c0826b506e2dc44b6.
For more details, read the devref added by this patch.
Partial-Bug: #1540844
Change-Id: I9924600c57648f7eccaa5abb6979419d9547a2ff
This change generalizes agent extension code so that all agents can take
advantage of a common mechanism.
Co-Authored-By: Margaret Frances <margaret_frances@cable.comcast.com>
Partially-Implements: blueprint l3-agent-extensions
Change-Id: I9380343c09d28eec67077c9e6d77c33a195e516b
Sometimes an object requires multiple disjoint actors to complete
a set of tasks before the status of the object should be transitioned
to ACTIVE. The main example of this is when a port is being created.
The L2 agent has to do its business to wire up the VIF, but at the same
time the DHCP agent has to setup the DHCP reservation. This led to
Nova booting the VM when the L2 agent was done even though the DHCP
agent may have been nowhere near ready.
This patch introduces a provisioning blocks mechansim that allows the
entities to be tracked that need to be involved to make a transition
to ACTIVE happen. See the devref in the dependent patch for a high-level
view of how this works.
The ML2 code is updated to use this new mechanism to prevent updating
the port status to ACTIVE without both the DHCP agent and L2 agent
reporting that the port is ready.
The DHCP RPC API required a version bump to allow the port ready
notification.
This also adds a devref doc for the provisioning_blocks
module with a high-level overview of how it works in addition
to a detailed description of how it is used specifically with
ML2, the L2 agents, and the DHCP agents.
Closes-Bug: #1453350
Change-Id: Id85ff6de1a14a550ab50baf4f79d3130af3680c8
The change Ib30f681ccd9177c45d2c17d28b4a46ac5462df7c tried to remove this
file, but it only removed the content of this file.
Change-Id: I82769091947d3c593509f1d2012cd6f0bc371da5
Instrumentation work from regxboi has stalled. For what we know this
feature may never materialize, and if it does it may end up looking
drastically different.
This reverts commit 1f329c5d01.
Change-Id: I872c6b981d123e7f291ac950b08c3fe5ed95769c
Introduce a generic mechanism to allow the user to set tags
on Neutron resources. This patch adds the function for "network"
resource with tags.
APIImpact
DocImpact: allow users to set tags on network resources
Partial-Implements: blueprint add-tags-to-core-resources
Related-Bug: #1489291
Change-Id: I4d9e80d2c46d07fc22de8015eac4bd3dacf4c03a
Implements an API extension for reporting availibility of IP
addresses on Neutron networks/subnets based on the blueprint
proposed at https://review.openstack.org/#/c/180803/
This provides an easy way for operators to count the number of
used and total IP addresses on any or all networks and/or
subnets.
Co-Authored-By: David Bingham <dbingham@godaddy.com>
Co-Authored-By: Craig Jellick <craig.jellick@gmail.com>
APIImpact
DocImpact: As a new API, will need all new docs. See devref for details.
Implements: blueprint network-ip-usage-api
Closes-Bug: 1457986
Change-Id: I81406054d46b2c0e0ffcd56e898e329f943ba46f
This firewall requires OVS 2.5+ version supporting conntrack and kernel
conntrack datapath support (kernel>=4.3). For more information, see
https://github.com/openvswitch/ovs/blob/master/FAQ.md
As part of this new entry points for current reference firewalls were
added.
Configuration:
in openvswitch_agent.ini:
- in securitygroup section set firewall_driver to openvswitch
DocImpact
Closes-bug: #1461000
Co-Authored-By: Miguel Angel Ajo Pelayo <mangelajo@redhat.com>
Co-Authored-By: Amir Sadoughi <amir.sadoughi@rackspace.com>
Change-Id: I13e5cda8b5f3a13a60b14d80e54f198f32d7a529
The L3 agent needs to know the address scope of each port of each
router it sets up in order to enforce isolation between scopes.
This commit adds a devref for the address scopes and subnet pools
features.
Change-Id: I6a7b3708fadefff1919d70ab1b8bc345b3fbe81c
Partially-Implements: blueprint address-scopes
Presents what instrumentation is available from VIFs in Nova,
Metering Lables and Rules, Linux Bridge, and OVS. Proposes
mappings for structures defined in RFC 2863 and RFC 4293 and
the method that will be followed for a data collection proof
of concept.
How to aggregate and consume these counters will be covered
in future patch sets that extend this devref.
Change-Id: I6c1ad0c4cf60d0069c5e057d77f75c12b04a020c
Partial-bug: #1475736
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
The page is intended to describe current upgrade features Neutron
supports, lay out potential improvements, describe testing strategy for
existing and planned upgrade features, and provide guidelines to
reviewers on where to look for potential upgrade breakages in proposed
patches.
Change-Id: I22e55bb2fe32b32d12fa5889b91ecb9f92b3e6a6
This commits moves the Neutron Stadium documentation to the toplevel
portion of the documentation tree.
Change-Id: I0ec585975c453a9dd08d22293bb68dbc775802e6
Signed-off-by: Kyle Mestery <mestery@mestery.com>
This patch adds a policy.rst document which describes how the
neutron.policy module works, how it uses oslo_policy and how
it's wired in API controllers.
The patch also remove an obsolete docstring from neutron/policy.py
Change-Id: I3353a23f4e97824a5a91eaedf57a91ace031a9de
Have you ever wondered why you were making the same mistakes
over and over again and wondered how you can remind yourself not
to repeat them again?
Have you ever felt like you posted review comments on some other
patch, only to see that the same anti-pattern was adopted in
someone else's?
Have you wondered what the heck that test was meant to validate?
Does that sound familiar to you? Yes? Great, we have the answer
for you! 'Effective Neutron' is the solution to all your problems!
From now on, everytime you bang your head against the monitor, do
not despair! You may find the answer to your grief in our collective
guide!
Go check it out!
(Your mileage may vary -- add your disclaimer here)
Change-Id: If1b8c1d04e1204985a4f9febb775c4fdb20305e9
I was involved lately in several boring attempts to fix broken
sub-project requirements and stable branches, and I think part of the
problem here is that we don't communicate our expectations to
sub-projects clear enough.
This is a first attempt to set brief and clear guidelines to sub-project
maintainers on how to maintain their repositories without much hassle.
Change-Id: I9180ee530f96a885b5667e050d141ce0ab52a8ce
This change adds tempate for ModelMigrationTest which should be
implemented in all driver/plugin repositoties that were split
out from Neutron.
Also split DRIVER_TABLES into separate lists for each driver.
This is needed for easier implementation of test.
Closes-bug:#1470678
Change-Id: I02100d15d71519014db7c8894bd2cb36c86d32a0
This patch adds developer documentation for quota management
and enforcement.
Partially-Implements blueprint better-quotas
Change-Id: Ia990484caf6f5818104109e3d28e2990b9347726
Note to reviewers: gerrit diff for merge patches is very limited, and
leaving comments in global section won't scale, so please comment here:
https://etherpad.openstack.org/p/qos-merge-back-review
This merge commit introduces QoS feature into Liberty release of
Neutron.
The feature is documented in: doc/source/devref/quality_of_service.rst
included with the merge patch.
It includes:
- QoS API service plugin with QoS policy and QoS bandwidth limit
(egress) rule support;
- core plugin mechanism to determine supported rule types, with its ML2
implementation;
- new agent extension manager;
- QoS agent extension with pluggable backend QoS drivers (Open vSwitch
and SR-IOV support is included).
To extend network and port core resources with qos_policy_id attribute,
a new ML2 extension driver (qos) was introduced that relies on the QoS
core resource extension (the idea is that eventually we'll get a core
resource extension manager that can be directly reused by core plugins).
Agent-server interaction is based on:
- get_device_details() method that is extended with qos_policy_id;
- a new push/pull mechanism that allows agents and servers to
communicate using oslo.versionedobjects based objects sent on the
wire.
The merge includes the following types of test coverage:
- unit tests;
- functional tests for OVS agent, QoS agent extension, and low level
ovs_lib changes;
- API tests to cover port/network qos_policy_id attribute and new QoS
resources.
The client changes can be found at:
* https://review.openstack.org/189655
* https://review.openstack.org/198277
The team also prepared fullstack test but it needs to wait for client
merge before it can pass in the gate:
* https://review.openstack.org/202492
Gerrit does not show diff for merge changes that did not result in any
conflict, so to facilitate review, rely on the following steps:
- fetch the patch locally
- git fetch origin
- git diff origin/master...
This merge also disables qos extension API tests until the service is
enabled in master gate.
Local changes apart from conflicts:
- updated down_revision for qos migration to reflect master expand head;
- disabled qos API tests with gate_hook.sh until we have it enabled in
master gate;
- bumped oslo.versionedobjects requirement to reflect what is in
openstack/requirements' global-requirements.txt
DocImpact
APIImpact
Partially-Implements: blueprint quantum-qos-api
Partially-Implements: blueprint ml2-qos
Partially-Implements: blueprint ml2-qos-ovs-bwlimiting
Partially-Implements: blueprint ml2-sriov-qos-with-bwlimiting
Change-Id: I92916d0e391791187e9a25ff172fb4b3504857b1
Sub-projects shall now register their independent alembic migrations
via entrypoints in setup.cfg, and neutron-db-manage will discover them
and run them automatically.
If a service or sub-project is specified explicitly, then
neutron-db-manage will run on only that service or sub-project.
The advanced services project are just special cases of sub-projects.
For example, specifying the CLI option '--service lbaas' is the same
as specifying '--subproject neutron-lbaas'.
Specifying no service or sub-project will cause neutron-db-manage to
run the command on neutron and all installed sub-projects.
Added and consolidated documentation into devref for alembic migrations.
Partial-Bug: #1471333
Partial-Bug: #1470625
Change-Id: I9a06de64ce35675af28adf819de6f22dc832390d
The lack of convention for heading levels among the independently
written devref documents was starting to make the Table of Contents
look rather messy when rendered in HTML.
This patch does not cover the "Neutron Internals" section since its
layout is reasonably OK for now.
Change-Id: I827c105599f05773bda7e4fc0a941ce04ebd51fa
Currently, there is no dns servers prioritization for subnets
for Neutron.
Generally speaking, it is useful to keep the order of dns
nameservers consistent. Add a new column named 'order' in table
'dnsnameservers' and add nameserver into DB one by one.
Closes-Bug: #1218629
Implements: blueprint keep-dns-nameserver-orderconsistency
Change-Id: Id937aea411397d39370368a4eb45be26c4eefa9e
Also applied the following fixes:
===
1. cleaned up some pylint failures that were not spotted before:
Module neutron.objects.qos.policy: Metaclass class method __new__ should
have 'mcs' as first argument
Module neutron.objects.qos.rule: Lambda may not be necessary
===
2. Revert "Introduce the AFTER_READ callback for ports and networks"
This reverts commit e3dba14241.
We don't use callbacks to extend resources anymore, instead relying on
ml2 extension drivers. No need for the patch to achieve QoS, and it also
breaks test_delete_subnet_with_callback that was added in master
recently.
===
3. updated requirements.txt and test-requirements.txt based on:
https://review.openstack.org/#/c/204398/
to avoid requirements gate checks failing due to incompatible
requirements comparing to global-requirements.txt
Change-Id: I744ab2d8327a428a5467f2d07d073a5f8c333520
Make sure we document the fact that neutron.openstack.common.* contents
are not meant to be used by external repositories (except, temporarily,
*aas repos).
If I could bootstrap the oslo-incubator subtree from scratch, I would
put it under neutron._openstack, to indicate that it's for internal
usage only. But we can't do it now, so instead I update devref.
Change-Id: I42252a7b0a07759c57995b2fc1f8d20ecba7d33b
This is a publisher/subscriber messaging mechanism optimized
for agent consumption and server production without the need
of creating new rpc messages when new resources are introduced.
Oslo versionedobjects are the perfect match to ensure
cross version compatibility even if the published/subscribed
resources format change over time.
This is still a basic stub allowing get_info of the resources,
and the next change will introduce the RPC methods to call
get_info: I0ac8a009e781b6edb283d8634b1a2f047db092dc
The plugin is returning stub objects to be consumed from the
agent to test the basic behaviour until we have DB.
TODO: Update documentation, according to code changes,
enforce versioned objects only doing deserial/serialization.
Co-Authored-By: Miguel Angel Ajo <mangelajo@redhat.com>
Co-Authored-By: Eran Gampel <eran@gampel.net>
Change-Id: I524cf5a14e99dc6bee4d4261557d98c75efa0809
Since I433126a8247e7e1c316f2c96bb21e15582b247ce, doc warnings are
considered as failures in gate.
Unlinked .rst file generates one, making docs job broken for
feature/qos. Same for an empty file with no title.
Change-Id: Iba82d9728e99238bcc55b12f0ab9eb936fd62147
The goal of this doc is to communicate what are full stack tests,
how they benefit you and when would you write such a test.
Additionally I'd like to communicate the way forward, and gather
feedback about any areas in the code that can benefit from full
stack tests, and any additional thoughts!
Change-Id: Ifd4ff9be0ed0184a49df6566d238c31a328cd23f
Akihiro Motoki