When routers are migrated from one Neutron agent to the other, the metering IPtables
rules are removed, which can cause some exceptions that can be ignored. The metering
agent already handled this situation. However, it logs the message as an ERROR, which
can triggers alarms. Therefore, we propose here to change the LOG message from error
to warning.
Closes-Bug: #1904874
Change-Id: I1805a07cef7fc7d7b041e582a4d79fb1a805df71
If a metering label is removed during a data collection and processing, and
error might happen. This patch address such situations, and creates an execution
flow that avoids such problems.
Change-Id: Iba68173c304dccf97521dcec9a87fce67e21cec2
Extend neutron metering agent to generate Granular metering data.
The rationale here is to have data (bytes and packets) not just in
a label basis, but also in tenant, router, and router-label, and tenant-label
basis. This allows operators to develop more complex network monitoring
solutions.
Moreover, I added documentation to explain what is the neutron metering agent,
its configs, and different message formats.
Change-Id: I7b6172f88efd4df89d7bed9a0af52f80c61acbe0
Implements: https://blueprints.launchpad.net/neutron/+spec/granular-metering-data
Closes-Bug: #1886949
Seems that is_enabled_and_bind_by_default() from
neutron.common.ipv6_utils was copied directly into
oslo_utils.netutils, so start using it instead.
Trivialfix
Change-Id: I00fa441e7a20fcd1115485bb8ab75750e6a8cf07
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues. Didn't think it was going to be
close to 100 files when I started.
Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
This change makes the metering agent retry creating the iptables
managers for each router and applying the metering rules.
This is needed in case the metering agent starts before some or all of
the namespaces are created.
Change-Id: Ifc565feb98c7f02df5c2831a3607c3e526a2e703
Closes-Bug: #1807153
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.
NeutronLibImpact
Depends-On: https://review.openstack.org/#/c/647836/
Change-Id: I3c2f28ecd18996a1cee1ae3af399166defe9da87
Currently the metering iptables driver is using the old method
to load the interface driver. This patch allows it to load
interface driver via stevedore aliases.
Change-Id: I9e3027c1ec695ec3a09db0c53a0388e406e4ced6
Related-Bug: #1504536
The existing method is to meter outgoing traffic from the entire VM
to a specific "remote_ip_addr"(in case of outbound),
but the method I suggested can meter outgoing traffic
from a particular subnet to external network.
From a metering point of view,
it does not matter which address/CIDR is the destination for outbound.
But, it is important to know where(subnet or VM) the packet leaves from.
That way you can see how much VMs are using the external network.
For outbound metering, dst is 0.0.0.0/0 and src is network of VMs
and for inbound metering, src is 0.0.0.0/0 and dst is network of VMs.
When creating a meter-label-rule, it is correct
to specify src address for "remote_ip_prefix" option for outbound
and specify dst address for inbound.
If you are confused by the term "remote_ip_prefix",
you need to clarify this option.
Change-Id: Ia3f3e26410a73d7b44abae637703fda41c4bc09a
Closes-Bug: #1716913
Change network namespace add/delete/list code to use
pyroute2 library instead of calling /sbin/ip.
Also changed all in-tree callers to use the new calls.
Closes-bug: #1717582
Related-bug: #1492714
Change-Id: Id802e77543177fbb95ff15c2c7361172e8824633
Refactoring neutron agent linux and ovsdb config opts
to be in neutron/conf/agent so that all the config options
reside in a centralized location. This simplifies the
process of looking up the config opts and provides an easy
way to import.
NeutronLibImpact
Change-Id: Ib1e0e63dec2985c417412d1ecc68e2a74ef87182
Partial-Bug: #1563069
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
according to https://wiki.openstack.org/wiki/Python3, now we should avoid
using six.iteritems and replace it with dict.items.
Change-Id: I58a399baa2275f280acc0e6d649f81838648ce5c
Closes-Bug: #1680761
If the metering-agent receives a label rule before it
has added the label, it will fail to update the iptables
rules as there are no existing chains.
When the action is "create", check if there is an existing
label, and create one and the corresponding iptables chains,
before trying to add the rule.
Closes-Bug: #1617248
Change-Id: Ia0ec1361188cca53023667d249c2b1e10bc22089
Refactoring Neutron configuration options for agent common config to be
in neutron/conf/agent/common. This will allow centralization of all
configuration options and provide an easy way to import.
Partial-Bug: #1563069
Change-Id: Iebac0cdd3bcfd0135349128921b7ad7a1a939ab8
Needed-By: Ib676003bbe909b5a9013a3178b12dbe291d936af
IPTables driver for metering was not handling the DVR router
namespaces properly for configuring the metering forward rules.
This patch addresses the issue by configuring the iptables
manager based on the availability of the namespace and selecting
the respective namespaces such as router namespace and snat
namespace with the right external device.
Change-Id: I6790d6ff42d9f8fa220e1a231ae94cbf8b60506c
Closes-Bug: #1506567
When syncing data from neutron server, metering-agent may receive
information about routers that are not hosted by the l3-agent on the
same host, because the server didn't filter them out. This could lead to
the following problems:
* metering-agent tries to setup iptables rules for a router that is not
on the host
* metering-agent tries to get get traffic counters for a router that
was once on the host but is already removed
* metering-agent not sets up iptables rules for a router that is
removed then added back to the host, because nothing about the
router is changed from metering-agent's perspective
This commit fixes the aforementioned problems by making metering-agent
only receive information about routers that are on the same host, and
update metering-agent's knowledge about which routers it should care.
However, there could still be problem if one removes then adds a router
back to the same l3-agent, or just sets the router's admin_state_up
property to False then True in a short time(shorter than the interval
between two syncs). Because the metering-agent sees nothing changed
while during the same time the router's namespace is removed and added
back on the host. Thus metering-agent will fail to get such router's
traffic counters. This commit also make iptables-driver to forget such
routers and leave the metering-agent to reconfigure them later.
Closes-Bug: #1580548
Change-Id: Ia6ae82c676582b06710d6f96b9938c215258182d
Signed-off-by: Hunt Xu <mhuntxu@gmail.com>
Subprojects have had enough time to switch to neutron-lib for
the deprecated items from these modules.
Change-Id: Ib2f4acf7e0153667389c57a52f2db33a23758b24
IPv6 utils is_enabled() doesn't actually determine if IPv6 is enabled on
the host. It checks if /proc/sys/net/ipv6/conf/default/disable_ipv6 is
present and is set to 0. This kernel configuration option controls if
the kernel will automatically assign IPv6 link-local addresses to newly
created network interfaces when their link state changes to up. The
existence of this /proc files does indicate that the Linux kernel has
the ipv6 module loaded or ipv6 was compiled in. Having this /proc file
set to zero does not indicate IPv6 is not available on the system, just
that newly created interfaces will inherit this configuration and will
not have IPv6 addresses bound to them unless the administrator changes
the interfaces specific /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6
configuration.
This check was added to Neutron so it could operate with distributions
which didn't load the ipv6 kernel module, preventing errors when
attempting to make IPv6 specific configurations in the iptables firewall
driver and the L3 agent. Removing it would break existing deployments.
Renaming this function to provide clarity for complex conditions tested
by this function. In fact it is a good security practice to set this
default disable_ipv6 option to 1, and explicitly enable IPv6 by setting
disable_ipv6=0 on individual interfaces which the administrator intends
to bind IPv6 addresses on. This establishes parity with IPv4 behavior
where interfaces are not active in an address family until the
administrator explicitly configures them to be active in that address
family. This practice does not currently work as expected with the
Neutron, since setting /proc/sys/net/ipv6/conf/default/disable_ipv6 to 1
unexpectedly disables creating IPv6 security group rules leaving
instances completely exposed via IPv6 regardless of security group
rules.
Change-Id: I844b992240a5db642766ec9c04e3b5fcab8e2e23
When update meter label or rule, iptables_manager will update iptables
rule in router's namespace. In order to, it will clean traffic counter
number collected in interval time, the other iptables always trashing
that will clean old iptalbes rule and generate new same significance
iptables rule.
Change-Id: Ide2b26c98587258175234acded38ce481b7e7f76
Closes-Bug: #1618879
If router has no gateway port when metering-agent wants to add
a metering-label-rule method _process_metering_label_rules() fails
with error "cannot concatenate 'str' and 'NoneType' objects"
because there is no check that router has an external gateway port.
This patch adds this check and adds some unit test.
Closes-bug: #1527274
Change-Id: Ic9f626db41bfb6343187742e209402dd7d5232d1
If state_less parameter is not specified then
neutron-postrouting-bottom rule goes up in POSTROUTING
chain, which causes premature NATing of traffic,
for ex. traffic between internal networks becomes NATed.
Closes-Bug: 1544508
Co-Authored-By: Sergey Belous <sbelous@mirantis.com>
Change-Id: I2e0011237d50a59d417cfee01dcd5f9d0da2e7f5
In the case of outbound traffic, set remote_ip to dst.
In the case of inbound traffic, set remote_ip to src.
Change-Id: I7f27b93efa67baf3efccaa94f6a1337d6886e230
Closes-Bug: #1528137
DocImpact: Clarify remote_ip_prefix description of metering label rule in API site
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
The use_namespaces option has been defined as a workaround to kernels
not properly supporting namespaces. This limitation is behind us, it's
time to remove use_namespaces after its deprecation in Kilo in order to
simplify code and remove a poorly tested case (use_namespaces=False).
This change prepares for removal pullup_route method[1] which was only
used when use_namespaces=False.
[1] neutron.agent.linux.ip_lib
DocImpact
UpgradeImpact
Closes-Bug: #1508188
Related-Bug: #1435382
Depends-On: I303038eec560a6d99421140c2822aed8b518470b
Depends-On: I4feb2a15c7e1e4bfdbed2531b18b8e7d798ab3cc
Change-Id: I2fbf65df1250d9f9f1656b3964ee3b6de1ef1118
oslo.log now provides a logging helper that is similar to custom neutron
helper (actually, the helper in oslo.log started from neutron version).
Now switching to library implementation.
Deprecated neutron.common.log.log
Change-Id: I85d5fc570950ff18cfdb8db20ad20b166e195299
Now we send all labels and rules per rule create/delete
and rebuild whole iptables chains.
In this patch we send only affected rule and create/
delete only this rule from iptables.
Change-Id: I58ebd8d810c62980c09a340ee1680be17c12b74a
Closes-Bug: #1400280
The metering agent will fail if one of the iptables chains
is missing, which will cause errors extracting data from
all the other chains. Add a simple try/except to let the
loop continue.
Closes-Bug: #1421037
Change-Id: I370ee0e2cc58ca7e1c5ef9bf4dbcce5abf7545a1
It's mostly a matter of changing imports to a new location.
Non-obvious changes needed:
* pass overwrite= argument to oslo_context since oslo.log reads context
from its thread local store and not local.store from incubator
* don't store context at local.store now that there is no code that
would consume it
* LOG.deprecated() -> versionutils.report_deprecated_feature()
* dropped LOG.audit check from hacking rule since now the method does
not exist
* WritableLogger is now located in oslo_log.loggers
Dropped log module from the tree. Also dropped local module that is now
of no use (and obsolete, as per oslo team).
Added versionutils back to openstack-common.conf since now we use the
module directly from neutron code and not just as a dependency of some
other oslo-incubator module.
Note: tempest tests are expected to be broken now, so instead of fixing
all the oslo.log related issues for the subtree in this patch, I only
added TODOs with directions for later fix.
Closes-Bug: #1425013
Change-Id: I310e059a815377579de6bb2aa204de168e72571e
This should remove the last bits of passing root_helper around
for execute(). Some tests store and pass the root_helper argument
outside of this, and have been left alone.
Depends-On: I1e3b64e5a1d6cff2aebc638710487bbdbdba61d4
Depends-On: I5d51fa5edc7f427d84d7ae49d3a4c972dca8ac37
Depends-On: I21c6e7795c8b2e8f87bbd2b00a8eb7cb215ece74
Depends-On: I6566e17a1ce07a3bebf2a1a3653ad7e7f397f0b9
Change-Id: I38c901c65fb0c8598589ec56d5afd50d2f882f4f
Partially-Implements: blueprint rootwrap-daemon-mode
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.
This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils
Added hacking check to enforce new import paths for all oslo libraries.
Updated setup.cfg entry points.
We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.
[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/
Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
Mostly trivial import changes.
- oslo.i18n no longer provide install() method to inject _() into
globals(), so removed all calls to it;
- removed Babel from dependencies (it will now be grabbed by oslo.i18n);
- updated tox.ini to ignore import violations for oslo.i18n.
Change-Id: I6623d551f512fb7fe9bf35ee734ed6d4c6cbc287
This change enforces log hints use and removes debug level log
translation, modifications are validated through a hacking rule.
Validate that hacking rules apply to directories:
- neutron/services/metering
Change-Id: Ic763e81e8b06973013115af45c7c09a3e69c673b
Partial-bug: #1320867
We have git to track authorship, so let's not pad source files
with it as well.
A hacking check has been added for this. The value is N322.
Change-Id: Iab0b64d417e0bb41a6b455e2ac377deee64ec3ee
The existing code incorrectly used the stale value from a previous list
comprehension - and deleted the chains for the wrong router :(
(Found via pylint)
Also: change to using a set() rather than a list(), since it is used for
repeated membership tests.
Also: refactor test cases to remove test case duplication.
Closes-Bug: #1362466
Change-Id: I4df400d57bab5427362db47a715576faa6340173
On systems where ipv6 module is not loaded in kernel we need to avoid
usage of ip6tables. This patch reads
/proc/sys/net/ipv6/conf/default/disable_ipv6 file and if ipv6 is
disabled then ip6tables are not used in IptablesManager
Closes-Bug: #1352893
Change-Id: I07e5851aa25eb98b7a97dff86b9850475df85f64
While adding iptables rule, cidr is added as destination for both ingress
and egress directions. Modified code to add -s for egress and -d for ingress.
Closes-bug: 1310589
Change-Id: Id9ca10855e6527d4bec689f8f9bcd6f681221954
Rajesh Tailor