Merge "Layer 3 IPs rate limit"

This commit is contained in:
Jenkins 2017-06-15 20:16:38 +00:00 committed by Gerrit Code Review
commit 61be2e9538
1 changed files with 363 additions and 0 deletions

View File

@ -0,0 +1,363 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
======================
Floating IP rate limit
======================
RFE: https://bugs.launchpad.net/neutron/+bug/1596611
Floating IP bandwidth is unrestricted now. But the NIC and the export of data
center may have a limitation for a cloud deployment. Then floating IP rate
limit is needed.
This spec describes how to add a built-in extension to limit the floating IP
bandwidth.
Problem Description
===================
Neutron now has floating IP whose bandwidth is not restricted, then there are
several reasons for adding rate limit to floating IP:
* a) Currently, neutron QoS implementation only affects neutron ports, more
detail is that the bandwidth restriction is based on the neutron port, so all
the VM traffic will be limited under that restriction, including L3 traffic.
L3 needs its IP based implementation.
* b) North/South traffic always rely on infrastructure networking capacity.
* c) In case of Floating IP Traffic, cloud deployment do not have enough
bandwidth to meet the total bandwidth requirement of all tenants at the
same time. Due to this, the SLA of user's network traffic cannot be
guaranteed. For example, if the VM NICs are given a high QoS value, then
to meet the requirement of each tenant, the DC network would have a heavy
load, thus it may be unable to facilitate additional requests and affect
not only the tenant network bandwidth from North-South, but also East-West
traffic.
* d) SNAT traffic in centralized network node will not meet the needs of all
tenant bandwidth either. Because a NIC has limited bandwidth.
Proposed Change
===============
Overview
--------
We want to limit the bandwidth of floating IP. At the same time, the east/west
traffic should have no effect.
Some agreements:
* Allow binding QoS policy to floating IP.
* Each floating IP should have only one QoS policy.
* L3 agent needs a QoS extension to process L3 IP related QoS rules.
* QoS policy parameter is nullable. Floating IP can have no QoS policy
assigned.
How the change works:
* Linux TC (Traffic Control) [1]_ will be used to implement such functionality.
And for egress traffic a HTB [2]_ qdisc will be used to limit the floating
IP outgoing bandwidth.
Solution Proposed
-----------------
Server side changes
+++++++++++++++++++
* New floating IP extension for QoS attribute (qos_policy_id).
* Adding a new relationship table between floating IP and QoS
policy ``QoSFIPPolicyBinding``.
* L3 plugin will return the floating IP list with its bonding QoS policy
during router sync RPC.
L3 agent side TC rules
++++++++++++++++++++++
Where to install the TC rules:
* For HA/legacy routers, rules for floating IP will be installed into network
node qrouter namespace, and the traffic control device will be qg-device.
* For DVR routers, it's compute node qrouter namespace, but the floating IP
traffic control device will be rfp-device (one of qrouter-namespace to
fip-namespace pair).
For all IPs, in the corresponding namespace, the egress and ingress qdisc
rules are basically the same. The ``{qdisc_id}`` will be used to create the
IPs' tc filter. Some tips:
* Remember all the following commands will be executed in a namespace.
* [a-device] represents the qg-device or rfp-device in a namespace.
* {egress/ingress_qdisc_id} is the qdisc handle id.
Create qdisc commands:
::
# egress
tc qdisc add dev [a-device] root htb
# ingress
tc qdisc add dev [a-device] ingress
Actual qdisc rules:
::
$ tc qdisc show dev [a-device]
# egress
qdisc htb {egress_qdisc_id} root refcnt 2 r2q 10 default 0 direct_packets_stat 400
# ingress
qdisc ingress {ingress_qdisc_id} parent ffff:fff1 ----------------
Then we can use that {ingress_qdisc_id} and {egress_qdisc_id} to create the
L3 IPs filters. Assuming we have a L3 IP 172.16.6.161, and it has a QoS policy
with rule {qos_rate_value: 1000Kbit, qos_burst_value: 1Mb}. The ingress and
egress tc filter create commands will be:
::
# ingress
$ tc filter add dev [a-device] parent {ingress_qdisc_id} protocol ip prio 1 \
u32 match ip dst 172.16.6.161 police \
rate 1000Kbit burst 1Mb drop flowid :1
# egress
$ tc filter add dev [a-device] parent {egress_qdisc_id} protocol ip prio 1 \
u32 match ip src 172.16.6.161 police \
rate 1000Kbit burst 1Mb drop flowid :1
Actual filter rules:
::
# ingress
$ tc -s -d -p filter show dev [a-device] parent {ingress_qdisc_id}
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
match IP dst 172.16.6.161/32 (success 0 )
police 0x67 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
...
# egress
$ tc -s -d -p filter show dev [a-device] parent {egress_qdisc_id}
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
match IP src 172.16.6.161/32 (success 0 )
police 0x68 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
...
The neutron basic workflow
--------------------------
Floating IP
+++++++++++
* 1. User create a floating IP.
* 2. Floating IP was associated with a port.
* 3. Create/find QoS policy and rule for floating IP.
* 4. Bind QoS policy to floating IP (floating IP update API).
* 5. L3 agent processes the router and its floating IP.
* 6. L3 agent sets the TC rules to the qrouter-namespace relevant device.
After this the floating IP will be set under a bandwidth restriction.
Example L3 agent side TC rules
------------------------------
Assuming that we have a legacy router: cf6951cb-b050-4543-9742-c63a4989edae,
gateway ip: 172.16.6.167, 1Mbps, floating IP: 172.16.10.69, 1Mbps, then in
its scheduled network node you can get the following rules:
::
# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
inet 127.0.0.1/8 scope host lo
140: qr-4aba9b05-36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
inet 192.168.233.1/24 brd 192.168.233.255 scope global qr-4aba9b05-36
141: qr-aef0d42d-f9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
inet 192.168.232.1/24 brd 192.168.232.255 scope global qr-aef0d42d-f9
143: qg-c99a5832-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN
inet 172.16.6.167/16 brd 172.16.255.255 scope global qg-c99a5832-7f
inet 172.16.10.69/32 brd 172.16.10.69 scope global qg-c99a5832-7f
valid_lft forever preferred_lft forever
# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae tc qdisc show dev qg-c99a5832-7f
qdisc htb 801b: root refcnt 2 r2q 10 default 0 direct_packets_stat 400
qdisc ingress ffff: parent ffff:fff1 ----------------
# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae tc -s -d -p filter show dev qg-c99a5832-7f parent 801b:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
match IP src 172.16.6.167/32 (success 0 )
police 0xa74 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
filter protocol ip pref 1 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
match IP src 172.16.10.69/32 (success 0 )
police 0xa76 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae tc -s -d -p filter show dev qg-c99a5832-7f parent ffff:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
match IP dst 172.16.6.167/32 (success 0 )
police 0xa73 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
filter protocol ip pref 1 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
match IP dst 172.16.10.69/32 (success 0 )
police 0xa75 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
And DVR router 0580788c-c919-447c-aea1-87d415aa173a with floating IP
172.16.6.161 1Mbps in a compute node:
::
# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
inet 127.0.0.1/8 scope host lo
24: rfp-0580788c-c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
inet 169.254.106.114/31 scope global rfp-0580788c-c
inet 172.16.6.161/32 brd 172.16.6.161 scope global rfp-0580788c-c
102: qr-43185e93-af: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
inet 192.168.199.1/24 brd 192.168.199.255 scope global qr-43185e93-af
104: qr-51635a61-46: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
inet 192.168.198.1/24 brd 192.168.198.255 scope global qr-51635a61-46
# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a tc qdisc show dev rfp-0580788c-c
qdisc htb 801e: root refcnt 2 r2q 10 default 0 direct_packets_stat 5
qdisc ingress ffff: parent ffff:fff1 ----------------
# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a tc -s -d -p filter show dev rfp-0580788c-c parent 801e:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
match IP src 172.16.6.161/32 (success 0 )
police 0x68 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a tc -s -d -p filter show dev rfp-0580788c-c parent ffff:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
match IP dst 172.16.6.161/32 (success 0 )
police 0x67 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
IPv6 Impact
-----------
Only IPv4 has the concept of floating IP.
Data Model Impact
-----------------
For floating IP:
A new table, QoSFIPPolicyBinding, will be created to represent the 1-1
association between floating IP and QoS policy, each policy has rules in both
ingress and egress direction.
REST API Impact
---------------
New floating IP API extension:
::
floatingip: {
...
'qos_policy_id': {'allow_post': True, 'allow_put': True,
'validate': {'type:uuid_or_none': None},
'is_visible': True,
'default': None},}
New floating IP commands for creation, deletion and update elements with
QoS property:
::
openstack floating ip create --qos-policy policy_id <network>
openstack floating ip set --qos-policy policy_id floating_ip_id
openstack floating ip unset --qos-policy floating_ip_id
Implementation
==============
Assignee(s)
-----------
* LIU Yulong <i@liuyulong.me>
Work Items
----------
* Create model tables and API extensions.
* New TC command wrapper for floating IP rate limit.
* L3 agent extension for QoS rule installation.
* CLI support.
* Testing.
* Documentation.
Dependencies
============
None
Testing
=======
Functionality
-------------
The bandwidth of floating IP should be restricted properly. Adding new
fullstack and tempest scenarios to test the bandwidth of ingress and egress.
References
==========
.. [1] `Linux Traffic Control <http://www.tldp.org/HOWTO/Traffic-Control-HOWTO>`_
.. [2] `Linux HTB Queuing <http://www.linuxjournal.com/article/7562>`_
.. [3] `[WIP] Floating IP rate limit <https://review.openstack.org/#/c/424466/>`_
.. [4] `[WIP] Router gateway rate limit (abandoned) <https://review.openstack.org/#/c/424468/>`_
.. [5] `[WIP] Adding L3 rate limit TC lib <https://review.openstack.org/#/c/453458/>`_