Open vSwitch¶
vxlan: non-ECT from ADDRESS with TOS=0x2¶
$ dmesg | tail -n 10
[22645061.223507] vxlan: non-ECT from 10.48.30.11 with TOS=0x2
[22645061.374522] vxlan: non-ECT from 10.48.30.11 with TOS=0x2
[22645061.374590] vxlan: non-ECT from 10.48.30.11 with TOS=0x2
[22645061.374599] vxlan: non-ECT from 10.48.30.11 with TOS=0x2
[22645061.376204] vxlan: non-ECT from 10.48.30.10 with TOS=0x2
[22645061.377116] vxlan: non-ECT from 10.48.30.10 with TOS=0x2
[22645061.411986] vxlan: non-ECT from 10.48.30.11 with TOS=0x2
[22645061.531827] vxlan: non-ECT from 10.48.30.11 with TOS=0x2
[22645062.894827] vxlan: non-ECT from 10.48.30.10 with TOS=0x2
[22645063.367127] vxlan: non-ECT from 10.48.30.11 with TOS=0x2
$ echo N | sudo tee /sys/module/vxlan/parameters/log_ecn_error
$ echo "options vxlan log_ecn_error=N" | sudo tee /etc/modprobe.d/osism-vxlan.conf
wakeup due to [POLLIN] on fd 38 (unknown anon_inode:[eventpoll]) at ../lib/dpif-netlink.c:2786 (99% CPU usage)¶
https://mail.openvswitch.org/pipermail/ovs-discuss/2019-May/048608.html
https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1827264
2020-04-07T05:22:46.785Z|04080|poll_loop(handler111)|INFO|Dropped 1336248 log messages in last 6 seconds (most recently, 0 seconds ago) due to excessive rate
2020-04-07T05:22:46.785Z|04081|poll_loop(handler111)|INFO|wakeup due to [POLLIN] on fd 38 (unknown anon_inode:[eventpoll]) at ../lib/dpif-netlink.c:2786 (99% CPU usage)
$ docker exec -it openvswitch_vswitchd ovs-appctl version
ovs-vswitchd (Open vSwitch) 2.10.0
$ docker exec -it openvswitch_vswitchd bash
# ovs-vsctl set Open_vSwitch . other-config:n-handler-threads=1
# ovs-vsctl set Open_vSwitch . other-config:n-revalidator-threads=1
OVS ports¶
Port with OVS tag
32
docker exec -it openvswitch_vswitchd ovs-vsctl show
...
Port "tap44874148-5b"
tag: 32
Interface "tap44874148-5b"
type: internal
...
Ports with tag 4095 - mark “dead”¶
Port with OVS tag
4095
docker exec -it openvswitch_vswitchd ovs-vsctl show
...
Port "tapc0f9a508-89"
tag: 4095
Interface "tapc0f9a508-89"
type: internal
...
Port in
OpenStack
# openstack --os-cloud admin port list | grep c0f9a508-89
| ID | Name | MAC Address | Fixed IP Addresses |
| c0f9a508-89... | | fa:16:... | {"subnet_id": "subnetUUID", "ip_address": "IP"} |
Sometimes there is no port in
OpenStack
# openstack --os-cloud admin port list | grep c0f9a508-89
| ID | Name | MAC Address | Fixed IP Addresses |
OpenStack
port show
# openstack --os-cloud admin port show c0f9a508-89...
+-----------------------+-----------------+
| Field | Value |
+-----------------------+-----------------+
...
| binding:vif_type | binding_failed |
...
+-----------------------+-----------------+
Port on host
DOWN
# ip address show tapc0f9a508-89
356: tapc0f9a508-89: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2a:fb:4b:a3:45:f2 brd ff:ff:ff:ff:ff:ff
Delete port in
OpenStack
if present
# openstack --os-cloud admin port delete c0f9a508-89...
Delete port in
OVS
# docker exec -it openvswitch_vswitchd ovs-vsctl del-port br-int tapc0f9a508-89
The following command could be used for
monitoring
those ports
# docker exec -it openvswitch_vswitchd ovs-vsctl show | grep -c -B1 "tag: 4095"
5
# docker exec -it openvswitch_vswitchd ovs-vsctl show | grep -B1 "tag: 4095"
Port "tapc0f9a508-89"
tag: 4095
...
Orphaned ports - without tag¶
docker exec -it openvswitch_vswitchd ovs-vsctl show
...
Port "tap7f14056f-61"
Interface "tap7f14056f-61"
type: internal
Port "tap646cf885-cf"
Interface "tap646cf885-cf"
type: internal
...
Those ports drop many packages, this costs CPU time
# docker exec -it openvswitch_vswitchd ovs-ofctl dump-ports br-int
...
port "tap7f14056f-61": rx pkts=, bytes=, drop=123456789, errs=, frame=, over=, crc=
tx pkts=, bytes=, drop=123456789, errs=, coll=
Port in
OpenStack
# openstack --os-cloud admin port list | grep 7f14056f-61
| ID | Name | MAC Address | Fixed IP Addresses |
| 7f14056f-61... | | fa:16:... | {"subnet_id": "subnetUUID", "ip_address": "IP"} |
Sometimes there is no port in
OpenStack
# openstack --os-cloud admin port list | grep 7f14056f-61
| ID | Name | MAC Address | Fixed IP Addresses |
Port on host
DOWN
# ip address show tap7f14056f-61
356: tap7f14056f-61: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2a:fb:4b:a3:45:f2 brd ff:ff:ff:ff:ff:ff
Delete port in
OpenStack
if present
# openstack --os-cloud admin port delete 7f14056f-61...
Delete port in
OVS
# docker exec -it openvswitch_vswitchd ovs-vsctl del-port br-int tap7f14056f-61
The following command could be used for
monitoring
those ports
# docker exec -it openvswitch_vswitchd ovs-vsctl show | grep -A1 "Port " | grep -v tag | grep -i interface | grep -c tap
5
# docker exec -it openvswitch_vswitchd ovs-vsctl show | grep -A1 "Port " | grep -v tag | grep -i interface | grep tap
Interface "tap7f14056f-61"
...