Hi -
We're trying to convert some of our Linux VMs from using the e1000 network driver to the VMXNET3 driver. We've encountered a problem with NFS when using the VMXNET3 driver, where performance is inconsistent and is frequently very bad. One of the benchmark tools we're using is called ioping. It shows that most requests are serviced in a reasonable timeframe, but that sometimes a request will stall for minutes. See the output from 30 pings worth of ioping data below:
# ioping -p 10 -s 3400000 .
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=1 time=272685.9 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=2 time=76.4 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=3 time=74.8 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=4 time=74.0 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=5 time=74.3 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=6 time=82.7 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=7 time=75.9 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=8 time=115.7 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=9 time=75.2 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=10 time=74.8 ms
73981 27340976 272685878 81781635
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=11 time=74.0 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=12 time=74.6 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=13 time=77.2 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=14 time=73.9 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=15 time=197838.4 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=16 time=127333.3 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=17 time=96.2 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=18 time=95.0 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=19 time=95.2 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=20 time=95.2 ms
73884 32585284 197838418 66884872
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=21 time=94.3 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=22 time=133.8 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=23 time=193947.3 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=24 time=138989.9 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=25 time=93.7 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=26 time=93.7 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=27 time=94.0 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=28 time=132.0 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=29 time=93.7 ms
3400000 bytes from . (nfs xx.xx.xxx.xx:/perf_test/): request=30 time=94.5 ms
93689 33376696 193947349 67671133
The VMXNET interface and the NFS server are on different physical switches, but are on the same VLAN.
Running tcpdump on the inteface puts the interface into promiscuous mode, which makes the problem go away. When in promiscuous mode, typical request times are in the 20-40 ms range, rather than the 70-95 ms range that the "good" pings here have.
I've reviewed Poor TCP performance may occur in Linux virtual machines with LRO enabled (1027511) which describes similar problems. But setting LRO on my NICs is not an avialable option.
We're running Oracle Linux 5.8 on the Guest OS. We're running VMWare ESXi 5.0.0, 914586. The NFS server is a Hitachi NAS (fka Bluearc). From the network side, the ESXi host is connected to a Cisco 5000 switch by 4 10g trunked connections. The HNAS is connected bia 10g connections to a different Cisco 5000 switch.
Any thoughts or suggestions would be greatly appreciated.
Thanks!
-steve