kubernetes - 为什么clusterip(iptables)的性能这么差？是不是我的配置有问题

Question

我用wrk测试集群性能，wrk和pod(nginx)都在同一个vm上。</p>

1、wrk->podIp

./wrk -H "Connection: Close" -t 4 -c 300 -d30  http://{podIp}:80
Running 30s test @ http://{podIp}:80
  4 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    20.75ms   24.25ms 838.73ms   97.79%
    Req/Sec     2.16k   450.42     3.49k    69.23%
  258686 requests in 30.07s, 208.46MB read
  Socket errors: connect 0, read 0, write 5, timeout 0
Requests/sec:   8603.95
Transfer/sec:      6.93MB

2、wrk->ClusterIp->pod(只有一个pod)

性能下降 10%

./wrk -H "Connection: Close" -t 4 -c 300 -d30  http://{ClusterIp}:80
Running 30s test @ http://{ClusterIp}:80
  4 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    29.38ms   14.66ms 249.18ms   90.07%
    Req/Sec     1.90k   351.30     3.31k    72.35%
  227505 requests in 30.05s, 183.34MB read
Requests/sec:   7571.81
Transfer/sec:      6.10MB

2、wrk->ClusterIp->pod(2个pod)

性能下降 30%。理论上，Requests/sec 应该接近 17206(8603 * 2)

./wrk -H "Connection: Close" -t 4 -c 300 -d30  http://{ClusterIp}:80
Running 30s test @ http://{ClusterIp}:80
  4 threads and 300 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    13.15ms    8.93ms 106.11ms   73.09%
    Req/Sec     3.00k     1.04k    6.32k    68.75%
  356342 requests in 30.10s, 287.16MB read
Requests/sec:  11837.60
Transfer/sec:      9.54MB

我的配置：在 /etc/sysctl.conf

net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_tw_reuse = 1

/etc/security/limits.conf

* soft noproc 102400
* hard noproc 102400