1

on a linux system I have a directory with about 100.000 files (they contain some statistics data). I have to access them using wildcards and now I experience some performance issues. When accessing a specific file it's really fast:

time ls 19991
19991

real    0m0.004s
user    0m0.000s
sys     0m0.000s

When using wildcards it's (as you would expect) slower:

time ls 19991*
19991

real    0m0.043s
user    0m0.020s
sys     0m0.020s

But: When I try to access the files concurrently ALL access get slow:

 i=1; while [ $i -le 10 ]; do (time ls 19991* &) ; let i=i+1; done 
19991
19991

real    0m0.248s
user    0m0.010s
sys     0m0.020s
19991

real    0m0.279s
user    0m0.000s
sys     0m0.040s
19991
19991
19991

real    0m0.306s
user    0m0.050s
sys     0m0.000s

real    0m0.236s
user    0m0.010s
sys     0m0.030s

real    0m0.257s
user    0m0.010s
sys     0m0.040s

real    0m0.263s
user    0m0.020s
sys     0m0.020s
19991
19991

real    0m0.196s
user    0m0.030s
sys     0m0.010s

real    0m0.175s
user    0m0.020s
sys     0m0.020s
19991

real    0m0.095s
user    0m0.040s
sys     0m0.000s
19991

real    0m0.158s
user    0m0.020s
sys     0m0.040s

Even if the access is serialized by the kernel, I would expect the first "ls" to take about 40ms, the second 80ms, the third 120ms, ... . But now even the fastest "ls" takes 95ms and most of them about 200ms.

This occurs when using a local filesystem (ext3) and also when using a network-mounted directory (nfs). So I think it has nothing to do with a specific filesystem.

Any ideas what causes this slowdown or how to fix it?

4

1 回答 1

0

使用的总时间是多少？

如果我查看用户时间和系统时间，那么它们仍然很小。所以也许这就是你所看到的：

init job1
init job2
init job3
...
init job10
run half of job1
...
run half of job10
finish jobs1
...
finish jobs10

比较这些：

$ time ( i=1; while [ $i -le 100 ]; do time ls 19991* &  let i=i+1; done ;wait)
real    0m1.278s
user    0m2.532s
sys     0m2.236s

$ time ( i=1; while [ $i -le 100 ]; do time ls 19991* ; let i=i+1; done ;wait)
real    0m3.197s
user    0m1.669s
sys     0m1.535s

用户和系统大致相同。并行版本中较长的 user+sys-time 可以用分叉的开销来解释。较低的实时性可以通过拥有多个 CPU 来解释。

因此，您的时间会受到更多并行运行的作业的严重影响。

于 2019-01-12T01:59:03.533 回答