python - 如何在 python 中确定合理的线程数？

Question

我正在编写一个简单的脚本，它应该进行大量检查。每个检查都是独立的，所以我决定把它放到多个线程中。但是我不知道设置脚本的机器有多快。我已经找到了非常好的工具来检查目标机器的基本参数，但我想知道是否有任何方法可以确定最大合理线程数量是多少（我的意思是新线程开始减慢进程而不是加快进程的那一刻)?

Answer 1

You can find out the number of cores your target machine has with

import multiprocessing

multiprocessing.cpu_count()

If you choose multiprocessing to manage your tasks then you can set the Pool size, or number of worker threads dependent on system load and .cpu_count().

As to what is a good number for you to choose for your program, you will have to decide for yourself :-)

Answer 2

在 python 中为了提高速度而使用线程并不是一个非常好的主意，尤其是对于 cpu 绑定操作。GIL 看到了来自多个 CPU 的任何潜在性能改进（其中 # 是线程速度增加的理论限制 - 尽管实际上是 YMMV）。

对于真正独立的“检查”，您最好查看多处理。

Answer 3

You might have your answer in your last sentence. You could measure the execution time, as it changes with thread additions. And adaptively add or remove (and requeue the removed thread's job) threads to preserve an execution time you like. If you want to get sophisticated check out control theory.