parallel-processing - Does CRYSTAL_WORKERS defaulting to 4 seem too low for a Crystal-lang app? Anyone else benchmark higher?

Question

I am very happy that Crystal-lang now has multithreading. Has anyone else done some multi-threaded benchmarking for Crystal-lang; if so, what CRYSTAL_WORKERS have you seen to be beneficial for what you benchmarked? (Yeah, I know, every benchmark is different, so mileage may vary.)

The default for CRYSTAL_WORKERS is 4, but it seems like it should be more, at least on higher-core systems. Could the benchmarks that lead to that 4 value possibly be tied to a side-effect of the number of cores used at the time or something else?

• https://crystal-lang.org/2019/09/06/parallelism-in-crystal.html

• https://github.com/crystal-lang/crystal/commit/f106d93f9bc10a0176d2da6f07f477ad34a9cda4#diff-390767d4904eb70109bd332570d8cc4R216

I did some Fibonacci comparisons on an 8-core (Intel) and a 16-core (AMD) CPU and got some large peaks and valley patterns ( https://github.com/drhuffman12/bench_vs/blob/master/threads/README.md ). (For simple calculations/transformations, the overhead of fibers/etc seem to not be worth it; but for more computationally-intensive code, like Fibinacci calcs, fibers w/ higher CRYSTAL_WORKERS values and same_thread: false seemed to potentially helpful.)

Answer 1

这个问题没有答案。工作人员的数量很大程度上取决于运行时系统上应用程序的类型和资源的可用性/管理。

当然，当您的系统有超过 4 个核心时，您会需要更多的核心，以便使用更多可用的计算能力。但是您不一定要使所有内核饱和，具体取决于系统上正在运行的其他内容。尤其是受 IO 限制的应用程序不一定会从拥有太多工作线程中受益。所以设置CRYSTAL_WORKERS=$(nprocs)可能不是一个好主意。

CRYSTAL_WORKERS=4只是一个默认值，因为需要有一个。这是一个非常常见的默认值，因为 a) 它应该比单线程执行提供合理的改进 b) 现代 CPU 通常至少有 4 个内核。

每当您将应用程序部署到运行时系统时，您都需要考虑适合该特定环境的工作人员编号并相应地进行配置。使用不同的默认值不会有任何帮助。