c++ - 有没有办法在运行时分析 MPI 应用程序？- 在运行时使用 MPI 调用所花费的时间

Question

我想知道是否有一种方法可以测量运行时 MPI 调用所花费的时间。因此，我可以使用它来计算新的负载平衡。

我知道如何使用 OpenMPI 或 Intel 的一些工具来分析和跟踪程序，但这些都是在运行后使用的。此外，我尝试过 FPMPI，但没有成功，因为无法构建最后一个版本。

“手动”测量在我的应用程序中没有任何意义，因为它太大了：/

Answer 1

First of all, do you really need to profile low-level communication such as MPI? Can't you simply time your high-level routines instead?

Anyway, it is pretty easy to write your own MPI profiler. Practically all MPI libraries (Open MPI included) export their functions (e.g., MPI_Send) as weak aliases of the same function symbols with prefix P (e.g., PMPI_Send). All you need to do is define your own functions with the same prototypes as the ones in the MPI library. Inside, update your call counters, start the timers, then call the original MPI function with a P prefix, and upon return stop the timers:

extern "C"
int MPI_Send(void *buf, int count, MPI_Datatype dtype, int dest, int tag, MPI_Comm comm) {
   // Update call counters and start the timer
   calls["MPI_Send"]++;
   timers["MPI_Send"].start();

   // Call the original MPI function
   int result = PMPI_Send(buf, count, dtype, dest, tag, comm);

   // Stop the timer
   timers["MPI_Send"].stop();

   return result;
}

The extern "C" part is important, otherwise you won't override the correct weak symbol if you write in C++.

This ability to override symbols from the MPI library is standardised - see #14.2 Profiling Interface in the current version of the standard.