2

当我在 InfiniBand 上运行 MPI 作业时，我得到以下磨损。我们使用扭矩管理器。

--------------------------------------------------------------------------
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory.  This can cause MPI jobs to
run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered.  You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module
parameters:

http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

Local host:              host1

Registerable memory:     65536 MiB

Total memory:            196598 MiB

Your MPI job will continue, but may be behave poorly and/or hang.

--------------------------------------------------------------------------

我已经阅读了警告消息上的链接，到目前为止我所做的是；

  1. 附加options mlx4_core log_num_mtt=20 log_mtts_per_seg=4在/etc/modprobe.d/mlx4_en.conf.
  2. 确保以下行写在/etc/security/limits.conf
    • * soft memlock unlimited
    • * hard memlock unlimited
  3. 附加session required pam_limits.so在/etc/pam.d/sshd
  4. 确保ulimit -c unlimited未注释/etc/init.d/pbs_mom

谁能帮我找出我错过了什么？

4

1 回答 1

3

您的mlx4_core参数只允许注册2^20 * 2^4 * 4 KiB = 64 GiB。每个节点有 192 GiB 的物理内存，并且建议至少有两倍的可注册内存，您应该设置log_num_mtt为 23，这会将限制增加到 512 GiB - 两个大于或等于两倍的最接近的幂RAM 量。请务必重新启动节点或卸载然后重新加载内核模块。

您还应该提交一个执行的简单 Torque 作业脚本，ulimit -l以验证锁定内存的限制并确保没有这样的限制。请注意，这ulimit -c unlimited并没有取消对锁定内存量的限制，而是对核心转储文件大小的限制。

于 2013-07-20T10:14:14.477 回答