0

尝试将 NVBLAS 与英特尔 Fortran 编译器一起使用时，我似乎遗漏了一些东西。

当我看到运行时 NVBLAS 初始化的反馈时，我似乎正在正确链接和使用 nvblas.conf。但是，NVBLAS 似乎没有拦截对 DGEMM 的调用，因为只执行了 CPU 实现。尽管使用了：

NVBLAS_CPU_RATIO_CGEMM 0.0

在 nvblas.conf 中（或完全删除它）。

如果我通过删除禁用对 CPU BLAS 实现的访问：

NVBLAS_CPU_BLAS_LIB  /ccc/home/wilkinson/EMPIRE-2064/src/dynamiclibs/libmkl_rt.so

正如我所料，程序在运行时崩溃。

我目前使用的编译器选项如下所示，我也尝试过手动链接 MKL，但结果相同。

# Compiler options
FFLAGS=-O3 -axAVX,SSE4.2 -msse3 -align array32byte -fpe1 -fno-alias -openmp -mkl=parallel -heap-arrays 32

 # Linker options
LDFLAGS= -L/ccc/home/wilkinson/EMPIRE-2064/src/dynamiclibs -lnvblas

# List of libraries used
LIBS= -L/ccc/home/wilkinson/EMPIRE-2064/src/dynamiclibs -lnvblas

调用 DGEMM 的示例如下：

call dgemm('N','T',nCols2,nCols1,nOcc(s),2.0d0/dble(nSpins),C2,nRowsP,C(:,:,s),nRowsP,0.0d0,P(i21,i11,s),nOrbsP)

不幸的是，我目前仅限于使用 Intel 编译器，但该限制将很快解除（此时我将使用 CUDA Fortran 来优化数据移动）。

4

1 回答 1

1

我不确定这里发生了什么。如果我举一个非常简单的 DGEMM 示例（直接摘自 MKL fortran 指南）：

      PROGRAM   MAIN

      IMPLICIT NONE

      DOUBLE PRECISION ALPHA, BETA
      INTEGER          M, K, N, I, J
      PARAMETER        (M=8000, K=8000, N=8000)
      DOUBLE PRECISION A(M,K), B(K,N), C(M,N)


      PRINT *, "Initializing data for matrix multiplication C=A*B for "
      PRINT 10, " matrix A(",M," x",K, ") and matrix B(", K," x", N, ")"
10    FORMAT(a,I5,a,I5,a,I5,a,I5,a)
      PRINT *, ""
      ALPHA = 1.0 
      BETA = 0.0

      PRINT *, "Intializing matrix data"
      PRINT *, ""
      DO I = 1, M
        DO J = 1, K
          A(I,J) = (I-1) * K + J
        END DO
      END DO

      DO I = 1, K
        DO J = 1, N
          B(I,J) = -((I-1) * N + J)
        END DO
      END DO

      DO I = 1, M
        DO J = 1, N
          C(I,J) = 0.0
        END DO
      END DO

      PRINT *, "Computing matrix product using DGEMM subroutine"
      CALL DGEMM('N','N',M,N,K,ALPHA,A,M,B,K,BETA,C,M)
      PRINT *, "Computations completed."
      PRINT *, ""

      PRINT *, "Top left corner of matrix A:"
      PRINT 20, ((A(I,J), J = 1,MIN(K,6)), I = 1,MIN(M,6))
      PRINT *, ""

      PRINT *, "Top left corner of matrix B:"
      PRINT 20, ((B(I,J),J = 1,MIN(N,6)), I = 1,MIN(K,6))
      PRINT *, ""

 20   FORMAT(6(F12.0,1x))

      PRINT *, "Top left corner of matrix C:"
      PRINT 30, ((C(I,J), J = 1,MIN(N,6)), I = 1,MIN(M,6))
      PRINT *, ""

 30   FORMAT(6(ES12.4,1x))

      PRINT *, "Example completed."
      STOP 

      END

如果我使用英特尔编译器 (12.1) 构建代码并在 nvprof 下运行它（注意我目前无法访问 MKL，所以我使用的是使用 ifort 构建的 OpenBLAS）：

$ ifort -o nvblas_test nvblas_test.f -L/opt/cuda-7.5/lib64 -lnvblas
$ echo -e "NVBLAS_CPU_BLAS_LIB  /opt/openblas/lib/libopenblas.so\nNVBLAS_AUTOPIN_MEM_ENABLED\n" > nvblas.conf

$ nvprof --print-gpu-summary ./nvblas_test
==23978== NVPROF is profiling process 23978, command: ./nvblas_test
[NVBLAS] Config parsed
 Initializing data for matrix multiplication C=A*B for 
 matrix A( 8000 x 8000) and matrix B( 8000 x 8000)

 Intializing matrix data

 Computing matrix product using DGEMM subroutine
 Computations completed.

 Top left corner of matrix A:
          1.           2.           3.           4.           5.           6.
       8001.        8002.        8003.        8004.        8005.        8006.
      16001.       16002.       16003.       16004.       16005.       16006.
      24001.       24002.       24003.       24004.       24005.       24006.
      32001.       32002.       32003.       32004.       32005.       32006.
      40001.       40002.       40003.       40004.       40005.       40006.

 Top left corner of matrix B:
         -1.          -2.          -3.          -4.          -5.          -6.
      -8001.       -8002.       -8003.       -8004.       -8005.       -8006.
     -16001.      -16002.      -16003.      -16004.      -16005.      -16006.
     -24001.      -24002.      -24003.      -24004.      -24005.      -24006.
     -32001.      -32002.      -32003.      -32004.      -32005.      -32006.
     -40001.      -40002.      -40003.      -40004.      -40005.      -40006.

 Top left corner of matrix C:
 -1.3653E+15  -1.3653E+15  -1.3653E+15  -1.3653E+15  -1.3653E+15  -1.3653E+15
 -3.4131E+15  -3.4131E+15  -3.4131E+15  -3.4131E+15  -3.4131E+15  -3.4131E+15
 -5.4608E+15  -5.4608E+15  -5.4608E+15  -5.4608E+15  -5.4608E+15  -5.4608E+15
 -7.5086E+15  -7.5086E+15  -7.5086E+15  -7.5086E+15  -7.5086E+15  -7.5086E+15
 -9.5563E+15  -9.5563E+15  -9.5563E+15  -9.5563E+15  -9.5563E+15  -9.5563E+15
 -1.1604E+16  -1.1604E+16  -1.1604E+16  -1.1604E+16  -1.1604E+16  -1.1604E+16

 Example completed.
==23978== Profiling application: ./nvblas_test
==23978== Profiling result:
Time(%)      Time     Calls       Avg       Min       Max  Name
 92.15%  8.56855s       512  16.736ms  9.6488ms  21.520ms  void magma_lds128_dgemm_kernel<bool=0, bool=0, int=5, int=5, int=3, int=3, int=3>(int, int, int, double const *, int, double const *, int, double*, int, int, int, double const *, double const *, double, double, int)
  7.38%  685.77ms      1025  669.04us     896ns  820.55us  [CUDA memcpy HtoD]
  0.47%  44.017ms        64  687.77us  504.56us  763.05us  [CUDA memcpy DtoH]

我得到了我所期望的——将 DGEMM 调用卸载到 GPU。当我这样做时：

$ echo "NVBLAS_GPU_DISABLED_DGEMM" >> nvblas.conf 
$ nvprof --print-gpu-summary ./nvblas_test
==23991== NVPROF is profiling process 23991, command: ./nvblas_test
[NVBLAS] Config parsed
 Initializing data for matrix multiplication C=A*B for 
 matrix A( 8000 x 8000) and matrix B( 8000 x 8000)

 Intializing matrix data

 Computing matrix product using DGEMM subroutine
 Computations completed.

 Top left corner of matrix A:
          1.           2.           3.           4.           5.           6.
       8001.        8002.        8003.        8004.        8005.        8006.
      16001.       16002.       16003.       16004.       16005.       16006.
      24001.       24002.       24003.       24004.       24005.       24006.
      32001.       32002.       32003.       32004.       32005.       32006.
      40001.       40002.       40003.       40004.       40005.       40006.

 Top left corner of matrix B:
         -1.          -2.          -3.          -4.          -5.          -6.
      -8001.       -8002.       -8003.       -8004.       -8005.       -8006.
     -16001.      -16002.      -16003.      -16004.      -16005.      -16006.
     -24001.      -24002.      -24003.      -24004.      -24005.      -24006.
     -32001.      -32002.      -32003.      -32004.      -32005.      -32006.
     -40001.      -40002.      -40003.      -40004.      -40005.      -40006.

 Top left corner of matrix C:
 -1.3653E+15  -1.3653E+15  -1.3653E+15  -1.3653E+15  -1.3653E+15  -1.3653E+15
 -3.4131E+15  -3.4131E+15  -3.4131E+15  -3.4131E+15  -3.4131E+15  -3.4131E+15
 -5.4608E+15  -5.4608E+15  -5.4608E+15  -5.4608E+15  -5.4608E+15  -5.4608E+15
 -7.5086E+15  -7.5086E+15  -7.5086E+15  -7.5086E+15  -7.5086E+15  -7.5086E+15
 -9.5563E+15  -9.5563E+15  -9.5563E+15  -9.5563E+15  -9.5563E+15  -9.5563E+15
 -1.1604E+16  -1.1604E+16  -1.1604E+16  -1.1604E+16  -1.1604E+16  -1.1604E+16

 Example completed.
==23991== Profiling application: ./nvblas_test
==23991== Profiling result:
Time(%)      Time     Calls       Avg       Min       Max  Name
100.00%     768ns         1     768ns     768ns     768ns  [CUDA memcpy HtoD]

我没有卸载到 GPU。如果你不能重现这个，那么问题要么出在你的编译器版本上（你还没有说你正在使用哪个版本），如果可以的话，那么也许你正在使用的有点花哨的构建选项正在与 NVBLAS 交互意想不到的方式

于 2016-01-21T09:19:13.107 回答