1

I'm trying to use Pardiso 6 sparse solver library in Python. The problem is that I can't seem to load the Pardiso shared object (SO). Here's the error that I get when calling

import ctypes
pardiso = ctypes.CDLL(pardiso_so_address)

Traceback (most recent call last):
  File "test.py", line 27, in <module>
    pardiso = ctypes.CDLL(lib720)
  File "/home/amin/anaconda3/envs/idp/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ./libpardiso600-GNU720-X86-64.so: undefined symbol: sgetrf_

I'd really appreciate it if someone could shed some light on this.

PS. I already contacted Pardiso developers and they told me that I need to link against optimized BLAS, but I already have MKL installed via conda.

Update 1: I installed mkl via conda, but it didn't help. Strangely, I added import scipy to the header and the error went away. The same thing happens if I add import mkl. So, for some reason, unless scipy or mkl are manually imported, the .so doesn't know that a lapack installation exists. Anyway, now another error is thrown, which I think might be related the libgfortran library. Here's the error

Traceback (most recent call last):
  File "test.py", line 34, in <module>
    pardiso = ctypes.CDLL(lib720)
  File "/home/amin/anaconda3/envs/test/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ./libpardiso600-GNU720-X86-64.so: undefined symbol: _gfortran_st_close

I double-checked to see if libgfortran is installed, and indeed it is:

(test) PyPardisoProject$ ldconfig -p | grep libgfortran
    libgfortran.so.5 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgfortran.so.5
    libgfortran.so.4 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgfortran.so.4

I think something similar might be at play, i.e. the library is there but it needs to be triggered (similar to what import scipy seems to have done for liblapack, but I have no idea how I can trigger it.

Note: I found an example in C on Pardiso website and tested the .so file against it via

$ gcc pardiso_sym.c -o pardiso_sym -L . -lpardiso600-GNU720-X86-64 -llapack -fopenmp -lgfortran
$ OMP_NUM_THREADS=1 ./pardiso_sym

and it worked with no problem (with the existing libraries on my machine). So, the .so works, it's just that I don't know how to inform it of its dependencies in Python.

Update 2: Here's the output of ldd pardiso_sym:

Scripts$ ldd pardiso_sym
    linux-vdso.so.1 (0x00007ffe7e982000)
    libpardiso600-GNU720-X86-64.so (0x00007f326802d000)
    liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f3267976000)
    libgfortran.so.4 => /lib/x86_64-linux-gnu/libgfortran.so.4 (0x00007f3267795000)
    libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f326775b000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3267568000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3267545000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f32673f6000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f32685df000)
    libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007f3267389000)
    libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f32670e9000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f32670cf000)
    libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f3267083000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f326707d000)

So, I added the common path, i.e. /lib/x86_64-linux-gnu and /lib64 to PATH and ran the Python script again via:

PATH=$PATH:/lib/x86_64-linux-gnu:/lib64 python padiso_script.py

but the same error is thrown. I also tried adding to LD_LIBRARY_PATH as well, but didn't work either.

4

3 回答 3

1

PATH诀窍是，您需要在加载 Pardiso 库之前在 Python 脚本中显式加载依赖项，即lapack,blas和，而不是将依赖项的位置添加到 system 。gfortran此外，您必须将可选mode=ctypes.RLTD_GLOBAL参数显式传递给ctypes.CDLL方法，以使依赖项全局可访问，因此 Pardiso 可以访问它们。

import ctypes
import ctypes.util

shared_libs = ["lapack", "blas", "omp", "gfortran"]
for lib in shared_libs:
    # Fetch the proper name of the dependency
    libname = ctypes.util.find_library(lib)
    # Load the dependency and make it globally accessible
    ctypes.CDLL(libname, mode=ctypes.RTLD_GLOBAL)
# Finally, load the Pardiso library
pardiso = ctypes.CDLL(pardiso_so_address)

以我的经验，如果您在已安装的conda环境中mkl，则只需将其gfortran列为依赖项，其余的将自动加载并可以访问，在这种情况下 set shared_libs = ["gfortran"].

于 2020-05-03T13:33:52.300 回答
1

Pardiso 6 sparse solver至少取决于 Lapack 函数sgetrf，该函数使用带有行交换的部分旋转来计算一般 M×N 矩阵 A 的 LU 分解。

从我们阅读的内容来看，libpardiso600-GNU720-X86-64.so它与共享的 Lapack 库动态链接。您需要提供一个包含一个实现的 PATH。

在启动 Python 之前，我建议您使用LD_LIBRARY_PATH并包含您正在使用的 BLAS/Lapack 库的路径。它可以是 netlib 实现、ATLAS 实现或 MKL 实现。

LD_LIRARY_PATH=$LD_LIRARY_PATH:/my_path_to_lapack \
python -c"import ctypes; pardiso = ctypes.CDLL(pardiso_so_address)"

如果使用conda，可以使用命令安装

conda install -c anaconda mkl

在这种情况下，安装可能会直接解决问题。

于 2020-05-01T06:02:19.363 回答
0

Pardiso 6 和 Intel MKL Pardiso 不兼容，因为它们具有不同的 API。您可以尝试从系统路径中删除 MKL，添加 OpenBLAS，然后再次尝试链接您的示例。

于 2020-05-03T01:27:55.140 回答