1

这是我在执行时收到的错误消息import theano

ImportError:动态模块未定义模块导出功能(PyInit_m3d1cf20adb1014f04986e6a344a55bde)

我在 Windows 10 上使用 Python 3.5.2。

我也在 GitHub 上向 Theano 的作者询问过这个问题。我在这里问希望有人能够帮助我,即使他/她不知道 Theano 的内部结构。我怀疑这与 Python 2 和 3 有更多关系。

这是完整的错误:

ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gpuarray\__init__.py", line 164, in <module>
    use(config.device)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gpuarray\__init__.py", line 151, in use
    init_dev(device)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gpuarray\__init__.py", line 66, in init_dev
    avail = dnn.dnn_available(name)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gpuarray\dnn.py", line 174, in dnn_available
    if not dnn_present():
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gpuarray\dnn.py", line 157, in dnn_present
    dnn_present.avail, dnn_present.msg = _dnn_check_version()
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gpuarray\dnn.py", line 130, in _dnn_check_version
    v = version()
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gpuarray\dnn.py", line 316, in version
    profile=False)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\compile\function.py", line 326, in function
    output_keys=output_keys)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\compile\pfunc.py", line 486, in pfunc
    output_keys=output_keys)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\compile\function_module.py", line 1795, in orig_function
    defaults)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\compile\function_module.py", line 1661, in create
    input_storage=input_storage_lists, storage_map=storage_map)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\link.py", line 699, in make_thunk
    storage_map=storage_map)[:3]
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\vm.py", line 1063, in make_all
    impl=impl))
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\op.py", line 924, in make_thunk
    no_recycling)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\op.py", line 828, in make_c_thunk
    output_storage=node_output_storage)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\cc.py", line 1190, in make_thunk
    keep_lock=keep_lock)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\cc.py", line 1131, in __compile__
    keep_lock=keep_lock)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\cc.py", line 1589, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\cmodule.py", line 1155, in module_from_key
    module = lnk.compile_cmodule(location)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\cc.py", line 1492, in compile_cmodule
    preargs=preargs)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\cmodule.py", line 2318, in compile_str
    return dlimport(lib_filename)
File "C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof\cmodule.py", line 302, in dlimport
    rval = __import__(module_name, {}, {}, [module_name])
ImportError: dynamic module does not define module export function (PyInit_m3d1cf20adb1014f04986e6a344a55bde)

据我所知,Theano 编译了一个 cpp 文件。以下是编译后的文件:

Directory of [..]\tmp04_uyjkc

04/02/2017  16:41    <DIR>          .
04/02/2017  16:41    <DIR>          ..
04/02/2017  16:08                 0 delete.me
04/02/2017  16:08           108.834 m3d1cf20adb1014f04986e6a344a55bde.pyd
04/02/2017  16:08             5.502 mod.cpp
04/02/2017  16:41                 0 ok.txt
04/02/2017  16:08                 0 __init__.py
04/02/2017  16:08    <DIR>          __pycache__
            5 File(s)        114.336 bytes

Directory of [..]\tmp04_uyjkc\__pycache__

04/02/2017  16:08    <DIR>          .
04/02/2017  16:08    <DIR>          ..
04/02/2017  16:08               239 __init__.cpython-35.pyc
            1 File(s)            239 bytes

这是 mod.cpp 的内容:

#include <Python.h>
#include <iostream>
#include "theano_mod_helper.h"
#include "cudnn.h"
//////////////////////
////  Support Code
//////////////////////

#if PY_MAJOR_VERSION >= 3
#define PyInt_FromLong PyLong_FromLong
#endif


    namespace {
    struct __struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde {
        PyObject* __ERROR;

        PyObject* storage_V1;


        __struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde() {
            // This is only somewhat safe because we:
            //  1) Are not a virtual class
            //  2) Do not use any virtual classes in the members
            //  3) Deal with mostly POD and pointers

            // If this changes, we would have to revise this, but for
            // now I am tired of chasing segfaults because
            // initialization code had an error and some pointer has
            // a junk value.
            memset(this, 0, sizeof(*this));
        }
        ~__struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde(void) {
            cleanup();
        }

        int init(PyObject* __ERROR, PyObject* storage_V1) {
            Py_XINCREF(storage_V1);
            this->storage_V1 = storage_V1;



            this->__ERROR = __ERROR;
            return 0;
        }
        void cleanup(void) {
            __label_1:

double __DUMMY_1;
__label_4:

double __DUMMY_4;

            Py_XDECREF(this->storage_V1);
        }
        int run(void) {
            int __failure = 0;

    PyObject* py_V1;

        PyObject* V1;

{

    py_V1 = Py_None;
    {Py_XINCREF(py_V1);}

        V1 = NULL;

{
// Op class DnnVersion

        V1 = PyTuple_Pack(2, PyInt_FromLong(CUDNN_VERSION), PyInt_FromLong(cudnnGetVersion()));
        __label_3:

double __DUMMY_3;

}
__label_2:

    if (!__failure) {

        assert(py_V1->ob_refcnt > 1);
        Py_DECREF(py_V1);
        py_V1 = V1 ? V1 : Py_None;
        Py_INCREF(py_V1);

      PyObject* old = PyList_GET_ITEM(storage_V1, 0);
      {Py_XINCREF(py_V1);}
      PyList_SET_ITEM(storage_V1, 0, py_V1);
      {Py_XDECREF(old);}
    }

        Py_XDECREF(V1);

    {Py_XDECREF(py_V1);}

double __DUMMY_2;

}


        if (__failure) {
            // When there is a failure, this code puts the exception
            // in __ERROR.
            PyObject* err_type = NULL;
            PyObject* err_msg = NULL;
            PyObject* err_traceback = NULL;
            PyErr_Fetch(&err_type, &err_msg, &err_traceback);
            if (!err_type) {err_type = Py_None;Py_INCREF(Py_None);}
            if (!err_msg) {err_msg = Py_None; Py_INCREF(Py_None);}
            if (!err_traceback) {err_traceback = Py_None; Py_INCREF(Py_None);}
            PyObject* old_err_type = PyList_GET_ITEM(__ERROR, 0);
            PyObject* old_err_msg = PyList_GET_ITEM(__ERROR, 1);
            PyObject* old_err_traceback = PyList_GET_ITEM(__ERROR, 2);
            PyList_SET_ITEM(__ERROR, 0, err_type);
            PyList_SET_ITEM(__ERROR, 1, err_msg);
            PyList_SET_ITEM(__ERROR, 2, err_traceback);
            {Py_XDECREF(old_err_type);}
            {Py_XDECREF(old_err_msg);}
            {Py_XDECREF(old_err_traceback);}
        }
        // The failure code is returned to index what code block failed.
        return __failure;

        }
    };
    }


        static int __struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde_executor(__struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde *self) {
            return self->run();
        }

        static void __struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde_destructor(PyObject *capsule) {
            __struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde *self = (__struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde *)PyCapsule_GetContext(capsule);
            delete self;
        }

//////////////////////
////  Functions
//////////////////////
static PyObject * instantiate(PyObject * self, PyObject *argtuple) {
  assert(PyTuple_Check(argtuple));
  if (2 != PyTuple_Size(argtuple)){ 
     PyErr_Format(PyExc_TypeError, "Wrong number of arguments, expected 2, got %i", (int)PyTuple_Size(argtuple));
     return NULL;
  }
  __struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde* struct_ptr = new __struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde();
  if (struct_ptr->init( PyTuple_GET_ITEM(argtuple, 0),PyTuple_GET_ITEM(argtuple, 1) ) != 0) {
    delete struct_ptr;
    return NULL;
  }
    PyObject* thunk = PyCapsule_New((void*)(&__struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde_executor), NULL, __struct_compiled_op_m3d1cf20adb1014f04986e6a344a55bde_destructor);
    if (thunk != NULL && PyCapsule_SetContext(thunk, struct_ptr) != 0) {
        PyErr_Clear();
        Py_DECREF(thunk);
        thunk = NULL;
    }

  return thunk; }

//////////////////////
////  Module init
//////////////////////
static PyMethodDef MyMethods[] = {
    {"instantiate", instantiate, METH_VARARGS, "undocumented"} ,
    {NULL, NULL, 0, NULL}
};
static struct PyModuleDef moduledef = {
      PyModuleDef_HEAD_INIT,
      "m3d1cf20adb1014f04986e6a344a55bde",
      NULL,
      -1,
      MyMethods,
};

PyMODINIT_FUNC PyInit_m3d1cf20adb1014f04986e6a344a55bde(void) {
    PyObject *m = PyModule_Create(&moduledef);
    return m;
}

功能PyInit_m3d1cf20adb1014f04986e6a344a55bde似乎存在,那么问题出在哪里?

编辑 1

我将文件复制到 d:\test 并设法使用 Theano 使用的相同命令重新编译 .pyd 文件(感谢 procmon):

import os

add_dir = r'C:\Users\Kiuhnm\Anaconda3\Library\mingw-w64\bin'
os.environ['PATH'] += os.pathsep + add_dir

cmd = r'""C:\Users\Kiuhnm\Anaconda3\Library\mingw-w64\bin\g++.exe"  -shared -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -Wl,-rpath, -march=haswell -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mbmi2 -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-clwb -mno-pcommit -mno-mwaitx --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=8192 -mtune=haswell -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -DMS_WIN64 -I"C:\Users\Kiuhnm\Anaconda3\lib\site-packages\numpy\core\include" -I"C:\Users\Kiuhnm\Anaconda3\include" -I"C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof" -L"C:\Users\Kiuhnm\Anaconda3\libs" -L"C:\Users\Kiuhnm\Anaconda3" -o d:\test\m3d1cf20adb1014f04986e6a344a55bde.pyd d:\test\mod.cpp -lcudnn -lpython35"'
os.system(cmd)

我读到 pyd 文件只是 DLL(至少在 Windows 上)。奇怪的是,编译后的 pyd 文件似乎没有导出任何函数。也许我不明白 Python 如何找到导出的函数:

D:\test>dumpbin /EXPORTS m3d1cf20adb1014f04986e6a344a55bde.pyd
Microsoft (R) COFF/PE Dumper Version 12.00.40629.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file m3d1cf20adb1014f04986e6a344a55bde.pyd

File Type: DLL

  Section contains the following exports for m3d1cf20adb1014f04986e6a344a55bde.pyd

    00000000 characteristics
    5896198F time date stamp Sat Feb 04 19:12:31 2017
        0.00 version
           1 ordinal base
           0 number of functions
           0 number of names

    ordinal hint RVA      name


  Summary

        1000 .CRT
        1000 .bss
        1000 .data
        2000 .debug_abbrev
        1000 .debug_aranges
        1000 .debug_frame
        9000 .debug_info
        2000 .debug_line
        3000 .debug_loc
        1000 .debug_ranges
        1000 .debug_str
        1000 .edata
        1000 .idata
        1000 .pdata
        1000 .rdata
        1000 .reloc
        2000 .text
        1000 .tls
        1000 .xdata
4

1 回答 1

1

我终于解决了这个难题!

我们再来看看g++命令:

"C:\Users\Kiuhnm\Anaconda3\Library\mingw-w64\bin\g++.exe"  -shared -g -O3 -fno-math-errno
 -Wno-unused-label -Wno-unused-variable -Wno-write-strings -Wl,-rpath, -march=haswell -mmmx
 -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul
 -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mbmi2 -mno-tbm -mavx -mavx2 -msse4.2
 -msse4.1 -mlzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mno-rdseed -mno-prfchw -mno-adx
 -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1
 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma
 -mno-avx512vbmi -mno-clwb -mno-pcommit -mno-mwaitx --param l1-cache-size=32 --param
 l1-cache-line-size=64 --param l2-cache-size=8192 -mtune=haswell
 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -DMS_WIN64
 -I"C:\Users\Kiuhnm\Anaconda3\lib\site-packages\numpy\core\include"
 -I"C:\Users\Kiuhnm\Anaconda3\include"
 -I"C:\Users\Kiuhnm\Anaconda3\lib\site-packages\theano\theano\gof"
 -L"C:\Users\Kiuhnm\Anaconda3\libs" -L"C:\Users\Kiuhnm\Anaconda3" -o
 d:\test\m3d1cf20adb1014f04986e6a344a55bde.pyd d:\test\mod.cpp -lcudnn -lpython35

-Wl,-rpath,在第二排看到了吗?好吧,事实证明,至少在 Windows 中,它吞噬了它后面的所有东西!

-Wl,a,b,c,d用于将“abc d”传递给链接器,但在 Windows 中,尾随逗号后跟空格是有问题的。也许这与 Windows 允许文件名中有空格的事实有关。

删除有问题的部分后,输出文件变大了 40 KB,表明它现在包含 mod.cpp 的目标代码。导入模块终于成功了,但现在我必须深入研究 Theano 代码以查看问题所在,因为该命令是动态生成的。

编辑:这并不难。所要做的就是添加

[dnn]
library_path=d:\whatever

.theanorc. 我使用了一个虚拟目录,因为 cuDNN 的文件已经可以在我的系统上访问(我必须已经更新了PATH)。

如果您没有安装 cuDNN,那么您永远不会遇到此问题。

现在-Wl,-rpath,应该是-Wl,-rpath,d:\whatever,但我没有检查。

为了完整起见,相关代码[...]Lib\site-packages\Theano\theano\gpuarray\dnn.py位于第 267 行:

def c_compile_args(self):
    return ['-Wl,-rpath,' + config.dnn.library_path]

config.dnn.library_path现在是“d:\whatever”,一切都很好。

于 2017-02-05T00:00:26.177 回答