2

首先为我犯的任何错误道歉。我是 StackOverflow 的新手，我还在习惯它。

背景：我尝试通过多处理训练多个 keras 模型。这些 Keras 模型存储在三个列表中（目前每个列表有 50 个模型）。我使用的函数有三个模型作为输入（每个列表一个）以及一些训练数据和一个 ID。输出应该是评估分数和给定的 ID。

我的第一次尝试是从标准 python 库中导入多处理。

from multiprocessing import Pool

事实证明，这个多处理应用程序不起作用，因为 pickle 作为序列化工具。所以我下载了 Pathos 和 Dill 来解决这个问题。我的代码是这样的：

    from pathos.multiprocessing import ProcessingPool as Pool
    p = Pool()
    from pathos.helpers import freeze_support
    import dill

    if __name__ == "__main__":
        freeze_support()

        result = p.amap(training.train_critic, models_actor_t, models_critic, models_critic_t, batch_states_list, batch_actions_list, batch_rewards_list, batch_new_states_list , batch_terminals_list, sub_epochs_list, id_list)

        print(result)

如果我运行这段代码，一切正常，我得到这个输出。

<multiprocess.pool.MapResult object at 0x000001D42CEC09E8>

如果我添加方法 result.get() 麻烦就开始了。

   if __name__ == "__main__":
        freeze_support()

        result = p.amap(training.train_critic, models_actor_t, models_critic, models_critic_t, batch_states_list, batch_actions_list, batch_rewards_list, batch_new_states_list , batch_terminals_list, sub_epochs_list, id_list)

        print(result)
        print(result.get())

输出是这样的：

Traceback (most recent call last):


File "C:/Users/Lennart/PycharmProjects/demonstrator_RLAlgorithm/GeneticAlgorithm/GAoperations.py", line 192, in <module>
    print(result.get())
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\multiprocess\pool.py", line 608, in get
    raise self._value
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\multiprocess\pool.py", line 385, in _handle_tasks
    put(task)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\multiprocess\connection.py", line 209, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\multiprocess\reduction.py", line 53, in dumps
    cls(buf, protocol).dump(obj)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 408, in dump
    self.save(obj)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 744, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 729, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 729, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 729, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 744, in save_tuple
    save(element)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 627, in save_reduce
    save(state)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\dill\dill.py", line 871, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 814, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 840, in _batch_setitems
    save(v)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 774, in save_list
    self._batch_appends(obj)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 801, in _batch_appends
    save(tmp[0])
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 520, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 627, in save_reduce
    save(state)
  File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\pickle.py", line 475, in save
    f(self, obj) # Call unbound method with explicit self

从这里开始的最后一部分重复了自己：

 File "C:\Users\Lennart\AppData\Local\Continuum\anaconda3\envs\Cense5\lib\site-packages\dill\dill.py", line 871, in save_module_dict
    StockPickler.save_dict(pickler, obj)

所以我的问题是：

  1. 有什么根本上的错误吗？
  2. Keras 模型存储在一个列表中。所以通过将它们插入到函数中，模型的权重仍然是持久的，对吧？

再次，如果我做了任何愚蠢的新手事情，我很抱歉。

谢谢，

伦纳特

4

0 回答 0