我有这样的代码:
def getModel():
model = Sequential()
model.Add(...)
.....
model = tf.contrib.tpu.keras_to_tpu_model(model,
strategy=tf.contrib.tpu.TPUDistributionStrategy(
tf.contrib.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
))
model.compile(loss='mse',
optimizer=tf.train.AdamOptimizer(learning_rate=1e-3, ))
return model
tpu_model = getModel()
## Main loop
....
tpu_model.predict(states)
tpu_model.fit(...)
请注意,我将相同tpu_model
的方法用于批量预测和训练。
tpu_model.predict()
似乎工作正常,但是当它运行时tpu_model.fit(...)
,它会引发以下错误:
WARNING:tensorflow:tpu_model (from tensorflow.contrib.tpu.python.tpu.keras_support) is experimental and may change or be removed at any time, and without warning.
INFO:tensorflow:New input shapes; (re-)compiling: mode=infer (# of cores 8), [TensorSpec(shape=(4, 7), dtype=tf.float32, name='dense_6_input_10')]
INFO:tensorflow:Overriding default placeholder.
INFO:tensorflow:Remapping placeholder for dense_6_input
INFO:tensorflow:Started compiling
INFO:tensorflow:Finished compiling. Time elapsed: 1.464857578277588 secs
INFO:tensorflow:Setting weights on TPU model.
...
...
...
RuntimeError Traceback (most recent call last)
--> 101 history = tpu_model.fit(states, target_f, epochs=1, verbose=0)
102 # Keeping track of loss
103 loss = history.history['loss'][0]
/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/tpu/python/tpu/keras_support.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
1505 validation_split, validation_data, shuffle,
1506 class_weight, sample_weight, initial_epoch,
-> 1507 steps_per_epoch, validation_steps, **kwargs)
1508 finally:
1509 self._numpy_to_infeed_manager_list = []
/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/tpu/python/tpu/keras_support.py in _pipeline_fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)
1578 steps_name='steps_per_epoch',
1579 steps=steps_per_epoch,
-> 1580 validation_split=validation_split)
1581
1582 # Prepare validation data
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, batch_size, check_steps, steps_name, steps, validation_split)
990 x, y, sample_weight = next_element
991 x, y, sample_weights = self._standardize_weights(x, y, sample_weight,
--> 992 class_weight, batch_size)
993 return x, y, sample_weights
994
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in _standardize_weights(self, x, y, sample_weight, class_weight, batch_size)
1036 if y is not None:
1037 if not self.optimizer:
-> 1038 raise RuntimeError('You must compile a model before '
1039 'training/testing. '
1040 'Use `model.compile(optimizer, loss)`.')
RuntimeError: You must compile a model before training/testing. Use `model.compile(optimizer, loss)`.
从日志中可以看出,在 TPU 上运行似乎有两种模式:
1. mode=infer
2.mode=training
看来两者不能同时进行。有没有办法解决?
我不能使用生成器,因为我正在做强化学习,其中批次基于动态添加到列表中的实时样本,从该列表中对批次进行采样、预测(并且更改某些值)和训练。