0

当我尝试使用 NER 训练 BilstmCRF 时出现错误

Dimensions must be equal, but are 100 and 24 for '{{node crf_loss/add_1}} = AddV2[T=DT_INT32](crf_loss/add, crf_loss/Cast)' with input shapes: [?,100], [?,100,24]

我知道问题是数据形状不相等，但实际上我不知道如何解决它。因为我调查了很多代码，这些代码看起来像我的，但我不明白为什么它们运行顺利。

我的数据形状

X_train.shape 是 (44950, 100)

np.array(y_train).shape 是 (44950, 100,24)

我如何处理文件：（Cr.https://github.com/SuphanutN/Thai-NER-BiLSTMCRF-WordCharEmbedding/blob/master/Thai_NER_WordCharacterEmbedding_Train.ipynb）

def prepare_sequence_word(input_text):
    idxs = list()
    for word in input_text:
        if word in thai2dict:
            idxs.append(thai2dict_to_ix[word])
        else:
            idxs.append(thai2dict_to_ix["unknown"]) #Use UNK tag for unknown word
    return idxs

def prepare_sequence_target(input_label):
    idxs = [ner_to_ix[w] for w in input_label]
    return idxs

input_sent = [[w,w,w,w,w],[w,w,w,w]]
train_target = [['B_PER', 'O', 'O', 'O', 'O'], ['B_ORG','O', 'O', 'O']]

X_train = [prepare_sequence_word(s) for s in input_sent]
X_train = pad_sequences(maxlen=max_len, sequences=X_train, value=thai2dict_to_ix["pad"], padding='post', truncating='post')
y_train = [prepare_sequence_target(s) for s in train_targets]
y_train = pad_sequences(maxlen=max_len, sequences=y_train, value=ner_to_ix["pad"], padding='post', truncating='post')
y_train = [to_categorical(i, num_classes=n_tag) for i in y_train]

X_train,X_test 看起来像

array([[ 6457,   705,  1826, ..., 27602, 27602, 27602],
   [ 6457, 21442,   415, ..., 27602, 27602, 27602],
   [   97,  5247,   197, ..., 27602, 27602, 27602],
   ...,
   [   19, 21442,   402, ..., 27602, 27602, 27602],
   [   97, 17666,     1, ..., 27602, 27602, 27602],
   [ 7306,  1003,  5164, ..., 27602, 27602, 27602]], dtype=int32)

np.array(y_train),np.array(y_test) 看起来像

array([[[0., 0., 0., ..., 0., 0., 0.],
    [0., 0., 0., ..., 0., 1., 0.],
    [0., 0., 0., ..., 0., 1., 0.],
    ...,
    [0., 0., 0., ..., 0., 0., 1.],
    [0., 0., 0., ..., 0., 0., 1.],
    [0., 0., 0., ..., 0., 0., 1.]],
  [[0., 0., 0., ..., 0., 1., 0.],
    [0., 0., 0., ..., 0., 1., 0.],
    [0., 0., 0., ..., 0., 1., 0.],
    ...,
    [0., 0., 0., ..., 0., 0., 1.],
    [0., 0., 0., ..., 0., 0., 1.],
    [0., 0., 0., ..., 0., 0., 1.]]], dtype=float32)

我的模型

EMBED_DIM = 300
BiRNN_UNITS = 48
class_labels_number = 21
EPOCHS = 2


model = Sequential()
model.add(Embedding(input_dim=n_thai2dict, output_dim=vector_dim, 
                weights=[thai2fit_weight], input_length=max_len, mask_zero=True)) 
model.add(Bidirectional(LSTM(BiRNN_UNITS // 2, return_sequences=True)))
model.add(CRF(class_labels_number))

model.compile('adam', loss=crf_loss, metrics=[crf_accuracy])
history = model.fit(X_train, np.array(y_train), epochs=EPOCHS, validation_data=(X_test, np.array(y_test)))
4

0 回答 0