0

我正在使用 Spark NLP 训练分类模型。我已按照本教程进行操作，以下大部分代码都来自那里。

这是我的训练脚本：

from pyspark.sql import SparkSession
from pyspark.ml import Pipeline

from sparknlp.annotator import *
from sparknlp.common import *
from sparknlp.base import *

import pandas as pd

import sparknlp
spark = sparknlp.start(gpu=True)

# has only 2 columns: category and description
DF = spark.read \
      .option("header", True) \
      .csv("data.csv")

(trainingData, testData) = DF.randomSplit([0.7, 0.3], seed = 100)

document_assembler = DocumentAssembler() \
      .setInputCol("description") \
      .setOutputCol("document")

sent_embeddings = BertSentenceEmbeddings.pretrained("sent_biobert_clinical_base_cased", "en") \
      .setInputCols("document") \
      .setOutputCol("sentence_embeddings")

classsifierdl = ClassifierDLApproach()\
      .setInputCols("sentence_embeddings")\
      .setOutputCol("class")\
      .setLabelColumn("category")\
      .setMaxEpochs(5)\
      .setLr(0.5)\
      .setDropout(0.5)\
      .setEnableOutputLogs(True)

clf_pipeline = Pipeline(
    stages=[document_assembler, 
            sent_embeddings,
            classsifierdl])

clf_pipelineModel = clf_pipeline.fit(trainingData)

from sklearn.metrics import classification_report, accuracy_score
df = clf_pipelineModel.transform(testData).select('category','description',"class.result").toPandas()
df['result'] = df['result'].apply(lambda x: x[0])
print(classification_report(df.category, df.result))
print(accuracy_score(df.category, df.result))

然而，分类器总是预测同一个类。我哪里错了？谢谢。

4

0 回答 0