4

我想使用 SparkNLPcolumn1使用默认训练模型对列上的 spark 数据集进行情感分析。这是我的代码：

DocumentAssembler docAssembler = (DocumentAssembler) new DocumentAssembler().setInputCol("column1")
                .setOutputCol("document");

Tokenizer tokenizer = (Tokenizer) ((Tokenizer) new Tokenizer().setInputCols(new String[] { "document" }))
                .setOutputCol("token");
String[] inputCols = new String[] { "token", "document" };

SentimentDetector sentiment = ((SentimentDetector) ((SentimentDetector) new SentimentDetector().setInputCols(inputCols)).setOutputCol("sentiment"));
Pipeline pipeline = new Pipeline().setStages(new PipelineStage[] { docAssembler, tokenizer, sentiment });

// Fit the pipeline to training documents.
PipelineModel pipelineFit = pipeline.fit(ds);
ds = pipelineFit.transform(ds);
ds.show();

这ds是Dataset<Row>包含列的列column1。我收到以下错误。

java.util.NoSuchElementException: Failed to find a default value for dictionary
at org.apache.spark.ml.param.Params$$anonfun$getOrDefault$2.apply(params.scala:780)
at org.apache.spark.ml.param.Params$$anonfun$getOrDefault$2.apply(params.scala:780)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.ml.param.Params$class.getOrDefault(params.scala:779)
at org.apache.spark.ml.PipelineStage.getOrDefault(Pipeline.scala:42)
at org.apache.spark.ml.param.Params$class.$(params.scala:786)
at org.apache.spark.ml.PipelineStage.$(Pipeline.scala:42)
at com.johnsnowlabs.nlp.annotators.sda.pragmatic.SentimentDetector.train(SentimentDetector.scala:62)
at com.johnsnowlabs.nlp.annotators.sda.pragmatic.SentimentDetector.train(SentimentDetector.scala:12)
at com.johnsnowlabs.nlp.AnnotatorApproach.fit(AnnotatorApproach.scala:45)
at org.apache.spark.ml.Pipeline$$anonfun$fit$2.apply(Pipeline.scala:153)
at org.apache.spark.ml.Pipeline$$anonfun$fit$2.apply(Pipeline.scala:149)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableViewLike$Transformed$class.foreach(IterableViewLike.scala:44)
at scala.collection.SeqViewLike$AbstractTransformed.foreach(SeqViewLike.scala:37)
at org.apache.spark.ml.Pipeline.fit(Pipeline.scala:149)

我已经浏览了示例，但我找不到任何使用默认模型在 java 中进行情感分析的明确示例/文档。

4

1 回答 1

3

所以最后我想通了。最终代码：

    DocumentAssembler docAssembler = (DocumentAssembler) new DocumentAssembler().setInputCol("column1")
                .setOutputCol("document");

Tokenizer tokenizer = (Tokenizer) ((Tokenizer) new Tokenizer().setInputCols(new String[] { "document" }))
                .setOutputCol("token");
String[] inputCols = new String[] { "token", "document" };

ViveknSentimentModel sentiment  = (ViveknSentimentModel) ViveknSentimentModel
.load("/path/to/pretained model folder");

Pipeline pipeline = new Pipeline().setStages(new PipelineStage[] { docAssembler, tokenizer, sentiment });

// Fit the pipeline to training documents.
PipelineModel pipelineFit = pipeline.fit(ds);
ds = pipelineFit.transform(ds);

模型可以从这里下载。

于 2019-09-24T18:57:14.940 回答