我尝试使用 spark 1.1.0 提供的新 TFIDF 算法。我正在用 Java 编写我的 MLLib 工作,但我不知道如何让 TFIDF 实现工作。出于某种原因,IDFModel只接受JavaRDD作为方法转换的输入,而不是简单的 Vector。如何使用给定的类为我的 LabledPoints 建模 TFIDF 向量?
注意:文档行格式为 [Label; 文本]
到目前为止,这是我的代码:
// 1.) Load the documents
JavaRDD<String> data = sc.textFile("/home/johnny/data.data.new");
// 2.) Hash all documents
HashingTF tf = new HashingTF();
JavaRDD<Tuple2<Double, Vector>> tupleData = data.map(new Function<String, Tuple2<Double, Vector>>() {
@Override
public Tuple2<Double, Vector> call(String v1) throws Exception {
String[] data = v1.split(";");
List<String> myList = Arrays.asList(data[1].split(" "));
return new Tuple2<Double, Vector>(Double.parseDouble(data[0]), tf.transform(myList));
}
});
tupleData.cache();
// 3.) Create a flat RDD with all vectors
JavaRDD<Vector> hashedData = tupleData.map(new Function<Tuple2<Double,Vector>, Vector>() {
@Override
public Vector call(Tuple2<Double, Vector> v1) throws Exception {
return v1._2;
}
});
// 4.) Create a IDFModel out of our flat vector RDD
IDFModel idfModel = new IDF().fit(hashedData);
// 5.) Create Labledpoint RDD with TFIDF
???
肖恩欧文的解决方案:
// 1.) Load the documents
JavaRDD<String> data = sc.textFile("/home/johnny/data.data.new");
// 2.) Hash all documents
HashingTF tf = new HashingTF();
JavaRDD<LabeledPoint> tupleData = data.map(v1 -> {
String[] datas = v1.split(";");
List<String> myList = Arrays.asList(datas[1].split(" "));
return new LabeledPoint(Double.parseDouble(datas[0]), tf.transform(myList));
});
// 3.) Create a flat RDD with all vectors
JavaRDD<Vector> hashedData = tupleData.map(label -> label.features());
// 4.) Create a IDFModel out of our flat vector RDD
IDFModel idfModel = new IDF().fit(hashedData);
// 5.) Create tfidf RDD
JavaRDD<Vector> idf = idfModel.transform(hashedData);
// 6.) Create Labledpoint RDD
JavaRDD<LabeledPoint> idfTransformed = idf.zip(tupleData).map(t -> {
return new LabeledPoint(t._2.label(), t._1);
});