我正在尝试使用 Scala 从 Spark MLLib 中获取一些频繁的项目集和关联规则。但实际上我什么也没得到，甚至没有错误。可以在此处找到代码（spark/databricks 笔记本）和数据输入文件。
该算法要么没有找到任何频繁项集和/或关联规则，但有一些证据表明这是错误的。我对 KNIME（一个非编程分析平台）做了同样的事情，但使用 Borgelt 算法进行关联规则学习。在那里，我得到了带有升力和所有其他所需比率的前因、后因映射。但是在使用 Scala 的 Spark 中，我什么也得不到。
%scala
import org.apache.spark.rdd.RDD
import org.apache.spark.mllib.fpm.AssociationRules
import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
// loading data
val data = sc.textFile("/FileStore/tables/onlinePurchasedProducts.txt")
val onlineTrx: RDD[Array[String]] = data.map(s => s.trim.split(' '))
println("Read: " + onlineTrx.count() + " online baskets")
// checking how transactions look like
val dataframe = onlineTrx.toDF()
println("Schema of transactions looks like: ")
dataframe.printSchema()
println("Content of transactions looks like: ")
dataframe.show()
val fpg = new FPGrowth()
val model = fpg
.setMinSupport(0.2)
.setNumPartitions(1)
.run(onlineTrx)
model.freqItemsets.collect().foreach { itemset =>
println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}
model.generateAssociationRules(0.4).collect().foreach { rule =>
println(s"${rule.antecedent.mkString("[", ",", "]")}=> " +
s"${rule.consequent .mkString("[", ",", "]")},${rule.confidence}")
}
这段代码的输出是：
Read: 42897 online baskets
Schema of transactions looks like:
root
|-- value: array (nullable = true)
| |-- element: string (containsNull = true)
Content of transactions looks like:
e+--------------------+
| value|
+--------------------+
| [34502, 70312]|
| [44247]|
| [45127]|
| [79560]|
| [74801]|
| [15500]|
| [74801]|
| [31149, 78707]|
| [74801]|
| [40774]|
| [76675]|
|[26507, 26638, 33...|
| [74801]|
| [78707]|
| [74801]|
| [21253]|
| [74801]|
|[75729, 10899, 26...|
| [24834]|
| [74801]|
+--------------------+
only showing top 20 rows
import org.apache.spark.rdd.RDD
import org.apache.spark.mllib.fpm.AssociationRules
import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
data: org.apache.spark.rdd.RDD[String]=
/FileStore/tables/onlinePurchasedProducts.txt MapPartitionsRDD[150] at
textFile at command-4263745371438753:8
onlineTrx: org.apache.spark.rdd.RDD[Array[String]] = MapPartitionsRDD[151] at map at command-4263745371438753:9
dataframe: org.apache.spark.sql.DataFrame = [value: array<string>]
fpg: org.apache.spark.mllib.fpm.FPGrowth = org.apache.spark.mllib.fpm.FPGrowth@23fd0c4
model: org.apache.spark.mllib.fpm.FPGrowthModel[String] = org.apache.spark.mllib.fpm.FPGrowthModel@41278271
任何想法，将不胜感激。