apache-spark - 使用 Spark 计算句子之间的相似度

Question

我的问题陈述有以下输入：-

 ID  -> List of Words
(101 -> Array("a1","b2","c4","d2"))
(102 -> Array("a6","b1","c5","d3"))
(103 -> Array("a1","b4","c4","d2"))
(104 -> Array("a2","b2","c3","d2"))
(105 -> Array("a7","b6","c1","d3"))

现在，我想找出这些输入语句之间的相似性。

例子：-

(101 -> Array("a1","b2","c4","d2"))
(103 -> Array("a1","b4","c4","d2"))
(104 -> Array("a2","b2","c3",",d2"))

在示例输出中，语句彼此非常相似。

如何使用 Spark 实现这一目标？我可以使用任何逻辑代码或任何机器学习算法。

谢谢