apache-spark - GraphFrames api 是否支持创建二分图？

Question

GraphFrames api 是否支持在当前版本中创建二分图？

当前版本：0.1.0

火花版本：1.6.1

Answer 1

正如对这个问题的评论所指出的，GraphFrames 和 GraphX 都没有对二分图的内置支持。但是，它们都具有足够的灵活性，可以让您创建二分图。对于 GraphX 解决方案，请参阅这个先前的答案。该解决方案使用不同顶点/对象类型之间的共享特征。虽然这适用RDDs于DataFrames. a 中的一行DataFrame有一个固定的模式——它有时不能包含一price列，有时不包含。它可以有一个price有时null为 的列，但该列必须存在于每一行中。

相反，解决方案GraphFrames似乎是您需要定义一个DataFrame本质上是二分图中两种类型对象的线性子类型 - 它必须包含两种类型对象的所有字段。这实际上很容易——一个joinwithfull_outer会给你这个。像这样的东西：

val players = Seq(
  (1,"dave", 34),
  (2,"griffin", 44)
).toDF("id", "name", "age")

val teams = Seq(
  (101,"lions","7-1"),
  (102,"tigers","5-3"),
  (103,"bears","0-9")
).toDF("id","team","record")

然后你可以像这样创建一个超集DataFrame：

val teamPlayer = players.withColumnRenamed("id", "l_id").join(
  teams.withColumnRenamed("id", "r_id"),
  $"r_id" === $"l_id", "full_outer"
).withColumn("l_id", coalesce($"l_id", $"r_id"))
 .drop($"r_id")
 .withColumnRenamed("l_id", "id")

teamPlayer.show

+---+-------+----+------+------+
| id|   name| age|  team|record|
+---+-------+----+------+------+
|101|   null|null| lions|   7-1|
|102|   null|null|tigers|   5-3|
|103|   null|null| bears|   0-9|
|  1|   dave|  34|  null|  null|
|  2|griffin|  44|  null|  null|
+---+-------+----+------+------+

您可以使用以下方法使其更清洁structs：

val tpStructs = players.select($"id" as "l_id", struct($"name", $"age") as "player").join(
  teams.select($"id" as "r_id", struct($"team",$"record") as "team"),
  $"l_id" === $"r_id",
  "full_outer"
).withColumn("l_id", coalesce($"l_id", $"r_id"))
 .drop($"r_id")
 .withColumnRenamed("l_id", "id")

tpStructs.show

+---+------------+------------+
| id|      player|        team|
+---+------------+------------+
|101|        null| [lions,7-1]|
|102|        null|[tigers,5-3]|
|103|        null| [bears,0-9]|
|  1|   [dave,34]|        null|
|  2|[griffin,44]|        null|
+---+------------+------------+

我还要指出，或多或少相同的解决方案适用GraphX于RDDs. 您总是可以通过加入两个case classes不共享的顶点来创建一个顶点traits：

case class Player(name: String, age: Int)
val playerRdd = sc.parallelize(Seq(
  (1L, Player("date", 34)),
  (2L, Player("griffin", 44))
))

case class Team(team: String, record: String)
val teamRdd = sc.parallelize(Seq(
  (101L, Team("lions", "7-1")),
  (102L, Team("tigers", "5-3")),
  (103L, Team("bears", "0-9"))
))

playerRdd.fullOuterJoin(teamRdd).collect foreach println
(101,(None,Some(Team(lions,7-1))))
(1,(Some(Player(date,34)),None))
(102,(None,Some(Team(tigers,5-3))))
(2,(Some(Player(griffin,44)),None))
(103,(None,Some(Team(bears,0-9))))

考虑到前面的答案，这似乎是一种更灵活的处理方式——无需trait在组合对象之间共享 a 。