scala - 创建RDD时未找到火花错误RDD类型

Question

我正在尝试创建案例类对象的 RDD。例如。，

// sqlContext from the previous example is used in this example.
// createSchemaRDD is used to implicitly convert an RDD to a SchemaRDD.
import sqlContext.createSchemaRDD

val people: RDD[Person] = ... // An RDD of case class objects, from the previous example.

// The RDD is implicitly converted to a SchemaRDD by createSchemaRDD, allowing it to be stored using        Parquet.
people.saveAsParquetFile("people.parquet")

我正在尝试通过给出来完成上一个示例中的部分

    case class Person(name: String, age: Int)

    // Create an RDD of Person objects and register it as a table.
    val people: RDD[Person] = sc.textFile("/user/root/people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt))
    people.registerAsTable("people")

我收到以下错误：

<console>:28: error: not found: type RDD
       val people: RDD[Person] =sc.textFile("/user/root/people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt))

知道出了什么问题吗？提前致谢！

Answer 1

这里的问题是显式RDD[String]类型注释。看起来RDD默认情况下没有导入 in spark-shell，这就是 Scala 抱怨找不到RDD类型的原因。先试试跑步import org.apache.spark.rdd.RDD。