scala - REPL returns RDD values but SBT won't compile

Question

When running the below method from a fresh spark shell REPL session everything works fine. However when I try to compile the class containing this method I get the following errors

Error:(21, 50) value values is not a member of org.apache.spark.rdd.RDD[(Long, org.apache.spark.mllib.recommendation.Rating)]
val training = ratings.filter(x => x._1 < 6).values.repartition(numPartitions).persist
                                             ^
Error:(22, 65) value values is not a member of org.apache.spark.rdd.RDD[(Long, org.apache.spark.mllib.recommendation.Rating)]
val validation = ratings.filter(x => x._1 >= 6 && x._1 < 8).values.repartition(numPartitions).persist
                                                            ^
Error:(23, 47) value values is not a member of org.apache.spark.rdd.RDD[(Long, org.apache.spark.mllib.recommendation.Rating)]
val test = ratings.filter(x => x._1 >= 8).values.persist
                                          ^

In both cases I'm using Spark 1.0.1 The code itself is as follows.

def createDataset(ratings: RDD[Tuple2[Long,Rating]]): List[RDD[Rating]] = {

    val training = ratings.filter(x => x._1 < 6).values.repartition(numPartitions).persist
    val validation = ratings.filter(x => x._1 >= 6 && x._1 < 8).values.repartition(numPartitions).persist
    val test = ratings.filter(x => x._1 >= 8).values.persist
    val numTraining = training.count
    val numValidation = validation.count
    val numTest = test.count

    println(" Number Of Training ::: " + numTraining + " numValidation ::: " + numValidation + " ::: " + numTest)
    List(training,validation,test)
  }

It is taken from the MLLib tutorial (Adapted slightly) , no idea whats going wrong.

Answer 1

你需要在你的代码中有这一行

import org.apache.spark.SparkContext._

PairRDDFunctions这将导入允许您调用的隐式转换values。spark REPL 会为您导入此内容，这就是您在解释器中看不到错误的原因。具体来说，这个函数在SparkContext做转换。

implicit def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
      (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null) = {
    new PairRDDFunctions(rdd)
  }