pyspark - Convert Dstream to dataframe using pyspark

Question

How i can convert a DStream to an dataframe? here is my actual code

localhost = "127.0.0.1"
addresses = [(localhost, 9999)]
schema = ['event', 'id', 'time','occurence']
flumeStream = FlumeUtils.createPollingStream(ssc, addresses)
counts = flumeStream.map(lambda line: str(line).split(",")) \
        .filter(lambda line: len(line)>1) \
        .map(lambda line: (line[29],line[30],line[67],1)) \
        .foreachRDD(lambda rdd: sqlContext.createDataFrame(rdd))

counts.show()

ssc.start()
ssc.awaitTerminationOrTimeout(62)
ssc.stop()

it gives me the following error:

AttributeError: 'NoneType' object has no attribute 'show'

Answer 1

将您的 DStream 转换为 RDD，然后转换为 DataFrame，即 dstrea.rdd.to_df