0

我成功地遵循了 DAVID TAIEB 的这个很棒的教程。问题是图表需要很长时间才能改变。它大约需要几分钟，而不是几秒钟，因为它应该是！

可能是什么问题呢 ？

当我在 Scala Notebook 中运行这个单元格时：

demo.startTwitterStreaming(sc)

输出如下所示：

Registering JaasConfiguration: /tmp/1sFcnUVoabfuBAyF/jaas.conf
Setting hadoop configuration for swift container
https://kafka-rest-prod01.messagehub.services.us-south.bluemix.net:443/admin/topics
https://kafka-rest-prod01.messagehub.services.us-south.bluemix.net:443/admin/topics
default location of ssl Trust store is: /usr/local/src/spark160master/ibm-java-x86_64-80/jre/lib/security/cacerts
Twitter stream started
Tweets are collected real-time and analyzed
To stop the streaming and start interacting with the data use: StreamingTwitter.stopTwitterStreaming
Receiver Started: KafkaReceiver-0
Batch started with 0 records
-------------------------------------------
Time: 1460572070000 ms
-------------------------------------------

Batch completed with 0 records
Batch started with 0 records
-------------------------------------------
Time: 1460572075000 ms
-------------------------------------------

Batch completed with 0 records
Batch started with 487 records
-------------------------------------------
Time: 1460572080000 ms
-------------------------------------------
(#TDS,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(30.5, 0.0, 16.5, 0.0, 19.5, 3.0, 38.5, 20.0, 5.0, 32.0, 28.0, 9.0, 0.0)))
(#DolceAmoreILoveYou,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(5.0, 0.0, 6.0, 0.0, 17.0, 1.0, 17.5, 12.5, 24.5, 34.0, 44.0, 2.0, 0.0)))
(#nasty,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(0.5, 0.0, 16.0, 0.0, 0.5, 34.5, 49.0, 1.0, 0.0, 46.0, 16.0, 2.5, 0.0)))
(#Job:,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(8.0, 0.0, 2.5, 0.0, 28.0, 1.5, 4.5, 2.5, 45.0, 19.5, 37.0, 3.5, 0.0)))
(#bigbrand,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(5.5, 0.0, 6.0, 0.0, 49.0, 4.0, 2.0, 15.0, 12.5, 3.5, 48.5, 5.5, 0.0)))
(#SwitchUp,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(47.5, 0.0, 3.5, 0.0, 34.5, 1.0, 47.0, 29.0, 11.5, 22.5, 1.0, 3.5, 0.0)))
(#Hiring,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(8.0, 0.0, 2.5, 0.0, 28.0, 1.5, 4.5, 2.5, 45.0, 19.5, 37.0, 3.5, 0.0)))
(#entrepreneurship,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(20.5, 25.5, 5.0, 0.0, 28.0, 9.0, 12.0, 3.0, 28.0, 4.0, 12.5, 2.5, 0.0)))
(#startup,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(5.5, 0.0, 6.0, 0.0, 49.0, 4.0, 2.0, 15.0, 12.5, 3.5, 48.5, 5.5, 0.0)))
(#porn,(1,List(agreeableness_big5, analytical, anger, confident, conscientiousness_big5, disgust, extraversion_big5, fear, joy, neuroticism_big5, openness_big5, sadness, tentative),List(0.5, 0.0, 16.0, 0.0, 0.5, 34.5, 49.0, 1.0, 0.0, 46.0, 16.0, 2.5, 0.0)))
...

它看起来不像教程显示的那样： https://i0.wp.com/developer.ibm.com/clouddataservices/wp-content/uploads/sites/47/2016/01/Running-Spark-Streaming.png？缩放=2&w=980

我希望你能帮助我。是什么导致我的图表每隔几秒就改变一次？

当我关闭应用程序和内核并尝试重新打开它们时，这就是我所做的：

1-重新启动内核 2-再次运行所有单元

这是正确的吗 ？

谢谢@DavidTaieb

4

3 回答 3

1

由于底层服务（MessageHub、MessageConnect 和 Watson Tone Analyzer）中的 api 更改，本教程最近进行了更新。作为更改的一部分，我添加了一个 StreamingListener 以提供更精细的日志消息。不幸的是，我忘记更新显示正在运行的单元格的屏幕截图。无论如何，您正在运行最新版本，这很棒。
从您包含的片段中，看起来微批处理已正确处理，并且结果已成功发送到 MessageHub/Kafka。仪表板配置为每隔几秒轮询一次 MessageHub，但您似乎需要一整分钟才能刷新。我在我的测试环境中重新运行了这个应用程序，我开始注意到同样的事情。查看日志，我注意到对 MessageHub 的其余调用开始返回 409 错误代码。重新启动前端应用程序确实有助于解决该问题。我建议你做同样的事情来解锁你。我将在未来几天进一步调查此错误，并在我得到可接受的修复后提供更新。

于 2016-04-14T03:58:57.973 回答
0

如果在前一个请求完成之前发出消费者请求，则消息中心设置为返回 409 错误代码。增加轮询间隔应该可以解决这个问题。

我在提供的示例中发现的另一个问题是，boostrap 服务器设置了端口 9094 和 9093。截至 3 月 3 日，端口 9094 已关闭。请确保您仅使用端口 9093。

于 2016-04-14T09:33:11.223 回答
0

当我在一小时前尝试教程时，从包含以下命令的单元格中收到以下错误：“demo.startTwitterStreaming(sc)”

注册 JaasConfiguration：/tmp/BA8rP92KZiZZSkpq/jaas.conf 为 swift 容器设置 hadoop 配置 https://kafka-rest-prod01.messagehub.services.us-south.bluemix.net:443/admin/topics https://kafka- rest-prod01.messagehub.services.us-south.bluemix.net:443/admin/topics

点击上面的链接说 - {"error_code":40101,"message":"Missing Key"}

输出[3]：

名称：java.lang.NullPointerException 消息：null StackTrace：org.apache.hadoop.fs.swift.http.SwiftRestClient$AuthenticationPost.extractResult(SwiftRestClient.java:1293) org.apache.hadoop.fs.swift.http.SwiftRestClient$ AuthenticationPost.extractResult(SwiftRestClient.java:1245)......

java.lang.Thread.run(Thread.java:785)

于 2016-05-10T12:57:34.117 回答