2

您能否提供为 Facebook 数据流配置 Flume 的步骤。

我们已经成功配置了从 Twitter 提取数据的水槽。

请查看我们为从 twitter 提取数据而创建的配置。

Twitter 的 Flume 配置

TwitterAgent.sources = Twitter 
TwitterAgent.channels = MemChannel 
TwitterAgent.sinks = HDFS

# Describing/Configuring the source 
TwitterAgent.sources.Twitter.type =          org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey =  4ENqf3q23iwdTSDJchv7w 
TwitterAgent.sources.Twitter.consumerSecret =      bAPTWfbRildBMWsEHo56SmZeXkftvZNCgvjHXbcUfAKoKzQjY0VIUOftTh6c 
TwitterAgent.sources.Twitter.accessToken = 736128293661855746-   rQIQYZNGCh9lW8XHCkjcnvwZH1BItnGi0XJ0gHM26F
TwitterAgent.sources.Twitter.accessTokenSecret = ehTsqX7GcU1aBqmekDcwPuu1csFOnfgzxc2EPtS0kudXOADeAAI 
TwitterAgent.sources.Twitter.keywords = modi, india elections, bjp,   congress, tdp,jana sena, pwan kalyan, mohanlal

# Describing/Configuring the sink 

TwitterAgent.sinks.HDFS.type = hdfs 
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://env11-hadoop-  master.trv.flytxt.com:54310/user/Hadoop/twitter_data
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = writable 
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1
TwitterAgent.sinks.HDFS.hdfs.rollSize = 1
TwitterAgent.sinks.HDFS.hdfs.rollCount = 1

# Describing/Configuring the channel TwitterAgent.channels.MemChannel.type =   memory 
TwitterAgent.channels.MemChannel.capacity = 10000 
TwitterAgent.channels.MemChannel.transactionCapacity = 100
TwitterAgent.channels.MemChannel.type = file

# Binding the source and sink to the channel 
TwitterAgent.sources.Twitter.channels = MemChannel

TwitterAgent.sinks.HDFS.channel = MemChannel

For configuring twitter we need the following configurations

TwitterAgent.sources.Twitter.consumerKey =   4ENqf3q23iwsdfmhadfjafjkemliSYs7w 
TwitterAgent.sources.Twitter.consumerSecret =   bAPTWfbRildangxvasxvhaxjasbxkjtvUfAKoKzQjY0VIUOftTh6c 
TwitterAgent.sources.Twitter.accessToken = 7361282936618557ZNbcvHJxjxbnH1BItnGi0XJ0gHM26F
TwitterAgent.sources.Twitter.accessTokenSecret = ehTsASNMGCxvashgvcxjAHvcSFGcjahgPuu1csFO2EPtS0kudXOADeAAI

但是如何为 facebook 获得相同的信息？或使用 Flume 为 Facebook 数据流提供工作配置

4

1 回答 1

-1

是的，您当然可以借助 Facebook 的 scribe 工具以日志形式从 Facebook 接收数据。

scribe的安装部分可以参考：http ://blog.octo.com/en/scribe-installation/

对于工作部分，您可以参考：http ://blog.octo.com/en/scribe-a-way-to-aggregate-data-and-why-not-to-directly-fill-the-hdfs/

于 2017-06-10T06:40:53.510 回答