1

根据此处的 Flume 文档

Flume 源使用外部源（如 Web 服务器）传递给它的事件。外部源以目标 Flume 源可识别的格式向 Flume 发送事件。例如，Avro Flume 源可用于从 Avro 客户端或流中其他从 Avro 接收器发送事件的 Flume 代理接收 Avro 事件。

为什么 Flume 源需要识别或理解消息的格式？虽然它所做的只是将消息转发到其中一个频道。

4

1 回答 1

0

根据我所了解的，Flume 将传输数据封装在由标头和有效负载（传输数据）组成的事件数据包中。从文档中：

Flume 事件被定义为具有字节有效负载和一组可选字符串属性的数据流单元。

在您的文档引用之前。

您指定的格式是事件数据包的格式，而不是数据的格式。

假设你有这个代理：

plain_to_avro_translator.sources = plain-source avro-source
plain_to_avro_translator.sinks = avro-sink local-file-sink
plain_to_avro_translator.channels = mem-channel1 mem-channel2

plain_to_avro_translator.sources.plain-source.channels = mem-channel1
plain_to_avro_translator.sources.plain-source.type = exec
plain_to_avro_translator.sources.plain-source.restart = true
plain_to_avro_translator.sources.plain-source.restartThrottle = 40000
plain_to_avro_translator.sources.plain-source.command = cat /home/user/data.log

plain_to_avro_translator.sinks.avro-sink.channel = mem-channel1
plain_to_avro_translator.sinks.avro-sink.type = thrift
plain_to_avro_translator.sinks.avro-sink.hostname = 192.168.200.43
plain_to_avro_translator.sinks.avro-sink.port = 6000

plain_to_avro_translator.channels.mem-channel1.type = memory
plain_to_avro_translator.channels.mem-channel1.capacity = 100
plain_to_avro_translator.channels.mem-channel1.transactionCapacity = 100

plain_to_avro_translator.sources.avro-source.channels = mem-channel2
plain_to_avro_translator.sources.avro-source.type = thrift
plain_to_avro_translator.sources.avro-source.bind = 0.0.0.0
plain_to_avro_translator.sources.avro-source.port = 6000

plain_to_avro_translator.channels.mem-channel2.type = memory
plain_to_avro_translator.channels.mem-channel2.capacity = 100
plain_to_avro_translator.channels.mem-channel2.transactionCapacity = 100

plain_to_avro_translator.sinks.local-file-sink.channel = mem-channel2
plain_to_avro_translator.sinks.local-file-sink.type = file_roll
plain_to_avro_translator.sinks.local-file-sink.sink.directory = /home/user/flume_output

这将毫无问题地工作，并且不依赖于 data.log 格式（您可以编写任何您需要的内容和任何格式）。如果您尝试将 avro-sink 类型设置为 avro 而不是 thrift，您将从 avro-source 收到错误，因为它需要 thrift 格式事件。

Sink 和 Source 需要知道如何解析事件包。

希望我得到它。如果我错了，请任何人纠正我。

于 2018-08-20T17:35:31.960 回答