2

下午好，

我正在尝试在 cosmos HDFS 中配置 Orion Context 的持久性。到目前为止，我已经创建了一个 Orion 上下文代理实例（orion-psb-image-R4.2），创建了一个 Cosmos 用户帐户和其中的文件夹“cardio”（/user/USERNAME/cardio）。

我还将 cygnus.conf 文件（/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf）配置为如下：

# APACHE_FLUME_HOME/conf/cygnus.conf
orionagent.sources = http-source
orionagent.sinks = hdfs-sink
orionagent.channels = notifications

# Flume source, must not be changed
orionagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# channel name where to write the notification events
orionagent.sources.http-source.channels = notifications
# listening port the Flume source will use for receiving incoming notifications
orionagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
orionagent.sources.http-source.handler = es.tid.fiware.orionconnectors.cosmosinjector.OrionRestHandler
# regular expression for the orion version the notifications will have in their headers
orionagent.sources.http-source.handler.orion_version = 0\.11\.*
# URL target
orionagent.sources.http-source.handler.notification_target = /notify

# channel name from where to read notification events
orionagent.sinks.hdfs-sink.channel = notifications
# Flume sink that will process and persist in HDFS the notification events, must not be changed
orionagent.sinks.hdfs-sink.type = es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink
# IP address of the Cosmos deployment where the notification events will be persisted
orionagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
orionagent.sinks.hdfs-sink.cosmos_port = 14000
# username allowed to write in HDFS (/user/myusername)
orionagent.sinks.hdfs-sink.cosmos_username = USERNAME
# dataset where to persist the data (/user/myusername/mydataset)
orionagent.sinks.hdfs-sink.cosmos_dataset = cardio
# HDFS backend type (webhdfs, httpfs or infinity)
orionagent.sinks.hdfs-sink.hdfs_api = httpfs

# channel name
orionagent.channels.notifications.type = memory
# capacity of the channel
orionagent.channels.notifications.capacity = 1000
# amount of bytes that can be sent per transaction
orionagent.channels.notifications.transactionCapacity = 100

问题是当我使用以下命令启动代理时：

/usr/local/apache-flume-1.4.0-bin/bin/flume-ng agent --conf /usr/local/apache-flume-1.4.0-bin/conf/ -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent -Dflume.root.logger=INFO,consoleapache-flume-1.4.0-bin

我收到以下错误消息：

2015-07-17 15:33:10,477 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)] HttpFS operation: PUT http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&user.name=arlourenco HTTP/1.1
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:71)] HttpFS response: HTTP/1.1 401 Unauthorized
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [ERROR - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] The  directory could not be created in HDFS. HttpFS response: 401 Unauthorized
2015-07-17 15:33:17,470 (910628961@qtp-586740701-0) [WARN - org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:186)] Received bad request from client.

和：

org.apache.flume.source.http.HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported

这是完整的日志：

+ exec /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/usr/local/apache-flume-1.4.0-bin/conf:/usr/local/apache-flume-1.4.0-bin/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf -n orionagent
2015-07-17 15:33:09,858 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2015-07-17 15:33:09,863 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:/usr/local/apache-flume-1.4.0-bin/conf/cygnus.conf
2015-07-17 15:33:09,871 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: hdfs-sink Agent: orionagent
2015-07-17 15:33:09,871 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,872 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,873 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:hdfs-sink
2015-07-17 15:33:09,885 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [orionagent]
2015-07-17 15:33:09,885 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:150)] Creating channels
2015-07-17 15:33:09,899 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:40)] Creating instance of channel notifications type memory
2015-07-17 15:33:09,908 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:205)] Created channel notifications
2015-07-17 15:33:09,912 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:39)] Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
2015-07-17 15:33:09,961 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: hdfs-sink, type: es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink
2015-07-17 15:33:09,967 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:119)] Channel notifications connected to [http-source, hdfs-sink]
2015-07-17 15:33:09,972 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:http-source,state:IDLE} }} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@e8234ec counterGroup:{ name:null counters:{} } }} channels:{notifications=org.apache.flume.channel.MemoryChannel{name: notifications}} }
2015-07-17 15:33:09,973 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel notifications
2015-07-17 15:33:10,027 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: CHANNEL, name: notifications, registered successfully.
2015-07-17 15:33:10,028 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: CHANNEL, name: notifications started
2015-07-17 15:33:10,028 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink hdfs-sink
2015-07-17 15:33:10,029 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source http-source
2015-07-17 15:33:10,069 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-07-17 15:33:10,149 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26
2015-07-17 15:33:10,306 (lifecycleSupervisor-1-3) [INFO - org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started SocketConnector@0.0.0.0:5050
2015-07-17 15:33:10,307 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: SOURCE, name: http-source, registered successfully.
2015-07-17 15:33:10,307 (lifecycleSupervisor-1-3) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: SOURCE, name: http-source started
2015-07-17 15:33:10,456 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.http.HttpClientFactory.<init>(HttpClientFactory.java:68)] Setting max total connections to 500 and default max connections per route to 100
2015-07-17 15:33:10,477 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)] HttpFS operation: PUT http://130.206.80.46:14000/webhdfs/v1/user/arlourenco/cardio/?op=mkdirs&user.name=arlourenco HTTP/1.1
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:71)] HttpFS response: HTTP/1.1 401 Unauthorized
2015-07-17 15:33:10,708 (lifecycleSupervisor-1-1) [ERROR - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] The  directory could not be created in HDFS. HttpFS response: 401 Unauthorized
2015-07-17 15:33:17,470 (910628961@qtp-586740701-0) [WARN - org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:186)] Received bad request from client.
org.apache.flume.source.http.HTTPBadRequestException: orion/0.19.0 libcurl/7.19.7 user agent not supported
        at es.tid.fiware.orionconnectors.cosmosinjector.OrionRestHandler.getEvents(OrionRestHandler.java:93)
        at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:184)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

可能是什么问题？

4

1 回答 1

0

首先，您使用的是相当旧的 Cygnus 版本；我猜它是预装在 orion-psb-image-R4.2 中的那个，它是 0.5.2。最新的是 0.8.2，所以请考虑升级，特别是在 Cosmos 的 WebHDFS 中启用 OAUth2 身份验证/授权后（您可以在此处找到有关升级的详细说明和有关在 WebHDFS 中使用 OAuth2 的详细说明）。

无论如何，问题是您发送的用户代理标头不同于orion/*, ie libcurl/7.19.7。是否可以避免从您的客户端发送这样的用户代理？如果不是，则应修改 Cygnus 代码以接受所有类型的用户代理（之前其他用户指出的东西，我承认它可能有用）。

于 2015-07-17T16:16:17.313 回答