fiware - 制作 Fiware-Cygnus 存储到 Apache-Phoenix 数据库

Question

我是 Cygnus 和 Phoenix db 的新手。我希望 Cygnus 将数据存储在 Apache Phoenix 数据库中，因此我必须配置 Phoenix Sink 以与 Cygnus 一起使用。

我知道 Cygnus 是 Apache Flume 的扩展，所以默认情况下（理论上）任何 Flume 代理都可以使用它。我agent_phoenix.conf为 Cygnus 配置了我的 Flume 代理文件，如下所示：

cygnus-ngsi.channels = memoryChannel
cygnus-ngsi.sinks = phoenix-sink

cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = memoryChannel
cygnus-ngsi.sources.http-source.port = 5051
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf

cygnus-ngsi.channels.memoryChannel.type=memory
cygnus-ngsi.channels.memoryChannel.transactionCapacity=100
cygnus-ngsi.channels.memoryChannel.byteCapacityBufferPercentage=20

cygnus-ngsi.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink
cygnus-ngsi.sinks.phoenix-sink.channel=memoryChannel
cygnus-ngsi.sinks.phoenix-sink.batchSize=100
cygnus-ngsi.sinks.phoenix-sink.table=PROVA_PHOENIX
cygnus-ngsi.sinks.phoenix-sink.ddl=CREATETABLEIFNOTEXISTSPROVA_PHOENIX(uidVARCHARNOTNULL,f_hostVARCHARCONSTRAINTpkPRIMARYKEY(uid),colonnaProvaVARCHAR)
cygnus-ngsi.sinks.phoenix-sink.zookeeperQuorum=localhost
cygnus-ngsi.sinks.phoenix-sink.serializer=REGEX
cygnus-ngsi.sinks.phoenix-sink.serializer.rowkeyType=uuid
cygnus-ngsi.sinks.phoenix-sink.serializer.columns=colonnaProva

并在 CentOS 命令行上使用此命令启动它：

/usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_phoenix.conf -n cygnus-ngsi -Dflume.root.logger=INFO,console

它开始进行一些初始化，但随后我收到以下消息：

76666 [lifecycleSupervisor-1-8] ERROR org.apache.phoenix.flume.serializer.BaseEventSerializer  - error No suitable driver found for jdbc:phoenix:localhost; occurred during initializing connection 
176667 [lifecycleSupervisor-1-8] ERROR org.apache.phoenix.flume.sink.PhoenixSink  - Error No suitable driver found for jdbc:phoenix:localhost; in initializing the serializer.
176667 [lifecycleSupervisor-1-8] ERROR org.apache.flume.lifecycle.LifecycleSupervisor  - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@29918f3 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at com.google.common.base.Throwables.propagate(Throwables.java:156)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:124)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.phoenix.flume.serializer.BaseEventSerializer.initialize(BaseEventSerializer.java:140)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:119)
... 10 more

HBase和Phoenix在这台机器上安装正确，实际上HBase可以启动并且Phoenix已经用SQLline文件测试过了。你有解决这个问题的方法吗？

----编辑----我只是向前迈了一步。我在 cygnus/conf 目录中编辑了我的flume-env.sh文件，并将包含PhoenixSinkjava 类的 .jar 文件指定为 FLUME_CLASSPATH。现在启动我在上面编写的相同命令开始配置（显然很好）然后我收到此错误消息：

time=2017-10-30T22:27:40.552Z | lvl=ERROR | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=run | msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[142] : Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load sink type: org.apache.phoenix.flume.sink.PhoenixSink, class: org.apache.phoenix.flume.sink.PhoenixSink
at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.phoenix.flume.sink.PhoenixSink
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:195)
at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
... 11 more

现在有什么问题？

-------------编辑 2--------------------

好的，我离得更近了，但是这个错误没有意义。我通过在类路径中复制正确的 phoenix JAR 解决了 jdbc 问题，然后我得到了一些依赖错误，我通过在类路径中复制其他 JAR 来解决它们。现在我得到了同样的 jdbc 旧错误。我真的无法弄清楚发生了什么：

[root@fiware fiware]# /usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_phoenix.conf -n cygnus-ngsi -Dflume.root.logger=INFO,console
Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
+ exec /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -Xms512m -Xmx1g -Dcom.sun.management.jmxremote -Dflume.root.logger=INFO,console -cp '/usr/cygnus/conf:/usr/cygnus/lib/*:/usr/cygnus/plugins.d/phoenix-sink/:/usr/cygnus/plugins.d/phoenix-sink/lib/*:/usr/cygnus/plugins.d/phoenix-sink/libext/*' -Djava.library.path= com.telefonica.iot.cygnus.nodes.CygnusApplication -f /usr/cygnus/conf/agent_phoenix.conf -n cygnus-ngsi
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/phoenix-sink/lib/cygnus-ngsi-1.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
time=2017-10-31T06:59:24.751Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp= | op=main | msg=com.telefonica.iot.cygnus.nodes.CygnusApplication[166] : Starting Cygnus, version 1.7.1.UNKNOWN
time=2017-10-31T06:59:24.897Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=main | msg=com.telefonica.iot.cygnus.nodes.CygnusApplication[289] : Waiting for valid Flume components references...
time=2017-10-31T06:59:24.900Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider[61] : Configuration provider starting
time=2017-10-31T06:59:24.909Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=run | msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[133] : Reloading configuration file:/usr/cygnus/conf/agent_phoenix.conf
time=2017-10-31T06:59:24.915Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.915Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.916Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.917Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.917Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[1016] : Processing:phoenix-sink
time=2017-10-31T06:59:24.917Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=addProperty | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[930] : Added sinks: phoenix-sink Agent: cygnus-ngsi
time=2017-10-31T06:59:24.917Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=isValid | msg=org.apache.flume.conf.FlumeConfiguration$AgentConfiguration[319] : Agent configuration for 'c' does not contain any channels. Marking it as invalid.
time=2017-10-31T06:59:24.919Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=validateConfiguration | msg=org.apache.flume.conf.FlumeConfiguration[127] : Agent configuration invalid for agent 'c'. It will be removed.
time=2017-10-31T06:59:24.934Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=validateConfiguration | msg=org.apache.flume.conf.FlumeConfiguration[140] : Post-validation flume configuration contains configuration for agents: [cygnus-ngsi]
time=2017-10-31T06:59:24.934Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=loadChannels | msg=org.apache.flume.node.AbstractConfigurationProvider[150] : Creating channels
time=2017-10-31T06:59:24.944Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=create | msg=org.apache.flume.channel.DefaultChannelFactory[40] : Creating instance of channel memoryChannel type com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
time=2017-10-31T06:59:24.949Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=loadChannels | msg=org.apache.flume.node.AbstractConfigurationProvider[205] : Created channel memoryChannel
time=2017-10-31T06:59:24.951Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=create | msg=org.apache.flume.source.DefaultSourceFactory[39] : Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource
time=2017-10-31T06:59:24.979Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=configure | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[156] : [NGSIRestHandler] Startup completed
time=2017-10-31T06:59:24.994Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=create | msg=org.apache.flume.sink.DefaultSinkFactory[40] : Creating instance of sink: phoenix-sink, type: org.apache.phoenix.flume.sink.PhoenixSink
time=2017-10-31T06:59:25.247Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=getConfiguration | msg=org.apache.flume.node.AbstractConfigurationProvider[119] : Channel memoryChannel connected to [http-source, phoenix-sink]
time=2017-10-31T06:59:25.255Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=startAllComponents | msg=org.apache.flume.node.Application[138] : Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{name:http-source,state:IDLE} }} sinkRunners:{phoenix-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6be236d0 counterGroup:{ name:null counters:{} } }} channels:{memoryChannel=com.telefonica.iot.cygnus.channels.CygnusMemoryChannel{name: memoryChannel}} }
time=2017-10-31T06:59:25.256Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=startAllComponents | msg=org.apache.flume.node.Application[145] : Starting Channel memoryChannel
time=2017-10-31T06:59:25.258Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=register | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[110] : Monitoried counter group for type: CHANNEL, name: memoryChannel, registered successfully.
time=2017-10-31T06:59:25.259Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component type: CHANNEL, name: memoryChannel started
time=2017-10-31T06:59:25.259Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=startAllComponents | msg=org.apache.flume.node.Application[173] : Starting Sink phoenix-sink
time=2017-10-31T06:59:25.261Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.phoenix.flume.sink.PhoenixSink[116] : Starting sink Phoenix Sink__1 
time=2017-10-31T06:59:25.268Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=register | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[110] : Monitoried counter group for type: SINK, name: Phoenix Sink__1, registered successfully.
time=2017-10-31T06:59:25.268Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=startAllComponents | msg=org.apache.flume.node.Application[184] : Starting Source http-source
time=2017-10-31T06:59:25.268Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component type: SINK, name: Phoenix Sink__1 started
time=2017-10-31T06:59:25.273Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=<init> | msg=com.telefonica.iot.cygnus.interceptors.CygnusGroupingRules[49] : No grouping rules have been read. Details: /opt/apache-flume/conf/grouping_rules.conf (No such file or directory)
time=2017-10-31T06:59:25.287Z | lvl=WARN | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=<init> | msg=com.telefonica.iot.cygnus.interceptors.CygnusGroupingRules[49] : No grouping rules have been read. Details: /opt/apache-flume/conf/grouping_rules.conf (No such file or directory)
time=2017-10-31T06:59:25.412Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=register | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[110] : Monitoried counter group for type: SOURCE, name: http-source, registered successfully.
time=2017-10-31T06:59:25.412Z | lvl=INFO | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component type: SOURCE, name: http-source started
time=2017-10-31T06:59:25.460Z | lvl=ERROR | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=initialize | msg=org.apache.phoenix.flume.serializer.BaseEventSerializer[200] : error No suitable driver found for jdbc:phoenix:localhost; occurred during initializing connection 
time=2017-10-31T06:59:25.460Z | lvl=ERROR | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=start | msg=org.apache.phoenix.flume.sink.PhoenixSink[123] : Error No suitable driver found for jdbc:phoenix:localhost; in initializing the serializer.
time=2017-10-31T06:59:25.462Z | lvl=ERROR | corr= | trans= | srv= | subsrv= | comp=cygnus-ngsi | op=run | msg=org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable[253] : Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6be236d0 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at com.google.common.base.Throwables.propagate(Throwables.java:156)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:124)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.phoenix.flume.serializer.BaseEventSerializer.initialize(BaseEventSerializer.java:140)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:119)
... 10 more

有人可以解释一下吗？

fiware - 制作 Fiware-Cygnus 存储到 Apache-Phoenix 数据库

0 回答 0

Related

Reference