0

运行命令时碰巧出现此错误: flume-ng agent -n TwitterAgent -f $FLUME_HOME/flume.conf

错误是: 22/01/01 14:02:27 错误 node.PollingPropertiesFileConfigurationProvider: 未处理的错误 java.lang.NoSuchMethodError: 'java.util.Map twitter4j.conf.Configuration.getRequestHeaders()' at twitter4j.StreamingReadTimeoutConfiguration.getRequestHeaders(TwitterStreamImpl.java: 664) 在 twitter4j.internal.http.HttpClientWrapper.(HttpClientWrapper.java:47) 在 twitter4j.TwitterStreamImpl.(TwitterStreamImpl.java:54) 在 twitter4j.TwitterStreamFactory.(TwitterStreamFactory.java:40) 在 org.apache.flume.source .twitter.TwitterSource.configure(TwitterSource.java:110) 在 org.apache.flume.conf.Configurables.configure(Configurables.java:41) 在 org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:325 ) 在 org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:105) 在 org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:145) 在 java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 在 java.base /java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) 在 java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) 在 java.base/java.util.concurrent .ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 在 java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 在 java.base/java.lang.Thread.run(Thread.java :829)Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run( ScheduledThreadPoolExecutor.java:305) 在 java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 在 java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)在 java.base/java.lang.Thread.run(Thread.java:829)Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run( ScheduledThreadPoolExecutor.java:305) 在 java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 在 java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)在 java.base/java.lang.Thread.run(Thread.java:829)ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 在 java.base/java.lang.Thread.run(Thread.java:829)ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 在 java.base/java.lang.Thread.run(Thread.java:829)

这是我的 Flume.conf:

 TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = *************
TwitterAgent.sources.Twitter.consumerSecret = ***********
TwitterAgent.sources.Twitter.accessToken = **************
TwitterAgent.sources.Twitter.accessTokenSecret = *****************

TwitterAgent.sources.Twitter.keywords = apple,iphone,review

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume/tweets/

TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

我该如何解决我使用 Hadoop 2.9.0 和 Flume 1.9.x 以后的错误。我还包含了 twitter4j-4.7.0 流、核心和异步文件,但发现它不起作用。我还在 flume/lib 文件夹中包含了 flume-sources-1.0-SNAPSHOT.jar

而flume-env.sh是这样的:

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.

# Enviroment variables can be set here.
 **export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64**

# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
# export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"

# Let Flume write raw event data and configuration information to its log files for debugging
# purposes. Enabling these flags is not recommended in production,
# as it may result in logging sensitive user information or encryption secrets.
# export JAVA_OPTS="$JAVA_OPTS -Dorg.apache.flume.log.rawdata=true -Dorg.apache.flume.log.printconfig=true "

# Note that the Flume conf directory is always included in the classpa
**export FLUME_CLASSPATH="/usr/local/flume/lib"
export CLASSPATH=$CLASSPATH:/FLUME_HOME/lib/*
export twitter4j.streamBaseURL=https://stream.twitter.com/1.1/**

4

0 回答 0