我正在使用 HBase 存储由 CDH4(当前为 4.5)管理的应用程序日志,升级到 cdh 4.6(与 4.7 相同)后,插入速度非常慢。我发现客户端正在连接到 regionserver 并立即关闭连接(使用 CDh 4.5 我没有遇到同样的问题)
区域服务器日志:
13:46:08,428 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ZK03:2181,ZK02:2181,ZK01:2181 sessionTimeout=60000 watcher=hconnection
13:46:08,429 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 19573@NODE01
13:46:08,429 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ZK03/10.1.243.170:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
13:46:08,429 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ZK03/10.1.243.170:2181, initiating session
13:46:08,431 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ZK03/10.1.243.170:2181, sessionid = 0x146a9fec35171f0, negotiated timeout = 60000
13:46:08,538 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x146a9fec35171f0
13:46:08,540 INFO org.apache.zookeeper.ZooKeeper: Session: 0x146a9fec35171f0 closed
13:46:08,540 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
客户端连接类:
private void initConnection(Configuration hConf) throws RuntimeException {
try {
//HConnectionManager.create(hConf);
hConnection = HConnectionManager.createConnection(hConf);
} catch (ZooKeeperConnectionException e) {
logAndThrow("Failed to init connection " + e.getMessage());
}
}
public Connection(Configuration hConf) {
initConnection(hConf);
}
public void closeConnection() throws IOException {
hConnection.close();
}
public HTableInterface getHTableInterface(String tableName) throws IOException {
HTableInterface htable = hConnection.getTable(tableName);
htable.setAutoFlush(false, true);
htable.setWriteBufferSize(1024*1024*12);
return htable;
}
进口:
Put put = new Put(rowKey.get(), tsWhole);
mainTableBuffer.add(put);
if(cfg_.maxBatchBufferSize <= mainTableBuffer.size()) {
mainTableInterface_.batch(mainTableBuffer);
mainTableBuffer.clear();
}