我们正在使用 elasticsearch REST Java 客户端(我们使用的是 Java 7,因此无法使用普通的 elasticsearch Java 客户端)与我们的 elasticsearch 服务器进行交互。除了我们尝试对大约 130 万个文档进行初始索引时,这一切都很好。这运行了一段时间,但在几十万个文件之后,我们得到了一个
20/06 21:27:33,153 ERROR [cid=51][stderr][write:71] (pool-837116-thread-1) Exception in thread "pool-837116-thread-1" java.lang.OutOfMemoryError: unable to create new native thread
20/06 21:27:33,154 ERROR [cid=51][stderr][write:71] (pool-837116-thread-1) at java.lang.Thread.start0(Native Method)
20/06 21:27:33,154 ERROR [cid=51][stderr][write:71] (pool-837116-thread-1) at java.lang.Thread.start(Thread.java:693)
20/06 21:27:33,154 ERROR [cid=51][stderr][write:71] (pool-837116-thread-1) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:334)
20/06 21:27:33,154 ERROR [cid=51][stderr][write:71] (pool-837116-thread-1) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
20/06 21:27:33,154 ERROR [cid=51][stderr][write:71] (pool-837116-thread-1) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
20/06 21:27:33,155 ERROR [cid=51][stderr][write:71] (pool-837116-thread-1) at java.lang.Thread.run(Thread.java:724)
其次是
java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:343)
at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:325)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191)
如您所见,Elasticsearch REST 客户端正在使用 apache http nio。我发现奇怪的是 nio 库正在为每个请求(或连接?)创建一个线程。从上面的日志中您可以看到线程 (pool-837116-thread-1)。还有很多 I/O 调度程序线程,数量不断增加。
不过,活动线程的总数似乎没有太大变化。因此,似乎不是重用线程,而是为每个连接周期创建一个(或两个实际上)新线程。上传基本上是:
1.创建客户端
restClient = RestClient.builder(new HttpHost(host.getHost(),host.getPort(),host.getProtocol())/*,new HttpHost(host.getHost(),host.getPort()+1,host.getProtocol())*/)
.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder
.setDefaultCredentialsProvider(credsProvider)
}
}).setMaxRetryTimeoutMillis(30000).build();
2. 发送带有json body的请求并关闭客户端
try{
HttpEntity entity = new NStringEntity(json,ContentType.APPLICATION_JSON);
Response indexResponse = restClient.performRequest("PUT", endpoint, parameters,entity,header);
log.debug("Response #0 #1", indexResponse,indexResponse.getStatusLine());
log.debug("Entity #0",indexResponse.getEntity());
}finally{
if(restClient!=null){
log.debug("Closing restClient #0", restClient);
restClient.close();
}
}
这是正常的吗?为什么 apache nio 不重用线程?这是弹性搜索 REST 客户端、apache nio 还是我的代码的问题?我打电话给restClient,不知道我还应该做什么。
我试图在 IO Reactor 上将线程数设置为 1:
restClient = RestClient.builder(new HttpHost(host.getHost(),host.getPort(),host.getProtocol())/*,new HttpHost(host.getHost(),host.getPort()+1,host.getProtocol())*/)
.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder
.setDefaultCredentialsProvider(credsProvider)
.setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(1).build()); //set to one thread
}
}).setMaxRetryTimeoutMillis(30000).build();
但这并没有改变关于线程重用的任何内容。