1

从 Google Storage 下载数据时，我收到大量 SocketTimeoutException。增加超时的最佳方法是什么？

用例

使用 Map/Reduce 从多个服务器并行下载 24 个文件（总共 10GB）。

源代码

  Storage storage = new Storage.Builder(
                new NetHttpTransport(),
                new JacksonFactory(),
                new GoogleCredential().setAccessToken(accessToken))
            .setApplicationName("FooBar")
            .build();

  Storage.Objects.Get getObject = storage.objects().get(bucket, fn);
  getObject.getMediaHttpDownloader().setDirectDownloadEnabled(true);
  getObject.executeMediaAndDownloadTo(outputStream);

堆栈跟踪

java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    at sun.net.www.MeteredStream.read(MeteredStream.java:134)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3052)
    at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3046)
    at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:51)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:63)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:261)
    at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:209)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:553)
    at com.google.api.services.storage.Storage$Objects$Get.executeMediaAndDownloadTo(Storage.java:4494)

编辑

这只发生在我从日本下载数据时。美国东部和美国西部的一切都很好。

4

2 回答 2

0

寻找一个名为“CommandTimeout”的变量，或者类似的东西。通常这决定了连接在标记错误之前可以持续多长时间。

“ConnectionTimeout”通常意味着程序可以尝试连接到您的源多长时间，这并不是您真正需要更改的，因为连接发生得很快，下载速度很慢。

于 2013-09-25T15:39:42.930 回答
0

尝试以下...

  1. 在 linux 框中设置代理。

  2. 在创建存储时 - new NetHttpTransport() 提供您自己的方式来创建套接字并在传输中添加代理。

    public HttpClient myHttpClient() throws Exception {
    SchemeRegistry schemeRegistry = new SchemeRegistry();
    //SetRegisrty for both HTTP and HTTPS - Check google for this.
    schemeRegistry.register(new Scheme("http", PlainSocketFactory
            .getSocketFactory(), YOUR_PROXY_PORT));
    schemeRegistry.register(new Scheme("https", SSLSocketFactory
            .getSocketFactory(), 443));

    HttpParams params = new BasicHttpParams();
    HttpConnectionParams.setConnectionTimeout(params, 30 * 1000);  // SET the timeout
    HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
    ClientConnectionManager connManager = new ThreadSafeClientConnManager(
            params, schemeRegistry);
    DefaultHttpClient httpClient = new DefaultHttpClient(connManager,
                params);
    try {
        int proxyPort = YOUR_PROXY_PORT;
        String proxyHost = "YOUR_PROXT_HOST_NAME";
        if (proxyPort > 0 && proxyHost != null && proxyHost.length() > 0) {
            System.setProperty("https.proxyHost", proxyHost);
            System.setProperty("https.proxyPort", proxyPort + "");
            System.setProperty("http.proxyHost", proxyHost);
            System.setProperty("http.proxyPort", proxyPort + "");
            HttpHost proxy = new HttpHost(proxyHost, proxyPort);
            httpClient.getParams().setParameter(
                    ConnRoutePNames.DEFAULT_PROXY, proxy);
        }
    } catch (NullPointerException e) {
        System.out.println("Proxy error here");
    }
    return httpClient;
}

public static HttpTransport myNetHttpTransport()
        throws Exception {
    return new ApacheHttpTransport(myHttpClient());
}

使用.setTransport(myNetHttpTransport())而不是NetHttpTransport().

我们在这上面花了很长时间。但截至目前，这似乎正在奏效。请让我知道是否有任何帮助...

于 2014-11-04T13:21:22.777 回答