具有并发多线程请求的Java OKHTTP SocketTimeoutException

Question

我的Java应用程序使用各种代理服务器从特定域收集数据。特别申请需要下列程序：

• 通过特定代理加载URL
• 等待5秒
• 通过同一个代理加载下一个url

为了使信息的加载（由于5秒的暂停）不需要永远，我总共使用400个线程。每个线程都使用自己的代理服务器，也就是使用自己的OKHTTP客户端：

MyHTTPClient = new OkHttpClient.Builder()
.connectTimeout(7, TimeUnit.SECONDS)
.writeTimeout(7, TimeUnit.SECONDS)
.readTimeout(7, TimeUnit.SECONDS)
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ProxyIP, ProxyPort)))
.proxyAuthenticator((Route route, Response response) - > {
    return response.request().newBuilder().header("Proxy-Authorization", Credentials.basic(ProxyUser, ProxyPass)).build();
})
.sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0])
.hostnameVerifier((String hostname, SSLSession session) - > true)
.addNetworkInterceptor(new LoggingInterceptor())
.retryOnConnectionFailure(true)
.build();

每个线程必须使用自己的代理，因此每个线程都有自己的OKHTTP客户端。总共有400个OKHTTP客户端。

 while (true) {
     MyAnswer = MyHTTPClient.newCall(
         new Request.Builder().url(https://www.example.com)
         .addHeader("referer", SomeReferrer)
         .addHeader("cache-control", "no-cache")
         .addHeader("pragma", "no-cache")
         .get().build())
     .execute();

     Body = MyAnswer.body().string();
     MyAnswer.body().close();

     Thread.sleep(5000);
 }

java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.waitForConnect(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:107)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.base/java.net.Socket.connect(Socket.java:608)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:120)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
at okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.kt:261)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:201)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)

我做了一些研究，发现在并发连接方面可能存在某种限制。我发现，无论何时发生此错误，连接池中都没有空闲连接：

MyHTTPClient.connectionPool().idleConnectionCount() //Always 0 when the Timeout-Exception occurs

因此，我尝试为每个okhttpclient增加连接池的大小，但这似乎完全没有效果：

.connectionPool(new ConnectionPool(5000, 100, TimeUnit.MILLISECONDS))

我还尝试增加和减少httpclient的超时值，这实际上似乎会影响应用程序在没有此错误的情况下运行多长时间，但实际上并没有阻止这种情况的发生。

Dispatcher MyDispatcher = new Dispatcher(Executors.newCachedThreadPool());
MyDispatcher.setMaxRequests(9999999);
MyDispatcher.setMaxRequestsPerHost(9999999);

Answer 1

首先创建一个OkHttpClient，然后使用OkHttpClient.newBuilder()自定义代理。这将导致客户端共享线程池和连接池。这可能不能解决你的问题，但它会使它更有效。

  final OkHttpClient sharedHttpClient = new OkHttpClient.Builder()
    .connectTimeout(7, TimeUnit.SECONDS)
    .writeTimeout(7, TimeUnit.SECONDS)
    .readTimeout(7, TimeUnit.SECONDS)
    .sslSocketFactory(sslSocketFactory, (X509TrustManager) trustAllCerts[0])
    .hostnameVerifier((String hostname, SSLSession session) - > true)
    .addNetworkInterceptor(new LoggingInterceptor())
    .retryOnConnectionFailure(true)
    .build();

MyHTTPClient = sharedHttpClient.newBuilder()
    .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ProxyIP, ProxyPort)))
    .proxyAuthenticator((Route route, Response response) - > {
    return response.request().newBuilder().header("Proxy-Authorization", Credentials.basic(ProxyUser, ProxyPass)).build();
})
    .build();