HTTP-прокси Camel-Jetty + проблема с большими данными ответа: превышена емкость буфера

Эксперт по пристани, у меня возникла ошибка при возврате больших данных через верблюжью пристань в качестве http-прокси, любой опыт/идея/предложение приветствуются, заранее спасибо. Код рутбилдера:

from("servlet:balancer?matchOnUriPrefix=true")
.to("jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?" + "bridgeEndpoint=true"
                                            + "&disableStreamCache=true"
                                            + "&throwExceptionOnFailure=false" 
                                            + "&bufferSize=1024000000"
                                            + "&inboundBufferSize=1024000000"
                                            + "&outboundBufferSize=1024000000"
                                            + "&eagerCheckContentAvailable=true"
                                            + "&httpClient.responseBufferSize=1024000000&responseBufferSize=1024000000&chunked=false")

ENV:

<jdk.version>1.8</jdk.version>
<camel.version>2.17.0</camel.version>
<spring.version>4.2.4.RELEASE</spring.version>
<spring.security.version>4.0.4.RELEASE</spring.security.version>

Получил ошибку, когда служба отдыха на основе http возвращает 5000 записей данных json, я пробовал много способов настроить размер буфера ответа, к сожалению, все еще не работает, он работает, когда сторона службы возвращает ‹1000 записей:

    [2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e
DirectByteBufferR@61e4dcc8[p=40262,l=47776,c=1024000000,r=7514]={umn12":"data4782"...4875","\r\n2000\r\n<<<column10":"data48...16":"data4892",>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1232] - parseNext s=CHUNK DirectByteBuffer@26a1b8da[p=47776,l=47776,c=1024000000,r=0]={umn12":"data4782"...16":"data4892",<<<>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint                         142] - filled 31280 SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,3/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        113] - Read 31280 bytes DirectByteBuffer@26a1b8da[p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} from SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,0/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1232] - parseNext s=CHUNK DirectByteBuffer@26a1b8da[p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e
DirectByteBufferR@5991d3e4[p=0,l=678,c=1024000000,r=678]={<<<"column17":"data4...a4894","column8>>>\r\n2000\r\n":"data48...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNK --> CHUNKED_CONTENT 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNKED_CONTENT --> CHUNK_SIZE 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNK_SIZE --> CHUNK 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e
DirectByteBufferR@2b0b1218[p=686,l=8878,c=1024000000,r=8192]={"column17":"data4...column8\r\n2000\r\n<<<":"data4894","col..."column6":"data>>>\r\n2000\r\n4913","co...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange                        228] - Failed HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497: req=false/rsp=true java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1555] - close HttpParser{s=CHUNK,2105344 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser                            1587] - CHUNK --> CLOSED 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        514] - Response failure HttpResponse[HTTP/1.1 200 OK]@54e3183e HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497 on HttpChannelOverHTTP@4b2502b4(exchange=HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]: java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        529] - Concurrent failure: response termination skipped, performed by helpers 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange                        259] - Aborted (true) while active HttpExchange@2e196e3f req=TERMINATED/null@null res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497: java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        541] - State update failed: TRANSIENT -> CONTENT: FAILURE 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange                        210] - Terminated response for HttpExchange@2e196e3f req=TERMINATED/null@null res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497, result: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]@66fc8f65 > HttpResponse[HTTP/1.1 200 OK]@54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        442] - Response complete HttpResponse[HTTP/1.1 200 OK]@54e3183e 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpChannel                         89] - HttpExchange@2e196e3f req=TERMINATED/null@null res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity exceeded@35d21497 disassociated true from HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.ConnectionPool                      257] - Connection removed HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] - pooled: 0 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint                         75] - oshut SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP         154] - HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] oshut 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint                        108] - onClose SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint                         115] - close SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.SelectorManager                         756] - Destroyed SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractConnection                      210] - onClose HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint                        108] - onClose SelectChannelEndPoint@7283e6d8{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP         157] - HttpConnectionOverHTTP@383f74f1(l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[HttpChannelOverHTTP@4b2502b4(exchange=null)[send=HttpSenderOverHTTP@325ed6d0(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP@4fc1b6a4(rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] closed 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        450] - Request/Response failed: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]@66fc8f65 > HttpResponse[HTTP/1.1 200 OK]@54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.DefaultErrorHandler               153] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). On delivery attempt: 0 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.Pipeline                          78] - Message exchange has failed: so breaking out of pipeline for exchange: Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8] Exception: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 
[2016/04/11 18:32:09 CST] [ERROR] [org.apache.camel.processor.DefaultErrorHandler               204] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]

Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
[route1            ] [route1            ] [servlet:/balancer?matchOnUriPrefix=true                                       ] [      3135]
[route1            ] [policy1           ] [policy[null]                                                                  ] [      3136]
[route1            ] [to1               ] [jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?bridgeEndpoint=true&d] [      3133]

Stacktrace
--------------------------------------------------------------------------------------------------------------------------------------- 
org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded]
    at org.apache.camel.component.jetty9.JettyContentExchange9.doTaskCompleted(JettyContentExchange9.java:155)
    at org.apache.camel.component.jetty9.JettyContentExchange9$2.onComplete(JettyContentExchange9.java:221)
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193)
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:185)
    at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:453)
    at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:434)
    at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:360)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:254)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:38)
    at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1499)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1272)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:156)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:117)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
    at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:89)
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:122)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Buffering capacity exceeded
    at org.eclipse.jetty.client.util.BufferingResponseListener.onContent(BufferingResponseListener.java:114)
    at org.eclipse.jetty.client.api.Response$Listener$Adapter.onContent(Response.java:245)
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:124)
    at org.eclipse.jetty.client.ResponseNotifier.access$100(ResponseNotifier.java:35)
    at org.eclipse.jetty.client.ResponseNotifier$ContentCallback.process(ResponseNotifier.java:274)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:117)
    at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:322)
    ... 13 more
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver                        158] - Parsed true, remaining 22402 HttpParser{s=CLOSED,2105344 of -1}

http-proxy apache-camel jetty-9
person Lori Sun    schedule 11.04.2016    source источник
comment
Я не знаю, как заставить Jetty ничего не буферизировать. Не уверен, что это возможно. Можете ли вы попробовать компонент http4 в качестве производителя вашего прокси-маршрута? В Camel 2.17.0 есть дополнительное преимущество: не кэшируется поток ответа, если кэширование потока отключено.   -  person Ralf    schedule 12.04.2016
comment
По какой-то другой причине мне нужно использовать верблюжью пристань, в любом случае, спасибо, я собираюсь выяснить, как ничего не кешировать, это первый выбор :), изменить размер буфера кеша ответов как вторичный...   -  person Lori Sun    schedule 12.04.2016

Ответы (1)


arrow_upward
3
arrow_downward

Причина в том, что у camel-jetty9-2.17.0.jar есть ограничение, org.apache.camel.component.jetty9.JettyContentExchange9#send

BufferingResponseListener responseListener = new BufferingResponseListener() {

    @Override
    public void onComplete(Result result) {
        if (result.isFailed()) {
            doTaskCompleted(result.getFailure());
        } else {
            onResponseComplete(result, getContent(), getMediaType());
        }
    }
};

В BufferingResponseListener по умолчанию maxLength составляет 2 МБ (2 * 1024 * 1024). И конструктор с одним параметром: BufferingResponseListener(int maxLength) - это единственный способ настроить maxLength. но верблюд использует конструктор по умолчанию, поэтому все ответы размером более 2 МБ получат такую ​​​​ошибку.

person Lori Sun    schedule 14.04.2016