pg.ddx.io pgsql-general@postgresql.org mailing list archive
help / color / mirror / Atom feedAn I/O error occured while sending to the backend
5+ messages / 3 participants
[nested] [flat]
* An I/O error occured while sending to the backend
@ 2022-11-16 05:04 gzh <gzhcoder@126.com>
0 siblings, 1 reply; 5+ messages in thread
From: gzh @ 2022-11-16 05:04 UTC (permalink / raw)
To: pgsql-general@lists.postgresql.org
Hi,
I have developed an application using the jdbc driver and
connecting to a postgresql database .
The application mainly does the following two things:
① read data from the CSV file and insert it into the database
② perform a database query
In my application, the above ① and ② are executed asynchronously.
Everything was working fine until I increase the amount of CSV data to more than 1000.
Randomly they receive an error, the stack trace of which is below.
com.nec.neosarf.exception.NeoSarfException: org.postgresql.util.PSQLException: An I/O error occured while sending to the backend
at com.nec.neosarf.services.core.controller.DutiesDao.DutiesDaoError(DutiesDao.java:2037)
at com.nec.neosarf.services.core.controller.DutiesDao.SelectData(DutiesDao.java:459)
at com.nec.neosarf.services.core.controller.DutiesDao.SelectData(DutiesDao.java:335)
at com.nec.neosarf.services.core.controller.CoreConductor.ConductorLv3_DataAccess(CoreConductor.java:2062)
at com.nec.neosarf.services.core.controller.CoreConductor.ConductorCore(CoreConductor.java:794)
at com.nec.neosarf.services.core.controller.CoreConductor.ConductorRun(CoreConductor.java:623)
at com.nec.neosarf.services.core.controller.CoreConductor.Conductor(CoreConductor.java:356)
at com.nec.neosarf.services.core.controller.CoreBody.ConductorCreation(CoreBody.java:4817)
at com.nec.neosarf.services.core.controller.CoreBody.ActionDirectorInteface(CoreBody.java:2783)
at com.nec.neosarf.services.core.controller.CoreBody.AjaxCheck(CoreBody.java:2465)
at com.nec.neosarf.services.core.controller.CoreBody.ExecutionPageRun(CoreBody.java:1693)
at com.nec.neosarf.services.core.servlet.NeoSarf.coreServlet(NeoSarf.java:386)
at com.nec.neosarf.services.core.servlet.NeoSarf.service(NeoSarf.java:139)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:711)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:353)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313)
at com.nec.neosarf.services.core.filter.dynamic.NeoSarfFilterChain.doFilter(NeoSarfFilterChain.java:36)
at com.nec.neosarf.services.debugger.filter.DebugSynchronizedFilter.doFilter(DebugSynchronizedFilter.java:72)
at com.nec.neosarf.services.core.filter.dynamic.NeoSarfFilterChain.doFilter(NeoSarfFilterChain.java:34)
at com.nec.neosarf.services.core.filter.dynamic.DynamicRequestFilter.executeServlet(DynamicRequestFilter.java:93)
at com.nec.neosarf.services.core.filter.RequestFilter.doFilter(RequestFilter.java:450)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.postgresql.util.PSQLException: An I/O error occured while sending to the backend
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:382)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:167)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:119)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:121)
at org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:121)
at com.nec.neosarf.db.DaoBase.selectReal(DaoBase.java:393)
at com.nec.neosarf.db.DaoBase.selectDuties(DaoBase.java:328)
at com.nec.neosarf.services.core.controller.DutiesDao.SelectData(DutiesDao.java:372)
... 42 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
at org.postgresql.core.PGStream.receiveChar(PGStream.java:453)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2119)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
... 51 more
Thanks for any help.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: An I/O error occured while sending to the backend
@ 2022-11-16 06:57 Laurenz Albe <laurenz.albe@cybertec.at>
parent: gzh <gzhcoder@126.com>
0 siblings, 2 replies; 5+ messages in thread
From: Laurenz Albe @ 2022-11-16 06:57 UTC (permalink / raw)
To: gzh <gzhcoder@126.com>; pgsql-general@lists.postgresql.org
On Wed, 2022-11-16 at 13:04 +0800, gzh wrote:
> I have developed an application using the jdbc driver and
> connecting to a postgresql database .
> The application mainly does the following two things:
> ① read data from the CSV file and insert it into the database
> ② perform a database query
> In my application, the above ① and ② are executed asynchronously.
> Everything was working fine until I increase the amount of CSV data to more than 1000.
> Randomly they receive an error, the stack trace of which is below.
>
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.base/java.net.SocketInputStream.socketRead0(Native Method)
> at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
> at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
> at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
> at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
> at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
> at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
> at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454)
> at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065)
> at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
Well, set a longer socket timeout if you need to run long SQL statements, or tune
those statements to be faster.
https://jdbc.postgresql.org/documentation/use/#connection-parameters has more
information about "socketTimeout".
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: An I/O error occured while sending to the backend
@ 2022-11-16 07:37 gzh <gzhcoder@126.com>
parent: Laurenz Albe <laurenz.albe@cybertec.at>
1 sibling, 1 reply; 5+ messages in thread
From: gzh @ 2022-11-16 07:37 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: pgsql-general@lists.postgresql.org
Thank you very much for your advice.
What I don't understand is that there is no problem when executing ① alone or ② alone,
the error occurs when ① and ② are executed together .
It works well when i let the application sleep after ① for 10 seconds before executing ② .
Is ② affected by ① ? Could you tell me why?
At 2022-11-16 14:57:41, "Laurenz Albe" <laurenz.albe@cybertec.at> wrote:
>On Wed, 2022-11-16 at 13:04 +0800, gzh wrote:
>> I have developed an application using the jdbc driver and
>> connecting to a postgresql database .
>> The application mainly does the following two things:
>> ① read data from the CSV file and insert it into the database
>> ② perform a database query
>> In my application, the above ① and ② are executed asynchronously.
>> Everything was working fine until I increase the amount of CSV data to more than 1000.
>> Randomly they receive an error, the stack trace of which is below.
>>
>> Caused by: java.net.SocketTimeoutException: Read timed out
>> at java.base/java.net.SocketInputStream.socketRead0(Native Method)
>> at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
>> at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
>> at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
>> at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
>> at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
>> at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
>> at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454)
>> at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065)
>> at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
>
>Well, set a longer socket timeout if you need to run long SQL statements, or tune
>those statements to be faster.
>
>https://jdbc.postgresql.org/documentation/use/#connection-parameters has more
>information about "socketTimeout".
>
>Yours,
>Laurenz Albe
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: An I/O error occured while sending to the backend
@ 2022-11-16 14:23 Rob Sargent <robjsargent@gmail.com>
parent: gzh <gzhcoder@126.com>
0 siblings, 0 replies; 5+ messages in thread
From: Rob Sargent @ 2022-11-16 14:23 UTC (permalink / raw)
To: pgsql-general@lists.postgresql.org
On 11/16/22 00:37, gzh wrote:
> Thank you very much for your advice.
>
> What I don't understand is that there is no problem when executing
> ① alone or ② alone,
> the error occurs when ① and ② are executed together .
> It works well when i let the application sleep after ① for 10 seconds
> before executing ② .
>
Is this true for large numbers of lines in you csv? Originally you said
something like 'adding 1000 lines' broke your app. Are you using
CopyManager?
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: An I/O error occured while sending to the backend
@ 2022-11-18 03:25 gzh <gzhcoder@126.com>
parent: Laurenz Albe <laurenz.albe@cybertec.at>
1 sibling, 0 replies; 5+ messages in thread
From: gzh @ 2022-11-18 03:25 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: pgsql-general@lists.postgresql.org
Hi, Rob
>Is this true for large numbers of lines in you csv? Originally you said
>something like 'adding 1000 lines' broke your app. Are you using
>CopyManager?
Thank you for your reply. I didn't use CopyManager.
②'s query statement query the data inserted by ①, and a large amount of data inserted will cause performance problem.
I misunderstood, it's not ① that affects ②, it should be that ②'s query statement has a performance problem.
The Encrypt function is used in ②'s query statement to encrypt the query result, and improper use of this function leads to poor performance.
Sometimes 2~4 seconds to return the result, sometimes 7~19 seconds to return the result.
We optimized the query statement to reduce the query time to less than 1 second and now the application works well.
At 2022-11-16 14:57:41, "Laurenz Albe" <laurenz.albe@cybertec.at> wrote:
>On Wed, 2022-11-16 at 13:04 +0800, gzh wrote:
>> I have developed an application using the jdbc driver and
>> connecting to a postgresql database .
>> The application mainly does the following two things:
>> ① read data from the CSV file and insert it into the database
>> ② perform a database query
>> In my application, the above ① and ② are executed asynchronously.
>> Everything was working fine until I increase the amount of CSV data to more than 1000.
>> Randomly they receive an error, the stack trace of which is below.
>>
>> Caused by: java.net.SocketTimeoutException: Read timed out
>> at java.base/java.net.SocketInputStream.socketRead0(Native Method)
>> at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
>> at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
>> at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
>> at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
>> at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
>> at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
>> at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454)
>> at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065)
>> at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
>
>Well, set a longer socket timeout if you need to run long SQL statements, or tune
>those statements to be faster.
>
>https://jdbc.postgresql.org/documentation/use/#connection-parameters has more
>information about "socketTimeout".
>
>Yours,
>Laurenz Albe
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2022-11-18 03:25 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-11-16 05:04 An I/O error occured while sending to the backend gzh <gzhcoder@126.com>
2022-11-16 06:57 ` Laurenz Albe <laurenz.albe@cybertec.at>
2022-11-16 07:37 ` gzh <gzhcoder@126.com>
2022-11-16 14:23 ` Rob Sargent <robjsargent@gmail.com>
2022-11-18 03:25 ` gzh <gzhcoder@126.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox