Subject: Re: CLOSE_WAIT pileup and Application Timeout

From: Adrian Klaver <[email-masked]>
Date: Fri, 4 Oct 2024 08:47:11 -0700

On 10/3/24 21:29, KK CHN wrote:
> List,
> 
> I am facing a  network (TCP IP connection closing issue) .
> 
> Running a  mobile tablet application, Android application to update the 
> status of vehicles fleet say around 1000 numbers installed with the app 
> on each vehicle along  with a  vehicle tracking  application server 
> solution based on Java and Wildfly with  PosrgreSQL16 backend.
> 

> 
> The  running vehicles may disconnect  or be unable to send the location 
> data in between if the mobile data coverage is less or absent in a 
> particular area where data coverage is nil or signal strength less.
> 
> The server on which the backend application runs most often ( a week's 
> time  or so) shows connection timeout and is unable to serve tracking  
> of  the vehicles further.
> 
> When we restart the  Wildfly server  the application returns to normal.  
> again the issue repeats  after a week or two.

Seems the issue is in the application server. What is not clear to me is 
whether the connection timeout you refer to is from the mobile devices 
to the application or the application to the Postgres server? I'm 
guessing the latter as I would expect the mobile devices to drop 
connections more often then weekly.
> 
> In the Server machine when this bottleneck occurs  I am seeing  a lot 
> of  TCP/IP CLOSE_WAIT   ( 3000 to 5000 ) when the server backend becomes 
> unresponsive.

Again not clear, are you referring to the application or the Postgres 
database running on the server?

> 
> What is the root cause of this issue ?   Is it due to the android 
> application unable to send the CLOSE_WAIT ACK due to poor mobile data 
> connectivity ?
> 
> 
>   If so, how do people  address this issue ?  and what may be a fix ?
> 
>   Any  directions / or reference material most welcome.
> 
> Thank you,
> Krishane
> 
> 
> 
> 
> 

-- 
Adrian Klaver
[email-masked]

---

From: KK CHN <[email-masked]>
Date: Sun, 6 Oct 2024 18:56:11 +0530

On Fri, Oct 4, 2024 at 9:17 PM Adrian Klaver <[email-masked]>
wrote:

> to the application or the application to the Postgres server?

its from mobile devices to application server.  When I do a restart of
application server everything backs to normal.  But after a period of time
again it cripples.  That time when I netstat on Application VM lots of
CLOSE_WAIT states as indicated.

> I'm

> Yes mobile devices may drops connections at any point of time if it
> reaches an area where signal strength is poor( eg; Underground parking or
> near the areas where mobile data coverage is poor.
> >

The topology is mobile devices  connect and update the location via
application VM then   finally in  PGSQL VM.

The application server and  Database server both separate virtual
machines.      Application server hangs most often not the database VM.
Since there are other applications which update to the database VM without
any issue.  The DB VM caters all the writes from other applications. But
those applications are different, not fleet management one.

---

From: Adrian Klaver <[email-masked]>
Date: Sun, 6 Oct 2024 08:52:46 -0700

On 10/6/24 06:26, KK CHN wrote:
> 
> 
> On Fri, Oct 4, 2024 at 9:17 PM Adrian Klaver <[email-masked] 

> application server everything backs to normal.  But after a period of 
> time again it cripples.  That time when I netstat on Application VM lots 
> of  CLOSE_WAIT states as indicated.
> 
>     I'm
>     reaches an area where signal strength is poor( eg; Underground
>     parking or near the areas where mobile data coverage is poor.
>      >
> 
> 
> Since there are other applications which update to the database VM 
> without any issue.  The DB VM caters all the writes from other 
> applications. But those applications are different, not fleet management 
> one.

 From what I see this really has nothing to do with the Postgres 
backend. It is a matter of communication, actually lack of 
communication, between the mobile devices and the application server. A 
broad answer is that something needs to be done to gracefully deal with 
mobile device connection drops