Date: Mon, 30 Mar 2020 11:00:21 +0100
From: Rory Campbell-Lange <rory@campbell-lange.net>
To: Stefan Knecht <knecht.stefan@gmail.com>
Cc: Daniele Varrazzo <daniele.varrazzo@gmail.com>, psycopg@postgresql.org
Subject: Re: psycopg3: a first report
Message-ID: <20200330100021.GA32517@campbell-lange.net>
References: 
 <CA+mi_8ZH-feQP0+k_j7_joygE=hMZCFnaNMv8Z1KCLKB9YLb0w@mail.gmail.com>
 <CAP50yQ_Mt=RtR0sXYoazJTuarphR1Kv8A6YTevCMiv+xZeD9=Q@mail.gmail.com>
 <20200330074917.GA28993@campbell-lange.net>
 <CAP50yQ9WSB6s_rfXe2S0V-8H8ByCRNSw6ninrGdn5NgLzWZ8bA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CAP50yQ9WSB6s_rfXe2S0V-8H8ByCRNSw6ninrGdn5NgLzWZ8bA@mail.gmail.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
Precedence: bulk

On 30/03/20, Stefan Knecht (knecht.stefan@gmail.com) wrote:
> Rory, this is about established connections, not new connections - psycopg2
> already offers a connection timeout, but that is a different thing. I don't
> want to drift too far off topic - but we are already using pgbouncer, and
> the problem isn't detected by it, either. I'm not a developer but I believe
> the problem is the generic nature of some blocking socket calls, which may
> hang under some odd circumstances, and they remain hanging until some odd
> ssl timeout is reached (15 minutes+ which is a very long time for any
> application to be hanging in limbo, but more so for our own monitoring
> tools which are written in Python).

My apologies for the misunderstanding.

This sounds like the hanging tcp/ip problem one can get with
disappearing web server clients. We've had that problem with mobiles
dropping out presumably because they go out of range, and the provider
not dropping the connection because they might come back.

We deal with this in apache by having a fairly aggressive
KeepAliveTimeout parameter.

Having this problem in one's own stack sounds scary.

I'm very intrigued by Daniele's suggestion that having a per-operation
timeout on the client is achievable.

Rory