From: Alvaro Herrera <alvherre@2ndquadrant.com>
To: Rui DeSousa <rui@crazybean.net>
Cc: Achilleas Mantzios <achill@matrix.gatewaynet.com>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: PostgreSQL 10.5 : Logical replication timeout results in PANIC in pg_wal "No space left on device"
Date: Fri, 23 Nov 2018 08:10:24 -0300
Message-ID: <20181123111024.73ck7f7zbzexfudz@alvherre.pgsql> (raw)
In-Reply-To: <946FBD41-7BAF-4414-8FB9-C0F16A0680F6@crazybean.net>
On 2018-Nov-14, Rui DeSousa wrote:
> > On Nov 14, 2018, at 3:31 AM, Achilleas Mantzios <achill@matrix.gatewaynet.com> wrote:
> >
> > Our sysadms (seasoned linux/network guys : we have been working here
> > for more than 10 yrs) were absolute in that we run no firewall or
> > other traffic shaping system between the two hosts. (if we did the
> > problem would manifest itself earlier). Can you recommend what to
> > look for exactly regarding both TCP stacks ? The subscriber node is
> > a clone of the primary. We have :
> >
> > # sysctl -a | grep -i keepaliv
> > net.ipv4.tcp_keepalive_intvl = 75
> > net.ipv4.tcp_keepalive_probes = 9
> > net.ipv4.tcp_keepalive_time = 7200
>
> Those keep alive settings are linux’s defaults and work out to be 18
> hours before the abandon connection is dropped. So, the WAL receiver
> should have corrected itself after that time. For reference, I run
> terminating abandon session within 15 mins as they take-up valuable
> database resources and could potentially hold on to locks, snapshots,
> etc.
Where does your 18h figure come from? As I understand it, these numbers
mean "wait 7200 seconds, then send 9 probes 75 seconds apart", kill the
connection if not reply is obtained. So that works out to about 131
minutes (modulo fencepost bug). Certainly not 18 hours ...
Now ... I have seen Linux kernel code that seemed to me to cause network
transmission get stuck *in the kernel* without any way out. Now I'm not
a kernel expert and I don't know if this applies to your case (maybe it
got fixed already), but it was definitely some process that was stuck
with "wchan" set to a network kernel call and way beyond TCP keepalives.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: alvherre@2ndquadrant.com, rui@crazybean.net, achill@matrix.gatewaynet.com, pgsql-admin@lists.postgresql.org
Subject: Re: PostgreSQL 10.5 : Logical replication timeout results in PANIC in pg_wal "No space left on device"
In-Reply-To: <20181123111024.73ck7f7zbzexfudz@alvherre.pgsql>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox