Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gMaiu-0001MQ-2N for pgsql-admin@arkaria.postgresql.org; Tue, 13 Nov 2018 15:35:28 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gMair-0007Dj-Up for pgsql-admin@arkaria.postgresql.org; Tue, 13 Nov 2018 15:35:25 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gMair-0007DB-Mt for pgsql-admin@lists.postgresql.org; Tue, 13 Nov 2018 15:35:25 +0000 Received: from mtaout003-public.msg.strl.va.charter.net ([68.114.190.28]) by magus.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gMaip-0007xd-4S for pgsql-admin@lists.postgresql.org; Tue, 13 Nov 2018 15:35:25 +0000 Received: from impout005 ([68.114.189.20]) by mtaout003.msg.strl.va.charter.net (InterMail vM.9.00.023.01 201-2473-194) with ESMTP id <20181113153515.ZKVC7355.mtaout003.msg.strl.va.charter.net@impout005>; Tue, 13 Nov 2018 09:35:15 -0600 Received: from cerberus.crazybean.net ([24.151.10.150]) by impout005 with charter.net id zTb41y00D3EE3TL01Tb4k3; Tue, 13 Nov 2018 09:35:15 -0600 X-Authority-Analysis: v=2.2 cv=Ts+WvHfh c=1 sm=1 tr=0 a=D2HTw/M6/8MmcHighGO7rg==:117 a=D2HTw/M6/8MmcHighGO7rg==:17 a=JHtHm7312UAA:10 a=Rh5JslpUAAAA:8 a=IG59EEACG4OzhSmbteQA:9 a=QEXdDO2ut3YA:10 a=tbR47KEUwDWZ4YwL:21 a=_W_S_7VecoQA:10 a=CRjGpuKUiiIuQkCM3vJl:22 Received: from [10.25.227.242] (144.121.139.193.lightower.net [144.121.139.193]) by cerberus.crazybean.net (OpenSMTPD) with ESMTPSA id ade33576 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Tue, 13 Nov 2018 10:35:04 -0500 (EST) From: Rui DeSousa Message-Id: <0BBC4BB3-4A59-4040-B065-82D09268A4F7@crazybean.net> Content-Type: multipart/alternative; boundary="Apple-Mail=_19AE5EBF-5EBB-4AB7-8E85-CB39E21B7D83" Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: PostgreSQL 10.5 : Logical replication timeout results in PANIC in pg_wal "No space left on device" Date: Tue, 13 Nov 2018 10:35:03 -0500 In-Reply-To: <838a3914-a078-88eb-ee5e-4059b7d14620@matrix.gatewaynet.com> Cc: pgsql-admin@lists.postgresql.org To: Achilleas Mantzios References: <838a3914-a078-88eb-ee5e-4059b7d14620@matrix.gatewaynet.com> X-Mailer: Apple Mail (2.3445.101.1) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --Apple-Mail=_19AE5EBF-5EBB-4AB7-8E85-CB39E21B7D83 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Nov 13, 2018, at 7:00 AM, Achilleas Mantzios = wrote: >=20 > Is there a way for the WAL receiver to not have detected the = termination of the replication stream? The teardown of the network socket on the upstream server should send a = reset packet to the downstream server and at that point the WAL receiver = would close its connection. Is there any firewalls, router, rules, etc = between the nodes that could have dropped the packet?=20 >=20 > Shouldn't normally the WAL receiver detect this and try again in = wal_retrieve_retry_interval ? Not really=E2=80=A6 if the connection has already been torn down; the = upstream server would send another reset packet on the next request and = in this case it would. However, if request packets at not reaching the = upstream server; i.e. due to firewall silently dropping the packets = (personally I believe firewall should always set reset packets to = friendly hosts) then what happens is the TCP/IP send queue builds up = with the requests packets instead =E2=80=94 a t this point waiting on = the OS to terminate the connection which can day or two depending on = your TCP/IP setting. What you want to use instead is wal_receiver_timeout to detect the given = case where upstream server either no longer exists or the firewall, etc = is silently dropping packets. --Apple-Mail=_19AE5EBF-5EBB-4AB7-8E85-CB39E21B7D83 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On Nov 13, 2018, at 7:00 AM, Achilleas Mantzios <achill@matrix.gatewaynet.com> wrote:

Is there a = way for the WAL receiver to not have detected the termination of the = replication stream?

The teardown of the network socket on the = upstream server should send a reset packet to the downstream server and = at that point the WAL receiver would close its connection.  Is = there any firewalls, router, rules, etc between the nodes that could = have dropped the packet? 


Shouldn't normally the WAL receiver detect this and try = again in wal_retrieve_retry_interval ?

Not really=E2=80=A6 if the connection = has already been torn down; the upstream server would send another reset = packet on the next request and in this case it would.  However, if = request packets at not reaching the upstream server; i.e. due to = firewall silently dropping the packets (personally I believe firewall = should always set reset packets to friendly hosts) then what happens is = the TCP/IP send queue builds up with the requests packets instead =E2=80=94= a t this point waiting on the OS to terminate the connection which can = day or two depending on your TCP/IP setting.

What you want to use instead is = wal_receiver_timeout to detect the given case where upstream server = either no longer exists or the firewall, etc is silently dropping = packets.


= --Apple-Mail=_19AE5EBF-5EBB-4AB7-8E85-CB39E21B7D83--