Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1gIyNc-0008BT-PL for pgsql-general@arkaria.postgresql.org; Sat, 03 Nov 2018 16:02:33 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gIyNZ-0002RM-KQ for pgsql-general@arkaria.postgresql.org; Sat, 03 Nov 2018 16:02:29 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1gIyNY-0002R5-Mj for pgsql-general@lists.postgresql.org; Sat, 03 Nov 2018 16:02:29 +0000 Received: from resqmta-ch2-10v.sys.comcast.net ([2001:558:fe21:29:69:252:207:42]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1gIyNQ-0001K8-UZ for pgsql-general@lists.postgresql.org; Sat, 03 Nov 2018 16:02:26 +0000 Received: from resomta-ch2-08v.sys.comcast.net ([69.252.207.104]) by resqmta-ch2-10v.sys.comcast.net with ESMTP id IwjSgIuXkzXYcIyNOgZj3Q; Sat, 03 Nov 2018 16:02:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20161114; t=1541260938; bh=eMuR42pw4WM8PbjLWUgHk5VLtm/uXlOT8KTFEpIt6+E=; h=Received:Received:Received:From:To:Subject:Date:Message-ID: MIME-Version:Content-Type; b=WVP3GwodFR+WtjTLeUPQcJEQDogZCNTdvtIfSRpZ6sT//Kqgd8g8l+0KOFDvl7B1D eVdt6Qq/xN0Po57BgaWqIFMLlRI+MY4g3VeLO6iuhX/BIl/VwgYxVsd9HgCv1Yn5wA B0e2xMzuSf6WOFJgWYwus0eZlaWjj04HM+rRr0Ta44WGluliZZfHKn7aufyae+Uopm /IX2J2YAI40EP/o2fn8/SHUTQeu2yw+UE9ko2EMk8MpGT8CUEqdt77CScBBNthhpOI 2eOqdRe7Y4PrxaxetKIAND+1ieuRnnmcZ8vWG0wmg5Z45SuKOOxcqDVQ6Io+kev73t AgKiUrDTKKuog== Received: from jsievers.comcast.net ([12.30.159.20]) by resomta-ch2-08v.sys.comcast.net with ESMTPA id IyNFgL8n4Ww14IyNFgUeD1; Sat, 03 Nov 2018 16:02:16 +0000 Received: from jsievers.enova.com (localhost [127.0.0.1]) by jsievers.comcast.net (Postfix) with ESMTP id ADFBA260056; Sat, 3 Nov 2018 11:02:08 -0500 (CDT) From: Jerry Sievers To: =?utf-8?B?QWxlxaEgWmVsZW7DvQ==?= Cc: pgsql-general@lists.postgresql.org Subject: Re: Logical replication hangs up. References: Date: Sat, 03 Nov 2018 11:02:08 -0500 In-Reply-To: (=?utf-8?Q?=22Ale=C5=A1_Zelen=C3=BD=22's?= message of "Sat, 3 Nov 2018 02:22:03 +0100") Message-ID: <87wopugnen.fsf@jsievers.enova.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-CMAE-Envelope: MS4wfNci6uycj35IQ1difa3F4a0Xp0dlLnnBxqgdjKKjdthprbtSZ8sbJDo2gTPz8S3qvr/vHk1qiZeenhhca1YOpkoDh5Ni1gkC+XzpySljpUasogDQ88SH 2b3Y262/cfafCECFNUS72T1a788FnkyyCcrHTSl4i6LLoeEg9YlCqKAOxVh/ptUxlQiARuT1We7PX4Z9dzEiKRsV6Wdx8lVEs40jMaMcJb2FpJbKyIBdTl0I List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Ale=C5=A1 Zelen=C3=BD writes: > Hello, > > we are suing logical replication on 10.4=C2=A0 and it now hangs. After > some timeout it is retarted again, replaying 18GB of data and then > hangs (while 7GB of wals remains to be proceeded). Timeout... Have a look at the 2 setting wal sender/receiver timeout and you probably need to raise the sender timeout value. HTH > > The backlog of 18GB comes from a failed migration adding new table to > replication while replication user was not granted to select the > table. This was clear from log files and once resolved by adding > select privilege, I thought that all will work as usual (same > happened in test env. many times and adding missing grant for select > was sufficient to get it working... these were issues on tests). > > RDBMS Version: > PostgreSQL 10.4 (Ubuntu 10.4-2.pgdg16.04+1) on x86_64-pc-linux-gnu, > compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609, > 64-bit > > Publication (master) instance error message from log: > > 2018-11-02 18:34:14 UTC 7974 5bdc8d27.1f26 7 192.168.23.11(58884) > master_prod repusr 0 5/0 sub_eur idle [00000]:LOG:=C2=A0 terminating > walsender process due to replication timeout > > Subscription instance log: > > 2018-11-02 18:34:14 UTC 8657 5bdc8d26.21d1 2=C2=A0=C2=A0=C2=A0 0 3/0=C2= =A0=C2=A0 > [XX000]:ERROR:=C2=A0 could not receive data from WAL stream: SSL > connection has been closed unexpectedly > 2018-11-02 18:34:14 UTC 24699 5b923b1c.607b 1209397=C2=A0=C2=A0=C2=A0 0= =C2=A0=C2=A0=C2=A0 > [00000]:LOG:=C2=A0 worker process: logical replication worker for > subscription 37932 (PID 8657) exited with exit code 1 > > While it seems to be a network issue, it is may be not - we have > checked the network and even monitoring, all the time some packets > were exchanged. > > We do have 3 subscriptions(thus repl. slots) for one publication, one > subscriber instance is within same datacenter as master, remainig > subscribers are remote. > > >>> select * from pg_replication_slots > +------------+----------+-----------+--------+-------------+-----------+-= -------+------------+------+--------------+--------------+-----------------= ----+ > | slot_name=C2=A0 | plugin=C2=A0=C2=A0 | slot_type | datoid | database=C2= =A0=C2=A0=C2=A0 | > temporary | active | active_pid | xmin | catalog_xmin | restart_lsn=C2=A0 > | confirmed_flush_lsn | > +------------+----------+-----------+--------+-------------+-----------+-= -------+------------+------+--------------+--------------+-----------------= ----+ > | sub_usd=C2=A0=C2=A0=C2=A0 | pgoutput | logical=C2=A0=C2=A0 | 16421=C2= =A0 | master_prod | > False=C2=A0=C2=A0=C2=A0=C2=A0 | True=C2=A0=C2=A0 | 16604=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 5536488=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 | 426/AAE55A68 > | 426/AAE55A68=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | > | sub_cad=C2=A0=C2=A0=C2=A0 | pgoutput | logical=C2=A0=C2=A0 | 16421=C2= =A0 | master_prod | > False=C2=A0=C2=A0=C2=A0=C2=A0 | True=C2=A0=C2=A0 | 22875=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 5536488=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 | 426/AAE55A68 > | 426/AAE55A68=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | > | sub_eur=C2=A0=C2=A0=C2=A0 | pgoutput | logical=C2=A0=C2=A0 | 16421=C2= =A0 | master_prod | > False=C2=A0=C2=A0=C2=A0=C2=A0 | True=C2=A0=C2=A0 | 16605=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 5536488=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 | 426/AAE55A68 > | 426/AAE55A68=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | > +------------+----------+-----------+--------+-------------+-----------+-= -------+------------+------+--------------+--------------+-----------------= ----+ > > > Once=C2=A0 after the error occurs, wal senders are re/started and they > reads wal files until they reach restart_lsn wal file: > >>> select pg_walfile_name('426/AAE55A68') > +--------------------------+ > | pg_walfile_name=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | > +--------------------------+ > | 0000000100000426000000AA | > +--------------------------+ > > # changing file names until they reach this one: > root@master-db:/pgsql/pgcluster/10/master_prod# lsof -p 1560 -p 5758 > -p 5790| grep pg_wal > postgres 1560 postgres=C2=A0=C2=A0 10r=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 REG= =C2=A0 259,3=C2=A0 16777216 115766007 / > pgsql/pgcluster/10/master_prod/pg_wal/000000010000042900000069 > postgres 5758 postgres=C2=A0=C2=A0 10r=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 REG= =C2=A0 259,3=C2=A0 16777216 115766007 / > pgsql/pgcluster/10/master_prod/pg_wal/000000010000042900000069 > postgres 5790 postgres=C2=A0=C2=A0 10r=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 REG= =C2=A0 259,3=C2=A0 16777216 115766007 / > pgsql/pgcluster/10/master_prod/pg_wal/000000010000042900000069 > > > At this moment sent_lsn stops growing and nothing is happening for a > while. > > select * from pg_stat_replication; > > =C2=A0 pid=C2=A0 | usesysid | usename |=C2=A0 application_name=C2=A0 | cl= ient_addr=C2=A0=C2=A0 | > client_hostname | client_port |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 backend_start=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | > backend_xmin |=C2=A0 state=C2=A0 |=C2=A0=C2=A0 sent_lsn=C2=A0=C2=A0 |=C2= =A0 write_lsn=C2=A0=C2=A0 |=C2=A0 flush_lsn=C2=A0=C2=A0 | > =C2=A0 replay_lsn=C2=A0 | write_lag | flush_lag | replay_lag | sync_prior= ity | > sync_state > -------+----------+---------+--------------------+---------------+-------= ----------+-------------+-------------------------------+--------------+---= ------+--------------+--------------+--------------+--------------+--------= ---+-----------+------------+---------------+------------ > =C2=A016604 |=C2=A0=C2=A0=C2=A0 37868 | repusr=C2=A0 | sub_usd=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 192.168.21.11 | > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 35010 | 2018= -11-02 23:52:22.059157+00 | > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 | catchup | 429/69E9CC60 | 426/AAE55A68 | 426/AAE55A68 > | 426/AAE55A68 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | > async > =C2=A016605 |=C2=A0=C2=A0=C2=A0 37868 | repusr=C2=A0 | sub_eur=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 192.168.23.11 | > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 36388 | 2018= -11-02 23:52:24.75038+00=C2=A0 | > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 | catchup | 429/69E9CC60 | 426/AAE55A68 | 426/AAE55A68 > | 426/AAE55A68 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | > async > =C2=A012387 |=C2=A0=C2=A0=C2=A0 37868 | repusr=C2=A0 | sub_cad=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 192.168.22.11 | > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 51982 | 2018= -11-02 23:30:31.372404+00 | > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 | catchup | 429/69E9CC60 | 426/AAE55A68 | 426/AAE55A68 > | 426/AAE55A68 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 | > async > (3 rows) > > What I found being done behind the scenes while from > pg_stat_replication "nothing is happening": > > On Master wal snaders strace look like: > > open("pg_replslot/sub_usd/xid-6040508-lsn-429-57000000.snap", > O_WRONLY|O_CREAT|O_APPEND, 0600) =3D 15 > write(15, "\334\0\0\0\0\0\0\0H\3\234W)\4\0\0\0\0\0\0\0\0\0\0\177\6\0\ > 0%@\0\0"..., 220) =3D 220 > close(15)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 0 > > On subscription instance, strice for wal reciever looks like this: > recvfrom(7, 0x55f56a3783c3, 5, 0, NULL, NULL) =3D -1 EAGAIN (Resource > temporarily unavailable) > epoll_create1(EPOLL_CLOEXEC)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 =3D 10 > epoll_ctl(10, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=3D > 1781282352, u64=3D94512536630832}}) =3D 0 > epoll_ctl(10, EPOLL_CTL_ADD, 9, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=3D > 1781282376, u64=3D94512536630856}}) =3D 0 > epoll_ctl(10, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=3D > 1781282400, u64=3D94512536630880}}) =3D 0 > epoll_wait(10, [], 1, 1000)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 0 > > tcpdump: > 21:33:16.811590 IP (tos 0x0, ttl 64, id 24478, offset 0, flags [DF], > proto TCP (6), length 52) > =C2=A0=C2=A0=C2=A0 192.168.20.11.5432 > 192.168.21.11.34744: Flags [.], c= ksum 0xe2ab > (correct), ack 1360, win 235, options [nop,nop,TS val 3699603 ecr > 2060668460], length 0 > 21:33:26.821860 IP (tos 0x0, ttl 64, id 64522, offset 0, flags [DF], > proto TCP (6), length 120) > =C2=A0=C2=A0=C2=A0 192.168.21.11.34744 > 192.168.20.11.5432: Flags [P.], = cksum > 0x3ecf (incorrect -> 0xbaf0), seq 1360:1428, ack 1, win 265, options > [nop,nop,TS val 2060670963 ecr 3699603], length 68 > 21:33:26.822156 IP (tos 0x0, ttl 64, id 24479, offset 0, flags [DF], > proto TCP (6), length 52) > =C2=A0=C2=A0=C2=A0 192.168.20.11.5432 > 192.168.21.11.34744: Flags [.], c= ksum 0xceda > (correct), ack 1428, win 235, options [nop,nop,TS val 3702105 ecr > 2060670963], length 0 > 21:33:36.832445 IP (tos 0x0, ttl 64, id 64523, offset 0, flags [DF], > proto TCP (6), length 120) > =C2=A0=C2=A0=C2=A0 192.168.21.11.34744 > 192.168.20.11.5432: Flags [P.], = cksum > 0x3ecf (incorrect -> 0x3881), seq 1428:1496, ack 1, win 265, options > [nop,nop,TS val 2060673465 ecr 3702105], length 68 > 21:33:36.832752 IP (tos 0x0, ttl 64, id 24480, offset 0, flags [DF], > proto TCP (6), length 52) > =C2=A0=C2=A0=C2=A0 192.168.20.11.5432 > 192.168.21.11.34744: Flags [.], c= ksum 0xbb09 > (correct), ack 1496, win 235, options [nop,nop,TS val 3704608 ecr > 2060673465], length 0 > 21:33:46.843063 IP (tos 0x0, ttl 64, id 64524, offset 0, flags [DF], > proto TCP (6), length 120) > =C2=A0=C2=A0=C2=A0 192.168.21.11.34744 > 192.168.20.11.5432: Flags [P.], = cksum > 0x3ecf (incorrect -> 0xcf34), seq 1496:1564, ack 1, win 265, options > [nop,nop,TS val 2060675968 ecr 3704608], length 68 > 21:33:46.843362 IP (tos 0x0, ttl 64, id 24481, offset 0, flags [DF], > proto TCP (6), length 52) > =C2=A0=C2=A0=C2=A0 192.168.20.11.5432 > 192.168.21.11.34744: Flags [.], c= ksum 0xa737 > (correct), ack 1564, win 235, options [nop,nop,TS val 3707111 ecr > 2060675968], length 0 > > Some checksum issues, but finally ack-ed. Tried to open psql session > from host running subscription instance to master instance and it > worked like a charm. > > > I've tried pg_waldump, but nothing what I'll be able to consider as > error... >>> select pg_walfile_name('426/AAE55A68') > +--------------------------+ > | pg_walfile_name=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | > +--------------------------+ > | 0000000100000426000000AA | > +--------------------------+ > /usr/lib/postgresql/10/bin/pg_waldump -s 426/AAE55A68 > 0000000100000426000000AA | less > > rmgr: Standby=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0=C2=A0=C2=A0=C2= =A0 50/=C2=A0=C2=A0=C2=A0 50, tx:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0, lsn: > 426/AAE55A68, prev 426/AAE559C0, desc: RUNNING_XACTS nextXid 5536488 > latestCompletedXid 5536487 oldestRunningXid 5536488 > rmgr: Standby=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0=C2=A0=C2=A0=C2= =A0 50/=C2=A0=C2=A0=C2=A0 50, tx:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0, lsn: > 426/AAE55AA0, prev 426/AAE55A68, desc: RUNNING_XACTS nextXid 5536488 > latestCompletedXid 5536487 oldestRunningXid 5536488 > rmgr: Standby=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0=C2=A0=C2=A0=C2= =A0 50/=C2=A0=C2=A0=C2=A0 50, tx:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0, lsn: > 426/AAE55AD8, prev 426/AAE55AA0, desc: RUNNING_XACTS nextXid 5536488 > latestCompletedXid 5536487 oldestRunningXid 5536488 > rmgr: Standby=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0=C2=A0=C2=A0=C2= =A0 50/=C2=A0=C2=A0=C2=A0 50, tx:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0, lsn: > 426/AAE55B10, prev 426/AAE55AD8, desc: RUNNING_XACTS nextXid 5536488 > latestCompletedXid 5536487 oldestRunningXid 5536488 > rmgr: Standby=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0=C2=A0=C2=A0=C2= =A0 50/=C2=A0=C2=A0=C2=A0 50, tx:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0, lsn: > 426/AAE55B48, prev 426/AAE55B10, desc: RUNNING_XACTS nextXid 5536488 > latestCompletedXid 5536487 oldestRunningXid 5536488 > rmgr: Standby=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0=C2=A0=C2=A0=C2= =A0 50/=C2=A0=C2=A0=C2=A0 50, tx:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0, lsn: > 426/AAE55B80, prev 426/AAE55B48, desc: RUNNING_XACTS nextXid 5536488 > latestCompletedXid 5536487 oldestRunningXid 5536488 > rmgr: XLOG=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0= =C2=A0=C2=A0=C2=A0 24/=C2=A0=C2=A0=C2=A0 24, tx:=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 0, lsn: > 426/AAE55BB8, prev 426/AAE55B80, desc: SWITCH > > > I've also did the same for sent_lsn: >>> select pg_walfile_name('429/69E9CC60'); > +--------------------------+ > | pg_walfile_name=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | > +--------------------------+ > | 000000010000042900000069 | > +--------------------------+ > > /usr/lib/postgresql/10/bin/pg_waldump -s 429/69E9CC60 > 000000010000042900000069 | less > rmgr: Heap=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0= =C2=A0=C2=A0=C2=A0 76/=C2=A0=C2=A0=C2=A0 76, tx:=C2=A0=C2=A0=C2=A0 5536495,= lsn: > 429/69E9CC60, prev 429/69E9CC10, desc: DELETE off 81 KEYS_UPDATED , > blkref #0: rel 1663/16421/38572 blk 40 > rmgr: Heap=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0= =C2=A0=C2=A0=C2=A0 76/=C2=A0=C2=A0=C2=A0 76, tx:=C2=A0=C2=A0=C2=A0 5536495,= lsn: > 429/69E9CCB0, prev 429/69E9CC60, desc: DELETE off 82 KEYS_UPDATED , > blkref #0: rel 1663/16421/38572 blk 40 > rmgr: Heap=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 len (rec/tot):=C2=A0= =C2=A0=C2=A0=C2=A0 76/=C2=A0=C2=A0=C2=A0 76, tx:=C2=A0=C2=A0=C2=A0 5536495,= lsn: > 429/69E9CD00, prev 429/69E9CCB0, desc: DELETE off 83 KEYS_UPDATED , > blkref #0: rel 1663/16421/38572 blk 40 > > The relation 38572 from list above is an ordinary non replicated > table. > > Any help or advice how to diagnose/get it working is highly > appreciated. > > Kind regards Ales Zeleny > > --=20 Jerry Sievers Postgres DBA/Development Consulting e: postgres.consulting@comcast.net p: 312.241.7800