Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nUr3P-0002Pf-Rf for pgsql-bugs@arkaria.postgresql.org; Thu, 17 Mar 2022 14:24:40 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1nUr3O-0008Gi-EQ for pgsql-bugs@arkaria.postgresql.org; Thu, 17 Mar 2022 14:24:38 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nUr3N-0008GY-SU for pgsql-bugs@lists.postgresql.org; Thu, 17 Mar 2022 14:24:38 +0000 Received: from lana.depesz.com ([88.198.49.178] helo=depesz.com) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1nUr3F-0000se-EQ for pgsql-bugs@postgresql.org; Thu, 17 Mar 2022 14:24:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=depesz.com; s=20170201; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Reply-To: Message-ID:Subject:To:Sender:From:Date:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=Sj9BhO2MvlZtiCS1Y62l/deAbQlIlOlrCdkbNMRc9Mg=; b=NpRGm78R7w1UANV7JcUeBD278g xSL1ctkL8mDKmGNTYvmPkkf4CPI3N09PZGlvyeC6czTfAwQcknZmCnIakoA/0sCJnegvBB++kw2ro DuVqYwqFDxRazb3deAKryw717rmXtjYPrbzG/PxbE/eycYEcrm0M3Ann39uAsdfToSRw=; Received: from lana.depesz.com ([88.198.49.178] helo=depesz.com) by depesz.com with esmtpa (Exim 4.92) (envelope-from ) id 1nUr3C-0003SQ-JY for pgsql-bugs@postgresql.org; Thu, 17 Mar 2022 15:24:26 +0100 Date: Thu, 17 Mar 2022 15:24:26 +0100 From: hubert depesz lubaczewski Sender: depesz@depesz.com To: pgsql-bugs mailing list Subject: Logical replication stops dropping used initial-sync replication slots Message-ID: <20220317142426.GA727@depesz.com> Reply-To: depesz@depesz.com MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.10.1 (2018-07-13) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Hi, I have following case: Primary: Ubuntu Bionic with Pg 12.9, and ~ 60k tables. Replica: Ubuntu Focal with Pg 14.2. Settings are: max_wal_senders │ 50 wal_sender_timeout │ 60000 max_sync_workers_per_subscription │ 10 max_worker_processes │ 50 I made publication using: create publication focal14; select format('ALTER PUBLICATION focal14 ADD TABLE ONLY %I.%I;', n.nspname, c.relname) from pg_class c join pg_namespace n on c.relnamespace = n.oid where n.nspname ~ 'shard' and c.relkind in ('p', 'r') order by n.nspname, c.relname \gexec Worked. Then on replica I did: create subscription focal14 connection 'host=${source_ip} port=5432 user=lrep_user dbname=${dbname}' publication focal14; This started initial sync. Happily it was working with 10 concurrent workers, but after ~ hour, I started to get, in logs on replica: 2022-03-17 13:32:48.119 UTC,,,2923553,,62333880.2c9c21,2,,2022-03-17 13:32:48 UTC,8/447946,23877409,ERROR,53400,"could not find free replication state slot for replication origin with OID 51",,"Increase max_replication_slots and try again.",,,,,,,"","logical replication worker",,0 And sure enough, there are MANY replication slots on primary: slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn ---------------------------------------------+----------+-----------+--------+----------+-----------+--------+------------+------+--------------+--------------+--------------------- focal14 | pgoutput | logical | 16606 | dbname | f | f | | | 160319873 | 8B7/89D938C0 | 8B7/93FD2380 pg_4962101_sync_4663369_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14139 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5F79B708 pg_4962101_sync_4794836_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160320058 | 8B8/27FE6668 | 8B8/27FE66A0 pg_4962101_sync_4613771_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14129 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5F79B708 pg_4962101_sync_4681288_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160320058 | 8B8/27FE6630 | 8B8/27FE6668 pg_4962101_sync_4596929_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160320058 | 8B8/27FE65F8 | 8B8/27FE6630 pg_4962101_sync_4394969_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14131 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5F79B708 pg_4962101_sync_4402927_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14130 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5EFFDE90 pg_4962101_sync_4458658_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160320058 | 8B8/27FE66A0 | 8B8/27FE66D8 pg_4962101_sync_4374836_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14132 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5EFFDE90 pg_4962101_sync_4562514_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14137 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5F79B708 pg_4962101_sync_4588090_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14138 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5EFFDE90 pg_4962101_sync_4640031_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4188 | 8B7/940D41C0 pg_4962101_sync_4602213_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D41C0 | 8B7/940D41F8 pg_4962101_sync_4530451_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D41F8 | 8B7/940D4230 pg_4962101_sync_4667494_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4230 | 8B7/940D4268 pg_4962101_sync_4452706_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4268 | 8B7/940D42A0 pg_4962101_sync_4553748_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D42A0 | 8B7/940D42D8 pg_4962101_sync_4592679_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D42D8 | 8B7/940D4310 pg_4962101_sync_4600185_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4310 | 8B7/940D4348 pg_4962101_sync_4659756_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4348 | 8B7/940D4380 pg_4962101_sync_4389357_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4380 | 8B7/940D43B8 pg_4962101_sync_4773549_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D43B8 | 8B7/940D43F0 pg_4962101_sync_4784350_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D43F0 | 8B7/940D4428 pg_4962101_sync_4794832_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4428 | 8B7/940D4460 pg_4962101_sync_4678346_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4460 | 8B7/940D4498 pg_4962101_sync_4703039_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4498 | 8B7/940D44D0 pg_4962101_sync_4496803_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D44D0 | 8B7/940D4508 pg_4962101_sync_4612692_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4508 | 8B7/940D4540 pg_4962101_sync_4779832_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4540 | 8B7/940D4578 pg_4962101_sync_4569154_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4578 | 8B7/940D45B0 pg_4962101_sync_4561772_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D45B0 | 8B7/940D45E8 pg_4962101_sync_4452829_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D45E8 | 8B7/940D4620 pg_4962101_sync_4490381_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4620 | 8B7/940D4658 pg_4962101_sync_4585733_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4658 | 8B7/940D4690 pg_4962101_sync_4459745_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14140 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5F79B708 pg_4962101_sync_4508354_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14141 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5F79B708 pg_4962101_sync_4451993_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | t | 14142 | | 160322211 | 8BF/4D9EC3F0 | 8BF/5F79B708 pg_4962101_sync_4491715_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4770 | 8B7/940D47A8 pg_4962101_sync_4404697_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D47A8 | 8B7/940D47E0 pg_4962101_sync_4520525_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D47E0 | 8B7/940D4818 pg_4962101_sync_4508992_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4818 | 8B7/940D4850 pg_4962101_sync_4474811_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4850 | 8B7/940D4888 pg_4962101_sync_4514152_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4888 | 8B7/940D48C0 pg_4962101_sync_4379697_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D48C0 | 8B7/940D48F8 pg_4962101_sync_4677962_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D48F8 | 8B7/940D4930 pg_4962101_sync_4554683_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4930 | 8B7/940D4968 pg_4962101_sync_4450272_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D4968 | 8B7/940D49A0 pg_4962101_sync_4717161_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D49A0 | 8B7/940D49D8 pg_4962101_sync_4784928_7073527418723003846 | pgoutput | logical | 16606 | dbname | f | f | | | 160319884 | 8B7/940D49D8 | 8B7/940D4A10 (50 rows) For some reason there are MANY inactive slots. But why? I checked in logs for some slot that worked well, and found: 2022-03-17 12:59:58.069 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,2,"authentication",2022-03-17 12:59:58 UTC,13/587955,0,LOG,00000,"replication connection authorized: user=lrep_user application_name=pg_4962101_sync_4512642_7073527418723003846 SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256, compression=off)",,,,,,,,,"" 2022-03-17 12:59:58.071 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,3,"SELECT",2022-03-17 12:59:58 UTC,13/0,0,LOG,00000,"duration: 0.381 ms statement: SELECT pg_catalog.set_config('search_path', '', false);",,,,,,,,,"pg_4962101_sync_4512642_7073527418723003846" 2022-03-17 12:59:58.072 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,4,"BEGIN",2022-03-17 12:59:58 UTC,13/587957,0,LOG,00000,"duration: 0.043 ms statement: BEGIN READ ONLY ISOLATION LEVEL REPEATABLE READ",,,,,,,,,"pg_4962101_sync_4512642_7073527418723003846" 2022-03-17 12:59:58.085 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,5,"idle in transaction",2022-03-17 12:59:58 UTC,13/587957,0,LOG,00000,"logical decoding found consistent point at 8AF/6FDC8110","There are no running transactions.",,,,,"CREATE_REPLICATION_SLOT ""pg_4962101_sync_4512642_7073527418723003846"" LOGICAL pgoutput USE_SNAPSHOT",,,"pg_4962101_sync_4512642_7073527418723003846" 2022-03-17 13:00:00.126 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,6,"SELECT",2022-03-17 12:59:58 UTC,13/587957,0,LOG,00000,"duration: 1.402 ms statement: SELECT c.oid, c.relreplident, c.relkind FROM pg_catalog.pg_class c INNER JOIN pg_catalog.pg_namespace n ON (c.relnamespace = n.oid) WHERE n.nspname = 'aaa_shard_bbb' AND c.relname = 'some_table'",,,,,,,,,"pg_4962101_sync_4512642_7073527418723003846" 2022-03-17 13:00:00.130 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,7,"SELECT",2022-03-17 12:59:58 UTC,13/587957,0,LOG,00000,"duration: 3.495 ms statement: SELECT a.attname, a.atttypid, a.attnum = ANY(i.indkey) FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_index i ON (i.indexrelid = pg_get_replica_identity_index(163890290)) WHERE a.attnum > 0::pg_catalog.int2 AND NOT a.attisdropped AND a.attgenerated = '' AND a.attrelid = 163890290 ORDER BY a.attnum",,,,,,,,,"pg_4962101_sync_4512642_7073527418723003846" 2022-03-17 13:00:00.134 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,8,"COPY",2022-03-17 12:59:58 UTC,13/587957,0,LOG,00000,"duration: 3.525 ms statement: COPY aaa_shard_bbb.some_table TO STDOUT",,,,,,,,,"pg_4962101_sync_4512642_7073527418723003846" 2022-03-17 13:00:00.135 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,9,"COMMIT",2022-03-17 12:59:58 UTC,13/0,0,LOG,00000,"duration: 0.039 ms statement: COMMIT",,,,,,,,,"pg_4962101_sync_4512642_7073527418723003846" 2022-03-17 13:00:01.162 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,10,"idle",2022-03-17 12:59:58 UTC,13/0,0,LOG,00000,"starting logical decoding for slot ""pg_4962101_sync_4512642_7073527418723003846""","Streaming transactions committing after 8AF/6FDC8148, reading WAL from 8AF/6FDC8110.",,,,,"START_REPLICATION SLOT ""pg_4962101_sync_4512642_7073527418723003846"" LOGICAL 8AF/6FDC8148 (proto_version '1', publication_names '""focal14""')",,,"pg_4962101_sync_4512642_7073527418723003846" 2022-03-17 13:00:01.162 UTC,"lrep_user","dbname",13435,"SOME_IP:50816",623330ce.347b,11,"idle",2022-03-17 12:59:58 UTC,13/0,0,LOG,00000,"logical decoding found consistent point at 8AF/6FDC8110","There are no running transactions.",,,,,"START_REPLICATION SLOT ""pg_4962101_sync_4512642_7073527418723003846"" LOGICAL 8AF/6FDC8148 (proto_version '1', publication_names '""focal14""')",,,"pg_4962101_sync_4512642_7073527418723003846" Not sure when this slot got removed, but it's no longer there. But, a slot that currently exists and is inactive: 2022-03-17 13:32:48.401 UTC,"lrep_user","dbname",28146,"SOME_IP:37178",62333880.6df2,2,"authentication",2022-03-17 13:32:48 UTC,11/150184,0,LOG,00000,"replication connection authorized: user=lrep_user application_name=pg_4962101_sync_4474811_7073527418723003846 SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256, compression=off)",,,,,,,,,"" 2022-03-17 13:32:48.403 UTC,"lrep_user","dbname",28146,"SOME_IP:37178",62333880.6df2,3,"SELECT",2022-03-17 13:32:48 UTC,11/0,0,LOG,00000,"duration: 0.408 ms statement: SELECT pg_catalog.set_config('search_path', '', false);",,,,,,,,,"pg_4962101_sync_4474811_7073527418723003846" 2022-03-17 13:32:48.405 UTC,"lrep_user","dbname",28146,"SOME_IP:37178",62333880.6df2,4,"BEGIN",2022-03-17 13:32:48 UTC,11/150186,0,LOG,00000,"duration: 0.061 ms statement: BEGIN READ ONLY ISOLATION LEVEL REPEATABLE READ",,,,,,,,,"pg_4962101_sync_4474811_7073527418723003846" 2022-03-17 13:32:48.419 UTC,"lrep_user","dbname",28146,"SOME_IP:37178",62333880.6df2,5,"idle in transaction",2022-03-17 13:32:48 UTC,11/150186,0,LOG,00000,"logical decoding found consistent point at 8B7/940D4850","There are no running transactions.",,,,,"CREATE_REPLICATION_SLOT ""pg_4962101_sync_4474811_7073527418723003846"" LOGICAL pgoutput USE_SNAPSHOT",,,"pg_4962101_sync_4474811_7073527418723003846" and some that is currently active: 2022-03-17 13:32:48.349 UTC,"lrep_user","dbname",28138,"10.2.210.180:37160",62333880.6dea,2,"authentication",2022-03-17 13:32:48 UTC,11/150181,0,LOG,00000,"replication connection authorized: user=lrep_user application_name=pg_4962101_sync_4663369_7073527418723003846 SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256, compression=off)",,,,,,,,,"" 2022-03-17 13:32:48.354 UTC,"lrep_user","dbname",28138,"10.2.210.180:37160",62333880.6dea,3,"SELECT",2022-03-17 13:32:48 UTC,11/0,0,LOG,00000,"duration: 0.397 ms statement: SELECT pg_catalog.set_config('search_path', '', false);",,,,,,,,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:32:48.356 UTC,"lrep_user","dbname",28138,"10.2.210.180:37160",62333880.6dea,4,"BEGIN",2022-03-17 13:32:48 UTC,11/150183,0,LOG,00000,"duration: 0.049 ms statement: BEGIN READ ONLY ISOLATION LEVEL REPEATABLE READ",,,,,,,,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:32:48.381 UTC,"lrep_user","dbname",28138,"10.2.210.180:37160",62333880.6dea,5,"idle in transaction",2022-03-17 13:32:48 UTC,11/150183,0,LOG,00000,"logical decoding found consistent point at 8B7/940D4690","There are no running transactions.",,,,,"CREATE_REPLICATION_SLOT ""pg_4962101_sync_4663369_7073527418723003846"" LOGICAL pgoutput USE_SNAPSHOT",,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:11.788 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,2,"authentication",2022-03-17 13:35:11 UTC,11/150814,0,LOG,00000,"replication connection authorized: user=lrep_user application_name=pg_4962101_sync_4663369_7073527418723003846 SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256, compression=off)",,,,,,,,,"" 2022-03-17 13:35:11.790 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,3,"SELECT",2022-03-17 13:35:11 UTC,11/0,0,LOG,00000,"duration: 0.414 ms statement: SELECT pg_catalog.set_config('search_path', '', false);",,,,,,,,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:11.793 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,4,"BEGIN",2022-03-17 13:35:11 UTC,11/150816,0,LOG,00000,"duration: 0.058 ms statement: BEGIN READ ONLY ISOLATION LEVEL REPEATABLE READ",,,,,,,,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:11.805 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,5,"idle in transaction",2022-03-17 13:35:11 UTC,11/150816,0,LOG,00000,"logical decoding found initial starting point at 8B8/2D0B3DB0","Waiting for transactions (approximately 1) older than 160320063 to end.",,,,,"CREATE_REPLICATION_SLOT ""pg_4962101_sync_4663369_7073527418723003846"" LOGICAL pgoutput USE_SNAPSHOT",,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:11.915 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,6,"idle in transaction",2022-03-17 13:35:11 UTC,11/150816,0,LOG,00000,"logical decoding found initial consistent point at 8B8/2D1859D0","Waiting for transactions (approximately 1) older than 160320064 to end.",,,,,"CREATE_REPLICATION_SLOT ""pg_4962101_sync_4663369_7073527418723003846"" LOGICAL pgoutput USE_SNAPSHOT",,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:12.035 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,7,"idle in transaction",2022-03-17 13:35:11 UTC,11/150816,0,LOG,00000,"logical decoding found consistent point at 8B8/2D1874C0","There are no running transactions.",,,,,"CREATE_REPLICATION_SLOT ""pg_4962101_sync_4663369_7073527418723003846"" LOGICAL pgoutput USE_SNAPSHOT",,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:12.050 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,8,"SELECT",2022-03-17 13:35:11 UTC,11/150816,0,LOG,00000,"duration: 1.056 ms statement: SELECT c.oid, c.relreplident, c.relkind FROM pg_catalog.pg_class c INNER JOIN pg_catalog.pg_namespace n ON (c.relnamespace = n.oid) WHERE n.nspname = 'aaa_shard_ccc' AND c.relname = 'OTHER_TABLE'",,,,,,,,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:12.055 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,9,"SELECT",2022-03-17 13:35:11 UTC,11/150816,0,LOG,00000,"duration: 3.441 ms statement: SELECT a.attname, a.atttypid, a.attnum = ANY(i.indkey) FROM pg_catalog.pg_attribute a LEFT JOIN pg_catalog.pg_index i ON (i.indexrelid = pg_get_replica_identity_index(308468275)) WHERE a.attnum > 0::pg_catalog.int2 AND NOT a.attisdropped AND a.attgenerated = '' AND a.attrelid = 308468275 ORDER BY a.attnum",,,,,,,,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:12.063 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,10,"COPY",2022-03-17 13:35:11 UTC,11/150816,0,LOG,00000,"duration: 7.618 ms statement: COPY aaa_shard_ccc.OTHER_TABLE TO STDOUT",,,,,,,,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:12.069 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,11,"COMMIT",2022-03-17 13:35:11 UTC,11/0,0,LOG,00000,"duration: 0.051 ms statement: COMMIT",,,,,,,,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:12.069 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,12,"idle",2022-03-17 13:35:11 UTC,11/0,0,LOG,00000,"starting logical decoding for slot ""pg_4962101_sync_4663369_7073527418723003846""","Streaming transactions committing after 8B8/2D1874F8, reading WAL from 8B8/2D0AB818.",,,,,"START_REPLICATION SLOT ""pg_4962101_sync_4663369_7073527418723003846"" LOGICAL 8B8/2D1874F8 (proto_version '1', publication_names '""focal14""')",,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:12.069 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,13,"idle",2022-03-17 13:35:11 UTC,11/0,0,LOG,00000,"logical decoding found initial starting point at 8B8/2D0B3DB0","Waiting for transactions (approximately 1) older than 160320063 to end.",,,,,"START_REPLICATION SLOT ""pg_4962101_sync_4663369_7073527418723003846"" LOGICAL 8B8/2D1874F8 (proto_version '1', publication_names '""focal14""')",,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:12.083 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,14,"idle",2022-03-17 13:35:11 UTC,11/0,0,LOG,00000,"logical decoding found initial consistent point at 8B8/2D1859D0","Waiting for transactions (approximately 1) older than 160320064 to end.",,,,,"START_REPLICATION SLOT ""pg_4962101_sync_4663369_7073527418723003846"" LOGICAL 8B8/2D1874F8 (proto_version '1', publication_names '""focal14""')",,,"pg_4962101_sync_4663369_7073527418723003846" 2022-03-17 13:35:12.090 UTC,"lrep_user","dbname",14139,"10.2.210.180:43592",6233390f.373b,15,"idle",2022-03-17 13:35:11 UTC,11/0,0,LOG,00000,"logical decoding found consistent point at 8B8/2D1874C0","There are no running transactions.",,,,,"START_REPLICATION SLOT ""pg_4962101_sync_4663369_7073527418723003846"" LOGICAL 8B8/2D1874F8 (proto_version '1', publication_names '""focal14""')",,,"pg_4962101_sync_4663369_7073527418723003846" Looks that, for some reason, it doesn't get data on next table to copy data from, and is kinda stuck in process... What could it be? How can I fix that? We need to use this (logical replication) as we want to upgrade OS, and there is locale problem, which prohibits usage of physical replication. Best regards, depesz