pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix the logical replication timeout during large transactions. 7+ messages / 1 participants [nested] [flat]
* pgsql: Fix the logical replication timeout during large transactions. @ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix the logical replication timeout during large transactions. The problem is that we don't send keep-alive messages for a long time while processing large transactions during logical replication where we don't send any data of such transactions. This can happen when the table modified in the transaction is not published or because all the changes got filtered. We do try to send the keep_alive if necessary at the end of the transaction (via WalSndWriteData()) but by that time the subscriber-side can timeout and exit. To fix this we try to send the keepalive message if required after processing certain threshold of changes. Reported-by: Fabrice Chapuis Author: Wang wei and Amit Kapila Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda Backpatch-through: 10 Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/d6da71fa8f28faa68823e163f318ffb38a7a9a54 Modified Files -------------- src/backend/replication/logical/logical.c | 27 ++++++++++++++++++ src/backend/replication/pgoutput/pgoutput.c | 44 +++++++++++++++++++++++++++-- src/backend/replication/walsender.c | 38 +++++++++++++++++++++---- src/include/replication/logical.h | 3 ++ 4 files changed, 105 insertions(+), 7 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix the logical replication timeout during large transactions. @ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix the logical replication timeout during large transactions. The problem is that we don't send keep-alive messages for a long time while processing large transactions during logical replication where we don't send any data of such transactions. This can happen when the table modified in the transaction is not published or because all the changes got filtered. We do try to send the keep_alive if necessary at the end of the transaction (via WalSndWriteData()) but by that time the subscriber-side can timeout and exit. To fix this we try to send the keepalive message if required after processing certain threshold of changes. Reported-by: Fabrice Chapuis Author: Wang wei and Amit Kapila Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda Backpatch-through: 10 Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/55558df2374167af38534df988ec3b9d0b0019f0 Modified Files -------------- src/backend/replication/logical/logical.c | 10 ++++++++ src/backend/replication/pgoutput/pgoutput.c | 40 ++++++++++++++++++++++++++++- src/backend/replication/walsender.c | 38 +++++++++++++++++++++++---- src/include/replication/logical.h | 3 +++ 4 files changed, 85 insertions(+), 6 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix the logical replication timeout during large transactions. @ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix the logical replication timeout during large transactions. The problem is that we don't send keep-alive messages for a long time while processing large transactions during logical replication where we don't send any data of such transactions. This can happen when the table modified in the transaction is not published or because all the changes got filtered. We do try to send the keep_alive if necessary at the end of the transaction (via WalSndWriteData()) but by that time the subscriber-side can timeout and exit. To fix this we try to send the keepalive message if required after processing certain threshold of changes. Reported-by: Fabrice Chapuis Author: Wang wei and Amit Kapila Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda Backpatch-through: 10 Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com Branch ------ REL_10_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/a4015ec0375da35e999e86696571ff354a8f706b Modified Files -------------- src/backend/replication/logical/logical.c | 8 ++++++ src/backend/replication/pgoutput/pgoutput.c | 38 ++++++++++++++++++++++++++++- src/backend/replication/walsender.c | 38 +++++++++++++++++++++++++---- src/include/replication/logical.h | 1 + 4 files changed, 79 insertions(+), 6 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix the logical replication timeout during large transactions. @ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix the logical replication timeout during large transactions. The problem is that we don't send keep-alive messages for a long time while processing large transactions during logical replication where we don't send any data of such transactions. This can happen when the table modified in the transaction is not published or because all the changes got filtered. We do try to send the keep_alive if necessary at the end of the transaction (via WalSndWriteData()) but by that time the subscriber-side can timeout and exit. To fix this we try to send the keepalive message if required after processing certain threshold of changes. Reported-by: Fabrice Chapuis Author: Wang wei and Amit Kapila Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda Backpatch-through: 10 Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/f832b5007c1c4b61bb1fd9a5000e0ca56b070373 Modified Files -------------- src/backend/replication/logical/logical.c | 10 ++++++++ src/backend/replication/pgoutput/pgoutput.c | 40 ++++++++++++++++++++++++++++- src/backend/replication/walsender.c | 38 +++++++++++++++++++++++---- src/include/replication/logical.h | 3 +++ 4 files changed, 85 insertions(+), 6 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix the logical replication timeout during large transactions. @ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix the logical replication timeout during large transactions. The problem is that we don't send keep-alive messages for a long time while processing large transactions during logical replication where we don't send any data of such transactions. This can happen when the table modified in the transaction is not published or because all the changes got filtered. We do try to send the keep_alive if necessary at the end of the transaction (via WalSndWriteData()) but by that time the subscriber-side can timeout and exit. To fix this we try to send the keepalive message if required after processing certain threshold of changes. Reported-by: Fabrice Chapuis Author: Wang wei and Amit Kapila Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda Backpatch-through: 10 Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/f95d53eded55ecbf037f6416ced6af29a2c3caca Modified Files -------------- src/backend/replication/logical/logical.c | 27 +++++++++++++++ src/backend/replication/pgoutput/pgoutput.c | 54 +++++++++++++++++++++++++---- src/backend/replication/walsender.c | 24 +++++++++++-- src/include/replication/logical.h | 2 ++ 4 files changed, 98 insertions(+), 9 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix the logical replication timeout during large transactions. @ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix the logical replication timeout during large transactions. The problem is that we don't send keep-alive messages for a long time while processing large transactions during logical replication where we don't send any data of such transactions. This can happen when the table modified in the transaction is not published or because all the changes got filtered. We do try to send the keep_alive if necessary at the end of the transaction (via WalSndWriteData()) but by that time the subscriber-side can timeout and exit. To fix this we try to send the keepalive message if required after processing certain threshold of changes. Reported-by: Fabrice Chapuis Author: Wang wei and Amit Kapila Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda Backpatch-through: 10 Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com Branch ------ REL_11_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/87c1dd246af8ace926645900f02886905b889718 Modified Files -------------- src/backend/replication/logical/logical.c | 10 ++++++++ src/backend/replication/pgoutput/pgoutput.c | 40 ++++++++++++++++++++++++++++- src/backend/replication/walsender.c | 38 +++++++++++++++++++++++---- src/include/replication/logical.h | 3 +++ 4 files changed, 85 insertions(+), 6 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Fix the logical replication timeout during large DDLs. @ 2023-02-08 02:46 Amit Kapila <akapila@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Amit Kapila @ 2023-02-08 02:46 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix the logical replication timeout during large DDLs. The DDLs like Refresh Materialized views that generate lots of temporary data due to rewrite rules may not be processed by output plugins (for example pgoutput). So, we won't send keep-alive messages for a long time while processing such commands and that can lead the subscriber side to timeout. We have previously fixed a similar case for large transactions in commit f95d53eded where the output plugin filters all or most of the changes but missed to handle the DDLs. We decided not to backpatch this as this adds a new callback in the existing exposed structure and moreover, users can increase the wal_sender_timeout and wal_receiver_timeout to avoid this problem. Author: Wang wei, Hou Zhijie Reviewed-by: Peter Smith, Ashutosh Bapat, Shi yu, Amit Kapila Discussion: https://postgr.es/m/OS3PR01MB6275478E5D29E4A563302D3D9E2B9@OS3PR01MB6275.jpnprd01.prod.outlook.com Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/8c58624df46222d4d09c5655d8350f3b037880c8 Modified Files -------------- src/backend/replication/logical/logical.c | 50 +++++++++++++++++++++++ src/backend/replication/logical/reorderbuffer.c | 20 +++++++++ src/backend/replication/pgoutput/pgoutput.c | 54 +++---------------------- src/include/replication/reorderbuffer.h | 12 ++++++ src/tools/pgindent/typedefs.list | 1 + 5 files changed, 89 insertions(+), 48 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2023-02-08 02:46 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org> 2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org> 2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org> 2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org> 2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org> 2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org> 2023-02-08 02:46 pgsql: Fix the logical replication timeout during large DDLs. Amit Kapila <akapila@postgresql.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox