pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: Fix the logical replication timeout during large transactions.
7+ messages / 1 participants
[nested] [flat]

* pgsql: Fix the logical replication timeout during large transactions.
@ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix the logical replication timeout during large transactions.

The problem is that we don't send keep-alive messages for a long time
while processing large transactions during logical replication where we
don't send any data of such transactions. This can happen when the table
modified in the transaction is not published or because all the changes
got filtered. We do try to send the keep_alive if necessary at the end of
the transaction (via WalSndWriteData()) but by that time the
subscriber-side can timeout and exit.

To fix this we try to send the keepalive message if required after
processing certain threshold of changes.

Reported-by: Fabrice Chapuis
Author: Wang wei and Amit Kapila
Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda
Backpatch-through: 10
Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/f95d53eded55ecbf037f6416ced6af29a2c3caca

Modified Files
--------------
src/backend/replication/logical/logical.c   | 27 +++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 54 +++++++++++++++++++++++++----
src/backend/replication/walsender.c         | 24 +++++++++++--
src/include/replication/logical.h           |  2 ++
4 files changed, 98 insertions(+), 9 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix the logical replication timeout during large transactions.
@ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix the logical replication timeout during large transactions.

The problem is that we don't send keep-alive messages for a long time
while processing large transactions during logical replication where we
don't send any data of such transactions. This can happen when the table
modified in the transaction is not published or because all the changes
got filtered. We do try to send the keep_alive if necessary at the end of
the transaction (via WalSndWriteData()) but by that time the
subscriber-side can timeout and exit.

To fix this we try to send the keepalive message if required after
processing certain threshold of changes.

Reported-by: Fabrice Chapuis
Author: Wang wei and Amit Kapila
Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda
Backpatch-through: 10
Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com

Branch
------
REL_14_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/d6da71fa8f28faa68823e163f318ffb38a7a9a54

Modified Files
--------------
src/backend/replication/logical/logical.c   | 27 ++++++++++++++++++
src/backend/replication/pgoutput/pgoutput.c | 44 +++++++++++++++++++++++++++--
src/backend/replication/walsender.c         | 38 +++++++++++++++++++++----
src/include/replication/logical.h           |  3 ++
4 files changed, 105 insertions(+), 7 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix the logical replication timeout during large transactions.
@ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix the logical replication timeout during large transactions.

The problem is that we don't send keep-alive messages for a long time
while processing large transactions during logical replication where we
don't send any data of such transactions. This can happen when the table
modified in the transaction is not published or because all the changes
got filtered. We do try to send the keep_alive if necessary at the end of
the transaction (via WalSndWriteData()) but by that time the
subscriber-side can timeout and exit.

To fix this we try to send the keepalive message if required after
processing certain threshold of changes.

Reported-by: Fabrice Chapuis
Author: Wang wei and Amit Kapila
Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda
Backpatch-through: 10
Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com

Branch
------
REL_13_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/55558df2374167af38534df988ec3b9d0b0019f0

Modified Files
--------------
src/backend/replication/logical/logical.c   | 10 ++++++++
src/backend/replication/pgoutput/pgoutput.c | 40 ++++++++++++++++++++++++++++-
src/backend/replication/walsender.c         | 38 +++++++++++++++++++++++----
src/include/replication/logical.h           |  3 +++
4 files changed, 85 insertions(+), 6 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix the logical replication timeout during large transactions.
@ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix the logical replication timeout during large transactions.

The problem is that we don't send keep-alive messages for a long time
while processing large transactions during logical replication where we
don't send any data of such transactions. This can happen when the table
modified in the transaction is not published or because all the changes
got filtered. We do try to send the keep_alive if necessary at the end of
the transaction (via WalSndWriteData()) but by that time the
subscriber-side can timeout and exit.

To fix this we try to send the keepalive message if required after
processing certain threshold of changes.

Reported-by: Fabrice Chapuis
Author: Wang wei and Amit Kapila
Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda
Backpatch-through: 10
Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com

Branch
------
REL_12_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/f832b5007c1c4b61bb1fd9a5000e0ca56b070373

Modified Files
--------------
src/backend/replication/logical/logical.c   | 10 ++++++++
src/backend/replication/pgoutput/pgoutput.c | 40 ++++++++++++++++++++++++++++-
src/backend/replication/walsender.c         | 38 +++++++++++++++++++++++----
src/include/replication/logical.h           |  3 +++
4 files changed, 85 insertions(+), 6 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix the logical replication timeout during large transactions.
@ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix the logical replication timeout during large transactions.

The problem is that we don't send keep-alive messages for a long time
while processing large transactions during logical replication where we
don't send any data of such transactions. This can happen when the table
modified in the transaction is not published or because all the changes
got filtered. We do try to send the keep_alive if necessary at the end of
the transaction (via WalSndWriteData()) but by that time the
subscriber-side can timeout and exit.

To fix this we try to send the keepalive message if required after
processing certain threshold of changes.

Reported-by: Fabrice Chapuis
Author: Wang wei and Amit Kapila
Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda
Backpatch-through: 10
Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com

Branch
------
REL_11_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/87c1dd246af8ace926645900f02886905b889718

Modified Files
--------------
src/backend/replication/logical/logical.c   | 10 ++++++++
src/backend/replication/pgoutput/pgoutput.c | 40 ++++++++++++++++++++++++++++-
src/backend/replication/walsender.c         | 38 +++++++++++++++++++++++----
src/include/replication/logical.h           |  3 +++
4 files changed, 85 insertions(+), 6 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix the logical replication timeout during large transactions.
@ 2022-05-11 05:51 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2022-05-11 05:51 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix the logical replication timeout during large transactions.

The problem is that we don't send keep-alive messages for a long time
while processing large transactions during logical replication where we
don't send any data of such transactions. This can happen when the table
modified in the transaction is not published or because all the changes
got filtered. We do try to send the keep_alive if necessary at the end of
the transaction (via WalSndWriteData()) but by that time the
subscriber-side can timeout and exit.

To fix this we try to send the keepalive message if required after
processing certain threshold of changes.

Reported-by: Fabrice Chapuis
Author: Wang wei and Amit Kapila
Reviewed By: Masahiko Sawada, Euler Taveira, Hou Zhijie, Hayato Kuroda
Backpatch-through: 10
Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com

Branch
------
REL_10_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/a4015ec0375da35e999e86696571ff354a8f706b

Modified Files
--------------
src/backend/replication/logical/logical.c   |  8 ++++++
src/backend/replication/pgoutput/pgoutput.c | 38 ++++++++++++++++++++++++++++-
src/backend/replication/walsender.c         | 38 +++++++++++++++++++++++++----
src/include/replication/logical.h           |  1 +
4 files changed, 79 insertions(+), 6 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: Fix the logical replication timeout during large DDLs.
@ 2023-02-08 02:46 Amit Kapila <akapila@postgresql.org>
  0 siblings, 0 replies; 7+ messages in thread

From: Amit Kapila @ 2023-02-08 02:46 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix the logical replication timeout during large DDLs.

The DDLs like Refresh Materialized views that generate lots of temporary
data due to rewrite rules may not be processed by output plugins (for
example pgoutput). So, we won't send keep-alive messages for a long time
while processing such commands and that can lead the subscriber side to
timeout. We have previously fixed a similar case for large transactions in
commit f95d53eded where the output plugin filters all or most of the
changes but missed to handle the DDLs.

We decided not to backpatch this as this adds a new callback in the
existing exposed structure and moreover, users can increase the
wal_sender_timeout and wal_receiver_timeout to avoid this problem.

Author: Wang wei, Hou Zhijie
Reviewed-by: Peter Smith, Ashutosh Bapat, Shi yu, Amit Kapila
Discussion: https://postgr.es/m/OS3PR01MB6275478E5D29E4A563302D3D9E2B9@OS3PR01MB6275.jpnprd01.prod.outlook.com
Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/8c58624df46222d4d09c5655d8350f3b037880c8

Modified Files
--------------
src/backend/replication/logical/logical.c       | 50 +++++++++++++++++++++++
src/backend/replication/logical/reorderbuffer.c | 20 +++++++++
src/backend/replication/pgoutput/pgoutput.c     | 54 +++----------------------
src/include/replication/reorderbuffer.h         | 12 ++++++
src/tools/pgindent/typedefs.list                |  1 +
5 files changed, 89 insertions(+), 48 deletions(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread


end of thread, other threads:[~2023-02-08 02:46 UTC | newest]

Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org>
2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org>
2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org>
2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org>
2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org>
2022-05-11 05:51 pgsql: Fix the logical replication timeout during large transactions. Amit Kapila <akapila@postgresql.org>
2023-02-08 02:46 pgsql: Fix the logical replication timeout during large DDLs. Amit Kapila <akapila@postgresql.org>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox