pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Skipping logical replication transactions on subscriber side
Date: Tue, 1 Jun 2021 17:35:44 +0200
Message-ID: <1f0cd48d-81b0-e8ae-df8c-f6207d00f44a@enterprisedb.com> (raw)
In-Reply-To: <CAA4eK1+2O68tkwdZsyfw3aZ7zB4YdejM4GzCciRtwcON6gBbTw@mail.gmail.com>
References: <CAD21AoDeScrsHhLyEPYqN3sydg6PxAPVBboK=30xJfUVihNZDA@mail.gmail.com>
	<CAA4eK1+MkrsHRTdFJiYF82UE3rXBU2QcQfzH-3i_u4RFzapCsw@mail.gmail.com>
	<CAD21AoCxC5rAR9hdAu5HLWbkNx5jR6boSeDfj=8ph5-fWtjxuQ@mail.gmail.com>
	<CAA4eK1+Ghs131feWdYZd1VJ9ozeRXEoH3gcF1OtNe3RW557wew@mail.gmail.com>
	<CAD21AoCmMahD+EQB+HK7n8oM_hjMqytNm73rDhLdFPPtoYG+-w@mail.gmail.com>
	<CAA4eK1LTE-AYtwatvLzAw+Vy53C92QHoB7-rVbX-9nf8ws2Vag@mail.gmail.com>
	<CAD21AoDyGQNVDUyY2b3=hfPm359GUP=Yep7i=ojL42cP+QAmpQ@mail.gmail.com>
	<CAA4eK1K=9Z1qTRQ8FDUuoK1r4te9TgUHka97cG0Ua65SaTyEPg@mail.gmail.com>
	<bd954841-596d-c212-9a13-6f9747cc8903@enterprisedb.com>
	<CAA4eK1+2O68tkwdZsyfw3aZ7zB4YdejM4GzCciRtwcON6gBbTw@mail.gmail.com>

On 01.06.21 06:01, Amit Kapila wrote:
> But, won't that be costly in cases where we have errors in the
> processing of very large transactions? Subscription has to process all
> the data before it gets an error. I think we can even imagine this
> feature to be extended to use commitLSN as a skip candidate in which
> case we can even avoid getting the data of that transaction from the
> publisher. So if this information is persistent, the user can even set
> the skip identifier after the restart before the publisher can send
> all the data.

At least in current practice, skipping parts of the logical replication 
stream on the subscriber is a rare, emergency-level operation when 
something that shouldn't have happened happened.  So it doesn't really 
matter how costly it is.  It's not going to be more costly than the 
error happening in the first place.  All you'd need is one shared memory 
slot per subscription to store a xid to skip.

We will also want some proper conflict handling at some point.  But I 
think what is being discussed here is meant to be a repair tool, not a 
policy tool, and I'm afraid it might get over-engineered.






view thread (590+ messages)  latest in thread

Message-ID: <1f0cd48d-81b0-e8ae-df8c-f6207d00f44a@enterprisedb.com>
Permalink:  ../1f0cd48d-81b0-e8ae-df8c-f6207d00f44a@enterprisedb.com/
Also on:    postgresql.org/message-id/1f0cd48d-81b0-e8ae-df8c-f6207d00f44a@enterprisedb.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: peter.eisentraut@enterprisedb.com, amit.kapila16@gmail.com, sawada.mshk@gmail.com
  Subject: Re: Skipping logical replication transactions on subscriber side
  In-Reply-To: <1f0cd48d-81b0-e8ae-df8c-f6207d00f44a@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox