Re: Skipping logical replication transactions on subscriber side

pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed

From: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: vignesh C <vignesh21@gmail.com>
Cc: Greg Nancarrow <gregn4422@gmail.com>
Cc: tanghy.fnst@fujitsu.com <tanghy.fnst@fujitsu.com>
Cc: osumi.takamichi@fujitsu.com <osumi.takamichi@fujitsu.com>
Cc: Alexey Lesovsky <lesovsky@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Skipping logical replication transactions on subscriber side
Date: Tue, 7 Dec 2021 15:44:18 +0100
Message-ID: <09f9584f-0d5b-9988-5b5b-891554e32ae8@enterprisedb.com> (raw)
In-Reply-To: <CAA4eK1LpbjYgEjK0=v+1wAyWQMXALPoh=BS5cQeFam7ye3rQKg@mail.gmail.com>
References: <CAA4eK1KzZp5YC34exW3Yb4Vk9v5iMW0YuJCQsWeTOeF7ejLdXw@mail.gmail.com>
	<CALDaNm2_WY2t2GnzW1-QL823+xk5cuC0zc4Eep8fjYeLB0jfRA@mail.gmail.com>
	<CAA4eK1JheT_MTmQiGfWZJeHSEJGzd-mgPPwBwA=YWjhcTfykPA@mail.gmail.com>
	<CAD21AoBu2f2R6Bc9qWpky8v_h822zdkm0419j60Ph_T9ciaVsw@mail.gmail.com>
	<CAD21AoAnexmfLz5dA4mPXT10b=wt0DEsp-CcAbLCZbhuh8zZoQ@mail.gmail.com>
	<OS0PR01MB5716D985D4F2430E817CE30B94689@OS0PR01MB5716.jpnprd01.prod.outlook.com>
	<CAA4eK1K_yFm_vLyNi-unqJNYk1edNkuOrqjBRWVTF5bFaTSsQg@mail.gmail.com>
	<CAD21AoAUogU_4+yPrwJeaZhOUV06USE+TFJHt5OD48O43NFvnQ@mail.gmail.com>
	<CAA4eK1JOsqPHZTOeifp+Q8pWgrn1qzks3ojZnSHp-1QZFcjyQQ@mail.gmail.com>
	<CAD21AoAqDVCVzSpv6DOMLqLeO+m4Jh=dKv9a8Cc4qthdpYzqFw@mail.gmail.com>
	<OS0PR01MB571603F90202D6709B5DF5B594689@OS0PR01MB5716.jpnprd01.prod.outlook.com>
	<CAA4eK1KD_C_0LSxaYB0UbG59VOgjf4mXBeSYbVWCLXAnnuqnPw@mail.gmail.com>
	<f716f584-65d0-fe83-2e84-53426631739a@enterprisedb.com>
	<CAA4eK1LpbjYgEjK0=v+1wAyWQMXALPoh=BS5cQeFam7ye3rQKg@mail.gmail.com>

On 03.12.21 03:53, Amit Kapila wrote:
>> I don't know how difficult it would be, but allowing multiple xids might
>> be desirable.
> 
> Are there many cases where there could be multiple xid failures that
> the user can skip? Apply worker always keeps looping at the same error
> failure so the user wouldn't know of the second xid failure (if any)
> till the first failure is resolved.

Yeah, nevermind, doesn't make sense.

> Yeah, I also think so. BTW, what do you think of providing extra
> flexibility of giving other options like 'operation', 'rel' along with
> xid? I think such options could be useful for large transactions that
> operate on multiple tables as it is quite possible that only a
> particular operation from the entire transaction is the cause of
> failure. Now, on one side, we can argue that skipping the entire
> transaction is better from the consistency point of view but I think
> it is already possible that we just skip a particular update/delete
> (if the corresponding tuple doesn't exist on the subscriber). For the
> sake of simplicity, we can just allow providing xid at this stage and
> then extend it later as required but I am not very sure of that point.

Skipping transactions partially sounds dangerous, especially when 
exposed as an option to users.  Needs more careful thought.

view thread (590+ messages)  latest in thread

Message-ID: <09f9584f-0d5b-9988-5b5b-891554e32ae8@enterprisedb.com>
Permalink:  ../09f9584f-0d5b-9988-5b5b-891554e32ae8@enterprisedb.com/
Also on:    postgresql.org/message-id/09f9584f-0d5b-9988-5b5b-891554e32ae8@enterprisedb.com

 ·

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: peter.eisentraut@enterprisedb.com, amit.kapila16@gmail.com, houzj.fnst@fujitsu.com, sawada.mshk@gmail.com, vignesh21@gmail.com, gregn4422@gmail.com, tanghy.fnst@fujitsu.com, osumi.takamichi@fujitsu.com, lesovsky@gmail.com
  Subject: Re: Skipping logical replication transactions on subscriber side
  In-Reply-To: <09f9584f-0d5b-9988-5b5b-891554e32ae8@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox