Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1i4IND-0006aG-0T for pgsql-hackers@arkaria.postgresql.org; Sun, 01 Sep 2019 05:25:59 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1i4INB-0003Jt-B3 for pgsql-hackers@arkaria.postgresql.org; Sun, 01 Sep 2019 05:25:57 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1i4INA-0003JS-W3 for pgsql-hackers@lists.postgresql.org; Sun, 01 Sep 2019 05:25:57 +0000 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1i4IN8-0000zy-3r for pgsql-hackers@postgresql.org; Sun, 01 Sep 2019 05:25:56 +0000 Received: by mail-pf1-x441.google.com with SMTP id h195so468698pfe.5 for ; Sat, 31 Aug 2019 22:25:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:message-id:in-reply-to:references:subject :mime-version; bh=xQDQWbSo8H57xhkRLLs5jN8c6+iaGLEnafpVLo0J+14=; b=YcmC995oFPPybymhJJUzxZ5JTZXu38u85j9HIr55PWa5UUthdZIi/mURiSAcafxJRV In4SNuE4ohdSzm9OPgOYAbXofmUO0WWQ11kb/h9u2CUj7FFmS5fi5Bhe5YkdeYm6YcWT /ocPAWbN60Ce+rpb8oXM/7rtNX8sVRUhCsliEbC6sO6g8o0tBefd4buLkRk9cSaKfBFb 337SueWfPEHzdaGRjfvizpb8dY+6RRTumIPkxUD1SSO/dqzc6LBlexoPnSCSXCRgP1fn rmSFPoBFTHxt8J/Gx2MSoPcWQpCqZQ4LacmYAw4NZAr5Ys7YDd3L/46nQIx3SvNVAoqD P61A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:message-id:in-reply-to :references:subject:mime-version; bh=xQDQWbSo8H57xhkRLLs5jN8c6+iaGLEnafpVLo0J+14=; b=TDxQKtH7aFYakWe4/o90kDg4blJgoKplRMvnWcD0CpoXywcadbCarxvOVCzdtJqdeU 0+GXqoo/wBz5kMvveBobDoGUes2wZ3GX7EcyfghsMqax+ZVtw1XxyFhnS8CUi4f0kjDb i7fchf4hocMZCFUHP/brBCg4iWnDwrHPgXHadE0K/vHvcJ2VLocdteY2HMQGnlaw2ZzK oE6aPKoxYPPHNk3SBRpbzwbm4Td+mAwc+p6sn1Qg3DIxR91kJvnbwkbNbIv7wPcYNQ9C UOh/5IO37KRErzftUs0F7ulezXtZrcEGLncKwR4wHBn6GSY0LUpNOjvVfcaEaXauFSOC e+fQ== X-Gm-Message-State: APjAAAUXPTEgUR+DvYyo76WA6p8sfbppRR8Ibt4buIxgHVkJVHm1huaL /kt3zUVWTnX4bMsl0l1+rNw= X-Google-Smtp-Source: APXvYqxbgfBvQIGUBtlVgyGtIwt8sUDCOeiX2OqxHvFdd12lL4W6YReBhXqzYi7WcR9KUCukWiLQ3Q== X-Received: by 2002:a17:90a:17ab:: with SMTP id q40mr7062571pja.106.1567315551452; Sat, 31 Aug 2019 22:25:51 -0700 (PDT) Received: from [192.168.13.2] ([188.166.216.51]) by smtp.gmail.com with ESMTPSA id z23sm518970pgi.78.2019.08.31.22.25.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 31 Aug 2019 22:25:50 -0700 (PDT) Date: Sun, 1 Sep 2019 12:25:41 +0700 From: Alexey Zagarin To: a.kondratov@postgrespro.ru, Euler Taveira Cc: Andres Freund , Petr Jelinek , pgsql-hackers , Suzuki Hironobu Message-ID: <66546be6-3daf-4918-9466-07bbc22c212c@Spark> In-Reply-To: References: <7106a0fc-8017-c0fe-a407-9466c9407ff8@2ndquadrant.com> <20190203101439.26xgw5sv7vhgdcz2@alap3.anarazel.de> Subject: Re: row filtering for logical replication X-Readdle-Message-ID: 66546be6-3daf-4918-9466-07bbc22c212c@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="5d6b565a_836c40e_180" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --5d6b565a_836c40e_180 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline I think that I also have found one shortcoming when using the setup descr= ibed by=C2=A0Alexey Kondratov. The problem that I face is that if both (c= loud and remote) tables already have data the moment I add the subscripti= on, then the whole table is copied in both directions initially. Which le= ads to duplicated data and broken replication because COPY doesn't take i= nto account the filtering condition. In case there are filters in a publi= cation, the COPY command that is executed when adding a subscription (or = altering one to refresh a publication) should also filter the data based = on the same condition, e.g.=C2=A0COPY (SELECT * =46ROM ... WHERE ...) TO = ... The current workaround is to always use WITH copy=5Fdata =3D false when s= ubscribing or refreshing, and then manually copy data with the above stat= ement. Alexey Zagarin On 1 Sep 2019 12:11 +0700, Euler Taveira , wrote:= > Em ter, 27 de ago de 2019 =C3=A0s 18:10, = escreveu: > > > > Do you have any plans for continuing working on this patch and > > submitting it again on the closest September commitfest=3F There are = only > > a few days left. Anyway, I will be glad to review the patch if you do= > > submit it, though I didn't yet dig deeply into the code. > > > Sure. See my last email to this thread. I appreciate if you can review = it. > > > Although almost all new tests are passed, there is a problem with DEL= ETE > > replication, so 1 out of 10 tests is failed. It isn't replicated if t= he > > record was created with is=5Fcloud=3DTRUE on cloud, replicated to rem= ote; > > then updated with is=5Fcloud=3D=46ALSE on remote, replicated to cloud= ; then > > deleted on remote. > > > That's because you don't include is=5Fcloud in PK or REPLICA IDENTITY. = I > add a small note in docs. > > > -- > Euler Taveira Timbira - > http://www.timbira.com.br/ > PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento > > > > --5d6b565a_836c40e_180 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
I think that I also have found one shortcoming when= using the setup described by&=23160;Alexey Kondratov. The problem that I= face is that if both (cloud and remote) tables already have data the mom= ent I add the subscription, then the whole table is copied in both direct= ions initially. Which leads to duplicated data and broken replication bec= ause COPY doesn't take into account the filtering condition. In case ther= e are filters in a publication, the COPY command that is executed when ad= ding a subscription (or altering one to refresh a publication) should als= o filter the data based on the same condition, e.g.&=23160;COPY (SEL= ECT * =46ROM ... WHERE ...) TO ...

The current workaround is to always use WITH c= opy=5Fdata =3D false when subscribing or refreshing, and then manually co= py data with the above statement.

Alexey Zagarin
On 1 Sep 2019 12:11 +0700, Euler Ta= veira <euler=40timbira.com.br>, wrote:
Em= ter, 27 de ago de 2019 =C3=A0s 18:10, <a.kondratov=40postgrespro.ru&g= t; escreveu:
Do you have any plans for continuing working on this patch and
submitting it again on the closest September commitfest=3F There are only=
a few days left. Anyway, I will be glad to review the patch if you do
submit it, though I didn't yet dig deeply into the code.

Sure. See my last email to this thread. I appreciate if you can review it= .

Al= though almost all new tests are passed, there is a problem with DELETE replication, so 1 out of 10 tests is failed. It isn't replicated if the record was created with is=5Fcloud=3DTRUE on cloud, replicated to remote;=
then updated with is=5Fcloud=3D=46ALSE on remote, replicated to cloud; th= en
deleted on remote.

That's because you don't include is=5Fcloud in PK or REPLICA IDENTITY. I<= br /> add a small note in docs.


--
Euler Taveira Timbira -
http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento




--5d6b565a_836c40e_180--