Subject: RE: Failed transaction statistics to measure the logical replication progress

From: "[email-masked]" <[email-masked]>
Date: Wed, 12 Jan 2022 12:34:49 +0000

On Thursday, December 23, 2021 6:37 PM Wang, Wei/王 威 <[email-masked]> wrote:
> On Wednesday, December 22, 2021 10:30 PM [email-masked]
> <[email-masked]> wrote:
> > On Wednesday, December 22, 2021 8:38 PM I wrote:
> > > Do we expect these commit counts which come from empty transactions ?
> > This is another issue discussed in [1] where the patch in the thread
> > is a work in progress, I think.
> > ......
> > IMHO, the conclusion is we are currently in the middle of fixing the behavior.
> 
> Thank you for telling me this.
> After applying v19-* and
> v15-0001-Skip-empty-transactions-for-logical-replication.patch,
> I retested v19-* patches. The result of previous case looks good to me.
> 
> But the results of following cases are also similar to previous unexpected result
> which increases commit_count or abort_count unexpectedly.
> [1]
> (Based on environment in the previous example, set TWO_PHASE=true)
> [Publisher] begin; insert into replica_test1 values(1,'1'); prepare transaction
> 'id'; commit prepared 'id';
> 
> In subscriber side, the commit_count of two records(sub1 and sub2) is
> increased.
> 
> [2]
> (Based on environment in the previous example, set STREAMING=on)
> [Publisher] begin; INSERT INTO replica_test1 SELECT i, md5(i::text) FROM
> generate_series(1, 5000) s(i); commit;
> 
> In subscriber side, the commit_count of two records(sub1 and sub2) is
> increased.
> 
> [3]
> (Based on environment in the previous example, set TWO_PHASE=true)
> [Publisher] begin; insert into replica_test1 values(1,'1'); prepare transaction
> 'id'; rollback prepared 'id';
> 
> In subscriber side, the abort_count of two records(sub1 and sub2) is
> increased.
> 
> I think the problem maybe is the patch you mentioned
> (Skip-empty-transactions-for-logical-replication.patch) is not finished yet.
> Share this information here.
Hi, thank you for your report.

Yes. As the patch's commit message mentions, the patch doesn't
cover streaming and two phase transactions.

In the above reply, I said that this was an independent issue
and we were in the middle of the modification of the behavior,
but empty transaction turned out to be harmful enough for this feature.
As far as I searched, the current logical replication sends
every transaction that is unrelated to publication specification.
It means that in a common scenario where some parts of tables are
replicated but others are not, the subscription statistics will
be buried by the updates by the empty transactions for the latter,
which damages this patch's value greatly.

Therefore, I included the description in the documentation
reported by you.

The attached v21 has a couple of other minor updates
like a modification of error message text.

Best Regards,
	Takamichi Osumi

---

From: Amit Kapila <[email-masked]>
Date: Thu, 17 Feb 2022 14:38:19 +0530

> but empty transaction turned out to be harmful enough for this feature.

Isn't that because of this patch? I mean the patch is reporting count
even when during apply we haven't started any transaction. In
particular, if you would have reported stats from
apply_handle_commit_internal() when IsTransactionState() returns true,
then it shouldn't have updated the stats for an empty transaction.
Similarly, I see for rollback_prepared, you don't need to report stats
if there is no prepared transaction to rollback. I think for
commit_prepare case, we can't do much for empty xacts but why would
that be a problem of this patch? I think as far as this patch goes, it
reports the number of completed xacts (committed/aborted) in the
subscription worker, so, it doesn't need to handle any special cases
like empty transactions.

-- 
With Regards,
Amit Kapila.