Re: logical decoding of two-phase transactions

pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed

From: Stas Kelvich <s.kelvich@postgrespro.ru>
To: Craig Ringer <craig@2ndquadrant.com>
Cc: Petr Jelinek <petr.jelinek@2ndquadrant.com>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Cc: Robert Haas <robertmhaas@gmail.com>
Subject: Re: logical decoding of two-phase transactions
Date: Mon, 27 Mar 2017 12:53:01 +0300
Message-ID: <DFC9E270-4554-4E46-B2F8-FD864C29610D@postgrespro.ru> (raw)
In-Reply-To: <CAMsr+YF0FDNjstkKzByeyG3G66SWf_mbH+cr3qhKAK3UYaZzaA@mail.gmail.com>
References: <02DA5F5E-CECE-4D9C-8B4B-418077E2C010@postgrespro.ru>
	<CAMsr+YH0aRgnk8XwyYypV1E10DNXwgGpWTLq_P2MGjACpiwMpA@mail.gmail.com>
	<89073436-925E-4299-A9AF-88D04E55C231@postgrespro.ru>
	<CAMsr+YGX6Q-VNXEhu3sriCf0wCS6hjQ0Lhj4rJ8VueVuuka+fg@mail.gmail.com>
	<3477E635-F590-4432-BD20-C59142B65CD3@postgrespro.ru>
	<CAMsr+YHos95GFyfd2Gk61-66wpxFTgXJh=sBV3Rtfhe3xzG1HQ@mail.gmail.com>
	<CAB7nPqQogmPYRkb=wwA4veeAWZXwDXQA64vw7fO_mHuDhAOHoA@mail.gmail.com>
	<CAMsr+YHQzGxnR-peT4SbX2-xiG2uApJMTgZ4a3TiRBM6COyfqg@mail.gmail.com>
	<1FE466EA-7058-484D-B0DB-42CD81FA59F0@postgrespro.ru>
	<CAMsr+YHVNW-Lh-XnYnkfhGL2QT8kK9kM4zkZ4+Sv=fszhujj-g@mail.gmail.com>
	<46ebc39d-82a0-e59f-2552-2ea8911a64e7@2ndquadrant.com>
	<DFE91C41-1410-4D89-B7D6-819F5FDC1437@postgrespro.ru>
	<CAMsr+YHbvCtVetrtJ1_LCKen98d3cG8zmHP_-nSm=aDj6yc0vQ@mail.gmail.com>
	<25A09FA8-5B0E-447D-97FB-F37F3B0E34E9@postgrespro.ru>
	<EEBD82AA-61EE-46F4-845E-05B94168E8F2@postgrespro.ru>
	<CAMsr+YE5UC7MeZFt7+rhNrcacAne8bogftJH3GtqGQwMN4a1Gg@mail.gmail.com>
	<8217d096-3641-8022-09e4-4e625633b278@2ndquadrant.com>
	<CAMsr+YFKAw2Cc5Z+Jbw72FeLDBzUhs0drECxG4pmgkTH7LsoJg@mail.gmail.com>
	<E6112EA9-2949-4E4A-8FE4-1C6A2096379D@postgrespro.ru>
	<CAMsr+YFsN5D=dPjoyXN33OBTGJgfYn3rrh-BH3GX5Qfbg-3VRA@mail.gmail.com>
	<EFF3D2BF-05BD-4EBA-A254-27EED342CD22@postgrespro.ru>
	<CAMsr+YHwPAeMrD2HEnQFy5dQr8CR=fwVjnokZNFua_K+QmMhBA@mail.gmail.com>
	<F5187EFB-8907-4D73-B961-9AC8B45B03CD@postgrespro.ru>
	<CAMsr+YGkEtwcsvQSU_6w2hnoY=oGEAxdK_ApYs3e-iwqSm06qg@mail.gmail.com>
	<CAMsr+YF0FDNjstkKzByeyG3G66SWf_mbH+cr3qhKAK3UYaZzaA@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>


> On 27 Mar 2017, at 12:26, Craig Ringer <craig@2ndquadrant.com> wrote:
> 
> On 27 March 2017 at 09:31, Craig Ringer <craig@2ndquadrant.com> wrote:
> 
>> We're in the last week of the CF. If you have a patch that's nearly
>> ready or getting there, now would be a good time to post it for help
>> and input from others.
>> 
>> I would really like to get this in, but we're running out of time.
>> 
>> Even if you just post your snapshot management work, with the cosmetic
>> changes discussed above, that would be a valuable start.
> 
> I'm going to pick up the last patch and:

I’m heavily underestimated amount of changes there, but almost finished
and will send updated patch in several hours.

> * Ensure we only add the GID to xact records for 2pc commits and aborts

And only during wal_level >= logical. Done.
Also patch adds origin info to prepares and aborts.

> * Add separate callbacks for prepare, abort prepared, and commit
> prepared (of xacts already processed during prepare), so we aren't
> overloading the "commit" callback and don't have to create fake empty
> transactions to pass to the commit callback;

Done.

> * Add another callback to determine whether an xact should be
> processed at PREPARE TRANSACTION or COMMIT PREPARED time.

Also done.

> * Rename the snapshot builder faux-commit stuff in the current patch
> so it's clearer what's going on.

Hm. Okay, i’ll leave that part to you.

> * Write tests covering DDL, abort-during-decode, etc

I’ve extended test, but it is good to have some more.

> Some special care is needed for the callback that decides whether to
> process a given xact as 2PC or not. It's called before PREPARE
> TRANSACTION to decide whether to decode any given xact at prepare time
> or wait until it commits. It's called again at COMMIT PREPARED time if
> we crashed after we processed PREPARE TRANSACTION and advanced our
> confirmed_flush_lsn such that we won't re-process the PREPARE
> TRANSACTION again. Our restart_lsn might've advanced past it so we
> never even decode it, so we can't rely on seeing it at all. It has
> access to the xid, gid and invalidations, all of which we have at both
> prepare and commit time, to make its decision from. It must have the
> same result at prepare and commit time for any given xact. We can
> probably use a cache in the reorder buffer to avoid the 2nd call on
> commit prepared if we haven't crashed/reconnected between the two.

Good point. Didn’t think about restart_lsn in case when we are skipping this
particular prepare (filter_prepared() -> true, in my terms). I think that should
work properly as it use the same code path as it was before, but I’ll look at it.

> This proposal does not provide a way to safely decode a 2pc xact that
> made catalog changes which may be aborted while being decoded. The
> plugin must lock such an xact so that it can't be aborted while being
> processed, or defer decoding until commit prepared. It can use the
> invalidations for the commit to decide.

I had played with that two-pass catalog scan and it seems to be
working but after some time I realised that it is not useful for the main
case when commit/abort is generated after receiver side will answer to
prepares. Also that two-pass scan is a massive change in relcache.c and
genam.c (FWIW there were no problems with cache, but some problems
with index scan and handling one-to-many queries to catalog, e.g. table
with it fields)

Finally i decided to throw it and switched to filter_prepare callback and
passed there txn structure to allow access to has_catalog_changes field.


Stas Kelvich
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

view thread (644+ messages)  latest in thread

Message-ID: <DFC9E270-4554-4E46-B2F8-FD864C29610D@postgrespro.ru>
Permalink:  ../DFC9E270-4554-4E46-B2F8-FD864C29610D@postgrespro.ru/
Also on:    postgresql.org/message-id/DFC9E270-4554-4E46-B2F8-FD864C29610D@postgrespro.ru

 ·

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: s.kelvich@postgrespro.ru, craig@2ndquadrant.com, petr.jelinek@2ndquadrant.com, robertmhaas@gmail.com
  Subject: Re: logical decoding of two-phase transactions
  In-Reply-To: <DFC9E270-4554-4E46-B2F8-FD864C29610D@postgrespro.ru>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox