Date: Fri, 30 May 2014 17:27:21 +0200
From: Andres Freund <andres@2ndquadrant.com>
To: Heikki Linnakangas <hlinnakangas@vmware.com>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>,
 Robert Haas <robertmhaas@gmail.com>, Bruce Momjian <bruce@momjian.us>,
 Rajeev rastogi <rajeev.rastogi@huawei.com>, Ants Aasma <ants@cybertec.at>,
 PostgreSQL-development <pgsql-hackers@postgresql.org>,
 Markus Wanner <markus@bluegap.ch>
Subject: Re: Proposal for CSN based snapshots
Message-ID: <20140530152721.GC30516@awork2.anarazel.de>
References: <51B6C97B.1070000@bluegap.ch>
 <BF2827DCCE55594C8D7A8F7FFD3AB7713DDBC758@SZXEML508-MBX.china.huawei.com>
 <5370D323.1070606@vmware.com>
 <20140512144112.GB9619@awork2.anarazel.de>
 <5370E267.2020904@vmware.com> <20140515183436.GD25053@momjian.us>
 <CA+TgmoYiqga4KWz_1=O8D6ivmTAJfY=-fxO6Nk+D-o3M0M60=Q@mail.gmail.com>
 <20140515200632.GB16014@awork2.anarazel.de>
 <20140515213714.GB7857@eldon.alvh.no-ip.org>
 <53889CCB.7030005@vmware.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <53889CCB.7030005@vmware.com>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

Hi,

On 2014-05-30 17:59:23 +0300, Heikki Linnakangas wrote:
> So, here's a first version of the patch. Still very much WIP.

Cool.

> One thorny issue came up in discussions with other hackers on this in PGCon:
> 
> When a transaction is committed asynchronously, it becomes visible to other
> backends before the commit WAL record is flushed. With CSN-based snapshots,
> the order that transactions become visible is always based on the LSNs of
> the WAL records. This is a problem when there is a mix of synchronous and
> asynchronous commits:
> 
> If transaction A commits synchronously with commit LSN 1, and transaction B
> commits asynchronously with commit LSN 2, B cannot become visible before A.
> And we cannot acknowledge B as committed to the client until it's visible to
> other transactions. That means that B will have to wait for A's commit
> record to be flushed to disk, before it can return, even though it was an
> asynchronous commit.

> I personally think that's annoying, but we can live with it. The most common
> usage of synchronous_commit=off is to run a lot of transactions in that
> mode, setting it in postgresql.conf. And it wouldn't completely defeat the
> purpose of mixing synchronous and asynchronous commits either: an
> asynchronous commit still only needs to wait for any already-logged
> synchronous commits to be flushed to disk, not the commit record of the
> asynchronous transaction itself.

I have a hard time believing that users won't hate us for such a
regression. It's pretty common to mix both sorts of transactions and
this will - by my guesstimate - dramatically reduce throughput for the
async backends.

> * Logical decoding is broken. I hacked on it enough that it looks roughly
> sane and it compiles, but didn't spend more time to debug.

I think we can live with it not working for the first few
iterations. I'll look into it once the patch has stabilized a bit.

> * I expanded pg_clog to 64-bits per XID, but people suggested keeping
> pg_clog as is, with two bits per commit, and adding a new SLRU for the
> commit LSNs beside it. Probably will need to do something like that to avoid
> bloating the clog.

It also influences how on-disk compatibility is dealt with. So: How are
you planning to deal with on-disk compatibility?

> * Add some kind of backend-private caching of clog, to make it faster to
> access. The visibility checks are now hitting the clog a lot more heavily
> than before, as you need to check the clog even if the hint bits are set, if
> the XID falls between xmin and xmax of the snapshot.

That'll hurt a lot in concurrent scenarios :/. Have you measured how
'wide' xmax-xmin usually is? I wonder if we could just copy a range of
values from the clog when we start scanning....

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers