From: Heikki Linnakangas <hlinnakangas@vmware.com>
To: Alvaro Herrera <alvherre@2ndquadrant.com>
To: Andres Freund <andres@2ndquadrant.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Bruce Momjian <bruce@momjian.us>
Cc: Rajeev rastogi <rajeev.rastogi@huawei.com>
Cc: Ants Aasma <ants@cybertec.at>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Cc: Markus Wanner <markus@bluegap.ch>
Subject: Re: Proposal for CSN based snapshots
Date: Fri, 30 May 2014 17:59:23 +0300
Message-ID: <53889CCB.7030005@vmware.com> (raw)
In-Reply-To: <20140515213714.GB7857@eldon.alvh.no-ip.org>
References: <51B1CB35.4050406@bluegap.ch>
<CA+CSw_uSzXidemci-nU5fJY-qXD0DE0ZWz-j1ASTT_ffBjXo9Q@mail.gmail.com>
<51B6C97B.1070000@bluegap.ch>
<BF2827DCCE55594C8D7A8F7FFD3AB7713DDBC758@SZXEML508-MBX.china.huawei.com>
<5370D323.1070606@vmware.com>
<20140512144112.GB9619@awork2.anarazel.de>
<5370E267.2020904@vmware.com>
<20140515183436.GD25053@momjian.us>
<CA+TgmoYiqga4KWz_1=O8D6ivmTAJfY=-fxO6Nk+D-o3M0M60=Q@mail.gmail.com>
<20140515200632.GB16014@awork2.anarazel.de>
<20140515213714.GB7857@eldon.alvh.no-ip.org>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
So, here's a first version of the patch. Still very much WIP.
One thorny issue came up in discussions with other hackers on this in PGCon:
When a transaction is committed asynchronously, it becomes visible to
other backends before the commit WAL record is flushed. With CSN-based
snapshots, the order that transactions become visible is always based on
the LSNs of the WAL records. This is a problem when there is a mix of
synchronous and asynchronous commits:
If transaction A commits synchronously with commit LSN 1, and
transaction B commits asynchronously with commit LSN 2, B cannot become
visible before A. And we cannot acknowledge B as committed to the client
until it's visible to other transactions. That means that B will have to
wait for A's commit record to be flushed to disk, before it can return,
even though it was an asynchronous commit.
I personally think that's annoying, but we can live with it. The most
common usage of synchronous_commit=off is to run a lot of transactions
in that mode, setting it in postgresql.conf. And it wouldn't completely
defeat the purpose of mixing synchronous and asynchronous commits
either: an asynchronous commit still only needs to wait for any
already-logged synchronous commits to be flushed to disk, not the commit
record of the asynchronous transaction itself.
Ants' original design with a separate commit-sequence-number that's
different from the commit LSN would not have this problem, because that
would allow the commits to become visible to others in out-of-WAL-order.
However, the WAL order == commit order is a nice and simple property,
with other advantages.
Some bigger TODO items:
* Logical decoding is broken. I hacked on it enough that it looks
roughly sane and it compiles, but didn't spend more time to debug.
* I expanded pg_clog to 64-bits per XID, but people suggested keeping
pg_clog as is, with two bits per commit, and adding a new SLRU for the
commit LSNs beside it. Probably will need to do something like that to
avoid bloating the clog.
* Add some kind of backend-private caching of clog, to make it faster to
access. The visibility checks are now hitting the clog a lot more
heavily than before, as you need to check the clog even if the hint bits
are set, if the XID falls between xmin and xmax of the snapshot.
* Transactions currently become visible immediately when a WAL record is
inserted, before it's flushed. That's wrong, but shouldn't be difficult
to fix (except for the async commit issue explained above).
- Heikki
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackersAttachments:
csn-1.patch.gz (application/gzip, 102.5K) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: hlinnakangas@vmware.com, alvherre@2ndquadrant.com, andres@2ndquadrant.com, robertmhaas@gmail.com, bruce@momjian.us, rajeev.rastogi@huawei.com, ants@cybertec.at, markus@bluegap.ch
Subject: Re: Proposal for CSN based snapshots
In-Reply-To: <53889CCB.7030005@vmware.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox