From: Heikki Linnakangas <hlinnaka@iki.fi>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Alexander Korotkov <a.korotkov@postgrespro.ru>
Cc: Simon Riggs <simon@2ndquadrant.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Jeff Davis <pgsql@j-davis.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: Greg Stark <stark@mit.edu>
Cc: Andres Freund <andres@2ndquadrant.com>
Cc: Rajeev rastogi <rajeev.rastogi@huawei.com>
Cc: Markus Wanner <markus@bluegap.ch>
Cc: Ants Aasma <ants@cybertec.at>
Cc: Bruce Momjian <bruce@momjian.us>
Cc: obartunov <obartunov@postgrespro.ru>
Cc: Teodor Sigaev <teodor@postgrespro.ru>
Subject: Re: Proposal for CSN based snapshots
Date: Mon, 22 Aug 2016 20:32:42 +0300
Message-ID: <ccbbd4e3-2999-cf29-96d0-66935c4ca9aa@iki.fi> (raw)
In-Reply-To: <CA+TgmobaHXcz+5cN9pFS5rVQNkxg9fuCD0poypas_auhUB4VDQ@mail.gmail.com>
References: <51B6C97B.1070000@bluegap.ch>
<53889CCB.7030005@vmware.com>
<20140530152721.GC30516@awork2.anarazel.de>
<5388D02D.6040305@vmware.com>
<CA+TgmoZ4cP_1Pavwn+VX74_ksDhHf-whO5FFR1j+NdZqONu8MA@mail.gmail.com>
<CAM-w4HMG-YeTfWc3r_excej3Kd04q4zDDXR0hxShVbij211ABA@mail.gmail.com>
<539AD153.9000004@vmware.com>
<1409043789.2335.330.camel@jeff-desktop>
<53FC6545.7000501@vmware.com>
<1409116988.2335.463.camel@jeff-desktop>
<53FD7D65.9040602@vmware.com>
<53FD7DCC.5020005@vmware.com>
<CAPpHfdsCUb4qBUHzfqNsUYALC_LEpHD9qB-ozvS_8esv1edCTw@mail.gmail.com>
<CANP8+jLAe3Smtq3FF49DMvuxtfoRQcCPQn28GEKuwAb2J41XCA@mail.gmail.com>
<CA+Tgmob1DmDsvc7iyd0SY5j4SFdesDwoef36bxG1pE_s1JQXOQ@mail.gmail.com>
<CANP8+jLyhL92eJi7RVX_yvbKjxm9=OBKcmNnodaYmeE+o2i1+Q@mail.gmail.com>
<CAPpHfdv7BMwGv=OfUg3S-jGVFKqHi79pR_ZK1Wsk-13oZ+cy5g@mail.gmail.com>
<69041c5a-ec60-3c54-49d0-5df071b27f52@iki.fi>
<389b45d1-f4f8-a333-03ef-efb94c8d6087@iki.fi>
<CA+TgmobaHXcz+5cN9pFS5rVQNkxg9fuCD0poypas_auhUB4VDQ@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
On 08/22/2016 07:49 PM, Robert Haas wrote:
> Nice to see you working on this again.
>
> On Mon, Aug 22, 2016 at 12:35 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> A sequential scan of a table like that with 10 million rows took about 700
>> ms on my laptop, when the hint bits are set, without this patch. With this
>> patch, if there's a snapshot holding back the xmin horizon, so that we need
>> to check the CSN log for every XID, it took about 30000 ms. So we have some
>> optimization work to do :-). I'm not overly worried about that right now, as
>> I think there's a lot of room for improvement in the SLRU code. But that's
>> the next thing I'm going to work.
>
> So the worst case for this patch is obviously bad right now and, as
> you say, that means that some optimization work is needed.
>
> But what about the best case? If we create a scenario where there are
> no open read-write transactions at all and (somehow) lots and lots of
> ProcArrayLock contention, how much does this help?
I ran some quick pgbench tests on my laptop, but didn't see any
meaningful benefit. I think the best I could see is about 5% speedup,
when running "pgbench -S", with 900 idle connections sitting in the
background. On the positive side, I didn't see much slowdown either.
(Sorry, I didn't record the details of those tests, as I was testing
many different options and I didn't see a clear difference either way.)
It seems that Amit's PGPROC batch clearing patch was very effective. I
remember seeing ProcArrayLock contention very visible earlier, but I
can't hit that now. I suspect you'd still see contention on bigger
hardware, though, my laptop has oly 4 cores. I'll have to find a real
server for the next round of testing.
> Because there's only a purpose to trying to minimize the losses if
> there are some gains to which we can look forward.
Aside from the potential performance gains, this slashes a lot of
complicated code:
70 files changed, 2429 insertions(+), 6066 deletions(-)
That removed code is quite mature at this point, and I'm sure we'll add
some code back to this patch as it evolves, but still.
Also, I'm looking forward for a follow-up patch, to track snapshots in
backends at a finer level, so that vacuum could remove tuples more
aggressively, if you have pg_dump running for days. CSN snapshots isn't
a strict requirement for that, but it makes it simpler, when you can
represent a snapshot with a small fixed-size integer.
Yes, seeing some direct performance gains would be nice too.
- Heikki
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: hlinnaka@iki.fi, robertmhaas@gmail.com, a.korotkov@postgrespro.ru, simon@2ndquadrant.com, dilipbalaut@gmail.com, pgsql@j-davis.com, alvherre@2ndquadrant.com, stark@mit.edu, andres@2ndquadrant.com, rajeev.rastogi@huawei.com, markus@bluegap.ch, ants@cybertec.at, bruce@momjian.us, obartunov@postgrespro.ru, teodor@postgrespro.ru
Subject: Re: Proposal for CSN based snapshots
In-Reply-To: <ccbbd4e3-2999-cf29-96d0-66935c4ca9aa@iki.fi>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox