Subject: Re: Proposal for CSN based snapshots
To: Robert Haas <robertmhaas@gmail.com>
References: <51B6C97B.1070000@bluegap.ch> <53889CCB.7030005@vmware.com>
 <20140530152721.GC30516@awork2.anarazel.de> <5388D02D.6040305@vmware.com>
 <CA+TgmoZ4cP_1Pavwn+VX74_ksDhHf-whO5FFR1j+NdZqONu8MA@mail.gmail.com>
 <CAM-w4HMG-YeTfWc3r_excej3Kd04q4zDDXR0hxShVbij211ABA@mail.gmail.com>
 <539AD153.9000004@vmware.com> <1409043789.2335.330.camel@jeff-desktop>
 <53FC6545.7000501@vmware.com> <1409116988.2335.463.camel@jeff-desktop>
 <53FD7D65.9040602@vmware.com> <53FD7DCC.5020005@vmware.com>
 <CAPpHfdsCUb4qBUHzfqNsUYALC_LEpHD9qB-ozvS_8esv1edCTw@mail.gmail.com>
 <CANP8+jLAe3Smtq3FF49DMvuxtfoRQcCPQn28GEKuwAb2J41XCA@mail.gmail.com>
 <CA+Tgmob1DmDsvc7iyd0SY5j4SFdesDwoef36bxG1pE_s1JQXOQ@mail.gmail.com>
 <CANP8+jLyhL92eJi7RVX_yvbKjxm9=OBKcmNnodaYmeE+o2i1+Q@mail.gmail.com>
 <CAPpHfdv7BMwGv=OfUg3S-jGVFKqHi79pR_ZK1Wsk-13oZ+cy5g@mail.gmail.com>
 <69041c5a-ec60-3c54-49d0-5df071b27f52@iki.fi>
 <389b45d1-f4f8-a333-03ef-efb94c8d6087@iki.fi>
 <CA+TgmobaHXcz+5cN9pFS5rVQNkxg9fuCD0poypas_auhUB4VDQ@mail.gmail.com>
From: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Alexander Korotkov <a.korotkov@postgrespro.ru>,
 Simon Riggs <simon@2ndquadrant.com>, Dilip Kumar <dilipbalaut@gmail.com>,
 Jeff Davis <pgsql@j-davis.com>,
 PostgreSQL-development <pgsql-hackers@postgresql.org>,
 Alvaro Herrera <alvherre@2ndquadrant.com>, Greg Stark <stark@mit.edu>,
 Andres Freund <andres@2ndquadrant.com>,
 Rajeev rastogi <rajeev.rastogi@huawei.com>, Markus Wanner
 <markus@bluegap.ch>, Ants Aasma <ants@cybertec.at>,
 Bruce Momjian <bruce@momjian.us>, obartunov <obartunov@postgrespro.ru>,
 Teodor Sigaev <teodor@postgrespro.ru>
Message-ID: <ccbbd4e3-2999-cf29-96d0-66935c4ca9aa@iki.fi>
Date: Mon, 22 Aug 2016 20:32:42 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
 Icedove/45.2.0
MIME-Version: 1.0
In-Reply-To: 
 <CA+TgmobaHXcz+5cN9pFS5rVQNkxg9fuCD0poypas_auhUB4VDQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

On 08/22/2016 07:49 PM, Robert Haas wrote:
> Nice to see you working on this again.
>
> On Mon, Aug 22, 2016 at 12:35 PM, Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> A sequential scan of a table like that with 10 million rows took about 700
>> ms on my laptop, when the hint bits are set, without this patch. With this
>> patch, if there's a snapshot holding back the xmin horizon, so that we need
>> to check the CSN log for every XID, it took about 30000 ms. So we have some
>> optimization work to do :-). I'm not overly worried about that right now, as
>> I think there's a lot of room for improvement in the SLRU code. But that's
>> the next thing I'm going to work.
>
> So the worst case for this patch is obviously bad right now and, as
> you say, that means that some optimization work is needed.
>
> But what about the best case?  If we create a scenario where there are
> no open read-write transactions at all and (somehow) lots and lots of
> ProcArrayLock contention, how much does this help?

I ran some quick pgbench tests on my laptop, but didn't see any 
meaningful benefit. I think the best I could see is about 5% speedup, 
when running "pgbench -S", with 900 idle connections sitting in the 
background. On the positive side, I didn't see much slowdown either. 
(Sorry, I didn't record the details of those tests, as I was testing 
many different options and I didn't see a clear difference either way.)

It seems that Amit's PGPROC batch clearing patch was very effective. I 
remember seeing ProcArrayLock contention very visible earlier, but I 
can't hit that now. I suspect you'd still see contention on bigger 
hardware, though, my laptop has oly 4 cores. I'll have to find a real 
server for the next round of testing.

> Because there's only a purpose to trying to minimize the losses if
> there are some gains to which we can look forward.

Aside from the potential performance gains, this slashes a lot of 
complicated code:

  70 files changed, 2429 insertions(+), 6066 deletions(-)

That removed code is quite mature at this point, and I'm sure we'll add 
some code back to this patch as it evolves, but still.

Also, I'm looking forward for a follow-up patch, to track snapshots in 
backends at a finer level, so that vacuum could remove tuples more 
aggressively, if you have pg_dump running for days. CSN snapshots isn't 
a strict requirement for that, but it makes it simpler, when you can 
represent a snapshot with a small fixed-size integer.

Yes, seeing some direct performance gains would be nice too.

- Heikki


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers