From: Andres Freund <andres@2ndquadrant.com>
To: Jim Nasby <jim@nasby.net>
Cc: Ants Aasma <ants@cybertec.at>
Cc: Merlin Moncure <mmoncure@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Atri Sharma <atri.jiit@gmail.com>
Cc: Greg Stark <stark@mit.edu>
Cc: Amit Kapila <amit.kapila@huawei.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Page replacement algorithm in buffer cache
Date: Tue, 2 Apr 2013 12:32:39 +0200
Message-ID: <20130402103239.GC2415@alap2.anarazel.de> (raw)
In-Reply-To: <515A1093.8090403@nasby.net>
References: <CAOeZVicGOGQZJZKFXFYjm0wDFA63+7cmzzGQ63nP+165-5qiLQ@mail.gmail.com>
<CAHyXU0xMFvr2eiR+2Lv=dj+UON+muei_pjbUP2BMetiR-UTASw@mail.gmail.com>
<29181.1363981925@sss.pgh.pa.us>
<CAHyXU0yKC-HWk9hhBjUCPsnPMqhskvu_MEqB2cvmRu0d0+BsCw@mail.gmail.com>
<29748.1363983378@sss.pgh.pa.us>
<CAHyXU0wKmB5WXS+hAt2QmGfpTNdsz4Dx1nR3GCKvsNBOzt2ZbQ@mail.gmail.com>
<CA+CSw_sMmHt-Up+SiyL4OPkrgfECQVnMeOpyJz+=M7x=DNTzWg@mail.gmail.com>
<514D29CB.5080600@nasby.net>
<CA+CSw_tHoPr8p0HDBOOCvJeNV7n1AgQMFP8zG=T_5MYbWOKVmw@mail.gmail.com>
<515A1093.8090403@nasby.net>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
On 2013-04-01 17:56:19 -0500, Jim Nasby wrote:
> On 3/23/13 7:41 AM, Ants Aasma wrote:
> >Yes, having bgwriter do the actual cleaning up seems like a good idea.
> >The whole bgwriter infrastructure will need some serious tuning. There
> >are many things that could be shifted to background if we knew it
> >could keep up, like hint bit setting on dirty buffers being flushed
> >out. But again, we have the issue of having good tests to see where
> >the changes hurt.
>
> I think at some point we need to stop depending on just bgwriter for all this stuff. I believe it would be much cleaner if we had separate procs for everything we needed (although some synergies might exist; if we wanted to set hint bits during write then bgwriter *is* the logical place to put that).
>
> In this case, I don't think keeping stuff on the free list is close enough to checkpoints that we'd want bgwriter to handle both. At most we might want them to pass some metrics back in forth.
bgwriter isn't doing checkpoints anymore, there's the checkpointer since 9.2.
In my personal experience and measurement bgwriter is pretty close to
useless right now. I think - pretty similar to what Amit has done - it
should perform part of a real clock sweep instead of just looking ahead
of the current position without changing usagecounts and the sweep
position and put enough buffers on the freelist to sustain the need till
its next activity phase. I hacked around that one night in a hotel and
got impressive speedups (and quite some breakage) for bigger than s_b
workloads.
That would reduce quite a bit of pain points:
- fewer different processes/cpus looking at buffer headers ahead in the cycle
- fewer cpus changing usagecounts
- dirty pages are far more likely to be flushed out already when a new
page is needed
- stuff like the relation extension lock which right now frequently have
to search far and wide for new pages while holding the extension lock
exlusively should finish quite a bit faster
If the freelist lock is separated from the lock protecting the clock
sweep this should get quite a bit of a scalability boost without having
potential unfairness you can have with partitioning the lock or such.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: andres@2ndquadrant.com, jim@nasby.net, ants@cybertec.at, mmoncure@gmail.com, tgl@sss.pgh.pa.us, atri.jiit@gmail.com, stark@mit.edu, amit.kapila@huawei.com
Subject: Re: Page replacement algorithm in buffer cache
In-Reply-To: <20130402103239.GC2415@alap2.anarazel.de>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox