From: Greg Smith <greg@2ndQuadrant.com>
To: didier <did447@gmail.com>
Cc: KONDO Mitsumasa <kondo.mitsumasa@lab.ntt.co.jp>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Heikki Linnakangas <hlinnakangas@vmware.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Improvement of checkpoint IO scheduler for stable transaction responses
Date: Sat, 20 Jul 2013 12:28:58 -0400
Message-ID: <51EABACA.70701@2ndQuadrant.com> (raw)
In-Reply-To: <CAJRYxuK9bJptJUrp7A_g8Hh2AHQqNmqvpDKxV=ODfFccqWUtww@mail.gmail.com>
References: <51B5AFB1.5030404@lab.ntt.co.jp>
<51BDCB6C.1090507@vmware.com>
<51BEB6F5.7070907@lab.ntt.co.jp>
<51C40EF4.1080201@lab.ntt.co.jp>
<51C9D03B.10107@vmware.com>
<CA+TgmoZ+ofyg4e+hRBTnnfrC7M2ATX-Pg0k4AtyHx60A+_DFhQ@mail.gmail.com>
<51C9FD5C.5050706@vmware.com>
<CA+TgmoZj-0CFLbjceMtgi1t_=hpfe9z8uSa+D0g77sNYcFy04w@mail.gmail.com>
<51E2F865.8030406@2ndQuadrant.com>
<CA+TgmoY=GTANT6uNV4Ds6H6K0sRuWxSHErOaHSVDLvMPxYwqDA@mail.gmail.com>
<51E80CBC.10204@2ndQuadrant.com>
<51E8F080.4040506@lab.ntt.co.jp>
<51E943A1.9030702@2ndQuadrant.com>
<CAJRYxuK9bJptJUrp7A_g8Hh2AHQqNmqvpDKxV=ODfFccqWUtww@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
On 7/20/13 4:48 AM, didier wrote:
> With your tests did you try to write the hot buffers first? ie buffers
> with a high refcount, either by sorting them on refcount or at least
> sweeping the buffer list in reverse?
I never tried that version. After a few rounds of seeing that all
changes I tried were just rearranging the good and bad cases, I got
pretty bored with trying new changes in that same style.
> by writing to the OS the less likely to be recycle buffers first it may
> have less work to do at fsync time, hopefully they have been written by
> the OS background task during the spread and are not re-dirtied by other
> backends.
That is the theory. In practice write caches are so large now, there is
almost no pressure forcing writes to happen until the fsync calls show
up. It's easily possible to enter the checkpoint fsync phase only to
discover there are 4GB of dirty writes ahead of you, ones that have
nothing to do with the checkpoint's I/O.
Backends are constantly pounding the write cache with new writes in
situations with checkpoint spikes. The writes and fsync calls made by
the checkpoint process are only a fraction of the real I/O going on.
The volume of data being squeezed out by each fsync call is based on
total writes to that relation since the checkpoint. That's connected to
the writes to that relation happening during the checkpoint, but the
checkpoint writes can easily be the minority there.
It is not a coincidence that the next feature I'm working on attempts to
quantify the total writes to each 1GB relation chunk. That's the most
promising path forward on the checkpoint problem I've found.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: greg@2ndQuadrant.com, did447@gmail.com, kondo.mitsumasa@lab.ntt.co.jp, robertmhaas@gmail.com, hlinnakangas@vmware.com
Subject: Re: Improvement of checkpoint IO scheduler for stable transaction responses
In-Reply-To: <51EABACA.70701@2ndQuadrant.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox