From: Andres Freund <andres@2ndquadrant.com>
To: Scott Marlowe <scott.marlowe@gmail.com>
Cc: ik@postgresql-consulting.com, Robert Kaye <rob@musicbrainz.org>
Cc: Josh Krupka <jkrupka@gmail.com>
Cc: PostgreSQL Performance <pgsql-performance@postgresql.org>
Subject: Re: MusicBrainz postgres performance issues
Date: Sun, 15 Mar 2015 23:47:56 +0100
Message-ID: <20150315224756.GE29732@awork2.anarazel.de> (raw)
In-Reply-To: <CAOR=d=2rC8dYr=tY7JQTNK+eszx2J1pRHuqYFHPbqxVDXiN1sQ@mail.gmail.com>
References: <B008F2EE-45D7-4535-9163-56A8CDCA553C@musicbrainz.org>
<CAB6McgXxZdD6yuYRLg4tgDdhqzXKLs63aV+S_nOfZ9arSXUAhA@mail.gmail.com>
<9E71CCBE-4CDF-482E-878D-6C16027B6283@musicbrainz.org>
<20150315135022.GC19792@alap3.anarazel.de>
<CAOR=d=0YwcnVi3kGD21W7jYjE7DcWjp0v0GjncH-c2-oJEP5Rw@mail.gmail.com>
<CAOR=d=1Axcbj2VcUO=5tM1nv6RjuA0D+av5+Mt7RTztgTxaMnA@mail.gmail.com>
<20150315172048.GG9324@alap3.anarazel.de>
<CAG95seXv4EKGhDV04rkk_ARgvz8OQUJy3Xbx5Rw16Wxm3+1j5Q@mail.gmail.com>
<20150315174646.GH9324@alap3.anarazel.de>
<CAOR=d=2rC8dYr=tY7JQTNK+eszx2J1pRHuqYFHPbqxVDXiN1sQ@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
On 2015-03-15 12:25:07 -0600, Scott Marlowe wrote:
> Here's the problem with a large shared_buffers on a machine that's
> getting pushed into swap. It starts to swap BUFFERs. Once buffers
> start getting swapped you're not just losing performance, that huge
> shared_buffers is now working against you because what you THINK are
> buffers in RAM to make things faster are in fact blocks on a hard
> drive being swapped in and out during reads. It's the exact opposite
> of fast. :)
IMNSHO that's tackling things from the wrong end. If 12GB of shared
buffers drive your 48GB dedicated OLTP postgres server into swapping out
actively used pages, the problem isn't the 12GB of shared buffers, but
that you require so much memory for other things. That needs to be
fixed.
But! We haven't even established that swapping is an actual problem
here. The ~2GB of swapped out memory could just as well be the java raid
controller management monstrosity or something similar. Those pages
won't ever be used and thus can better be used to buffer IO.
You can check what's actually swapped out using:
grep ^VmSwap /proc/[0-9]*/status|grep -v '0 kB'
For swapping to be actually harmful you need to have pages that are
regularly swapped in. vmstat will tell.
In a concurrent OLTP workload (~450 established connections do suggest
that) with a fair amount of data keeping the hot data set in
shared_buffers can significantly reduce problems. Constantly searching
for victim buffers isn't a nice thing, and that will happen if your most
frequently used data doesn't fit into s_b. On the other hand, if your
data set is so large that even the hottest part doesn't fit into memory
(perhaps because there's no hottest part as there's no locality at all),
a smaller shared buffers can make things more efficient, because the
search for replacement buffers is cheaper with a smaller shared buffers
setting.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: andres@2ndquadrant.com, scott.marlowe@gmail.com, rob@musicbrainz.org, jkrupka@gmail.com
Subject: Re: MusicBrainz postgres performance issues
In-Reply-To: <20150315224756.GE29732@awork2.anarazel.de>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox