From: Shaun Thomas <sthomas@optionshouse.com>
To: Brio <brianoraas@gmail.com>
To: pgsql-performance@postgresql.org
Subject: Re: postgres files in use not staying in linux file cache
Date: Tue, 10 Jun 2014 16:07:02 -0500
Message-ID: <53977376.3040406@optionshouse.com> (raw)
In-Reply-To: <CAM+G8pRg9-Dvfr3dam9qG-etQGNGUQEO-cDPyhwR-_HTvWQABA@mail.gmail.com>
References: <CAM+G8pRg9-Dvfr3dam9qG-etQGNGUQEO-cDPyhwR-_HTvWQABA@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
On 06/05/2014 04:32 PM, Brio wrote:
> So here's where I'm stuck. How can reading a file not leave it in the
> Linux cache? I'd expect it to enter the inactive list (which is about
> 80GB), so I'd expect another 80GB would need to be read before it would
> be its turn to be evicted.... which should take a long time if my
> maximum read speed is 100MB/s.
So here's the thing. The Linux page reclamation code is *extremely
broken* in everything before 3.11. Take a look at this, then realize
that this is *only one patch* from several that target the memory
manager weightings:
http://linux-kernel.2935.n7.nabble.com/patch-v2-0-3-mm-improve-page-aging-fairness-between-zones-nod...
This is especially true of the 3.2 kernel you're using. It's extremely
aggressive about ageing pages out of memory when there's high memory
pressure from frequent disk reads. Chances of promotion into the active
set is dismal, so you end up with a constant churn between inactive and
disk. Worse, if you kick it hard enough by having too many PostgreSQL
backends using memory, it'll actively purge the active set while still
failing to promote the inactive set.
The dev that linked me to this patch said he tested it against 3.10,
meaning it probably went into 3.11 or 3.12. So I personally wouldn't
trust anything before 3.13. :p
Since you're using Ubuntu 12.04, I strongly suggest upgrading your core
to 12.04.4 and apply the linux-generic-lts-saucy pseudo-package to at
least get onto the 3.11 instead. The 3.2 kernel is pants-on-head
retarded; we've had a lot more luck with 3.8 and above.
Cheers!
--
Shaun Thomas
OptionsHouse, LLC | 141 W. Jackson Blvd. | Suite 800 | Chicago IL, 60604
312-676-8870
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: sthomas@optionshouse.com, brianoraas@gmail.com
Subject: Re: postgres files in use not staying in linux file cache
In-Reply-To: <53977376.3040406@optionshouse.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox