pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: david@lang.hm
To: Matthew Wakeling <matthew@flymine.org>
Cc: pgsql-performance@postgresql.org
Subject: Re: select on 22 GB table causes "An I/O error occured while sending to the backend." exception
Date: Thu, 28 Aug 2008 11:17:18 -0700 (PDT)
Message-ID: <alpine.DEB.1.10.0808281114450.2713@asgard.lang.hm> (raw)
In-Reply-To: <alpine.DEB.1.10.0808280949060.2713@asgard.lang.hm>
References: <BAY124-W57D3654AF1159CAF8EAC19F5660@phx.gbl>
	<82bpze3ou8.fsf@mid.bfk.de>
	<alpine.DEB.1.10.0808271436210.907@asgard.lang.hm>
	<alpine.DEB.1.10.0808281411220.4454@aragorn.flymine.org>
	<alpine.DEB.1.10.0808280949060.2713@asgard.lang.hm>

On Thu, 28 Aug 2008, david@lang.hm wrote:

> On Thu, 28 Aug 2008, Matthew Wakeling wrote:
>
>> On Wed, 27 Aug 2008, david@lang.hm wrote:
>>> if memory overcommit is disabled, the kernel checks to see if you have an 
>>> extra 1G of ram available, if you do it allows the process to continue, if 
>>> you don't it tries to free memory (by throwing away cache, swapping to 
>>> disk, etc), and if it can't free the memory will return a memroy 
>>> allocation error (which I believe will cause firefox to exit).
>> 
>> Remember that the memory overcommit check is checking against the amount of 
>> RAM + swap you have - not just the amount of RAM. When a fork occurs, 
>> hardly any extra actual RAM is used (due to copy on write), but the 
>> potential is there for the process to use it. If overcommit is switched 
>> off, then you just need to make sure there is *plenty* of swap to convince 
>> the kernel that it can actually fulfil all of the memory requests if all 
>> the processes behave badly and all shared pages become unshared. Then the 
>> consequences of processes actually using that memory are that the machine 
>> will swap, rather than the OOM killer having to act.
>
> if you are correct that it just checks against memory+swap then it's not a 
> big deal, but I don't think it does that. I think it actually allocates the 
> memory, and if it does that it will push things out of ram to do the 
> allocation, I don't believe that it will allocate swap space directly.

I just asked on the kernel mailing list and Alan Cox responded.

he is saying that you are correct, it only allocates against the total 
available, it doesn't actually allocate ram.

so with sufficiant swap overcommit off should be fine.

but you do need to allocate more swap as the total memory 'used' can be 
significantly higher that with overcommit on.

David Lang

> David Lang
>
>> Of course, it's generally bad to run a machine with more going on than will 
>> fit in RAM.
>> 
>> Neither swapping nor OOM killing are particularly good - it's just a 
>> consequence of the amount of memory needed being unpredictable.
>> 
>> Probably the best solution is to just tell the kernel somehow to never kill 
>> the postmaster.
>> 
>> Matthew
>> 
>> 
>



view thread (65+ messages)  latest in thread

Message-ID: <alpine.DEB.1.10.0808281114450.2713@asgard.lang.hm>
Permalink:  ../alpine.DEB.1.10.0808281114450.2713@asgard.lang.hm/
Also on:    postgresql.org/message-id/alpine.DEB.1.10.0808281114450.2713@asgard.lang.hm

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-performance@postgresql.org
  Cc: david@lang.hm, matthew@flymine.org
  Subject: Re: select on 22 GB table causes "An I/O error occured while sending to the backend." exception
  In-Reply-To: <alpine.DEB.1.10.0808281114450.2713@asgard.lang.hm>
  References: <BAY124-W57D3654AF1159CAF8EAC19F5660@phx.gbl> <82bpze3ou8.fsf@mid.bfk.de> <alpine.DEB.1.10.0808271436210.907@asgard.lang.hm> <alpine.DEB.1.10.0808281411220.4454@aragorn.flymine.org> <alpine.DEB.1.10.0808280949060.2713@asgard.lang.hm> <alpine.DEB.1.10.0808281114450.2713@asgard.lang.hm>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox