Message-ID: <45614E3F.4080008@modgraph-usa.com>
Date: Sun, 19 Nov 2006 22:42:07 -0800
From: "Craig A. James" <cjames@modgraph-usa.com>
User-Agent: Thunderbird 1.5.0.8 (X11/20061025)
MIME-Version: 1.0
To: Bruno Wolff III <bruno@wolff.to>,
	"Craig A. James" <cjames@modgraph-usa.com>,
	pgsql-performance@postgresql.org,  mstone+postgres@mathom.us
Subject: Re: Postgres server crash
References: <19583.1163700841@sss.pgh.pa.us>
	<Pine.LNX.4.33.0611181713170.19262-100000@denzel.in>
	<20061119184105.GC2366@mathom.us>
	<4560D6B1.8060306@modgraph-usa.com>
	<20061120015952.GA10935@wolff.to>
In-Reply-To: <20061120015952.GA10935@wolff.to>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

We're getting off-topic for this group except that this is *why* we're plagued with this problem, so I'll make one more observation.

Bruno Wolff III wrote:
>> They would allocate 
>> 100's of GB of swap space rather than taking the time to design proper data 
>> structures.  If the program crashed every week or two, no big deal -- just 
>> run it again.  Overallocating memory is a valuable technique for such 
>> applications.
> 
> I don't think the above applies generally. Programmers need to be aware of
> the working set of CPU bound apps. If the program is constantly paging,
> the performance is going to be abysmal.

You're doing planetary number crunching, so you allocate an (x,y,z) space with 10^6 points on every axis, or 10^18 points, for roughly 2^60 64-bit floating-point numbers, or 2^68 bytes.  But your space is mostly empty (just like real space).  Where there's a planet or star, you're actually using memory, everywhere else, the memory is never referenced.   So you're using just an ordinary amount of memory, but you can access your planet's data by simply referencing a three-dimensional array in FORTRAN.  Just allow infinite overcommit, let the OS do the rest, and it actually works.  A lot of molecular modeling works this way too.

This is how these applications were/are actually written.  I'm not defending the method, just point out that it's real and it's one of the reasons virtual swap was invented.

Craig