From: Craig A. James <cjames@modgraph-usa.com>
To: Bruno Wolff III <bruno@wolff.to>
To: Craig A. James <cjames@modgraph-usa.com>
To: pgsql-performance@postgresql.org
To: mstone+postgres@mathom.us
Subject: Re: Postgres server crash
Date: Sun, 19 Nov 2006 22:42:07 -0800
Message-ID: <45614E3F.4080008@modgraph-usa.com> (raw)
In-Reply-To: <20061120015952.GA10935@wolff.to>
References: <19583.1163700841@sss.pgh.pa.us>
<Pine.LNX.4.33.0611181713170.19262-100000@denzel.in>
<20061119184105.GC2366@mathom.us>
<4560D6B1.8060306@modgraph-usa.com>
<20061120015952.GA10935@wolff.to>
We're getting off-topic for this group except that this is *why* we're plagued with this problem, so I'll make one more observation.
Bruno Wolff III wrote:
>> They would allocate
>> 100's of GB of swap space rather than taking the time to design proper data
>> structures. If the program crashed every week or two, no big deal -- just
>> run it again. Overallocating memory is a valuable technique for such
>> applications.
>
> I don't think the above applies generally. Programmers need to be aware of
> the working set of CPU bound apps. If the program is constantly paging,
> the performance is going to be abysmal.
You're doing planetary number crunching, so you allocate an (x,y,z) space with 10^6 points on every axis, or 10^18 points, for roughly 2^60 64-bit floating-point numbers, or 2^68 bytes. But your space is mostly empty (just like real space). Where there's a planet or star, you're actually using memory, everywhere else, the memory is never referenced. So you're using just an ordinary amount of memory, but you can access your planet's data by simply referencing a three-dimensional array in FORTRAN. Just allow infinite overcommit, let the OS do the rest, and it actually works. A lot of molecular modeling works this way too.
This is how these applications were/are actually written. I'm not defending the method, just point out that it's real and it's one of the reasons virtual swap was invented.
Craig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: cjames@modgraph-usa.com, bruno@wolff.to, mstone+postgres@mathom.us
Subject: Re: Postgres server crash
In-Reply-To: <45614E3F.4080008@modgraph-usa.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox