From: Mark Mielke <mark@mark.mielke.cc>
To: Matthew <matthew@flymine.org>
Cc: pgsql-performance@postgresql.org
Subject: Re: RAID arrays and performance
Date: Tue, 04 Dec 2007 11:32:14 -0500
Message-ID: <4755810E.4000707@mark.mielke.cc> (raw)
In-Reply-To: <Pine.LNX.4.58.0712041603490.3731@aragorn.flymine.org>
References: <Pine.LNX.4.58.0712041213490.3731@aragorn.flymine.org>
<87lk8aiqf3.fsf@oxford.xeocode.com>
<Pine.LNX.4.58.0712041247540.3731@aragorn.flymine.org>
<475559F4.7060104@mark.mielke.cc>
<Pine.LNX.4.58.0712041349380.3731@aragorn.flymine.org>
<4755648C.90905@mark.mielke.cc>
<Pine.LNX.4.58.0712041454110.3731@aragorn.flymine.org>
<87bq96ii3b.fsf@oxford.xeocode.com>
<Pine.LNX.4.58.0712041603490.3731@aragorn.flymine.org>
Matthew wrote:
> On Tue, 4 Dec 2007, Gregory Stark wrote:
>
>> Fwiw, what made you bring up this topic now? You're the second person in about
>> two days to bring up precisely this issue and it was an issue I had been
>> planning to bring up on -hackers as it was.
>>
> I only just joined the performance mailing list to talk about R-trees. I
> would probably have brought it up earlier if I had been here earlier.
> However, we're thinking of buying this large machine, and that reminded
> me.
>
> I have been biting at the bit for my bosses to allow me time to write an
> indexing system for transient data - a lookup table backed by disc,
> looking up from an integer to get an object, native in Java. Our system
> often needs to fetch a list of a thousand different objects by a key like
> that, and Postgres just doesn't do that exact thing fast. I was going to
> implement it with full asynchronous IO, to do that particular job very
> fast, so I have done a reasonable amount of research into the topic. In
> Java, that is. It would add a little bit more performance for our system.
> That wouldn't cover us - we still need to do complex queries with the same
> problem, and that'll have to stay in Postgres
So much excitement and zeal - refreshing to see. And yet, no numbers! :-)
You describe a new asynchronous I/O system to map integers to Java
objects above. Why would you write this? Have you tried BerkeleyDB or
BerkeleyDB JE? BerkeleyDB with BDB as a backend or their full Java
backend gives you a Java persistence API that will allow you to map Java
objects (including integers) to other Java objects. They use generated
Java run time instructions instead of reflection to store and lock your
Java objects. If it came to a bet, I would bet that their research and
tuning over several years, and many people, would beat your initial
implementation, asynchronous I/O or not.
Asynchronous I/O is no more a magic bullet than threading. It requires a
lot of work to get it right, and if one gets it wrong, it can be slower
than the regular I/O or single threaded scenarios. Both look sexy on
paper, neither may be the solution to your problem. Or they may be. We
wouldn't know without numbers.
Cheers,
mark
--
Mark Mielke <mark@mielke.cc>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: mark@mark.mielke.cc, matthew@flymine.org
Subject: Re: RAID arrays and performance
In-Reply-To: <4755810E.4000707@mark.mielke.cc>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox