To: "Jim C. Nasby" <jnasby@pervasive.com>
cc: Mark Lewis <mark.lewis@mir3.com>,
	Jeffrey Tenny <jeffrey.tenny@comcast.net>, pgsql-performance@postgresql.org
Subject: Re: performance question (something to do w/ 
In-reply-to: <20060508175013.GY99570@pervasive.com> 
References: <445F7FF8.6070707@comcast.net>
	<1147110142.32369.7.camel@archimedes>
	<20060508175013.GY99570@pervasive.com>
Comments: In-reply-to "Jim C. Nasby" <jnasby@pervasive.com>
	message dated "Mon, 08 May 2006 12:50:13 -0500"
Date: Mon, 08 May 2006 13:59:56 -0400
Message-ID: <28288.1147111196@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>

"Jim C. Nasby" <jnasby@pervasive.com> writes:
> On Mon, May 08, 2006 at 10:42:21AM -0700, Mark Lewis wrote:
>> Doing a SELECT with a large list of variables inside an IN runs slowly
>> on every database we've tested.  We've tested mostly in Oracle and
>> PostgreSQL, and both get very slow very quickly (actually Oracle refuses
>> to process the query at all after it gets too many bind parameters).
>> 
>> In our case, we have a (potentially quite large) set of external values
>> that we want to look up in the database.  We originally thought that
>> doing a single select with a large IN clause was the way to go, but then
>> we did some performance analysis on the optimal batch size (number of
>> items to include per IN clause), and discovered that for most databases,
>> the optimal batch size was 1.  For PostgreSQL I think it was 2.
>> 
>> The moral of the story is that you're probably better off running a
>> bunch of small selects than in trying to optimize things with one
>> gargantuan select.

> Ever experiment with loading the parameters into a temp table and
> joining to that?

Also, it might be worth re-testing that conclusion with PG CVS tip
(or 8.2 when it comes out).  The reimplementation of IN as = ANY that
I did a couple months ago might well change the results.

			regards, tom lane