X-Original-To: pgsql-performance@postgresql.org Received: from localhost (developer.postgresql.org [64.117.224.193]) by developer.postgresql.org (Postfix) with ESMTP id 578E69259DA for ; Wed, 28 May 2003 16:13:19 -0400 (EDT) Received: from developer.postgresql.org ([64.117.224.193]) by localhost (developer.postgresql.org [64.117.224.193:10024]) (amavisd-new) with ESMTP id 28975-03 for ; Wed, 28 May 2003 16:13:18 -0400 (EDT) Received: from sccrmhc03.attbi.com (sccrmhc03.attbi.com [204.127.202.63]) by developer.postgresql.org (Postfix) with ESMTP id 7CAF69258F9 for ; Wed, 28 May 2003 16:13:18 -0400 (EDT) Received: from attbi.com (dtenny.ne.client2.attbi.com[24.60.113.221]) by attbi.com (sccrmhc03) with SMTP id <2003052820131800300ant5ie>; Wed, 28 May 2003 20:13:18 +0000 Message-ID: <3ED5185D.2000102@attbi.com> Date: Wed, 28 May 2003 16:13:17 -0400 From: Dave Tenny User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3.1) Gecko/20030425 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Bruno Wolff III Cc: Shridhar Daithankar , pgsql-performance@postgresql.org Subject: Re: IN list processing performance (yet again) References: <3ED4B0E5.3050703@attbi.com> <200305281846.38928.shridhar_daithankar@nospam.persistent.co.in> <3ED4F8B6.7030905@attbi.com> <20030528184150.GB6118@wolff.to> In-Reply-To: <20030528184150.GB6118@wolff.to> Content-Type: multipart/alternative; boundary="------------010307030702020301050308" X-Virus-Scanned: by amavisd-new X-Archive-Number: 200305/248 X-Sequence-Number: 2027 This is a multi-part message in MIME format. --------------010307030702020301050308 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Bruno Wolff III wrote: >On Wed, May 28, 2003 at 13:58:14 -0400, > Dave Tenny wrote: > > >>A join isn't an option, these elements come a a selection of entity ID's >>that are specific to some client context. >>Some other people suggested joins too. >> >> > >You can union the values together and then join (or use where exists) with the >result. This may not be faster and you may not be able to union several >thousand selects together in a single statement. But it shouldn't be too >much work to test it out. > > I assume you mean something like: test=# select million.id, million.val from million, (select 10000 as a union select 20000 as a) t2 where million.id = t2.a; id | val -------+------- 10000 | 0 20000 | 10000 (2 rows) Ouch! That's deviant. Haven't tried it yet and I cringe at the thought of it, but I might take a run at it. However that's going to run up the buffer space quickly. That was one of my as yet unsnaswered questions, what is the pragmatic buffer size limit for queries? I'm /really/ hoping we'll come up with something better, like an understanding of why IN lists are non-linear in the first place when the column is indexed, and whether it's fixable through some other means or whether it's a bug that should be fixed. After all, I'm trying to support multiple databases, and other databases kick butt on this. It's just postgresql that's having difficulty. (Btw, I've also tried statement batching, but that's a lose for now, at least with the current JDBC drivers and 7.3.2). --------------010307030702020301050308 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit Bruno Wolff III wrote:
On Wed, May 28, 2003 at 13:58:14 -0400,
  Dave Tenny <tenny@attbi.com> wrote:
  
A join isn't an option, these elements come a a selection of entity ID's 
that are specific to some client context.
Some other people suggested joins too. 
    

You can union the values together and then join (or use where exists) with the
result. This may not be faster and you may not be able to union several
thousand selects together in a single statement. But it shouldn't be too
much work to test it out.
  
I assume you mean something like:

test=# select million.id, million.val from million, (select 10000 as a union select 20000 as a) t2 where million.id = t2.a;
  id   |  val
-------+-------
 10000 |     0
 20000 | 10000
(2 rows)

Ouch!  That's deviant.   Haven't tried it yet and I cringe at the thought of it, but I might take a run at it.  However that's going to
run up the buffer space quickly.  That was one of my as yet unsnaswered questions, what is the pragmatic buffer size limit
for queries?

I'm really hoping we'll come up with something better, like an understanding of why IN lists are non-linear in the first
place when the column is indexed, and whether it's fixable through some other means or whether it's a bug that should be fixed.

After all, I'm trying to support multiple databases, and other databases kick butt on this.  It's just postgresql that's
having difficulty.

(Btw, I've also tried statement batching, but that's a lose for now, at least with the current JDBC drivers and 7.3.2).

--------------010307030702020301050308--