DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com;
	h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
	b=G8JAG1DJcegHVZbP1slMm6bxbmNjyRI343NNLB4Vna9QXGQJzlFfV0oxmiL/rFjDOJVsdewX4qOGfh+ghT7zz05cjb4a67a/TT19KrMHAdSdNedji+/vb4xUz3bH/NJimRngpPFmnEYHLbO7YaDkQx4TSbFZ/ZTFAE0MDWGsYTY=
Message-ID: <758d5e7f0605090943p1bbfdae2p4cb28ba128288316@mail.gmail.com>
Date: Tue, 9 May 2006 18:43:23 +0200
From: "Dawid Kuroczko" <qnex42@gmail.com>
To: PFC <lists@peufeu.com>
Subject: Re: [HACKERS] Big IN() clauses etc : feature proposal
Cc: "Christian Kratzer" <ck@cksoft.de>, "Tom Lane" <tgl@sss.pgh.pa.us>,
	pgsql-performance@postgresql.org, pgsql-hackers@postgresql.org
In-Reply-To: <op.s8916gm6cigqcu@apollo13>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
References: <445F7FF8.6070707@comcast.net>
	<1147110142.32369.7.camel@archimedes>
	<20060508175013.GY99570@pervasive.com>
	<28288.1147111196@sss.pgh.pa.us> <op.s89zl30wcigqcu@apollo13>
	<20060509105214.P90693@vesihiisi.cksoft.de>
	<op.s8916gm6cigqcu@apollo13>

On 5/9/06, PFC <lists@peufeu.com> wrote:
> > You might consider just selecting your primary key or a set of
> > primary keys to involved relations in your search query.  If you
> > currently use "select *" this can make your result set very large.
> >
> > Copying all the result set to the temp. costs you additional IO
> > that you propably dont need.
>
>         It is a bit of a catch : I need this information, because the pur=
pose of
> the query is to retrieve these objects. I can first store the ids, then
> retrieve the objects, but it's one more query.
>
> > Also you might try:
> >       SELECT * FROM somewhere JOIN result USING (id)
> > Instead of:
> >       SELECT * FROM somewhere WHERE id IN (SELECT id FROM result)
>
>         Yes you're right in this case ; however the query to retrieve the=
 owners
> needs to eliminate duplicates, which IN() does.

Well, you can either
  SELECT * FROM somewhere JOIN (SELECT id FROM result GROUP BY id) AS
a USING (id);
or even, for large number of ids:
  CREATE TEMPORARY TABLE result_ids AS SELECT id FROM RESULT GROUP BY id;
  SELECT * FROM somewhere JOIN result_ids USING (id);


> > On the other hand if your search query runs in 10ms it seems to be fast
> > enough for you to run it multiple times.  Theres propably no point in
> > optimizing anything in such case.
>
>         I don't think so :
>         - 10 ms is a mean time, sometimes it can take much more time, som=
etimes
> it's faster.
>         - Repeating the query might yield different results if records we=
re added
> or deleted in the meantime.

You may SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
though locking might bite you. :)

>         - Complex search queries have imprecise rowcount estimates ; henc=
e the
> joins that I would add to them will get suboptimal plans.
>
>         Using a temp table is really the cleanest solution now ; but it's=
 too
> slow so I reverted to generating big IN() clauses in the application.

A thought, haven't checked it though, but...

You might want to use PL to store values, say PLperl, or even C, say:

create or replace function perl_store(name text, val int) returns void
as $$ my $name =3D shift; push @{$foo{$name}}, shift; return $$ LANGUAGE
plperl;

select perl_store('someids', id) from something group by id;
(you may need to warp it inside count())

Then use it:

create or replace function perl_retr(name text) returns setof int as
$$ my $name =3D shift; return $foo{$name} $$ LANGUAGE plperl;

select * from someother join perl_retr('someids') AS a(id) using (id);

All is in the memory.  Of course, you need to do some cleanup, test it,
etc, etc, etc. :)

Should work faster than a in-application solution :)

  Regards,
      Dawid