X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id 16C76D1D064 for ; Wed, 7 Apr 2004 09:31:35 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 73273-03 for ; Wed, 7 Apr 2004 09:31:35 -0300 (ADT) Received: from anchor-post-33.mail.demon.net (anchor-post-33.mail.demon.net [194.217.242.91]) by svr1.postgresql.org (Postfix) with ESMTP id 5F09FD1BC5C for ; Wed, 7 Apr 2004 09:31:31 -0300 (ADT) Received: from mwynhau.demon.co.uk ([193.237.186.96] helo=mainbox.archonet.com) by anchor-post-33.mail.demon.net with esmtp (Exim 3.35 #1) id 1BBCDO-000BEf-0X; Wed, 07 Apr 2004 13:31:34 +0100 Received: by mainbox.archonet.com (Postfix, from userid 529) id 2EB95165A8; Wed, 7 Apr 2004 13:31:33 +0100 (BST) Received: from client17.archonet.com (client17.archonet.com [192.168.1.17]) by mainbox.archonet.com (Postfix) with ESMTP id 922E11639A; Wed, 7 Apr 2004 13:31:30 +0100 (BST) From: Richard Huxton To: Ken Geis , pgsql-performance@postgresql.org Subject: Re: plan problem Date: Wed, 7 Apr 2004 13:31:30 +0100 User-Agent: KMail/1.5 References: <40731252.2030002@speakeasy.org> <200404070938.11482.dev@archonet.com> <4073C3DF.2070307@speakeasy.org> In-Reply-To: <4073C3DF.2070307@speakeasy.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200404071331.30925.dev@archonet.com> X-Bogosity: No, tests=bogofilter, spamicity=0.000000, version=0.15.3 X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200404/83 X-Sequence-Number: 6383 On Wednesday 07 April 2004 10:03, Ken Geis wrote: > Richard Huxton wrote: > > On Tuesday 06 April 2004 21:25, Ken Geis wrote: > >>I am trying to find an efficient way to draw a random sample from a > >>complex query. I also want it to be easy to use within my application. > >> > >>So I've defined a view that encapsulates the query. The id in the > >>"driving" table is exposed, and I run a query like: > >> > >>select * from stats_record_view > >> where id in (select id from driver_stats > >> order by random() > >> limit 30000); > > > > How about a join? > > > > SELECT s.* > > FROM > > stats_record_view s > > JOIN > > (SELECT id FROM driver_stats ORDER BY random() LIMIT 30000) AS r > > ON s.id = r.id; > > Yes, I tried this too after I sent the first mail, and this was somewhat > better. I ended up adding a random column to the driving table, putting > an index on it, and exposing that column in the view. Now I can say > > SELECT * FROM stats_record_view WHERE random < 0.093; > > For my application, it's OK if the same sample is picked time after time > and it may change if data is added. Fair enough - that'll certainly do it. > > Also worth checking the various list archives - this has come up in the > > past, but some time ago. > > There are some messages in the archives about how to get a random > sample. I know how to do that, and that's not why I posted my message. > Are you saying that the planner behavior I spoke of is in the > archives? I wouldn't know what to search on to find that thread. Does > anyone think that the planner issue has merit to address? Can someone > help me figure out what code I would look at? I was assuming after getting a random subset they'd see the same problem you are. If not, probably worth looking at. In which case, an EXPLAIN ANALYZE of your original query would be good. -- Richard Huxton Archonet Ltd