X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 8E2EC329E01 for ; Mon, 13 Sep 2004 14:59:40 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 02838-08 for ; Mon, 13 Sep 2004 13:59:32 +0000 (GMT) Received: from boutiquenumerique.com (gailleton-2-82-67-9-10.fbx.proxad.net [82.67.9.10]) by svr1.postgresql.org (Postfix) with ESMTP id CDCD6329E1C for ; Mon, 13 Sep 2004 14:59:32 +0100 (BST) Received: (qmail 2431 invoked from network); 13 Sep 2004 15:59:42 +0200 Received: from unknown (HELO musicbox) (192.168.0.2) by gailleton-2-82-67-9-10.fbx.proxad.net with SMTP; 13 Sep 2004 15:59:42 +0200 To: pgsql-performance@postgresql.org Subject: Re: Help with extracting large volumes of records across related tables References: <005801c49997$d4a91a20$6e01a8c0@a31p005> Message-ID: Date: Mon, 13 Sep 2004 16:00:35 +0200 From: =?iso-8859-15?Q?Pierre-Fr=E9d=E9ric_Caillaud?= Organization: =?iso-8859-15?Q?La_Boutique_Num=E9rique?= Content-Type: text/plain; format=flowed; delsp=yes; charset=iso-8859-15 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit In-Reply-To: <005801c49997$d4a91a20$6e01a8c0@a31p005> User-Agent: Opera M2/7.53 (Linux, build 737) X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200409/88 X-Sequence-Number: 8216 Thanks for the thanks ! Generally, when grouping stuff together, it is a good idea to have two sorted lists, and to scan them simultaneously. I have already used this solution several times outside of Postgres, and it worked very well (it was with Berkeley DB and there were 3 lists to scan in order). The fact that Python can very easily virtualize these lists using generators makes it possible to do it without consuming too much memory. > Pierre-Frederic, Paul, > > Thanks for your fast response (especially for the python code and > performance figure) - I'll chase this up as a solution - looks most > promising! > > Cheers, > > Damien > > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org >