X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id DA1628B9ED8 for ; Tue, 15 Feb 2005 21:51:52 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 28450-03 for ; Tue, 15 Feb 2005 21:51:44 +0000 (GMT) Received: from boutiquenumerique.com (boutiquenumerique.com [82.67.9.10]) by svr1.postgresql.org (Postfix) with ESMTP id CACAC8B9E19 for ; Tue, 15 Feb 2005 21:51:42 +0000 (GMT) Received: (qmail 10233 invoked from network); 15 Feb 2005 22:52:14 +0100 Received: from unknown (HELO musicbox) (boutiquenumerique-lists@192.168.0.2) by boutiquenumerique.com with SMTP; 15 Feb 2005 22:52:14 +0100 Date: Tue, 15 Feb 2005 22:55:32 +0100 To: "Magnus Hagander" , "Merlin Moncure" , "Josh Berkus" Subject: Re: seq scan cache vs. index cache smackdown Cc: pgsql-performance@postgresql.org References: <6BCB9D8A16AC4241919521715F4D8BCE4768A7@algol.sollentuna.se> From: PFC Organization: =?iso-8859-15?Q?La_Boutique_Num=E9rique?= Content-Type: text/plain; format=flowed; delsp=yes; charset=iso-8859-15 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID: In-Reply-To: <6BCB9D8A16AC4241919521715F4D8BCE4768A7@algol.sollentuna.se> User-Agent: Opera M2/7.54u1 (Linux, build 892) X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.004 tagged_above=0 required=5 tests=AWL X-Spam-Level: X-Archive-Number: 200502/245 X-Sequence-Number: 10574 In the 'wishful hand waving' department : read index -> determine (tuple id,page) to hit in table -> for each of these, tell the OS 'I'm gonna need these' via a NON BLOCKING call. Non blocking because you feed the information to the OS as you read the index, streaming it. Meanwhile, the OS accumulates the requests in an internal FIFO, reorganizes them according to the order best suited to good disk head movements, then reads them in clusters, and calls a callback inside the application when it has data available. Or the application polls it once in a while to get a bucketload of pages. The 'I'm gonna need these()' syscall would also sometimes return 'hey, I'm full, read the pages I have here waiting for you before asking for new ones'. A flag would tell the OS if the application wanted the results in any order, or with order preserved. Without order preservation, if the application has requested twice the same page with different tuple id's, the OS would call the callback only once, giving it a list of the tuple id's associated with that page. It involves a tradeoff between memory and performance : as the size of the FIFO increases, likelihood of good contiguous disk reading increases. However, the memory structure would only contain page numbers and tuple id's, so it could be pretty small. Returning the results in-order would also need more memory. It could be made very generic if instead of 'tuple id' you read 'opaque application data', and instead of 'page' you read '(offset, length)'. This structure actually exists already in the Linux Kernel, it's called the Elevator or something, but it works for scheduling reads between threads. You can also read 'internal not yet developed postgres cache manager' instead of OS if you don't feel like talking kernel developers into implementing this thing. > (Those are ReadFileScatter and WriteFileGather) >