X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by postgresql.org (Postfix) with ESMTP id 0A3CA9DCAD4 for ; Tue, 29 Nov 2005 01:54:48 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 14873-04 for ; Tue, 29 Nov 2005 01:54:46 -0400 (AST) X-Greylist: from auto-whitelisted by SQLgrey- Received: from floppy.pyrenet.fr (news.pyrenet.fr [194.116.145.2]) by postgresql.org (Postfix) with ESMTP id E036D9DCA9F for ; Tue, 29 Nov 2005 01:54:44 -0400 (AST) Received: by floppy.pyrenet.fr (Postfix, from userid 106) id AEE5431F4D; Tue, 29 Nov 2005 06:54:44 +0100 (MET) From: "Qingqing Zhou" X-Newsgroups: pgsql.hackers Subject: Re: ice-broker scan thread Date: Tue, 29 Nov 2005 00:55:36 -0500 Organization: Hub.Org Networking Services Lines: 34 Message-ID: References: <438BE53B.1090106@boreham.org> X-Complaints-To: usenet@news.hub.org X-Priority: 3 X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2900.2180 X-RFC2646: Format=Flowed; Original X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 To: pgsql-hackers@postgresql.org X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, score=0.046 required=5 tests=[AWL=0.046] X-Spam-Score: 0.046 X-Spam-Level: X-Archive-Number: 200511/1465 X-Sequence-Number: 76747 "David Boreham" wrote >> > I don't think your NT overlapped I/O code is quite right. At least > I think it will issue reads at a high rate without waiting for any of them > to complete. Beyond some point that has to give the kernel gut-rot. > [also with reply to Gavin] look up dictionary for "gut-rot", got it ... Uh, this behavior is intended - I try to push enough requests shortly to kernel so that it understands that I am doing sequential scan, so it would pull the data from disk to file system cache more efficiently. Some file systems may have "free-behind" mechanism, but our main thread (who really process the query) should be fast enough before the data vanished. > > You could re-write your program to have a single thread but use aio. > In that case it should show the same read ahead benefit that you see > with the thread. > I guess this is also Gavin's point - I understand that will be two different methodologies to handle "read-ahead". If no other thread/process involved, then the main thread will be responsible to grab a free buffer page from bufferpool and ask the kernel to put the data there by sync IO (current PostgreSQL does) or async IOs. And that's what I want to avoid. I'd like to use a dedicated thread/process to "break the ice" only, i.e., pull data from disk to file system cache, so that the main thread will only issue *logical* read. Regards, Qingqing