X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by postgresql.org (Postfix) with ESMTP id B6D339DCCAF for ; Wed, 30 Nov 2005 10:38:38 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 21454-02 for ; Wed, 30 Nov 2005 10:38:41 -0400 (AST) X-Greylist: from auto-whitelisted by SQLgrey- Received: from toad.mtbrook.bozemanpass.com (toad.mtbrook.bozemanpass.com [69.145.82.195]) by postgresql.org (Postfix) with ESMTP id 123D49DCC3E for ; Wed, 30 Nov 2005 10:38:36 -0400 (AST) Received: from [69.145.82.253] (unknown [69.145.82.253]) by toad.mtbrook.bozemanpass.com (Postfix) with ESMTP id BE71011027B for ; Wed, 30 Nov 2005 06:38:40 -0800 (PST) Message-ID: <438DB972.5050201@boreham.org> Date: Wed, 30 Nov 2005 07:38:42 -0700 From: David Boreham User-Agent: Mozilla Thunderbird 1.0.7 (Windows/20050923) X-Accept-Language: en-us, en MIME-Version: 1.0 To: pgsql-hackers@postgresql.org Subject: Re: ice-broker scan thread References: <438BE53B.1090106@boreham.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, score=0.017 required=5 tests=[AWL=0.017] X-Spam-Score: 0.017 X-Spam-Level: X-Archive-Number: 200511/1533 X-Sequence-Number: 76815 Qingqing Zhou wrote: >[also with reply to Gavin] look up dictionary for "gut-rot", got it ... Uh, >this behavior is intended - I try to push enough requests shortly to kernel >so that it understands that I am doing sequential scan, so it would pull the >data from disk to file system cache more efficiently. Some file systems may >have "free-behind" mechanism, but our main thread (who really process the >query) should be fast enough before the data vanished. > > I guess I was concerned that very large numbers of concurrent operations on the same file handle in flight at the same time might lead to poor performance or even instability. e.g. the kernel may make long linked lists, it might create lock contention with itself, that kind of bad stuff. My thinking being that the kernel wasn't designed with applications that fire off 10,000 concurrent reads against the same file. >I guess this is also Gavin's point - I understand that will be two different >methodologies to handle "read-ahead". If no other thread/process involved, >then the main thread will be responsible to grab a free buffer page from >bufferpool and ask the kernel to put the data there by sync IO (current >PostgreSQL does) or async IOs. And that's what I want to avoid. I'd like to >use a dedicated thread/process to "break the ice" only, i.e., pull data from >disk to file system cache, so that the main thread will only issue *logical* >read. > > Right, understood. My point was that a thread with sync I/O and the query thread with async I/O are in fact logically identical. They're just two different implementation techniques for the same fundemental functionality. In some cases the non-thread implementation might be prefered (for example on a platform with no threads).