X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 9974732A61F for ; Fri, 15 Oct 2004 02:08:47 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 20828-05 for ; Fri, 15 Oct 2004 01:08:41 +0000 (GMT) Received: from sccrmhc13.comcast.net (sccrmhc13.comcast.net [204.127.202.64]) by svr1.postgresql.org (Postfix) with ESMTP id 6781B32A518 for ; Fri, 15 Oct 2004 02:08:42 +0100 (BST) Received: from sysexperts.com (c-24-6-183-218.client.comcast.net[24.6.183.218]) by comcast.net (sccrmhc13) with ESMTP id <20041015010842016000kbste>; Fri, 15 Oct 2004 01:08:43 +0000 Received: from localhost (localhost [127.0.0.1]) (uid 1000) by filer with local; Thu, 14 Oct 2004 18:08:41 -0700 id 000071AD.416F2319.00001B0A Date: Thu, 14 Oct 2004 18:08:41 -0700 From: Kevin Brown To: pgsql-performance@postgresql.org Subject: Re: mmap (was First set of OSDL Shared Mem scalability results, some wierdness ... Message-ID: <20041015010841.GE665@filer> Mail-Followup-To: Kevin Brown , pgsql-performance@postgresql.org References: <157f64840410141725162e43b5@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline In-Reply-To: <157f64840410141725162e43b5@mail.gmail.com> Organization: Frobozzco International User-Agent: Mutt/1.5.6+20040818i X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200410/203 X-Sequence-Number: 8679 Aaron Werman wrote: > pg to my mind is unique in not trying to avoid OS buffering. Other > dbmses spend a substantial effort to create a virtual OS (task > management, I/O drivers, etc.) both in code and support. Choosing mmap > seems such a limiting an option - it adds OS dependency and limits > kernel developer options (2G limits, global mlock serializations, > porting problems, inability to schedule or parallelize I/O, still > having to coordinate writers and readers). I'm not sure I entirely agree with this. Whether you access a file via mmap() or via read(), the end result is that you still have to access it, and since PG has significant chunks of system-dependent code that it heavily relies on as it is (e.g., locking mechanisms, shared memory), writing the I/O subsystem in a similar way doesn't seem to me to be that much of a stretch (especially since PG already has the storage manager), though it might involve quite a bit of work. As for parallelization of I/O, the use of mmap() for reads should signficantly improve parallelization -- now instead of issuing read() system calls, possibly for the same set of blocks, all the backends would essentially be examining the same data directly. The performance improvements as a result of accessing the kernel's cache pages directly instead of having it do buffer copies to process-local memory should increase as concurrency goes up. But see below. > More to the point, I think it is very hard to effectively coordinate > multithreaded I/O, and mmap seems used mostly to manage relatively > simple scenarios. PG already manages and coordinates multithreaded I/O. The mechanisms used to coordinate writes needn't change at all. But the way reads are done relative to writes might have to be rethought, since an mmap()ed buffer always reflects what's actually in kernel space at the time the buffer is accessed, while a buffer retrieved via read() reflects the state of the file at the time of the read(). If it's necessary for the state of the buffers to be fixed at examination time, then mmap() will be at best a draw, not a win. > mmap doesn't look that promising. This ultimately depends on two things: how much time is spent copying buffers around in kernel memory, and how much advantage can be gained by freeing up the memory used by the backends to store the backend-local copies of the disk pages they use (and thus making that memory available to the kernel to use for additional disk buffering). The gains from the former are likely small. The gains from the latter are probably also small, but harder to estimate. The use of mmap() is probably one of those optimizations that should be done when there's little else left to optimize, because the potential gains are possibly (if not probably) relatively small and the amount of work involved may be quite large. So I agree -- compared with other, much lower-hanging fruit, mmap() doesn't look promising. -- Kevin Brown kevin@sysexperts.com