Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 9F76C1337B68 for ; Thu, 16 Dec 2010 18:34:15 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 54582-04 for ; Thu, 16 Dec 2010 22:34:08 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from exprod7og112.obsmtp.com (exprod7og112.obsmtp.com [64.18.2.177]) by mail.postgresql.org (Postfix) with SMTP id B281C1337B49 for ; Thu, 16 Dec 2010 18:34:07 -0400 (AST) Received: from source ([209.85.215.49]) by exprod7ob112.postini.com ([64.18.6.12]) with SMTP ID DSNKTQqT3xirH1xtS+eIHIQHy91q9A1hIvY2@postini.com; Thu, 16 Dec 2010 14:34:08 PST Received: by ewy20 with SMTP id 20so24500ewy.22 for ; Thu, 16 Dec 2010 14:34:06 -0800 (PST) Received: by 10.213.4.209 with SMTP id 17mr864969ebs.38.1292538845504; Thu, 16 Dec 2010 14:34:05 -0800 (PST) Received: from [192.168.1.183] (dsl-hkibrasgw2-ff7ac300-240.dhcp.inet.fi [88.195.122.240]) by mx.google.com with ESMTPS id u1sm2245204eeh.16.2010.12.16.14.34.04 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 16 Dec 2010 14:34:05 -0800 (PST) Message-ID: <4D0A93DA.30308@enterprisedb.com> Date: Fri, 17 Dec 2010 00:34:02 +0200 From: Heikki Linnakangas User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fi-FI; rv:1.9.1.15) Gecko/20101030 Icedove/3.0.10 MIME-Version: 1.0 To: Andres Freund CC: pgsql-hackers@postgresql.org, Joachim Wieland , Greg Smith , Tom Lane , =?ISO-8859-1?Q?Jos=E9_Arthur_Benetasso?= =?ISO-8859-1?Q?_Villanova?= Subject: Re: directory archive format for pg_dump References: <4D0A50D6.5070602@enterprisedb.com> <201012162329.51796.andres@anarazel.de> In-Reply-To: <201012162329.51796.andres@anarazel.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-4.2 tagged_above=-5 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3 X-Spam-Level: X-Archive-Number: 201012/1494 X-Sequence-Number: 176756 On 17.12.2010 00:29, Andres Freund wrote: > On Thursday 16 December 2010 19:33:10 Joachim Wieland wrote: >> On Thu, Dec 16, 2010 at 12:48 PM, Heikki Linnakangas >> >> wrote: >>> As soon as we have parallel pg_dump, the next big thing is going to be >>> parallel dump of the same table using multiple processes. Perhaps we >>> should prepare for that in the directory archive format, by allowing the >>> data of a single table to be split into multiple files. That way >>> parallel pg_dump is simple, you just split the table in chunks of >>> roughly the same size, say 10GB each, and launch a process for each >>> chunk, writing to a separate file. >> >> How exactly would you "just split the table in chunks of roughly the >> same size" ? Which queries should pg_dump send to the backend? If it >> just sends a bunch of WHERE queries, the server would still scan the >> same data several times since each pg_dump client would result in a >> seqscan over the full table. > I would suggest implementing< > support for tidscans and doing it in segment > size... I don't think there's any particular gain from matching the server's data file segment size, although 1GB does sound like a good chunk size for this too. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com