Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 5DF871337B4A for ; Thu, 16 Dec 2010 14:05:10 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 00552-01 for ; Thu, 16 Dec 2010 18:05:03 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from exprod7og124.obsmtp.com (exprod7og124.obsmtp.com [64.18.2.26]) by mail.postgresql.org (Postfix) with SMTP id 8B3691336B42 for ; Thu, 16 Dec 2010 14:05:02 -0400 (AST) Received: from source ([209.85.215.172]) (using TLSv1) by exprod7ob124.postini.com ([64.18.6.12]) with SMTP ID DSNKTQpUzVqOb0Xrkot0bwgbkWBS6ucs32jG@postini.com; Thu, 16 Dec 2010 10:05:02 PST Received: by mail-ey0-f172.google.com with SMTP id 10so2095396eyd.3 for ; Thu, 16 Dec 2010 10:05:01 -0800 (PST) Received: by 10.213.33.201 with SMTP id i9mr660990ebd.92.1292522701198; Thu, 16 Dec 2010 10:05:01 -0800 (PST) Received: from [192.168.1.183] (dsl-hkibrasgw2-ff7ac300-240.dhcp.inet.fi [88.195.122.240]) by mx.google.com with ESMTPS id q58sm2048111eeh.15.2010.12.16.10.04.58 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 16 Dec 2010 10:04:59 -0800 (PST) Message-ID: <4D0A54C6.3090502@enterprisedb.com> Date: Thu, 16 Dec 2010 20:04:54 +0200 From: Heikki Linnakangas User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fi-FI; rv:1.9.1.15) Gecko/20101030 Icedove/3.0.10 MIME-Version: 1.0 To: Robert Haas CC: Joachim Wieland , Greg Smith , Tom Lane , =?ISO-8859-1?Q?Jos=E9_Arthur_Benetasso_Villanova?= , pgsql-hackers@postgresql.org Subject: Re: directory archive format for pg_dump References: <4CEA9F29.5060106@enterprisedb.com> <28890.1290445623@sss.pgh.pa.us> <4CEAD61A.6060207@enterprisedb.com> <4CF40B5F.1010407@enterprisedb.com> <4CF655A3.4090308@enterprisedb.com> <4CF6562E.60506@enterprisedb.com> <4CF74F99.4050008@enterprisedb.com> <4CF7F8FB.1040305@enterprisedb.com> <4D09E624.4030706@2ndquadrant.com> <4D0A2EFF.9020104@enterprisedb.com> <4D0A50D6.5070602@enterprisedb.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-4.2 tagged_above=-5 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3 X-Spam-Level: X-Archive-Number: 201012/1462 X-Sequence-Number: 176724 On 16.12.2010 19:58, Robert Haas wrote: > On Thu, Dec 16, 2010 at 12:48 PM, Heikki Linnakangas > wrote: >> One more thing: the motivation behind this patch is to allow parallel >> pg_dump in the future, so we should be make sure this patch caters well for >> that. >> >> As soon as we have parallel pg_dump, the next big thing is going to be >> parallel dump of the same table using multiple processes. Perhaps we should >> prepare for that in the directory archive format, by allowing the data of a >> single table to be split into multiple files. That way parallel pg_dump is >> simple, you just split the table in chunks of roughly the same size, say >> 10GB each, and launch a process for each chunk, writing to a separate file. >> >> It should be a quite simple add-on to the current patch, but will make life >> so much easier for parallel pg_dump. It would also be helpful to work around >> file size limitations on some filesystems. > > Sounds reasonable. Are you planning to do this and commit? I'll defer to Joachim, assuming he has the time & energy. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com