Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id EEB9C1336B42 for ; Thu, 16 Dec 2010 13:48:20 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 55415-06 for ; Thu, 16 Dec 2010 17:48:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from exprod7og127.obsmtp.com (exprod7og127.obsmtp.com [64.18.2.210]) by mail.postgresql.org (Postfix) with SMTP id 1025A1336971 for ; Thu, 16 Dec 2010 13:48:12 -0400 (AST) Received: from source ([209.85.215.170]) (using TLSv1) by exprod7ob127.postini.com ([64.18.6.12]) with SMTP ID DSNKTQpQ22+7jWFnhPucUz/nUyW9ItZSrnBb@postini.com; Thu, 16 Dec 2010 09:48:13 PST Received: by mail-ey0-f170.google.com with SMTP id 5so1871138eyf.29 for ; Thu, 16 Dec 2010 09:48:11 -0800 (PST) Received: by 10.213.3.16 with SMTP id 16mr658658ebl.78.1292521691620; Thu, 16 Dec 2010 09:48:11 -0800 (PST) Received: from [192.168.1.183] (dsl-hkibrasgw2-ff7ac300-240.dhcp.inet.fi [88.195.122.240]) by mx.google.com with ESMTPS id x54sm2034139eeh.17.2010.12.16.09.48.10 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 16 Dec 2010 09:48:11 -0800 (PST) Message-ID: <4D0A50D6.5070602@enterprisedb.com> Date: Thu, 16 Dec 2010 19:48:06 +0200 From: Heikki Linnakangas User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fi-FI; rv:1.9.1.15) Gecko/20101030 Icedove/3.0.10 MIME-Version: 1.0 To: Joachim Wieland CC: Greg Smith , Tom Lane , =?ISO-8859-1?Q?Jos=E9_Arthur_Benetasso_Villanova?= , pgsql-hackers@postgresql.org Subject: Re: directory archive format for pg_dump References: <4CEA9F29.5060106@enterprisedb.com> <28890.1290445623@sss.pgh.pa.us> <4CEAD61A.6060207@enterprisedb.com> <4CF40B5F.1010407@enterprisedb.com> <4CF655A3.4090308@enterprisedb.com> <4CF6562E.60506@enterprisedb.com> <4CF74F99.4050008@enterprisedb.com> <4CF7F8FB.1040305@enterprisedb.com> <4D09E624.4030706@2ndquadrant.com> <4D0A2EFF.9020104@enterprisedb.com> In-Reply-To: <4D0A2EFF.9020104@enterprisedb.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-4.2 tagged_above=-5 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3 X-Spam-Level: X-Archive-Number: 201012/1457 X-Sequence-Number: 176719 On 16.12.2010 17:23, Heikki Linnakangas wrote: > On 16.12.2010 12:12, Greg Smith wrote: >> There's a number of small things that I'd like to see improved in new >> rev of this code >> ... > > In addition to those: >... One more thing: the motivation behind this patch is to allow parallel pg_dump in the future, so we should be make sure this patch caters well for that. As soon as we have parallel pg_dump, the next big thing is going to be parallel dump of the same table using multiple processes. Perhaps we should prepare for that in the directory archive format, by allowing the data of a single table to be split into multiple files. That way parallel pg_dump is simple, you just split the table in chunks of roughly the same size, say 10GB each, and launch a process for each chunk, writing to a separate file. It should be a quite simple add-on to the current patch, but will make life so much easier for parallel pg_dump. It would also be helpful to work around file size limitations on some filesystems. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com