Message-ID: <4D0A5E56.1080109@enterprisedb.com>
Date: Thu, 16 Dec 2010 20:45:42 +0200
From: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fi-FI;
	rv:1.9.1.15) Gecko/20101030 Icedove/3.0.10
MIME-Version: 1.0
To: Joachim Wieland <joe@mcknight.de>
CC: Greg Smith <greg@2ndquadrant.com>, Tom Lane <tgl@sss.pgh.pa.us>,
	=?ISO-8859-1?Q?Jos=E9_Arthur_Benetasso_Villanova?=
	<jose.arthur@gmail.com>, pgsql-hackers@postgresql.org
Subject: Re: directory archive format for pg_dump
References: <AANLkTi=AEi3F6jC0Zy3dG9njiwJJC8Y6V090-hMonyyn@mail.gmail.com>
	<AANLkTik+PR2HCsqS7PVAdETPyKSGn2hb-eTeg9QZpRew@mail.gmail.com>
	<4CEA9F29.5060106@enterprisedb.com>	<28890.1290445623@sss.pgh.pa.us>
	<4CEAD61A.6060207@enterprisedb.com>
	<AANLkTi=neaBqvLa2ZnDpuAMSen3P4ZHE8__Q7L2pmjD0@mail.gmail.com>
	<4CF40B5F.1010407@enterprisedb.com>	<4CF655A3.4090308@enterprisedb.com>
	<4CF6562E.60506@enterprisedb.com>
	<AANLkTinauQsCGt-yeyeN_QWuN+tao+q126rwKnO2QWLP@mail.gmail.com>
	<4CF74F99.4050008@enterprisedb.com>	<4CF7F8FB.1040305@enterprisedb.com>
	<AANLkTik=3XBZq-PbC9CNTV1X6+-+Yd9gOdwf7rYjCpBp@mail.gmail.com>
	<4D09E624.4030706@2ndquadrant.com>	<4D0A2EFF.9020104@enterprisedb.com>
	<4D0A50D6.5070602@enterprisedb.com>
	<AANLkTim8vp_iEvzmWwnOG9--3m3sw3gr_zuMWyK31PkP@mail.gmail.com>
In-Reply-To: <AANLkTim8vp_iEvzmWwnOG9--3m3sw3gr_zuMWyK31PkP@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 16.12.2010 20:33, Joachim Wieland wrote:
> On Thu, Dec 16, 2010 at 12:48 PM, Heikki Linnakangas
> <heikki.linnakangas@enterprisedb.com>  wrote:
>> As soon as we have parallel pg_dump, the next big thing is going to be
>> parallel dump of the same table using multiple processes. Perhaps we should
>> prepare for that in the directory archive format, by allowing the data of a
>> single table to be split into multiple files. That way parallel pg_dump is
>> simple, you just split the table in chunks of roughly the same size, say
>> 10GB each, and launch a process for each chunk, writing to a separate file.
>
> How exactly would you "just split the table in chunks of roughly the
> same size" ?

Check pg_class.relpages, and divide that evenly across the processes. 
That should be good enough.

> Which queries should pg_dump send to the backend? If it
> just sends a bunch of WHERE queries, the server would still scan the
> same data several times since each pg_dump client would result in a
> seqscan over the full table.

Hmm, I was thinking of "SELECT * FROM table WHERE ctid BETWEEN ? AND ?", 
but we don't support TidScans for ranges. Perhaps we could add that.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com