pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
To: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers@postgresql.org, Joachim Wieland <joe@mcknight.de>
Cc: Greg Smith <greg@2ndquadrant.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: José Arthur Benetasso Villanova <jose.arthur@gmail.com>
Subject: Re: directory archive format for pg_dump
Date: Fri, 17 Dec 2010 00:34:02 +0200
Message-ID: <4D0A93DA.30308@enterprisedb.com> (raw)
In-Reply-To: <201012162329.51796.andres@anarazel.de>
References: <AANLkTi=AEi3F6jC0Zy3dG9njiwJJC8Y6V090-hMonyyn@mail.gmail.com>
	<4D0A50D6.5070602@enterprisedb.com>
	<AANLkTim8vp_iEvzmWwnOG9--3m3sw3gr_zuMWyK31PkP@mail.gmail.com>
	<201012162329.51796.andres@anarazel.de>

On 17.12.2010 00:29, Andres Freund wrote:
> On Thursday 16 December 2010 19:33:10 Joachim Wieland wrote:
>> On Thu, Dec 16, 2010 at 12:48 PM, Heikki Linnakangas
>>
>> <heikki.linnakangas@enterprisedb.com>  wrote:
>>> As soon as we have parallel pg_dump, the next big thing is going to be
>>> parallel dump of the same table using multiple processes. Perhaps we
>>> should prepare for that in the directory archive format, by allowing the
>>> data of a single table to be split into multiple files. That way
>>> parallel pg_dump is simple, you just split the table in chunks of
>>> roughly the same size, say 10GB each, and launch a process for each
>>> chunk, writing to a separate file.
>>
>> How exactly would you "just split the table in chunks of roughly the
>> same size" ? Which queries should pg_dump send to the backend? If it
>> just sends a bunch of WHERE queries, the server would still scan the
>> same data several times since each pg_dump client would result in a
>> seqscan over the full table.
> I would suggest implementing<  >  support for tidscans and doing it in segment
> size...

I don't think there's any particular gain from matching the server's 
data file segment size, although 1GB does sound like a good chunk size 
for this too.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com



view thread (40+ messages)  latest in thread

Message-ID: <4D0A93DA.30308@enterprisedb.com>
Permalink:  ../4D0A93DA.30308@enterprisedb.com/
Also on:    postgresql.org/message-id/4D0A93DA.30308@enterprisedb.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: heikki.linnakangas@enterprisedb.com, andres@anarazel.de, joe@mcknight.de, greg@2ndquadrant.com, tgl@sss.pgh.pa.us, jose.arthur@gmail.com
  Subject: Re: directory archive format for pg_dump
  In-Reply-To: <4D0A93DA.30308@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox