From: Dimitri Fontaine <dimitri@2ndQuadrant.fr>
To: Hannu Krosing <hannu@2ndQuadrant.com>
Cc: Dimitri Fontaine <dimitri@2ndQuadrant.fr>,
 Josh Berkus <josh@agliodbs.com>,
	postgres hackers <pgsql-hackers@postgresql.org>
Subject: Re: Simplifying replication
Organization: 2ndQuadrant
References: <4CBCE356.2080703@agliodbs.com> <m28w1ushwz.fsf@2ndQuadrant.fr>
	<1288996741.2313.370.camel@hvost> <m2pquicrn1.fsf@2ndQuadrant.fr>
	<1289067736.2313.1896.camel@hvost>
User-Mail-Address: dimitri@2ndQuadrant.fr
Date: Sat, 06 Nov 2010 23:52:29 +0100
In-Reply-To: <1289067736.2313.1896.camel@hvost> (Hannu Krosing's message of
	"Sat, 06 Nov 2010 20:22:16 +0200")
Message-ID: <m2eiayawuq.fsf@2ndQuadrant.fr>
User-Agent: Gnus/5.110011 (No Gnus v0.11) Emacs/24.0.50 (darwin)
MIME-Version: 1.0
Content-Type: text/plain

Hannu Krosing <hannu@2ndQuadrant.com> writes:
>> > To make pg_basebackup.py self-sufficient it should also open 2nd
>> > connection to the same master and make sure that all WAL files are
>> > copied for the duration of base copy.

Done now, please have a look and try it if possible:

  https://github.com/dimitri/pg_basebackup

> btw, as next step you could backport this to 8.x and have most of the
> benefits of SR. It should not be very hard to keep track of wal position
> inside a pl/pythonu function and send one or more records back in form of

You now have the -x and -D options to set that up, but I didn't check
the backport part: it still depends on the pg_bb_list_files() function
to get the recursive listing of the pg_xlog directory, and it does that
using a WITH RECURSIVE query.

The way I did it is to only copy the (whole) WAL again if its ctime
changed since last loop. Also pg_basebackup won't start a backup if you
run it on its own, I don't think that's what you want here.

Oh, as I needed to fork() a process to care for the pg_xlog in a loop
while the base backup is sill ongoing, I added a -j --jobs option so
that you can hammer the master some more by having more than one process
doing the copying.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support