agora inbox for pgsql-hackers@postgresql.org  

help / color / mirror / Atom feed

From: Dimitri Fontaine <dimitri@2ndQuadrant.fr>
To: Hannu Krosing <hannu@2ndQuadrant.com>
Cc: Dimitri Fontaine <dimitri@2ndQuadrant.fr>
Cc: Josh Berkus <josh@agliodbs.com>
Cc: postgres hackers <pgsql-hackers@postgresql.org>
Subject: Re: Simplifying replication
Date: Sat, 06 Nov 2010 23:52:29 +0100
Message-ID: <m2eiayawuq.fsf@2ndQuadrant.fr> (raw)
In-Reply-To: <1289067736.2313.1896.camel@hvost>
References: <4CBCE356.2080703@agliodbs.com>
	<m28w1ushwz.fsf@2ndQuadrant.fr>
	<1288996741.2313.370.camel@hvost>
	<m2pquicrn1.fsf@2ndQuadrant.fr>
	<1289067736.2313.1896.camel@hvost>

Hannu Krosing <hannu@2ndQuadrant.com> writes:
>> > To make pg_basebackup.py self-sufficient it should also open 2nd
>> > connection to the same master and make sure that all WAL files are
>> > copied for the duration of base copy.

Done now, please have a look and try it if possible:

  https://github.com/dimitri/pg_basebackup

> btw, as next step you could backport this to 8.x and have most of the
> benefits of SR. It should not be very hard to keep track of wal position
> inside a pl/pythonu function and send one or more records back in form of

You now have the -x and -D options to set that up, but I didn't check
the backport part: it still depends on the pg_bb_list_files() function
to get the recursive listing of the pg_xlog directory, and it does that
using a WITH RECURSIVE query.

The way I did it is to only copy the (whole) WAL again if its ctime
changed since last loop. Also pg_basebackup won't start a backup if you
run it on its own, I don't think that's what you want here.

Oh, as I needed to fork() a process to care for the pg_xlog in a loop
while the base backup is sill ongoing, I added a -j --jobs option so
that you can hammer the master some more by having more than one process
doing the copying.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

view thread (50+ messages)  latest in thread

Message-ID: <m2eiayawuq.fsf@2ndQuadrant.fr>
Permalink:  ../m2eiayawuq.fsf@2ndQuadrant.fr/
Also on:    postgresql.org/message-id/m2eiayawuq.fsf@2ndQuadrant.fr

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: dimitri@2ndQuadrant.fr, hannu@2ndQuadrant.com, josh@agliodbs.com
  Subject: Re: Simplifying replication
  In-Reply-To: <m2eiayawuq.fsf@2ndQuadrant.fr>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox