agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedFrom: Hannu Krosing <hannu@2ndQuadrant.com>
To: Dimitri Fontaine <dimitri@2ndQuadrant.fr>
Cc: Josh Berkus <josh@agliodbs.com>
Cc: postgres hackers <pgsql-hackers@postgresql.org>
Subject: Re: Simplifying replication
Date: Sat, 06 Nov 2010 00:39:01 +0200
Message-ID: <1288996741.2313.370.camel@hvost> (raw)
In-Reply-To: <m28w1ushwz.fsf@2ndQuadrant.fr>
References: <4CBCE356.2080703@agliodbs.com>
<m28w1ushwz.fsf@2ndQuadrant.fr>
On Tue, 2010-10-19 at 10:31 +0200, Dimitri Fontaine wrote:
> > 4. I can start a new replica off the master by running a single command-line
> > utility on the standby and giving it connection information to the master.
> > Using this connection, it should be able to start a backup snapshot, copy
> > the entire database and any required logs, and then come up in standby mode.
> > All that should be required for this is one or two highport connections to
> > the master. No recovery.conf file is required, or exists.
>
> There's a prototype to stream a base backup from a libpq connection, I
> think someone here wanted to integrate that into the replication
> protocol itself. It should be doable with a simple libpq connection and
> all automated.
>
> The pg_basebackup python client software is 100 lines of code. It's
> mainly a recursive query to get the list of files within the master,
> then two server side functions to get binary file chunks,
> compressed. Then client side, a loop to decompress and write the chunks
> at the right place. That's it.
>
> http://github.com/dimitri/pg_basebackup/blob/master/pg_basebackup.py
>
> I could prepare a patch given some advice on the replication protocol
> integration. For one, is streaming a base backup something that
> walsender should care about?
To make pg_basebackup.py self-sufficient it should also open 2nd
connection to the same master and make sure that all WAL files are
copied for the duration of base copy.
This way you don't need to do anything extra to make sure you have
enough wal files. And if you fail because of filling up disks, you fail
on slave side, where it is less of a problem.
> >
--
-------
Hannu Krosing
PostgreSQL Infinite Scalability and Preformance Consultant
PG Admin Book: http://www.2ndQuadrant.com/books/
view thread (50+ messages) latest in thread
Message-ID: <1288996741.2313.370.camel@hvost>
Permalink: ../1288996741.2313.370.camel@hvost/
Also on: postgresql.org/message-id/1288996741.2313.370.camel@hvost
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: hannu@2ndQuadrant.com, dimitri@2ndQuadrant.fr, josh@agliodbs.com
Subject: Re: Simplifying replication
In-Reply-To: <1288996741.2313.370.camel@hvost>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox