pg.ddx.io pgsql-general@postgresql.org mailing list archive
help / color / mirror / Atom feedpg_receivexlog or archive_command
7+ messages / 5 participants
[nested] [flat]
* pg_receivexlog or archive_command
@ 2019-09-23 08:25 Vikas Sharma <shavikas@gmail.com>
2019-09-23 08:55 ` Re: pg_receivexlog or archive_command Andreas Kretschmer <andreas@a-kretschmer.de>
2019-09-23 18:31 ` Re: pg_receivexlog or archive_command Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-10-02 06:16 ` Re: pg_receivexlog or archive_command Stephen Frost <sfrost@snowman.net>
0 siblings, 3 replies; 7+ messages in thread
From: Vikas Sharma @ 2019-09-23 08:25 UTC (permalink / raw)
To: pgsql-general@lists.postgresql.org
Hi,
I am wondering which one is the best way to archive the xlogs for Backup
and Recovery - pg_receivexlog or archive_command.
pg_receivexlog seems best suited because the copied/archived file is
streamed as it is being written to in xlog while archive_command only
copies when the WAL is fully written to.
Best wishes
Vikas
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_receivexlog or archive_command
2019-09-23 08:25 pg_receivexlog or archive_command Vikas Sharma <shavikas@gmail.com>
@ 2019-09-23 08:55 ` Andreas Kretschmer <andreas@a-kretschmer.de>
2019-09-23 11:44 ` Re: pg_receivexlog or archive_command Luca Ferrari <fluca1978@gmail.com>
2 siblings, 1 reply; 7+ messages in thread
From: Andreas Kretschmer @ 2019-09-23 08:55 UTC (permalink / raw)
To: pgsql-general@lists.postgresql.org
Am 23.09.19 um 10:25 schrieb Vikas Sharma:
> Hi,
>
> I am wondering which one is the best way to archive the xlogs for
> Backup and Recovery - pg_receivexlog or archive_command.
>
> pg_receivexlog seems best suited because the copied/archived file is
> streamed as it is being written to in xlog while archive_command only
> copies when the WAL is fully written to.
>
you can use both of them, and you should consider "Barman".
Regards, Andreas
--
2ndQuadrant - The PostgreSQL Support Company.
www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_receivexlog or archive_command
2019-09-23 08:25 pg_receivexlog or archive_command Vikas Sharma <shavikas@gmail.com>
2019-09-23 08:55 ` Re: pg_receivexlog or archive_command Andreas Kretschmer <andreas@a-kretschmer.de>
@ 2019-09-23 11:44 ` Luca Ferrari <fluca1978@gmail.com>
2019-09-23 11:56 ` Re: pg_receivexlog or archive_command Andreas Kretschmer <andreas@a-kretschmer.de>
0 siblings, 1 reply; 7+ messages in thread
From: Luca Ferrari @ 2019-09-23 11:44 UTC (permalink / raw)
To: Andreas Kretschmer <andreas@a-kretschmer.de>; +Cc: pgsql-general <pgsql-general@lists.postgresql.org>
On Mon, Sep 23, 2019 at 10:55 AM Andreas Kretschmer
<andreas@a-kretschmer.de> wrote:
> you can use both of them, and you should consider "Barman".
If I remember well Barman uses pg_receivexlog when streaming, and
archive_command when doing a "normal" backup.
Also pgbackrest is another good tool for backup.
The idea here should be not reinventing the wheel.
Luca
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_receivexlog or archive_command
2019-09-23 08:25 pg_receivexlog or archive_command Vikas Sharma <shavikas@gmail.com>
2019-09-23 08:55 ` Re: pg_receivexlog or archive_command Andreas Kretschmer <andreas@a-kretschmer.de>
2019-09-23 11:44 ` Re: pg_receivexlog or archive_command Luca Ferrari <fluca1978@gmail.com>
@ 2019-09-23 11:56 ` Andreas Kretschmer <andreas@a-kretschmer.de>
0 siblings, 0 replies; 7+ messages in thread
From: Andreas Kretschmer @ 2019-09-23 11:56 UTC (permalink / raw)
To: Luca Ferrari <fluca1978@gmail.com>; +Cc: pgsql-general <pgsql-general@lists.postgresql.org>
Am 23.09.19 um 13:44 schrieb Luca Ferrari:
> On Mon, Sep 23, 2019 at 10:55 AM Andreas Kretschmer
> <andreas@a-kretschmer.de> wrote:
>> you can use both of them, and you should consider "Barman".
> If I remember well Barman uses pg_receivexlog when streaming, and
> archive_command when doing a "normal" backup.
Barman < version 2 can only archive_command, version 2 and higher can both.
> Also pgbackrest is another good tool for backup.
> The idea here should be not reinventing the wheel.
ack.
Regards, Andreas
> --
> 2ndQuadrant - The PostgreSQL Support Company.
> www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_receivexlog or archive_command
2019-09-23 08:25 pg_receivexlog or archive_command Vikas Sharma <shavikas@gmail.com>
@ 2019-09-23 18:31 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-10-02 06:17 ` Re: pg_receivexlog or archive_command Stephen Frost <sfrost@snowman.net>
2 siblings, 1 reply; 7+ messages in thread
From: Peter Eisentraut @ 2019-09-23 18:31 UTC (permalink / raw)
To: Vikas Sharma <shavikas@gmail.com>; pgsql-general@lists.postgresql.org
On 2019-09-23 10:25, Vikas Sharma wrote:
> I am wondering which one is the best way to archive the xlogs for Backup
> and Recovery - pg_receivexlog or archive_command.
I recommend using pg_receivexlog. It has two important advantages over
archive_command: 1) You can have multiple instances of pg_receivexlog
running and copying things to different places. This is complicated to
do correctly with archive_command. 2) pg_receivexlog will fsync the
files it writes. This is also complicated to do correctly with
archive_command.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_receivexlog or archive_command
2019-09-23 08:25 pg_receivexlog or archive_command Vikas Sharma <shavikas@gmail.com>
2019-09-23 18:31 ` Re: pg_receivexlog or archive_command Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
@ 2019-10-02 06:17 ` Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 7+ messages in thread
From: Stephen Frost @ 2019-10-02 06:17 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Vikas Sharma <shavikas@gmail.com>; pgsql-general@lists.postgresql.org
Greetings,
* Peter Eisentraut (peter.eisentraut@2ndquadrant.com) wrote:
> On 2019-09-23 10:25, Vikas Sharma wrote:
> > I am wondering which one is the best way to archive the xlogs for Backup
> > and Recovery - pg_receivexlog or archive_command.
>
> I recommend using pg_receivexlog. It has two important advantages over
> archive_command: 1) You can have multiple instances of pg_receivexlog
> running and copying things to different places. This is complicated to
> do correctly with archive_command. 2) pg_receivexlog will fsync the
> files it writes. This is also complicated to do correctly with
> archive_command.
Yes, it definitely is difficult to write your own archive_command, in
general, so, please, just don't. Use one of the existing tools that
have been well tested and written specifically to work with PG and to
provide the guarantees that an archive command should.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: pg_receivexlog or archive_command
2019-09-23 08:25 pg_receivexlog or archive_command Vikas Sharma <shavikas@gmail.com>
@ 2019-10-02 06:16 ` Stephen Frost <sfrost@snowman.net>
2 siblings, 0 replies; 7+ messages in thread
From: Stephen Frost @ 2019-10-02 06:16 UTC (permalink / raw)
To: Vikas Sharma <shavikas@gmail.com>; +Cc: pgsql-general@lists.postgresql.org
Greetings,
* Vikas Sharma (shavikas@gmail.com) wrote:
> I am wondering which one is the best way to archive the xlogs for Backup
> and Recovery - pg_receivexlog or archive_command.
>
> pg_receivexlog seems best suited because the copied/archived file is
> streamed as it is being written to in xlog while archive_command only
> copies when the WAL is fully written to.
This really ends up depending on what your WAL volume is. As mentioned,
you really don't want to write your own utility for archive_command,
since you really want to make sure that the WAL has actually been
archived and sync'd- so definitely use an existing tool which does that
for you. The downside of pg_receivewal when it comes to high WAL volume
is that a single-threaded process just simply can't keep up, while
something being called from archive_command can be parallelized.
Perhaps, down the road, there'll be a tool that could parallelize
streaming of WAL also, though it would be pretty tricky to get right,
and if you're doing that much WAL, is it really an issue that it's
already chunked up nicely for archive_command..?
Note that there is also archive_timeout which you can set, to make sure
that you don't go too long with writes on the primary that haven't been
sent to the archive and stored. If your transations are particularly
valuable, then having a synchronous standby setup (likely with two
replicas in a quorum-based sync setup) is probably the direction to
go in, so you can have a highly available environment. Anything that's
async will mean you have a good chance of having some data loss if
things go wrong (even with pg_receivewal..).
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2019-10-02 06:17 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-09-23 08:25 pg_receivexlog or archive_command Vikas Sharma <shavikas@gmail.com>
2019-09-23 08:55 ` Andreas Kretschmer <andreas@a-kretschmer.de>
2019-09-23 11:44 ` Luca Ferrari <fluca1978@gmail.com>
2019-09-23 11:56 ` Andreas Kretschmer <andreas@a-kretschmer.de>
2019-09-23 18:31 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-10-02 06:17 ` Stephen Frost <sfrost@snowman.net>
2019-10-02 06:16 ` Stephen Frost <sfrost@snowman.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox