pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedPerformance of pg_basebackup
3+ messages / 2 participants
[nested] [flat]
* Performance of pg_basebackup
@ 2012-06-12 14:54 Shaun Thomas <sthomas@optionshouse.com>
0 siblings, 1 reply; 3+ messages in thread
From: Shaun Thomas @ 2012-06-12 14:54 UTC (permalink / raw)
To: pgsql-performance
Hey everyone,
I was wondering if anyone has found a way to get pg_basebackup to be...
faster. Currently we do our backups something like this:
tar -c -I pigz -f /db/backup_yyyy-mm-dd.tar.gz -C /db pgdata
Which basically calls pigz to do parallel compression because with RAIDs
and ioDrives all over the place, it's the compression that's the
bottleneck. Otherwise, only one of our 24 CPUs is actually doing anything.
I can't seem to find anything like this for pg_basebackup. It just uses
its internal compression method. I could see this being the case for
pg_dump, but pg_basebackup just produces regular tar.gz files. Is there
any way to either fake a parallel compression here, or should this be a
feature request for pg_basebackup?
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Performance of pg_basebackup
@ 2012-06-12 14:57 Magnus Hagander <magnus@hagander.net>
parent: Shaun Thomas <sthomas@optionshouse.com>
0 siblings, 1 reply; 3+ messages in thread
From: Magnus Hagander @ 2012-06-12 14:57 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: pgsql-performance
On Tue, Jun 12, 2012 at 4:54 PM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> Hey everyone,
>
> I was wondering if anyone has found a way to get pg_basebackup to be...
> faster. Currently we do our backups something like this:
>
> tar -c -I pigz -f /db/backup_yyyy-mm-dd.tar.gz -C /db pgdata
>
> Which basically calls pigz to do parallel compression because with RAIDs and
> ioDrives all over the place, it's the compression that's the bottleneck.
> Otherwise, only one of our 24 CPUs is actually doing anything.
>
> I can't seem to find anything like this for pg_basebackup. It just uses its
> internal compression method. I could see this being the case for pg_dump,
> but pg_basebackup just produces regular tar.gz files. Is there any way to
> either fake a parallel compression here, or should this be a feature request
> for pg_basebackup?
If you have a single tablespace you can have pg_basebackup write the
output to stdout and then pipe that through pigz.
--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Performance of pg_basebackup
@ 2012-06-12 15:00 Shaun Thomas <sthomas@optionshouse.com>
parent: Magnus Hagander <magnus@hagander.net>
0 siblings, 0 replies; 3+ messages in thread
From: Shaun Thomas @ 2012-06-12 15:00 UTC (permalink / raw)
To: Magnus Hagander <magnus@hagander.net>; +Cc: pgsql-performance
On 06/12/2012 09:57 AM, Magnus Hagander wrote:
> If you have a single tablespace you can have pg_basebackup write the
> output to stdout and then pipe that through pigz.
Yeah, I saw that. Unfortunately we have tiered storage and hence two
tablespaces. :(
To be fair, my current process cheats by following symlinks. I haven't
yet modified it to explicitly handle tablespaces. I'll probably just
steal the directory method pg_basebackup uses until it can natively call
an external compression program.
Thanks, Magnus!
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2012-06-12 15:00 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-06-12 14:54 Performance of pg_basebackup Shaun Thomas <sthomas@optionshouse.com>
2012-06-12 14:57 ` Magnus Hagander <magnus@hagander.net>
2012-06-12 15:00 ` Shaun Thomas <sthomas@optionshouse.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox