pg.ddx.io pgsql-docs@postgresql.org mailing list archive
help / color / mirror / Atom feedAnother pg_dump using split and gzip for large databases
12+ messages / 4 participants
[nested] [flat]
* Another pg_dump using split and gzip for large databases
@ 2021-07-17 15:09 PG Doc comments form <noreply@postgresql.org>
0 siblings, 1 reply; 12+ messages in thread
From: PG Doc comments form @ 2021-07-17 15:09 UTC (permalink / raw)
To: pgsql-docs@lists.postgresql.org; +Cc: jim@jdoherty.net
The following documentation comment has been logged on the website:
Page: https://www.postgresql.org/docs/10/backup-dump.html
Description:
pg_dump ccbank | split -b 2048m --filter='gzip > $FILE.gz'
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-28 23:46 Bruce Momjian <bruce@momjian.us>
parent: PG Doc comments form <noreply@postgresql.org>
0 siblings, 2 replies; 12+ messages in thread
From: Bruce Momjian @ 2021-07-28 23:46 UTC (permalink / raw)
To: jim@jdoherty.net; pgsql-docs@lists.postgresql.org
On Sat, Jul 17, 2021 at 03:09:52PM +0000, PG Doc comments form wrote:
> The following documentation comment has been logged on the website:
>
> Page: https://www.postgresql.org/docs/10/backup-dump.html
> Description:
>
> pg_dump ccbank | split -b 2048m --filter='gzip > $FILE.gz'
Wow, odd we didn't have an example of using gzip and split together;
patch attached.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
Attachments:
[text/x-diff] split.diff (1.4K, ../../20210728234605.GA9600@momjian.us/2-split.diff)
download | inline diff:
diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 8c9186d277..96e7f06485 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -273,10 +273,10 @@ cat <replaceable class="parameter">filename</replaceable>.gz | gunzip | psql <re
The <command>split</command> command
allows you to split the output into smaller files that are
acceptable in size to the underlying file system. For example, to
- make chunks of 1 megabyte:
+ make 2 gigabyte chunks:
<programlisting>
-pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 1m - <replaceable class="parameter">filename</replaceable>
+pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 2G - <replaceable class="parameter">filename</replaceable>
</programlisting>
Reload with:
@@ -284,6 +284,14 @@ pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 1m - <rep
<programlisting>
cat <replaceable class="parameter">filename</replaceable>* | psql <replaceable class="parameter">dbname</replaceable>
</programlisting>
+
+ It is possible to use <command>split</command> and <application>gzip</application>:
+
+<programlisting>
+pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 2G --filter='gzip > $FILE.gz'
+</programlisting>
+
+ It can be restored using <command>zcat</command>.
</para>
</formalpara>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-29 01:07 James Doherty <jim@jdoherty.net>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 0 replies; 12+ messages in thread
From: James Doherty @ 2021-07-29 01:07 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: pgsql-docs@lists.postgresql.org
They key to my comment is splits —filter command which allows gzip each of
the split files
On Wed, Jul 28, 2021 at 18:46 Bruce Momjian <bruce@momjian.us> wrote:
> On Sat, Jul 17, 2021 at 03:09:52PM +0000, PG Doc comments form wrote:
> > The following documentation comment has been logged on the website:
> >
> > Page: https://www.postgresql.org/docs/10/backup-dump.html
> > Description:
> >
> > pg_dump ccbank | split -b 2048m --filter='gzip > $FILE.gz'
>
> Wow, odd we didn't have an example of using gzip and split together;
> patch attached.
>
> --
> Bruce Momjian <bruce@momjian.us> https://momjian.us
> EDB https://enterprisedb.com
>
> If only the physical world exists, free will is an illusion.
>
> --
James M Doherty President Revive Consulting LLC
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-29 09:27 Daniel Gustafsson <daniel@yesql.se>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 2 replies; 12+ messages in thread
From: Daniel Gustafsson @ 2021-07-29 09:27 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: jim@jdoherty.net; pgsql-docs@lists.postgresql.org
> On 29 Jul 2021, at 01:46, Bruce Momjian <bruce@momjian.us> wrote:
> On Sat, Jul 17, 2021 at 03:09:52PM +0000, PG Doc comments form wrote:
>> pg_dump ccbank | split -b 2048m --filter='gzip > $FILE.gz'
>
> Wow, odd we didn't have an example of using gzip and split together;
> patch attached.
Maybe because the --filter option is only available in GNU split and not in
POSIX, and thus not on FreeBSD and macOS etc. I'm not sure about what level of
compatibility we strive for in the examples though.
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-29 10:48 James Doherty <jim@jdoherty.net>
parent: Daniel Gustafsson <daniel@yesql.se>
1 sibling, 0 replies; 12+ messages in thread
From: James Doherty @ 2021-07-29 10:48 UTC (permalink / raw)
To: daniel@yesql.se; +Cc: Bruce Momjian <bruce@momjian.us>; pgsql-docs@lists.postgresql.org
Good point.
On Thu, Jul 29, 2021 at 04:27 Daniel Gustafsson <daniel@yesql.se> wrote:
> > On 29 Jul 2021, at 01:46, Bruce Momjian <bruce@momjian.us> wrote:
> > On Sat, Jul 17, 2021 at 03:09:52PM +0000, PG Doc comments form wrote:
>
> >> pg_dump ccbank | split -b 2048m --filter='gzip > $FILE.gz'
> >
> > Wow, odd we didn't have an example of using gzip and split together;
> > patch attached.
>
> Maybe because the --filter option is only available in GNU split and not in
> POSIX, and thus not on FreeBSD and macOS etc. I'm not sure about what
> level of
> compatibility we strive for in the examples though.
>
> --
> Daniel Gustafsson https://vmware.com/
>
> --
James M Doherty President Revive Consulting LLC
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-29 14:58 Bruce Momjian <bruce@momjian.us>
parent: Daniel Gustafsson <daniel@yesql.se>
1 sibling, 1 reply; 12+ messages in thread
From: Bruce Momjian @ 2021-07-29 14:58 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: jim@jdoherty.net; pgsql-docs@lists.postgresql.org
On Thu, Jul 29, 2021 at 11:27:20AM +0200, Daniel Gustafsson wrote:
> > On 29 Jul 2021, at 01:46, Bruce Momjian <bruce@momjian.us> wrote:
> > On Sat, Jul 17, 2021 at 03:09:52PM +0000, PG Doc comments form wrote:
>
> >> pg_dump ccbank | split -b 2048m --filter='gzip > $FILE.gz'
> >
> > Wow, odd we didn't have an example of using gzip and split together;
> > patch attached.
>
> Maybe because the --filter option is only available in GNU split and not in
> POSIX, and thus not on FreeBSD and macOS etc. I'm not sure about what level of
> compatibility we strive for in the examples though.
OK, I can adjust it to say "GNU split" or just skip this addition.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-29 15:23 Daniel Gustafsson <daniel@yesql.se>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 12+ messages in thread
From: Daniel Gustafsson @ 2021-07-29 15:23 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: jim@jdoherty.net; pgsql-docs@lists.postgresql.org
> On 29 Jul 2021, at 16:58, Bruce Momjian <bruce@momjian.us> wrote:
> OK, I can adjust it to say "GNU split" or just skip this addition.
I don't have strong feelings either way, with a slight preference for writing
it as "GNU split" since it's still a good tip for a large amount of readers.
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-29 15:29 Bruce Momjian <bruce@momjian.us>
parent: Daniel Gustafsson <daniel@yesql.se>
0 siblings, 1 reply; 12+ messages in thread
From: Bruce Momjian @ 2021-07-29 15:29 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: jim@jdoherty.net; pgsql-docs@lists.postgresql.org
On Thu, Jul 29, 2021 at 05:23:20PM +0200, Daniel Gustafsson wrote:
> > On 29 Jul 2021, at 16:58, Bruce Momjian <bruce@momjian.us> wrote:
>
> > OK, I can adjust it to say "GNU split" or just skip this addition.
>
> I don't have strong feelings either way, with a slight preference for writing
> it as "GNU split" since it's still a good tip for a large amount of readers.
Yes, I am on the fence for the same reasons but lean toward moving
forward for your same reasons.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
Attachments:
[text/x-diff] split.diff (1.4K, ../../20210729152925.GJ9600@momjian.us/2-split.diff)
download | inline diff:
diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 8c9186d277..62dead73ce 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -273,10 +273,10 @@ cat <replaceable class="parameter">filename</replaceable>.gz | gunzip | psql <re
The <command>split</command> command
allows you to split the output into smaller files that are
acceptable in size to the underlying file system. For example, to
- make chunks of 1 megabyte:
+ make 2 gigabyte chunks:
<programlisting>
-pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 1m - <replaceable class="parameter">filename</replaceable>
+pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 2G - <replaceable class="parameter">filename</replaceable>
</programlisting>
Reload with:
@@ -284,6 +284,15 @@ pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 1m - <rep
<programlisting>
cat <replaceable class="parameter">filename</replaceable>* | psql <replaceable class="parameter">dbname</replaceable>
</programlisting>
+
+ If using <application>GNU split</application>, it is possible to
+ use it and <application>gzip</application> together:
+
+<programlisting>
+pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 2G --filter='gzip > $FILE.gz'
+</programlisting>
+
+ It can be restored using <command>zcat</command>.
</para>
</formalpara>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-29 17:15 Daniel Gustafsson <daniel@yesql.se>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 12+ messages in thread
From: Daniel Gustafsson @ 2021-07-29 17:15 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: jim@jdoherty.net; pgsql-docs@lists.postgresql.org
> On 29 Jul 2021, at 17:29, Bruce Momjian <bruce@momjian.us> wrote:
>
> On Thu, Jul 29, 2021 at 05:23:20PM +0200, Daniel Gustafsson wrote:
>>> On 29 Jul 2021, at 16:58, Bruce Momjian <bruce@momjian.us> wrote:
>>
>>> OK, I can adjust it to say "GNU split" or just skip this addition.
>>
>> I don't have strong feelings either way, with a slight preference for writing
>> it as "GNU split" since it's still a good tip for a large amount of readers.
>
> Yes, I am on the fence for the same reasons but lean toward moving
> forward for your same reasons.
Making it two separate examples is a good idea, tipping the scale to a +1 from me.
+ If using <application>GNU split</application>, it is possible to
A small nitpick is that we write this as GNU <application>..</application>
elsewhere on this page (like GNU tar for example).
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-29 19:10 Bruce Momjian <bruce@momjian.us>
parent: Daniel Gustafsson <daniel@yesql.se>
0 siblings, 2 replies; 12+ messages in thread
From: Bruce Momjian @ 2021-07-29 19:10 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: jim@jdoherty.net; pgsql-docs@lists.postgresql.org
On Thu, Jul 29, 2021 at 07:15:21PM +0200, Daniel Gustafsson wrote:
> > On 29 Jul 2021, at 17:29, Bruce Momjian <bruce@momjian.us> wrote:
> >
> > On Thu, Jul 29, 2021 at 05:23:20PM +0200, Daniel Gustafsson wrote:
> >>> On 29 Jul 2021, at 16:58, Bruce Momjian <bruce@momjian.us> wrote:
> >>
> >>> OK, I can adjust it to say "GNU split" or just skip this addition.
> >>
> >> I don't have strong feelings either way, with a slight preference for writing
> >> it as "GNU split" since it's still a good tip for a large amount of readers.
> >
> > Yes, I am on the fence for the same reasons but lean toward moving
> > forward for your same reasons.
>
> Making it two separate examples is a good idea, tipping the scale to a +1 from me.
>
> + If using <application>GNU split</application>, it is possible to
> A small nitpick is that we write this as GNU <application>..</application>
> elsewhere on this page (like GNU tar for example).
OK, updated patch attached. FYI, I put the GNU inside the application
name in the first version because we specifically only want the GNU
version of that application, but I think taking it out for consistency
is fine.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
Attachments:
[text/x-diff] split.diff (1.4K, ../../20210729191034.GU9600@momjian.us/2-split.diff)
download | inline diff:
diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml
index 8c9186d277..cba32b6eb3 100644
--- a/doc/src/sgml/backup.sgml
+++ b/doc/src/sgml/backup.sgml
@@ -273,10 +273,10 @@ cat <replaceable class="parameter">filename</replaceable>.gz | gunzip | psql <re
The <command>split</command> command
allows you to split the output into smaller files that are
acceptable in size to the underlying file system. For example, to
- make chunks of 1 megabyte:
+ make 2 gigabyte chunks:
<programlisting>
-pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 1m - <replaceable class="parameter">filename</replaceable>
+pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 2G - <replaceable class="parameter">filename</replaceable>
</programlisting>
Reload with:
@@ -284,6 +284,15 @@ pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 1m - <rep
<programlisting>
cat <replaceable class="parameter">filename</replaceable>* | psql <replaceable class="parameter">dbname</replaceable>
</programlisting>
+
+ If using GNU <application>split</application>, it is possible to
+ use it and <application>gzip</application> together:
+
+<programlisting>
+pg_dump <replaceable class="parameter">dbname</replaceable> | split -b 2G --filter='gzip > $FILE.gz'
+</programlisting>
+
+ It can be restored using <command>zcat</command>.
</para>
</formalpara>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-07-29 19:12 Daniel Gustafsson <daniel@yesql.se>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 0 replies; 12+ messages in thread
From: Daniel Gustafsson @ 2021-07-29 19:12 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; +Cc: James Doherty <jim@jdoherty.net>; pgsql-docs@lists.postgresql.org
> On 29 Jul 2021, at 21:10, Bruce Momjian <bruce@momjian.us> wrote:
>
> On Thu, Jul 29, 2021 at 07:15:21PM +0200, Daniel Gustafsson wrote:
>>> On 29 Jul 2021, at 17:29, Bruce Momjian <bruce@momjian.us> wrote:
>>>
>>> On Thu, Jul 29, 2021 at 05:23:20PM +0200, Daniel Gustafsson wrote:
>>>>> On 29 Jul 2021, at 16:58, Bruce Momjian <bruce@momjian.us> wrote:
>>>>
>>>>> OK, I can adjust it to say "GNU split" or just skip this addition.
>>>>
>>>> I don't have strong feelings either way, with a slight preference for writing
>>>> it as "GNU split" since it's still a good tip for a large amount of readers.
>>>
>>> Yes, I am on the fence for the same reasons but lean toward moving
>>> forward for your same reasons.
>>
>> Making it two separate examples is a good idea, tipping the scale to a +1 from me.
>>
>> + If using <application>GNU split</application>, it is possible to
>> A small nitpick is that we write this as GNU <application>..</application>
>> elsewhere on this page (like GNU tar for example).
>
> OK, updated patch attached. FYI, I put the GNU inside the application
> name in the first version because we specifically only want the GNU
> version of that application, but I think taking it out for consistency
> is fine.
LGTM
--
Daniel Gustafsson https://vmware.com/
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Another pg_dump using split and gzip for large databases
@ 2021-08-03 14:58 Bruce Momjian <bruce@momjian.us>
parent: Bruce Momjian <bruce@momjian.us>
1 sibling, 0 replies; 12+ messages in thread
From: Bruce Momjian @ 2021-08-03 14:58 UTC (permalink / raw)
To: Daniel Gustafsson <daniel@yesql.se>; +Cc: jim@jdoherty.net; pgsql-docs@lists.postgresql.org
On Thu, Jul 29, 2021 at 03:10:34PM -0400, Bruce Momjian wrote:
> On Thu, Jul 29, 2021 at 07:15:21PM +0200, Daniel Gustafsson wrote:
> > > On 29 Jul 2021, at 17:29, Bruce Momjian <bruce@momjian.us> wrote:
> > >
> > > On Thu, Jul 29, 2021 at 05:23:20PM +0200, Daniel Gustafsson wrote:
> > >>> On 29 Jul 2021, at 16:58, Bruce Momjian <bruce@momjian.us> wrote:
> > >>
> > >>> OK, I can adjust it to say "GNU split" or just skip this addition.
> > >>
> > >> I don't have strong feelings either way, with a slight preference for writing
> > >> it as "GNU split" since it's still a good tip for a large amount of readers.
> > >
> > > Yes, I am on the fence for the same reasons but lean toward moving
> > > forward for your same reasons.
> >
> > Making it two separate examples is a good idea, tipping the scale to a +1 from me.
> >
> > + If using <application>GNU split</application>, it is possible to
> > A small nitpick is that we write this as GNU <application>..</application>
> > elsewhere on this page (like GNU tar for example).
>
> OK, updated patch attached. FYI, I put the GNU inside the application
> name in the first version because we specifically only want the GNU
> version of that application, but I think taking it out for consistency
> is fine.
Patch applied to all supported versions.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2021-08-03 14:58 UTC | newest]
Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-07-17 15:09 Another pg_dump using split and gzip for large databases PG Doc comments form <noreply@postgresql.org>
2021-07-28 23:46 ` Bruce Momjian <bruce@momjian.us>
2021-07-29 01:07 ` Re: Another pg_dump using split and gzip for large databases James Doherty <jim@jdoherty.net>
2021-07-29 09:27 ` Daniel Gustafsson <daniel@yesql.se>
2021-07-29 10:48 ` Re: Another pg_dump using split and gzip for large databases James Doherty <jim@jdoherty.net>
2021-07-29 14:58 ` Bruce Momjian <bruce@momjian.us>
2021-07-29 15:23 ` Daniel Gustafsson <daniel@yesql.se>
2021-07-29 15:29 ` Bruce Momjian <bruce@momjian.us>
2021-07-29 17:15 ` Daniel Gustafsson <daniel@yesql.se>
2021-07-29 19:10 ` Bruce Momjian <bruce@momjian.us>
2021-07-29 19:12 ` Daniel Gustafsson <daniel@yesql.se>
2021-08-03 14:58 ` Bruce Momjian <bruce@momjian.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox