MIME-Version: 1.0
References: <s0491qrn-343s-0757-8sn5-120rr8610qqq@tzk.arg>
In-Reply-To: <s0491qrn-343s-0757-8sn5-120rr8610qqq@tzk.arg>
From: Greg Sabino Mullane <htamfids@gmail.com>
Date: Wed, 27 Aug 2025 10:06:40 -0400
Message-ID: <CAKAnmmLY1EVyV-V0Rmb98F63TDHbx4LQFJ5Jv7qK8ourbCoEKQ@mail.gmail.com>
Subject: Re: In-order pg_dump (or in-order COPY TO)
To: Dimitrios Apostolou <jimis@gmx.net>
Cc: pgsql-general@lists.postgresql.org
Content-Type: multipart/alternative; boundary="00000000000059f719063d594e33"
Archived-At: <https://www.postgresql.org/message-id/CAKAnmmLY1EVyV-V0Rmb98F63TDHbx4LQFJ5Jv7qK8ourbCoEKQ%40mail.gmail.com>
Precedence: bulk

--00000000000059f719063d594e33
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, Aug 26, 2025 at 3:44=E2=80=AFPM Dimitrios Apostolou <jimis@gmx.net>=
 wrote:

> I am storing dumps of a database (pg_dump custom format) in a
> de-duplicating backup server. Each dump is many terabytes in size, so
> deduplication is very important. And de-duplication itself is based on
> rolling checksums which is pretty flexible, it can compensate for blocks
> moving by some offset.
>

I suggest looking into pgBackRest, and it's block incremental feature,
which sounds similar to what you are doing. But it also does it with
parallel processes, and can do things like grab backup files from your
replicas, plus a lot of other features.

Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support

--00000000000059f719063d594e33
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">On Tue, Aug 26, 2025 at 3:44=E2=80=AFPM D=
imitrios Apostolou &lt;<a href=3D"mailto:jimis@gmx.net">jimis@gmx.net</a>&g=
t; wrote:</div><div class=3D"gmail_quote gmail_quote_container"><blockquote=
 class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px so=
lid rgb(204,204,204);padding-left:1ex">I am storing dumps of a database (pg=
_dump custom format) in a <br>
de-duplicating backup server. Each dump is many terabytes in size, so <br>
deduplication is very important. And de-duplication itself is based on <br>
rolling checksums which is pretty flexible, it can compensate for blocks <b=
r>
moving by some offset.<br></blockquote><div><br></div><div>I suggest lookin=
g into pgBackRest, and it&#39;s block incremental feature, which sounds=C2=
=A0similar to what you are doing. But it also does it with parallel process=
es, and can do things like grab backup files from your replicas, plus a lot=
 of other features.</div><div>=C2=A0</div></div><div dir=3D"ltr" class=3D"g=
mail_signature"><div dir=3D"ltr"><div>Cheers,</div><div>Greg</div><div><br>=
</div><div>--</div><div>Crunchy Data - <a href=3D"https://www.crunchydata.c=
om" target=3D"_blank">https://www.crunchydata.com</a></div><div>Enterprise =
Postgres Software Products &amp; Tech Support</div><div><br></div></div></d=
iv></div>

--00000000000059f719063d594e33--