public inbox for [email protected]  

help / color / mirror / Atom feed

From: Dimitrios Apostolou <[email protected]>
To: Adrian Klaver <[email protected]>
Cc: [email protected]
Subject: Re: In-order pg_dump (or in-order COPY TO)
Date: Wed, 27 Aug 2025 14:09:39 +0200 (CEST)
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
	<[email protected]>

On Wednesday 2025-08-27 00:54, Adrian Klaver wrote:

>Date: Wed, 27 Aug 2025 00:54:52
>From: Adrian Klaver <[email protected]>
>To: Dimitrios Apostolou <[email protected]>, [email protected]
>Subject: Re: In-order pg_dump (or in-order COPY TO)
>
> On 8/26/25 12:43, Dimitrios Apostolou wrote:
>>  Hello list,
>>
>>  I am storing dumps of a database (pg_dump custom format) in a de-
>>  duplicating backup server. Each dump is many terabytes in size, so
>>  deduplication is very important. And de-duplication itself is based on
>>  rolling checksums which is pretty flexible, it can compensate for blocks
>>  moving by some offset.
>>
>>  Unfortunately after I did pg_restore to a new server, I notice that the
>>  dumps from the new server are not being de-duplicated, all blocks are
>>  considered new.
>

> What are the pg_dump/pg_restore commands?
>
> What are the Postgres versions involved?
>
> Are they community versions of Postgres or something else?
>
> What is the depduplication program?
>
>


Dump is from PostgreSQL 16, it's pg_dump writing to stdout:

pg_dump -v --format=custom --compress=none --no-toast-compression --serializable-deferrable db_name  |  borg create ...


As you can see the backup (and deduplicating) program is borgbackup.


Restore is in PostgreSQL 17:

I first create the empty tables by running the DDL commands in version 
control to setup the database. And then I do pg_restore --data-only:

pg_restore -vvvv -j 8 -U db_owner -d db_name --schema=public --section=data  dump_file


Worth noting is that the above pg_restore goes through the WAL, i.e. all 
writes are done by walwriter, not the backend directly.

Postgres is standard open source running on own server. It has a couple 
of custom patches that shouldn't matter in this codepath.


>>  Thanks in advance,
>>  Dimitris

view thread (22+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: In-order pg_dump (or in-order COPY TO)
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox