public inbox for [email protected]  

help / color / mirror / Atom feed

From: Dimitrios Apostolou <[email protected]>
To: Adrian Klaver <[email protected]>
To: Dimitrios Apostolou <[email protected]>
Cc: [email protected]
Subject: Re: In-order pg_dump (or in-order COPY TO)
Date: Sun, 31 Aug 2025 19:52:55 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>
	<[email protected]>

When I first said "dump file without TOC" I actually meant "without offsets in the TOC".

The fact that you get a TOC printed does not prove that the dump file includes a TOC with offsets.

All pg_dump -Fc commands that write to stdout, produce a file without offsets in the TOC. It has nothing to do with borg. ToC offsets must be filled in right before streaming each table, but this is impossible when the whole TOC has already been written to stdout in the beginning.

Dimitris

On 31 August 2025 17:41:34 CEST, Adrian Klaver <[email protected]> wrote:
>On 8/30/25 18:21, Dimitrios Apostolou wrote:
>> Sorry I was not remembering the details. Probably there is a TOC in your dump file, but it does not contain any positions for the data. The pg_restore command has to scan the whole file in advance, and fill in the TOC offsets in memory.
>> 
>> This scanning happens in a very inefficient way, with many seek calls and small block reads. Try strace to see them. This initial phase can take hours in a huge dump file, before even starting any actual restoration.
>
>It may be that my coffee is not strong enough, but I don't understand what you are trying to say.
>
>Are you using, from previous post, the following?:
>
>"Two patches for speeding up scanning an archive without TOC, like the one I'm having (because it is piped into borg, instead of written to file). These were activated, but shouldn't matter. They only build the TOC in pg_restore's memory. "
>
>The part I don't see is how you get a dump file without a TOC?
>
>When I do the pg_dump and pipe it to Borg the resulting file has a TOC.
>
>Can you show the rest of the  |  borg create ...  command?
>
>> 
>> Thank you for testing.
>> Dimitris
>> 
>> On 30 August 2025 20:19:13 CEST, Adrian Klaver <[email protected]> wrote:
>
>
>

view thread (22+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: In-order pg_dump (or in-order COPY TO)
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox