Subject: Re: Adding pg_dump flag for parallel export to pipes

From: Nitin Motiani <[email-masked]>
Date: Fri, 20 Feb 2026 14:38:17 +0530

Hi,

I fixed the failing tap test. Attaching rebased patches along with the
new test. The last patch is still WIP because I need to add more test
cases.

Thanks,
Nitin Motiani
Google

---

From: Nitin Motiani <[email-masked]>
Date: Thu, 21 May 2026 15:26:43 +0530

Changed how pipe commands are quoted in the Windows test. The latest
versions are attached.

Thanks

Nitin Motiani
Google

---

From: solai v <[email-masked]>
Date: Fri, 22 May 2026 16:04:23 +0530

Hi all,

Thank you for the updated patch.

> Changed how pipe commands are quoted in the Windows test. The latest

I worked on reproducing the current limitation around parallel dumps
and then tested the latest v16 patch adding --pipe support for
pg_dump. To begin with, I verified the existing behavior.
For example:
pg_dump postgres | gzip > dump.sql.gz works, but does not support parallelism,
whereas:
pg_dump -Fd -j 4 -f dumpdir postgres
du -sh dumpdir
21M dumpdir
requires intermediate disk storage. This demonstrates the current
limitation where users must choose between parallelism and streaming
pipelines.
I then tested the patch introducing --pipe support. The feature is
quite useful for modern workflows where users want to stream dump
output directly to compression or upload pipelines without relying on
intermediate storage. Basic functionality worked as expected.
For example:
pg_dump -p 55432 -Fd -j 4 --pipe="cat > dump.out" postgres, produced a
~38MB output file,
and:
pg_dump -p 55432 -Fd -j 4 --pipe="gzip > dump.gz" postgres produced, a
compressed file (~11MB).
The initial contents appeared valid:
gunzip -c dump.gz | head
1
2
3
...
Also, no intermediate directory was created, confirming that the patch
enables streaming without filesystem-backed staging. Error handling
also behaved correctly.
For example:
--pipe="invalid_cmd"
resulted in:
pg_dump: error: pipe command failed: command not found
and:
--pipe="gzip | false"
resulted in:
pg_dump: error: pipe command failed: child process exited with exit code 1
However, I observed an important issue when using the feature with
multiple parallel workers. Since the pipe command is executed per
output file, using: --pipe="gzip > dump.gz", it results in multiple
workers invoking independent gzip processes that all write to the same
output file. This leads to corrupted or truncated output.
In my testing:
gunzip -c dump.gz > dump.sql
failed with:
gzip: dump.gz: unexpected end of file
This suggests that concurrent writes to a shared output target are not
coordinated and can result in invalid dumps. It would be helpful to
clarify expected usage patterns here. For example: whether users are
expected to generate distinct outputs per worker, or whether
safeguards should be implemented to prevent multiple workers from
writing to the same destination. Additionally, during failure
scenarios I observed backend logs such as:
FATAL: connection to client lost
Broken pipe
While this is expected when the pipe terminates prematurely, it may be
worth considering whether error messaging or cleanup behavior can be
made clearer from the user perspective.
Overall, the feature is valuable and aligns well with modern backup
workflows. However, behavior in multi-worker scenarios with shared
pipe targets may need further clarification or safeguards to avoid
data corruption. Looking forward to more feedback.

Regards.
Solai

---

From: Nitin Motiani <[email-masked]>
Date: Tue, 9 Jun 2026 12:04:01 +0530

On Fri, May 22, 2026 at 4:04 PM solai v <[email-masked]> wrote:
> intermediate storage.

Thank you for the feedback.

> writing to the same destination.

I added a warning for cases where the pipe command provided with
parallel dump and restore doesn't contain a `%f`. We can also add it
to the documentation. Let me know what you think.

> made clearer from the user perspective.

I added the failed command to the error message. I'm not sure if we
can do any auto-cleanup commands which succeeded.

Thanks & Regards

Nitin Motiani
Google

---

From: solai v <[email-masked]>
Date: Fri, 12 Jun 2026 10:50:46 +0530

Hi all,

> can do any auto-cleanup commands which succeeded.

Thank you for the updated patches. I reviewed and tested the latest
v17 patch series in my current cluster. I tested the new --pipe
functionality with both serial and parallel directory dumps and
correctly generated:
pipe_dump/
    toc.dat
    3475.dat
    3476.dat
The same behavior was verified with parallel jobs (-j 4). I also
tested compression through an external command:
pg_dump -p 55432 -Fd -j 4 \
    --pipe='gzip > pipe_dump/%f.gz' \
    postgres
which successfully generated:
pipe_dump/
    toc.dat.gz
    3475.dat.gz
    3476.dat.gz
without creating an intermediate dump directory.
I verified error handling (--pipe='invalid_cmd') as well and correctly
reported the error as "pg_dump: error: pipe command failed:
"invalid_cmd": command not found".
Similarly, --pipe='gzip | false' correctly propagated the child
process failure. I also performed an end-to-end validation of the new
functionality by restoring the generated archive using pg_restore
--pipe. The restore completed successfully, and row counts for the
test tables matched the original database confirming that dump
generation, compression, restore, and data integrity all worked
correctly. One particularly useful improvement I noticed is the
handling of parallel jobs when the %f placeholder is omitted,
producing the warning: "pg_dump: warning: parallel jobs with --pipe
usually require the "%f" placeholder to avoid data corruption from
multiple workers writing to the same file". This is an usability
improvement, as it explicitly warns users against a common misuse that
could otherwise lead to corrupted output when multiple workers write
to the same destination file. Overall, the patch series worked well
and the new --pipe support behaves correctly in my testing, also the
newly added warning for missing %f significantly improves the user
experience for parallel jobs. The patch looks good to me.

Regards,
Solai