pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Chao Li <li.evan.chao@gmail.com>
To: stefangrassl@gmx.net
To: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #19754: pg_restore + zlib 1.3.2: intact files rejected, truncated files accepted
Date: Tue, 13 Oct 2026 16:55:51 +0800
Message-ID: <25266C45-5546-4496-87E6-E66306936008@gmail.com> (raw)
In-Reply-To: <19754-b335a019b76fd89b@postgresql.org>
References: <19754-b335a019b76fd89b@postgresql.org>



> On Oct 12, 2026, at 15:53, PG Bug reporting form <noreply@postgresql.org> wrote:
> 
> The following bug has been logged on the website:
> 
> Bug reference:      19754
> Logged by:          Stefan Graßl
> Email address:      stefangrassl@gmx.net
> PostgreSQL version: 18.6
> Operating system:   Windows (EDB binaries); reproduced on Debian 13
> Description:        
> 
> zlib 1.3.2 (released 2026-02-17) changed how gzread() handles the end of a
> gzip member. The change breaks the way pg_restore finds damaged data files
> in
> gzip-compressed directory-format dumps. pg_restore relies on gzclose()
> returning Z_BUF_ERROR to detect a data file that ends in the middle of a
> gzip
> stream ("could not close data file"). With zlib 1.3.2 that signal is wrong
> in
> both directions:
> 
> 1. Intact files are rejected. A data file fails when its compressed size is
>   a multiple of 8192 (zlib's default input buffer) and its uncompressed
> size
>   is a multiple of pg_restore's read size (4096 in 14 to 18). pg_restore
>   issues one more gzread() to detect EOF. zlib 1.3.2 then starts looking
> for
>   a second gzip member without first checking whether input is left. It
>   hits EOF and records Z_BUF_ERROR "unexpected end of file". gzclose()
>   returns it, and pg_restore stops:
> 
>     pg_restore: error: could not close data file "d/3434.dat": Success
> 
>   ("No error" on Windows). The table gets zero rows. With one more byte of
>   FEXTRA padding in the gzip header, the same file restores fine; the
>   deflate stream and the trailer stay byte-identical.
> 
> 2. Truncated files are no longer detected. zlib 1.3.2 clears a pending
>   Z_BUF_ERROR at the start of every gzread() call. The final gzread(),
>   which returns 0, therefore wipes out the error that the previous call
>   recorded, and gzclose() returns Z_OK. pg_restore exits 0 after loading
>   only the first part of the table. In the reproduction below, a data file
>   cut in half restores 10073 of 20001 rows with exit code 0. With zlib
>   1.3.1, the same file makes pg_restore fail as it should.
> 
> Both regressions are fixed on zlib's develop branch but not released yet:
> 
>  a05eacf  Fix bug where a complete gzip stream can be reported as
> truncated.
>  930cdf0  Restore the proper detection of truncated gzip streams.
>           ("Part of this was lost in 1.3.2.")
>  ec63cd6  Fix one more truncated stream detection bug.
> 
> Gzip_close() itself does what zlib documents. zlib.h says about gzread():
> "This error is deferred until gzclose(), which will return Z_BUF_ERROR if
> the last gzread ended in the middle of a gzip stream." The reason for
> reporting it here is the Windows binaries. In the EDB binary archives of
> 14.24 and 15.19, pg_dump.exe and pg_restore.exe load the bundled
> zlib1.dll, which is 1.3.2. In 18.6, zlib 1.3.2 is linked statically into
> both programs, so replacing the DLL does not help there; the programs have
> to be rebuilt against a fixed zlib. We reproduced (1) with the Windows
> binaries of 18.6 and 14.24. Every Windows user who restores a gzip
> directory-format dump is therefore exposed to silent partial restores (2),
> and sometimes to (1). Other platforms are affected wherever PostgreSQL is
> built against zlib 1.3.2.
> 
> Related: e01fc63 and e41b01f (2026-10-09, back-patched to 19) added checks
> for truncated compressed data, in pg_restore for zstd and LZ4 and in
> pg_verifybackup also for gzip. The gzip reader that pg_restore uses for
> directory-format dumps still relies on gzclose() alone. With zlib 1.3.2,
> gzip is therefore the one compression method whose truncated data files
> pg_restore no longer notices.
> 
> We ran into (1) during a production upgrade. One data file out of about
> 236,000 met both size conditions. It took weeks to see that this file was
> intact, because the message points to corruption, and "Success" or
> "No error" gives nothing to go on.
> 
> Suggestions:
> - Packagers: build the binaries for the November 12 releases against a zlib
>  that contains the three commits above, or against 1.3.1. That release is
>  the last one for 14, so 14 would otherwise stay affected for good.
> - Give the directory-format gzip reader the same kind of check that
>  e01fc63 adds for zstd and LZ4: read with inflate() and require
>  Z_STREAM_END, as e41b01f now does for gzip in pg_verifybackup. pg_restore
>  would then no longer depend on how gzread() and gzclose() report the end
>  of the stream, and neither (1) nor (2) could occur.
> - Perhaps mention this as a known issue, since pg_restore can silently
>  load only part of a table's data.
> - The message prints %m, but no system call failed, so errno is 0
>  (EndCompressFileHandle() sets it to 0 before closing). 3cac2c8 expected
>  errno-less gzclose() failures to be "basically can't-happen if we're not
>  misusing zlib". Printing the zlib status instead, as Kyotaro Horiguchi
>  proposed in 2020, would have pointed straight at zlib:
> https://www.postgresql.org/message-id/20201021.152022.1604120629445501521.horikyota.ntt@gmail.com
>  Those 2020 reports (10.6, 11.7) predate zlib 1.3.2 and involved damaged
>  files, so they are a different problem with the same message.
> 
> 19 and master: DEFAULT_IO_BUFFER_SIZE is now 128 kB. A zlib-level probe
> using that read size shows (1) at usize % 131072 == 0 and shows (2)
> unchanged. We have not tested this with a build of 19 or master.
> 
> Reproduction (official postgres image, about two minutes). The script builds
> zlib 1.3.1, 1.3.2 and develop and runs the real pg_restore against each one
> via LD_LIBRARY_PATH:
> 
>  docker run -d --name pgz -e POSTGRES_HOST_AUTH_METHOD=trust postgres:18
>  docker cp repro.sh pgz:/repro.sh
>  docker exec pgz bash /repro.sh
> 
> ---- repro.sh ----
> #!/bin/bash
> # pg_restore against zlib 1.3.1 / 1.3.2 / develop, on an intact data file
> whose
> # sizes sit on the buffer boundaries, and on a truncated data file.
> # Run inside the official postgres image (server already up, trust auth).
> set -euo pipefail
> export PGUSER=postgres
> PGBIN=$(ls -d /usr/lib/postgresql/*/bin)
> W=/tmp/work; rm -rf $W; mkdir -p $W; cd $W
> 
> apt-get -qq update >/dev/null
> apt-get -qq install -y gcc make curl ca-certificates >/dev/null 2>&1
> for t in v1.3.1 v1.3.2 develop; do
>  case $t in
>    develop)
> url=https://github.com/madler/zlib/archive/refs/heads/develop.tar.gz ;;
>    *)       url=https://github.com/madler/zlib/archive/refs/tags/$t.tar.gz
> ;;
>  esac
>  mkdir zlib-$t && curl -sfL $url | tar xz -C zlib-$t --strip-components=1
>  (cd zlib-$t && ./configure >/dev/null && make -s >/dev/null 2>&1)
> done
> 
> until pg_isready -q; do sleep 0.5; done
> psql -qX -c "create database src" -c "create database dst"
> psql -qX -d src -c "create table t (id int, v text)" \
>  -c "insert into t select i, md5(i::text) from generate_series(1, 20000) i"
> \
>  -c "insert into t values (0, '')"
> 
> isize() { perl -e 'open F, "<:raw", $ARGV[0] or die; seek F, -4, 2; read F,
> $b, 4; print unpack("V", $b)' "$1"; }
> dump()  { rm -rf d; pg_dump -d src -Fd -f d -t t; f=$(ls d/*.dat.gz); }
> 
> # make the uncompressed data file a multiple of 4096 bytes
> dump; u=$(isize $f)
> psql -qX -d src -c "update t set v = repeat('y', $(( (4096 - u % 4096) %
> 4096 ))) where id = 0"
> dump; cp $f intact.gz
> pg_restore -d dst --section=pre-data d
> 
> # insert an FEXTRA field so that the file size is congruent to $2 mod 8192;
> # deflate stream and trailer stay byte-identical
> fextra() {
>  perl -e '
>    open F, "<:raw", $ARGV[0] or die; local $/; $d = <F>; close F;
>    $x = (8192 + $ARGV[1] - (length($d) + 2) % 8192) % 8192;
>    substr($d, 3, 1) = chr(ord(substr($d, 3, 1)) | 4);
>    $d = substr($d, 0, 10) . pack("v", $x) . ("\0" x $x) . substr($d, 10);
>    open F, ">:raw", $ARGV[0] or die; print F $d; close F' "$1" "$2"
> }
> 
> try() {
>  local label=$1 lib=$2 rc
>  psql -qX -d dst -c "truncate t"
>  set +e; LD_LIBRARY_PATH=$lib $PGBIN/pg_restore -d dst --section=data d
> 2>err.txt; rc=$?; set -e
>  printf '  %-8s %-32s exit=%d rows=%-6s %s\n' "$label"
> "$(LD_LIBRARY_PATH=$lib ldd $PGBIN/pg_restore | awk '/libz\.so/{print $3}')"
> \
>    $rc "$(psql -qXtA -d dst -c 'select count(*) from t')" "$(tr '\n' ' ' <
> err.txt)"
> }
> 
> run_all() {
>  echo "$1  (csize=$(stat -c %s $f), csize%8192=$(( $(stat -c %s $f) % 8192
> )), usize=$(isize $f), usize%4096=$(( $(isize $f) % 4096 )))"
>  try system ""
>  for t in v1.3.1 v1.3.2 develop; do try $t $W/zlib-$t; done
> }
> 
> echo "$(pg_restore --version), expected rows: $(psql -qXtA -d src -c 'select
> count(*) from t')"
> cp intact.gz $f; fextra $f 0; run_all "A: intact, both sizes on the
> boundaries"
> cp intact.gz $f; fextra $f 1; run_all "B: same file, one more padding byte"
> cp intact.gz $f; head -c $(( $(stat -c %s intact.gz) / 2 )) intact.gz > $f;
> run_all "C: truncated to half"
> ------------------
> 
> Output with postgres:18 (paths shortened; the usize shown for the truncated
> file is just the garbage in its last four bytes). postgres:14 (14.24) gives
> identical results:
> 
> pg_restore (PostgreSQL) 18.6 (Debian 18.6-1.pgdg13+2), expected rows: 20001
> A: intact, both sizes on the boundaries  (csize=442368, csize%8192=0,
> usize=770048, usize%4096=0)
>  system   /lib/x86_64-linux-gnu/libz.so.1  exit=0 rows=20001
>  v1.3.1   zlib-v1.3.1/libz.so.1  exit=0 rows=20001
>  v1.3.2   zlib-v1.3.2/libz.so.1  exit=1 rows=0      pg_restore: error:
> could not close data file "d/3434.dat": Success
>  develop  zlib-develop/libz.so.1 exit=0 rows=20001
> B: same file, one more padding byte  (csize=442369, csize%8192=1,
> usize=770048, usize%4096=0)
>  system   /lib/x86_64-linux-gnu/libz.so.1  exit=0 rows=20001
>  v1.3.1   zlib-v1.3.1/libz.so.1  exit=0 rows=20001
>  v1.3.2   zlib-v1.3.2/libz.so.1  exit=0 rows=20001
>  develop  zlib-develop/libz.so.1 exit=0 rows=20001
> C: truncated to half  (csize=217495, csize%8192=4503, usize=3204074365,
> usize%4096=2941)
>  system   /lib/x86_64-linux-gnu/libz.so.1  exit=1 rows=0      pg_restore:
> error: could not close data file "d/3434.dat": Success
>  v1.3.1   zlib-v1.3.1/libz.so.1  exit=1 rows=0      pg_restore: error:
> could not close data file "d/3434.dat": Success
>  v1.3.2   zlib-v1.3.2/libz.so.1  exit=0 rows=10073
>  develop  zlib-develop/libz.so.1 exit=1 rows=0      pg_restore: error:
> could not close data file "d/3434.dat": Success
> 

I can reproduce the issue on my MacBook as well, so it doesn’t seem like platform dependent.

We can reuse the same logic of astream_gzip, so that pg_restore would no longer depend on gzread() and gzclose(). I have created a patch in that way, and running tests, I may post the patch tomorrow.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/










view thread (2+ messages)

Message-ID: <25266C45-5546-4496-87E6-E66306936008@gmail.com>
Permalink:  ../25266C45-5546-4496-87E6-E66306936008@gmail.com/
Also on:    postgresql.org/message-id/25266C45-5546-4496-87E6-E66306936008@gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-bugs@postgresql.org
  Cc: li.evan.chao@gmail.com, stefangrassl@gmx.net
  Subject: Re: BUG #19754: pg_restore + zlib 1.3.2: intact files rejected, truncated files accepted
  In-Reply-To: <25266C45-5546-4496-87E6-E66306936008@gmail.com>
  References: <19754-b335a019b76fd89b@postgresql.org> <25266C45-5546-4496-87E6-E66306936008@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox