Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xGYJF-00000002TJJ-06Kb for pgsql-bugs@arkaria.postgresql.org; Tue, 13 Oct 2026 08:56:33 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xGYJD-0000000CyCX-2bB4 for pgsql-bugs@arkaria.postgresql.org; Tue, 13 Oct 2026 08:56:31 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xGYJD-0000000CyCO-0xTa for pgsql-bugs@lists.postgresql.org; Tue, 13 Oct 2026 08:56:31 +0000 Received: from mail-dy1-x1332.google.com ([2607:f8b0:4864:20::1332]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1xGYJA-00000001lpr-2fGh for pgsql-bugs@lists.postgresql.org; Tue, 13 Oct 2026 08:56:30 +0000 Received: by mail-dy1-x1332.google.com with SMTP id 5a478bee46e88-354ba08cfd1so4698193eec.1 for ; Tue, 13 Oct 2026 01:56:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791881787; x=1792486587; darn=lists.postgresql.org; h=message-id:in-reply-to:to:references:date:subject:mime-version :content-transfer-encoding:content-type:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uK3LW/nd/xR2py1trgv05IJ7lBY+cUB6W1L6U9gRmaI=; b=Tn3qMb3AWg8msqLusT22mV9hbN1rI3j/u4d31z70aJp8mOMj02Z57wrn80W3RadWp+ ot9kEdYzfoUQjvvKVxdcSIdFb+1UG5O6prw1msI8ogsWyHAzNEqbEj3eos04WhPQOVpP wBPrhjE8RjHAIlPT5DtKv/YIuTUGWYTkZOjpeWUUJZfTjS4SpDPlPD8wzJ+R2jVEHfKe ERbSbsaZYCZm/VQST/wgMIqkANIaEaCvlTSkrJf2So9Ig8+kfNLG9Pifq2Rer/xhFQ3t y2d7nktUjudnzmHhnkMNIrz9oRYJJXnaZsJjmw6f9p0eL1FImvYTX63+eLY7AXXfzubb Cd2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791881787; x=1792486587; h=message-id:in-reply-to:to:references:date:subject:mime-version :content-transfer-encoding:content-type:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=uK3LW/nd/xR2py1trgv05IJ7lBY+cUB6W1L6U9gRmaI=; b=MC1isYOmqMjJ4zdhcBpyOKJm5Fm+yjdEGx9KfTzWk+kFsG8dmqsytUgqLhrN+2NBQf ejNK1Z81pb8nE4DpQOHt5go33/D80lLhnprfHK7qegYFk1mvO8ldbQ+JkstitArdhE/c q6zb/GghPTC7ZwpYFp12NiOVZbaAvnWbqSsWlns+NAEcqt+BtEaDP5POVBCHwHndEKSU hv6VPHjn7UsPzs4ZR4S0VMAKkuzGNZQ4E55uJM5iEEEgAcF/PWQ5NDXDNnw6QbP35Po3 7n4ZUmNxqo1CLciTi7gXZwt0lgwX5l5lO61GQBD6WhZ2rbTDH6bv4RNNYN8p8cfLmShE oOKQ== X-Forwarded-Encrypted: i=1; AKwUvBwHyUGvQDs0pbfkJU433lHVEmeTQ2zh/GdJVmVmJOx2rUP9NI24+nnJWHoouI7OFNDXX7UAMgg3WWXz@lists.postgresql.org X-Gm-Message-State: AFq9FYIHmGONCW7ht7zKNh9RJmKrJPFCG3gUl0CggYDRblvWibdfbw0j 42siehOv+hcV9HdBZLZhX54Z5ZRvg4JqdziTQ2D6hVp5R18m0cNaPQCd X-Gm-Gg: AYBFou25PuCuBXInLUdPle8XCRA3dcVSlvoxtdZBhwSU+UGStfHqyxeLz7lNUU3vmoT f2NtJRBq5g68Eu+F5a+EHE7nCC503yLIjJteiF3ZcK/epSnJ4PGsQsKnSZ3Nkm+zx2F0RZKXaoz ETm8Tzb9fTSJ0bauoGgLx+s6ocArFTXwIwhGirOInj4ZIp/o++ip6r4VjJG2zjYAFi1CO7vVxJN SNGp8LqDi74jkxyheo8cuoxVwo3gNFvN7ZPvnZlRf04h5ZdVHhFaYk1IHSdCjiKgBO/Z1yHFsbK +9e2lE9G5fHF98Gtppl0+TUJt+JtR/riaS78KQgvChMop4K+gZ23CDOaGkF6A7ezWTReVg5yF31 hXREko4AosUTaR9vyOASX5qDNkQgVEEVSj5C2cPKXElkVp5/795DFcJH8v1YiLDAo249o3/lS9R Mud28Kp1YjBo9X0cQow+e2ULMPtgNifkrW9KciAq0n6P3tBbniVjcOqaqjo1QtFFdtAv5j53wx6 cl4JMQ= X-Received: by 2002:a05:7300:c7c4:b0:353:7b29:54ec with SMTP id 5a478bee46e88-3537e2d8ec8mr17058417eec.40.1791881787201; Tue, 13 Oct 2026 01:56:27 -0700 (PDT) Received: from smtpclient.apple ([185.135.79.125]) by smtp.gmail.com with ESMTPSA id 5a478bee46e88-3537ca1b556sm55359568eec.9.2026.10.13.01.56.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 13 Oct 2026 01:56:26 -0700 (PDT) From: Chao Li Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.700.51.1.1\)) Subject: Re: BUG #19754: pg_restore + zlib 1.3.2: intact files rejected, truncated files accepted Date: Tue, 13 Oct 2026 16:55:51 +0800 References: <19754-b335a019b76fd89b@postgresql.org> To: stefangrassl@gmx.net, pgsql-bugs@lists.postgresql.org In-Reply-To: <19754-b335a019b76fd89b@postgresql.org> Message-Id: <25266C45-5546-4496-87E6-E66306936008@gmail.com> X-Mailer: Apple Mail (2.3864.700.51.1.1) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk > On Oct 12, 2026, at 15:53, PG Bug reporting form = wrote: >=20 > The following bug has been logged on the website: >=20 > Bug reference: 19754 > Logged by: Stefan Gra=C3=9Fl > Email address: stefangrassl@gmx.net > PostgreSQL version: 18.6 > Operating system: Windows (EDB binaries); reproduced on Debian 13 > Description: =20 >=20 > zlib 1.3.2 (released 2026-02-17) changed how gzread() handles the end = of a > gzip member. The change breaks the way pg_restore finds damaged data = files > in > gzip-compressed directory-format dumps. pg_restore relies on gzclose() > returning Z_BUF_ERROR to detect a data file that ends in the middle of = a > gzip > stream ("could not close data file"). With zlib 1.3.2 that signal is = wrong > in > both directions: >=20 > 1. Intact files are rejected. A data file fails when its compressed = size is > a multiple of 8192 (zlib's default input buffer) and its = uncompressed > size > is a multiple of pg_restore's read size (4096 in 14 to 18). = pg_restore > issues one more gzread() to detect EOF. zlib 1.3.2 then starts = looking > for > a second gzip member without first checking whether input is left. = It > hits EOF and records Z_BUF_ERROR "unexpected end of file". gzclose() > returns it, and pg_restore stops: >=20 > pg_restore: error: could not close data file "d/3434.dat": Success >=20 > ("No error" on Windows). The table gets zero rows. With one more = byte of > FEXTRA padding in the gzip header, the same file restores fine; the > deflate stream and the trailer stay byte-identical. >=20 > 2. Truncated files are no longer detected. zlib 1.3.2 clears a pending > Z_BUF_ERROR at the start of every gzread() call. The final gzread(), > which returns 0, therefore wipes out the error that the previous = call > recorded, and gzclose() returns Z_OK. pg_restore exits 0 after = loading > only the first part of the table. In the reproduction below, a data = file > cut in half restores 10073 of 20001 rows with exit code 0. With zlib > 1.3.1, the same file makes pg_restore fail as it should. >=20 > Both regressions are fixed on zlib's develop branch but not released = yet: >=20 > a05eacf Fix bug where a complete gzip stream can be reported as > truncated. > 930cdf0 Restore the proper detection of truncated gzip streams. > ("Part of this was lost in 1.3.2.") > ec63cd6 Fix one more truncated stream detection bug. >=20 > Gzip_close() itself does what zlib documents. zlib.h says about = gzread(): > "This error is deferred until gzclose(), which will return Z_BUF_ERROR = if > the last gzread ended in the middle of a gzip stream." The reason for > reporting it here is the Windows binaries. In the EDB binary archives = of > 14.24 and 15.19, pg_dump.exe and pg_restore.exe load the bundled > zlib1.dll, which is 1.3.2. In 18.6, zlib 1.3.2 is linked statically = into > both programs, so replacing the DLL does not help there; the programs = have > to be rebuilt against a fixed zlib. We reproduced (1) with the Windows > binaries of 18.6 and 14.24. Every Windows user who restores a gzip > directory-format dump is therefore exposed to silent partial restores = (2), > and sometimes to (1). Other platforms are affected wherever PostgreSQL = is > built against zlib 1.3.2. >=20 > Related: e01fc63 and e41b01f (2026-10-09, back-patched to 19) added = checks > for truncated compressed data, in pg_restore for zstd and LZ4 and in > pg_verifybackup also for gzip. The gzip reader that pg_restore uses = for > directory-format dumps still relies on gzclose() alone. With zlib = 1.3.2, > gzip is therefore the one compression method whose truncated data = files > pg_restore no longer notices. >=20 > We ran into (1) during a production upgrade. One data file out of = about > 236,000 met both size conditions. It took weeks to see that this file = was > intact, because the message points to corruption, and "Success" or > "No error" gives nothing to go on. >=20 > Suggestions: > - Packagers: build the binaries for the November 12 releases against a = zlib > that contains the three commits above, or against 1.3.1. That release = is > the last one for 14, so 14 would otherwise stay affected for good. > - Give the directory-format gzip reader the same kind of check that > e01fc63 adds for zstd and LZ4: read with inflate() and require > Z_STREAM_END, as e41b01f now does for gzip in pg_verifybackup. = pg_restore > would then no longer depend on how gzread() and gzclose() report the = end > of the stream, and neither (1) nor (2) could occur. > - Perhaps mention this as a known issue, since pg_restore can silently > load only part of a table's data. > - The message prints %m, but no system call failed, so errno is 0 > (EndCompressFileHandle() sets it to 0 before closing). 3cac2c8 = expected > errno-less gzclose() failures to be "basically can't-happen if we're = not > misusing zlib". Printing the zlib status instead, as Kyotaro = Horiguchi > proposed in 2020, would have pointed straight at zlib: > = https://www.postgresql.org/message-id/20201021.152022.1604120629445501521.= horikyota.ntt@gmail.com > Those 2020 reports (10.6, 11.7) predate zlib 1.3.2 and involved = damaged > files, so they are a different problem with the same message. >=20 > 19 and master: DEFAULT_IO_BUFFER_SIZE is now 128 kB. A zlib-level = probe > using that read size shows (1) at usize % 131072 =3D=3D 0 and shows = (2) > unchanged. We have not tested this with a build of 19 or master. >=20 > Reproduction (official postgres image, about two minutes). The script = builds > zlib 1.3.1, 1.3.2 and develop and runs the real pg_restore against = each one > via LD_LIBRARY_PATH: >=20 > docker run -d --name pgz -e POSTGRES_HOST_AUTH_METHOD=3Dtrust = postgres:18 > docker cp repro.sh pgz:/repro.sh > docker exec pgz bash /repro.sh >=20 > ---- repro.sh ---- > #!/bin/bash > # pg_restore against zlib 1.3.1 / 1.3.2 / develop, on an intact data = file > whose > # sizes sit on the buffer boundaries, and on a truncated data file. > # Run inside the official postgres image (server already up, trust = auth). > set -euo pipefail > export PGUSER=3Dpostgres > PGBIN=3D$(ls -d /usr/lib/postgresql/*/bin) > W=3D/tmp/work; rm -rf $W; mkdir -p $W; cd $W >=20 > apt-get -qq update >/dev/null > apt-get -qq install -y gcc make curl ca-certificates >/dev/null 2>&1 > for t in v1.3.1 v1.3.2 develop; do > case $t in > develop) > url=3Dhttps://github.com/madler/zlib/archive/refs/heads/develop.tar.gz = ;; > *) = url=3Dhttps://github.com/madler/zlib/archive/refs/tags/$t.tar.gz > ;; > esac > mkdir zlib-$t && curl -sfL $url | tar xz -C zlib-$t = --strip-components=3D1 > (cd zlib-$t && ./configure >/dev/null && make -s >/dev/null 2>&1) > done >=20 > until pg_isready -q; do sleep 0.5; done > psql -qX -c "create database src" -c "create database dst" > psql -qX -d src -c "create table t (id int, v text)" \ > -c "insert into t select i, md5(i::text) from generate_series(1, = 20000) i" > \ > -c "insert into t values (0, '')" >=20 > isize() { perl -e 'open F, "<:raw", $ARGV[0] or die; seek F, -4, 2; = read F, > $b, 4; print unpack("V", $b)' "$1"; } > dump() { rm -rf d; pg_dump -d src -Fd -f d -t t; f=3D$(ls = d/*.dat.gz); } >=20 > # make the uncompressed data file a multiple of 4096 bytes > dump; u=3D$(isize $f) > psql -qX -d src -c "update t set v =3D repeat('y', $(( (4096 - u % = 4096) % > 4096 ))) where id =3D 0" > dump; cp $f intact.gz > pg_restore -d dst --section=3Dpre-data d >=20 > # insert an FEXTRA field so that the file size is congruent to $2 mod = 8192; > # deflate stream and trailer stay byte-identical > fextra() { > perl -e ' > open F, "<:raw", $ARGV[0] or die; local $/; $d =3D ; close F; > $x =3D (8192 + $ARGV[1] - (length($d) + 2) % 8192) % 8192; > substr($d, 3, 1) =3D chr(ord(substr($d, 3, 1)) | 4); > $d =3D substr($d, 0, 10) . pack("v", $x) . ("\0" x $x) . substr($d, = 10); > open F, ">:raw", $ARGV[0] or die; print F $d; close F' "$1" "$2" > } >=20 > try() { > local label=3D$1 lib=3D$2 rc > psql -qX -d dst -c "truncate t" > set +e; LD_LIBRARY_PATH=3D$lib $PGBIN/pg_restore -d dst = --section=3Ddata d > 2>err.txt; rc=3D$?; set -e > printf ' %-8s %-32s exit=3D%d rows=3D%-6s %s\n' "$label" > "$(LD_LIBRARY_PATH=3D$lib ldd $PGBIN/pg_restore | awk = '/libz\.so/{print $3}')" > \ > $rc "$(psql -qXtA -d dst -c 'select count(*) from t')" "$(tr '\n' ' = ' < > err.txt)" > } >=20 > run_all() { > echo "$1 (csize=3D$(stat -c %s $f), csize%8192=3D$(( $(stat -c %s = $f) % 8192 > )), usize=3D$(isize $f), usize%4096=3D$(( $(isize $f) % 4096 )))" > try system "" > for t in v1.3.1 v1.3.2 develop; do try $t $W/zlib-$t; done > } >=20 > echo "$(pg_restore --version), expected rows: $(psql -qXtA -d src -c = 'select > count(*) from t')" > cp intact.gz $f; fextra $f 0; run_all "A: intact, both sizes on the > boundaries" > cp intact.gz $f; fextra $f 1; run_all "B: same file, one more padding = byte" > cp intact.gz $f; head -c $(( $(stat -c %s intact.gz) / 2 )) intact.gz = > $f; > run_all "C: truncated to half" > ------------------ >=20 > Output with postgres:18 (paths shortened; the usize shown for the = truncated > file is just the garbage in its last four bytes). postgres:14 (14.24) = gives > identical results: >=20 > pg_restore (PostgreSQL) 18.6 (Debian 18.6-1.pgdg13+2), expected rows: = 20001 > A: intact, both sizes on the boundaries (csize=3D442368, = csize%8192=3D0, > usize=3D770048, usize%4096=3D0) > system /lib/x86_64-linux-gnu/libz.so.1 exit=3D0 rows=3D20001 > v1.3.1 zlib-v1.3.1/libz.so.1 exit=3D0 rows=3D20001 > v1.3.2 zlib-v1.3.2/libz.so.1 exit=3D1 rows=3D0 pg_restore: = error: > could not close data file "d/3434.dat": Success > develop zlib-develop/libz.so.1 exit=3D0 rows=3D20001 > B: same file, one more padding byte (csize=3D442369, csize%8192=3D1, > usize=3D770048, usize%4096=3D0) > system /lib/x86_64-linux-gnu/libz.so.1 exit=3D0 rows=3D20001 > v1.3.1 zlib-v1.3.1/libz.so.1 exit=3D0 rows=3D20001 > v1.3.2 zlib-v1.3.2/libz.so.1 exit=3D0 rows=3D20001 > develop zlib-develop/libz.so.1 exit=3D0 rows=3D20001 > C: truncated to half (csize=3D217495, csize%8192=3D4503, = usize=3D3204074365, > usize%4096=3D2941) > system /lib/x86_64-linux-gnu/libz.so.1 exit=3D1 rows=3D0 = pg_restore: > error: could not close data file "d/3434.dat": Success > v1.3.1 zlib-v1.3.1/libz.so.1 exit=3D1 rows=3D0 pg_restore: = error: > could not close data file "d/3434.dat": Success > v1.3.2 zlib-v1.3.2/libz.so.1 exit=3D0 rows=3D10073 > develop zlib-develop/libz.so.1 exit=3D1 rows=3D0 pg_restore: = error: > could not close data file "d/3434.dat": Success >=20 I can reproduce the issue on my MacBook as well, so it doesn=E2=80=99t = seem like platform dependent. We can reuse the same logic of astream_gzip, so that pg_restore would no = longer depend on gzread() and gzclose(). I have created a patch in that = way, and running tests, I may post the patch tomorrow. Best regards, -- Chao Li (Evan) HighGo Software Co., Ltd. https://www.highgo.com/