Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xJmBR-00000000QYH-0fVz for pgsql-bugs@arkaria.postgresql.org; Thu, 22 Oct 2026 06:21:49 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xJmBO-00000009A74-36pq for pgsql-bugs@arkaria.postgresql.org; Thu, 22 Oct 2026 06:21:46 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xJ0jo-0000000AJfa-2LjZ for pgsql-bugs@lists.postgresql.org; Tue, 20 Oct 2026 03:42:08 +0000 Received: from mahout.postgresql.org ([2001:4800:3e1:1::227]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xJ0jm-000000035WF-0QrS for pgsql-bugs@lists.postgresql.org; Tue, 20 Oct 2026 03:42:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Message-ID:Date:Reply-To:Cc:From:To:Subject: Content-Transfer-Encoding:MIME-Version:Content-Type:Sender:Content-ID: Content-Description:In-Reply-To:References; bh=eYcMOGcLhySF48RFNz/m9g4e4iBzMPePrgJ65SOL3dU=; b=MJI5hbPgrf0F4F9XJ82Bj4ueSG FjEULJ0exXa9X1RK+tpPY7edbpVpiTa1gVpLkN1bvXb4icgHkSJg70a8JgWTkCb0sD5JKeJgHLs2Z +v0LAjftF/A+zdn4pG+YD11yDiX6oaOu8UZhg8fLk4L137VmNz/YTWk7PW2nxaw9cktCnMeNSI/58 Tslhb2/EXHGK3Kdjady5wAE1GFh3x7WebXRL38ybeA504MqZ4e1+8lgtVe12aAhNG/ND2TqNc8Bx7 M1y8eHvXReNyPqS/F0U83VIGVEf3VT6vA8KVsuETM6Cgny4+RXBfzE2n/5UO8IohiX1r9MBy+UDMf hwMY9CKA==; Received: from wrigleys.postgresql.org ([2a02:16a8:dc51::60]) by mahout.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xJ0jk-00000008RXT-3Raw for pgsql-bugs@lists.postgresql.org; Tue, 20 Oct 2026 03:42:04 +0000 Received: from localhost ([127.0.0.1] helo=wrigleys.postgresql.org) by wrigleys.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xJ0jj-00000007cj9-3BUL for pgsql-bugs@lists.postgresql.org; Tue, 20 Oct 2026 03:42:03 +0000 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup To: pgsql-bugs@lists.postgresql.org From: PG Bug reporting form Cc: kehan5800@gmail.com Reply-To: kehan5800@gmail.com, pgsql-bugs@lists.postgresql.org Date: Tue, 20 Oct 2026 03:41:07 +0000 Message-ID: <19776-c9e815f92ff29628@postgresql.org> X-Auto-Response-Suppress: All Auto-Submitted: auto-generated List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk The following bug has been logged on the website: Bug reference: 19776 Logged by: Ke Email address: kehan5800@gmail.com PostgreSQL version: 18.6 Operating system: Ubuntu 22.04 (Linux 5.15, x86_64) Description: =20 This happens with a chain of a full backup F and two incremental backups, I1 and I2. VACUUM truncates a table between F and I1, and COPY makes it grow again between I1 and I2. "pg_combinebackup F I1 I2" then fills blocks that did not exist when I1 was taken with their contents from F, so rows deleted before I1 reappear in the restored table. Sequential scans return those rows, the primary key index does not contain them, and bt_index_check() fails. pg_verifybackup reports all three input backups as valid, and "pg_combinebackup F I1" alone restores correctly. Reproduced on master (45fc51fcbf) and 18.6, built from source with ./configure --without-icu (contrib amcheck installed). ---- repro.sh: PGBIN=3D/usr/local/pgsql/bin sh repro.sh /tmp/chain ---- #!/bin/sh set -e W=3D$1; P=3D5460 rm -rf "$W"; mkdir -p "$W" "$PGBIN/initdb" -D "$W/src" -A trust --no-locale >/dev/null cat >> "$W/src/postgresql.conf" </dev/null q() { "$PGBIN/psql" -X -h "$W" -p "$1" -d postgres -Atq -c "$2"; } SEQ=3D"SET enable_indexscan =3D off; SET enable_indexonlyscan =3D off; SET enable_bitmapscan =3D off;" q $P "CREATE TABLE t (id int PRIMARY KEY, pad text)" q $P "INSERT INTO t SELECT g, repeat('h', 20) FROM generate_series(1, 10000) g" q $P "INSERT INTO t SELECT 1000000 + g, repeat('x', 20) FROM generate_series(1, 20000) g" q $P "VACUUM (FREEZE) t" "$PGBIN/pg_basebackup" -h "$W" -p $P -D "$W/F" -c fast q $P "DELETE FROM t WHERE id > 1000000" q $P "VACUUM t" # truncates the heap "$PGBIN/pg_basebackup" -h "$W" -p $P -D "$W/I1" -c fast --incremental "$W/F/backup_manifest" seq 2000001 2005000 | awk '{print $1 "\tnew"}' > "$W/copy.txt" q $P "COPY t FROM '$W/copy.txt'" # regrows the heap "$PGBIN/pg_basebackup" -h "$W" -p $P -D "$W/I2" -c fast --incremental "$W/I1/backup_manifest" echo "source: $(q $P "$SEQ SELECT count(*), count(*) FILTER (WHERE id BETWEEN 1000001 AND 1999999), pg_relation_size('t') / 8192 FROM t")" REL=3D$(q $P "SELECT pg_relation_filepath('t')") "$PGBIN/pg_ctl" -D "$W/src" -w stop >/dev/null for b in I1 I2; do set -- $(od -A n -t u4 -N 12 "$W/$b/$(dirname $REL)/INCREMENTAL.$(basename $REL)") echo "$b/$(dirname $REL)/INCREMENTAL.$(basename $REL): $2 blocks, truncation_block_length $3" done "$PGBIN/pg_combinebackup" -o "$W/R" "$W/F" "$W/I1" "$W/I2" echo "port =3D $((P + 1))" >> "$W/R/postgresql.conf" "$PGBIN/pg_ctl" -D "$W/R" -l "$W/R.log" -w start >/dev/null P=3D$((P + 1)) echo "restored: $(q $P "$SEQ SELECT count(*), count(*) FILTER (WHERE id BETWEEN 1000001 AND 1999999), pg_relation_size('t') / 8192 FROM t")" echo "restored, blocks holding deleted rows: $(q $P "$SEQ SELECT min((ctid::text::point)[0]) || '..' || max((ctid::text::point)[0]) FROM t WHERE id BETWEEN 1000001 AND 1999999")" echo "restored, index scan: $(q $P "SET enable_seqscan =3D off; SET enable_bitmapscan =3D off; SELECT count(*) FROM t WHERE id BETWEEN 1000001 = AND 1999999")" q $P "CREATE EXTENSION amcheck" q $P "SELECT bt_index_check('t_pkey', true)" 2>&1 || true "$PGBIN/pg_ctl" -D "$W/R" -w stop >/dev/null ---- end ---- Output on master (identical on 18.6): source: 15000|0|110 I1/base/5/INCREMENTAL.16384: 1 blocks, truncation_block_length 74 I2/base/5/INCREMENTAL.16384: 23 blocks, truncation_block_length 110 restored: 16904|1904|110 restored, blocks holding deleted rows: 96..109 restored, index scan: 0 ERROR: heap tuple (96,1) from table "t" lacks matching index tuple within index "t_pkey" HINT: Retrying verification using the function bt_index_parent_check() might provide a more specific error. Expected: "restored: 15000|0|110", and bt_index_check() succeeds. In the source cluster, blocks 96..109 of t are all-zero pages that COPY's bulk extension added and no WAL record touched, so neither I1 nor I2 contains them. The two INCREMENTAL files are consistent with that. I1 records that the file had 74 blocks at that point, and I2 has truncation_block_length 110 and holds the WAL-logged blocks. reconstruct_from_incremental_file() walks the chain from newest to oldest. For every older backup, it uses the newest file's truncation_block_length (110) as the bound for the blocks it may take from that backup: https://github.com/postgres/postgres/blob/45fc51fcbf4534ea2f6f85e36f8155dbd= 5ba9851/src/bin/pg_combinebackup/reconstruct.c#L227 https://github.com/postgres/postgres/blob/45fc51fcbf4534ea2f6f85e36f8155dbd= 5ba9851/src/bin/pg_combinebackup/reconstruct.c#L267 It never lowers that bound to an intermediate file's truncation_block_length. So blocks 96..109 come from F, where they hold the rows deleted before I1. They should be zero-filled, because I1 says that the file ended at block 74. Using the running minimum of the truncation lengths fixes it. With this patch, pg_combinebackup run on the same (unchanged) three backups writes blocks 96..109 as zeroes, and the script shows "restored: 15000|0|110" with no bt_index_check() error: --- a/src/bin/pg_combinebackup/reconstruct.c +++ b/src/bin/pg_combinebackup/reconstruct.c @@ -105,6 +105,7 @@ reconstruct_from_incremental_file(char *input_filename, rfile **sourcemap; off_t *offsetmap; unsigned block_length; + unsigned limit; /* older backups may supply blocks below this */ unsigned sidx =3D n_prior_backups; bool full_copy_possible =3D true; int copy_source_index =3D -1; @@ -128,6 +129,7 @@ reconstruct_from_incremental_file(char *input_filename, latest_source =3D make_incremental_rfile(input_filename); source[n_prior_backups] =3D latest_source; block_length =3D find_reconstructed_block_length(latest_source); + limit =3D latest_source->truncation_block_length; /* * For each block in the output file, we need to know from which file we @@ -224,7 +226,7 @@ reconstruct_from_incremental_file(char *input_filename, * later backup is one that we can't validate. */ blocklength =3D sb.st_size / BLCKSZ; - for (b =3D 0; b < latest_source->truncation_block_length; ++b) + for (b =3D 0; b < limit; ++b) { if (sourcemap[b] =3D=3D NULL && b < blockle= ngth) { @@ -238,7 +240,7 @@ reconstruct_from_incremental_file(char *input_filename, * should be exactly as long as the source file is. If so, a full * copy is acceptable, otherwise not. */ - if (full_copy_possible) + if (full_copy_possible && limit =3D=3D latest_source->truncation_block_length) { uint64 expected_length; @@ -264,7 +266,7 @@ reconstruct_from_incremental_file(char *input_filename, { BlockNumber b =3D s->relative_block_numbers[i]; - if (b < latest_source->truncation_block_length && + if (b < limit && sourcemap[b] =3D=3D NULL) { sourcemap[b] =3D s; @@ -278,6 +280,9 @@ reconstruct_from_incremental_file(char *input_filename, full_copy_possible =3D false; } } + /* blocks at or above this file's truncation length did not exist then */ + if (s->truncation_block_length < limit) + limit =3D s->truncation_block_length; } /* Related: "Incremental backup inconsistency after vacuum truncation + COPY regrowth" (pgsql-hackers, 2026-07-29, no replies so far, https://www.postgresql.org/message-id/CAPnDWbagY0Y91qZmJkuJX4+byVJ-dJysnx4Q= mW23wgR_AvECVg@mail.gmail.com) covers the truncation and the COPY regrowth within a single incremental backup. In that case the INCREMENTAL file itself cannot tell the zero pages apart, and the change proposed there is on the server side. In the chain case shown here, the information is in I1 and pg_combinebackup does not use it. A pg_combinebackup fix would also make chains that already exist restore correctly.