pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
12+ messages / 4 participants
[nested] [flat]

* BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
@ 2026-10-20 03:41 PG Bug reporting form <noreply@postgresql.org>
  0 siblings, 0 replies; 12+ messages in thread

From: PG Bug reporting form @ 2026-10-20 03:41 UTC (permalink / raw)
  To: pgsql-bugs@lists.postgresql.org; +Cc: kehan5800@gmail.com

The following bug has been logged on the website:

Bug reference:      19776
Logged by:          Ke
Email address:      kehan5800@gmail.com
PostgreSQL version: 18.6
Operating system:   Ubuntu 22.04 (Linux 5.15, x86_64)
Description:        

This happens with a chain of a full backup F and two incremental backups, I1
and I2. VACUUM truncates a table between F and I1, and COPY makes it grow
again
between I1 and I2. "pg_combinebackup F I1 I2" then fills blocks that did not
exist when I1 was taken with their contents from F, so rows deleted before
I1
reappear in the restored table. Sequential scans return those rows, the
primary
key index does not contain them, and bt_index_check() fails. pg_verifybackup
reports all three input backups as valid, and "pg_combinebackup F I1" alone
restores correctly.

Reproduced on master (45fc51fcbf) and 18.6, built from source with
./configure --without-icu (contrib amcheck installed).

---- repro.sh: PGBIN=/usr/local/pgsql/bin sh repro.sh /tmp/chain ----
#!/bin/sh
set -e
W=$1; P=5460
rm -rf "$W"; mkdir -p "$W"
"$PGBIN/initdb" -D "$W/src" -A trust --no-locale >/dev/null
cat >> "$W/src/postgresql.conf" <<EOC
port = $P
unix_socket_directories = '$W'
listen_addresses = ''
summarize_wal = on
autovacuum = off
EOC
"$PGBIN/pg_ctl" -D "$W/src" -l "$W/src.log" -w start >/dev/null
q() { "$PGBIN/psql" -X -h "$W" -p "$1" -d postgres -Atq -c "$2"; }
SEQ="SET enable_indexscan = off; SET enable_indexonlyscan = off; SET
enable_bitmapscan = off;"
q $P "CREATE TABLE t (id int PRIMARY KEY, pad text)"
q $P "INSERT INTO t SELECT g, repeat('h', 20) FROM generate_series(1, 10000)
g"
q $P "INSERT INTO t SELECT 1000000 + g, repeat('x', 20) FROM
generate_series(1, 20000) g"
q $P "VACUUM (FREEZE) t"
"$PGBIN/pg_basebackup" -h "$W" -p $P -D "$W/F" -c fast
q $P "DELETE FROM t WHERE id > 1000000"
q $P "VACUUM t"                             # truncates the heap
"$PGBIN/pg_basebackup" -h "$W" -p $P -D "$W/I1" -c fast --incremental
"$W/F/backup_manifest"
seq 2000001 2005000 | awk '{print $1 "\tnew"}' > "$W/copy.txt"
q $P "COPY t FROM '$W/copy.txt'"            # regrows the heap
"$PGBIN/pg_basebackup" -h "$W" -p $P -D "$W/I2" -c fast --incremental
"$W/I1/backup_manifest"
echo "source:   $(q $P "$SEQ SELECT count(*), count(*) FILTER (WHERE id
BETWEEN 1000001 AND 1999999), pg_relation_size('t') / 8192 FROM t")"
REL=$(q $P "SELECT pg_relation_filepath('t')")
"$PGBIN/pg_ctl" -D "$W/src" -w stop >/dev/null
for b in I1 I2; do
  set -- $(od -A n -t u4 -N 12 "$W/$b/$(dirname $REL)/INCREMENTAL.$(basename
$REL)")
  echo "$b/$(dirname $REL)/INCREMENTAL.$(basename $REL): $2 blocks,
truncation_block_length $3"
done
"$PGBIN/pg_combinebackup" -o "$W/R" "$W/F" "$W/I1" "$W/I2"
echo "port = $((P + 1))" >> "$W/R/postgresql.conf"
"$PGBIN/pg_ctl" -D "$W/R" -l "$W/R.log" -w start >/dev/null
P=$((P + 1))
echo "restored: $(q $P "$SEQ SELECT count(*), count(*) FILTER (WHERE id
BETWEEN 1000001 AND 1999999), pg_relation_size('t') / 8192 FROM t")"
echo "restored, blocks holding deleted rows: $(q $P "$SEQ SELECT
min((ctid::text::point)[0]) || '..' || max((ctid::text::point)[0]) FROM t
WHERE id BETWEEN 1000001 AND 1999999")"
echo "restored, index scan: $(q $P "SET enable_seqscan = off; SET
enable_bitmapscan = off; SELECT count(*) FROM t WHERE id BETWEEN 1000001 AND
1999999")"
q $P "CREATE EXTENSION amcheck"
q $P "SELECT bt_index_check('t_pkey', true)" 2>&1 || true
"$PGBIN/pg_ctl" -D "$W/R" -w stop >/dev/null
---- end ----

Output on master (identical on 18.6):

source:   15000|0|110
I1/base/5/INCREMENTAL.16384: 1 blocks, truncation_block_length 74
I2/base/5/INCREMENTAL.16384: 23 blocks, truncation_block_length 110
restored: 16904|1904|110
restored, blocks holding deleted rows: 96..109
restored, index scan: 0
ERROR:  heap tuple (96,1) from table "t" lacks matching index tuple within
index "t_pkey"
HINT:  Retrying verification using the function bt_index_parent_check()
might provide a more specific error.

Expected: "restored: 15000|0|110", and bt_index_check() succeeds.

In the source cluster, blocks 96..109 of t are all-zero pages that COPY's
bulk
extension added and no WAL record touched, so neither I1 nor I2 contains
them.
The two INCREMENTAL files are consistent with that. I1 records that the file
had 74 blocks at that point, and I2 has truncation_block_length 110 and
holds
the WAL-logged blocks.

reconstruct_from_incremental_file() walks the chain from newest to oldest.
For
every older backup, it uses the newest file's truncation_block_length (110)
as
the bound for the blocks it may take from that backup:

https://github.com/postgres/postgres/blob/45fc51fcbf4534ea2f6f85e36f8155dbd5ba9851/src/bin/pg_combin...
https://github.com/postgres/postgres/blob/45fc51fcbf4534ea2f6f85e36f8155dbd5ba9851/src/bin/pg_combin...

It never lowers that bound to an intermediate file's
truncation_block_length.
So blocks 96..109 come from F, where they hold the rows deleted before I1.
They
should be zero-filled, because I1 says that the file ended at block 74.

Using the running minimum of the truncation lengths fixes it. With this
patch, pg_combinebackup run on the same (unchanged) three backups writes
blocks 96..109 as zeroes, and the script shows "restored: 15000|0|110" with
no
bt_index_check() error:

inline patch 1: src/bin/pg_combinebackup/reconstruct.c (+1/-0 lines) download inline-1.patch--- a/src/bin/pg_combinebackup/reconstruct.c +++ b/src/bin/pg_combinebackup/reconstruct.c @@ -105,6 +105,7 @@ reconstruct_from_incremental_file(char *input_filename, rfile **sourcemap; off_t *offsetmap; unsigned block_length; + unsigned limit; /* older backups may supply
blocks below this */ unsigned sidx = n_prior_backups; bool full_copy_possible = true; int copy_source_index = -1; @@ -128,6 +129,7 @@ reconstruct_from_incremental_file(char *input_filename, latest_source = make_incremental_rfile(input_filename); source[n_prior_backups] = latest_source; block_length = find_reconstructed_block_length(latest_source); + limit = latest_source->truncation_block_length; /* * For each block in the output file, we need to know from which file we @@ -224,7 +226,7 @@ reconstruct_from_incremental_file(char *input_filename, * later backup is one that we can't validate. */ blocklength = sb.st_size / BLCKSZ; - for (b = 0; b < latest_source->truncation_block_length; ++b) + for (b = 0; b < limit; ++b) { if (sourcemap[b] == NULL && b < blocklength) { @@ -238,7 +240,7 @@ reconstruct_from_incremental_file(char *input_filename, * should be exactly as long as the source file is. If so, a full * copy is acceptable, otherwise not. */ - if (full_copy_possible) + if (full_copy_possible && limit == latest_source->truncation_block_length) { uint64 expected_length; @@ -264,7 +266,7 @@ reconstruct_from_incremental_file(char *input_filename, { BlockNumber b = s->relative_block_numbers[i]; - if (b < latest_source->truncation_block_length && + if (b < limit && sourcemap[b] == NULL) { sourcemap[b] = s; @@ -278,6 +280,9 @@ reconstruct_from_incremental_file(char *input_filename, full_copy_possible = false; } } + /* blocks at or above this file's truncation length did not exist then */ + if (s->truncation_block_length < limit) + limit = s->truncation_block_length; } /* Related: "Incremental backup inconsistency after vacuum truncation + COPY regrowth" (pgsql-hackers, 2026-07-29, no replies so far, https://www.postgresql.org/message-id/CAPnDWbagY0Y91qZmJkuJX4+byVJ-dJysnx4QmW23wgR_AvECVg@mail.gmail...) covers the truncation and the COPY regrowth within a single incremental backup. In that case the INCREMENTAL file itself cannot tell the zero pages apart, and the change proposed there is on the server side. In the chain case shown here, the information is in I1 and pg_combinebackup does not use it. A pg_combinebackup fix would also make chains that already exist restore correctly. ^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
@ 2026-10-28 13:16 Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Rajesh Kumar Thandapani @ 2026-10-28 13:16 UTC (permalink / raw)
  To: Ke <kehan5800@gmail.com>; pgsql-bugs@lists.postgresql.org; +Cc: pgsql-hackers@lists.postgresql.org, Robert Haas <robertmhaas@gmail.com>; Dilip Kumar <dilipkumarb@google.com>

Hi Ke,

> This problem occurs because in reconstruct_from_incremental_file(),
> pg_combinebackup determines which blocks require reconstruction based only on
> latest_source->truncation_block_length, without considering the
> truncation_block_length of intermediate backups.

Thanks for the report and reproducer.

While walking backward through the backup chain in
reconstruct_from_incremental_file(), we only check
b < latest_source->truncation_block_length when deciding whether an
unassigned block in sourcemap[] can be taken from an earlier backup. If an
intermediate backup truncated the relation and a later backup extended it
again without writing every block in that range (e.g., unwritten pages at
the end of a relation after bulk extension in COPY), those blocks remain
NULL in sourcemap[] and end up being copied from a backup prior to the
truncation.

Attached is a patch that tracks the current truncation_block_length upper
bound as we step backward through the chain and clamps it whenever an
incremental backup has a smaller s->truncation_block_length. Any remaining
unassigned blocks at or above that limit stay NULL in sourcemap[] and are
written as zeroes.

I also extended 011_ib_truncation.pl to cover an intermediate truncation
followed by COPY bulk extension.

Regards,
Rajesh Kumar Thandapani

Attachments:

v1-0001-Fix-pg_combinebackup-with-truncated-intermediate-.patch (application/x-patch, 5.9K) download From 9c8937ac6ce822b13fbe0903eaca33625990e819 Mon Sep 17 00:00:00 2001 From: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Date: Wed, 28 Oct 2026 10:08:53 +0000 Subject: [PATCH v1] Fix pg_combinebackup with truncated intermediate backups. When reconstructing a relation from a chain of backups, reconstruct_from_incremental_file() walks backward from the latest incremental backup to the full backup, filling sourcemap[] for any block b < latest_source->truncation_block_length that has not yet been assigned a source. However, if an intermediate incremental backup truncated the relation (s->truncation_block_length < latest_source->truncation_block_length) and a later backup extended the relation again without writing every block in the truncated range (for example, via bulk relation extension), those unwritten blocks still had sourcemap[b] == NULL when the loop moved past the intermediate backup. As a result, they were sourced from an earlier backup taken before the truncation, resurrecting stale rows. Fix by tracking the current truncation_block_length upper bound as we walk backward through the backup chain and tightening it whenever an incremental backup has a smaller s->truncation_block_length. Author: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Bug: #19776 Reported-by: Ke <kehan5800@gmail.com> Discussion: https://postgr.es/m/19776-c9e815f92ff29628@postgresql.org --- src/bin/pg_combinebackup/reconstruct.c | 17 +++++- .../pg_combinebackup/t/011_ib_truncation.pl | 59 ++++++++++++++++++- 2 files changed, 73 insertions(+), 3 deletions(-) diff --git a/src/bin/pg_combinebackup/reconstruct.c b/src/bin/pg_combinebackup/reconstruct.c index 38e756b..eb98455 100644 --- a/src/bin/pg_combinebackup/reconstruct.c +++ b/src/bin/pg_combinebackup/reconstruct.c @@ -105,6 +105,7 @@ reconstruct_from_incremental_file(char *input_filename, rfile **sourcemap; off_t *offsetmap; unsigned block_length; + unsigned truncation_block_length; unsigned i; unsigned sidx = n_prior_backups; bool full_copy_possible = true; @@ -129,6 +130,7 @@ reconstruct_from_incremental_file(char *input_filename, latest_source = make_incremental_rfile(input_filename); source[n_prior_backups] = latest_source; block_length = find_reconstructed_block_length(latest_source); + truncation_block_length = latest_source->truncation_block_length; /* * For each block in the output file, we need to know from which file we @@ -225,7 +227,7 @@ reconstruct_from_incremental_file(char *input_filename, * later backup is one that we can't validate. */ blocklength = sb.st_size / BLCKSZ; - for (b = 0; b < latest_source->truncation_block_length; ++b) + for (b = 0; b < truncation_block_length; ++b) { if (sourcemap[b] == NULL && b < blocklength) { @@ -265,7 +267,7 @@ reconstruct_from_incremental_file(char *input_filename, { BlockNumber b = s->relative_block_numbers[i]; - if (b < latest_source->truncation_block_length && + if (b < truncation_block_length && sourcemap[b] == NULL) { sourcemap[b] = s; @@ -279,6 +281,17 @@ reconstruct_from_incremental_file(char *input_filename, full_copy_possible = false; } } + + /* + * Any blocks at or beyond this incremental file's + * truncation_block_length did not exist at the time of this backup, + * so we must not source them from any earlier backup. + */ + if (s->truncation_block_length < truncation_block_length) + { + truncation_block_length = s->truncation_block_length; + full_copy_possible = false; + } } /* diff --git a/src/bin/pg_combinebackup/t/011_ib_truncation.pl b/src/bin/pg_combinebackup/t/011_ib_truncation.pl index 1174928..d6d6427 100644 --- a/src/bin/pg_combinebackup/t/011_ib_truncation.pl +++ b/src/bin/pg_combinebackup/t/011_ib_truncation.pl @@ -115,7 +115,64 @@ my $restored_count = is($restored_count, $rows_after_truncation, 'Restored backup has correct row count'); -$primary->stop; $restored->stop; +# Test that blocks truncated in an intermediate backup are not resurrected +# when a later backup bulk-extends the relation with unwritten zero-filled +# pages. Increase shared_buffers so COPY can bulk-extend by multiple blocks +# at once, grow the table to 30 blocks, truncate it in an intermediate +# backup, and then bulk-extend it with COPY. +$primary->append_conf('postgresql.conf', 'shared_buffers = 8MB'); +$primary->restart; + +my $full_rows = 30; +$primary->safe_psql( + 'postgres', + "INSERT INTO t + SELECT i, + repeat('0123456789ABCDEF0123456789ABCDEF', ($block_size / (2*32))) + FROM generate_series($rows_after_truncation + 1, $full_rows) i; + VACUUM t;" +); + +my $incr_backup = "$backup_path/incr"; +$primary->backup('incr2', + backup_options => [ '--incremental', "$incr_backup/backup_manifest" ]); + +$primary->safe_psql('postgres', + "DELETE FROM t WHERE id > $rows_after_truncation; + VACUUM (TRUNCATE) t;"); + +my $incr2_backup = "$backup_path/incr2"; +$primary->backup('incr3', + backup_options => [ '--incremental', "$incr2_backup/backup_manifest" ]); + +my $copy_rows = 17; +my $row_data = '0123456789ABCDEF0123456789ABCDEF' x ($block_size / (2 * 32)); +my $copy_input = "COPY t FROM STDIN;\n"; +for my $i (1 .. $copy_rows) +{ + $copy_input .= ($full_rows + $i) . "\t$row_data\n"; +} +$copy_input .= "\\.\n"; +$primary->safe_psql('postgres', $copy_input); + +my $incr3_backup = "$backup_path/incr3"; +$primary->backup('incr4', + backup_options => [ '--incremental', "$incr3_backup/backup_manifest" ]); + +my $restored2 = PostgreSQL::Test::Cluster->new('node3'); +$restored2->init_from_backup( + $primary, 'incr4', + combine_with_prior => [ 'full', 'incr', 'incr2', 'incr3' ]); +$restored2->start(); + +$restored_count = + $restored2->safe_psql('postgres', "SELECT count(*) FROM t;"); +is($restored_count, $rows_after_truncation + $copy_rows, + 'Restored backup after intermediate truncation and bulk extension has correct row count'); + +$primary->stop; +$restored2->stop; + done_testing(); -- 2.56.0.307.g5fe7bd680c-goog
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
@ 2026-11-01 07:32 ` Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-03 11:01   ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Rajesh Kumar Thandapani @ 2026-11-01 07:32 UTC (permalink / raw)
  To: Ke <kehan5800@gmail.com>; pgsql-bugs@lists.postgresql.org; +Cc: pgsql-hackers@lists.postgresql.org, Robert Haas <robertmhaas@gmail.com>; Dilip Kumar <dilipkumarb@google.com>

Hi all,

Attaching a rebased v2 of the patch so it applies cleanly against
current master for cfbot. No functional changes from v1.

Thanks,
Rajesh

Attachments:

v2-0001-Fix-pg_combinebackup-with-truncated-intermediate-.patch (application/x-patch, 6.0K) download From d91ccacc29b80dd502cd620f8d62bfe02e24d94b Mon Sep 17 00:00:00 2001 From: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Date: Sun, 1 Nov 2026 07:27:49 +0000 Subject: [PATCH v2] Fix pg_combinebackup with truncated intermediate backups. When reconstructing a relation from a chain of backups, reconstruct_from_incremental_file() walks backward from the latest incremental backup to the full backup, filling sourcemap[] for any block b < latest_source->truncation_block_length that has not yet been assigned a source. However, if an intermediate incremental backup truncated the relation (s->truncation_block_length < latest_source->truncation_block_length) and a later backup extended the relation again without writing every block in the truncated range (for example, via bulk relation extension), those unwritten blocks still had sourcemap[b] == NULL when the loop moved past the intermediate backup. As a result, they were sourced from an earlier backup taken before the truncation, resurrecting stale rows. Fix by tracking the current truncation_block_length upper bound as we walk backward through the backup chain and tightening it whenever an incremental backup has a smaller s->truncation_block_length. Author: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Bug: #19776 Reported-by: Ke <kehan5800@gmail.com> Discussion: https://postgr.es/m/19776-c9e815f92ff29628@postgresql.org --- src/bin/pg_combinebackup/reconstruct.c | 17 +++++- .../pg_combinebackup/t/011_ib_truncation.pl | 59 ++++++++++++++++++- 2 files changed, 73 insertions(+), 3 deletions(-) diff --git a/src/bin/pg_combinebackup/reconstruct.c b/src/bin/pg_combinebackup/reconstruct.c index 5900c5b..5d875cc 100644 --- a/src/bin/pg_combinebackup/reconstruct.c +++ b/src/bin/pg_combinebackup/reconstruct.c @@ -105,6 +105,7 @@ reconstruct_from_incremental_file(char *input_filename, rfile **sourcemap; off_t *offsetmap; unsigned block_length; + unsigned truncation_block_length; unsigned sidx = n_prior_backups; bool full_copy_possible = true; int copy_source_index = -1; @@ -128,6 +129,7 @@ reconstruct_from_incremental_file(char *input_filename, latest_source = make_incremental_rfile(input_filename); source[n_prior_backups] = latest_source; block_length = find_reconstructed_block_length(latest_source); + truncation_block_length = latest_source->truncation_block_length; /* * For each block in the output file, we need to know from which file we @@ -224,7 +226,7 @@ reconstruct_from_incremental_file(char *input_filename, * later backup is one that we can't validate. */ blocklength = sb.st_size / BLCKSZ; - for (b = 0; b < latest_source->truncation_block_length; ++b) + for (b = 0; b < truncation_block_length; ++b) { if (sourcemap[b] == NULL && b < blocklength) { @@ -264,7 +266,7 @@ reconstruct_from_incremental_file(char *input_filename, { BlockNumber b = s->relative_block_numbers[i]; - if (b < latest_source->truncation_block_length && + if (b < truncation_block_length && sourcemap[b] == NULL) { sourcemap[b] = s; @@ -278,6 +280,17 @@ reconstruct_from_incremental_file(char *input_filename, full_copy_possible = false; } } + + /* + * Any blocks at or beyond this incremental file's + * truncation_block_length did not exist at the time of this backup, + * so we must not source them from any earlier backup. + */ + if (s->truncation_block_length < truncation_block_length) + { + truncation_block_length = s->truncation_block_length; + full_copy_possible = false; + } } /* diff --git a/src/bin/pg_combinebackup/t/011_ib_truncation.pl b/src/bin/pg_combinebackup/t/011_ib_truncation.pl index 1174928..d6d6427 100644 --- a/src/bin/pg_combinebackup/t/011_ib_truncation.pl +++ b/src/bin/pg_combinebackup/t/011_ib_truncation.pl @@ -115,7 +115,64 @@ my $restored_count = is($restored_count, $rows_after_truncation, 'Restored backup has correct row count'); -$primary->stop; $restored->stop; +# Test that blocks truncated in an intermediate backup are not resurrected +# when a later backup bulk-extends the relation with unwritten zero-filled +# pages. Increase shared_buffers so COPY can bulk-extend by multiple blocks +# at once, grow the table to 30 blocks, truncate it in an intermediate +# backup, and then bulk-extend it with COPY. +$primary->append_conf('postgresql.conf', 'shared_buffers = 8MB'); +$primary->restart; + +my $full_rows = 30; +$primary->safe_psql( + 'postgres', + "INSERT INTO t + SELECT i, + repeat('0123456789ABCDEF0123456789ABCDEF', ($block_size / (2*32))) + FROM generate_series($rows_after_truncation + 1, $full_rows) i; + VACUUM t;" +); + +my $incr_backup = "$backup_path/incr"; +$primary->backup('incr2', + backup_options => [ '--incremental', "$incr_backup/backup_manifest" ]); + +$primary->safe_psql('postgres', + "DELETE FROM t WHERE id > $rows_after_truncation; + VACUUM (TRUNCATE) t;"); + +my $incr2_backup = "$backup_path/incr2"; +$primary->backup('incr3', + backup_options => [ '--incremental', "$incr2_backup/backup_manifest" ]); + +my $copy_rows = 17; +my $row_data = '0123456789ABCDEF0123456789ABCDEF' x ($block_size / (2 * 32)); +my $copy_input = "COPY t FROM STDIN;\n"; +for my $i (1 .. $copy_rows) +{ + $copy_input .= ($full_rows + $i) . "\t$row_data\n"; +} +$copy_input .= "\\.\n"; +$primary->safe_psql('postgres', $copy_input); + +my $incr3_backup = "$backup_path/incr3"; +$primary->backup('incr4', + backup_options => [ '--incremental', "$incr3_backup/backup_manifest" ]); + +my $restored2 = PostgreSQL::Test::Cluster->new('node3'); +$restored2->init_from_backup( + $primary, 'incr4', + combine_with_prior => [ 'full', 'incr', 'incr2', 'incr3' ]); +$restored2->start(); + +$restored_count = + $restored2->safe_psql('postgres', "SELECT count(*) FROM t;"); +is($restored_count, $rows_after_truncation + $copy_rows, + 'Restored backup after intermediate truncation and bulk extension has correct row count'); + +$primary->stop; +$restored2->stop; + done_testing(); -- 2.56.0.385.gac671c819a-goog
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
@ 2026-11-03 11:01   ` Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-04 14:51     ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Raja Sai pranav @ 2026-11-03 11:01 UTC (permalink / raw)
  To: rajeshkumarth@google.com; +Cc: kehan5800@gmail.com; pgsql-bugs@lists.postgresql.org; pgsql-hackers@lists.postgresql.org; robertmhaas@gmail.com; dilipkumarb@google.com

Hi Rajesh,

I reviewed and tested v2.

v2 applies cleanly to master (316861101e8). Linux aarch64, --enable-cassert
--enable-debug --enable-tap-tests, no warnings.

- With master's reconstruct.c, the new 011_ib_truncation test fails 3 of 3
(restored count 25, expected 24). With the patch it passes 3 of 3, and the
whole pg_combinebackup suite passes.

- On a build with --with-segsize-blocks=6 (one of the CI configurations),
the new test passes even without the fix, 3 of 3. There the truncation
empties whole segments: in incr3, 16384.2 to 16384.4 are zero-length full
files, so for the segment that COPY later re-extends (INCREMENTAL.16384.4
in incr4, no blocks) the walk back stops at incr3's empty full file and
never reaches the stale copy. I would expect it to show only when the
truncation point falls inside the segment that is re-extended (I did not
try to build that case). The fix itself is fine there (the suite passes);
only the test can't see the bug in that configuration. Probably acceptable,
but worth knowing.

- The logic looks right to me. The bound only tightens as we walk back, it
is applied after the current file's own blocks are taken (so blocks the
intermediate backup itself re-extended past its truncation_block_length are
still used), and it is only applied to incremental files.

- Nothing tests the full_copy_possible = false in the new block: with that
line removed, the whole suite still passes. I think the line is needed,
though: without it, a chain where no block comes from an incremental file
and the full file happens to be as long as the latest
truncation_block_length would be copied whole, including the blocks the
intermediate backup had truncated away. I couldn't build that case through
the server (regrowing the relation writes blocks that end up in the later
incremental file), so a test may not be practical. A short comment saying
why the line is there would help.

- Small wording nit in the new comment: blocks at or beyond an incremental
file's truncation_block_length are not necessarily blocks that "did not
exist at the time of this backup"; they may have been truncated away at
some point during that backup's WAL range. Something like "Blocks at or
beyond this file's truncation_block_length were truncated away at some
point before this backup, so their contents in any earlier backup are
stale" might be clearer.

Regards,
Sai Pranav

On Mon, Nov 02, 2026 04:32 PM, Rajesh Kumar Thandapani <
rajeshkumarth@google.com> wrote:

> Hi all,
>
> Attaching a rebased v2 of the patch so it applies cleanly against
> current master for cfbot. No functional changes from v1.
>
> Thanks,
> Rajesh
>

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-03 11:01   ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
@ 2026-11-04 14:51     ` Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-07 15:21       ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Rajesh Kumar Thandapani @ 2026-11-04 14:51 UTC (permalink / raw)
  To: Raja Sai pranav <rajasaipranav0@gmail.com>; pgsql-hackers@lists.postgresql.org; +Cc: Ke <kehan5800@gmail.com>; pgsql-bugs@lists.postgresql.org, Fujii Masao <masao.fujii@gmail.com>; Robert Haas <robertmhaas@gmail.com>; Dilip Kumar <dilipkumarb@google.com>

Hi Sai Pranav,

> - Nothing tests the full_copy_possible = false in the new block: with that
> line removed, the whole suite still passes. I think the line is needed,
> though: without it, a chain where no block comes from an incremental file
> and the full file happens to be as long as the latest
> truncation_block_length would be copied whole, including the blocks the
> intermediate backup had truncated away. I couldn't build that case through
> the server (regrowing the relation writes blocks that end up in the later
> incremental file), so a test may not be practical. A short comment saying
> why the line is there would help.
>
> - Small wording nit in the new comment: blocks at or beyond an incremental
> file's truncation_block_length are not necessarily blocks that "did not
> exist at the time of this backup"; they may have been truncated away at
> some point during that backup's WAL range. Something like "Blocks at or
> beyond this file's truncation_block_length were truncated away at some
> point before this backup, so their contents in any earlier backup are
> stale" might be clearer.

Thanks for reviewing and testing the patch! Both points make sense.
I updated the comment in v3 to clarify the truncation wording and
explain why we also clear full_copy_possible there.

Regards,
Rajesh

Attachments:

v3-0001-Fix-pg_combinebackup-with-truncated-intermediate-.patch (application/x-patch, 6.2K) download From a75c1629d07e6d8a78a6d653377d6127262aa87a Mon Sep 17 00:00:00 2001 From: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Date: Wed, 4 Nov 2026 11:41:32 +0000 Subject: [PATCH v3] Fix pg_combinebackup with truncated intermediate backups. When reconstructing a relation from a chain of backups, reconstruct_from_incremental_file() walks backward from the latest incremental backup to the full backup, filling sourcemap[] for any block b < latest_source->truncation_block_length that has not yet been assigned a source. However, if an intermediate incremental backup truncated the relation (s->truncation_block_length < latest_source->truncation_block_length) and a later backup extended the relation again without writing every block in the truncated range (for example, via bulk relation extension), those unwritten blocks still had sourcemap[b] == NULL when the loop moved past the intermediate backup. As a result, they were sourced from an earlier backup taken before the truncation, resurrecting stale rows. Fix by tracking the current truncation_block_length upper bound as we walk backward through the backup chain and tightening it whenever an incremental backup has a smaller s->truncation_block_length. Author: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Reviewed-by: Raja Sai Pranav <rajasaipranav0@gmail.com> Bug: #19776 Reported-by: Ke <kehan5800@gmail.com> Discussion: https://postgr.es/m/19776-c9e815f92ff29628@postgresql.org --- src/bin/pg_combinebackup/reconstruct.c | 19 +++++- .../pg_combinebackup/t/011_ib_truncation.pl | 59 ++++++++++++++++++- 2 files changed, 75 insertions(+), 3 deletions(-) diff --git a/src/bin/pg_combinebackup/reconstruct.c b/src/bin/pg_combinebackup/reconstruct.c index 5900c5b..f33ad2a 100644 --- a/src/bin/pg_combinebackup/reconstruct.c +++ b/src/bin/pg_combinebackup/reconstruct.c @@ -105,6 +105,7 @@ reconstruct_from_incremental_file(char *input_filename, rfile **sourcemap; off_t *offsetmap; unsigned block_length; + unsigned truncation_block_length; unsigned sidx = n_prior_backups; bool full_copy_possible = true; int copy_source_index = -1; @@ -128,6 +129,7 @@ reconstruct_from_incremental_file(char *input_filename, latest_source = make_incremental_rfile(input_filename); source[n_prior_backups] = latest_source; block_length = find_reconstructed_block_length(latest_source); + truncation_block_length = latest_source->truncation_block_length; /* * For each block in the output file, we need to know from which file we @@ -224,7 +226,7 @@ reconstruct_from_incremental_file(char *input_filename, * later backup is one that we can't validate. */ blocklength = sb.st_size / BLCKSZ; - for (b = 0; b < latest_source->truncation_block_length; ++b) + for (b = 0; b < truncation_block_length; ++b) { if (sourcemap[b] == NULL && b < blocklength) { @@ -264,7 +266,7 @@ reconstruct_from_incremental_file(char *input_filename, { BlockNumber b = s->relative_block_numbers[i]; - if (b < latest_source->truncation_block_length && + if (b < truncation_block_length && sourcemap[b] == NULL) { sourcemap[b] = s; @@ -278,6 +280,19 @@ reconstruct_from_incremental_file(char *input_filename, full_copy_possible = false; } } + + /* + * Blocks at or beyond this incremental file's truncation_block_length + * were either not present or truncated away before this backup, so + * their contents in any earlier backup are stale and must not be + * used. This also prevents copying an earlier full file whole even if + * its size happens to match latest_source->truncation_block_length. + */ + if (s->truncation_block_length < truncation_block_length) + { + truncation_block_length = s->truncation_block_length; + full_copy_possible = false; + } } /* diff --git a/src/bin/pg_combinebackup/t/011_ib_truncation.pl b/src/bin/pg_combinebackup/t/011_ib_truncation.pl index 1174928..d6d6427 100644 --- a/src/bin/pg_combinebackup/t/011_ib_truncation.pl +++ b/src/bin/pg_combinebackup/t/011_ib_truncation.pl @@ -115,7 +115,64 @@ my $restored_count = is($restored_count, $rows_after_truncation, 'Restored backup has correct row count'); -$primary->stop; $restored->stop; +# Test that blocks truncated in an intermediate backup are not resurrected +# when a later backup bulk-extends the relation with unwritten zero-filled +# pages. Increase shared_buffers so COPY can bulk-extend by multiple blocks +# at once, grow the table to 30 blocks, truncate it in an intermediate +# backup, and then bulk-extend it with COPY. +$primary->append_conf('postgresql.conf', 'shared_buffers = 8MB'); +$primary->restart; + +my $full_rows = 30; +$primary->safe_psql( + 'postgres', + "INSERT INTO t + SELECT i, + repeat('0123456789ABCDEF0123456789ABCDEF', ($block_size / (2*32))) + FROM generate_series($rows_after_truncation + 1, $full_rows) i; + VACUUM t;" +); + +my $incr_backup = "$backup_path/incr"; +$primary->backup('incr2', + backup_options => [ '--incremental', "$incr_backup/backup_manifest" ]); + +$primary->safe_psql('postgres', + "DELETE FROM t WHERE id > $rows_after_truncation; + VACUUM (TRUNCATE) t;"); + +my $incr2_backup = "$backup_path/incr2"; +$primary->backup('incr3', + backup_options => [ '--incremental', "$incr2_backup/backup_manifest" ]); + +my $copy_rows = 17; +my $row_data = '0123456789ABCDEF0123456789ABCDEF' x ($block_size / (2 * 32)); +my $copy_input = "COPY t FROM STDIN;\n"; +for my $i (1 .. $copy_rows) +{ + $copy_input .= ($full_rows + $i) . "\t$row_data\n"; +} +$copy_input .= "\\.\n"; +$primary->safe_psql('postgres', $copy_input); + +my $incr3_backup = "$backup_path/incr3"; +$primary->backup('incr4', + backup_options => [ '--incremental', "$incr3_backup/backup_manifest" ]); + +my $restored2 = PostgreSQL::Test::Cluster->new('node3'); +$restored2->init_from_backup( + $primary, 'incr4', + combine_with_prior => [ 'full', 'incr', 'incr2', 'incr3' ]); +$restored2->start(); + +$restored_count = + $restored2->safe_psql('postgres', "SELECT count(*) FROM t;"); +is($restored_count, $rows_after_truncation + $copy_rows, + 'Restored backup after intermediate truncation and bulk extension has correct row count'); + +$primary->stop; +$restored2->stop; + done_testing(); -- 2.56.0.398.g882cbad275-goog
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-03 11:01   ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-04 14:51     ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
@ 2026-11-07 15:21       ` Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 05:12         ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Raja Sai pranav @ 2026-11-07 15:21 UTC (permalink / raw)
  To: rajeshkumarth@google.com; +Cc: pgsql-hackers@lists.postgresql.org; kehan5800@gmail.com; pgsql-bugs@lists.postgresql.org; masao.fujii@gmail.com; robertmhaas@gmail.com; dilipkumarb@google.com

Hi Rajesh,

Thanks - v3 addresses both points from the v2 review.

- v3 applies cleanly to current master (72b9498df9a), including past
fcfd2456078 ("Reject out-of-range block numbers in incremental files"),
which also touches reconstruct.c.
- The reconstruct.c change is the same logic I tested on v2 (track the
running
truncation_block_length upper bound, tighten it on each intermediate backup
with a smaller s->truncation_block_length, applied after the file's own
blocks and only to incremental files, and clear full_copy_possible there).
In my v2 testing that fixed the 011 test (which failed 3/3 on master without
it).
- v3 now adds the bulk-extension regression test I'd flagged as missing: the
node3 / incr2-incr4 chain that bumps shared_buffers so COPY bulk-extends the
relation past an intermediate truncation point, leaving zero-filled
unwritten
pages in the truncated range. That exercises exactly the full_copy_possible
=
false / "unwritten blocks sourced from a pre-truncation backup" path, which
is what the fix guards.
- The reworded comment reads clearly now.

LGTM.

(Note on testing: I re-confirmed v3 applies to current master and re-read
the
diff; the reconstruct.c fix is unchanged from the v2 I built and tested. I
did
not re-run the TAP suite on v3 this round - my macOS TAP environment aborts
in
pg_regress --config-auth independent of the patch - so the pass/fail
evidence
is inherited from the identical v2 fix plus a read of the new test.)

Regards,
Sai Pranav

On Wed, Nov 04, 2026 08:21 PM, Rajesh Kumar Thandapani <
rajeshkumarth@google.com> wrote:

> Hi Sai Pranav,
>
> > - Nothing tests the full_copy_possible = false in the new block: with
> that
> > line removed, the whole suite still passes. I think the line is needed,
> > though: without it, a chain where no block comes from an incremental file
> > and the full file happens to be as long as the latest
> > truncation_block_length would be copied whole, including the blocks the
> > intermediate backup had truncated away. I couldn't build that case
> through
> > the server (regrowing the relation writes blocks that end up in the later
> > incremental file), so a test may not be practical. A short comment saying
> > why the line is there would help.
> >
> > - Small wording nit in the new comment: blocks at or beyond an
> incremental
> > file's truncation_block_length are not necessarily blocks that "did not
> > exist at the time of this backup"; they may have been truncated away at
> > some point during that backup's WAL range. Something like "Blocks at or
> > beyond this file's truncation_block_length were truncated away at some
> > point before this backup, so their contents in any earlier backup are
> > stale" might be clearer.
>
> Thanks for reviewing and testing the patch! Both points make sense.
> I updated the comment in v3 to clarify the truncation wording and
> explain why we also clear full_copy_possible there.
>
> Regards,
> Rajesh
>

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-03 11:01   ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-04 14:51     ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-07 15:21       ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
@ 2026-11-16 05:12         ` Fujii Masao <masao.fujii@gmail.com>
  2026-11-16 06:03           ` Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Fujii Masao @ 2026-11-16 05:12 UTC (permalink / raw)
  To: Raja Sai pranav <rajasaipranav0@gmail.com>; +Cc: rajeshkumarth@google.com; pgsql-hackers@lists.postgresql.org; kehan5800@gmail.com; pgsql-bugs@lists.postgresql.org; robertmhaas@gmail.com; dilipkumarb@google.com

On Sun, Nov 8, 2026 at 12:21 AM Raja Sai pranav
<rajasaipranav0@gmail.com> wrote:
> - v3 now adds the bulk-extension regression test I'd flagged as missing: the
> node3 / incr2-incr4 chain that bumps shared_buffers so COPY bulk-extends the
> relation past an intermediate truncation point, leaving zero-filled unwritten
> pages in the truncated range.

I just wonder if the added test might be fragile. It relies on internal
logics, such as COPY's batching behavior ($copy_rows = 17), buffer pin
limits (shared_buffers = 8MB), and the bulk-extension strategy leaving
pages unwritten. Changes to any of these could easily make the test pass
even without the fix. No?

If so, isn't it better to leave out the test for now? Also because
the reproducer has already demonstrated the bug and the fix.

Regards,


-- 
Fujii Masao





^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-03 11:01   ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-04 14:51     ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-07 15:21       ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 05:12         ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Fujii Masao <masao.fujii@gmail.com>
@ 2026-11-16 06:03           ` Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 13:36             ` Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Raja Sai pranav @ 2026-11-16 06:03 UTC (permalink / raw)
  To: masao.fujii@gmail.com; +Cc: rajeshkumarth@google.com; pgsql-hackers@lists.postgresql.org; kehan5800@gmail.com; pgsql-bugs@lists.postgresql.org; robertmhaas@gmail.com; dilipkumarb@google.com

Hi Fujii-san,

I checked this by running
https://www.google.com/url?q=http://011_ib_truncation.pl&source=gmail&ust=1794895380937000&a...
against current master without the reconstruct.c change, varying $copy_rows
(1, 4, 16, 17, 18, 32, 100) and shared_buffers (8MB, 16MB, 128MB). The new
subtest catches the bug in only 6 of the 21 combinations (copy_rows 16-17
with 8MB, 17-18 with 16MB and 128MB); in all the others it passes without
the fix. So I agree it depends too much on COPY's batching to be a reliable
regression test, and dropping it makes sense, since the reproducer upthread
already demonstrates the bug and the fix.

Regards,
Sai Pranav

On Mon, Nov 16, 2026 10:42 AM, Fujii Masao <masao.fujii@gmail.com> wrote:

> On Sun, Nov 8, 2026 at 12:21 AM Raja Sai pranav
> <rajasaipranav0@gmail.com> wrote:
> > - v3 now adds the bulk-extension regression test I'd flagged as missing:
> the
> > node3 / incr2-incr4 chain that bumps shared_buffers so COPY bulk-extends
> the
> > relation past an intermediate truncation point, leaving zero-filled
> unwritten
> > pages in the truncated range.
>
> I just wonder if the added test might be fragile. It relies on internal
> logics, such as COPY's batching behavior ($copy_rows = 17), buffer pin
> limits (shared_buffers = 8MB), and the bulk-extension strategy leaving
> pages unwritten. Changes to any of these could easily make the test pass
> even without the fix. No?
>
> If so, isn't it better to leave out the test for now? Also because
> the reproducer has already demonstrated the bug and the fix.
>
> Regards,
>
>
> --
> Fujii Masao
>

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-03 11:01   ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-04 14:51     ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-07 15:21       ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 05:12         ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Fujii Masao <masao.fujii@gmail.com>
  2026-11-16 06:03           ` Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
@ 2026-11-16 13:36             ` Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-17 04:17               ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Rajesh Kumar Thandapani @ 2026-11-16 13:36 UTC (permalink / raw)
  To: Raja Sai pranav <rajasaipranav0@gmail.com>; +Cc: masao.fujii@gmail.com; pgsql-hackers@lists.postgresql.org; kehan5800@gmail.com; pgsql-bugs@lists.postgresql.org; robertmhaas@gmail.com; dilipkumarb@google.com

Hi Fujii-san, Sai Pranav,

Thank you both for checking that. I agree that relying on COPY
batching and shared_buffers
makes the test too sensitive to internal heuristics, so I have removed the test
change in the attached v4 patch.

One alternative I was wondering about: in 011_ib_truncation.pl, after taking
the first incremental backup (where table t is truncated from 8 blocks
to 7 blocks),
we could append one block of zeroes directly to the relation file on disk from
Perl before taking a second incremental backup:

my $relpath =
    $primary->safe_psql('postgres', "SELECT pg_relation_filepath('t');");
  my $segfile = $primary->data_dir . "/$relpath";
  $segfile .= ".1" if -f "$segfile.1";
  open(my $fh, '>>:raw', $segfile) or die "open $segfile: $!";
  print $fh ("\0" x $block_size);
  close($fh);

That simulates an unwritten zero-filled page at block 7 without relying on COPY
or shared_buffers, and deterministically reproduces the bug (returning 8 rows
instead of 7 without the fix).

Do you think a test like that is worth adding, or is it better to leave out the
test and go with v4 as-is?
I am okay with either option; I'm just sharing my thoughts on adding a test
without assuming any internal huristics.

Thanks.

Regards,
Rajesh Kumar

On Mon, Nov 16, 2026 at 11:33 AM Raja Sai pranav <rajasaipranav0@gmail.com>
wrote:

> Hi Fujii-san,
>
> I checked this by running
> https://www.google.com/url?q=http://011_ib_truncation.pl&source=gmail&ust=1794895380937000&a...
> against current master without the reconstruct.c change, varying $copy_rows
> (1, 4, 16, 17, 18, 32, 100) and shared_buffers (8MB, 16MB, 128MB). The new
> subtest catches the bug in only 6 of the 21 combinations (copy_rows 16-17
> with 8MB, 17-18 with 16MB and 128MB); in all the others it passes without
> the fix. So I agree it depends too much on COPY's batching to be a reliable
> regression test, and dropping it makes sense, since the reproducer upthread
> already demonstrates the bug and the fix.
>
> Regards,
> Sai Pranav
>
> On Mon, Nov 16, 2026 10:42 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
>
>> On Sun, Nov 8, 2026 at 12:21 AM Raja Sai pranav
>> <rajasaipranav0@gmail.com> wrote:
>> > - v3 now adds the bulk-extension regression test I'd flagged as
>> missing: the
>> > node3 / incr2-incr4 chain that bumps shared_buffers so COPY
>> bulk-extends the
>> > relation past an intermediate truncation point, leaving zero-filled
>> unwritten
>> > pages in the truncated range.
>>
>> I just wonder if the added test might be fragile. It relies on internal
>> logics, such as COPY's batching behavior ($copy_rows = 17), buffer pin
>> limits (shared_buffers = 8MB), and the bulk-extension strategy leaving
>> pages unwritten. Changes to any of these could easily make the test pass
>> even without the fix. No?
>>
>> If so, isn't it better to leave out the test for now? Also because
>> the reproducer has already demonstrated the bug and the fix.
>>
>> Regards,
>>
>>
>> --
>> Fujii Masao
>>
>

Attachments:

v4-0001-Fix-pg_combinebackup-with-truncated-intermediate-.patch (application/x-patch, 3.8K) download From 2a9c12d89c3597c3f30954909464840ab8f227da Mon Sep 17 00:00:00 2001 From: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Date: Mon, 16 Nov 2026 13:12:36 +0000 Subject: [PATCH v4] Fix pg_combinebackup with truncated intermediate backups. When reconstructing a relation from a chain of backups, reconstruct_from_incremental_file() walks backward from the latest incremental backup to the full backup, filling sourcemap[] for any block b < latest_source->truncation_block_length that has not yet been assigned a source. However, if an intermediate incremental backup truncated the relation (s->truncation_block_length < latest_source->truncation_block_length) and a later backup extended the relation again without writing every block in the truncated range (for example, via bulk relation extension), those unwritten blocks still had sourcemap[b] == NULL when the loop moved past the intermediate backup. As a result, they were sourced from an earlier backup taken before the truncation, resurrecting stale rows. Fix by tracking the current truncation_block_length upper bound as we walk backward through the backup chain and tightening it whenever an incremental backup has a smaller s->truncation_block_length. Author: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Reviewed-by: Raja Sai Pranav <rajasaipranav0@gmail.com> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Bug: #19776 Reported-by: Ke <kehan5800@gmail.com> Discussion: https://postgr.es/m/19776-c9e815f92ff29628@postgresql.org --- src/bin/pg_combinebackup/reconstruct.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/src/bin/pg_combinebackup/reconstruct.c b/src/bin/pg_combinebackup/reconstruct.c index 5900c5b..f33ad2a 100644 --- a/src/bin/pg_combinebackup/reconstruct.c +++ b/src/bin/pg_combinebackup/reconstruct.c @@ -105,6 +105,7 @@ reconstruct_from_incremental_file(char *input_filename, rfile **sourcemap; off_t *offsetmap; unsigned block_length; + unsigned truncation_block_length; unsigned sidx = n_prior_backups; bool full_copy_possible = true; int copy_source_index = -1; @@ -128,6 +129,7 @@ reconstruct_from_incremental_file(char *input_filename, latest_source = make_incremental_rfile(input_filename); source[n_prior_backups] = latest_source; block_length = find_reconstructed_block_length(latest_source); + truncation_block_length = latest_source->truncation_block_length; /* * For each block in the output file, we need to know from which file we @@ -224,7 +226,7 @@ reconstruct_from_incremental_file(char *input_filename, * later backup is one that we can't validate. */ blocklength = sb.st_size / BLCKSZ; - for (b = 0; b < latest_source->truncation_block_length; ++b) + for (b = 0; b < truncation_block_length; ++b) { if (sourcemap[b] == NULL && b < blocklength) { @@ -264,7 +266,7 @@ reconstruct_from_incremental_file(char *input_filename, { BlockNumber b = s->relative_block_numbers[i]; - if (b < latest_source->truncation_block_length && + if (b < truncation_block_length && sourcemap[b] == NULL) { sourcemap[b] = s; @@ -278,6 +280,19 @@ reconstruct_from_incremental_file(char *input_filename, full_copy_possible = false; } } + + /* + * Blocks at or beyond this incremental file's truncation_block_length + * were either not present or truncated away before this backup, so + * their contents in any earlier backup are stale and must not be + * used. This also prevents copying an earlier full file whole even if + * its size happens to match latest_source->truncation_block_length. + */ + if (s->truncation_block_length < truncation_block_length) + { + truncation_block_length = s->truncation_block_length; + full_copy_possible = false; + } } /* -- 2.56.1.703.gf848ed807c-goog
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-03 11:01   ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-04 14:51     ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-07 15:21       ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 05:12         ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Fujii Masao <masao.fujii@gmail.com>
  2026-11-16 06:03           ` Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 13:36             ` Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
@ 2026-11-17 04:17               ` Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-17 06:54                 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Raja Sai pranav @ 2026-11-17 04:17 UTC (permalink / raw)
  To: rajeshkumarth@google.com; +Cc: masao.fujii@gmail.com; pgsql-hackers@lists.postgresql.org; kehan5800@gmail.com; pgsql-bugs@lists.postgresql.org; robertmhaas@gmail.com; dilipkumarb@google.com

Hi Rajesh,

I tried the zero-block variant on current master (5ffc821037) with v4's
reconstruct.c, adding the append plus a second incremental backup and a
third node to
https://www.google.com/url?q=http://011_ib_truncation.pl&source=gmail&ust=1794975425465000&a...,
and ran it with shared_buffers at 8MB and 128MB. With the fix it passes at
both sizes; without the fix it fails at both with 8 rows instead of 7. So
unlike the v3 subtest it reproduces the bug deterministically, and nothing
in it depends on COPY batching.

I think it is worth adding. The only thing I would add is a comment saying
why a zero page is appended to the heap file directly, since that is the
exact shape the bug mishandles (an unwritten block past the truncation
point).

Regards,
Sai Pranav

On Mon, Nov 16, 2026 07:06 PM, Rajesh Kumar Thandapani <
rajeshkumarth@google.com> wrote:

> Hi Fujii-san, Sai Pranav,
>
> Thank you both for checking that. I agree that relying on COPY batching and
> shared_buffers makes the test too sensitive to internal heuristics, so I have
> removed the test change in the attached v4 patch.
>
> One alternative I was wondering about: in 011_ib_truncation.pl, after taking
> the first incremental backup (where table t is truncated from 8 blocks to 7
> blocks), we could append one block of zeroes directly to the relation file
> on disk from Perl before taking a second incremental backup:
>
> my $relpath =
>     $primary->safe_psql('postgres', "SELECT pg_relation_filepath('t');");
>   my $segfile = $primary->data_dir . "/$relpath";
>   $segfile .= ".1" if -f "$segfile.1";
>   open(my $fh, '>>:raw', $segfile) or die "open $segfile: $!";
>   print $fh ("\0" x $block_size);
>   close($fh);
>
> That simulates an unwritten zero-filled page at block 7 without relying on
> COPY or shared_buffers, and deterministically reproduces the bug (returning
> 8 rows instead of 7 without the fix).
>
> Do you think a test like that is worth adding, or is it better to leave out
> the test and go with v4 as-is?
> I am okay with either option; I'm just sharing my thoughts on adding a
> test without assuming any internal huristics.
>
> Thanks.
>
> Regards,
> Rajesh Kumar
>
> On Mon, Nov 16, 2026 at 11:33 AM Raja Sai pranav <rajasaipranav0@gmail.com>
> wrote:
>
>> Hi Fujii-san,
>>
>> I checked this by running https://www.google.com/url?q=
>> http://011_ib_truncation.pl&source=gmail&ust=1794895380937000&sa=E
>> against current master without the reconstruct.c change, varying $copy_rows
>> (1, 4, 16, 17, 18, 32, 100) and shared_buffers (8MB, 16MB, 128MB). The new
>> subtest catches the bug in only 6 of the 21 combinations (copy_rows 16-17
>> with 8MB, 17-18 with 16MB and 128MB); in all the others it passes without
>> the fix. So I agree it depends too much on COPY's batching to be a reliable
>> regression test, and dropping it makes sense, since the reproducer upthread
>> already demonstrates the bug and the fix.
>>
>> Regards,
>> Sai Pranav
>>
>> On Mon, Nov 16, 2026 10:42 AM, Fujii Masao <masao.fujii@gmail.com> wrote:
>>
>>> On Sun, Nov 8, 2026 at 12:21 AM Raja Sai pranav
>>> <rajasaipranav0@gmail.com> wrote:
>>> > - v3 now adds the bulk-extension regression test I'd flagged as
>>> missing: the
>>> > node3 / incr2-incr4 chain that bumps shared_buffers so COPY
>>> bulk-extends the
>>> > relation past an intermediate truncation point, leaving zero-filled
>>> unwritten
>>> > pages in the truncated range.
>>>
>>> I just wonder if the added test might be fragile. It relies on internal
>>> logics, such as COPY's batching behavior ($copy_rows = 17), buffer pin
>>> limits (shared_buffers = 8MB), and the bulk-extension strategy leaving
>>> pages unwritten. Changes to any of these could easily make the test pass
>>> even without the fix. No?
>>>
>>> If so, isn't it better to leave out the test for now? Also because
>>> the reproducer has already demonstrated the bug and the fix.
>>>
>>> Regards,
>>>
>>>
>>> --
>>> Fujii Masao
>>>
>>

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-03 11:01   ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-04 14:51     ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-07 15:21       ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 05:12         ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Fujii Masao <masao.fujii@gmail.com>
  2026-11-16 06:03           ` Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 13:36             ` Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-17 04:17               ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
@ 2026-11-17 06:54                 ` Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-18 13:05                   ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Rajesh Kumar Thandapani @ 2026-11-17 06:54 UTC (permalink / raw)
  To: Raja Sai pranav <rajasaipranav0@gmail.com>; rajeshkumarth@google.com; +Cc: masao.fujii@gmail.com; pgsql-hackers@lists.postgresql.org; kehan5800@gmail.com; pgsql-bugs@lists.postgresql.org; robertmhaas@gmail.com; dilipkumarb@google.com

Hi Sai Pranav, Fujii-san,

Thanks for testing that, Sai Pranav!

I have attached v5 split into two patches so Fujii-san can easily pick
either option:
1. v5-0001 contains just the reconstruct.c fix (same as v4).
2. v5-0002 adds the deterministic zero-block append test to
011_ib_truncation.pl (with the primary stopped while appending to the file,
and a comment explaining why the zero page is appended directly).

I am okay with either patch (with or without the test) though I feel the
test adds value by catching any issues immediately.
Thanks.

Regards,
Rajesh Kumar Thandapani

On Tue, Nov 17, 2026 at 4:17 AM Raja Sai pranav <rajasaipranav0@gmail.com>
wrote:

> Hi Rajesh,
>
> I tried the zero-block variant on current master (5ffc821037) with v4's
> reconstruct.c, adding the append plus a second incremental backup and a
> third node to
> https://www.google.com/url?q=http://011_ib_truncation.pl&source=gmail&ust=1794975425465000&a...,
> and ran it with shared_buffers at 8MB and 128MB. With the fix it passes at
> both sizes; without the fix it fails at both with 8 rows instead of 7. So
> unlike the v3 subtest it reproduces the bug deterministically, and nothing
> in it depends on COPY batching.
>
> I think it is worth adding. The only thing I would add is a comment saying
> why a zero page is appended to the heap file directly, since that is the
> exact shape the bug mishandles (an unwritten block past the truncation
> point).
>
> Regards,
> Sai Pranav
>
> On Mon, Nov 16, 2026 07:06 PM, Rajesh Kumar Thandapani <
> rajeshkumarth@google.com> wrote:
>
>> Hi Fujii-san, Sai Pranav,
>>
>> Thank you both for checking that. I agree that relying on COPY batching and
>> shared_buffers makes the test too sensitive to internal heuristics, so I have
>> removed the test change in the attached v4 patch.
>>
>> One alternative I was wondering about: in 011_ib_truncation.pl, after taking
>> the first incremental backup (where table t is truncated from 8 blocks to
>> 7 blocks), we could append one block of zeroes directly to the relation file
>> on disk from Perl before taking a second incremental backup:
>>
>> my $relpath =
>>     $primary->safe_psql('postgres', "SELECT pg_relation_filepath('t');");
>>   my $segfile = $primary->data_dir . "/$relpath";
>>   $segfile .= ".1" if -f "$segfile.1";
>>   open(my $fh, '>>:raw', $segfile) or die "open $segfile: $!";
>>   print $fh ("\0" x $block_size);
>>   close($fh);
>>
>> That simulates an unwritten zero-filled page at block 7 without relying on
>> COPY or shared_buffers, and deterministically reproduces the bug (returning
>> 8 rows instead of 7 without the fix).
>>
>> Do you think a test like that is worth adding, or is it better to leave out
>> the test and go with v4 as-is?
>> I am okay with either option; I'm just sharing my thoughts on adding a
>> test without assuming any internal huristics.
>>
>> Thanks.
>>
>> Regards,
>> Rajesh Kumar
>>
>> On Mon, Nov 16, 2026 at 11:33 AM Raja Sai pranav <
>> rajasaipranav0@gmail.com> wrote:
>>
>>> Hi Fujii-san,
>>>
>>> I checked this by running
>>> https://www.google.com/url?q=http://011_ib_truncation.pl&source=gmail&ust=1794895380937000&a...
>>> against current master without the reconstruct.c change, varying $copy_rows
>>> (1, 4, 16, 17, 18, 32, 100) and shared_buffers (8MB, 16MB, 128MB). The new
>>> subtest catches the bug in only 6 of the 21 combinations (copy_rows 16-17
>>> with 8MB, 17-18 with 16MB and 128MB); in all the others it passes without
>>> the fix. So I agree it depends too much on COPY's batching to be a reliable
>>> regression test, and dropping it makes sense, since the reproducer upthread
>>> already demonstrates the bug and the fix.
>>>
>>> Regards,
>>> Sai Pranav
>>>
>>> On Mon, Nov 16, 2026 10:42 AM, Fujii Masao <masao.fujii@gmail.com>
>>> wrote:
>>>
>>>> On Sun, Nov 8, 2026 at 12:21 AM Raja Sai pranav
>>>> <rajasaipranav0@gmail.com> wrote:
>>>> > - v3 now adds the bulk-extension regression test I'd flagged as
>>>> missing: the
>>>> > node3 / incr2-incr4 chain that bumps shared_buffers so COPY
>>>> bulk-extends the
>>>> > relation past an intermediate truncation point, leaving zero-filled
>>>> unwritten
>>>> > pages in the truncated range.
>>>>
>>>> I just wonder if the added test might be fragile. It relies on internal
>>>> logics, such as COPY's batching behavior ($copy_rows = 17), buffer pin
>>>> limits (shared_buffers = 8MB), and the bulk-extension strategy leaving
>>>> pages unwritten. Changes to any of these could easily make the test pass
>>>> even without the fix. No?
>>>>
>>>> If so, isn't it better to leave out the test for now? Also because
>>>> the reproducer has already demonstrated the bug and the fix.
>>>>
>>>> Regards,
>>>>
>>>>
>>>> --
>>>> Fujii Masao
>>>>
>>>

Attachments: download all (2 files, .tar.gz)

v5-0001-Fix-pg_combinebackup-with-truncated-intermediate-.patch (application/x-patch, 3.8K) download From 35770da48efe7cdb4d208e8f1a9c1b1096045d37 Mon Sep 17 00:00:00 2001 From: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Date: Mon, 16 Nov 2026 13:12:36 +0000 Subject: [PATCH v5 1/2] Fix pg_combinebackup with truncated intermediate backups. When reconstructing a relation from a chain of backups, reconstruct_from_incremental_file() walks backward from the latest incremental backup to the full backup, filling sourcemap[] for any block b < latest_source->truncation_block_length that has not yet been assigned a source. However, if an intermediate incremental backup truncated the relation (s->truncation_block_length < latest_source->truncation_block_length) and a later backup extended the relation again without writing every block in the truncated range (for example, via bulk relation extension), those unwritten blocks still had sourcemap[b] == NULL when the loop moved past the intermediate backup. As a result, they were sourced from an earlier backup taken before the truncation, resurrecting stale rows. Fix by tracking the current truncation_block_length upper bound as we walk backward through the backup chain and tightening it whenever an incremental backup has a smaller s->truncation_block_length. Author: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Reviewed-by: Raja Sai Pranav <rajasaipranav0@gmail.com> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Bug: #19776 Reported-by: Ke <kehan5800@gmail.com> Discussion: https://postgr.es/m/19776-c9e815f92ff29628@postgresql.org --- src/bin/pg_combinebackup/reconstruct.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/src/bin/pg_combinebackup/reconstruct.c b/src/bin/pg_combinebackup/reconstruct.c index 5900c5b..f33ad2a 100644 --- a/src/bin/pg_combinebackup/reconstruct.c +++ b/src/bin/pg_combinebackup/reconstruct.c @@ -105,6 +105,7 @@ reconstruct_from_incremental_file(char *input_filename, rfile **sourcemap; off_t *offsetmap; unsigned block_length; + unsigned truncation_block_length; unsigned sidx = n_prior_backups; bool full_copy_possible = true; int copy_source_index = -1; @@ -128,6 +129,7 @@ reconstruct_from_incremental_file(char *input_filename, latest_source = make_incremental_rfile(input_filename); source[n_prior_backups] = latest_source; block_length = find_reconstructed_block_length(latest_source); + truncation_block_length = latest_source->truncation_block_length; /* * For each block in the output file, we need to know from which file we @@ -224,7 +226,7 @@ reconstruct_from_incremental_file(char *input_filename, * later backup is one that we can't validate. */ blocklength = sb.st_size / BLCKSZ; - for (b = 0; b < latest_source->truncation_block_length; ++b) + for (b = 0; b < truncation_block_length; ++b) { if (sourcemap[b] == NULL && b < blocklength) { @@ -264,7 +266,7 @@ reconstruct_from_incremental_file(char *input_filename, { BlockNumber b = s->relative_block_numbers[i]; - if (b < latest_source->truncation_block_length && + if (b < truncation_block_length && sourcemap[b] == NULL) { sourcemap[b] = s; @@ -278,6 +280,19 @@ reconstruct_from_incremental_file(char *input_filename, full_copy_possible = false; } } + + /* + * Blocks at or beyond this incremental file's truncation_block_length + * were either not present or truncated away before this backup, so + * their contents in any earlier backup are stale and must not be + * used. This also prevents copying an earlier full file whole even if + * its size happens to match latest_source->truncation_block_length. + */ + if (s->truncation_block_length < truncation_block_length) + { + truncation_block_length = s->truncation_block_length; + full_copy_possible = false; + } } /* -- 2.56.0.679.gc527c65306-goog
v5-0002-Add-regression-test-for-truncated-intermediate-ba.patch (application/x-patch, 2.7K) download From 31114c4bc6235ac8e5046946c03dcf08cff27be7 Mon Sep 17 00:00:00 2001 From: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Date: Tue, 17 Nov 2026 05:09:03 +0000 Subject: [PATCH v5 2/2] Add regression test for truncated intermediate backups. Extend 011_ib_truncation.pl to cover a backup chain where an intermediate incremental backup truncates a relation and a subsequent incremental backup sees the relation extended again with an unwritten zero-filled block (appended directly to the stopped primary's heap file to avoid relying on COPY or bulk-extension heuristics). Author: Rajesh Kumar Thandapani <rajeshkumarth@google.com> Reviewed-by: Raja Sai Pranav <rajasaipranav0@gmail.com> Bug: #19776 Discussion: https://postgr.es/m/19776-c9e815f92ff29628@postgresql.org --- .../pg_combinebackup/t/011_ib_truncation.pl | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/src/bin/pg_combinebackup/t/011_ib_truncation.pl b/src/bin/pg_combinebackup/t/011_ib_truncation.pl index 1174928..999560b 100644 --- a/src/bin/pg_combinebackup/t/011_ib_truncation.pl +++ b/src/bin/pg_combinebackup/t/011_ib_truncation.pl @@ -115,7 +115,45 @@ my $restored_count = is($restored_count, $rows_after_truncation, 'Restored backup has correct row count'); +my $relpath = + $primary->safe_psql('postgres', "SELECT pg_relation_filepath('t');"); + $primary->stop; $restored->stop; +# Test that blocks truncated away in an intermediate incremental backup are +# not resurrected from an earlier backup when a later backup extends the +# relation with unwritten zero-filled pages (for example, via bulk relation +# extension). Append a zero-filled block directly to the heap file on disk +# rather than relying on COPY or bulk-extension heuristics so that block 7 is +# guaranteed to exist past incr's truncation point without any WAL record +# referencing it. +my $segfile = $primary->data_dir . "/$relpath"; +$segfile .= ".1" if -f "$segfile.1"; +open(my $fh, '>>:raw', $segfile) or die "open $segfile: $!"; +print $fh ("\0" x $block_size); +close($fh) or die "close $segfile: $!"; + +$primary->start; + +my $incr_backup = "$backup_path/incr"; +$primary->backup('incr2', + backup_options => [ '--incremental', "$incr_backup/backup_manifest" ]); + +$primary->stop; + +my $restored2 = PostgreSQL::Test::Cluster->new('node3'); +$restored2->init_from_backup( + $primary, 'incr2', + combine_with_prior => [ 'full', 'incr' ]); +$restored2->start(); + +$restored_count = + $restored2->safe_psql('postgres', "SELECT count(*) FROM t;"); +is($restored_count, $rows_after_truncation, + 'Restored backup does not resurrect blocks truncated in an intermediate backup' +); + +$restored2->stop; + done_testing(); -- 2.56.0.679.gc527c65306-goog
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup
  2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-01 07:32 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-03 11:01   ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-04 14:51     ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-07 15:21       ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 05:12         ` Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Fujii Masao <masao.fujii@gmail.com>
  2026-11-16 06:03           ` Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-16 13:36             ` Re: Re: Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
  2026-11-17 04:17               ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Raja Sai pranav <rajasaipranav0@gmail.com>
  2026-11-17 06:54                 ` Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
@ 2026-11-18 13:05                   ` Raja Sai pranav <rajasaipranav0@gmail.com>
  0 siblings, 0 replies; 12+ messages in thread

From: Raja Sai pranav @ 2026-11-18 13:05 UTC (permalink / raw)
  To: rajeshkumarth@google.com; +Cc: masao.fujii@gmail.com; pgsql-hackers@lists.postgresql.org; pgsql-bugs@lists.postgresql.org; kehan5800@gmail.com; robertmhaas@gmail.com; dilipkumarb@google.com

Hi Rajesh,

I tested v5 on master (45047c51aa).

v5-0001's reconstruct.c change is the same as v4. With v5-0001 and v5-0002
applied,
https://www.google.com/url?q=http://011_ib_truncation.pl&source=gmail&ust=1795093501571000&a...
passes. With only v5-0002 (master's reconstruct.c), the new subtest fails
with "got 8, expected 7", so the test catches the bug.

Both look good to me. Which to take is Fujii-san's call, but I'd include
0002, since it doesn't depend on COPY batching or shared_buffers.

Thanks,
Sai Pranav

On Tue, Nov 17, 2026 12:24 PM, Rajesh Kumar Thandapani <
rajeshkumarth@google.com> wrote:

> Hi Sai Pranav, Fujii-san,
>
> Thanks for testing that, Sai Pranav!
>
> I have attached v5 split into two patches so Fujii-san can easily pick
> either option:
> 1. v5-0001 contains just the reconstruct.c fix (same as v4).
> 2. v5-0002 adds the deterministic zero-block append test to
> 011_ib_truncation.pl (with the primary stopped while appending to the
> file, and a comment explaining why the zero page is appended directly).
>
> I am okay with either patch (with or without the test) though I feel the
> test adds value by catching any issues immediately.
> Thanks.
>
> Regards,
> Rajesh Kumar Thandapani
>
> On Tue, Nov 17, 2026 at 4:17 AM Raja Sai pranav <rajasaipranav0@gmail.com>
> wrote:
>
>> Hi Rajesh,
>>
>> I tried the zero-block variant on current master (5ffc821037) with v4's
>> reconstruct.c, adding the append plus a second incremental backup and a
>> third node to https://www.google.com/url?q=http://011_ib_truncation.pl;
>> source=gmail&ust=1794975425465000&sa=E, and ran it with shared_buffers
>> at 8MB and 128MB. With the fix it passes at both sizes; without the fix it
>> fails at both with 8 rows instead of 7. So unlike the v3 subtest it
>> reproduces the bug deterministically, and nothing in it depends on COPY
>> batching.
>>
>> I think it is worth adding. The only thing I would add is a comment
>> saying why a zero page is appended to the heap file directly, since that is
>> the exact shape the bug mishandles (an unwritten block past the truncation
>> point).
>>
>> Regards,
>> Sai Pranav
>>
>> On Mon, Nov 16, 2026 07:06 PM, Rajesh Kumar Thandapani <
>> rajeshkumarth@google.com> wrote:
>>
>>> Hi Fujii-san, Sai Pranav,
>>>
>>> Thank you both for checking that. I agree that relying on COPY batching and
>>> shared_buffers makes the test too sensitive to internal heuristics, so I
>>> have removed the test change in the attached v4 patch.
>>>
>>> One alternative I was wondering about: in 011_ib_truncation.pl, after taking
>>> the first incremental backup (where table t is truncated from 8 blocks to
>>> 7 blocks), we could append one block of zeroes directly to the relation file
>>> on disk from Perl before taking a second incremental backup:
>>>
>>> my $relpath =
>>>     $primary->safe_psql('postgres', "SELECT pg_relation_filepath('t');");
>>>   my $segfile = $primary->data_dir . "/$relpath";
>>>   $segfile .= ".1" if -f "$segfile.1";
>>>   open(my $fh, '>>:raw', $segfile) or die "open $segfile: $!";
>>>   print $fh ("\0" x $block_size);
>>>   close($fh);
>>>
>>> That simulates an unwritten zero-filled page at block 7 without relying on
>>> COPY or shared_buffers, and deterministically reproduces the bug (returning
>>> 8 rows instead of 7 without the fix).
>>>
>>> Do you think a test like that is worth adding, or is it better to leave out
>>> the test and go with v4 as-is?
>>> I am okay with either option; I'm just sharing my thoughts on adding a
>>> test without assuming any internal huristics.
>>>
>>> Thanks.
>>>
>>> Regards,
>>> Rajesh Kumar
>>>
>>> On Mon, Nov 16, 2026 at 11:33 AM Raja Sai pranav <
>>> rajasaipranav0@gmail.com> wrote:
>>>
>>>> Hi Fujii-san,
>>>>
>>>> I checked this by running https://www.google.com/url?q=
>>>> http://011_ib_truncation.pl&source=gmail&ust=1794895380937000&sa=E
>>>> against current master without the reconstruct.c change, varying $copy_rows
>>>> (1, 4, 16, 17, 18, 32, 100) and shared_buffers (8MB, 16MB, 128MB). The new
>>>> subtest catches the bug in only 6 of the 21 combinations (copy_rows 16-17
>>>> with 8MB, 17-18 with 16MB and 128MB); in all the others it passes without
>>>> the fix. So I agree it depends too much on COPY's batching to be a reliable
>>>> regression test, and dropping it makes sense, since the reproducer upthread
>>>> already demonstrates the bug and the fix.
>>>>
>>>> Regards,
>>>> Sai Pranav
>>>>
>>>> On Mon, Nov 16, 2026 10:42 AM, Fujii Masao <masao.fujii@gmail.com>
>>>> wrote:
>>>>
>>>>> On Sun, Nov 8, 2026 at 12:21 AM Raja Sai pranav
>>>>> <rajasaipranav0@gmail.com> wrote:
>>>>> > - v3 now adds the bulk-extension regression test I'd flagged as
>>>>> missing: the
>>>>> > node3 / incr2-incr4 chain that bumps shared_buffers so COPY
>>>>> bulk-extends the
>>>>> > relation past an intermediate truncation point, leaving zero-filled
>>>>> unwritten
>>>>> > pages in the truncated range.
>>>>>
>>>>> I just wonder if the added test might be fragile. It relies on internal
>>>>> logics, such as COPY's batching behavior ($copy_rows = 17), buffer pin
>>>>> limits (shared_buffers = 8MB), and the bulk-extension strategy leaving
>>>>> pages unwritten. Changes to any of these could easily make the test
>>>>> pass
>>>>> even without the fix. No?
>>>>>
>>>>> If so, isn't it better to leave out the test for now? Also because
>>>>> the reproducer has already demonstrated the bug and the fix.
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> --
>>>>> Fujii Masao
>>>>>
>>>>

^ permalink  raw  reply  [nested|flat] 12+ messages in thread


end of thread, other threads:[~2026-11-18 13:05 UTC | newest]

Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-20 03:41 BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup PG Bug reporting form <noreply@postgresql.org>
2026-10-28 13:16 Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup Rajesh Kumar Thandapani <rajeshkumarth@google.com>
2026-11-01 07:32 ` Rajesh Kumar Thandapani <rajeshkumarth@google.com>
2026-11-03 11:01   ` Raja Sai pranav <rajasaipranav0@gmail.com>
2026-11-04 14:51     ` Rajesh Kumar Thandapani <rajeshkumarth@google.com>
2026-11-07 15:21       ` Raja Sai pranav <rajasaipranav0@gmail.com>
2026-11-16 05:12         ` Fujii Masao <masao.fujii@gmail.com>
2026-11-16 06:03           ` Raja Sai pranav <rajasaipranav0@gmail.com>
2026-11-16 13:36             ` Rajesh Kumar Thandapani <rajeshkumarth@google.com>
2026-11-17 04:17               ` Raja Sai pranav <rajasaipranav0@gmail.com>
2026-11-17 06:54                 ` Rajesh Kumar Thandapani <rajeshkumarth@google.com>
2026-11-18 13:05                   ` Raja Sai pranav <rajasaipranav0@gmail.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox