pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: Avoid truncated or zero-padded copies of replaced files in backu
6+ messages / 1 participants
[nested] [flat]

* pgsql: Avoid truncated or zero-padded copies of replaced files in backu
@ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Avoid truncated or zero-padded copies of replaced files in backups

Base backups call stat() on each file while scanning the data directory,
and later send exactly that many bytes.  If the file is replaced by for
example a concurrent rename(), the backup contains the new file
truncated or zero-padded to the size of the old one.  This is harmless
for relation files as WAL replay repairs them.  However, it causes harm
for non-relation files manipulated as temporary file renamed in place.

Here is a list of non-relation files that use a temporary file followed
by a final rename() (I may be missing some):
* postgresql.auto.conf
* pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check
and prevents startup)
* pg_prewarm's autoprewarm.blocks
* pg_stash_advice.tsv.

When sending files that are not relation files, fstat() is called again
on the opened file and the refreshed stats are used for the tar header
and the backup manifest.  Since the rename
is atomic, the opened file is always complete.  Relation files keep using
the earlier stat, to avoid copying data that WAL replay would restore
anyway (because that's cheaper).

Back-patch down to v15.  v14 is going to be EOL soon, so I have decided
to leave it out of the picture (risk/reward thing).

Author: David Steele <david@pgbackrest.org>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com
Backpatch-through: 15

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/51d7bf5edcaf4f8a95bb10f96b7af76d00a2f83f

Modified Files
--------------
src/backend/backup/basebackup.c | 33 ++++++++++++++++++++++++---------
1 file changed, 24 insertions(+), 9 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Avoid truncated or zero-padded copies of replaced files in backu
@ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Avoid truncated or zero-padded copies of replaced files in backups

Base backups call stat() on each file while scanning the data directory,
and later send exactly that many bytes.  If the file is replaced by for
example a concurrent rename(), the backup contains the new file
truncated or zero-padded to the size of the old one.  This is harmless
for relation files as WAL replay repairs them.  However, it causes harm
for non-relation files manipulated as temporary file renamed in place.

Here is a list of non-relation files that use a temporary file followed
by a final rename() (I may be missing some):
* postgresql.auto.conf
* pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check
and prevents startup)
* pg_prewarm's autoprewarm.blocks
* pg_stash_advice.tsv.

When sending files that are not relation files, fstat() is called again
on the opened file and the refreshed stats are used for the tar header
and the backup manifest.  Since the rename
is atomic, the opened file is always complete.  Relation files keep using
the earlier stat, to avoid copying data that WAL replay would restore
anyway (because that's cheaper).

Back-patch down to v15.  v14 is going to be EOL soon, so I have decided
to leave it out of the picture (risk/reward thing).

Author: David Steele <david@pgbackrest.org>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com
Backpatch-through: 15

Branch
------
REL_19_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/2bfed4d18788586940c926e16ee0cebab9e47780

Modified Files
--------------
src/backend/backup/basebackup.c | 33 ++++++++++++++++++++++++---------
1 file changed, 24 insertions(+), 9 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Avoid truncated or zero-padded copies of replaced files in backu
@ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Avoid truncated or zero-padded copies of replaced files in backups

Base backups call stat() on each file while scanning the data directory,
and later send exactly that many bytes.  If the file is replaced by for
example a concurrent rename(), the backup contains the new file
truncated or zero-padded to the size of the old one.  This is harmless
for relation files as WAL replay repairs them.  However, it causes harm
for non-relation files manipulated as temporary file renamed in place.

Here is a list of non-relation files that use a temporary file followed
by a final rename() (I may be missing some):
* postgresql.auto.conf
* pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check
and prevents startup)
* pg_prewarm's autoprewarm.blocks
* pg_stash_advice.tsv.

When sending files that are not relation files, fstat() is called again
on the opened file and the refreshed stats are used for the tar header
and the backup manifest.  Since the rename
is atomic, the opened file is always complete.  Relation files keep using
the earlier stat, to avoid copying data that WAL replay would restore
anyway (because that's cheaper).

Back-patch down to v15.  v14 is going to be EOL soon, so I have decided
to leave it out of the picture (risk/reward thing).

Author: David Steele <david@pgbackrest.org>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com
Backpatch-through: 15

Branch
------
REL_18_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/b09b26ac8e2efa2837c0a4b7780d7df0ce15d0c2

Modified Files
--------------
src/backend/backup/basebackup.c | 33 ++++++++++++++++++++++++---------
1 file changed, 24 insertions(+), 9 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Avoid truncated or zero-padded copies of replaced files in backu
@ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Avoid truncated or zero-padded copies of replaced files in backups

Base backups call stat() on each file while scanning the data directory,
and later send exactly that many bytes.  If the file is replaced by for
example a concurrent rename(), the backup contains the new file
truncated or zero-padded to the size of the old one.  This is harmless
for relation files as WAL replay repairs them.  However, it causes harm
for non-relation files manipulated as temporary file renamed in place.

Here is a list of non-relation files that use a temporary file followed
by a final rename() (I may be missing some):
* postgresql.auto.conf
* pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check
and prevents startup)
* pg_prewarm's autoprewarm.blocks
* pg_stash_advice.tsv.

When sending files that are not relation files, fstat() is called again
on the opened file and the refreshed stats are used for the tar header
and the backup manifest.  Since the rename
is atomic, the opened file is always complete.  Relation files keep using
the earlier stat, to avoid copying data that WAL replay would restore
anyway (because that's cheaper).

Back-patch down to v15.  v14 is going to be EOL soon, so I have decided
to leave it out of the picture (risk/reward thing).

Author: David Steele <david@pgbackrest.org>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com
Backpatch-through: 15

Branch
------
REL_17_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/1fbe31b9d25a6a31977a8a94be469fffe4597f77

Modified Files
--------------
src/backend/backup/basebackup.c | 33 ++++++++++++++++++++++++---------
1 file changed, 24 insertions(+), 9 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Avoid truncated or zero-padded copies of replaced files in backu
@ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Avoid truncated or zero-padded copies of replaced files in backups

Base backups call stat() on each file while scanning the data directory,
and later send exactly that many bytes.  If the file is replaced by for
example a concurrent rename(), the backup contains the new file
truncated or zero-padded to the size of the old one.  This is harmless
for relation files as WAL replay repairs them.  However, it causes harm
for non-relation files manipulated as temporary file renamed in place.

Here is a list of non-relation files that use a temporary file followed
by a final rename() (I may be missing some):
* postgresql.auto.conf
* pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check
and prevents startup)
* pg_prewarm's autoprewarm.blocks
* pg_stash_advice.tsv.

When sending files that are not relation files, fstat() is called again
on the opened file and the refreshed stats are used for the tar header
and the backup manifest.  Since the rename
is atomic, the opened file is always complete.  Relation files keep using
the earlier stat, to avoid copying data that WAL replay would restore
anyway (because that's cheaper).

Back-patch down to v15.  v14 is going to be EOL soon, so I have decided
to leave it out of the picture (risk/reward thing).

Author: David Steele <david@pgbackrest.org>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com
Backpatch-through: 15

Branch
------
REL_16_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/61aec3f3bc91c42cd08d1947794ddcc258020869

Modified Files
--------------
src/backend/backup/basebackup.c | 51 +++++++++++++++++++++++++++++------------
1 file changed, 36 insertions(+), 15 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Avoid truncated or zero-padded copies of replaced files in backu
@ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Avoid truncated or zero-padded copies of replaced files in backups

Base backups call stat() on each file while scanning the data directory,
and later send exactly that many bytes.  If the file is replaced by for
example a concurrent rename(), the backup contains the new file
truncated or zero-padded to the size of the old one.  This is harmless
for relation files as WAL replay repairs them.  However, it causes harm
for non-relation files manipulated as temporary file renamed in place.

Here is a list of non-relation files that use a temporary file followed
by a final rename() (I may be missing some):
* postgresql.auto.conf
* pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check
and prevents startup)
* pg_prewarm's autoprewarm.blocks
* pg_stash_advice.tsv.

When sending files that are not relation files, fstat() is called again
on the opened file and the refreshed stats are used for the tar header
and the backup manifest.  Since the rename
is atomic, the opened file is always complete.  Relation files keep using
the earlier stat, to avoid copying data that WAL replay would restore
anyway (because that's cheaper).

Back-patch down to v15.  v14 is going to be EOL soon, so I have decided
to leave it out of the picture (risk/reward thing).

Author: David Steele <david@pgbackrest.org>
Reviewed-by: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Reviewed-by: Shihao Zhong <zhong950419@gmail.com>
Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com
Backpatch-through: 15

Branch
------
REL_15_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/1fff28b14af838e3a0e948bfffc98edc02ee5bd3

Modified Files
--------------
src/backend/backup/basebackup.c | 51 +++++++++++++++++++++++++++++------------
1 file changed, 36 insertions(+), 15 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread


end of thread, other threads:[~2026-10-27 04:56 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz>
2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz>
2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz>
2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz>
2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz>
2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox