pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Avoid truncated or zero-padded copies of replaced files in backu 6+ messages / 1 participants [nested] [flat]
* pgsql: Avoid truncated or zero-padded copies of replaced files in backu @ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Avoid truncated or zero-padded copies of replaced files in backups Base backups call stat() on each file while scanning the data directory, and later send exactly that many bytes. If the file is replaced by for example a concurrent rename(), the backup contains the new file truncated or zero-padded to the size of the old one. This is harmless for relation files as WAL replay repairs them. However, it causes harm for non-relation files manipulated as temporary file renamed in place. Here is a list of non-relation files that use a temporary file followed by a final rename() (I may be missing some): * postgresql.auto.conf * pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check and prevents startup) * pg_prewarm's autoprewarm.blocks * pg_stash_advice.tsv. When sending files that are not relation files, fstat() is called again on the opened file and the refreshed stats are used for the tar header and the backup manifest. Since the rename is atomic, the opened file is always complete. Relation files keep using the earlier stat, to avoid copying data that WAL replay would restore anyway (because that's cheaper). Back-patch down to v15. v14 is going to be EOL soon, so I have decided to leave it out of the picture (risk/reward thing). Author: David Steele <david@pgbackrest.org> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Shihao Zhong <zhong950419@gmail.com> Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com Backpatch-through: 15 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/51d7bf5edcaf4f8a95bb10f96b7af76d00a2f83f Modified Files -------------- src/backend/backup/basebackup.c | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Avoid truncated or zero-padded copies of replaced files in backu @ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Avoid truncated or zero-padded copies of replaced files in backups Base backups call stat() on each file while scanning the data directory, and later send exactly that many bytes. If the file is replaced by for example a concurrent rename(), the backup contains the new file truncated or zero-padded to the size of the old one. This is harmless for relation files as WAL replay repairs them. However, it causes harm for non-relation files manipulated as temporary file renamed in place. Here is a list of non-relation files that use a temporary file followed by a final rename() (I may be missing some): * postgresql.auto.conf * pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check and prevents startup) * pg_prewarm's autoprewarm.blocks * pg_stash_advice.tsv. When sending files that are not relation files, fstat() is called again on the opened file and the refreshed stats are used for the tar header and the backup manifest. Since the rename is atomic, the opened file is always complete. Relation files keep using the earlier stat, to avoid copying data that WAL replay would restore anyway (because that's cheaper). Back-patch down to v15. v14 is going to be EOL soon, so I have decided to leave it out of the picture (risk/reward thing). Author: David Steele <david@pgbackrest.org> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Shihao Zhong <zhong950419@gmail.com> Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com Backpatch-through: 15 Branch ------ REL_19_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/2bfed4d18788586940c926e16ee0cebab9e47780 Modified Files -------------- src/backend/backup/basebackup.c | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Avoid truncated or zero-padded copies of replaced files in backu @ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Avoid truncated or zero-padded copies of replaced files in backups Base backups call stat() on each file while scanning the data directory, and later send exactly that many bytes. If the file is replaced by for example a concurrent rename(), the backup contains the new file truncated or zero-padded to the size of the old one. This is harmless for relation files as WAL replay repairs them. However, it causes harm for non-relation files manipulated as temporary file renamed in place. Here is a list of non-relation files that use a temporary file followed by a final rename() (I may be missing some): * postgresql.auto.conf * pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check and prevents startup) * pg_prewarm's autoprewarm.blocks * pg_stash_advice.tsv. When sending files that are not relation files, fstat() is called again on the opened file and the refreshed stats are used for the tar header and the backup manifest. Since the rename is atomic, the opened file is always complete. Relation files keep using the earlier stat, to avoid copying data that WAL replay would restore anyway (because that's cheaper). Back-patch down to v15. v14 is going to be EOL soon, so I have decided to leave it out of the picture (risk/reward thing). Author: David Steele <david@pgbackrest.org> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Shihao Zhong <zhong950419@gmail.com> Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com Backpatch-through: 15 Branch ------ REL_18_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/b09b26ac8e2efa2837c0a4b7780d7df0ce15d0c2 Modified Files -------------- src/backend/backup/basebackup.c | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Avoid truncated or zero-padded copies of replaced files in backu @ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Avoid truncated or zero-padded copies of replaced files in backups Base backups call stat() on each file while scanning the data directory, and later send exactly that many bytes. If the file is replaced by for example a concurrent rename(), the backup contains the new file truncated or zero-padded to the size of the old one. This is harmless for relation files as WAL replay repairs them. However, it causes harm for non-relation files manipulated as temporary file renamed in place. Here is a list of non-relation files that use a temporary file followed by a final rename() (I may be missing some): * postgresql.auto.conf * pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check and prevents startup) * pg_prewarm's autoprewarm.blocks * pg_stash_advice.tsv. When sending files that are not relation files, fstat() is called again on the opened file and the refreshed stats are used for the tar header and the backup manifest. Since the rename is atomic, the opened file is always complete. Relation files keep using the earlier stat, to avoid copying data that WAL replay would restore anyway (because that's cheaper). Back-patch down to v15. v14 is going to be EOL soon, so I have decided to leave it out of the picture (risk/reward thing). Author: David Steele <david@pgbackrest.org> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Shihao Zhong <zhong950419@gmail.com> Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com Backpatch-through: 15 Branch ------ REL_17_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/1fbe31b9d25a6a31977a8a94be469fffe4597f77 Modified Files -------------- src/backend/backup/basebackup.c | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Avoid truncated or zero-padded copies of replaced files in backu @ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Avoid truncated or zero-padded copies of replaced files in backups Base backups call stat() on each file while scanning the data directory, and later send exactly that many bytes. If the file is replaced by for example a concurrent rename(), the backup contains the new file truncated or zero-padded to the size of the old one. This is harmless for relation files as WAL replay repairs them. However, it causes harm for non-relation files manipulated as temporary file renamed in place. Here is a list of non-relation files that use a temporary file followed by a final rename() (I may be missing some): * postgresql.auto.conf * pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check and prevents startup) * pg_prewarm's autoprewarm.blocks * pg_stash_advice.tsv. When sending files that are not relation files, fstat() is called again on the opened file and the refreshed stats are used for the tar header and the backup manifest. Since the rename is atomic, the opened file is always complete. Relation files keep using the earlier stat, to avoid copying data that WAL replay would restore anyway (because that's cheaper). Back-patch down to v15. v14 is going to be EOL soon, so I have decided to leave it out of the picture (risk/reward thing). Author: David Steele <david@pgbackrest.org> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Shihao Zhong <zhong950419@gmail.com> Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com Backpatch-through: 15 Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/61aec3f3bc91c42cd08d1947794ddcc258020869 Modified Files -------------- src/backend/backup/basebackup.c | 51 +++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 15 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Avoid truncated or zero-padded copies of replaced files in backu @ 2026-10-27 04:56 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2026-10-27 04:56 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Avoid truncated or zero-padded copies of replaced files in backups Base backups call stat() on each file while scanning the data directory, and later send exactly that many bytes. If the file is replaced by for example a concurrent rename(), the backup contains the new file truncated or zero-padded to the size of the old one. This is harmless for relation files as WAL replay repairs them. However, it causes harm for non-relation files manipulated as temporary file renamed in place. Here is a list of non-relation files that use a temporary file followed by a final rename() (I may be missing some): * postgresql.auto.conf * pg_logical/replorigin_checkpoint (a damaged copy fails its CRC check and prevents startup) * pg_prewarm's autoprewarm.blocks * pg_stash_advice.tsv. When sending files that are not relation files, fstat() is called again on the opened file and the refreshed stats are used for the tar header and the backup manifest. Since the rename is atomic, the opened file is always complete. Relation files keep using the earlier stat, to avoid copying data that WAL replay would restore anyway (because that's cheaper). Back-patch down to v15. v14 is going to be EOL soon, so I have decided to leave it out of the picture (risk/reward thing). Author: David Steele <david@pgbackrest.org> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Shihao Zhong <zhong950419@gmail.com> Discussion: https://postgr.es/m/CAGRkXqS_On-pLgcv6xjH8mvGhdcPmfWtythW3r8pYKUG+hsiuQ@mail.gmail.com Backpatch-through: 15 Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/1fff28b14af838e3a0e948bfffc98edc02ee5bd3 Modified Files -------------- src/backend/backup/basebackup.c | 51 +++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 15 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2026-10-27 04:56 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz> 2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz> 2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz> 2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz> 2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz> 2026-10-27 04:56 pgsql: Avoid truncated or zero-padded copies of replaced files in backu Michael Paquier <michael@paquier.xyz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox