pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fix bug in archive streamer with LZ4 decompression 5+ messages / 1 participants [nested] [flat]
* pgsql: Fix bug in archive streamer with LZ4 decompression @ 2025-07-02 04:49 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 5+ messages in thread From: Michael Paquier @ 2025-07-02 04:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix bug in archive streamer with LZ4 decompression When decompressing some input data, the calculation for the initial starting point and the initial size were incorrect, potentially leading to failures when decompressing contents with LZ4. These initialization points are fixed in this commit, bringing the logic closer to what exists for gzip and zstd. The contents of the compressed data is clear (for example backups taken with LZ4 can still be decompressed with a "lz4" command), only the decompression part reading the input data was impacted by this issue. This code path impacts pg_basebackup and pg_verifybackup, which can use the LZ4 decompression routines with an archive streamer, or any tools that try to use the archive streamers in src/fe_utils/. The issue is easier to reproduce with files that have a low-compression rate, like ones filled with random data, for a size of at least 512kB, but this could happen with anything as long as it is stored in a data folder. Some tests are added based on this idea, with a file filled with random bytes grabbed from the backend, written at the root of the data folder. This is proving good enough to reproduce the original problem. Author: Mikhail Gribkov <youzhick@gmail.com> Discussion: https://postgr.es/m/CAMEv5_uQS1Hg6KCaEP2JkrTBbZ-nXQhxomWrhYQvbdzR-zy-wA@mail.gmail.com Backpatch-through: 15 Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/5c639523f721e9bd6139916200470e151d690728 Modified Files -------------- src/bin/pg_basebackup/bbstreamer_lz4.c | 4 ++-- src/bin/pg_verifybackup/t/008_untar.pl | 24 ++++++++++++++++++++++++ src/bin/pg_verifybackup/t/010_client_untar.pl | 25 +++++++++++++++++++++++++ 3 files changed, 51 insertions(+), 2 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix bug in archive streamer with LZ4 decompression @ 2025-07-02 04:49 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 5+ messages in thread From: Michael Paquier @ 2025-07-02 04:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix bug in archive streamer with LZ4 decompression When decompressing some input data, the calculation for the initial starting point and the initial size were incorrect, potentially leading to failures when decompressing contents with LZ4. These initialization points are fixed in this commit, bringing the logic closer to what exists for gzip and zstd. The contents of the compressed data is clear (for example backups taken with LZ4 can still be decompressed with a "lz4" command), only the decompression part reading the input data was impacted by this issue. This code path impacts pg_basebackup and pg_verifybackup, which can use the LZ4 decompression routines with an archive streamer, or any tools that try to use the archive streamers in src/fe_utils/. The issue is easier to reproduce with files that have a low-compression rate, like ones filled with random data, for a size of at least 512kB, but this could happen with anything as long as it is stored in a data folder. Some tests are added based on this idea, with a file filled with random bytes grabbed from the backend, written at the root of the data folder. This is proving good enough to reproduce the original problem. Author: Mikhail Gribkov <youzhick@gmail.com> Discussion: https://postgr.es/m/CAMEv5_uQS1Hg6KCaEP2JkrTBbZ-nXQhxomWrhYQvbdzR-zy-wA@mail.gmail.com Backpatch-through: 15 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/3369a3b49b0bc0a4205062e45623af297240c8c6 Modified Files -------------- src/bin/pg_verifybackup/t/008_untar.pl | 22 ++++++++++++++++++++++ src/bin/pg_verifybackup/t/010_client_untar.pl | 22 ++++++++++++++++++++++ src/fe_utils/astreamer_lz4.c | 4 ++-- 3 files changed, 46 insertions(+), 2 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix bug in archive streamer with LZ4 decompression @ 2025-07-02 04:49 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 5+ messages in thread From: Michael Paquier @ 2025-07-02 04:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix bug in archive streamer with LZ4 decompression When decompressing some input data, the calculation for the initial starting point and the initial size were incorrect, potentially leading to failures when decompressing contents with LZ4. These initialization points are fixed in this commit, bringing the logic closer to what exists for gzip and zstd. The contents of the compressed data is clear (for example backups taken with LZ4 can still be decompressed with a "lz4" command), only the decompression part reading the input data was impacted by this issue. This code path impacts pg_basebackup and pg_verifybackup, which can use the LZ4 decompression routines with an archive streamer, or any tools that try to use the archive streamers in src/fe_utils/. The issue is easier to reproduce with files that have a low-compression rate, like ones filled with random data, for a size of at least 512kB, but this could happen with anything as long as it is stored in a data folder. Some tests are added based on this idea, with a file filled with random bytes grabbed from the backend, written at the root of the data folder. This is proving good enough to reproduce the original problem. Author: Mikhail Gribkov <youzhick@gmail.com> Discussion: https://postgr.es/m/CAMEv5_uQS1Hg6KCaEP2JkrTBbZ-nXQhxomWrhYQvbdzR-zy-wA@mail.gmail.com Backpatch-through: 15 Branch ------ REL_17_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/074003431895624197230de80217886b186e4208 Modified Files -------------- src/bin/pg_basebackup/bbstreamer_lz4.c | 4 ++-- src/bin/pg_verifybackup/t/008_untar.pl | 24 ++++++++++++++++++++++++ src/bin/pg_verifybackup/t/010_client_untar.pl | 25 +++++++++++++++++++++++++ 3 files changed, 51 insertions(+), 2 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix bug in archive streamer with LZ4 decompression @ 2025-07-02 04:49 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 5+ messages in thread From: Michael Paquier @ 2025-07-02 04:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix bug in archive streamer with LZ4 decompression When decompressing some input data, the calculation for the initial starting point and the initial size were incorrect, potentially leading to failures when decompressing contents with LZ4. These initialization points are fixed in this commit, bringing the logic closer to what exists for gzip and zstd. The contents of the compressed data is clear (for example backups taken with LZ4 can still be decompressed with a "lz4" command), only the decompression part reading the input data was impacted by this issue. This code path impacts pg_basebackup and pg_verifybackup, which can use the LZ4 decompression routines with an archive streamer, or any tools that try to use the archive streamers in src/fe_utils/. The issue is easier to reproduce with files that have a low-compression rate, like ones filled with random data, for a size of at least 512kB, but this could happen with anything as long as it is stored in a data folder. Some tests are added based on this idea, with a file filled with random bytes grabbed from the backend, written at the root of the data folder. This is proving good enough to reproduce the original problem. Author: Mikhail Gribkov <youzhick@gmail.com> Discussion: https://postgr.es/m/CAMEv5_uQS1Hg6KCaEP2JkrTBbZ-nXQhxomWrhYQvbdzR-zy-wA@mail.gmail.com Backpatch-through: 15 Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/d44efe87ea753f9e5c764f8577cd131b74eb6e21 Modified Files -------------- src/bin/pg_basebackup/bbstreamer_lz4.c | 4 ++-- src/bin/pg_verifybackup/t/008_untar.pl | 24 ++++++++++++++++++++++++ src/bin/pg_verifybackup/t/010_client_untar.pl | 25 +++++++++++++++++++++++++ 3 files changed, 51 insertions(+), 2 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
* pgsql: Fix bug in archive streamer with LZ4 decompression @ 2025-07-02 04:49 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 5+ messages in thread From: Michael Paquier @ 2025-07-02 04:49 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix bug in archive streamer with LZ4 decompression When decompressing some input data, the calculation for the initial starting point and the initial size were incorrect, potentially leading to failures when decompressing contents with LZ4. These initialization points are fixed in this commit, bringing the logic closer to what exists for gzip and zstd. The contents of the compressed data is clear (for example backups taken with LZ4 can still be decompressed with a "lz4" command), only the decompression part reading the input data was impacted by this issue. This code path impacts pg_basebackup and pg_verifybackup, which can use the LZ4 decompression routines with an archive streamer, or any tools that try to use the archive streamers in src/fe_utils/. The issue is easier to reproduce with files that have a low-compression rate, like ones filled with random data, for a size of at least 512kB, but this could happen with anything as long as it is stored in a data folder. Some tests are added based on this idea, with a file filled with random bytes grabbed from the backend, written at the root of the data folder. This is proving good enough to reproduce the original problem. Author: Mikhail Gribkov <youzhick@gmail.com> Discussion: https://postgr.es/m/CAMEv5_uQS1Hg6KCaEP2JkrTBbZ-nXQhxomWrhYQvbdzR-zy-wA@mail.gmail.com Backpatch-through: 15 Branch ------ REL_18_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/d09d1379346eac4b3aa45ee92a5a96bd315bfa4b Modified Files -------------- src/bin/pg_verifybackup/t/008_untar.pl | 22 ++++++++++++++++++++++ src/bin/pg_verifybackup/t/010_client_untar.pl | 22 ++++++++++++++++++++++ src/fe_utils/astreamer_lz4.c | 4 ++-- 3 files changed, 46 insertions(+), 2 deletions(-) ^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2025-07-02 04:49 UTC | newest] Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-07-02 04:49 pgsql: Fix bug in archive streamer with LZ4 decompression Michael Paquier <michael@paquier.xyz> 2025-07-02 04:49 pgsql: Fix bug in archive streamer with LZ4 decompression Michael Paquier <michael@paquier.xyz> 2025-07-02 04:49 pgsql: Fix bug in archive streamer with LZ4 decompression Michael Paquier <michael@paquier.xyz> 2025-07-02 04:49 pgsql: Fix bug in archive streamer with LZ4 decompression Michael Paquier <michael@paquier.xyz> 2025-07-02 04:49 pgsql: Fix bug in archive streamer with LZ4 decompression Michael Paquier <michael@paquier.xyz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox