pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Fail hard on out-of-memory failures in xlogreader.c 6+ messages / 1 participants [nested] [flat]
* pgsql: Fail hard on out-of-memory failures in xlogreader.c @ 2023-10-03 01:26 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2023-10-03 01:26 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fail hard on out-of-memory failures in xlogreader.c This commit changes the WAL reader routines so as a FATAL for the backend or exit(FAILURE) for the frontend is triggered if an allocation for a WAL record decode fails in walreader.c, rather than treating this case as bogus data, which would be equivalent to the end of WAL. The key is to avoid palloc_extended(MCXT_ALLOC_NO_OOM) in walreader.c, relying on plain palloc() calls. The previous behavior could make WAL replay finish too early than it should. For example, crash recovery finishing earlier may corrupt clusters because not all the WAL available locally was replayed to ensure a consistent state. Out-of-memory failures would show up randomly depending on the memory pressure on the host, but one simple case would be to generate a large record, then replay this record after downsizing a host, as Ethan Mertz originally reported. This relies on bae868caf222, as the WAL reader routines now do the memory allocation required for a record only once its header has been fully read and validated, making xl_tot_len trustable. Making the WAL reader react differently on out-of-memory or bogus record data would require ABI changes, so this is the safest choice for stable branches. Also, it is worth noting that 3f1ce973467a has been using a plain palloc() in this code for some time now. Thanks to Noah Misch and Thomas Munro for the discussion. Like the other commit, backpatch down to 12, leaving out v11 that will be EOL'd soon. The behavior of considering a failed allocation as bogus data comes originally from 0ffe11abd3a0, where the record length retrieved from its header was not entirely trustable. Reported-by: Ethan Mertz Discussion: https://postgr.es/m/ZRKKdI5-RRlta3aF@paquier.xyz Backpatch-through: 12 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/6b18b3fe2c2f375d4a5d8a69b67bc59d13d4c844 Modified Files -------------- src/backend/access/transam/xlogreader.c | 47 ++++++--------------------------- 1 file changed, 8 insertions(+), 39 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fail hard on out-of-memory failures in xlogreader.c @ 2023-10-03 01:26 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2023-10-03 01:26 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fail hard on out-of-memory failures in xlogreader.c This commit changes the WAL reader routines so as a FATAL for the backend or exit(FAILURE) for the frontend is triggered if an allocation for a WAL record decode fails in walreader.c, rather than treating this case as bogus data, which would be equivalent to the end of WAL. The key is to avoid palloc_extended(MCXT_ALLOC_NO_OOM) in walreader.c, relying on plain palloc() calls. The previous behavior could make WAL replay finish too early than it should. For example, crash recovery finishing earlier may corrupt clusters because not all the WAL available locally was replayed to ensure a consistent state. Out-of-memory failures would show up randomly depending on the memory pressure on the host, but one simple case would be to generate a large record, then replay this record after downsizing a host, as Ethan Mertz originally reported. This relies on bae868caf222, as the WAL reader routines now do the memory allocation required for a record only once its header has been fully read and validated, making xl_tot_len trustable. Making the WAL reader react differently on out-of-memory or bogus record data would require ABI changes, so this is the safest choice for stable branches. Also, it is worth noting that 3f1ce973467a has been using a plain palloc() in this code for some time now. Thanks to Noah Misch and Thomas Munro for the discussion. Like the other commit, backpatch down to 12, leaving out v11 that will be EOL'd soon. The behavior of considering a failed allocation as bogus data comes originally from 0ffe11abd3a0, where the record length retrieved from its header was not entirely trustable. Reported-by: Ethan Mertz Discussion: https://postgr.es/m/ZRKKdI5-RRlta3aF@paquier.xyz Backpatch-through: 12 Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/a06efbc3ac72a455a63635c9cbc56ad3338988cc Modified Files -------------- src/backend/access/transam/xlogreader.c | 47 ++++++--------------------------- 1 file changed, 8 insertions(+), 39 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fail hard on out-of-memory failures in xlogreader.c @ 2023-10-03 01:26 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2023-10-03 01:26 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fail hard on out-of-memory failures in xlogreader.c This commit changes the WAL reader routines so as a FATAL for the backend or exit(FAILURE) for the frontend is triggered if an allocation for a WAL record decode fails in walreader.c, rather than treating this case as bogus data, which would be equivalent to the end of WAL. The key is to avoid palloc_extended(MCXT_ALLOC_NO_OOM) in walreader.c, relying on plain palloc() calls. The previous behavior could make WAL replay finish too early than it should. For example, crash recovery finishing earlier may corrupt clusters because not all the WAL available locally was replayed to ensure a consistent state. Out-of-memory failures would show up randomly depending on the memory pressure on the host, but one simple case would be to generate a large record, then replay this record after downsizing a host, as Ethan Mertz originally reported. This relies on bae868caf222, as the WAL reader routines now do the memory allocation required for a record only once its header has been fully read and validated, making xl_tot_len trustable. Making the WAL reader react differently on out-of-memory or bogus record data would require ABI changes, so this is the safest choice for stable branches. Also, it is worth noting that 3f1ce973467a has been using a plain palloc() in this code for some time now. Thanks to Noah Misch and Thomas Munro for the discussion. Like the other commit, backpatch down to 12, leaving out v11 that will be EOL'd soon. The behavior of considering a failed allocation as bogus data comes originally from 0ffe11abd3a0, where the record length retrieved from its header was not entirely trustable. Reported-by: Ethan Mertz Discussion: https://postgr.es/m/ZRKKdI5-RRlta3aF@paquier.xyz Backpatch-through: 12 Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/afc79591de8e2facc3e22f8746ee2b7e59a2a741 Modified Files -------------- src/backend/access/transam/xlogreader.c | 47 ++++++--------------------------- 1 file changed, 8 insertions(+), 39 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fail hard on out-of-memory failures in xlogreader.c @ 2023-10-03 01:26 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2023-10-03 01:26 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fail hard on out-of-memory failures in xlogreader.c This commit changes the WAL reader routines so as a FATAL for the backend or exit(FAILURE) for the frontend is triggered if an allocation for a WAL record decode fails in walreader.c, rather than treating this case as bogus data, which would be equivalent to the end of WAL. The key is to avoid palloc_extended(MCXT_ALLOC_NO_OOM) in walreader.c, relying on plain palloc() calls. The previous behavior could make WAL replay finish too early than it should. For example, crash recovery finishing earlier may corrupt clusters because not all the WAL available locally was replayed to ensure a consistent state. Out-of-memory failures would show up randomly depending on the memory pressure on the host, but one simple case would be to generate a large record, then replay this record after downsizing a host, as Ethan Mertz originally reported. This relies on bae868caf222, as the WAL reader routines now do the memory allocation required for a record only once its header has been fully read and validated, making xl_tot_len trustable. Making the WAL reader react differently on out-of-memory or bogus record data would require ABI changes, so this is the safest choice for stable branches. Also, it is worth noting that 3f1ce973467a has been using a plain palloc() in this code for some time now. Thanks to Noah Misch and Thomas Munro for the discussion. Like the other commit, backpatch down to 12, leaving out v11 that will be EOL'd soon. The behavior of considering a failed allocation as bogus data comes originally from 0ffe11abd3a0, where the record length retrieved from its header was not entirely trustable. Reported-by: Ethan Mertz Discussion: https://postgr.es/m/ZRKKdI5-RRlta3aF@paquier.xyz Backpatch-through: 12 Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/baeb8542c1cc849e58c016f401687b9c4d3e7f20 Modified Files -------------- src/backend/access/transam/xlogreader.c | 31 +++++-------------------------- 1 file changed, 5 insertions(+), 26 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fail hard on out-of-memory failures in xlogreader.c @ 2023-10-03 01:26 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2023-10-03 01:26 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fail hard on out-of-memory failures in xlogreader.c This commit changes the WAL reader routines so as a FATAL for the backend or exit(FAILURE) for the frontend is triggered if an allocation for a WAL record decode fails in walreader.c, rather than treating this case as bogus data, which would be equivalent to the end of WAL. The key is to avoid palloc_extended(MCXT_ALLOC_NO_OOM) in walreader.c, relying on plain palloc() calls. The previous behavior could make WAL replay finish too early than it should. For example, crash recovery finishing earlier may corrupt clusters because not all the WAL available locally was replayed to ensure a consistent state. Out-of-memory failures would show up randomly depending on the memory pressure on the host, but one simple case would be to generate a large record, then replay this record after downsizing a host, as Ethan Mertz originally reported. This relies on bae868caf222, as the WAL reader routines now do the memory allocation required for a record only once its header has been fully read and validated, making xl_tot_len trustable. Making the WAL reader react differently on out-of-memory or bogus record data would require ABI changes, so this is the safest choice for stable branches. Also, it is worth noting that 3f1ce973467a has been using a plain palloc() in this code for some time now. Thanks to Noah Misch and Thomas Munro for the discussion. Like the other commit, backpatch down to 12, leaving out v11 that will be EOL'd soon. The behavior of considering a failed allocation as bogus data comes originally from 0ffe11abd3a0, where the record length retrieved from its header was not entirely trustable. Reported-by: Ethan Mertz Discussion: https://postgr.es/m/ZRKKdI5-RRlta3aF@paquier.xyz Backpatch-through: 12 Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/6103d2ccaed48f8e9599d67f96302d9bae208a8e Modified Files -------------- src/backend/access/transam/xlogreader.c | 32 +++++--------------------------- 1 file changed, 5 insertions(+), 27 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Fail hard on out-of-memory failures in xlogreader.c @ 2023-10-03 01:26 Michael Paquier <michael@paquier.xyz> 0 siblings, 0 replies; 6+ messages in thread From: Michael Paquier @ 2023-10-03 01:26 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fail hard on out-of-memory failures in xlogreader.c This commit changes the WAL reader routines so as a FATAL for the backend or exit(FAILURE) for the frontend is triggered if an allocation for a WAL record decode fails in walreader.c, rather than treating this case as bogus data, which would be equivalent to the end of WAL. The key is to avoid palloc_extended(MCXT_ALLOC_NO_OOM) in walreader.c, relying on plain palloc() calls. The previous behavior could make WAL replay finish too early than it should. For example, crash recovery finishing earlier may corrupt clusters because not all the WAL available locally was replayed to ensure a consistent state. Out-of-memory failures would show up randomly depending on the memory pressure on the host, but one simple case would be to generate a large record, then replay this record after downsizing a host, as Ethan Mertz originally reported. This relies on bae868caf222, as the WAL reader routines now do the memory allocation required for a record only once its header has been fully read and validated, making xl_tot_len trustable. Making the WAL reader react differently on out-of-memory or bogus record data would require ABI changes, so this is the safest choice for stable branches. Also, it is worth noting that 3f1ce973467a has been using a plain palloc() in this code for some time now. Thanks to Noah Misch and Thomas Munro for the discussion. Like the other commit, backpatch down to 12, leaving out v11 that will be EOL'd soon. The behavior of considering a failed allocation as bogus data comes originally from 0ffe11abd3a0, where the record length retrieved from its header was not entirely trustable. Reported-by: Ethan Mertz Discussion: https://postgr.es/m/ZRKKdI5-RRlta3aF@paquier.xyz Backpatch-through: 12 Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/22b2e6e9df54f4d38e2fd20a882505d87c1f169f Modified Files -------------- src/backend/access/transam/xlogreader.c | 32 +++++--------------------------- 1 file changed, 5 insertions(+), 27 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2023-10-03 01:26 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-10-03 01:26 pgsql: Fail hard on out-of-memory failures in xlogreader.c Michael Paquier <michael@paquier.xyz> 2023-10-03 01:26 pgsql: Fail hard on out-of-memory failures in xlogreader.c Michael Paquier <michael@paquier.xyz> 2023-10-03 01:26 pgsql: Fail hard on out-of-memory failures in xlogreader.c Michael Paquier <michael@paquier.xyz> 2023-10-03 01:26 pgsql: Fail hard on out-of-memory failures in xlogreader.c Michael Paquier <michael@paquier.xyz> 2023-10-03 01:26 pgsql: Fail hard on out-of-memory failures in xlogreader.c Michael Paquier <michael@paquier.xyz> 2023-10-03 01:26 pgsql: Fail hard on out-of-memory failures in xlogreader.c Michael Paquier <michael@paquier.xyz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox