pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Reduce memory block size for decoded tuple storage to 8kB. 7+ messages / 1 participants [nested] [flat]
* pgsql: Reduce memory block size for decoded tuple storage to 8kB. @ 2024-10-16 19:09 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-10-16 19:09 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Reduce memory block size for decoded tuple storage to 8kB. Commit a4ccc1cef introduced the Generation Context and modified the logical decoding process to use a Generation Context with a fixed block size of 8MB for storing tuple data decoded during logical decoding (i.e., rb->tup_context). Several reports have indicated that the logical decoding process can be terminated due to out-of-memory (OOM) situations caused by excessive memory usage in rb->tup_context. This issue can occur when decoding a workload involving several concurrent transactions, including a long-running transaction that modifies tuples. By design, the Generation Context does not free a memory block until all chunks within that block are released. Consequently, if tuples modified by the long-running transaction are stored across multiple memory blocks, these blocks remain allocated until the long-running transaction completes, leading to substantial memory fragmentation. The memory usage during logical decoding, tracked by rb->size, does not account for memory fragmentation, resulting in potentially much higher memory consumption than the value of the logical_decoding_work_mem parameter. Various improvement strategies were discussed in the relevant thread. This change reduces the block size of the Generation Context used in rb->tup_context from 8MB to 8kB. This modification significantly decreases the likelihood of substantial memory fragmentation occurring and is relatively straightforward to backport. Performance testing across multiple platforms has confirmed that this change will not introduce any performance degradation that would impact actual operation. Backport to all supported branches. Reported-by: Alex Richman, Michael Guissine, Avi Weinberg Reviewed-by: Amit Kapila, Fujii Masao, David Rowley Tested-by: Hayato Kuroda, Shlok Kyal Discussion: https://postgr.es/m/CAD21AoBTY1LATZUmvSXEssvq07qDZufV4AF-OHh9VD2pC0VY2A%40mail.gmail.com Backpatch-through: 12 Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/53fa68b3bc3a3668524dda86904fd5ddfcf3cb9d Modified Files -------------- src/backend/replication/logical/reorderbuffer.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Reduce memory block size for decoded tuple storage to 8kB. @ 2024-10-16 19:09 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-10-16 19:09 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Reduce memory block size for decoded tuple storage to 8kB. Commit a4ccc1cef introduced the Generation Context and modified the logical decoding process to use a Generation Context with a fixed block size of 8MB for storing tuple data decoded during logical decoding (i.e., rb->tup_context). Several reports have indicated that the logical decoding process can be terminated due to out-of-memory (OOM) situations caused by excessive memory usage in rb->tup_context. This issue can occur when decoding a workload involving several concurrent transactions, including a long-running transaction that modifies tuples. By design, the Generation Context does not free a memory block until all chunks within that block are released. Consequently, if tuples modified by the long-running transaction are stored across multiple memory blocks, these blocks remain allocated until the long-running transaction completes, leading to substantial memory fragmentation. The memory usage during logical decoding, tracked by rb->size, does not account for memory fragmentation, resulting in potentially much higher memory consumption than the value of the logical_decoding_work_mem parameter. Various improvement strategies were discussed in the relevant thread. This change reduces the block size of the Generation Context used in rb->tup_context from 8MB to 8kB. This modification significantly decreases the likelihood of substantial memory fragmentation occurring and is relatively straightforward to backport. Performance testing across multiple platforms has confirmed that this change will not introduce any performance degradation that would impact actual operation. Backport to all supported branches. Reported-by: Alex Richman, Michael Guissine, Avi Weinberg Reviewed-by: Amit Kapila, Fujii Masao, David Rowley Tested-by: Hayato Kuroda, Shlok Kyal Discussion: https://postgr.es/m/CAD21AoBTY1LATZUmvSXEssvq07qDZufV4AF-OHh9VD2pC0VY2A%40mail.gmail.com Backpatch-through: 12 Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/cb988b04d0637f5d55bfb5a01b23dd4c39beccfa Modified Files -------------- src/backend/replication/logical/reorderbuffer.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Reduce memory block size for decoded tuple storage to 8kB. @ 2024-10-16 19:10 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-10-16 19:10 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Reduce memory block size for decoded tuple storage to 8kB. Commit a4ccc1cef introduced the Generation Context and modified the logical decoding process to use a Generation Context with a fixed block size of 8MB for storing tuple data decoded during logical decoding (i.e., rb->tup_context). Several reports have indicated that the logical decoding process can be terminated due to out-of-memory (OOM) situations caused by excessive memory usage in rb->tup_context. This issue can occur when decoding a workload involving several concurrent transactions, including a long-running transaction that modifies tuples. By design, the Generation Context does not free a memory block until all chunks within that block are released. Consequently, if tuples modified by the long-running transaction are stored across multiple memory blocks, these blocks remain allocated until the long-running transaction completes, leading to substantial memory fragmentation. The memory usage during logical decoding, tracked by rb->size, does not account for memory fragmentation, resulting in potentially much higher memory consumption than the value of the logical_decoding_work_mem parameter. Various improvement strategies were discussed in the relevant thread. This change reduces the block size of the Generation Context used in rb->tup_context from 8MB to 8kB. This modification significantly decreases the likelihood of substantial memory fragmentation occurring and is relatively straightforward to backport. Performance testing across multiple platforms has confirmed that this change will not introduce any performance degradation that would impact actual operation. Backport to all supported branches. Reported-by: Alex Richman, Michael Guissine, Avi Weinberg Reviewed-by: Amit Kapila, Fujii Masao, David Rowley Tested-by: Hayato Kuroda, Shlok Kyal Discussion: https://postgr.es/m/CAD21AoBTY1LATZUmvSXEssvq07qDZufV4AF-OHh9VD2pC0VY2A%40mail.gmail.com Backpatch-through: 12 Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/5c1ed0a516813b588e9b2ddf042f1eda0bbabec6 Modified Files -------------- src/backend/replication/logical/reorderbuffer.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Reduce memory block size for decoded tuple storage to 8kB. @ 2024-10-16 19:10 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-10-16 19:10 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Reduce memory block size for decoded tuple storage to 8kB. Commit a4ccc1cef introduced the Generation Context and modified the logical decoding process to use a Generation Context with a fixed block size of 8MB for storing tuple data decoded during logical decoding (i.e., rb->tup_context). Several reports have indicated that the logical decoding process can be terminated due to out-of-memory (OOM) situations caused by excessive memory usage in rb->tup_context. This issue can occur when decoding a workload involving several concurrent transactions, including a long-running transaction that modifies tuples. By design, the Generation Context does not free a memory block until all chunks within that block are released. Consequently, if tuples modified by the long-running transaction are stored across multiple memory blocks, these blocks remain allocated until the long-running transaction completes, leading to substantial memory fragmentation. The memory usage during logical decoding, tracked by rb->size, does not account for memory fragmentation, resulting in potentially much higher memory consumption than the value of the logical_decoding_work_mem parameter. Various improvement strategies were discussed in the relevant thread. This change reduces the block size of the Generation Context used in rb->tup_context from 8MB to 8kB. This modification significantly decreases the likelihood of substantial memory fragmentation occurring and is relatively straightforward to backport. Performance testing across multiple platforms has confirmed that this change will not introduce any performance degradation that would impact actual operation. Backport to all supported branches. Reported-by: Alex Richman, Michael Guissine, Avi Weinberg Reviewed-by: Amit Kapila, Fujii Masao, David Rowley Tested-by: Hayato Kuroda, Shlok Kyal Discussion: https://postgr.es/m/CAD21AoBTY1LATZUmvSXEssvq07qDZufV4AF-OHh9VD2pC0VY2A%40mail.gmail.com Backpatch-through: 12 Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/4a675f31868a6a7f0ef436ebfbaedddc74d1c2a1 Modified Files -------------- src/backend/replication/logical/reorderbuffer.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Reduce memory block size for decoded tuple storage to 8kB. @ 2024-10-16 19:10 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-10-16 19:10 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Reduce memory block size for decoded tuple storage to 8kB. Commit a4ccc1cef introduced the Generation Context and modified the logical decoding process to use a Generation Context with a fixed block size of 8MB for storing tuple data decoded during logical decoding (i.e., rb->tup_context). Several reports have indicated that the logical decoding process can be terminated due to out-of-memory (OOM) situations caused by excessive memory usage in rb->tup_context. This issue can occur when decoding a workload involving several concurrent transactions, including a long-running transaction that modifies tuples. By design, the Generation Context does not free a memory block until all chunks within that block are released. Consequently, if tuples modified by the long-running transaction are stored across multiple memory blocks, these blocks remain allocated until the long-running transaction completes, leading to substantial memory fragmentation. The memory usage during logical decoding, tracked by rb->size, does not account for memory fragmentation, resulting in potentially much higher memory consumption than the value of the logical_decoding_work_mem parameter. Various improvement strategies were discussed in the relevant thread. This change reduces the block size of the Generation Context used in rb->tup_context from 8MB to 8kB. This modification significantly decreases the likelihood of substantial memory fragmentation occurring and is relatively straightforward to backport. Performance testing across multiple platforms has confirmed that this change will not introduce any performance degradation that would impact actual operation. Backport to all supported branches. Reported-by: Alex Richman, Michael Guissine, Avi Weinberg Reviewed-by: Amit Kapila, Fujii Masao, David Rowley Tested-by: Hayato Kuroda, Shlok Kyal Discussion: https://postgr.es/m/CAD21AoBTY1LATZUmvSXEssvq07qDZufV4AF-OHh9VD2pC0VY2A%40mail.gmail.com Backpatch-through: 12 Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/05e982cdc4dfcc340f2db9bfd9dbd92844cf9b01 Modified Files -------------- src/backend/replication/logical/reorderbuffer.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Reduce memory block size for decoded tuple storage to 8kB. @ 2024-10-16 19:10 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-10-16 19:10 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Reduce memory block size for decoded tuple storage to 8kB. Commit a4ccc1cef introduced the Generation Context and modified the logical decoding process to use a Generation Context with a fixed block size of 8MB for storing tuple data decoded during logical decoding (i.e., rb->tup_context). Several reports have indicated that the logical decoding process can be terminated due to out-of-memory (OOM) situations caused by excessive memory usage in rb->tup_context. This issue can occur when decoding a workload involving several concurrent transactions, including a long-running transaction that modifies tuples. By design, the Generation Context does not free a memory block until all chunks within that block are released. Consequently, if tuples modified by the long-running transaction are stored across multiple memory blocks, these blocks remain allocated until the long-running transaction completes, leading to substantial memory fragmentation. The memory usage during logical decoding, tracked by rb->size, does not account for memory fragmentation, resulting in potentially much higher memory consumption than the value of the logical_decoding_work_mem parameter. Various improvement strategies were discussed in the relevant thread. This change reduces the block size of the Generation Context used in rb->tup_context from 8MB to 8kB. This modification significantly decreases the likelihood of substantial memory fragmentation occurring and is relatively straightforward to backport. Performance testing across multiple platforms has confirmed that this change will not introduce any performance degradation that would impact actual operation. Backport to all supported branches. Reported-by: Alex Richman, Michael Guissine, Avi Weinberg Reviewed-by: Amit Kapila, Fujii Masao, David Rowley Tested-by: Hayato Kuroda, Shlok Kyal Discussion: https://postgr.es/m/CAD21AoBTY1LATZUmvSXEssvq07qDZufV4AF-OHh9VD2pC0VY2A%40mail.gmail.com Backpatch-through: 12 Branch ------ REL_17_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/eef9cc4dc28fbe17d621eeb773c56e9ac3f065ff Modified Files -------------- src/backend/replication/logical/reorderbuffer.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Reduce memory block size for decoded tuple storage to 8kB. @ 2024-10-16 19:10 Masahiko Sawada <msawada@postgresql.org> 0 siblings, 0 replies; 7+ messages in thread From: Masahiko Sawada @ 2024-10-16 19:10 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Reduce memory block size for decoded tuple storage to 8kB. Commit a4ccc1cef introduced the Generation Context and modified the logical decoding process to use a Generation Context with a fixed block size of 8MB for storing tuple data decoded during logical decoding (i.e., rb->tup_context). Several reports have indicated that the logical decoding process can be terminated due to out-of-memory (OOM) situations caused by excessive memory usage in rb->tup_context. This issue can occur when decoding a workload involving several concurrent transactions, including a long-running transaction that modifies tuples. By design, the Generation Context does not free a memory block until all chunks within that block are released. Consequently, if tuples modified by the long-running transaction are stored across multiple memory blocks, these blocks remain allocated until the long-running transaction completes, leading to substantial memory fragmentation. The memory usage during logical decoding, tracked by rb->size, does not account for memory fragmentation, resulting in potentially much higher memory consumption than the value of the logical_decoding_work_mem parameter. Various improvement strategies were discussed in the relevant thread. This change reduces the block size of the Generation Context used in rb->tup_context from 8MB to 8kB. This modification significantly decreases the likelihood of substantial memory fragmentation occurring and is relatively straightforward to backport. Performance testing across multiple platforms has confirmed that this change will not introduce any performance degradation that would impact actual operation. Backport to all supported branches. Reported-by: Alex Richman, Michael Guissine, Avi Weinberg Reviewed-by: Amit Kapila, Fujii Masao, David Rowley Tested-by: Hayato Kuroda, Shlok Kyal Discussion: https://postgr.es/m/CAD21AoBTY1LATZUmvSXEssvq07qDZufV4AF-OHh9VD2pC0VY2A%40mail.gmail.com Backpatch-through: 12 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/1b9b6cc3456be0f6ab929107293b31c333270bc1 Modified Files -------------- src/backend/replication/logical/reorderbuffer.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2024-10-16 19:10 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-10-16 19:09 pgsql: Reduce memory block size for decoded tuple storage to 8kB. Masahiko Sawada <msawada@postgresql.org> 2024-10-16 19:09 pgsql: Reduce memory block size for decoded tuple storage to 8kB. Masahiko Sawada <msawada@postgresql.org> 2024-10-16 19:10 pgsql: Reduce memory block size for decoded tuple storage to 8kB. Masahiko Sawada <msawada@postgresql.org> 2024-10-16 19:10 pgsql: Reduce memory block size for decoded tuple storage to 8kB. Masahiko Sawada <msawada@postgresql.org> 2024-10-16 19:10 pgsql: Reduce memory block size for decoded tuple storage to 8kB. Masahiko Sawada <msawada@postgresql.org> 2024-10-16 19:10 pgsql: Reduce memory block size for decoded tuple storage to 8kB. Masahiko Sawada <msawada@postgresql.org> 2024-10-16 19:10 pgsql: Reduce memory block size for decoded tuple storage to 8kB. Masahiko Sawada <msawada@postgresql.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox