pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements 7+ messages / 1 participants [nested] [flat]
* pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements @ 2022-08-02 22:05 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-02 22:05 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Be more wary about 32-bit integer overflow in pg_stat_statements. We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so #2 is a reasonable defense, and #3 could explain the reported performance difficulties. (See also commit 8bbe4cbd9, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/c67c2e2a29392b85ba7c728d3ceed986808eeec3 Modified Files -------------- contrib/pg_stat_statements/pg_stat_statements.c | 26 +++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements @ 2022-08-02 22:05 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-02 22:05 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Be more wary about 32-bit integer overflow in pg_stat_statements. We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so #2 is a reasonable defense, and #3 could explain the reported performance difficulties. (See also commit 8bbe4cbd9, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/82ebc70d1c7fd9b301e15cec658696d28df01835 Modified Files -------------- contrib/pg_stat_statements/pg_stat_statements.c | 26 +++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements @ 2022-08-02 22:05 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-02 22:05 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Be more wary about 32-bit integer overflow in pg_stat_statements. We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so #2 is a reasonable defense, and #3 could explain the reported performance difficulties. (See also commit 8bbe4cbd9, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/17fd203b414e9a1d649fb22ab11afd8355947476 Modified Files -------------- contrib/pg_stat_statements/pg_stat_statements.c | 26 +++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements @ 2022-08-02 22:05 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-02 22:05 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Be more wary about 32-bit integer overflow in pg_stat_statements. We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so #2 is a reasonable defense, and #3 could explain the reported performance difficulties. (See also commit 8bbe4cbd9, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/6b67db10c366ee825345ef81dcca57d29ad4c7f1 Modified Files -------------- contrib/pg_stat_statements/pg_stat_statements.c | 26 +++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements @ 2022-08-02 22:05 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-02 22:05 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Be more wary about 32-bit integer overflow in pg_stat_statements. We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so #2 is a reasonable defense, and #3 could explain the reported performance difficulties. (See also commit 8bbe4cbd9, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/6608a43056365dc866fc8bd2c9f323aea4725210 Modified Files -------------- contrib/pg_stat_statements/pg_stat_statements.c | 26 +++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements @ 2022-08-02 22:05 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-02 22:05 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Be more wary about 32-bit integer overflow in pg_stat_statements. We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so #2 is a reasonable defense, and #3 could explain the reported performance difficulties. (See also commit 8bbe4cbd9, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com Branch ------ REL_11_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/06f6a07ba465a6c2731697a7548e7be363cf4c57 Modified Files -------------- contrib/pg_stat_statements/pg_stat_statements.c | 26 +++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements @ 2022-08-02 22:05 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-02 22:05 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Be more wary about 32-bit integer overflow in pg_stat_statements. We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so #2 is a reasonable defense, and #3 could explain the reported performance difficulties. (See also commit 8bbe4cbd9, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com Branch ------ REL_10_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/dd414bf4e047e55028db28172e8184fcd2ee1201 Modified Files -------------- contrib/pg_stat_statements/pg_stat_statements.c | 26 +++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2022-08-02 22:05 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-08-02 22:05 pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements Tom Lane <tgl@sss.pgh.pa.us> 2022-08-02 22:05 pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements Tom Lane <tgl@sss.pgh.pa.us> 2022-08-02 22:05 pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements Tom Lane <tgl@sss.pgh.pa.us> 2022-08-02 22:05 pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements Tom Lane <tgl@sss.pgh.pa.us> 2022-08-02 22:05 pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements Tom Lane <tgl@sss.pgh.pa.us> 2022-08-02 22:05 pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements Tom Lane <tgl@sss.pgh.pa.us> 2022-08-02 22:05 pgsql: Be more wary about 32-bit integer overflow in pg_stat_statements Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox