public inbox for [email protected]
help / color / mirror / Atom feedFrom: Alvaro Herrera <[email protected]>
To: Tom Lane <[email protected]>
Cc: Michael Paquier <[email protected]>
Cc: Hongxu Ma <[email protected]>
Cc: Jelte Fennema <[email protected]>
Cc: David G. Johnston <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: PSQL error: total cell count of XXX exceeded
Date: Mon, 20 Nov 2023 21:48:35 +0100
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
On 2023-Sep-12, Tom Lane wrote:
> I'm more than a bit skeptical about trying to do something about this,
> simply because this range of query result sizes is far past what is
> practical. The OP clearly hasn't tested his patch on actually
> overflowing query results, and I don't care to either.
I think we're bound to hit this limit at some point in the future, and
it seems easy enough to solve. I propose the attached, which is pretty
much what Hongxu last submitted, with some minor changes.
Having this make a difference requires some 128GB of RAM, so it's not a
piece of cake, but it's an amount that can be reasonably expected to be
physically installed in real machines nowadays.
(I first thought we could just use pg_mul_s32_overflow during
printTableInit and raise an error if that returns true, but that just
postpones the problem.)
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
Subversion to GIT: the shortest path to happiness I've ever heard of
(Alexey Klyukin)
Attachments:
[text/x-diff] v5-0001-Avoid-overflow-in-fe_utils-printTable-API.patch (3.7K, ../[email protected]/2-v5-0001-Avoid-overflow-in-fe_utils-printTable-API.patch)
download | inline diff:
From 75abe5a485532cbc03db1eade2e1a6099914f98f Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <[email protected]>
Date: Mon, 20 Nov 2023 17:35:18 +0100
Subject: [PATCH v5] Avoid overflow in fe_utils' printTable API
We were using 32-bit integer arithmetic to compute the total number of
cells, which can overflow when used with large resultsets. We still
don't allow more than 4 billion rows, but now the total number of cells
can exceed that.
Author: Hongxu Ma <[email protected]>
Discussion: https://postgr.es/m/TYBP286MB0351B057B101C90D7C1239E6B4E2A@TYBP286MB0351.JPNP286.PROD.OUTLOOK.COM
---
src/fe_utils/print.c | 22 ++++++++++++++--------
src/include/fe_utils/print.h | 2 +-
2 files changed, 15 insertions(+), 9 deletions(-)
diff --git a/src/fe_utils/print.c b/src/fe_utils/print.c
index 7af1ccb6b5..565cbc6d13 100644
--- a/src/fe_utils/print.c
+++ b/src/fe_utils/print.c
@@ -3172,6 +3172,8 @@ void
printTableInit(printTableContent *const content, const printTableOpt *opt,
const char *title, const int ncolumns, const int nrows)
{
+ int64 total_cells;
+
content->opt = opt;
content->title = title;
content->ncolumns = ncolumns;
@@ -3179,7 +3181,8 @@ printTableInit(printTableContent *const content, const printTableOpt *opt,
content->headers = pg_malloc0((ncolumns + 1) * sizeof(*content->headers));
- content->cells = pg_malloc0((ncolumns * nrows + 1) * sizeof(*content->cells));
+ total_cells = (int64) ncolumns * nrows;
+ content->cells = pg_malloc0((total_cells + 1) * sizeof(*content->cells));
content->cellmustfree = NULL;
content->footers = NULL;
@@ -3249,15 +3252,17 @@ void
printTableAddCell(printTableContent *const content, char *cell,
const bool translate, const bool mustfree)
{
+ int64 total_cells;
+
#ifndef ENABLE_NLS
(void) translate; /* unused parameter */
#endif
- if (content->cellsadded >= content->ncolumns * content->nrows)
+ total_cells = (int64) content->ncolumns * content->nrows;
+ if (content->cellsadded >= total_cells)
{
- fprintf(stderr, _("Cannot add cell to table content: "
- "total cell count of %d exceeded.\n"),
- content->ncolumns * content->nrows);
+ fprintf(stderr, _("Cannot add cell to table content: total cell count of %lld exceeded.\n"),
+ (long long int) total_cells);
exit(EXIT_FAILURE);
}
@@ -3273,7 +3278,7 @@ printTableAddCell(printTableContent *const content, char *cell,
{
if (content->cellmustfree == NULL)
content->cellmustfree =
- pg_malloc0((content->ncolumns * content->nrows + 1) * sizeof(bool));
+ pg_malloc0((total_cells + 1) * sizeof(bool));
content->cellmustfree[content->cellsadded] = true;
}
@@ -3341,9 +3346,10 @@ printTableCleanup(printTableContent *const content)
{
if (content->cellmustfree)
{
- int i;
+ int64 total_cells;
- for (i = 0; i < content->nrows * content->ncolumns; i++)
+ total_cells = (int64) content->ncolumns * content->nrows;
+ for (int64 i = 0; i < total_cells; i++)
{
if (content->cellmustfree[i])
free(unconstify(char *, content->cells[i]));
diff --git a/src/include/fe_utils/print.h b/src/include/fe_utils/print.h
index 13ebb00362..a697d0ba8d 100644
--- a/src/include/fe_utils/print.h
+++ b/src/include/fe_utils/print.h
@@ -171,7 +171,7 @@ typedef struct printTableContent
const char **cells; /* NULL-terminated array of cell content
* strings */
const char **cell; /* Pointer to the last added cell */
- long cellsadded; /* Number of cells added this far */
+ int64 cellsadded; /* Number of cells added this far */
bool *cellmustfree; /* true for cells that need to be free()d */
printTableFooter *footers; /* Pointer to the first footer */
printTableFooter *footer; /* Pointer to the last added footer */
--
2.39.2
view thread (6+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: PSQL error: total cell count of XXX exceeded
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox