public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tom Lane <[email protected]>
To: Pavel Stehule <[email protected]>
Cc: Shaozhong SHI <[email protected]>
Cc: pgsql-sql <[email protected]>
Subject: Re: Memory allocation error
Date: Fri, 14 Jul 2023 07:13:00 -0400
Message-ID: <[email protected]> (raw)
In-Reply-To: <CAFj8pRBDkNo1viX2DYdLQLkO+Qf7__xZVNRYtzHQ_RvumgvLpQ@mail.gmail.com>
References: <CA+i5JwYtVS9z2E71PcNKAVPbOn4R2wuj-LqbJsYr_XOz73q7dQ@mail.gmail.com>
<CAFj8pRCr=Qvsys2FKYscjWw5VR7wOYMFsHRoUwLsTZjgRV5yjQ@mail.gmail.com>
<CA+i5JwYy-=tbN8P59mxd3rEE6aWo9dy=5PrYM3Fq1f67eA1ytw@mail.gmail.com>
<CAFj8pRBDkNo1viX2DYdLQLkO+Qf7__xZVNRYtzHQ_RvumgvLpQ@mail.gmail.com>
Pavel Stehule <[email protected]> writes:
> pá 14. 7. 2023 v 9:36 odesílatel Shaozhong SHI <[email protected]>
> napsal:
>> On Fri, 14 Jul 2023 at 08:14, Pavel Stehule <[email protected]>
>> wrote:
>>> It depends what you do. Postgres doesn't allow to allocate bigger blocks
>>> than 1GB. Maybe you create too big string or too big value of some other
>>> type. But it can be signal of some cache bloating.
> The best way - do it all in one recursive query without any recursive
> function.
> This issue you can fix only by rewriting your code.
Yeah, but nonetheless this error message is pretty user-unfriendly.
The given example is too incomplete to run as-is, but I guessed that
maybe the array_agg() was accumulating too many values, and sure
enough it's possible to reproduce:
regression=# select array_agg(x::text) from generate_series(1,100000000) x;
ERROR: invalid memory alloc request size 1073741824
We can do better than that. The attached patch causes the error to be
regression=# select array_agg(x::text) from generate_series(1,100000000) x;
ERROR: array size exceeds the maximum allowed (1073741823)
I'm not wedded to that wording, but it's an existing translatable string
that at least points you in the direction of "my array is too big".
(This is also what you get if the eventual array construction overruns
the 1G limit, cf construct_md_array().)
regards, tom lane
Attachments:
[text/x-diff] better-error-for-array_agg-overflow.patch (732B, 2-better-error-for-array_agg-overflow.patch)
download | inline diff:
diff --git a/src/backend/utils/adt/arrayfuncs.c b/src/backend/utils/adt/arrayfuncs.c
index 4359dbd83d..7828a6264b 100644
--- a/src/backend/utils/adt/arrayfuncs.c
+++ b/src/backend/utils/adt/arrayfuncs.c
@@ -5317,6 +5317,12 @@ accumArrayResult(ArrayBuildState *astate,
if (astate->nelems >= astate->alen)
{
astate->alen *= 2;
+ /* give an array-related error if we go past MaxAllocSize */
+ if (!AllocSizeIsValid(astate->alen * sizeof(Datum)))
+ ereport(ERROR,
+ (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ errmsg("array size exceeds the maximum allowed (%d)",
+ (int) MaxAllocSize)));
astate->dvalues = (Datum *)
repalloc(astate->dvalues, astate->alen * sizeof(Datum));
astate->dnulls = (bool *)
view thread (7+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: Re: Memory allocation error
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox