From: Tom Lane <tgl@sss.pgh.pa.us>
To: Pavel Stehule <pavel.stehule@gmail.com>
cc: Shaozhong SHI <shishaozhong@gmail.com>,
        pgsql-sql <pgsql-sql@lists.postgresql.org>
Subject: Re: Memory allocation error
In-reply-to: 
 <CAFj8pRBDkNo1viX2DYdLQLkO+Qf7__xZVNRYtzHQ_RvumgvLpQ@mail.gmail.com>
References: 
 <CA+i5JwYtVS9z2E71PcNKAVPbOn4R2wuj-LqbJsYr_XOz73q7dQ@mail.gmail.com>
 <CAFj8pRCr=Qvsys2FKYscjWw5VR7wOYMFsHRoUwLsTZjgRV5yjQ@mail.gmail.com>
 <CA+i5JwYy-=tbN8P59mxd3rEE6aWo9dy=5PrYM3Fq1f67eA1ytw@mail.gmail.com>
 <CAFj8pRBDkNo1viX2DYdLQLkO+Qf7__xZVNRYtzHQ_RvumgvLpQ@mail.gmail.com>
Comments: In-reply-to Pavel Stehule <pavel.stehule@gmail.com>
	message dated "Fri, 14 Jul 2023 09:49:51 +0200"
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0"
Content-ID: <3288455.1689333136.0@sss.pgh.pa.us>
Content-Transfer-Encoding: 8bit
Date: Fri, 14 Jul 2023 07:13:00 -0400
Message-ID: <3288506.1689333180@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/3288506.1689333180%40sss.pgh.pa.us>
Precedence: bulk

------- =_aaaaaaaaaa0
Content-Type: text/plain; charset="UTF-8"
Content-ID: <3288455.1689333136.1@sss.pgh.pa.us>
Content-Transfer-Encoding: 8bit

Pavel Stehule <pavel.stehule@gmail.com> writes:
> pá 14. 7. 2023 v 9:36 odesílatel Shaozhong SHI <shishaozhong@gmail.com>
> napsal:
>> On Fri, 14 Jul 2023 at 08:14, Pavel Stehule <pavel.stehule@gmail.com>
>> wrote:
>>> It depends what you do. Postgres doesn't allow to allocate bigger blocks
>>> than 1GB. Maybe you create too big string or too big value of some other
>>> type. But it can be signal of some cache bloating.

> The best way - do it all in one recursive query without any recursive
> function.
> This issue you can fix only by rewriting your code.

Yeah, but nonetheless this error message is pretty user-unfriendly.

The given example is too incomplete to run as-is, but I guessed that
maybe the array_agg() was accumulating too many values, and sure
enough it's possible to reproduce:

regression=# select array_agg(x::text) from generate_series(1,100000000) x;
ERROR:  invalid memory alloc request size 1073741824

We can do better than that.  The attached patch causes the error to be

regression=# select array_agg(x::text) from generate_series(1,100000000) x;
ERROR:  array size exceeds the maximum allowed (1073741823)

I'm not wedded to that wording, but it's an existing translatable string
that at least points you in the direction of "my array is too big".
(This is also what you get if the eventual array construction overruns
the 1G limit, cf construct_md_array().)

			regards, tom lane


------- =_aaaaaaaaaa0
Content-Type: text/x-diff; name="better-error-for-array_agg-overflow.patch";
	charset="us-ascii"
Content-ID: <3288455.1689333136.2@sss.pgh.pa.us>
Content-Description: better-error-for-array_agg-overflow.patch
Content-Transfer-Encoding: quoted-printable

diff --git a/src/backend/utils/adt/arrayfuncs.c b/src/backend/utils/adt/ar=
rayfuncs.c
index 4359dbd83d..7828a6264b 100644
--- a/src/backend/utils/adt/arrayfuncs.c
+++ b/src/backend/utils/adt/arrayfuncs.c
@@ -5317,6 +5317,12 @@ accumArrayResult(ArrayBuildState *astate,
 	if (astate->nelems >=3D astate->alen)
 	{
 		astate->alen *=3D 2;
+		/* give an array-related error if we go past MaxAllocSize */
+		if (!AllocSizeIsValid(astate->alen * sizeof(Datum)))
+			ereport(ERROR,
+					(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+					 errmsg("array size exceeds the maximum allowed (%d)",
+							(int) MaxAllocSize)));
 		astate->dvalues =3D (Datum *)
 			repalloc(astate->dvalues, astate->alen * sizeof(Datum));
 		astate->dnulls =3D (bool *)

------- =_aaaaaaaaaa0--