From: Tom Lane <tgl@sss.pgh.pa.us>
To: Alvaro Herrera <alvherre@alvh.no-ip.org>
cc: uwe.binder@pass-consulting.com, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #18080: to_tsvector fails for long text input
In-reply-to: <3300287.1694786029@sss.pgh.pa.us>
References: <202309151141.pq2zpi5kxdvn@alvherre.pgsql>
 <3300287.1694786029@sss.pgh.pa.us>
Comments: In-reply-to Tom Lane <tgl@sss.pgh.pa.us>
	message dated "Fri, 15 Sep 2023 09:53:49 -0400"
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0"
Content-ID: <1056570.1695404930.0@sss.pgh.pa.us>
Date: Fri, 22 Sep 2023 13:48:59 -0400
Message-ID: <1056584.1695404939@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/1056584.1695404939%40sss.pgh.pa.us>
Precedence: bulk

------- =_aaaaaaaaaa0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <1056570.1695404930.1@sss.pgh.pa.us>

I wrote:
> Yeah.  My thought about blocking the error had been to limit
> prs.lenwords to MaxAllocSize/sizeof(ParsedWord) in this code.

Concretely, as attached.  This allows the given test case to
complete, since it doesn't actually create very many distinct
words.  In other cases we could expect to fail when the array
has to get enlarged, but that's just a normal implementation
limitation.

I looked for other places that might initialize lenwords
to not-sane values, and didn't find any.

BTW, the field order in ParsedWord is such that there's a fair
amount of wasted pad space on 64-bit builds.  I doubt we can
get away with rearranging it in released branches; but maybe
it's worth doing something about that in HEAD, to push out
the point at which you hit the 1Gb limit.

			regards, tom lane


------- =_aaaaaaaaaa0
Content-Type: text/x-diff; name="bound-lenwords-in-to_tsvector_byid.patch";
	charset="us-ascii"
Content-ID: <1056570.1695404930.2@sss.pgh.pa.us>
Content-Description: bound-lenwords-in-to_tsvector_byid.patch

diff --git a/src/backend/tsearch/to_tsany.c b/src/backend/tsearch/to_tsany.c
index 3b6d41f9e8..fe39d6c4b9 100644
--- a/src/backend/tsearch/to_tsany.c
+++ b/src/backend/tsearch/to_tsany.c
@@ -252,6 +252,8 @@ to_tsvector_byid(PG_FUNCTION_ARGS)
 												 * number */
 	if (prs.lenwords < 2)
 		prs.lenwords = 2;
+	else if (prs.lenwords > MaxAllocSize / sizeof(ParsedWord))
+		prs.lenwords = MaxAllocSize / sizeof(ParsedWord);
 	prs.curwords = 0;
 	prs.pos = 0;
 	prs.words = (ParsedWord *) palloc(sizeof(ParsedWord) * prs.lenwords);

------- =_aaaaaaaaaa0--