Feedback-ID: ia2694551:Fastmail
Date: Fri, 15 Sep 2023 13:41:56 +0200
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: uwe.binder@pass-consulting.com, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #18080: to_tsvector fails for long text input
Message-ID: <202309151141.pq2zpi5kxdvn@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="x3yqaoy2fs4quapw"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <18080-d5c5e58fef8c99b7@postgresql.org>
Archived-At: 
 <https://www.postgresql.org/message-id/202309151141.pq2zpi5kxdvn%40alvherre.pgsql>
Precedence: bulk


--x3yqaoy2fs4quapw
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

On 2023-Sep-04, PG Bug reporting form wrote:

> SELECT to_tsvector('english'::regconfig, (REPEAT('<Long123456789/>'::text,
> 20000000)));
> results in
> ERROR:  invalid memory alloc request size 2133333320

This is because to_tsvector_byid does this:

    prs.lenwords = VARSIZE_ANY_EXHDR(in) / 6;    /* just estimation of word's
                                                 * number */
    if (prs.lenwords < 2)
        prs.lenwords = 2;
    prs.curwords = 0;
    prs.pos = 0;
    prs.words = (ParsedWord *) palloc(sizeof(ParsedWord) * prs.lenwords);

where sizeof(ParsedWord) is 40 (in my laptop).  So this tries to
allocate more memory than palloc() is willing to give it.  The attached
patch fixes just the query you supplied and nothing else.

I wonder if we want to support this kind of thing; I suspect we don't.
Other parts of text-search would fail in the same way and would also
need to receive similar fixes.  However, the real problem comes when we
try to store such huge tsvectors, because that means we end up with
"huge" tuples on disk that need I/O support.  Eventually AFAIR you run
into the size limit in the FE/BE protocol and all crashes and burns
because that one cannot be changed without bumping the version.

So I don't think this patch actually does you any good.

-- 
Álvaro Herrera        Breisgau, Deutschland  —  https://www.EnterpriseDB.com/

--x3yqaoy2fs4quapw
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment; filename="huge_tsvector.patch"

diff --git a/src/backend/tsearch/to_tsany.c b/src/backend/tsearch/to_tsany.c
index 3b6d41f9e8..d0f49a357f 100644
--- a/src/backend/tsearch/to_tsany.c
+++ b/src/backend/tsearch/to_tsany.c
@@ -254,7 +254,8 @@ to_tsvector_byid(PG_FUNCTION_ARGS)
 		prs.lenwords = 2;
 	prs.curwords = 0;
 	prs.pos = 0;
-	prs.words = (ParsedWord *) palloc(sizeof(ParsedWord) * prs.lenwords);
+	prs.words = (ParsedWord *) palloc_extended(sizeof(ParsedWord) * prs.lenwords,
+											   MCXT_ALLOC_HUGE);
 
 	parsetext(cfgId, &prs, VARDATA_ANY(in), VARSIZE_ANY_EXHDR(in));
 
diff --git a/src/backend/tsearch/wparser_def.c b/src/backend/tsearch/wparser_def.c
index fb80fdd63f..70ad1f8680 100644
--- a/src/backend/tsearch/wparser_def.c
+++ b/src/backend/tsearch/wparser_def.c
@@ -307,12 +307,14 @@ TParserInit(char *str, int len)
 			 * char2wchar doesn't work for C-locale and sizeof(pg_wchar) could
 			 * be different from sizeof(wchar_t)
 			 */
-			prs->pgwstr = (pg_wchar *) palloc(sizeof(pg_wchar) * (prs->lenstr + 1));
+			prs->pgwstr = (pg_wchar *) palloc_extended(sizeof(pg_wchar) * (prs->lenstr + 1),
+													   MCXT_ALLOC_HUGE);
 			pg_mb2wchar_with_len(prs->str, prs->pgwstr, prs->lenstr);
 		}
 		else
 		{
-			prs->wstr = (wchar_t *) palloc(sizeof(wchar_t) * (prs->lenstr + 1));
+			prs->wstr = (wchar_t *) palloc_extended(sizeof(wchar_t) * (prs->lenstr + 1),
+													MCXT_ALLOC_HUGE);
 			char2wchar(prs->wstr, prs->lenstr + 1, prs->str, prs->lenstr,
 					   mylocale);
 		}

--x3yqaoy2fs4quapw--