Feedback-ID: idd01497b:Fastmail
Mime-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8
Date: Fri, 05 Jun 2026 20:46:00 +0000
Message-Id: <DJ1ERDAO9GGX.3TNILXYMEE8KO@partin.io>
Cc: "pgsql-hackers" <pgsql-hackers@postgresql.org>
Subject: Re: dict_synonym.c: fix truncation of multibyte sequence
To: "Jeff Davis" <pgsql@j-davis.com>
From: "Tristan Partin" <tristan@partin.io>
References: <1101e1a3afbbabb503317069c40374b82e6f4cac.camel@j-davis.com>
 <DJ18MRKAVFG9.1X19THWF6N43F@partin.io>
 <8cf296c265a367e08bf221781c4ba6c3f3726fda.camel@j-davis.com>
In-Reply-To: <8cf296c265a367e08bf221781c4ba6c3f3726fda.camel@j-davis.com>
Archived-At: 
 <https://www.postgresql.org/message-id/DJ1ERDAO9GGX.3TNILXYMEE8KO%40partin.io>
Precedence: bulk

On Fri Jun 5, 2026 at 5:37 PM UTC, Jeff Davis wrote:
> On Fri, 2026-06-05 at 15:57 +0000, Tristan Partin wrote:
>> > In any case, the input comes from a trusted
>> > source (dictionary configuration), so it's not very serious.
>>=20
>> The fix looks and sounds good. Do we have any way to test this, so it
>> doesn't regress in the future?
>
>   -- =C8=BA is 2 bytes, '=E2=B1=A5' is 3 bytes
>   $ echo "foo bar=C8=BA" > /path/to/postgres/share/tsearch_data/mbtest.sy=
n
>
>   CREATE TEXT SEARCH DICTIONARY mb_syn (
>     TEMPLATE =3D synonym,
>     SYNONYMS =3D mbtest);
>
>   SELECT ts_lexize('mb_syn', 'foo');
>
>   =3D# SELECT ts_lexize('mb_syn', 'foo'); -- before patch
>    ts_lexize=20
>   -----------
>    {bar}
>   (1 row)
>
>   =3D# SELECT ts_lexize('mb_syn', 'foo'); -- after patch
>    ts_lexize=20
>   -----------
>    {bar=E2=B1=A5}
>   (1 row)
>
> It requires a specially-crafted synonym file, and I'm not sure it's
> worth much effort to add a test for this specific path. If we see
> similar bugs, it's more likely to be somewhere else that makes the same
> faulty assumption.
>
> If you do think we should add tests, we should probably add a set of
> dictionary-related files (.syn, .dict, .ths, etc.) that contain a
> variety of adversarial Unicode cases.
>
> I'd be inclined to just commit this fix for now. It needs backpatching,
> and I don't think we want to backpatch a large set of tests with it.

I would say proceed as you see fit. I guess I am generally of the=20
opinion that additional testing is generally always better, but I don't=20
want to push for something if others don't see the same value. I'd be=20
happy to provide a patch for the test in a subsequent discussion if that=20
is a good middle ground?

--=20
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)