Message-ID: <8cf296c265a367e08bf221781c4ba6c3f3726fda.camel@j-davis.com>
Subject: Re: dict_synonym.c: fix truncation of multibyte sequence
From: Jeff Davis <pgsql@j-davis.com>
To: Tristan Partin <tristan@partin.io>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Date: Fri, 05 Jun 2026 10:37:03 -0700
In-Reply-To: <DJ18MRKAVFG9.1X19THWF6N43F@partin.io>
References: <1101e1a3afbbabb503317069c40374b82e6f4cac.camel@j-davis.com>
	 <DJ18MRKAVFG9.1X19THWF6N43F@partin.io>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.52.3-0ubuntu1.1 
MIME-Version: 1.0
Archived-At: 
 <https://www.postgresql.org/message-id/8cf296c265a367e08bf221781c4ba6c3f3726fda.camel%40j-davis.com>
Precedence: bulk

On Fri, 2026-06-05 at 15:57 +0000, Tristan Partin wrote:
> > In any case, the input comes from a trusted
> > source (dictionary configuration), so it's not very serious.
>=20
> The fix looks and sounds good. Do we have any way to test this, so it
> doesn't regress in the future?

  -- =C8=BA is 2 bytes, '=E2=B1=A5' is 3 bytes
  $ echo "foo bar=C8=BA" > /path/to/postgres/share/tsearch_data/mbtest.syn

  CREATE TEXT SEARCH DICTIONARY mb_syn (
    TEMPLATE =3D synonym,
    SYNONYMS =3D mbtest);

  SELECT ts_lexize('mb_syn', 'foo');

  =3D# SELECT ts_lexize('mb_syn', 'foo'); -- before patch
   ts_lexize=20
  -----------
   {bar}
  (1 row)

  =3D# SELECT ts_lexize('mb_syn', 'foo'); -- after patch
   ts_lexize=20
  -----------
   {bar=E2=B1=A5}
  (1 row)

It requires a specially-crafted synonym file, and I'm not sure it's
worth much effort to add a test for this specific path. If we see
similar bugs, it's more likely to be somewhere else that makes the same
faulty assumption.

If you do think we should add tests, we should probably add a set of
dictionary-related files (.syn, .dict, .ths, etc.) that contain a
variety of adversarial Unicode cases.

I'd be inclined to just commit this fix for now. It needs backpatching,
and I don't think we want to backpatch a large set of tests with it.

Regards,
	Jeff Davis