From: Tom Lane <tgl@sss.pgh.pa.us>
To: Stanislav Kozlovski <Stanislav_Kozlovski@outlook.com>
cc: "pgsql-general@lists.postgresql.org" <pgsql-general@lists.postgresql.org>
Subject: Re: tsvector limitations - why and how
In-reply-to: <DU0PR10MB60604534BEBB91C67FD743A78A942@DU0PR10MB6060.EURPRD10.PROD.OUTLOOK.COM>
References: <DU0PR10MB60604534BEBB91C67FD743A78A942@DU0PR10MB6060.EURPRD10.PROD.OUTLOOK.COM>
Comments: In-reply-to Stanislav Kozlovski <Stanislav_Kozlovski@outlook.com>
	message dated "Tue, 27 Aug 2024 20:38:03 -0000"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3150735.1724797492.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Tue, 27 Aug 2024 18:24:52 -0400
Message-ID: <3150736.1724797492@sss.pgh.pa.us>
Archived-At: <https://www.postgresql.org/message-id/3150736.1724797492%40sss.pgh.pa.us>
Precedence: bulk

Stanislav Kozlovski <Stanislav_Kozlovski@outlook.com> writes:
> I was aware of the limitations of FTS<https://www.postgresql.org/docs/17=
/textsearch-limitations.html> and tried to ensure I didn't hit any - but w=
hat I missed was that the maximum allowed lexeme position was 16383 and ev=
erything above silently gets set to 16383. I was searching for a phrase (t=
wo words) at the end of the book and couldn't find it. After debugging I r=
ealized that my phrase's lexemes were being set to 16383, which was inaccu=
rate.
> ...
> The problem I had is that it breaks FOLLOWED BY queries, essentially sto=
pping you from being able to match on phrases (more than one word) on larg=
e text.

Yeah.  FOLLOWED BY didn't exist when the tsvector storage
representation was designed, so the possible inaccuracy of the
lexeme positions wasn't such a big deal.

> Why is this still the case?

Because nobody's done the significant amount of work needed to make
it better.  I think an acceptable patch would have to support both
the current tsvector representation and a "big" version that's able
to handle anything up to the 1GB varlena limit.  (If you were hoping
for documents bigger than that, you'd be needing a couple more
orders of magnitude worth of work.)  We might also find that there
are performance bottlenecks that'd have to be improved, but even just
making the code cope with two representations would be a big patch.

There has been some cursory talk about this, I think, but I don't
believe anyone's actually worked on it since the 2017 patch you
mentioned.  I'm not sure if that patch is worth using as the basis
for a fresh try: it looks like it had some performance issues, and
AFAICS it didn't really improve the lexeme-position limit.

(Wanders away wondering if the expanded-datum infrastructure could
be exploited here...)

			regards, tom lane