From: Tom Lane <tgl@sss.pgh.pa.us>
To: esemmano@gmail.com
cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #18479: websearch_to_tsquery inconsistent behavior for german
 when using parentheses
In-reply-to: <18479-d9b46e2fc242c33e@postgresql.org>
References: <18479-d9b46e2fc242c33e@postgresql.org>
Comments: In-reply-to PG Bug reporting form <noreply@postgresql.org>
	message dated "Fri, 24 May 2024 07:02:28 -0000"
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0"
Content-ID: <2130868.1718316178.0@sss.pgh.pa.us>
Date: Thu, 13 Jun 2024 18:04:20 -0400
Message-ID: <2130969.1718316260@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/2130969.1718316260%40sss.pgh.pa.us>
Precedence: bulk

------- =_aaaaaaaaaa0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <2130868.1718316178.1@sss.pgh.pa.us>

PG Bug reporting form <noreply@postgresql.org> writes:
> Although the docs
> https://www.postgresql.org/docs/current/textsearch-controls.html say nothing
> about websearch_to_tsquery supporting parentheses in queries, I noticed some
> inconsistent behaviour when using multiple 'or' keywords with parentheses in
> postgres 15.4

The definition of websearch_to_tsquery says pretty plainly that
"Other punctuation is ignored".  So I'd expect parens to do nothing.
That makes this problematic:

> select websearch_to_tsquery('german', 'foo or baz bar or (ding dong)');
>           websearch_to_tsquery
> -----------------------------------------
>  'foo' | 'baz' & 'bar' | 'ding' & 'dong'

> select websearch_to_tsquery('german', 'foo or (baz bar) or (ding dong)');
>               websearch_to_tsquery
> ------------------------------------------------
>  'foo' | 'baz' & 'bar' & 'or' & 'ding' & 'dong'


I found what seems to be the issue in gettoken_query_websearch: it
ignores ISOPERATOR chars (including parens) in WAITOPERAND state,
but not in WAITOPERATOR state.  That results in switching back to
WAITOPERAND state which will consume the "or" as a regular word.
So a minimal fix could look like the attached.

It's fairly confusing that this code manages to ignore not-ISOPERATOR
punctuation.  It seems like that gets eaten by gettoken_tsvector()
and then later we decide there's not really a word there.

I'm also confused how come the same thing doesn't happen in the
english tsconfig.  Not sure it's worth poking at more, though.

			regards, tom lane


------- =_aaaaaaaaaa0
Content-Type: text/x-diff; name="draft-bug18479-fix.patch";
	charset="us-ascii"
Content-ID: <2130868.1718316178.2@sss.pgh.pa.us>
Content-Description: draft-bug18479-fix.patch
Content-Transfer-Encoding: quoted-printable

diff --git a/src/backend/utils/adt/tsquery.c b/src/backend/utils/adt/tsque=
ry.c
index 690a80d774..eb08e912ea 100644
--- a/src/backend/utils/adt/tsquery.c
+++ b/src/backend/utils/adt/tsquery.c
@@ -492,6 +492,12 @@ gettoken_query_websearch(TSQueryParserState state, in=
t8 *operator,
 					*operator =3D OP_OR;
 					return PT_OPR;
 				}
+				else if (ISOPERATOR(state->buf))
+				{
+					/* ignore other operators here too */
+					state->buf++;
+					continue;
+				}
 				else if (*state->buf =3D=3D '\0')
 				{
 					return PT_END;

------- =_aaaaaaaaaa0--