Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lBnGe-0006Xx-UH for pgsql-bugs@arkaria.postgresql.org; Mon, 15 Feb 2021 23:27:00 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lBnGd-0006nS-MC for pgsql-bugs@arkaria.postgresql.org; Mon, 15 Feb 2021 23:26:59 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lBnGd-0006nK-Fy for pgsql-bugs@lists.postgresql.org; Mon, 15 Feb 2021 23:26:59 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lBnGV-0002oG-SY for pgsql-bugs@lists.postgresql.org; Mon, 15 Feb 2021 23:26:58 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 11FNQmIb2288865; Mon, 15 Feb 2021 18:26:48 -0500 From: Tom Lane To: =?UTF-8?Q?Dimitri_N=C3=BCscheler?= cc: pgsql-bugs@lists.postgresql.org, Pavel Borisov Subject: Re: BUG #16865: Regression: GIN Negated prefix search returns results that contain the search term In-reply-to: References: <16865-4ffdc3e682e6d75b@postgresql.org> Comments: In-reply-to =?UTF-8?Q?Dimitri_N=C3=BCscheler?= message dated "Sun, 14 Feb 2021 23:05:41 +0100" MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0" Content-ID: <2288829.1613431562.0@sss.pgh.pa.us> Date: Mon, 15 Feb 2021 18:26:48 -0500 Message-ID: <2288864.1613431608@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk ------- =_aaaaaaaaaa0 Content-Type: text/plain; charset="us-ascii" Content-ID: <2288829.1613431562.1@sss.pgh.pa.us> =?UTF-8?Q?Dimitri_N=C3=BCscheler?= writes: > Meanwhile I managed to anonymize the data. I put it in this archive > https://www.violetsky.ch/postgres-issue-anonymized.tar.gz Thanks. I've reproduced the issue and it boils down to doing the wrong thing when there are GIN_MAYBE values in the input data for checkcondition_gin, specifically when the bitmap holding the original GIN index results has become lossy. We correctly get a TS_MAYBE result out of the calculation in TS_execute_recurse, but then TS_execute figures it can throw that detail away and just return TS_YES. I think that this problem existed before 2f2007fbb, but we were masking it by always forcing rechecks. Anyway, this seems to put the final nail in the coffin of the idea that it's sufficient for TS_execute to return a boolean. At least the tsginidx.c callers really need to see the 3-way result. Hence I propose the attached patch. It fixes the given test case, but I wonder if you can try it and see if you see any remaining problems. I wasted quite a bit of time trying to devise a test case that is short enough to be reasonable to include in our regression tests, without success ... you need quite a bit of data to make the GIN bitmap become lossy. So no test case here, but I'm not super comfortable with that. regards, tom lane ------- =_aaaaaaaaaa0 Content-Type: text/x-diff; name="fix-lossy-gin-index-behavior.patch"; charset="us-ascii" Content-ID: <2288829.1613431562.2@sss.pgh.pa.us> Content-Description: fix-lossy-gin-index-behavior.patch Content-Transfer-Encoding: quoted-printable diff --git a/src/backend/utils/adt/tsginidx.c b/src/backend/utils/adt/tsgi= nidx.c index 6c913baaba..8eed376708 100644 --- a/src/backend/utils/adt/tsginidx.c +++ b/src/backend/utils/adt/tsginidx.c @@ -246,10 +246,22 @@ gin_tsquery_consistent(PG_FUNCTION_ARGS) gcv.map_item_operand =3D (int *) (extra_data[0]); gcv.need_recheck =3D recheck; = - res =3D TS_execute(GETQUERY(query), - &gcv, - TS_EXEC_PHRASE_NO_POS, - checkcondition_gin); + switch (TS_execute_ternary(GETQUERY(query), + &gcv, + TS_EXEC_PHRASE_NO_POS, + checkcondition_gin)) + { + case TS_NO: + res =3D false; + break; + case TS_YES: + res =3D true; + break; + case TS_MAYBE: + res =3D true; + *recheck =3D true; + break; + } } = PG_RETURN_BOOL(res); @@ -284,11 +296,12 @@ gin_tsquery_triconsistent(PG_FUNCTION_ARGS) gcv.map_item_operand =3D (int *) (extra_data[0]); gcv.need_recheck =3D &recheck; = - if (TS_execute(GETQUERY(query), - &gcv, - TS_EXEC_PHRASE_NO_POS, - checkcondition_gin)) - res =3D recheck ? GIN_MAYBE : GIN_TRUE; + res =3D TS_execute_ternary(GETQUERY(query), + &gcv, + TS_EXEC_PHRASE_NO_POS, + checkcondition_gin); + if (res =3D=3D GIN_TRUE && recheck) + res =3D GIN_MAYBE; } = PG_RETURN_GIN_TERNARY_VALUE(res); diff --git a/src/backend/utils/adt/tsvector_op.c b/src/backend/utils/adt/t= svector_op.c index 2939fb5c21..9236ebcc8f 100644 --- a/src/backend/utils/adt/tsvector_op.c +++ b/src/backend/utils/adt/tsvector_op.c @@ -1854,6 +1854,18 @@ TS_execute(QueryItem *curitem, void *arg, uint32 fl= ags, return TS_execute_recurse(curitem, arg, flags, chkcond) !=3D TS_NO; } = +/* + * Evaluate tsquery boolean expression. + * + * This is the same as TS_execute except that TS_MAYBE is returned as-is. + */ +TSTernaryValue +TS_execute_ternary(QueryItem *curitem, void *arg, uint32 flags, + TSExecuteCallback chkcond) +{ + return TS_execute_recurse(curitem, arg, flags, chkcond); +} + /* * TS_execute recursion for operators above any phrase operator. Here we= do * not need to worry about lexeme positions. As soon as we hit an OP_PHR= ASE diff --git a/src/include/tsearch/ts_utils.h b/src/include/tsearch/ts_utils= .h index 69a9ba8524..4266560151 100644 --- a/src/include/tsearch/ts_utils.h +++ b/src/include/tsearch/ts_utils.h @@ -199,6 +199,9 @@ typedef TSTernaryValue (*TSExecuteCallback) (void *arg= , QueryOperand *val, = extern bool TS_execute(QueryItem *curitem, void *arg, uint32 flags, TSExecuteCallback chkcond); +extern TSTernaryValue TS_execute_ternary(QueryItem *curitem, void *arg, + uint32 flags, + TSExecuteCallback chkcond); extern bool tsquery_requires_match(QueryItem *curitem); = /* ------- =_aaaaaaaaaa0--