From: Victor Drobny <v.drobny@postgrespro.ru>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] new function for tsquery creartion
Date: Tue, 28 Nov 2017 09:56:10 +0300
Message-ID: <15517a04afd2be779e2e6d69611c348b@postgrespro.ru> (raw)
In-Reply-To: <bff75704-a832-e240-d216-5ba6a8db5d13@2ndquadrant.com>
References: <fe931111ff7e9ad79196486ada79e268@postgrespro.ru>
<CA+TgmoYSBLFAq14ePMWC=JBQ4D4Crt08emkcDPa65SQ6sbVAoQ@mail.gmail.com>
<CAEepm=2S-rXC-G3iGSJLKwThF2v9zWG0pJB4R6v5LVVphwQkRw@mail.gmail.com>
<69234700d26674fc7c171e48b850824c@postgrespro.ru>
<bff75704-a832-e240-d216-5ba6a8db5d13@2ndquadrant.com>
On 2017-11-19 04:30, Tomas Vondra wrote:
Hello,
> Hi,
>
> On 09/13/2017 10:57 AM, Victor Drobny wrote:
>> On 2017-09-09 06:03, Thomas Munro wrote:
>>> Please send a rebased version of the patch for people to review and
>>> test as that one has bit-rotted.
>> Hello,
>> Thank you for interest. In the attachment you can find rebased
>> version(based on 69835bc8988812c960f4ed5aeee86b62ac73602a commit)
>>
>
> I did a quick review of the patch today. The patch unfortunately no
> longer applies, so I had to use an older commit from September. Please
> rebase to current master.
Thank you for your time. In the attachment you can find rebased version.
(based on e842791b0 commit)
> I've only looked on the diff at this point, will do more testing once
> the rebase happens.
>
> Some comments:
>
> 1) This seems to mix multiple improvements into one patch. There's the
> support for alternative query syntax, and then there are the new
> operators (AROUND and <m,n>). I propose to split the patch into two or
> more parts, each addressing one of those bits.
I agree. I have split it in 3 parts: support for around operator,
queryto_tsquery function and documentation.
> I guess there will be two or three parts - first adding the syntax,
> second adding <m,n> and third adding the AROUND(n). Seems reasonable?
>
> 2) I don't think we should mention Google in the docs explicitly. Not
> that I'm somehow anti-google, but this syntax was certainly not
> invented
> by Google - I vividly remember using something like that on Altavista
> (yeah, I'm old). And it's used by pretty much every other web search
> engine out there ...
Yes, those syntax is not introduced by google, but, as for me, it was
the
easiest way to give a brief description of it. Of cause it can be
changed,
I just don't know how. Any suggestions are welcomed! ;)
> 3) In the SGML docs, please use <literal></literal> instead of just
> quoting the values. So it should be <literal>|</literal> instead of '|'
> etc. Just like in the parts describing plainto_tsquery, for example.
Fixed. I hope that i didn't miss anything.
> 4) Also, I recommend adding a brief explanation what the examples do.
> Right now there's just a bunch of queryto_tsquery, and the reader is
> expected to understand the output. I suggest adding a sentence or two,
> explaining what's happening (just like for plainto_tsquery examples).
>
> 5) I'm not sure about negative values in the <n,m> operator. I don't
> find it particularly confusing - once you understand that (a <n,m> b)
> means "there are 'k' words between 'a' and 'b' (n <= k <= m)", then
> negative values seem like a fairly straightforward extension.
>
> But I guess the main question is "Is there really a demand for the new
> <n,m> operator, or have we just invented if because we can?"
The operator <n,m> is not introduced yet. It's just a concept. It were
our
thoughts about implementation AROUND operator through <n,m> in future.
> 6) There seem to be some new constants defined, with names not
> following
> the naming convention. I mean this
>
> #define WAITOPERAND 1
> #define WAITOPERATOR 2
> #define WAITFIRSTOPERAND 3
> #define WAITSINGLEOPERAND 4
> #define INSIDEQUOTES 5 <-- the new one
>
> and
>
> #define TSPO_L_ONLY 0x01
> #define TSPO_R_ONLY 0x02
> #define TSPO_BOTH 0x04
> #define TS_NOT_EXAC 0x08 <-- the new one
>
> Perhaps that's OK, but it seems a bit inconsistent.
I agree. I have fixed it.
>
> regards
--
Victor Drobny
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:download all (3 files, .tar.gz)queryto_tsquery3_01_around.patch (text/x-diff, 12.0K) downloaddiff --git a/src/backend/tsearch/ts_selfuncs.c b/src/backend/tsearch/ts_selfuncs.cindex 046f543..375cb5c 100644--- a/src/backend/tsearch/ts_selfuncs.c+++ b/src/backend/tsearch/ts_selfuncs.c@@ -396,6 +396,7 @@ tsquery_opr_selec(QueryItem *item, char *operand,
break;
case OP_PHRASE:
+ case OP_AROUND:
case OP_AND:
s1 = tsquery_opr_selec(item + 1, operand,
lookup, length, minfreq);
diff --git a/src/backend/utils/adt/tsginidx.c b/src/backend/utils/adt/tsginidx.cindex aba456e..1c852bd 100644--- a/src/backend/utils/adt/tsginidx.c+++ b/src/backend/utils/adt/tsginidx.c@@ -239,10 +239,11 @@ TS_execute_ternary(GinChkVal *gcv, QueryItem *curitem, bool in_phrase)
return !result;
case OP_PHRASE:
+ case OP_AROUND:
/*
* GIN doesn't contain any information about positions, so treat
- * OP_PHRASE as OP_AND with recheck requirement+ * OP_PHRASE and OP_AROUND as OP_AND with recheck requirement
*/
*(gcv->need_recheck) = true;
/* Pass down in_phrase == true in case there's a NOT below */
diff --git a/src/backend/utils/adt/tsquery.c b/src/backend/utils/adt/tsquery.cindex 5cdfe4d..07d56b0 100644--- a/src/backend/utils/adt/tsquery.c+++ b/src/backend/utils/adt/tsquery.c@@ -29,7 +29,8 @@ const int tsearch_op_priority[OP_COUNT] =
4, /* OP_NOT */
2, /* OP_AND */
1, /* OP_OR */
- 3 /* OP_PHRASE */+ 3, /* OP_PHRASE */+ 3 /* OP_AROUND */
};
struct TSQueryParserStateData
@@ -58,10 +59,10 @@ struct TSQueryParserStateData
};
/* parser's states */
-#define WAITOPERAND 1-#define WAITOPERATOR 2-#define WAITFIRSTOPERAND 3-#define WAITSINGLEOPERAND 4+#define WAITOPERAND 1+#define WAITOPERATOR 2+#define WAITFIRSTOPERAND 3+#define WAITSINGLEOPERAND 4
/*
* subroutine to parse the modifiers (weight and prefix flag currently)
@@ -210,6 +211,69 @@ typedef enum
PT_CLOSE = 5
} ts_tokentype;
+static bool+has_prefix(char * str, char * prefix)+{+ if (strlen(prefix) > strlen(str))+ {+ return false;+ }+ while (*prefix != '\0')+ {+ if (*(str++) != *(prefix++))+ {+ return false;+ }+ }+ return true;+}++/*+ * Parse around operator. The operator+ * have the following form:+ *+ * a AROUND(X) b (distance is no greater than X)+ *+ * The buffer should begins with "AROUND(" prefix+ */+static char *+parse_around_operator(char *buf, int16 *distance)+{+ char *ptr = buf;+ char *endptr;+ long l = 1;++ Assert(has_prefix(ptr, "AROUND("));++ ptr += strlen("AROUND(");++ while (t_isspace(ptr))+ ptr++;++ l = strtol(ptr, &endptr, 10);+ if (ptr == endptr)+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("Invalid AROUND(X) operator!")));+ else if (errno == ERANGE || l > MAXENTRYPOS)+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("distance in AROUND operator should not be greater than %d",+ MAXENTRYPOS)));++ ptr = endptr;+ *distance = l;+ while (t_isspace(ptr))+ ptr++;++ if (!t_iseq(ptr, ')'))+ ereport(ERROR,+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),+ errmsg("Missing ')' in AROUND(X) operator")));++ return ++ptr;+}+
/*
* get token from query string
*
@@ -301,6 +365,16 @@ gettoken_query(TSQueryParserState state,
return PT_ERR;
return PT_OPR;
}
+ else if (has_prefix(state->buf, "AROUND("))+ {+ state->state = WAITOPERAND;+ *operator = OP_AROUND;+ /* weight var is used as storage for distance */+ state->buf = parse_around_operator(state->buf, weight);+ if (*weight < 0)+ return PT_ERR;+ return PT_OPR;+ }
else if (t_iseq(state->buf, ')'))
{
(state->buf)++;
@@ -336,12 +410,13 @@ pushOperator(TSQueryParserState state, int8 oper, int16 distance)
{
QueryOperator *tmp;
- Assert(oper == OP_NOT || oper == OP_AND || oper == OP_OR || oper == OP_PHRASE);+ Assert(oper == OP_NOT || oper == OP_AND || oper == OP_OR+ || oper == OP_PHRASE || oper == OP_AROUND);
tmp = (QueryOperator *) palloc0(sizeof(QueryOperator));
tmp->type = QI_OPR;
tmp->oper = oper;
- tmp->distance = (oper == OP_PHRASE) ? distance : 0;+ tmp->distance = (oper == OP_PHRASE || oper == OP_AROUND) ? distance : 0;
/* left is filled in later with findoprnd */
state->polstr = lcons(tmp, state->polstr);
@@ -555,7 +630,8 @@ findoprnd_recurse(QueryItem *ptr, uint32 *pos, int nnodes, bool *needcleanup)
Assert(curitem->oper == OP_AND ||
curitem->oper == OP_OR ||
- curitem->oper == OP_PHRASE);+ curitem->oper == OP_PHRASE ||+ curitem->oper == OP_AROUND);
(*pos)++;
@@ -884,6 +960,9 @@ infix(INFIX *in, int parentPriority, bool rightPhraseOp)
else
sprintf(in->cur, " <-> %s", nrm.buf);
break;
+ case OP_AROUND:+ sprintf(in->cur, " AROUND(%d) %s", distance, nrm.buf);+ break;
default:
/* OP_NOT is handled in above if-branch */
elog(ERROR, "unrecognized operator type: %d", op);
@@ -966,7 +1045,8 @@ tsquerysend(PG_FUNCTION_ARGS)
break;
case QI_OPR:
pq_sendint8(&buf, item->qoperator.oper);
- if (item->qoperator.oper == OP_PHRASE)+ if (item->qoperator.oper == OP_PHRASE+ || item->qoperator.oper == OP_AROUND)
pq_sendint16(&buf, item->qoperator.distance);
break;
default:
@@ -1062,14 +1142,15 @@ tsqueryrecv(PG_FUNCTION_ARGS)
int8 oper;
oper = (int8) pq_getmsgint(buf, sizeof(int8));
- if (oper != OP_NOT && oper != OP_OR && oper != OP_AND && oper != OP_PHRASE)+ if (oper != OP_NOT && oper != OP_OR && oper != OP_AND+ && oper != OP_PHRASE && oper != OP_AROUND)
elog(ERROR, "invalid tsquery: unrecognized operator type %d",
(int) oper);
if (i == size - 1)
elog(ERROR, "invalid pointer to right operand");
item->qoperator.oper = oper;
- if (oper == OP_PHRASE)+ if (oper == OP_PHRASE || oper == OP_AROUND)
item->qoperator.distance = (int16) pq_getmsgint(buf, sizeof(int16));
}
else
diff --git a/src/backend/utils/adt/tsquery_cleanup.c b/src/backend/utils/adt/tsquery_cleanup.cindex 350171c..bb7fbd5 100644--- a/src/backend/utils/adt/tsquery_cleanup.c+++ b/src/backend/utils/adt/tsquery_cleanup.c@@ -161,7 +161,8 @@ clean_NOT_intree(NODE *node)
NODE *res = node;
Assert(node->valnode->qoperator.oper == OP_AND ||
- node->valnode->qoperator.oper == OP_PHRASE);+ node->valnode->qoperator.oper == OP_PHRASE ||+ node->valnode->qoperator.oper == OP_AROUND);
node->left = clean_NOT_intree(node->left);
node->right = clean_NOT_intree(node->right);
@@ -277,7 +278,8 @@ clean_stopword_intree(NODE *node, int *ladd, int *radd)
node->right = clean_stopword_intree(node->right, &rladd, &rradd);
/* Check if current node is OP_PHRASE, get its distance */
- isphrase = (node->valnode->qoperator.oper == OP_PHRASE);+ isphrase = (node->valnode->qoperator.oper == OP_PHRASE ||+ node->valnode->qoperator.oper == OP_AROUND);
ndistance = isphrase ? node->valnode->qoperator.distance : 0;
if (node->left == NULL && node->right == NULL)
diff --git a/src/backend/utils/adt/tsquery_op.c b/src/backend/utils/adt/tsquery_op.cindex 755c3e9..7cf9b8a 100644--- a/src/backend/utils/adt/tsquery_op.c+++ b/src/backend/utils/adt/tsquery_op.c@@ -37,7 +37,7 @@ join_tsqueries(TSQuery a, TSQuery b, int8 operator, uint16 distance)
res->valnode = (QueryItem *) palloc0(sizeof(QueryItem));
res->valnode->type = QI_OPR;
res->valnode->qoperator.oper = operator;
- if (operator == OP_PHRASE)+ if (operator == OP_PHRASE || operator == OP_AROUND)
res->valnode->qoperator.distance = distance;
res->child = (QTNode **) palloc0(sizeof(QTNode *) * 2);
diff --git a/src/backend/utils/adt/tsquery_util.c b/src/backend/utils/adt/tsquery_util.cindex 971bb81..548a846 100644--- a/src/backend/utils/adt/tsquery_util.c+++ b/src/backend/utils/adt/tsquery_util.c@@ -121,7 +121,7 @@ QTNodeCompare(QTNode *an, QTNode *bn)
return res;
}
- if (ao->oper == OP_PHRASE && ao->distance != bo->distance)+ if ((ao->oper == OP_PHRASE || ao->oper == OP_AROUND) && ao->distance != bo->distance)
return (ao->distance > bo->distance) ? -1 : 1;
return 0;
@@ -171,7 +171,8 @@ QTNSort(QTNode *in)
for (i = 0; i < in->nchild; i++)
QTNSort(in->child[i]);
- if (in->nchild > 1 && in->valnode->qoperator.oper != OP_PHRASE)+ if (in->nchild > 1 && in->valnode->qoperator.oper != OP_PHRASE+ && in->valnode->qoperator.oper != OP_AROUND)
qsort((void *) in->child, in->nchild, sizeof(QTNode *), cmpQTN);
}
diff --git a/src/backend/utils/adt/tsrank.c b/src/backend/utils/adt/tsrank.cindex 4577bcc..b62b14d 100644--- a/src/backend/utils/adt/tsrank.c+++ b/src/backend/utils/adt/tsrank.c@@ -366,7 +366,8 @@ calc_rank(const float *w, TSVector t, TSQuery q, int32 method)
/* XXX: What about NOT? */
res = (item->type == QI_OPR && (item->qoperator.oper == OP_AND ||
- item->qoperator.oper == OP_PHRASE)) ?+ item->qoperator.oper == OP_PHRASE ||+ item->qoperator.oper == OP_AROUND)) ?
calc_rank_and(w, t, q) :
calc_rank_or(w, t, q);
diff --git a/src/backend/utils/adt/tsvector_op.c b/src/backend/utils/adt/tsvector_op.cindex 8225202..e8bc9eb 100644--- a/src/backend/utils/adt/tsvector_op.c+++ b/src/backend/utils/adt/tsvector_op.c@@ -1429,6 +1429,7 @@ checkcondition_str(void *checkval, QueryOperand *val, ExecPhraseData *data)
#define TSPO_L_ONLY 0x01 /* emit positions appearing only in L */
#define TSPO_R_ONLY 0x02 /* emit positions appearing only in R */
#define TSPO_BOTH 0x04 /* emit positions appearing in both L&R */
+#define TSPO_NOT_EXAC 0x08 /* not exact distance for AROUND(X) */
static bool
TS_phrase_output(ExecPhraseData *data,
@@ -1473,8 +1474,18 @@ TS_phrase_output(ExecPhraseData *data,
Rpos = INT_MAX;
}
+ /* Processing OP_AROUND */+ if ((emit & TSPO_NOT_EXAC) &&+ Lpos - Rpos >= 0 &&+ Lpos - Rpos <= (Loffset + Roffset) * 2 - Rdata->width + Ldata->width)+ {+ if (emit & TSPO_BOTH)+ output_pos = Rpos;+ Lindex++;+ Rindex++;+ }
/* Merge-join the two input lists */
- if (Lpos < Rpos)+ else if (Lpos < Rpos)
{
/* Lpos is not matched in Rdata, should we output it? */
if (emit & TSPO_L_ONLY)
@@ -1625,6 +1636,7 @@ TS_phrase_execute(QueryItem *curitem, void *arg, uint32 flags,
}
case OP_PHRASE:
+ case OP_AROUND:
case OP_AND:
memset(&Ldata, 0, sizeof(Ldata));
memset(&Rdata, 0, sizeof(Rdata));
@@ -1647,7 +1659,7 @@ TS_phrase_execute(QueryItem *curitem, void *arg, uint32 flags,
(Rdata.npos == 0 && !Rdata.negate))
return (flags & TS_EXEC_PHRASE_NO_POS) ? true : false;
- if (curitem->qoperator.oper == OP_PHRASE)+ if (curitem->qoperator.oper == OP_PHRASE || curitem->qoperator.oper == OP_AROUND)
{
/*
* Compute Loffset and Roffset suitable for phrase match, and
@@ -1703,7 +1715,8 @@ TS_phrase_execute(QueryItem *curitem, void *arg, uint32 flags,
{
/* straight AND */
return TS_phrase_output(data, &Ldata, &Rdata,
- TSPO_BOTH,+ TSPO_BOTH |+ (curitem->qoperator.oper == OP_AROUND ? TSPO_NOT_EXAC : 0),
Loffset, Roffset,
Min(Ldata.npos, Rdata.npos));
}
@@ -1843,6 +1856,7 @@ TS_execute(QueryItem *curitem, void *arg, uint32 flags,
return TS_execute(curitem + 1, arg, flags, chkcond);
case OP_PHRASE:
+ case OP_AROUND:
return TS_phrase_execute(curitem, arg, flags, chkcond, NULL);
default:
@@ -1882,9 +1896,10 @@ tsquery_requires_match(QueryItem *curitem)
return false;
case OP_PHRASE:
+ case OP_AROUND:
/*
- * Treat OP_PHRASE as OP_AND here+ * Treat OP_PHRASE and OP_AROUND as OP_AND here
*/
case OP_AND:
/* If either side requires a match, we're good */
diff --git a/src/include/tsearch/ts_type.h b/src/include/tsearch/ts_type.hindex 30d7c4b..ee1a184 100644--- a/src/include/tsearch/ts_type.h+++ b/src/include/tsearch/ts_type.h@@ -166,8 +166,9 @@ typedef struct
#define OP_NOT 1
#define OP_AND 2
#define OP_OR 3
-#define OP_PHRASE 4 /* highest code, tsquery_cleanup.c */-#define OP_COUNT 4+#define OP_PHRASE 4+#define OP_AROUND 5 /* highest code, tsquery_cleanup.c */+#define OP_COUNT 5
extern const int tsearch_op_priority[OP_COUNT];
queryto_tsquery3_02_queryto_tsquery.patch (text/x-diff, 18.5K) downloaddiff --git a/src/backend/tsearch/to_tsany.c b/src/backend/tsearch/to_tsany.cindex cf55e39..2972540 100644--- a/src/backend/tsearch/to_tsany.c+++ b/src/backend/tsearch/to_tsany.c@@ -390,7 +390,8 @@ add_to_tsvector(void *_state, char *elem_value, int elem_len)
* and different variants are ORed together.
*/
static void
-pushval_morph(Datum opaque, TSQueryParserState state, char *strval, int lenval, int16 weight, bool prefix)+pushval_morph(Datum opaque, TSQueryParserState state, char *strval, int lenval,+ int16 weight, bool prefix, bool isphrase)
{
int32 count = 0;
ParsedText prs;
@@ -423,7 +424,12 @@ pushval_morph(Datum opaque, TSQueryParserState state, char *strval, int lenval,
/* put placeholders for each missing stop word */
pushStop(state);
if (cntpos)
- pushOperator(state, data->qoperator, 1);+ {+ if (isphrase)+ pushOperator(state, OP_PHRASE, 1);+ else+ pushOperator(state, data->qoperator, 1);+ }
cntpos++;
pos++;
}
@@ -464,7 +470,10 @@ pushval_morph(Datum opaque, TSQueryParserState state, char *strval, int lenval,
if (cntpos)
{
/* distance may be useful */
- pushOperator(state, data->qoperator, 1);+ if (isphrase)+ pushOperator(state, OP_PHRASE, 1);+ else+ pushOperator(state, data->qoperator, 1);
}
cntpos++;
@@ -490,6 +499,7 @@ to_tsquery_byid(PG_FUNCTION_ARGS)
query = parse_tsquery(text_to_cstring(in),
pushval_morph,
PointerGetDatum(&data),
+ false,
false);
PG_RETURN_TSQUERY(query);
@@ -520,7 +530,8 @@ plainto_tsquery_byid(PG_FUNCTION_ARGS)
query = parse_tsquery(text_to_cstring(in),
pushval_morph,
PointerGetDatum(&data),
- true);+ true,+ false);
PG_RETURN_POINTER(query);
}
@@ -551,7 +562,8 @@ phraseto_tsquery_byid(PG_FUNCTION_ARGS)
query = parse_tsquery(text_to_cstring(in),
pushval_morph,
PointerGetDatum(&data),
- true);+ true,+ false);
PG_RETURN_TSQUERY(query);
}
@@ -567,3 +579,36 @@ phraseto_tsquery(PG_FUNCTION_ARGS)
ObjectIdGetDatum(cfgId),
PointerGetDatum(in)));
}
++Datum+queryto_tsquery_byid(PG_FUNCTION_ARGS)+{+ text *in = PG_GETARG_TEXT_PP(1);+ MorphOpaque data;+ TSQuery query = NULL;++ data.cfg_id = PG_GETARG_OID(0);++ data.qoperator = OP_AND;++ query = parse_tsquery(text_to_cstring(in),+ pushval_morph,+ PointerGetDatum(&data),+ false,+ true);++ PG_RETURN_TSQUERY(query);+}++Datum+queryto_tsquery(PG_FUNCTION_ARGS)+{+ text *in = PG_GETARG_TEXT_PP(0);+ Oid cfgId;++ cfgId = getTSCurrentConfig(true);+ PG_RETURN_DATUM(DirectFunctionCall2(queryto_tsquery_byid,+ ObjectIdGetDatum(cfgId),+ PointerGetDatum(in)));++}
\ No newline at end of file
diff --git a/src/backend/utils/adt/tsquery.c b/src/backend/utils/adt/tsquery.cindex 07d56b0..53c9fc7 100644--- a/src/backend/utils/adt/tsquery.c+++ b/src/backend/utils/adt/tsquery.c@@ -63,6 +63,7 @@ struct TSQueryParserStateData
#define WAITOPERATOR 2
#define WAITFIRSTOPERAND 3
#define WAITSINGLEOPERAND 4
+#define WAITQUOTE 5
/*
* subroutine to parse the modifiers (weight and prefix flag currently)
@@ -285,7 +286,8 @@ parse_around_operator(char *buf, int16 *distance)
static ts_tokentype
gettoken_query(TSQueryParserState state,
int8 *operator,
- int *lenval, char **strval, int16 *weight, bool *prefix)+ int *lenval, char **strval, int16 *weight, bool *prefix,+ bool isquery)
{
*weight = 0;
*prefix = false;
@@ -296,7 +298,8 @@ gettoken_query(TSQueryParserState state,
{
case WAITFIRSTOPERAND:
case WAITOPERAND:
- if (t_iseq(state->buf, '!'))+ if (t_iseq(state->buf, '!') ||+ (isquery && t_iseq(state->buf, '-')))
{
(state->buf)++; /* can safely ++, t_iseq guarantee that
* pg_mblen()==1 */
@@ -318,6 +321,20 @@ gettoken_query(TSQueryParserState state,
errmsg("syntax error in tsquery: \"%s\"",
state->buffer)));
}
+ else if (isquery && t_iseq(state->buf, '"'))+ {+ char *quote = strchr(state->buf + 1, '"');+ if (quote == NULL)+ {+ state->buf++;+ continue;+ }+ *strval = state->buf + 1;+ *lenval = quote - state->buf - 1;+ state->buf = quote + 1;+ state->state = WAITQUOTE;+ return PT_VAL;+ }
else if (!t_isspace(state->buf))
{
/*
@@ -355,6 +372,13 @@ gettoken_query(TSQueryParserState state,
(state->buf)++;
return PT_OPR;
}
+ else if (isquery && has_prefix(state->buf, "OR "))+ {+ state->state = WAITOPERAND;+ *operator = OP_OR;+ (state->buf) += 3;+ return PT_OPR;+ }
else if (t_iseq(state->buf, '<'))
{
state->state = WAITOPERAND;
@@ -365,7 +389,7 @@ gettoken_query(TSQueryParserState state,
return PT_ERR;
return PT_OPR;
}
- else if (has_prefix(state->buf, "AROUND("))+ else if (isquery && has_prefix(state->buf, "AROUND("))
{
state->state = WAITOPERAND;
*operator = OP_AROUND;
@@ -381,8 +405,23 @@ gettoken_query(TSQueryParserState state,
state->count--;
return (state->count < 0) ? PT_ERR : PT_CLOSE;
}
+ else if (t_iseq(state->buf, '('))+ {+ *operator = OP_AND;+ state->state = WAITOPERAND;+ return PT_OPR;+ }
else if (*(state->buf) == '\0')
return (state->count) ? PT_ERR : PT_END;
+ else if (isquery &&+ (t_isalpha(state->buf) || t_iseq(state->buf, '!')+ || t_iseq(state->buf, '-')+ || t_iseq(state->buf, '"')))+ {+ state->state = WAITOPERAND;+ *operator = OP_AND;+ return PT_OPR;+ }
else if (!t_isspace(state->buf))
return PT_ERR;
break;
@@ -394,6 +433,9 @@ gettoken_query(TSQueryParserState state,
state->buf += strlen(state->buf);
state->count++;
return PT_VAL;
+ case WAITQUOTE:+ state->state = WAITOPERATOR;+ continue;
default:
return PT_ERR;
break;
@@ -550,7 +592,8 @@ cleanOpStack(TSQueryParserState state,
static void
makepol(TSQueryParserState state,
PushFunction pushval,
- Datum opaque)+ Datum opaque,+ bool isquery)
{
int8 operator = 0;
ts_tokentype type;
@@ -564,19 +607,20 @@ makepol(TSQueryParserState state,
/* since this function recurses, it could be driven to stack overflow */
check_stack_depth();
- while ((type = gettoken_query(state, &operator, &lenval, &strval, &weight, &prefix)) != PT_END)+ while ((type = gettoken_query(state, &operator, &lenval, &strval, &weight,+ &prefix, isquery)) != PT_END)
{
switch (type)
{
case PT_VAL:
- pushval(opaque, state, strval, lenval, weight, prefix);+ pushval(opaque, state, strval, lenval, weight, prefix, state->state == WAITQUOTE);
break;
case PT_OPR:
cleanOpStack(state, opstack, &lenstack, operator);
pushOpStack(opstack, &lenstack, operator, weight);
break;
case PT_OPEN:
- makepol(state, pushval, opaque);+ makepol(state, pushval, opaque, isquery);
break;
case PT_CLOSE:
cleanOpStack(state, opstack, &lenstack, OP_OR /* lowest */ );
@@ -681,7 +725,8 @@ TSQuery
parse_tsquery(char *buf,
PushFunction pushval,
Datum opaque,
- bool isplain)+ bool isplain,+ bool isquery)
{
struct TSQueryParserStateData state;
int i;
@@ -708,7 +753,7 @@ parse_tsquery(char *buf,
*(state.curop) = '\0';
/* parse query & make polish notation (postfix, but in reverse order) */
- makepol(&state, pushval, opaque);+ makepol(&state, pushval, opaque, isquery);
close_tsvector_parser(state.valstate);
@@ -779,7 +824,7 @@ parse_tsquery(char *buf,
static void
pushval_asis(Datum opaque, TSQueryParserState state, char *strval, int lenval,
- int16 weight, bool prefix)+ int16 weight, bool prefix, bool isphrase)
{
pushValue(state, strval, lenval, weight, prefix);
}
@@ -792,7 +837,7 @@ tsqueryin(PG_FUNCTION_ARGS)
{
char *in = PG_GETARG_CSTRING(0);
- PG_RETURN_TSQUERY(parse_tsquery(in, pushval_asis, PointerGetDatum(NULL), false));+ PG_RETURN_TSQUERY(parse_tsquery(in, pushval_asis, PointerGetDatum(NULL), false, false));
}
/*
diff --git a/src/backend/utils/adt/tsquery_cleanup.c b/src/backend/utils/adt/tsquery_cleanup.cindex e679bb5..bb7fbd5 100644--- a/src/backend/utils/adt/tsquery_cleanup.c+++ b/src/backend/utils/adt/tsquery_cleanup.c@@ -278,7 +278,7 @@ clean_stopword_intree(NODE *node, int *ladd, int *radd)
node->right = clean_stopword_intree(node->right, &rladd, &rradd);
/* Check if current node is OP_PHRASE, get its distance */
- isphrase = (node->valnode->qoperator.oper == OP_PHRASE ||+ isphrase = (node->valnode->qoperator.oper == OP_PHRASE ||
node->valnode->qoperator.oper == OP_AROUND);
ndistance = isphrase ? node->valnode->qoperator.distance : 0;
diff --git a/src/backend/utils/adt/tsvector_op.c b/src/backend/utils/adt/tsvector_op.cindex ce7ee25..e8bc9eb 100644--- a/src/backend/utils/adt/tsvector_op.c+++ b/src/backend/utils/adt/tsvector_op.c@@ -1715,7 +1715,7 @@ TS_phrase_execute(QueryItem *curitem, void *arg, uint32 flags,
{
/* straight AND */
return TS_phrase_output(data, &Ldata, &Rdata,
- TSPO_BOTH |+ TSPO_BOTH |
(curitem->qoperator.oper == OP_AROUND ? TSPO_NOT_EXAC : 0),
Loffset, Roffset,
Min(Ldata.npos, Rdata.npos));
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.hindex c969375..9abde15 100644--- a/src/include/catalog/pg_proc.h+++ b/src/include/catalog/pg_proc.h@@ -4906,6 +4906,8 @@ DATA(insert OID = 3746 ( to_tsquery PGNSP PGUID 12 100 0 0 0 f f f f t f i s 2
DESCR("make tsquery");
DATA(insert OID = 3747 ( plainto_tsquery PGNSP PGUID 12 100 0 0 0 f f f f t f i s 2 0 3615 "3734 25" _null_ _null_ _null_ _null_ _null_ plainto_tsquery_byid _null_ _null_ _null_ ));
DESCR("transform to tsquery");
+DATA(insert OID = 8889 ( queryto_tsquery PGNSP PGUID 12 100 0 0 0 f f f f t f i s 2 0 3615 "3734 25" _null_ _null_ _null_ _null_ _null_ queryto_tsquery_byid _null_ _null_ _null_ ));+DESCR("transform to tsquery");
DATA(insert OID = 5006 ( phraseto_tsquery PGNSP PGUID 12 100 0 0 0 f f f f t f i s 2 0 3615 "3734 25" _null_ _null_ _null_ _null_ _null_ phraseto_tsquery_byid _null_ _null_ _null_ ));
DESCR("transform to tsquery");
DATA(insert OID = 3749 ( to_tsvector PGNSP PGUID 12 100 0 0 0 f f f f t f s s 1 0 3614 "25" _null_ _null_ _null_ _null_ _null_ to_tsvector _null_ _null_ _null_ ));
@@ -4914,6 +4916,8 @@ DATA(insert OID = 3750 ( to_tsquery PGNSP PGUID 12 100 0 0 0 f f f f t f s s 1
DESCR("make tsquery");
DATA(insert OID = 3751 ( plainto_tsquery PGNSP PGUID 12 100 0 0 0 f f f f t f s s 1 0 3615 "25" _null_ _null_ _null_ _null_ _null_ plainto_tsquery _null_ _null_ _null_ ));
DESCR("transform to tsquery");
+DATA(insert OID = 8890 ( queryto_tsquery PGNSP PGUID 12 100 0 0 0 f f f f t f s s 1 0 3615 "25" _null_ _null_ _null_ _null_ _null_ queryto_tsquery _null_ _null_ _null_ ));+DESCR("transform to tsquery");
DATA(insert OID = 5001 ( phraseto_tsquery PGNSP PGUID 12 100 0 0 0 f f f f t f s s 1 0 3615 "25" _null_ _null_ _null_ _null_ _null_ phraseto_tsquery _null_ _null_ _null_ ));
DESCR("transform to tsquery");
DATA(insert OID = 4209 ( to_tsvector PGNSP PGUID 12 100 0 0 0 f f f f t f s s 1 0 3614 "3802" _null_ _null_ _null_ _null_ _null_ jsonb_to_tsvector _null_ _null_ _null_ ));
diff --git a/src/include/tsearch/ts_utils.h b/src/include/tsearch/ts_utils.hindex 782548c..ffb1762 100644--- a/src/include/tsearch/ts_utils.h+++ b/src/include/tsearch/ts_utils.h@@ -44,11 +44,12 @@ typedef void (*PushFunction) (Datum opaque, TSQueryParserState state,
char *token, int tokenlen,
int16 tokenweights, /* bitmap as described in
* QueryOperand struct */
- bool prefix);+ bool prefix,+ bool isphrase);
extern TSQuery parse_tsquery(char *buf,
PushFunction pushval,
- Datum opaque, bool isplain);+ Datum opaque, bool isplain, bool isquery);
/* Functions for use by PushFunction implementations */
extern void pushValue(TSQueryParserState state,
diff --git a/src/test/regress/expected/tsearch.out b/src/test/regress/expected/tsearch.outindex d63fb12..2eb410b 100644--- a/src/test/regress/expected/tsearch.out+++ b/src/test/regress/expected/tsearch.out@@ -1672,3 +1672,115 @@ select * from phrase_index_test where fts @@ phraseto_tsquery('english', 'fat ca
(1 row)
set enable_seqscan = on;
++--test queryto_tsquery function+select queryto_tsquery('My brand new smartphone');+ queryto_tsquery +-------------------------------+ 'brand' & 'new' & 'smartphon'+(1 row)++select queryto_tsquery('My brand "new smartphone"');+ queryto_tsquery +---------------------------------+ 'brand' & 'new' <-> 'smartphon'+(1 row)++select queryto_tsquery('"A fat cat" has just eaten a -rat.');+ queryto_tsquery +------------------------------------+ 'fat' <-> 'cat' & 'eaten' & !'rat'+(1 row)++select queryto_tsquery('"A fat cat" has just eaten OR -rat.');+ queryto_tsquery +------------------------------------+ 'fat' <-> 'cat' & 'eaten' | !'rat'+(1 row)++select queryto_tsquery('"A fat cat" has just (eaten OR -rat)');+ queryto_tsquery +----------------------------------------+ 'fat' <-> 'cat' & ( 'eaten' | !'rat' )+(1 row)++-- testing AROUND operator evaluation+select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"gnu debugger" AROUND(5) runs');+ ?column? +----------+ f+(1 row)++select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('run AROUND(5) "gnu debugger"');+ ?column? +----------+ f+(1 row)++select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"gnu debugger" AROUND(6) runs');+ ?column? +----------+ t+(1 row)++select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('run AROUND(6) "gnu debugger"');+ ?column? +----------+ t+(1 row)++select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"many programming languages" AROUND(10) "portable debugger"');+ ?column? +----------+ f+(1 row)++select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"portable debugger" AROUND(10) "many programming languages"');+ ?column? +----------+ f+(1 row)++select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"many programming languages" AROUND(11) "portable debugger"');+ ?column? +----------+ t+(1 row)++select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"portable debugger" AROUND(11) "many programming languages"');+ ?column? +----------+ t+(1 row)++select queryto_tsquery('"fat cat AROUND(5) rat"');+ queryto_tsquery +------------------------------------------------+ 'fat' <-> 'cat' <-> 'around' <-> '5' <-> 'rat'+(1 row)++select queryto_tsquery('simple','"fat cat OR rat"');+ queryto_tsquery +------------------------------------+ 'fat' <-> 'cat' <-> 'or' <-> 'rat'+(1 row)++select queryto_tsquery('fat*rat');+ queryto_tsquery +-----------------+ 'fat' & 'rat'+(1 row)++select queryto_tsquery('fat-rat');+ queryto_tsquery +---------------------------+ 'fat-rat' & 'fat' & 'rat'+(1 row)diff --git a/src/test/regress/sql/tsearch.sql b/src/test/regress/sql/tsearch.sqlindex 1c8520b..65e71da 100644--- a/src/test/regress/sql/tsearch.sql+++ b/src/test/regress/sql/tsearch.sql@@ -539,3 +539,34 @@ create index phrase_index_test_idx on phrase_index_test using gin(fts);
set enable_seqscan = off;
select * from phrase_index_test where fts @@ phraseto_tsquery('english', 'fat cat');
set enable_seqscan = on;
++--test queryto_tsquery function+select queryto_tsquery('My brand new smartphone');+select queryto_tsquery('My brand "new smartphone"');+select queryto_tsquery('"A fat cat" has just eaten a -rat.');+select queryto_tsquery('"A fat cat" has just eaten OR -rat.');+select queryto_tsquery('"A fat cat" has just (eaten OR -rat)');++-- testing AROUND operator evaluation+select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"gnu debugger" AROUND(5) runs');+select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('run AROUND(5) "gnu debugger"');+select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"gnu debugger" AROUND(6) runs');+select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('run AROUND(6) "gnu debugger"');++select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"many programming languages" AROUND(10) "portable debugger"');+select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"portable debugger" AROUND(10) "many programming languages"');+select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"many programming languages" AROUND(11) "portable debugger"');+select to_tsvector('The GNU Debugger is a portable debugger that runs on many Unix like systems and works for many programming languages') @@+queryto_tsquery('"portable debugger" AROUND(11) "many programming languages"');++select queryto_tsquery('"fat cat AROUND(5) rat"');+select queryto_tsquery('simple','"fat cat OR rat"');+select queryto_tsquery('fat*rat');+select queryto_tsquery('fat-rat');
\ No newline at end of file
queryto_tsquery3_03_documentation.patch (text/x-diff, 4.9K) downloaddiff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgmlindex 4dd9d02..b4d95ed 100644--- a/doc/src/sgml/func.sgml+++ b/doc/src/sgml/func.sgml@@ -9535,6 +9535,18 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<row>
<entry>
<indexterm>
+ <primary>queryto_tsquery</primary>+ </indexterm>+ <literal><function>queryto_tsquery(<optional> <replaceable class="parameter">config</replaceable> <type>regconfig</type> , </optional> <replaceable class="parameter">query</replaceable> <type>text</type>)</function></literal>+ </entry>+ <entry><type>tsquery</type></entry>+ <entry>produce <type>tsquery</type> from google like query</entry>+ <entry><literal>queryto_tsquery('english', 'The Fat Rats')</literal></entry>+ <entry><literal>'fat' & 'rat'</literal></entry>+ </row>+ <row>+ <entry>+ <indexterm>
<primary>querytree</primary>
</indexterm>
<literal><function>querytree(<replaceable class="parameter">query</replaceable> <type>tsquery</type>)</function></literal>
diff --git a/doc/src/sgml/textsearch.sgml b/doc/src/sgml/textsearch.sgmlindex 4dc52ec..5e05985 100644--- a/doc/src/sgml/textsearch.sgml+++ b/doc/src/sgml/textsearch.sgml@@ -797,13 +797,15 @@ UPDATE tt SET ti =
<para>
<productname>PostgreSQL</productname> provides the
functions <function>to_tsquery</function>,
- <function>plainto_tsquery</function>, and- <function>phraseto_tsquery</function>+ <function>plainto_tsquery</function>,+ <function>phraseto_tsquery</function> and+ <function>queryto_tsquery</function>
for converting a query to the <type>tsquery</type> data type.
<function>to_tsquery</function> offers access to more features
than either <function>plainto_tsquery</function> or
<function>phraseto_tsquery</function>, but it is less forgiving
- about its input.+ about its input. <function>queryto_tsquery</function> provides a + different, Google like syntax to create tsquery.
</para>
<indexterm>
@@ -960,8 +962,72 @@ SELECT phraseto_tsquery('english', 'The Fat & Rats:C');
-----------------------------
'fat' <-> 'rat' <-> 'c'
</screen>
- </para>+</para>++<synopsis>+queryto_tsquery(<optional> <replaceable class="parameter">config</replaceable> <type>regconfig</type>, </optional> <replaceable class="parameter">querytext</replaceable> <type>text</type>) returns <type>tsquery</type>+</synopsis>+ <para>+ <function>queryto_tsquery</function> creates a <type>tsquery</type> from a unformated text.+ But instead of <function>plainto_tsquery</function> and <function>phraseto_tsquery</function> it won't+ ignore already placed operations. This function supports following operators:+ <itemizedlist spacing="compact" mark="bullet">+ <listitem>+ <para>+ <literal>some text</literal> - any text inside quote signs will be treated as a phrase and will be+ performed like in <function>phraseto_tsquery</function>.+ </para>+ </listitem>+ <listitem>+ <para>+ <literal>OR</literal> - standard logical operator. It is just an alias for <literal>|</literal> sign.+ </para>+ </listitem>+ <listitem>+ <para>+ <literal>terma AROUND(N) termb</literal> - this operation will match if the distance between + terma and termb is less than N.+ </para>+ </listitem>+ <listitem>+ <para>+ <literal>-</literal> - standard logical negation sign. It is an alias for <literal>!</literal> sign.+ </para>+ </listitem>+ </itemizedlist>+ Other missing operators will be replaced by AND like in <function>plainto_tsquery</function>.+ </para>+ <para>+ Examples:+ <screen>+ select queryto_tsquery('The fat rats');+ queryto_tsquery + -----------------+ 'fat' & 'rat'+ (1 row)+ </screen>+ <screen>+ select queryto_tsquery('"supernovae stars" AND -crab');+ queryto_tsquery + ----------------------------------+ 'supernova' <-> 'star' & !'crab'+ (1 row)+ </screen>+ <screen>+ select queryto_tsquery('-run AROUND(5) "gnu debugger" OR "I like bananas"');+ queryto_tsquery + -----------------------------------------------------------+ !'run' AROUND(5) 'gnu' <-> 'debugg' | 'like' <-> 'banana'+ (1 row)+ </screen>+ </para>+ <para>+ Note that in the examples <function>queryto_tsquery</function> didn't ignore+ operators if they were placed and put required operations+ if they were skipped. In case of inquote text <function>queryto_tsquery</function>+ has placed phrase operator and & in other cases.+ </para>
</sect2>
<sect2 id="textsearch-ranking">
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: v.drobny@postgrespro.ru, tomas.vondra@2ndquadrant.com, thomas.munro@enterprisedb.com, robertmhaas@gmail.com
Subject: Re: [HACKERS] new function for tsquery creartion
In-Reply-To: <15517a04afd2be779e2e6d69611c348b@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox