From: Tom Lane <tgl@sss.pgh.pa.us>
To: pgsql-docs@postgreSQL.org
Cc: Oleg Bartunov <oleg@sai.msu.su>
Cc: Teodor Sigaev <teodor@sigaev.ru>
Subject: Use of "token" vs "lexeme" in text search documentation
Date: Mon, 15 Oct 2007 15:22:12 -0400
Message-ID: <20167.1192476132@sss.pgh.pa.us> (raw)
The current documentation seems a bit inconsistent in its use of the
terms "token" and "lexeme". The majority of the text seems to use
"lexeme" exclusively, which is inconsistent with the fact that the
term "token" is exposed by ts_token_type() and friends. But there
are a few places that seem to use "lexeme" to mean something returned
by a dictionary.
I was considering trying to adopt these conventions:
* What a parser returns is a "token".
* When a dictionary recognizes a token, what it returns is a "lexeme".
This would make the phrase "normalized lexeme" redundant, since we
don't call it a lexeme at all unless it's been normalized.
Comments?
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-docs@postgresql.org
Cc: tgl@sss.pgh.pa.us, oleg@sai.msu.su, teodor@sigaev.ru
Subject: Re: Use of "token" vs "lexeme" in text search documentation
In-Reply-To: <20167.1192476132@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox