pg.ddx.io pgsql-docs@postgresql.org mailing list archivehelp / color / mirror / Atom feed
Use of "token" vs "lexeme" in text search documentation 2+ messages / 2 participants [nested] [flat]
* Use of "token" vs "lexeme" in text search documentation @ 2007-10-15 19:22 Tom Lane <tgl@sss.pgh.pa.us> 2007-10-16 16:04 ` Re: Use of "token" vs "lexeme" in text search documentation Oleg Bartunov <oleg@sai.msu.su> 0 siblings, 1 reply; 2+ messages in thread From: Tom Lane @ 2007-10-15 19:22 UTC (permalink / raw) To: pgsql-docs; +Cc: Oleg Bartunov <oleg@sai.msu.su>; Teodor Sigaev <teodor@sigaev.ru> The current documentation seems a bit inconsistent in its use of the terms "token" and "lexeme". The majority of the text seems to use "lexeme" exclusively, which is inconsistent with the fact that the term "token" is exposed by ts_token_type() and friends. But there are a few places that seem to use "lexeme" to mean something returned by a dictionary. I was considering trying to adopt these conventions: * What a parser returns is a "token". * When a dictionary recognizes a token, what it returns is a "lexeme". This would make the phrase "normalized lexeme" redundant, since we don't call it a lexeme at all unless it's been normalized. Comments? regards, tom lane ^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: Use of "token" vs "lexeme" in text search documentation 2007-10-15 19:22 Use of "token" vs "lexeme" in text search documentation Tom Lane <tgl@sss.pgh.pa.us> @ 2007-10-16 16:04 ` Oleg Bartunov <oleg@sai.msu.su> 0 siblings, 0 replies; 2+ messages in thread From: Oleg Bartunov @ 2007-10-16 16:04 UTC (permalink / raw) To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-docs@postgreSQL.org, Teodor Sigaev <teodor@sigaev.ru> On Mon, 15 Oct 2007, Tom Lane wrote: > The current documentation seems a bit inconsistent in its use of the > terms "token" and "lexeme". The majority of the text seems to use > "lexeme" exclusively, which is inconsistent with the fact that the > term "token" is exposed by ts_token_type() and friends. But there > are a few places that seem to use "lexeme" to mean something returned > by a dictionary. > > I was considering trying to adopt these conventions: > > * What a parser returns is a "token". > > * When a dictionary recognizes a token, what it returns is a "lexeme". > > This would make the phrase "normalized lexeme" redundant, since we > don't call it a lexeme at all unless it's been normalized. > > Comments? Hmm, you say what I always thought. I'd be happy if you stress this in docs. > > regards, tom lane > Regards, Oleg _____________________________________________________________ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83 ^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2007-10-16 16:04 UTC | newest] Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2007-10-15 19:22 Use of "token" vs "lexeme" in text search documentation Tom Lane <tgl@sss.pgh.pa.us> 2007-10-16 16:04 ` Oleg Bartunov <oleg@sai.msu.su>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox