Received: from localhost (unknown [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id D7BE19F99B2 for ; Mon, 15 Oct 2007 16:22:17 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 31911-07 for ; Mon, 15 Oct 2007 16:22:15 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey-1.7.5 Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by postgresql.org (Postfix) with ESMTP id 16AF79F9928 for ; Mon, 15 Oct 2007 16:22:14 -0300 (ADT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.1/8.14.1) with ESMTP id l9FJMDcE020168; Mon, 15 Oct 2007 15:22:13 -0400 (EDT) To: pgsql-docs@postgreSQL.org cc: Oleg Bartunov , Teodor Sigaev Subject: Use of "token" vs "lexeme" in text search documentation Date: Mon, 15 Oct 2007 15:22:12 -0400 Message-ID: <20167.1192476132@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200710/49 X-Sequence-Number: 4470 The current documentation seems a bit inconsistent in its use of the terms "token" and "lexeme". The majority of the text seems to use "lexeme" exclusively, which is inconsistent with the fact that the term "token" is exposed by ts_token_type() and friends. But there are a few places that seem to use "lexeme" to mean something returned by a dictionary. I was considering trying to adopt these conventions: * What a parser returns is a "token". * When a dictionary recognizes a token, what it returns is a "lexeme". This would make the phrase "normalized lexeme" redundant, since we don't call it a lexeme at all unless it's been normalized. Comments? regards, tom lane