Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pQJ1v-000711-JU for pgsql-general@arkaria.postgresql.org; Fri, 10 Feb 2023 02:20:51 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pQJ1t-0001ob-KP for pgsql-general@arkaria.postgresql.org; Fri, 10 Feb 2023 02:20:49 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pQJ1t-0001oS-A3 for pgsql-general@lists.postgresql.org; Fri, 10 Feb 2023 02:20:49 +0000 Received: from mx6.networkz.ch ([2a01:4f8:1c1c:f346::1] helo=mx4.networkz.ch) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pQJ1m-0000yp-53 for pgsql-general@lists.postgresql.org; Fri, 10 Feb 2023 02:20:48 +0000 Received: from [192.168.178.81] (unknown [157.143.48.26]) by mx4.networkz.ch (Postfix) with ESMTPSA id B2932FA839 for ; Fri, 10 Feb 2023 03:20:37 +0100 (CET) Message-ID: <4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch> Date: Fri, 10 Feb 2023 03:20:36 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 To: pgsql-general@lists.postgresql.org Content-Language: en-US From: Chris Subject: pg_trgm vs. Solr ngram Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Hello list I'm pondering migrating an FTS application from Solr to Postgres, just because we use Postgres for everything else. The application is basically fgrep with a web frontend. However the indexed documents are very computer network specific and contain a lot of hyphenated hostnames with dot-separated domains, as well as IPv4 and IPv6 addresses. In Solr I was using ngrams and customized the TokenizerFactories until more or less only whitespace was as separator, while [.:-_\d] remains part of the ngrams. This allows to search for ".12.255/32" or "xzy-eth5.example.org" without any false positives. It looks like a straight conversion of this method is not possible since the tokenization in pg_trgm is not configurable afaict. Is there some other good method to search for a random substring including all the punctuation using an index? Or a pg_trgm-style module that is more flexible like the Solr/Lucene variant? Or maybe hacking my own pg_trgm wouldn't be so hard and could be fun, do I pretty much just need to change the emitted tokens or will this lead to significant complications in the operators, indexes etc.? thanks for any hints & cheers Christian