pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Chris <rc@networkz.ch>
To: pgsql-general@lists.postgresql.org
Subject: pg_trgm vs. Solr ngram
Date: Fri, 10 Feb 2023 03:20:36 +0100
Message-ID: <4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch> (raw)

Hello list

I'm pondering migrating an FTS application from Solr to Postgres, just 
because we use Postgres for everything else.

The application is basically fgrep with a web frontend. However the 
indexed documents are very computer network specific and contain a lot 
of hyphenated hostnames with dot-separated domains, as well as IPv4 and 
IPv6 addresses. In Solr I was using ngrams and customized the 
TokenizerFactories until more or less only whitespace was as separator, 
while [.:-_\d] remains part of the ngrams. This allows to search for 
".12.255/32" or "xzy-eth5.example.org" without any false positives.

It looks like a straight conversion of this method is not possible since 
the tokenization in pg_trgm is not configurable afaict. Is there some 
other good method to search for a random substring including all the 
punctuation using an index? Or a pg_trgm-style module that is more 
flexible like the Solr/Lucene variant?

Or maybe hacking my own pg_trgm wouldn't be so hard and could be fun, do 
I pretty much just need to change the emitted tokens or will this lead 
to significant complications in the operators, indexes etc.?

thanks for any hints & cheers
Christian





view thread (5+ messages)  latest in thread

Message-ID: <4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch>
Permalink:  ../4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch/
Also on:    postgresql.org/message-id/4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-general@postgresql.org
  Cc: rc@networkz.ch
  Subject: Re: pg_trgm vs. Solr ngram
  In-Reply-To: <4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch>
  References: <4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox