From: Chris <rc@networkz.ch>
To: pgsql-general@lists.postgresql.org
Subject: pg_trgm vs. Solr ngram
Date: Fri, 10 Feb 2023 03:20:36 +0100
Message-ID: <4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch> (raw)
Hello list
I'm pondering migrating an FTS application from Solr to Postgres, just
because we use Postgres for everything else.
The application is basically fgrep with a web frontend. However the
indexed documents are very computer network specific and contain a lot
of hyphenated hostnames with dot-separated domains, as well as IPv4 and
IPv6 addresses. In Solr I was using ngrams and customized the
TokenizerFactories until more or less only whitespace was as separator,
while [.:-_\d] remains part of the ngrams. This allows to search for
".12.255/32" or "xzy-eth5.example.org" without any false positives.
It looks like a straight conversion of this method is not possible since
the tokenization in pg_trgm is not configurable afaict. Is there some
other good method to search for a random substring including all the
punctuation using an index? Or a pg_trgm-style module that is more
flexible like the Solr/Lucene variant?
Or maybe hacking my own pg_trgm wouldn't be so hard and could be fun, do
I pretty much just need to change the emitted tokens or will this lead
to significant complications in the operators, indexes etc.?
thanks for any hints & cheers
Christian
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: rc@networkz.ch
Subject: Re: pg_trgm vs. Solr ngram
In-Reply-To: <4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch>
References: <4628c3f6-e2c5-1484-71cf-62446cec984d@networkz.ch>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox