Subject: Question: consolidating strpos searches?

From: James Addison <[email-masked]>
Date: Sat, 4 Jan 2025 17:16:06 +0000

Hello and Happy New Year,

I'd like to validate or reject a performance-related PostgreSQL patch
idea, regarding multiple strpos calls that operate on a common text
input.

---

From: Tom Lane <[email-masked]>
Date: Sat, 04 Jan 2025 12:45:11 -0500

James Addison <[email-masked]> writes:
> In other words: each additional strpos(value, ...) expression
> increased the evaluation time by a similar, significant duration of
> two seconds.  This seems to confirm the basis that each expression is
> currently evaluated separately, by an independent read from the input
> text.

That's correct.  We have not felt a need to try to identify duplicate
function calls: the cost of looking for them would seldom be repaid,
so that such an optimization would be a net loss for most people.
(In your example, they're not even truly duplicates.)

> I'd like to suggest the introduction of a documented multiple string
> matching algorithm[1], to yield results for each of multiple strpos
> calls while only reading through their common input text at-most once.

This seems like something with a pretty narrow audience.  I'd suggest
building it as an extension rather than trying to persuade people
it belongs in core Postgres.

			regards, tom lane

---

From: Greg Sabino Mullane <[email-masked]>
Date: Sat, 4 Jan 2025 14:03:59 -0500

On Sat, Jan 4, 2025 at 12:16 PM James Addison <[email-masked]> wrote:

> In the context of considering writing a patch: would the complexity of
> implementing such a feature for PostgreSQL be worth the potential
> performance benefits?

Probably not. As Tom said, this sounds like it should be tried as an
extension.

And either way, is there more I should learn about and consider?  How would
> I provide convincing supporting
> evidence if I do write a patch?
>

As this is the performance mailing list, it might help to describe the
real-world problem being encountered here. There are other ways to solve
this particular issue. Among them would be using OR not AND in your
contrived example, using partial indexes, using pg_trgm,  using regular
expressions ( i.e. WHERE value ~ '(known|suffix)' ), redesigning your table
and/or queries, and outsourcing the searching of large strings to a system
more suitable for it.

Cheers,
Greg

---

From: James Addison <[email-masked]>
Date: Wed, 15 Jan 2025 23:58:59 +0000

On Sat, 4 Jan 2025 at 19:04, Greg Sabino Mullane <[email-masked]> wrote:
>> In the context of considering writing a patch: would the complexity of implementing such a feature for PostgreSQL be worth the potential
>> performance benefits?
>
> Probably not. As Tom said, this sounds like it should be tried as an extension.

Will do; thanks, both of you.

>> And either way, is there more I should learn about and consider?  How would I provide convincing supporting
> As this is the performance mailing list, it might help to describe the real-world problem being encountered here. There are other ways to solve this particular issue. Among them would be using OR not AND in your contrived example, using partial indexes, using pg_trgm,  using regular expressions ( i.e. WHERE value ~ '(known|suffix)' ), redesigning your table and/or queries, and outsourcing the searching of large strings to a system more suitable for it.

The example is indeed contrived, and the idea doesn't resolve a
problem I've encountered -- in fact, my interest stems from an open
TODO item to implement Boyer-Moore string search.  I began considering
how to implement multiple string pattern search in that context -- but
LIKE/ILIKE introduce a few non-trivial considerations -- notably
wildcard patterns -- compared to strpos.  Whether to require strict
ordering of search results can also be relevant, depending on the
pattern match approach (and boolean operators, as noted) involved.