pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Peter Eisentraut <peter@eisentraut.org>
To: Jeff Davis <pgsql@j-davis.com>
To: Chao Li <li.evan.chao@gmail.com>
Cc: Thomas Munro <thomas.munro@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-hackers@postgresql.org
Subject: Re: Remaining dependency on setlocale()
Date: Wed, 17 Dec 2025 11:39:05 +0100
Message-ID: <dd0cdd1f-e786-426e-b336-1ffa9b2f1fc6@eisentraut.org> (raw)
In-Reply-To: <0e186a9a92634f0c5675a618ff5685d00cd8f836.camel@j-davis.com>
References: <4c5da86af36a0d5e430eee3f60ce5e06f1b5cd34.camel@j-davis.com>
	<ed394f839d813762ddcd65cfcab0e062f8b5e0b1.camel@j-davis.com>
	<CA+hUKGLYcbnHBXhHk_OFwZ03LE+D=cAirm7TVN8fMhT9rgcTbA@mail.gmail.com>
	<f040113cf384ada69558ec004a04a3ddb3e40a26.camel@j-davis.com>
	<CA+hUKGJxSDFcTZSGi++PTC=jM49q1oK=v7Lcdf96VQXXvLhoow@mail.gmail.com>
	<cdd20efc0f5418c597791a1e3a5abb68fa3ce35b.camel@j-davis.com>
	<d9657a6e51aa20702447bb2386b32fea6218670f.camel@j-davis.com>
	<b1ea6756-279a-4924-9146-e125af2057b2@eisentraut.org>
	<8186b28a1a39e61a0d833a4c25a8909ebbbabd48.camel@j-davis.com>
	<450ceb6260cad30d7afdf155d991a9caafee7c0d.camel@j-davis.com>
	<DDD50039-02B4-4033-8D22-B086BC448E1C@gmail.com>
	<78edc741a168c54d57cc898a5877b6443527a2c8.camel@j-davis.com>
	<83F6E6A9-FF29-41C9-9E15-EA5360E912D0@gmail.com>
	<5f65b85740197ba6249ea507cddf609f84a6188b.camel@j-davis.com>
	<38f3f4a6-e95e-47a7-914f-110dd6191301@eisentraut.org>
	<0e186a9a92634f0c5675a618ff5685d00cd8f836.camel@j-davis.com>

On 12.12.25 21:11, Jeff Davis wrote:
>> case '\xc7':        /* C with cedilla */
>>
>> so the premise that "fuzzystrmatch is designed for ASCII" does not
>> appear to be correct.  Needs more analysis.
>>
>> (But apparently it's not multibyte aware at all, so I don't know what
>> to
>> do about that.)
> I didn't notice that, thank you. Agreed, we need a bit more discussion
> around this case as well as soundex().

Soundex is an ASCII-only algorithm, there is no expectation that the 
algorithm does anything useful with non-ASCII characters, and it doesn't 
do so now.  So I think using pg_ascii_toupper() is ok.  (Users could for 
example use unaccent to preprocess text.)

One might wonder if the presence of non-ASCII characters should be an 
error, but that doesn't have to be the subject of this thread.  I 
noticed that the Wikipedia page for Soundex even calls out PostgreSQL 
for doing things slightly different than everyone else, but I haven't 
studied the details.

For Metaphone, I found the reference implementation linked from its 
Wikipedia page, and it looks like our implementation is pretty closely 
aligned to that.  That reference implementation also contains the 
C-with-cedilla case explicitly.  The correct fix here would probably be 
to change the implementation to work on wide characters.  But I think 
for the moment you could try a shortcut like, use pg_ascii_toupper(), 
but if the encoding is LATIN1 (or LATIN9 or whichever other encodings 
also contain C-with-cedilla at that code point), then explicitly 
uppercase that one as well.  This would preserve the existing behavior.

Note that the documentation calls out: "At present, the soundex, 
metaphone, dmetaphone, and dmetaphone_alt functions do not work well 
with multibyte encodings (such as UTF-8)."






view thread (96+ messages)  latest in thread

Message-ID: <dd0cdd1f-e786-426e-b336-1ffa9b2f1fc6@eisentraut.org>
Permalink:  ../dd0cdd1f-e786-426e-b336-1ffa9b2f1fc6@eisentraut.org/
Also on:    postgresql.org/message-id/dd0cdd1f-e786-426e-b336-1ffa9b2f1fc6@eisentraut.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: peter@eisentraut.org, pgsql@j-davis.com, li.evan.chao@gmail.com, thomas.munro@gmail.com, tgl@sss.pgh.pa.us
  Subject: Re: Remaining dependency on setlocale()
  In-Reply-To: <dd0cdd1f-e786-426e-b336-1ffa9b2f1fc6@eisentraut.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox