public inbox for [email protected]  

help / color / mirror / Atom feed

From: Tom Lane <[email protected]>
To: Robert Haas <[email protected]>
Cc: Jeroen Vermeulen <[email protected]>
Cc: VASUKI M <[email protected]>
Cc: [email protected]
Subject: Re: BUG #19354: JOHAB rejects valid byte sequences
Date: Tue, 16 Dec 2025 10:41:46 -0500
Message-ID: <[email protected]> (raw)
In-Reply-To: <CA+TgmoZaoc37ohnhF5inoPxWzfoznV483xQw8Fmw+ELFScv47g@mail.gmail.com>
References: <[email protected]>
	<CA+TgmoaRGSezRaA7x00X495Qho8WGTzggbDSUt-JsruXceZWug@mail.gmail.com>
	<CA+zULE4L4rA2DLAcfy=eQL7w_ZexV4P5zpQRbP=_qrhJBEOzjg@mail.gmail.com>
	<[email protected]>
	<CAE2r8H5vaSyaC_t1FcpHBo-BB_=SrFj7GFnOC-SxC6WDf5c9VA@mail.gmail.com>
	<CA+zULE47EXZOp7qKYODd+mjSgDiR-WX5ZNBkwdKnj-Zc0FT58w@mail.gmail.com>
	<CA+TgmoZaoc37ohnhF5inoPxWzfoznV483xQw8Fmw+ELFScv47g@mail.gmail.com>

Robert Haas <[email protected]> writes:
> ... So I went looking for
> where we got the mapping tables from. UCS_to_JOHAB.pl expects to read
> from a file JOHAB.TXT, of which the latest version seems to be found
> here:
> https://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/KSC/JOHAB.TXT
> And indeed, if I run UCS_to_JOHAB.pl on that JOHAB.txt file, it
> regenerates the current mapping files.

Thanks for doing that research!

> So apparently we've
> got the "right" mappings, but you can only actually the ones that
> match the code's rules for something to be a valid multi-byte
> character, which aren't actually in sync with the mapping table.

Yeah.  Looking at the code in wchar.c, it's clear that it thinks
that JOHAB has the same character-length rules as EUC_KR, which is
something that one might guess based on available documentation that
says it's related to that encoding.  So I can see how we got here.

However, that doesn't mean we can fix pg_johab_mblen() and we're done.
I'm still quite afraid that we'd be introducing security-grade
inconsistencies of interpretation between different PG versions.

> I'm
> left with the conclusions that (1) nobody ever actually tried using
> this encoding for anything real until 3 days ago and (2) we don't have
> any testing infrastructure that verifies that the characters in the
> mapping tables are actually accepted by pg_verifymbstr(). I wonder how
> many other encodings we have that don't actually work?

Indeed.  Anyone want to do some testing?

			regards, tom lane

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
  Subject: Re: BUG #19354: JOHAB rejects valid byte sequences
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox