public inbox for [email protected]  

help / color / mirror / Atom feed

From: Evan Jones <[email protected]>
To: [email protected]
Subject: [PATCH] hstore: Fix parsing on Mac OS X: isspace() is locale specific
Date: Mon, 5 Jun 2023 11:26:56 -0400
Message-ID: <CA+HWA9awUW0+RV_gO9r1ABZwGoZxPztcJxPy8vMFSTbTfi4jig@mail.gmail.com> (raw)

This patch fixes a rare parsing bug with unicode characters on Mac OS X.
The problem is that isspace() on Mac OS X changes its behaviour with the
locale. Use scanner_isspace instead, which only returns true for ASCII
whitespace. It appears other places in the Postgres code have already run
into this, since a number of places use scanner_isspace instead. However,
there are still a lot of other calls to isspace(). I'll try to take a quick
look to see if there might be other instances of this bug.

The bug is that in the following hstore value, the unicode character
"disappears", and is replaced with "key\xc4", because it is parsed
incorrectly:

select E'keyą=>value'::hstore;
     hstore
-----------------
 "keyą"=>"value"
(1 row)

select 'keyą=>value'::hstore::text::bytea;
              bytea
----------------------------------
 \x226b6579c4223d3e2276616c756522
(1 row)

The correct result should be:

     hstore
-----------------
 "keyą"=>"value"
(1 row)

That query is added to the regression test. The query works on Linux, but
failed on Mac OS X.

For a more detailed explanation of how isspace() works, on Mac OS X, see:
https://github.com/evanj/isspace_locale

Thanks!

Evan Jones


Attachments:

  [application/octet-stream] hstore-isspace.patch (2.5K, ../CA+HWA9awUW0+RV_gO9r1ABZwGoZxPztcJxPy8vMFSTbTfi4jig@mail.gmail.com/3-hstore-isspace.patch)
  download | inline diff:
diff --git a/contrib/hstore/expected/hstore.out b/contrib/hstore/expected/hstore.out
index 1836c9acf3..26815cbc7f 100644
--- a/contrib/hstore/expected/hstore.out
+++ b/contrib/hstore/expected/hstore.out
@@ -243,6 +243,14 @@ select '	'::hstore;
  
 (1 row)
 
+-- UTF-8 Mac OS X en_US.UTF-8 locale bug: isspace(0x85) returns true
+-- \u0105 encodes as 0xc4 0x85 in UTF-8; the 0x85 was interpreted as whitespace
+select E'key\u0105=>value'::hstore;
+     hstore      
+-----------------
+ "keyą"=>"value"
+(1 row)
+
 -- invalid input
 select '  =>null'::hstore;
 ERROR:  syntax error in hstore, near "=" at position 2
diff --git a/contrib/hstore/hstore_io.c b/contrib/hstore/hstore_io.c
index cec7df71a2..999ddad76d 100644
--- a/contrib/hstore/hstore_io.c
+++ b/contrib/hstore/hstore_io.c
@@ -13,6 +13,7 @@
 #include "lib/stringinfo.h"
 #include "libpq/pqformat.h"
 #include "nodes/miscnodes.h"
+#include "parser/scansup.h"
 #include "utils/builtins.h"
 #include "utils/json.h"
 #include "utils/jsonb.h"
@@ -118,7 +119,7 @@ get_val(HSParser *state, bool ignoreeq, bool *escaped)
 			{
 				st = GV_WAITESCIN;
 			}
-			else if (!isspace((unsigned char) *(state->ptr)))
+			else if (!scanner_isspace((unsigned char) *(state->ptr)))
 			{
 				*(state->cur) = *(state->ptr);
 				state->cur++;
@@ -141,7 +142,7 @@ get_val(HSParser *state, bool ignoreeq, bool *escaped)
 				state->ptr--;
 				return true;
 			}
-			else if (isspace((unsigned char) *(state->ptr)))
+			else if (scanner_isspace((unsigned char) *(state->ptr)))
 			{
 				return true;
 			}
@@ -255,7 +256,7 @@ parse_hstore(HSParser *state)
 			{
 				PRSEOF;
 			}
-			else if (!isspace((unsigned char) *(state->ptr)))
+			else if (!scanner_isspace((unsigned char) *(state->ptr)))
 			{
 				PRSSYNTAXERROR;
 			}
@@ -309,7 +310,7 @@ parse_hstore(HSParser *state)
 			{
 				return true;
 			}
-			else if (!isspace((unsigned char) *(state->ptr)))
+			else if (!scanner_isspace((unsigned char) *(state->ptr)))
 			{
 				PRSSYNTAXERROR;
 			}
diff --git a/contrib/hstore/sql/hstore.sql b/contrib/hstore/sql/hstore.sql
index efef91292a..36c7422018 100644
--- a/contrib/hstore/sql/hstore.sql
+++ b/contrib/hstore/sql/hstore.sql
@@ -53,6 +53,10 @@ select e'\\"a=>q"w'::hstore;
 select ''::hstore;
 select '	'::hstore;
 
+-- UTF-8 Mac OS X en_US.UTF-8 locale bug: isspace(0x85) returns true
+-- \u0105 encodes as 0xc4 0x85 in UTF-8; the 0x85 was interpreted as whitespace
+select E'key\u0105=>value'::hstore;
+
 -- invalid input
 select '  =>null'::hstore;
 select 'aa=>"'::hstore;

view thread (2+ messages)

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected]
  Subject: Re: [PATCH] hstore: Fix parsing on Mac OS X: isspace() is locale specific
  In-Reply-To: <CA+HWA9awUW0+RV_gO9r1ABZwGoZxPztcJxPy8vMFSTbTfi4jig@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox