pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
ERROR: unsupported Unicode escape sequence - in JSON-type column
6+ messages / 4 participants
[nested] [flat]

* ERROR: unsupported Unicode escape sequence - in JSON-type column
@ 2023-02-27 06:28 Jan Bilek <jan.bilek@eftlab.com.au>
  2023-02-27 12:13 ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Laurenz Albe <laurenz.albe@cybertec.at>
  0 siblings, 1 reply; 6+ messages in thread

From: Jan Bilek @ 2023-02-27 06:28 UTC (permalink / raw)
  To: pgsql-general

Hi team,

Our customer was able to sneak in an Unicode data into a column of a JSON Type and now that record fails on select.

[cid:part1.hzRCQplU.TJZnhVJk@eftlab.com.au]

Would you be able to suggest any way out of this? E.g. finding infringing row, updating its data ... ?

Thanks & Cheers,
Jan

--
Jan Bilek - CTO at EFTlab Pty Ltd.


Attachments:

q0s2w56c4oOuSUld.png (image/png, 18.0K) download view image
^ permalink raw reply [nested|flat] 6+ messages in thread

* Re: ERROR: unsupported Unicode escape sequence - in JSON-type column
  2023-02-27 06:28 ERROR: unsupported Unicode escape sequence - in JSON-type column Jan Bilek <jan.bilek@eftlab.com.au>
@ 2023-02-27 12:13 ` Laurenz Albe <laurenz.albe@cybertec.at>
  2023-02-27 15:17   ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Erik Wienhold <ewie@ewie.name>
  2023-02-28 00:55   ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Jan Bilek <jan.bilek@eftlab.com.au>
  0 siblings, 2 replies; 6+ messages in thread

From: Laurenz Albe @ 2023-02-27 12:13 UTC (permalink / raw)
  To: Jan Bilek <jan.bilek@eftlab.com.au>; pgsql-general

On Mon, 2023-02-27 at 06:28 +0000, Jan Bilek wrote:
> Our customer was able to sneak in an Unicode data into a column of a JSON Type and now that record fails on select.
> Would you be able to suggest any way out of this? E.g. finding infringing row, updating its data ... ?

I'd be curious to know how the customer managed to do that.
Perhaps there is a loophole in PostgreSQL that needs to be fixed.

First, find the table that contains the column.
Then you can try something like

  DO
  $$DECLARE
     pkey bigint;
  BEGIN
     FOR pkey IN SELECT id FROM jsontab LOOP
        BEGIN  -- starts block with exception handler
           PERFORM jsoncol -> 'creationDateTime'
           FROM jsontab
           WHERE id = pkey;
        EXCEPTION
           WHEN untranslatable_character THEN
              RAISE NOTICE 'bad character in line with id = %', pkey;
        END;
     END LOOP;
  END;$$;

Yours,
Laurenz Albe
-- 
Cybertec | https://www.cybertec-postgresql.com





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: ERROR: unsupported Unicode escape sequence - in JSON-type column
  2023-02-27 06:28 ERROR: unsupported Unicode escape sequence - in JSON-type column Jan Bilek <jan.bilek@eftlab.com.au>
  2023-02-27 12:13 ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Laurenz Albe <laurenz.albe@cybertec.at>
@ 2023-02-27 15:17   ` Erik Wienhold <ewie@ewie.name>
  2023-02-27 16:12     ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Tom Lane <tgl@sss.pgh.pa.us>
  2023-02-28 01:03     ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Jan Bilek <jan.bilek@eftlab.com.au>
  1 sibling, 2 replies; 6+ messages in thread

From: Erik Wienhold @ 2023-02-27 15:17 UTC (permalink / raw)
  To: Laurenz Albe <laurenz.albe@cybertec.at>; Jan Bilek <jan.bilek@eftlab.com.au>; pgsql-general

> On 27/02/2023 13:13 CET Laurenz Albe <laurenz.albe@cybertec.at> wrote:
>
> I'd be curious to know how the customer managed to do that.
> Perhaps there is a loophole in PostgreSQL that needs to be fixed.

Probably via some data access layer and not directly via Postgres.  It's easy
to reproduce with psycopg:

	import psycopg

	with psycopg.connect() as con:
	    con.execute('create temp table jsontab (jsoncol json)')
	    con.execute(
	        'insert into jsontab (jsoncol) values (%s)',
	        [psycopg.types.json.Json('\0')],
	    )

	    with con.execute('select jsoncol from jsontab') as cur:
	        print(cur.fetchall())

	    try:
	        with con.execute('select jsoncol::jsonb from jsontab') as cur:
	            pass
	        raise AssertionError("jsonb should fail")
	    except psycopg.errors.UntranslatableCharacter:
	        pass

Another reason to prefer jsonb over json to reject such inputs right away.
The documentation states that json does not validate inputs in constrast to
jsonb.

Of course the OP now has to deal with json.  The data can be sanitized by
replacing all null character escape sequences:

	update jsontab
	set jsoncol = replace(jsoncol::text, '\u0000', '')::json
	where strpos(jsoncol::text, '\u0000') > 0;

But the data access layer (or whatever got the json into the database) must be
fixed as well to reject or sanitize those inputs in the future.

--
Erik





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: ERROR: unsupported Unicode escape sequence - in JSON-type column
  2023-02-27 06:28 ERROR: unsupported Unicode escape sequence - in JSON-type column Jan Bilek <jan.bilek@eftlab.com.au>
  2023-02-27 12:13 ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Laurenz Albe <laurenz.albe@cybertec.at>
  2023-02-27 15:17   ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Erik Wienhold <ewie@ewie.name>
@ 2023-02-27 16:12     ` Tom Lane <tgl@sss.pgh.pa.us>
  1 sibling, 0 replies; 6+ messages in thread

From: Tom Lane @ 2023-02-27 16:12 UTC (permalink / raw)
  To: Erik Wienhold <ewie@ewie.name>; +Cc: Laurenz Albe <laurenz.albe@cybertec.at>; Jan Bilek <jan.bilek@eftlab.com.au>; pgsql-general

Erik Wienhold <ewie@ewie.name> writes:
>> On 27/02/2023 13:13 CET Laurenz Albe <laurenz.albe@cybertec.at> wrote:
>> I'd be curious to know how the customer managed to do that.
>> Perhaps there is a loophole in PostgreSQL that needs to be fixed.

> Another reason to prefer jsonb over json to reject such inputs right away.
> The documentation states that json does not validate inputs in constrast to
> jsonb.

It's not that it doesn't validate, it's that the validation rules are
different.  Per the manual [1]:

    RFC 7159 permits JSON strings to contain Unicode escape sequences
    denoted by \uXXXX. In the input function for the json type, Unicode
    escapes are allowed regardless of the database encoding, and are
    checked only for syntactic correctness (that is, that four hex digits
    follow \u). However, the input function for jsonb is stricter: it
    disallows Unicode escapes for characters that cannot be represented in
    the database encoding. The jsonb type also rejects \u0000 (because
    that cannot be represented in PostgreSQL's text type), and it insists
    that any use of Unicode surrogate pairs to designate characters
    outside the Unicode Basic Multilingual Plane be correct.

You can certainly quibble with our decisions here, but I think they
are reasonably consistent.  json is for data that you'd like a syntax
check on (else you might as well store it as "text"), but no more than
a syntax check, because you're going to do the actual JSON processing
elsewhere and you don't want Postgres opining on what semi-standard
JSON constructs mean.  If you're actually going to process the data
inside the database, jsonb is a better choice.  The extra restrictions
in jsonb are to ensure that a string value represented in JSON can be
extracted into a valid string of our text datatype.

Storing data in json and then casting to jsonb on-the-fly seems like
about the worst possible combination of choices.

			regards, tom lane

[1] https://www.postgresql.org/docs/current/datatype-json.html





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: ERROR: unsupported Unicode escape sequence - in JSON-type column
  2023-02-27 06:28 ERROR: unsupported Unicode escape sequence - in JSON-type column Jan Bilek <jan.bilek@eftlab.com.au>
  2023-02-27 12:13 ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Laurenz Albe <laurenz.albe@cybertec.at>
  2023-02-27 15:17   ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Erik Wienhold <ewie@ewie.name>
@ 2023-02-28 01:03     ` Jan Bilek <jan.bilek@eftlab.com.au>
  1 sibling, 0 replies; 6+ messages in thread

From: Jan Bilek @ 2023-02-28 01:03 UTC (permalink / raw)
  To: Erik Wienhold <ewie@ewie.name>; Laurenz Albe <laurenz.albe@cybertec.at>; pgsql-general

On 2/28/23 01:17, Erik Wienhold wrote:
>> On 27/02/2023 13:13 CET Laurenz Albe <laurenz.albe@cybertec.at> wrote:
>>
>> I'd be curious to know how the customer managed to do that.
>> Perhaps there is a loophole in PostgreSQL that needs to be fixed.
> Probably via some data access layer and not directly via Postgres.  It's easy
> to reproduce with psycopg:
>
> 	import psycopg
>
> 	with psycopg.connect() as con:
> 	    con.execute('create temp table jsontab (jsoncol json)')
> 	    con.execute(
> 	        'insert into jsontab (jsoncol) values (%s)',
> 	        [psycopg.types.json.Json('\0')],
> 	    )
>
> 	    with con.execute('select jsoncol from jsontab') as cur:
> 	        print(cur.fetchall())
>
> 	    try:
> 	        with con.execute('select jsoncol::jsonb from jsontab') as cur:
> 	            pass
> 	        raise AssertionError("jsonb should fail")
> 	    except psycopg.errors.UntranslatableCharacter:
> 	        pass
>
> Another reason to prefer jsonb over json to reject such inputs right away.
> The documentation states that json does not validate inputs in constrast to
> jsonb.
>
> Of course the OP now has to deal with json.  The data can be sanitized by
> replacing all null character escape sequences:
>
> 	update jsontab
> 	set jsoncol = replace(jsoncol::text, '\u0000', '')::json
> 	where strpos(jsoncol::text, '\u0000') > 0;
>
> But the data access layer (or whatever got the json into the database) must be
> fixed as well to reject or sanitize those inputs in the future.
>
> --
> Erik

Hi Erik,

No, it didn't go through any foreign data access layer - it went in 
straight through the Postgresql variable bind using pre-cached insert 
statement using PostgreSQL 14.5, connected over UNIX sockets.

Strange thing happened afterwards - that locating that record was on & 
off - I couldn't pin-point it in DB as it seemed to be failing on 
multiple places ... until using that trick from Laurenz. Felt like a 
PostgreSQL memory corruption, but system remained stable without any 
complaints.

Thanks & Cheers,
Jan

-- 
Jan Bilek - CTO at EFTlab Pty Ltd.



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: ERROR: unsupported Unicode escape sequence - in JSON-type column
  2023-02-27 06:28 ERROR: unsupported Unicode escape sequence - in JSON-type column Jan Bilek <jan.bilek@eftlab.com.au>
  2023-02-27 12:13 ` Re: ERROR: unsupported Unicode escape sequence - in JSON-type column Laurenz Albe <laurenz.albe@cybertec.at>
@ 2023-02-28 00:55   ` Jan Bilek <jan.bilek@eftlab.com.au>
  1 sibling, 0 replies; 6+ messages in thread

From: Jan Bilek @ 2023-02-28 00:55 UTC (permalink / raw)
  To: Laurenz Albe <laurenz.albe@cybertec.at>; pgsql-general

On 2/27/23 22:13, Laurenz Albe wrote:

On Mon, 2023-02-27 at 06:28 +0000, Jan Bilek wrote:


Our customer was able to sneak in an Unicode data into a column of a JSON Type and now that record fails on select.
Would you be able to suggest any way out of this? E.g. finding infringing row, updating its data ... ?



I'd be curious to know how the customer managed to do that.
Perhaps there is a loophole in PostgreSQL that needs to be fixed.

First, find the table that contains the column.
Then you can try something like

  DO
  $$DECLARE
     pkey bigint;
  BEGIN
     FOR pkey IN SELECT id FROM jsontab LOOP
        BEGIN  -- starts block with exception handler
           PERFORM jsoncol -> 'creationDateTime'
           FROM jsontab
           WHERE id = pkey;
        EXCEPTION
           WHEN untranslatable_character THEN
              RAISE NOTICE 'bad character in line with id = %', pkey;
        END;
     END LOOP;
  END;$$;

Yours,
Laurenz Albe


Hi Laurenz,

Thank you and yes, that's how we managed to go through that - one of our devs found similar approach described here: https://stackoverflow.com/questions/31671634/handling-unicode-sequences-in-postgresql (see the null_if_invalid_string function there + credits to Hendrik) and we reapplied it. FYI with a bit of tinkering we've been able to retrieve following (corrupted) data:

[cid:part1.XnBLfLs0.ptVnd3Gz@eftlab.com.au]

(It comes from a PROD system so I don't have it in a text form for you to experiment on that.)

Anyway, your solution points in exactly same direction.

How'd customer managed to do that? Still no idea ... looks like they restarted TCP connection on our middle-ware, but any partial packets should be dropped as not matching TCP length header. Also records are deserialized on receive so that would fail. Still, that record had to make it somehow in the PostgreSQL. We are still looking.

Thanks & Cheers,
Jan

--
Jan Bilek - CTO at EFTlab Pty Ltd.


Attachments:

dcuZLHUsas1qwrL0.png (image/png, 18.3K) download view image
^ permalink raw reply [nested|flat] 6+ messages in thread


end of thread, other threads:[~2023-02-28 01:03 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-02-27 06:28 ERROR: unsupported Unicode escape sequence - in JSON-type column Jan Bilek <jan.bilek@eftlab.com.au>
2023-02-27 12:13 ` Laurenz Albe <laurenz.albe@cybertec.at>
2023-02-27 15:17   ` Erik Wienhold <ewie@ewie.name>
2023-02-27 16:12     ` Tom Lane <tgl@sss.pgh.pa.us>
2023-02-28 01:03     ` Jan Bilek <jan.bilek@eftlab.com.au>
2023-02-28 00:55   ` Jan Bilek <jan.bilek@eftlab.com.au>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox