pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Erik Wienhold <ewie@ewie.name>
Cc: Laurenz Albe <laurenz.albe@cybertec.at>
Cc: Jan Bilek <jan.bilek@eftlab.com.au>
Cc: pgsql-general@postgresql.org <pgsql-general@postgresql.org>
Subject: Re: ERROR: unsupported Unicode escape sequence - in JSON-type column
Date: Mon, 27 Feb 2023 11:12:19 -0500
Message-ID: <368156.1677514339@sss.pgh.pa.us> (raw)
In-Reply-To: <1435849180.1100046.1677511030342@office.mailbox.org>
References: <e3cb6a78-8ff2-dcf1-2fdd-26e3093d3106@eftlab.com.au>
	<43a45b6e-08f6-b40d-de6d-ec63e21eb496@eftlab.com.au>
	<959c233d610ac3e9ee2c1f7e47ea7775e0d1f1f7.camel@cybertec.at>
	<1435849180.1100046.1677511030342@office.mailbox.org>

Erik Wienhold <ewie@ewie.name> writes:
>> On 27/02/2023 13:13 CET Laurenz Albe <laurenz.albe@cybertec.at> wrote:
>> I'd be curious to know how the customer managed to do that.
>> Perhaps there is a loophole in PostgreSQL that needs to be fixed.

> Another reason to prefer jsonb over json to reject such inputs right away.
> The documentation states that json does not validate inputs in constrast to
> jsonb.

It's not that it doesn't validate, it's that the validation rules are
different.  Per the manual [1]:

    RFC 7159 permits JSON strings to contain Unicode escape sequences
    denoted by \uXXXX. In the input function for the json type, Unicode
    escapes are allowed regardless of the database encoding, and are
    checked only for syntactic correctness (that is, that four hex digits
    follow \u). However, the input function for jsonb is stricter: it
    disallows Unicode escapes for characters that cannot be represented in
    the database encoding. The jsonb type also rejects \u0000 (because
    that cannot be represented in PostgreSQL's text type), and it insists
    that any use of Unicode surrogate pairs to designate characters
    outside the Unicode Basic Multilingual Plane be correct.

You can certainly quibble with our decisions here, but I think they
are reasonably consistent.  json is for data that you'd like a syntax
check on (else you might as well store it as "text"), but no more than
a syntax check, because you're going to do the actual JSON processing
elsewhere and you don't want Postgres opining on what semi-standard
JSON constructs mean.  If you're actually going to process the data
inside the database, jsonb is a better choice.  The extra restrictions
in jsonb are to ensure that a string value represented in JSON can be
extracted into a valid string of our text datatype.

Storing data in json and then casting to jsonb on-the-fly seems like
about the worst possible combination of choices.

			regards, tom lane

[1] https://www.postgresql.org/docs/current/datatype-json.html





view thread (6+ messages)  latest in thread

Message-ID: <368156.1677514339@sss.pgh.pa.us>
Permalink:  ../368156.1677514339@sss.pgh.pa.us/
Also on:    postgresql.org/message-id/368156.1677514339@sss.pgh.pa.us

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-general@postgresql.org
  Cc: tgl@sss.pgh.pa.us, ewie@ewie.name, laurenz.albe@cybertec.at, jan.bilek@eftlab.com.au
  Subject: Re: ERROR: unsupported Unicode escape sequence - in JSON-type column
  In-Reply-To: <368156.1677514339@sss.pgh.pa.us>
  References: <e3cb6a78-8ff2-dcf1-2fdd-26e3093d3106@eftlab.com.au> <43a45b6e-08f6-b40d-de6d-ec63e21eb496@eftlab.com.au> <959c233d610ac3e9ee2c1f7e47ea7775e0d1f1f7.camel@cybertec.at> <1435849180.1100046.1677511030342@office.mailbox.org> <368156.1677514339@sss.pgh.pa.us>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox