From: Tom Lane <tgl@sss.pgh.pa.us>
To: Erik Wienhold <ewie@ewie.name>
Cc: Laurenz Albe <laurenz.albe@cybertec.at>
Cc: Jan Bilek <jan.bilek@eftlab.com.au>
Cc: pgsql-general@postgresql.org <pgsql-general@postgresql.org>
Subject: Re: ERROR: unsupported Unicode escape sequence - in JSON-type column
Date: Mon, 27 Feb 2023 11:12:19 -0500
Message-ID: <368156.1677514339@sss.pgh.pa.us> (raw)
In-Reply-To: <1435849180.1100046.1677511030342@office.mailbox.org>
References: <e3cb6a78-8ff2-dcf1-2fdd-26e3093d3106@eftlab.com.au>
<43a45b6e-08f6-b40d-de6d-ec63e21eb496@eftlab.com.au>
<959c233d610ac3e9ee2c1f7e47ea7775e0d1f1f7.camel@cybertec.at>
<1435849180.1100046.1677511030342@office.mailbox.org>
Erik Wienhold <ewie@ewie.name> writes:
>> On 27/02/2023 13:13 CET Laurenz Albe <laurenz.albe@cybertec.at> wrote:
>> I'd be curious to know how the customer managed to do that.
>> Perhaps there is a loophole in PostgreSQL that needs to be fixed.
> Another reason to prefer jsonb over json to reject such inputs right away.
> The documentation states that json does not validate inputs in constrast to
> jsonb.
It's not that it doesn't validate, it's that the validation rules are
different. Per the manual [1]:
RFC 7159 permits JSON strings to contain Unicode escape sequences
denoted by \uXXXX. In the input function for the json type, Unicode
escapes are allowed regardless of the database encoding, and are
checked only for syntactic correctness (that is, that four hex digits
follow \u). However, the input function for jsonb is stricter: it
disallows Unicode escapes for characters that cannot be represented in
the database encoding. The jsonb type also rejects \u0000 (because
that cannot be represented in PostgreSQL's text type), and it insists
that any use of Unicode surrogate pairs to designate characters
outside the Unicode Basic Multilingual Plane be correct.
You can certainly quibble with our decisions here, but I think they
are reasonably consistent. json is for data that you'd like a syntax
check on (else you might as well store it as "text"), but no more than
a syntax check, because you're going to do the actual JSON processing
elsewhere and you don't want Postgres opining on what semi-standard
JSON constructs mean. If you're actually going to process the data
inside the database, jsonb is a better choice. The extra restrictions
in jsonb are to ensure that a string value represented in JSON can be
extracted into a valid string of our text datatype.
Storing data in json and then casting to jsonb on-the-fly seems like
about the worst possible combination of choices.
regards, tom lane
[1] https://www.postgresql.org/docs/current/datatype-json.html
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: tgl@sss.pgh.pa.us, ewie@ewie.name, laurenz.albe@cybertec.at, jan.bilek@eftlab.com.au
Subject: Re: ERROR: unsupported Unicode escape sequence - in JSON-type column
In-Reply-To: <368156.1677514339@sss.pgh.pa.us>
References: <e3cb6a78-8ff2-dcf1-2fdd-26e3093d3106@eftlab.com.au> <43a45b6e-08f6-b40d-de6d-ec63e21eb496@eftlab.com.au> <959c233d610ac3e9ee2c1f7e47ea7775e0d1f1f7.camel@cybertec.at> <1435849180.1100046.1677511030342@office.mailbox.org> <368156.1677514339@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox