pg.ddx.io  psycopg/psycopg GitHub issues and pull requests  
help / color / mirror / Atom feed
From: dvarrazzo (@dvarrazzo) <noreply+dvarrazzo@github.com>
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
Subject: Re: [psycopg/psycopg] PR #1505: Decode JSON data using the connection encoding
Date: Thu, 19 Nov 2026 22:27:17 +0000
Message-ID: <gh-psycopg-psycopg-1505-c7023964406@github.com> (raw)
In-Reply-To: <gh-psycopg-psycopg-1505@github.com>
References: <gh-psycopg-psycopg-1505@github.com>

I played a bit with the issues and I think that the loader side is now largely correct. 

I've run some tests, and there is still a corner case in which we get things wrong. If 1) a databasein non-utf8 encoding 2) contains json with non-ascii chars which is 3) read with a client in SQL_ASCII. I think that this combination is pathological enough that we might not care about it: SQL_ASCII is a pretty special configuration, as it means reading raw data from the server, so it's not really even compatible with reading JSON and expecting a Python value. We should return objects that have bytes instead of strings, so it wouldn't even be valid json.

As a fallback we could consider to use `server_encoding`, but again using SQL_ASCII means wanting to read the raw data, so I'd expect that if someone is using this combination of configuration they would read the `::text` cast of the json field, in order to get a bytes string containing some jsonish value, and they would know themselves what to do with it. Or they would use a custom load function.

What is actually a more real problem is on the dumping side. What happens if there's a non-utf8 connection and someone dumps a non-ascii char? By default, everything works, because we use by default [`json.dumps()`](https://docs.python.org/3/library/json.html#json.dump) which uses by default `ensure_ascii=True`. But if someone uses a custom dump function that emits bytes in utf8 we would send corrupted data to the server. An `€` sent to a latin9 connection would be stored as `€`.

So I think we should fix the dumper too, I propose this way:

- if the dump function returns `str`, encode it in the client encoding (currently we encode utf8);
- if the dump function returns `bytes`:
  - if the database is utf8 or SQL_ASCII, use it as it is;
  - otherwise, decode the utf8 and encode it in the client_encoding.

The latter is kinda expensive but it is what it is, and it only affects the non-utf8 databases, which are now no more relevant - and better slow and correct than fast and wrong.

Could you please add a test to verify the issue, and verify the fix in the three cases? I think we could use the following grid of parameters for the tests:
- encoding: connection in utf8, ascii, latin9
- dumper: the following two functions
  - one producing non-ascii str (e.g. `lambda obj: jsong.dumps(obj, ensure_ascii=False)`
  - one producing utf8 encoded bytes (e.g. lambda obj: jsong.dumps(obj, ensure_ascii=False).encode()`)
- type: json and jsonb of course.

view thread (10+ messages)  latest in thread

Message-ID: <gh-psycopg-psycopg-1505-c7023964406@github.com>
Permalink:  ../gh-psycopg-psycopg-1505-c7023964406@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox