Date: Fri, 16 Feb 2024 11:47:29 +0100
From: Karsten Hilbert <Karsten.Hilbert@gmx.net>
To: psycopg@lists.postgresql.org,
	"psycopg@postgresql.org" <psycopg@postgresql.org>
Subject: Re: Reporting UnicodeEncodeError info on arbitrary data sent to PG
 with psycopg3
Message-ID: <Zc89QYhCp0fM6GQB@hermes.hilbert.loc>
Mail-Followup-To: psycopg@lists.postgresql.org,
	"psycopg@postgresql.org" <psycopg@postgresql.org>
References: <20240213193732.28cb8329@slate.karlpinc.com>
 <CA+mi_8Y0jzEZA+3kCnTtWCS8cvWTUFcD=0J+ihiamh=y8GvOxg@mail.gmail.com>
 <20240214094203.52d7e22d@slate.karlpinc.com>
 <CA+mi_8bKxi286PjqYrt9Epcd2dxL+vFtMDJuyurV2edsOEMqJg@mail.gmail.com>
 <20240215234515.0ab71d58@slate.karlpinc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240215234515.0ab71d58@slate.karlpinc.com>
Ma_X_il-Followup-to: d
Re_X_turn-receipt-to: Karsten.Hilbert@gmx.net
Di_X_sposition-Notification-To: Karsten.Hilbert@gmx.net
X-Confi_X_rm-Reading-To: Karsten.Hilbert@gmx.net
X-Pri_X_ority: 2 (High)
Sender: <karsten.hilbert@gmx.net>
X-Provags-ID: V03:K1:+fH6Dck9OZD6XNYc8DshdTdOzixvaS965Fps513uvWTGYpluSaH
 fEtqF4xG1IjhcBZB+KeeiPnRKhdpQYluSP/R8T+uUYxZ2b/F4Ht9ozYNvobSjHi5di5hd2t
 +oXmGGcBPmaW7tqlXGPTtw/3/C8GlTMrm4Vp8227j/bz4wdeO/u6T3bh7/YylWHbDj9C+sc
 an8x1PyKCN8StJ3px8/bw==
X-Spam-Flag: NO
UI-OutboundReport: notjunk:1;M01:P0:UfI2M5/CsGI=;iGubA24KhsunLN2rCWnd1JNhKaC
 gGC3SCXFW4Vlw+cWcoVCudYGQ3cxKlvd33FojHBW3MC5C1cd11IdQfisJMEoNI2DDnqNaQv7i
 I0JJuDU0sJbn9iVwtM+HyJFnZeqQpat04qDkucbEkwQKpQz0aKuKSSWGTJtxPwTRSj0R29BnL
 zb2D3viFKaxLnL0ZdziSOLFXpofwnJCE+0sc06gYe7qV/mwd6h26vr0r0lHxQWwsLUPgXFfqH
 AaEKPE6yFqR/e/neiGuDyXOGLQC1Ekj1vVeikR6mlnWdNGSvYnG9crqCQZA0HFIHChc17Bk/Q
 SgiAUZilOJ/CZQiHKRbA6Zje5eQfrzfmIrLM6yS95qPgFtKuVgSZn13EmLBNWpDO1TB7DNUgb
 MHSKfPOXTXti6sIR8cJr/VuzAlHsKK0eHJOZTN3eIwVZ0BYJeq/nO9FJMYFCusBt4yOcwOwj9
 JzshPS75zuyUD3UDRaJ/0veH+ePGSlE2zmfo5fGUfHBUjuIuU+Zuuq+ijmbUiJMnBGkSMyUSY
 vjFQkaxFiDjzwj1wrt3y2hWzYdlaob2XyeHVLZI8qeYRfgK1SqN1wpafLx2QOCP0Ouf+BLZal
 uUMgHWoEYZUl0RyrN8UoztWfg7hHgumhpZ2ERrz7JrppVx3nG7GuK0OFaN5TJj3Zb/LYH6Qkn
 ZwUjmOFImz+7q9tJgfGvXNVnIPkmKWld8eFcJjAkG6liktXNI5by7tO+pfDbXcikbl923wwY0
 CmVQCQkBww6K0gsYNVmSp6aplBSpNOAsjsvGzb4EMSyP2n5OjZUiJWyAGr3xsZce1WHAIU0m9
 Fczon9IJO678qLCH/3jw1OKWMF0OGIY2RK/I7FdN3HZCw=
Content-Transfer-Encoding: quoted-printable
List-Id: <psycopg.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:psycopg@lists.postgresql.org>
List-Owner: <mailto:psycopg-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/psycopg>
Archived-At: 
 <https://www.postgresql.org/message-id/Zc89QYhCp0fM6GQB%40hermes.hilbert.loc>
Precedence: bulk

Am Thu, Feb 15, 2024 at 11:45:15PM -0600 schrieb Karl O. Pinc:

>   Today there is no substitute for knowing the encoding of the
> text your application obtains from the outside world.
> This can be highly system dependent because when reading
> files open()-ed as text, Python decodes (into UTF-8) the bytes read.

Not quite. Python assumes the bytes in the file *are* encoded
by whatever encoding is passed to open(), including, if so
UTF-8). It then decodes said bytes into *unicode code
points*. If we want them back as UTF-8 we need to encode them
as such.

> By default decoding from the system locale's character encoding.
> And when writing files open()-ed as text Python encodes (from UTF-8)

again, from unicode, that is:

	https://docs.python.org/3/howto/unicode.html

>   No matter how you get your data, to put your data into
> the database as text, its bytes must first have their external
> encoding decoded to UTF-8.  Because Python strings are
> UTF-8.

unicode code points, but, yeah

>   Once in Python, psycopg converts the UTF-8 text to the database

unicode

> It's important to get the encoding right so I think it'd be
> good to talk about it.

+1

Karsten
=2D-
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B