Subject: Re: Problem with error messages
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
 pgsql-translators@postgresql.org
References: 
 <CAHbMG0U0E_vOEwOwChrZ4Nq06xx5W7hdevwABo9uMjQ6urkZ3g@mail.gmail.com>
 <337ce77f-4159-0f43-cdb7-fcac30e3e219@2ndquadrant.com>
Cc: Walter Willmertinger <willmis@gmail.com>
From: Alexander Law <exclusion@gmail.com>
Message-ID: <578F1B63.5000301@gmail.com>
Date: Wed, 20 Jul 2016 09:34:11 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.8.0
MIME-Version: 1.0
In-Reply-To: <337ce77f-4159-0f43-cdb7-fcac30e3e219@2ndquadrant.com>
Content-Type: multipart/alternative;
 boundary="------------060702090500010008030608"
Precedence: bulk
Sender: pgsql-translators-owner@postgresql.org

This is a multi-part message in MIME format.
--------------060702090500010008030608
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hello,

I think, we had two issues here:
1. We have UTF-encoded translation to some language, which could contain 
extended (non-letter) characters such as », — , …
and we have localized clients expecting to read the server messages in 
their language using non-UTF-encoding.
I think we shouldn't leave them with the untranslated messages, instead 
we just should TRANSLIT option of iconv.
For example:

>echo 'HINWEIS: Tabelle »mickeymouse« existiert nicht, wird übersprungen' | iconv -f UTF-8 -t ASCII
HINWEIS: Tabelle iconv: illegal input sequence at position 17

But

>echo 'HINWEIS: Tabelle »mickeymouse« existiert nicht, wird übersprungen' | iconv -f UTF-8 -t ASCII//TRANSLIT
HINWEIS: Tabelle >>mickeymouse<< existiert nicht, wird ubersprungen

Is much better and still better than to show untranslated message with 
the notice about failed conversion.

2. Language of the localization could be not compatible with the client 
encoding at all. For example, we can't convert Russian to LATIN1 with 
TRANSLIT. In that case I would just print the original error message.

Best regards,
Alexander


20.07.2016 03:58, Peter Eisentraut пишет:
> I finally had a moment to think this through.
>
> The problem is that if an error message is not representable in the
> client encoding, all you get as a client are encoding errors.  This can
> happen in any translation (or untranslated, except those generally only
> have ASCII characters).
>
> In this particular case, the German translated error messages contain
> Unicode characters not in LATIN1/LATIN9.  This might have been a bad
> choice in retrospect, and can be fixed.  But this same thing might also
> happen if you happen to connect to a, say, a database with locale
> ru_RU.utf8 while your client encoding is LATIN9.  You will not be able
> to get any error message other than an encoding error.  (Ironically, the
> backend will first try to send the encoding error in translated form,
> which will again fail, and finally it will send it in English.)
>
> I think the recovery path should be changed so that it sends the
> original error message in untranslated form, possibly preceded by a
> notice that encoding conversion failed.
>
> Comments?
>
>
> On 6/30/15 9:31 AM, Walter Willmertinger wrote:
>> We already wrote in pgsql-bugs (#11550), but there it was recommended to
>> transfer this topic to the translator team.
>>
>> Now the problem:
>> If we set client_encoding to Latin9 (as we are here in Germany), we get
>> as nearly every error message from PostgreSQL: character with byte
>> sequence 0xe2 0x80 0x9e in encoding UTF8 has no
>> equivalent in LATIN9
>>
>> Why:
>> In 9.x we have new characters for delimiting words.
>>
>> An example:
>> "Drop table if exists mickeymouse;"
>> delivers in PG-8.4
>>
>> HINWEIS: Tabelle „mickeymouse“ existiert nicht, wird übersprungen
>>
>> but delivers in PG-9.3
>>
>> HINWEIS: Tabelle »mickeymouse« existiert nicht, wird übersprungen
>>
>> If we set client_encoding to Latin9 (as we are here in Germany), we get
>> an error message from PostgreSQL:
>> character with byte sequence 0xe2 0x80 0x9e in encoding UTF8 has no
>> equivalent in LATIN9
>>
>> but we do not see the real error message: "Table ... does not exist"
>>
>> So a proposal: Please change these delimiters to something which can be
>> compiled to any foreign language, as "  or similar
>>
>> Regards
>> Walter
>>
>>
>> -- 
>>
>> Viele Grüße
>>
>> Walter Willmertinger
>>
>


--------------060702090500010008030608
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">Hello,<br>
      <br>
      I think, we had two issues here:<br>
      1. We have UTF-encoded translation to some language, which could
      contain extended (non-letter) characters such as », — ,
      <meta http-equiv="content-type" content="text/html; charset=utf-8">
      … <br>
      <title></title>
      <meta name="generator" content="LibreOffice 5.1.4.2 (Linux)">
      <style type="text/css">
		@page { margin: 2cm }
		p { margin-bottom: 0.25cm; line-height: 120% }
		a:link { so-language: zxx }
	</style>and we have localized clients expecting to read the server
      messages in their language using non-UTF-encoding.<br>
      I think we shouldn't leave them with the untranslated messages,
      instead we just should TRANSLIT option of iconv.<br>
      For example:<br>
      <pre>&gt;echo 'HINWEIS: Tabelle »mickeymouse« existiert nicht, wird übersprungen' | iconv -f UTF-8 -t ASCII
HINWEIS: Tabelle iconv: illegal input sequence at position 17</pre>
      But <br>
      <pre>&gt;echo 'HINWEIS: Tabelle »mickeymouse« existiert nicht, wird übersprungen' | iconv -f UTF-8 -t ASCII//TRANSLIT
HINWEIS: Tabelle &gt;&gt;mickeymouse&lt;&lt; existiert nicht, wird ubersprungen</pre>
      Is much better and still better than to show untranslated message
      with the notice about failed conversion.<br>
      <br>
      2. Language of the localization could be not compatible with the
      client encoding at all. For example, we can't convert Russian to
      LATIN1 with TRANSLIT. In that case I would just print the original
      error message.<br>
      <br>
      Best regards,<br>
      Alexander<br>
      <br>
      <br>
      20.07.2016 03:58, Peter Eisentraut пишет:<br>
    </div>
    <blockquote
      cite="mid:337ce77f-4159-0f43-cdb7-fcac30e3e219@2ndquadrant.com"
      type="cite">
      <pre wrap="">I finally had a moment to think this through.

The problem is that if an error message is not representable in the
client encoding, all you get as a client are encoding errors.  This can
happen in any translation (or untranslated, except those generally only
have ASCII characters).

In this particular case, the German translated error messages contain
Unicode characters not in LATIN1/LATIN9.  This might have been a bad
choice in retrospect, and can be fixed.  But this same thing might also
happen if you happen to connect to a, say, a database with locale
ru_RU.utf8 while your client encoding is LATIN9.  You will not be able
to get any error message other than an encoding error.  (Ironically, the
backend will first try to send the encoding error in translated form,
which will again fail, and finally it will send it in English.)

I think the recovery path should be changed so that it sends the
original error message in untranslated form, possibly preceded by a
notice that encoding conversion failed.

Comments?


On 6/30/15 9:31 AM, Walter Willmertinger wrote:
</pre>
      <blockquote type="cite">
        <pre wrap="">We already wrote in pgsql-bugs (#11550), but there it was recommended to
transfer this topic to the translator team.

Now the problem:
If we set client_encoding to Latin9 (as we are here in Germany), we get
as nearly every error message from PostgreSQL: character with byte
sequence 0xe2 0x80 0x9e in encoding UTF8 has no
equivalent in LATIN9

Why:
In 9.x we have new characters for delimiting words.

An example:
"Drop table if exists mickeymouse;"
delivers in PG-8.4 

HINWEIS: Tabelle „mickeymouse“ existiert nicht, wird übersprungen

but delivers in PG-9.3

HINWEIS: Tabelle »mickeymouse« existiert nicht, wird übersprungen

If we set client_encoding to Latin9 (as we are here in Germany), we get
an error message from PostgreSQL:
character with byte sequence 0xe2 0x80 0x9e in encoding UTF8 has no
equivalent in LATIN9

but we do not see the real error message: "Table ... does not exist"

So a proposal: Please change these delimiters to something which can be
compiled to any foreign language, as "  or similar

Regards
Walter


-- 

Viele Grüße

Walter Willmertinger

</pre>
      </blockquote>
      <pre wrap="">

</pre>
    </blockquote>
    <br>
  </body>
</html>

--------------060702090500010008030608--