Received: from maia.hub.org (unknown [200.46.208.211])
	by mail.postgresql.org (Postfix) with ESMTP id 6257363380D
	for <pgsql-admin-postgresql.org@mail.postgresql.org>;
	Tue, 23 Mar 2010 05:12:39 -0300 (ADT)
Received: from mail.postgresql.org ([200.46.204.86])
	by maia.hub.org (mx1.hub.org [200.46.208.211]) (amavisd-maia,
	port 10024) with ESMTP id 33700-04 for
	<pgsql-admin-postgresql.org@mail.postgresql.org>;
	Tue, 23 Mar 2010 08:12:13 +0000 (UTC)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
Received: from hermes.vectorsf.com (firewall.vectorsf.com [62.82.80.62])
	by mail.postgresql.org (Postfix) with ESMTP id C8AC56337DB
	for <pgsql-admin@postgresql.org>; Tue, 23 Mar 2010 05:12:06 -0300 (ADT)
Received: from localhost (localhost.localdomain [127.0.0.1])
	by hermes.vectorsf.com (Postfix) with ESMTP id E1AF011C003C;
	Tue, 23 Mar 2010 09:12:03 +0100 (CET)
X-Virus-Scanned: amavisd-new at hermes.vectorsf.com
Received: from hermes.vectorsf.com ([127.0.0.1])
	by localhost (hermes.vectorsf.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id scCOTnfCn7uJ; Tue, 23 Mar 2010 09:11:58 +0100 (CET)
Received: from [10.70.4.200] (unknown [10.70.4.200])
	(Authenticated sender: imartinez)
	by hermes.vectorsf.com (Postfix) with ESMTP id 3C42D11C0015;
	Tue, 23 Mar 2010 09:11:58 +0100 (CET)
Subject: Re: Bad encoded chars in being inserted into database
From: =?ISO-8859-1?Q?I=F1igo?= Martinez Lasala <imartinez@vectorsf.com>
To: Scott Marlowe <scott.marlowe@gmail.com>
Cc: pgsql-admin@postgresql.org
In-Reply-To: <dcc563d11003221505w7a12d704j3d0ce0b50a5f83c0@mail.gmail.com>
References: <AcrJnH1ewzMtxpQ8RNe8stLAPzTRcg==>
	<050f01cac99c$7dafe100$790fa300$@com>
	<dcc563d11003221505w7a12d704j3d0ce0b50a5f83c0@mail.gmail.com>
Content-Type: multipart/alternative; boundary="=-1MNaoAgT4Wz27SRT5hZf"
Date: Tue, 23 Mar 2010 09:11:58 +0100
Message-ID: <1269331918.8928.12.camel@deimos>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.1 
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Spam-Status: No, hits=-2.117 tagged_above=-10 required=5 tests=AWL=0.481,
	BAYES_00=-2.599, HTML_MESSAGE=0.001
X-Spam-Level: 
X-Archive-Number: 201003/162
X-Sequence-Number: 33961


--=-1MNaoAgT4Wz27SRT5hZf
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit

We are working with 8.1 and migrating to 8.4....
We will see if after migration this behavior has disappeared. ;-) 

Thank you, Scott.

-----Original Message-----
From: Scott Marlowe <scott.marlowe@gmail.com>
To: Iñigo Martinez Lasala <imartinez@vectorsf.com>
Cc: pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Bad encoded chars in being inserted into database
Date: Mon, 22 Mar 2010 16:05:44 -0600


On Mon, Mar 22, 2010 at 2:48 AM, Iñigo Martinez Lasala
<imartinez@vectorsf.com> wrote:
> Hi everybody.
>
>
>
> I have a doubt about how postgres deal with bad encoded characters into
> database.
>
> We have several gforge application. They are using postgres as database.
>
> If we export a database and import again, we have to deal with several bad
> encoded chars. These bad chars always come from copy & paste emails from
> Lotus Notes mail client. OK, I understand the Notes client people is using
> is an ancient application and does not deal very well with some Unicode
> chars…
>
> What I cannot understand is why postgres accept these bad enconded
> characters into database, exports them without problema but does not allow
> them when importing again.
>
> This has been happening since postgers 7.3. However, until 7.4.XX (y don’t
> remember what minor version) you could import database without ERRORs.
> However, since 7.4.XX it’s impossible and it’s imperative to clean bad
> characters (using iconv, for example) prior importing tables.

This is because postgresql's support for UTF-8 encoding (and all
encoding really) has gotten tighter over time, so that the filter to
catch improperly encoded UTF has gotten better with each major
release.



--=-1MNaoAgT4Wz27SRT5hZf
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
  <META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
  <META NAME="GENERATOR" CONTENT="GtkHTML/3.28.1">
</HEAD>
<BODY>
We are working with 8.1 and migrating to 8.4....<BR>
We will see if after migration this behavior has disappeared. ;-) <BR>
<BR>
Thank you, Scott.<BR>
<BR>
-----Original Message-----<BR>
<B>From</B>: Scott Marlowe &lt;<A HREF="mailto:Scott%20Marlowe%20%3cscott.marlowe@gmail.com%3e">scott.marlowe@gmail.com</A>&gt;<BR>
<B>To</B>: I&#241;igo Martinez Lasala &lt;<A HREF="mailto:%3d%3fISO-8859-1%3fQ%3fI%3dF1igo%3f%3d%20Martinez%20Lasala%20%3cimartinez@vectorsf.com%3e">imartinez@vectorsf.com</A>&gt;<BR>
<B>Cc</B>: <A HREF="mailto:pgsql-admin@postgresql.org">pgsql-admin@postgresql.org</A><BR>
<B>Subject</B>: Re: [ADMIN] Bad encoded chars in being inserted into database<BR>
<B>Date</B>: Mon, 22 Mar 2010 16:05:44 -0600<BR>
<BR>
<PRE>
On Mon, Mar 22, 2010 at 2:48 AM, I&#241;igo Martinez Lasala
&lt;<A HREF="mailto:imartinez@vectorsf.com">imartinez@vectorsf.com</A>&gt; wrote:
&gt; Hi everybody.
&gt;
&gt;
&gt;
&gt; I have a doubt about how postgres deal with bad encoded characters into
&gt; database.
&gt;
&gt; We have several gforge application. They are using postgres as database.
&gt;
&gt; If we export a database and import again, we have to deal with several bad
&gt; encoded chars. These bad chars always come from copy &amp; paste emails from
&gt; Lotus Notes mail client. OK, I understand the Notes client people is using
&gt; is an ancient application and does not deal very well with some Unicode
&gt; chars&#8230;
&gt;
&gt; What I cannot understand is why postgres accept these bad enconded
&gt; characters into database, exports them without problema but does not allow
&gt; them when importing again.
&gt;
&gt; This has been happening since postgers 7.3. However, until 7.4.XX (y don&#8217;t
&gt; remember what minor version) you could import database without ERRORs.
&gt; However, since 7.4.XX it&#8217;s impossible and it&#8217;s imperative to clean bad
&gt; characters (using iconv, for example) prior importing tables.

This is because postgresql's support for UTF-8 encoding (and all
encoding really) has gotten tighter over time, so that the filter to
catch improperly encoded UTF has gotten better with each major
release.
</PRE>
<BR>
</BODY>
</HTML>

--=-1MNaoAgT4Wz27SRT5hZf--