X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by postgresql.org (Postfix) with ESMTP id 307149DCB8C for ; Sun, 4 Dec 2005 12:41:03 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 12357-01 for ; Sun, 4 Dec 2005 12:40:59 -0400 (AST) X-Greylist: from auto-whitelisted by SQLgrey- Received: from mirth.inuus.com (h-68-166-39-71.snvacaid.covad.net [68.166.39.71]) by postgresql.org (Postfix) with ESMTP id 683659DCABC for ; Sun, 4 Dec 2005 12:40:59 -0400 (AST) Received: from mirth.inuus.com (localhost.localdomain [127.0.0.1]) by mirth.inuus.com (8.13.4/8.13.4) with ESMTP id jB4GesJU005509 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 4 Dec 2005 08:40:54 -0800 Received: (from lindner@localhost) by mirth.inuus.com (8.13.4/8.13.4/Submit) id jB4Gess9005508; Sun, 4 Dec 2005 08:40:54 -0800 X-Authentication-Warning: mirth.inuus.com: lindner set sender to lindner@inuus.com using -f Date: Sun, 4 Dec 2005 08:40:54 -0800 From: Paul Lindner To: Tom Lane Cc: Paul Lindner , Bruce Momjian , Neil Conway , pgsql-hackers@postgresql.org Subject: Re: Upcoming PG re-releases Message-ID: <20051204164054.GE10317@inuus.com> References: <1133625371.9297.3.camel@localhost.localdomain> <200512031554.jB3Fs8h10927@candle.pha.pa.us> <20051204162520.GD10317@inuus.com> <8284.1133714056@sss.pgh.pa.us> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0H629O+sVkh21xTi" Content-Disposition: inline In-Reply-To: <8284.1133714056@sss.pgh.pa.us> User-Agent: Mutt/1.4.2.1i X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, score=0 required=5 tests=[none] X-Spam-Score: 0 X-Spam-Level: X-Archive-Number: 200512/183 X-Sequence-Number: 77031 --0H629O+sVkh21xTi Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Dec 04, 2005 at 11:34:16AM -0500, Tom Lane wrote: > Paul Lindner writes: > > To convert your pre-8.1 database to 8.1 you may have to remove and/or > > fix the offending characters. One simple way to fix the problem is to > > run your pg_dump output through the iconv command like this: >=20 > > iconv -c -f UTF8 -t UTF8 -o fixed.sql dump.sql >=20 > Is that really a one-size-fits-all solution? Especially with -c? >=20 I'd say yes, and the -c flag is needed so iconv strips out the invalid characters. =20 This technique worked for some smaller databases I converted and croaked with out-of-memory on the larger ones. It certainly doesn't make the problem worse. If one wanted to fix this in the general case one could duplicate the iconv behavior in the Postgres code via some kind of special flag/setting that is only used for imports.. set strip_bad_utf8 =3D on --=20 Paul Lindner ||||| | | | | | | | | | lindner@inuus.com --0H629O+sVkh21xTi Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDkxwW+FkoUnP1gFQRAq8OAKCnDyI+Z45fFEsZUoXTH++q41ESAACePtPV krnEcJGLyR7kD7rkqUjwrks= =VA7E -----END PGP SIGNATURE----- --0H629O+sVkh21xTi--