DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com;
	h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
	b=IujbsgRS4ZlnKoOJNUn4JyWS/pkA2bG8VHIt9C3gSdZOpg64KH8pqMC4/ruBEbpuEIOddRF/fnl11AexP7qg21Jezc8MSJIeib0Z8k6OafXJCeM29XzCGXN8tZ9H3k6Shz2vMwKkZNTy+wN8qUn107kXMOx9lGnZXGLNnOD8Ib8=
Message-ID: <e692861c0512081454u560e6cc2h41f09293d4e5f2d1@mail.gmail.com>
Date: Thu, 8 Dec 2005 17:54:35 -0500
From: Gregory Maxwell <gmaxwell@gmail.com>
To: Bruce Momjian <pgman@candle.pha.pa.us>
Subject: Re: Upcoming PG re-releases
Cc: Gavin Sherry <swm@linuxworld.com.au>, Peter Eisentraut <peter_e@gmx.net>,
	pgsql-hackers@postgresql.org
In-Reply-To: <200512082244.jB8MiYT02161@candle.pha.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
References: <Pine.LNX.4.58.0512090932030.12041@linuxworld.com.au>
	<200512082244.jB8MiYT02161@candle.pha.pa.us>

On 12/8/05, Bruce Momjian <pgman@candle.pha.pa.us> wrote:
> >       A script which identifies non-utf-8 characters and provides some
> >       context, line numbers, etc, will greatly speed up the process of
> >       remedying the situation.
>
> I think the best we can do is the "iconv -c with the diff" idea, which
> is already in the release notes.  I suppose we could merge the iconv and
> diff into a single command, but I don't see a portable way to output the
> iconv output to stdout., /dev/stdin not being portable.

No, what is needed for people who care about fixing their data is a
loadable strip_invalid_utf8() that works in older versions.. then just
select * from bar where foo !=3D strip_invalid_utf8(foo);  The function
would be useful in general, for example, if you have an application
which doesn't already have much utf8 logic, you want to use a text
field, and stripping is the behaviour you want. For example, lots of
simple web applications.