Date: Sun, 12 May 2019 01:13:47 -0700
From: Noah Misch <noah@leadboat.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andrew Dunstan <andrew.dunstan@2ndquadrant.com>,
	PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Subject: Re: Augment every test postgresql.conf
Message-ID: <20190512081347.GA1173385@rfd.leadboat.com>
References: <20181229021950.GA3302966@rfd.leadboat.com>
 <20181230024014.GA181668@gust.leadboat.com>
 <16623.1546141591@sss.pgh.pa.us>
 <20181230055346.GB156585@gust.leadboat.com>
 <f8938ff6-4dc8-2beb-636b-1d524db78e2f@2ndQuadrant.com>
 <20190407064156.GC1914190@rfd.leadboat.com>
 <CAA8=A786VMBGfUbBswhP6oP_6YpNy+2KrcP1rB-Kj1yX05_H5Q@mail.gmail.com>
 <20190512015615.GD1124997@rfd.leadboat.com>
 <25383.1557629039@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <25383.1557629039@sss.pgh.pa.us>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk

On Sat, May 11, 2019 at 10:43:59PM -0400, Tom Lane wrote:
> Noah Misch <noah@leadboat.com> writes:
> > Pushed.  This broke 010_dump_connstr.pl on bowerbird, introducing 'invalid
> > byte sequence for encoding "UTF8"' errors.  That's because log_connections
> > renders this 010_dump_connstr.pl solution insufficient:
> 
> Ugh.
> 
> > 4. If GetMessageEncoding()==PG_SQL_ASCII, make pgwin32_message_to_UTF16()
> >    return NULL.  The caller will always send untranslated bytes to write() or
> >    ReportEventA().  This seems consistent with the SQL_ASCII concept and with
> >    pg_do_encoding_conversion()'s interpretation of SQL_ASCII.
> 
> > 5. When including a datname or rolname value in a message, hex-escape
> >    non-ASCII bytes.  They are byte sequences, not text of known encoding.
> >    This preserves the most information, but it's overkill and ugly in the
> >    probably-common case of one encoding across all databases of a cluster.
> 
> > I'm inclined to do (1) in back branches and (4) in HEAD only.  (If starting
> > fresh today, I would store the encoding of each rolname and dbname or just use
> > UTF8 for those particular fields.)  Other preferences?
> 
> I agree that (4) is a fairly reasonable thing to do, and wouldn't mind
> back-patching that.

Okay.  Absent objections, I'll just do it that way.

> Taking a wider view, this seems closely related
> to something I've been thinking about in connection with the recent
> pg_stat_activity contretemps: that mechanism is also shoving strings
> across database boundaries without a lot of worry about encodings.
> Maybe we should try to develop a common solution.
> 
> One difference from the datname/rolname situation is that for
> pg_stat_activity we can know the source encoding --- we aren't storing
> it now, but we easily could.  If we're thinking of a future solution
> only, adding a "name encoding" field to relevant shared catalogs makes
> sense perhaps.  Alternatively, requiring names in shared catalogs to be
> UTF8 might be a reasonable answer too.
> 
> In all these cases, throwing an error when we can't translate a character
> into the destination encoding is not very pleasant.  For pg_stat_activity,
> I was imagining that translating such characters to '?' might be the best
> answer.  I don't know if we can get away with that for the datname/rolname
> case --- at the very least, it opens problems with apparent duplication of
> names that should be unique.  I don't much like your hex-encoding answer,
> though; that has its own uniqueness-violation hazards, plus it's ugly.

Another case of byte sequence masquerading as text is pg_settings.setting.

In most contexts, it's important to convey exact values.  Error messages can
use '?'.  I wouldn't let dump/reload of a rolname corrupt it that way, and I
wouldn't recognize the '?' version for authentication.  While
pg_stat_activity.query could use '?', I'd encourage adding bytea and encoding
columns for exact transmission.  pg_stat_activity can't standardize on UTF8
without shrinking the set of valid queries or inaccurately reporting some,
neither of which is attractive.

datname/rolname could afford to be more prescriptive, since non-ASCII names
are full of bugs today.  A useful consequence of UTF8 datname/rolname would be
today's "pg_dumpall --globals" remaining simple.  If we were to support
arbitrary encodings with a "name encoding" field, the general-case equivalent
of "pg_dumpall --globals" would connect to several databases of different
encodings in order to dump all objects, perhaps even creating a temporary
database if no suitable-encoding database existed.

MULE_INTERNAL presents trouble since we don't have a UTF8<->MULE_INTERNAL
conversion.  If we standardized cross-database strings on UTF8, it would be
impossible to read such strings, create roles, etc. from a MULE_INTERNAL
database.  I suppose we'd either add the conversion or deprecate
MULE_INTERNAL, forbidding its use as the initdb encoding.