Subject: Re: Why don't we support external input/output functions for the composite types

From: Robert Haas <[email-masked]>
Date: Thu, 25 Apr 2024 08:50:38 -0400

This thread caught my eye this morning, and I'm confused.

On Thu, Apr 25, 2024 at 12:44 AM Tom Lane <[email-masked]> wrote:
> The primary outcome would be to require a huge amount of new work
> to be done by a lot of software, much of it not under our control.

What, what is the "lot of software" that would have to be changed? It
can't be existing extensions, because they wouldn't be forced into
using this feature. Are you thinking that drivers or admin tools would
need to support it? To me it seems like the only required changes
would be to things that know how to parse the output of record_out(),
and there is probably some of that, but the language you're using here
is so emphatic as to make me suspect that you anticipate some larger
impact.

> And the impact wouldn't only be to software that would prefer not
> to know about this.  For example, how likely do you think it is
> that these hypothetical user-defined I/O functions would cope
> well with ALTER TABLE/ALTER TYPE commands that change those
> rowtypes?

Hmm. Dilip mentioned changing the storage format, but if you do that,
you have bigger problems, like my_record_type.x no longer working. At
that point I don't see why what you have is properly called a record
type at all. So I guess what you're imagining here is that ALTER TABLE
.. ALTER TYPE would try COERCION_PATH_COERCEVIAIO, but, uh so what? We
could probably fix it so that such coercions were handled in some
other way, but even if we didn't, it just means the user has to
provide a USING clause, which is no different than what happens in any
other case where coerce-via-I/O doesn't work out.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

---

From: Tom Lane <[email-masked]>
Date: Thu, 25 Apr 2024 12:34:19 -0400

> What, what is the "lot of software" that would have to be changed?

I think this potentially affects stuff as low-level as drivers,
particularly those that deal in binary not text I/O.  It's not
unreasonable for client code to assume that any type with
typtype 'c' (composite) will adhere to the specifications at
https://www.postgresql.org/docs/current/rowtypes.html#ROWTYPES-IO-SYNTAX
especially because that section pretty much says in so many words
that that's the case.

> It
> need to support it?

Yes.  We've heard that argument about "this only affects extensions
that choose to use it" before, and it's nonsense.  As soon as you
extend system-wide APIs, the consequences are system-wide: everybody
has to cope with the new definition.

>> For example, how likely do you think it is

> type at all.

Yup, I agree.

> So I guess what you're imagining here is that ALTER TABLE
> .. ALTER TYPE would try COERCION_PATH_COERCEVIAIO, but, uh so what?

Uh, no.  My point is that if you make a custom output function
for "type complex (real float8, imaginary float8)", that function
will probably crash pretty hard if what it's handed is something
other than two float8s.  But there is nothing to stop somebody
from trying to ALTER the type to be two numerics or whatever.
Conversely, the type's custom input function would likely keep on
producing two float8s, yielding corrupt data so far as the rest
of the system is concerned.

You could imagine preventing such trouble by forbidding ALTER TYPE
on types with custom I/O functions.  But that just makes it even
more obvious that what this is is a poorly-thought-through hack,
rather than a feature that interoperates well with the rest of
Postgres.

I think that to the extent that there's a need for custom I/O
of something like this, it should be handled by bespoke types,
similar to (say) type point.

			regards, tom lane

---

From: Robert Haas <[email-masked]>
Date: Thu, 25 Apr 2024 13:12:39 -0400

> has to cope with the new definition.

Sure. Any new feature has this problem to some extent.

> of the system is concerned.

I'm not sure I really buy this. Changing the column definitions
amounts to changing the on-disk format, and no data type can survive a
change to the on-disk format without updating the I/O functions to
match.

> similar to (say) type point.

I somewhat agree with this. The main disadvantage of that approach is
that you lose the ability to directly refer to the members, which in
some cases would be quite nice. I bet a lot of people would enjoy
being able to write my_point.x and my_point.y instead of my_point[0]
and my_point[1], for example. But maybe the solution to that is not
$SUBJECT.

A related problem is that, even if my_point behaved like a composite
type, you'd have to write (my_point).x and (my_point).y to avoid
something like:

ERROR:  missing FROM-clause entry for table "my_point"

I think it's confusing and counterintuitive that putting parentheses
around a subexpression completely changes the meaning. I don't know of
any other programming language that behaves that way, and I find the
way the "indirection" productions are coded in gram.y to be highly
questionable. I suspect everything we currently treat as an
indirection_el should instead be a way of constructing a new a_expr or
c_expr or something like that, but I strongly suspect if I try to make
the work I'll discover horrible problems I can't fix. Still, it's
awful.

---

From: Tom Lane <[email-masked]>
Date: Thu, 25 Apr 2024 17:05:32 -0400

Robert Haas <[email-masked]> writes:
>> other than two float8s.

> change to the on-disk format without updating the I/O functions to

What I'm trying to say is: given that the command "alter type T alter
attribute A type foo" exists, users would reasonably expect the system
to honor that on its own for any composite type, because that's what
it does today.  But it can't if T has custom I/O functions, at least
not without understanding how to rewrite those functions.

> and my_point[1], for example. But maybe the solution to that is not

Nope, it isn't IMO.  We already added infrastructure to allow
arbitrary custom types to define subscripting operations.  I think a
case could be made to allow them to define field selection, too.

> any other programming language that behaves that way,

I take it that you also don't believe that "2 + 3 * 4" should yield
different results from "(2 + 3) * 4"?

I could get behind offering an alternative notation, eg "a.b->c does
the same thing as (a.b).c", if we could find a reasonable notation
that doesn't infringe on user operator namespace.  I think that might
be hard to do though, and I don't think the existing notation is so
awful than we should break existing operators to have an alternative.
The business with deprecating => operators a few years ago had the
excuse that "the standard says so", but we don't have that
justification here.

			regards, tom lane

---

From: Robert Haas <[email-masked]>
Date: Fri, 26 Apr 2024 10:34:30 -0400

> not without understanding how to rewrite those functions.

I understand your point, but I don't agree with it. Ordinary users
wouldn't be able to create types like this anyway, because there's no
way we can allow an unprivileged user to set an input or output
function. It would have to be restricted to superusers, just as we do
for base types. And IMHO those have basically the same issue: you have
to ensure that all the functions and operators that operate on the
type, and any subscripting operations, are on the same page about what
the underlying storage is. This doesn't seem different. It may well
still be a bad idea for other reasons, or just kind of useless, but I
disagree that it's a bad idea for that particular reason.

> case could be made to allow them to define field selection, too.

That would be cool!

> different results from "(2 + 3) * 4"?

Isaac's rebuttal to this particular point was perfect; I have nothing to add.

---

From: Tom Lane <[email-masked]>
Date: Fri, 26 Apr 2024 11:55:57 -0400

> for base types.

Well, that would be one way of making the consistency problem be not
our problem, but it would be a sad restriction.  It'd void a lot of
the arguable use-case for this feature, if you ask me.  I realize
that non-superusers couldn't create the C-language I/O functions that
would be most at risk here, but you could imagine people building
I/O functions in some other PL.

(We'd have to remove the restriction that cstring isn't an allowed
input or return type for user-defined functions; but AFAIK that's
just a holdover from days when cstring was a lot more magic than
it is now.)

Maybe there's an argument that PL functions already have to be
proof enough against datatype inconsistencies that nothing really
awful could happen.  Not sure.

In any case, if we have to put strange restrictions on a composite
type when it has custom I/O functions, then that still is an
indication that the feature is a hack that doesn't play nice
with the rest of the system.  So I remain of the opinion that
we shouldn't go there.  If field selection support for custom
types will solve the use-case, I find that a lot more attractive.

> Isaac's rebuttal to this particular point was perfect; I have nothing to add.

As far as I could tell, Isaac's rebuttal was completely off-point.

			regards, tom lane