Subject: Re: Pluggable toaster

From: Aleksander Alekseev <[email-masked]>
Date: Mon, 6 Feb 2023 17:38:34 +0300

Hi,

> The main reason behind this decision is that keeping the first implementation
> on the side of the vanilla (I mean rebasing it) over time is very difficult due
> to the very invasive nature of this solution.
>
> So we decided to reduce changes in the core to the minimum necessary
> to make it available through the hooks, because the hooks part is very
> lightweight and simple to keep rebasing onto the vanilla core. We plan
> to keep this extension free with the PostgreSQL license, so any PostgreSQL
> user could benefit from the TOAST on steroids, and sometimes in the
> future it will be a much simpler task to integrate the Pluggable TOAST into
> the vanilla, along with our advanced TOAST implementations which
> we plan to keep under Open Source licenses too.

That's great to hear. I'm looking forward to the link to the
corresponding GitHub repository. Please let us know when this effort
will be available for testing and benchmarking!

I would like to point out however that there were several other pieces
of feedback that could have been missed:

* No one wants to see this as an extension. This was my original
proposal (adding ZSON to /contrib/) and it was rejected. The community
explicitly wants this to be a core feature with its syntax,
autocompletion, documentation and stuff.
* The community wants the feature to have a simple implementation. You
said yourself that the idea of type-aware TOASTers is very invasive,
and I completely agree.
* People also want this to be simple from the user perspective, as
simple as just CREATE COMPRESSED TABLE ... [USING lz4|zstd];

At least this is my personal summary/impression from following the mailing list.

Anyhow since we are back to the stage where we discuss the RFC I
suggest continuing it in the compression dictionaries thread [1] since
we made noticeable progress there already.

[1]: https://postgr.es/m/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22%3D5xVBg7S4vr5rQ%40mail.gmail.com

-- 
Best regards,
Aleksander Alekseev

---

From: Matthias van de Meent <[email-masked]>
Date: Mon, 6 Feb 2023 21:14:54 +0100

> autocompletion, documentation and stuff.

I believe that is a misrepresentation of the situation. ZSON had
(has?) several systemic issues and could not be accepted to /contrib/
in the way it was implemented, and it was commented that it would make
sense that the feature of compression assisted by dictionaries would
be implemented in core. Yet still, that feature is only closely
related to pluggable TOAST, but it is not the same as making TOAST
pluggable.

> said yourself that the idea of type-aware TOASTers is very invasive,

I'm not sure that this is correct either. Compression (and TOAST) is
inherently complex, and I don't think we should reject improvements
because they are complex.
The problem that I see being raised here, is that there was little
discussion and no observed community consensus about the design of
this complex feature *before* this patch with high complexity was
provided.
The next action that was requested is to take a step back and decide
how we would want to implement type-aware TOASTing (and the associated
patch compression dictionaries) *before* we look into the type-aware
toasting.

> simple as just CREATE COMPRESSED TABLE ... [USING lz4|zstd];

Could you provide a reference to this? I have yet to see a COMPRESSED
TABLE feature or syntax, let alone users asking for TOAST to be as
easily usable as that feature or syntax.

Kind regards,

Matthias van de Meent

---

From: Nikita Malakhov <[email-masked]>
Date: Mon, 6 Feb 2023 23:21:45 +0300

Hi,

It'll be great to see more opinions on the approach as a whole.

>patch compression dictionaries) *before* we look into the type-aware

We decided to put this improvement as a patch because we thought
that the most complex and questionable part would be the TOAST
implementations (the Toasters) itself, and the Pluggable TOAST is
just a tool to make plugging different TOAST implementations clean
and simple.

--
Nikita Malakhov
Postgres Professional
https://postgrespro.ru/

---

From: Aleksander Alekseev <[email-masked]>
Date: Tue, 7 Feb 2023 13:05:45 +0300

Hi,

> this complex feature *before* this patch with high complexity was

Strictly speaking there is no such thing as "community opinion". There
are different people, everyone has their own opinion. To make things
more interesting the opinions change with time.

I did my best to make a brief summary of 100+ messages from different
people in something like 4 threads. These are things that were
requested and/or no one disagrees with (at least no one said "no, put
all this out of the core! and make it complicated too!"). Focusing on
something (almost) no one disagrees with seems to be more productive
than arguing about something everyone disagrees with.

As I see it, the goal is not to be right, but rather to find a
consensus most of us will be not unhappy with.

> patch compression dictionaries) *before* we look into the type-aware

Yes, I thought we already agreed to forget about type-aware TOASTing
and compression dictionaries, and are looking for a consensus now.

To clarify, I don't think that pluggable TOASTing is an absolutely bad
idea. We are just not discussing this particular idea anymore, at
least for now.

> easily usable as that feature or syntax.

I was referring to the recent discussion of the new RFC. Please see
[1] and below.

[1]: https://www.postgresql.org/message-id/flat/20230203095540.zutul5vmsbmantbm%40alvherre.pgsql#7cce6acef0cb7eb2490715ec9d835e74

-- 
Best regards,
Aleksander Alekseev

---

From: Pavel Borisov <[email-masked]>
Date: Tue, 7 Feb 2023 14:38:20 +0400

Hi, hackers!

Maybe I've read the thread too superficially, but for me, it seems
like more of a discussion on what TOAST should NOT be. Maybe someone
more in the topic could explain what is the consensus on what we
require and what we like to to have in a new TOAST?

For me, a good toast should be chunk-updatable, so that we don't need
to rewrite the whole TOAST and WAL-replicate the whole thing at every
small attribute modification. But obviously, it's just another
opinion.

Kind regards,
Pavel Borisov

---

From: Nikita Malakhov <[email-masked]>
Date: Wed, 14 Jun 2023 13:05:24 +0300

Hi!

I have a question for the community - why was this patch silently put to a
"rejected" status?
Should there no be some kind of explanation?

During this discussion I got the impression that for some reason some
members of the community
do not want the TOAST functionality, which has such drawbacks that make it
really a curse for
in many ways very good DBMS, to be touched. We cannot get rid of it because
of backwards
compatibility, so the best way is to make it more adaptable and extensible
- this is what this thread
is about. We proposed our vision on how to extend the TOAST Postgres-way,
like Pluggable
Storage some time before.

There are some very complex subjects left in this topic that really need a
community' attention.
I've mentioned them above, but there was no feedback on them.

Pavel, we've already had an update implementation for TOAST. But it is a
part of a Pluggable
TOAST and it hardly would be here without it. I've started another thread
on extending the TOAST
pointer, maybe you would want to participate there [1].

We still would be grateful for feedback.

[1] Extending the TOAST Pointer
<https://www.postgresql.org/message-id/flat/CAJ7c6TNAYyeMYKVkiwOZChy7UpE_CkjpYOk73gcWTXMkLkEyzw%40mail.gmail.com#59aacdde27dd61277fe7c46c61c84b2c>

-- 
Regards,
Nikita Malakhov
Postgres Professional
https://postgrespro.ru/

---

From: Pavel Borisov <[email-masked]>
Date: Wed, 14 Jun 2023 14:21:05 +0400

On Wed, 14 Jun 2023 at 14:05, Nikita Malakhov <[email-masked]> wrote:
>
> Hi!
>
> I have a question for the community - why was this patch silently put to a "rejected" status?
> During this discussion I got the impression that for some reason some members of the community
> do not want the TOAST functionality, which has such drawbacks that make it really a curse for
> in many ways very good DBMS, to be touched. We cannot get rid of it because of backwards
> compatibility, so the best way is to make it more adaptable and extensible - this is what this thread
> is about. We proposed our vision on how to extend the TOAST Postgres-way, like Pluggable
> There are some very complex subjects left in this topic that really need a community' attention.
> Pavel, we've already had an update implementation for TOAST. But it is a part of a Pluggable
> TOAST and it hardly would be here without it. I've started another thread on extending the TOAST
I don't see a clear reason it's rejected, besides technically it's
Waiting on Author since January. If it's a mistake and the patch is
up-to-date you can set an appropriate status.

Regards,
Pavel Borisov,
Supabase.

---

From: Aleksander Alekseev <[email-masked]>
Date: Wed, 14 Jun 2023 13:56:34 +0300

Hi,

> > Should there no be some kind of explanation?

I wouldn't say that it happened "silently" nor that the reason is so
mysterious [1].

[1]: https://www.postgresql.org/message-id/flat/CAM-w4HPjg7NwEWBtXn1empgAg3fqJHifHo_nhgqFWopiYaNxYg%40mail.gmail.com

-- 
Best regards,
Aleksander Alekseev

---

From: Nikita Malakhov <[email-masked]>
Date: Fri, 16 Jun 2023 13:18:49 +0300

Hi,

Seems that I missed the thread mentioned above. I strongly disagree
with such statement, Pluggable TOAST could not be a part or Compression
Dictionaries thread because the TOAST improvement is a more general subject,
it involves much deeper and tricky changes in the core. And also is much
more
promising in terms of performance and storage improvements.

We already have a lot of changes in Pluggable TOAST that were not committed
to the main GIT branch of this thread, so it seems that I have to merge
them and
reopen it.

--
Regards,
Nikita Malakhov
Postgres Professional
The Russian Postgres Company
https://postgrespro.ru/

---

From: Aleksander Alekseev <[email-masked]>
Date: Fri, 16 Jun 2023 16:20:38 +0300

Hi Nikita,

> to the main GIT branch of this thread, so it seems that I have to merge them and
> reopen it.

Pretty sure that reopening an already rejected patch that is competing
with compression dictionaries (which the rest of us are currently
focusing on) will achieve anything. Consider joining the compression
dictionaries effort instead [1]. During the discussion with the
community it ended up being a TOAST improvement after all. So we could
use your expertise in this area.

[1]: https://commitfest.postgresql.org/43/3626/

-- 
Best regards,
Aleksander Alekseev

---

From: Aleksander Alekseev <[email-masked]>
Date: Fri, 16 Jun 2023 17:44:37 +0300

Hi,

> focusing on) will achieve anything.

Ooops, I didn't mean to be sarcastic:

s/will achieve/will not achieve/

My apologies.

> Consider joining the compression

-- 
Best regards,
Aleksander Alekseev