MIME-Version: 1.0
From: =?UTF-8?Q?Erik_Nordstr=C3=B6m?= <erik@timescale.com>
Date: Thu, 21 Nov 2024 17:08:49 +0100
Message-ID: <CACAa4VJ+QY4pY7M0ECq29uGkrOygikYtao1UG9yCDFosxaps9g@mail.gmail.com>
Subject: Changed behavior in rewriteheap
To: pgsql-hackers@lists.postgresql.org
Content-Type: multipart/alternative; boundary="000000000000fdf91706276e7b32"
Archived-At: <https://www.postgresql.org/message-id/CACAa4VJ%2BQY4pY7M0ECq29uGkrOygikYtao1UG9yCDFosxaps9g%40mail.gmail.com>
Precedence: bulk

--000000000000fdf91706276e7b32
Content-Type: text/plain; charset="UTF-8"

Hello,

I've noticed a change in behavior of the heap rewrite functionality in
PostgreSQL 17, used by, e.g., CLUSTER. I've been experimenting with the
functionality to implement a way to merge partitions in TimescaleDB. I am
using table_relation_copy_for_cluster() to write the data of several tables
to a single merged table, and then I do a heap swap on one of the original
tables while dropping the others. So, if you have 3 partitions and want to
merge them to one, then I write all three partitions to a temporary heap,
swap the new heap on partition 1 and then drop partitions 2 and 3.

Now, this approach worked fine for PostgreSQL 15 and 16, but 17 introduced
some changes that altered the behavior so that I only see data from one of
the partitions after merge (the last one written).

The commit that I think is responsible is the following:
https://github.com/postgres/postgres/commit/8af256524893987a3e534c6578dd60edfb782a77

Now, I realize that this is not a problem for PostgreSQL itself since the
rewrite functionality isn't used for the purpose I am using it. To my
defense, the rewrite code seems to imply that it should be possible to
write more data to an existing heap according to this comment in
begin_heap_rewrite: /* new_heap needn't be empty, just locked */.

I've also tried recompiling PG17 with the rewriteheap.c file from PG16 and
then it works again. I haven't yet been able to figure out exactly what is
different but I will continue to try to narrow it down. In the meantime,
maybe someone on the mailing list has some insight on what could be the
issue and whether my approach is viable?

Regards,
Erik

-- 
Database Architect, Timescale

--000000000000fdf91706276e7b32
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hello,</div><div><br></div><div>I&#39;ve noticed a ch=
ange in behavior of the heap rewrite functionality in PostgreSQL 17, used b=
y, e.g., CLUSTER. I&#39;ve been experimenting with the functionality to imp=
lement a way to merge partitions in TimescaleDB. I am using table_relation_=
copy_for_cluster() to write the data of several tables to a single merged t=
able, and then I do a heap swap on one of the original tables while droppin=
g the others. So, if you have 3 partitions and want to merge them to one, t=
hen I write all three partitions to a temporary heap, swap the new heap on =
partition 1 and then drop partitions 2 and 3.</div><div><br></div><div>Now,=
 this approach worked fine for PostgreSQL 15 and 16, but 17 introduced some=
 changes that altered the behavior so that I only see data from one of the =
partitions after merge (the last one written).<br></div><div><br></div><div=
>The commit that I think is responsible is the following: <a href=3D"https:=
//github.com/postgres/postgres/commit/8af256524893987a3e534c6578dd60edfb782=
a77">https://github.com/postgres/postgres/commit/8af256524893987a3e534c6578=
dd60edfb782a77</a></div><div><br></div><div>Now, I realize that this is not=
 a problem for PostgreSQL itself since the rewrite functionality isn&#39;t =
used for the purpose I am using it. To my defense, the rewrite code seems t=
o imply that it should be possible to write more data to an existing heap a=
ccording to this comment in begin_heap_rewrite: <code class=3D"gmail-diff-t=
ext gmail-syntax-highlighted-line"><span class=3D"gmail-pl-c">/* new_heap n=
eedn&#39;t be empty, just locked */.</span></code></div><br><div>I&#39;ve a=
lso tried recompiling PG17 with the rewriteheap.c file from PG16 and then i=
t works again. I haven&#39;t yet been able to figure out exactly what is di=
fferent but I will continue to try to narrow it down. In the meantime, mayb=
e someone on the mailing list has some insight on what could be the issue a=
nd whether my approach is viable?</div><div><br></div><div>Regards,</div><d=
iv>Erik</div><div><span class=3D"gmail_signature_prefix"><br></span></div><=
div><span class=3D"gmail_signature_prefix">-- </span></div><div dir=3D"ltr"=
 class=3D"gmail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"l=
tr">Database Architect, Timescale<br></div></div></div>

--000000000000fdf91706276e7b32--