MIME-Version: 1.0
References: <CAFCRh-8FOZiycyfX4uPB8MTHQTxqNVuW0pdKBuFNQneEZy1PwQ@mail.gmail.com>
 <7f90e1f3-7e0b-4b87-8cb6-f2862755fd3c@aklaver.com> <CAFCRh-9SWkL2Evdc8vWrDREGiuDeo58fVqqT4XgpnaKPj7FumQ@mail.gmail.com>
 <f5a5bef6-56ca-49cb-b30c-a517ece77255@aklaver.com> <CA+v5N43bO5Xuiu8Bunn+0RJbVepr2Yn4-0f8U6-hNZe0ctt23A@mail.gmail.com>
 <CAFCRh-_3oKeCDKOFWQ=bxBPNZWtrvL1ucG5wjOxh2HgEbz-qtw@mail.gmail.com>
In-Reply-To: <CAFCRh-_3oKeCDKOFWQ=bxBPNZWtrvL1ucG5wjOxh2HgEbz-qtw@mail.gmail.com>
From: Merlin Moncure <mmoncure@gmail.com>
Date: Fri, 11 Jul 2025 14:30:45 -0600
Message-ID: <CAHyXU0znf84Cnv4cTdG7SQqEaXs_eiX+W4oOERq89K5g_tBk9A@mail.gmail.com>
Subject: Re: Aggregate versions of hashing functions (md5, sha1, etc...)
To: Dominique Devienne <ddevienne@gmail.com>
Cc: Florents Tselai <florents.tselai@gmail.com>, Adrian Klaver <adrian.klaver@aklaver.com>, 
	pgsql-general <pgsql-general@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000fce0f00639ad2ff6"
Archived-At: <https://www.postgresql.org/message-id/CAHyXU0znf84Cnv4cTdG7SQqEaXs_eiX%2BW4oOERq89K5g_tBk9A%40mail.gmail.com>
Precedence: bulk

--000000000000fce0f00639ad2ff6
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Jul 11, 2025 at 10:17=E2=80=AFAM Dominique Devienne <ddevienne@gmai=
l.com>
wrote:

> On Fri, Jul 11, 2025 at 6:05=E2=80=AFPM Florents Tselai
> <florents.tselai@gmail.com> wrote:
> > On Fri, Jul 11, 2025, 18:27 Adrian Klaver <adrian.klaver@aklaver.com>
> wrote:
> >> [...] create an extension that incorporates the code.
> >
> > That's an ideal use case for an extension indeed .
>
> Extensions are of no use to me, unfortunately, unless built-in and
> official. So if I have to wait for v19, so be it. But the ball has to
> get rolling at least.
>

Right -- exactly.   The problem is cloud providers do not allow 3rd party
extensions.  You can work around this if your extensions are available at
the SQL level, or can be built with standard extensions.

Candidly, it's going to be tough sledding to get your needs incorporated
into contrib.   I'm not saying it wont happen -- let's just say holding
breath until solution is not advisable.  I know exactly where you're at.

This is why I built an SQL available extension that does lz4 compression;
it's the only way to compress data locally before sending it out to AWS via
the s3 API.

Aside: This may be an unpopular position, but I think the postgres
extension system is useless for 3rd party contributions until there is some
way to introduce items in the vein of npm, pip, etc.

 I think the only short term path I know of:
1. write or find a C library that does something similar to what you need
2. compile that to plv8 with emscriptem
3. Wrap with plv8 function handlers

Getting it to reasonable performance is possible if you compile WASM and
work out amortizing start up costs. However, due to how plv8 and
emscripten interact from memory standpoint, there's a lot of copying to
move memory in/out, and you will have to work under a manageable buffer
size, say 1mb.  If you're curious I can take you through it.

merlin

--000000000000fce0f00639ad2ff6
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">On Fri, Jul 11, 2025 at 10:17=E2=80=AFAM =
Dominique Devienne &lt;<a href=3D"mailto:ddevienne@gmail.com">ddevienne@gma=
il.com</a>&gt; wrote:</div><div class=3D"gmail_quote gmail_quote_container"=
><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border=
-left:1px solid rgb(204,204,204);padding-left:1ex">On Fri, Jul 11, 2025 at =
6:05=E2=80=AFPM Florents Tselai<br>
&lt;<a href=3D"mailto:florents.tselai@gmail.com" target=3D"_blank">florents=
.tselai@gmail.com</a>&gt; wrote:<br>
&gt; On Fri, Jul 11, 2025, 18:27 Adrian Klaver &lt;<a href=3D"mailto:adrian=
.klaver@aklaver.com" target=3D"_blank">adrian.klaver@aklaver.com</a>&gt; wr=
ote:<br>
&gt;&gt; [...] create an extension that incorporates the code.<br>
&gt;<br>
&gt; That&#39;s an ideal use case for an extension indeed .<br>
<br>
Extensions are of no use to me, unfortunately, unless built-in and<br>
official. So if I have to wait for v19, so be it. But the ball has to<br>
get rolling at least.<br></blockquote><div><br></div><div>Right -- exactly.=
=C2=A0 =C2=A0The problem is cloud providers=C2=A0do not allow 3rd party ext=
ensions.=C2=A0 You can work around this if your extensions are available at=
 the SQL level, or can be built with standard extensions.=C2=A0=C2=A0</div>=
<div><br></div><div>Candidly, it&#39;s going to be tough sledding to get yo=
ur needs incorporated into contrib.=C2=A0 =C2=A0I&#39;m not saying it wont =
happen -- let&#39;s just say holding breath until solution is not advisable=
.=C2=A0 I know exactly where you&#39;re at.=C2=A0=C2=A0</div><div><br></div=
><div>This is why I built an SQL available extension that does lz4 compress=
ion; it&#39;s the only way to compress data locally before sending it out t=
o AWS via the s3 API.=C2=A0=C2=A0</div><div><br></div><div>Aside: This may =
be an unpopular position, but I think the postgres extension system is usel=
ess for 3rd party contributions until there is some way to introduce items =
in the vein of npm, pip, etc.</div><div><br></div><div>=C2=A0I think the on=
ly short term path I=C2=A0know of:</div><div>1. write or find a C library t=
hat does something similar to what you need</div><div>2. compile that to pl=
v8 with emscriptem</div><div>3. Wrap with plv8 function handlers</div><div>=
<br></div><div>Getting it to reasonable performance is possible if you comp=
ile WASM and work out amortizing start up costs. However, due to how plv8 a=
nd emscripten=C2=A0interact from memory standpoint, there&#39;s a lot of co=
pying to move memory in/out, and you will have to work under a manageable b=
uffer size, say 1mb.=C2=A0 If you&#39;re curious I can take you through it.=
</div><div><br></div><div>merlin</div></div></div>

--000000000000fce0f00639ad2ff6--