Subject: Re: pgbench with libevent?

From: Fabien COELHO <[email-masked]>
Date: Mon, 14 Aug 2023 02:35:26 +0200 (CEST)

> Pgbench is managing clients I/Os manually with select or poll. Much of this 
> could be managed by libevent.

Or maybe libuv (used by nodejs?).

>From preliminary testing libevent seems not too good at fine grain time 
management which are used for throttling, whereas libuv advertised that it 
is good at it, although what it does is yet to be seen.

Note: libev had no updates in 8 years.

-- 
Fabien.

---

From: Thomas Munro <[email-masked]>
Date: Mon, 14 Aug 2023 14:58:26 +1200

> is good at it, although what it does is yet to be seen.

Do you think our WaitEventSet stuff could be good here, if made
frontend-friendly?

---

From: Tatsuo Ishii <[email-masked]>
Date: Mon, 14 Aug 2023 15:06:59 +0900 (JST)

> Do you think our WaitEventSet stuff could be good here, if made

Interesting. In my understanding this also needs to make Latch
frontend-friendly?

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

---

From: Thomas Munro <[email-masked]>
Date: Mon, 14 Aug 2023 19:15:20 +1200

> Interesting. In my understanding this also needs to make Latch

It could be refactored to support a different subset of event types --
maybe just sockets, no latches and obviously no 'postmaster death'.
But figuring out how to make latches work between threads might also
be interesting for future projects...

Maybe Fabien has completion-based I/O in mind (not just "readiness").
That's something that some of those libraries can do, IIUC.  For
example, when your thread wakes up, it tells you "your socket read is
finished, the data is already in your target buffer".  As opposed to
"you can now call recv() without blocking", so you avoid another trip
into the kernel.  But that's also something we'll eventually want to
figure out in the server.

---

From: Fabien COELHO <[email-masked]>
Date: Mon, 14 Aug 2023 12:09:52 +0200 (CEST)

Hello Thomas,

> Do you think our WaitEventSet stuff could be good here, if made

Interesting question.

AFAICS, the answer is that it could indeed probably fit the task, but it 
would require significant work to make it thread-compatible, and to 
untangle it from IsUnderPosmaster/postmaster death, memory context, 
elog/ereport, and other back-end specific stuff.

If you remove all that with a clean abstraction (quite a task), then once 
done the question could be why not use libevent/libuv/… in the backend 
instead of maintaining more or less the same thing inside postgres?

So ISTM that as far as pgbench is concerned it would be much simpler to 
use libevent/libuv/… directly if the pros are enough and the cons not 
redhibitory, and provided that the needed detailed features are really 
there.

-- 
Fabien.

---

From: Fabien COELHO <[email-masked]>
Date: Mon, 14 Aug 2023 12:22:24 +0200 (CEST)

> Maybe Fabien has completion-based I/O in mind (not just "readiness").

Pgbench is really a primitive client on top of libpq. ISTM that 
completion-based I/O would require to enhance libpq asynchronous-ity, not 
just expose its underlying fd to allow asynchronous implementations.
Currently pgbench only actuall "waits" for results from the server
and testing PQisBusy to check whether they are there.

> finished, the data is already in your target buffer".

Indeed, libevent has a higher level "buffer" oriented API.

> As opposed to "you can now call recv() without blocking", so you avoid 
> another trip into the kernel.  But that's also something we'll 
> eventually want to figure out in the server.

-- 
Fabien.