public inbox for [email protected]
help / color / mirror / Atom feedfuture of PQfn()
18+ messages / 5 participants
[nested] [flat]
* future of PQfn()
@ 2026-05-26 16:05 Nathan Bossart <[email protected]>
2026-05-26 16:53 ` Re: future of PQfn() Christoph Berg <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 21:36 ` Re: future of PQfn() Jelte Fennema-Nio <[email protected]>
0 siblings, 3 replies; 18+ messages in thread
From: Nathan Bossart @ 2026-05-26 16:05 UTC (permalink / raw)
To: pgsql-hackers
PQfn() was marked "somewhat obsolete" in commit efc3a25bb0 (2003), and was
later marked "unsafe" in commit bd48114937 (2026). I looked around for
third-party code that uses this interface but found none, and the only
internal usage is for the frontend LO interface. As part of the latter
commit, a special PQnfn() function was added for use by the fronend LO
functions, but this was not exported by libpq.
Given the above, I'd like to propose retiring PQfn() in v20. Since it's an
exported symbol, we can't just delete the code, but we could have it
unconditionally error. Assuming folks are okay with that, I'm wondering
what we should do with the relevant documentation. Should we leave a stub
with a note about its removal, or should we just wipe all mentions? I'm
currently leaning towards leaving a note, but I could see the argument
that's not even worth doing given the lack of uptake.
The other question is what to do with the frontend LO code. The simplest
thing we can do is to leave PQnfn() around as an internal function that is
only used by this interface. Alternatively, we could take our own advice
and used a prepared statement with binary transmission of params/results,
but that has two key problems: 1) potential name collisions with
user-created prepared statements and 2) breakage after DISCARD/DEALLOCATE,
which I haven't come up with a good way to deal with. Another approach we
could take is to just send the query via PQexecParams(), but a simple test
(creating and unlinking 10K LOs) showed a ~41% slowdown compared to HEAD.
So, I guess we'll need to keep PQnfn() around for now...
Thoughts?
--
nathan
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
@ 2026-05-26 16:53 ` Christoph Berg <[email protected]>
2026-05-26 16:55 ` Re: future of PQfn() Christoph Berg <[email protected]>
2 siblings, 1 reply; 18+ messages in thread
From: Christoph Berg @ 2026-05-26 16:53 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: pgsql-hackers
Re: Nathan Bossart
> PQfn() was marked "somewhat obsolete" in commit efc3a25bb0 (2003), and was
> later marked "unsafe" in commit bd48114937 (2026). I looked around for
> third-party code that uses this interface but found none
I found some references via Debian's codesearch:
https://sources.debian.org/src/fpc/3.2.2+dfsg-50/fpcsrc/packages/postgres/src/postgres3dyn.pp?hl=141...
PQfn : function (conn:PPGconn; fnid:longint; result_buf:Plongint; result_len:Plongint; result_is_int:longint;args:PPQArgBlock; nargs:longint):PPGresult;cdecl;
{ Accessor functions for PGresult objects }
https://sources.debian.org/src/clisp/1:2.49.20250504.gitf662209-2/modules/postgresql/postgresql.lisp...
;; "Fast path" interface --- not really recommended for application use
(def-call-out PQfn (:return-type PGresult)
(:arguments (conn PGconn) (fnid int) (result_buf (c-ptr int) :out)
(result_len (c-ptr int) :out) (result_is_int int)
(args (c-array-ptr PQArgBlock)) ; at least nargs
(nargs int)))
https://sources.debian.org/src/rust-pq-sys/0.7.5-1/src/bindings_linux_32.rs?hl=745#L745
unsafe extern "C" {
pub fn PQfn(
conn: *mut PGconn,
fnid: ::std::os::raw::c_int,
result_buf: *mut ::std::os::raw::c_int,
result_len: *mut ::std::os::raw::c_int,
result_is_int: ::std::os::raw::c_int,
args: *const PQArgBlock,
nargs: ::std::os::raw::c_int,
) -> *mut PGresult;
}
https://sources.debian.org/src/vala/0.56.19-1/vapi/libpq.vapi?hl=349#L349
[CCode (cname = "PQfn")]
public Result fn (int fnid, [CCode (array_length = false)] int[] result_buf, out int result_len, int result_is_int, ArgBlock[] args);
https://sources.debian.org/src/libdbi-drivers/0.9.0-13/drivers/pgsql/dbd_pgsql.h?hl=650#L650
#define PGSQL_CUSTOM_FUNCTIONS { \
...
"PQfn", \
All seem to be libpq wrappers.
Then there is a handful of packages with a copy of libpq-fe.h that
includes the function declaration.
... plus two other mentioning PQfn as "unimplemented".
> Given the above, I'd like to propose retiring PQfn() in v20. Since it's an
> exported symbol, we can't just delete the code, but we could have it
> unconditionally error.
Keeping the symbol but making it return an error seems ok from the
above I think.
Christoph
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 16:53 ` Re: future of PQfn() Christoph Berg <[email protected]>
@ 2026-05-26 16:55 ` Christoph Berg <[email protected]>
0 siblings, 0 replies; 18+ messages in thread
From: Christoph Berg @ 2026-05-26 16:55 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; pgsql-hackers
Re: To Nathan Bossart
> I found some references via Debian's codesearch:
I meant to include the URL for that:
https://codesearch.debian.net/search?q=PQfn%5Cb&literal=0
PQnfn has no hits outside of postgresql-18.
Christoph
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
@ 2026-05-26 17:42 ` Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2026-05-26 17:42 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: pgsql-hackers
On Tue, May 26, 2026 at 9:05 AM Nathan Bossart <[email protected]> wrote:
> Given the above, I'd like to propose retiring PQfn() in v20. Since it's an
> exported symbol, we can't just delete the code, but we could have it
> unconditionally error.
That seems reasonable to me. (We may want to exercise the
PqMsg_FunctionCall message more explicitly in our test suite at the
same time we do that...)
> Assuming folks are okay with that, I'm wondering
> what we should do with the relevant documentation. Should we leave a stub
> with a note about its removal, or should we just wipe all mentions? I'm
> currently leaning towards leaving a note, but I could see the argument
> that's not even worth doing given the lack of uptake.
I think we can probably wipe it out, personally.
> The other question is what to do with the frontend LO code. The simplest
> thing we can do is to leave PQnfn() around as an internal function that is
> only used by this interface. Alternatively, we could take our own advice
> and used a prepared statement with binary transmission of params/results,
> but that has two key problems: 1) potential name collisions with
> user-created prepared statements and 2) breakage after DISCARD/DEALLOCATE,
> which I haven't come up with a good way to deal with. Another approach we
> could take is to just send the query via PQexecParams(), but a simple test
> (creating and unlinking 10K LOs) showed a ~41% slowdown compared to HEAD.
> So, I guess we'll need to keep PQnfn() around for now...
Short-term, keeping it around seems fine.
Long-term, it doesn't feel great that the alternatives we tell other
people to use are... worse. Surely other clients of libpq run into the
layering violation problem with prepared statements, as well?
--Jacob
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
@ 2026-05-26 19:55 ` Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Nathan Bossart @ 2026-05-26 19:55 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: pgsql-hackers
On Tue, May 26, 2026 at 10:42:47AM -0700, Jacob Champion wrote:
> Short-term, keeping it around seems fine.
>
> Long-term, it doesn't feel great that the alternatives we tell other
> people to use are... worse. Surely other clients of libpq run into the
> layering violation problem with prepared statements, as well?
Yup. Here's a related note in JDBC:
https://github.com/pgjdbc/pgjdbc/blob/cf2d89ec/docs/content/documentation/server-prepare.md?plain=1#...
I wonder how difficult it would be to teach the protocol to advise clients
when prepared statements are deallocated...
FWIW I'm less concerned about the name collision problem. I was thinking
we could just document that libpq manages statements with a prefix like
"libpq_internal_". Any problems in that area seem likely to be intentional
breakage that we needn't worry about.
--
nathan
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
@ 2026-05-27 21:39 ` Jacob Champion <[email protected]>
2026-05-28 15:17 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 15:14 ` Re: future of PQfn() Nathan Bossart <[email protected]>
0 siblings, 2 replies; 18+ messages in thread
From: Jacob Champion @ 2026-05-27 21:39 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: pgsql-hackers
On Tue, May 26, 2026 at 12:55 PM Nathan Bossart
<[email protected]> wrote:
> I wonder how difficult it would be to teach the protocol to advise clients
> when prepared statements are deallocated...
Probably not too difficult. But it seems like most, if not all, of the
stuff in the DISCARD ALL umbrella is a target for a feature like
that... This feels a lot like the perennial request for proxies to be
able to separate their own context from the per-application/per-user
contexts running on top of them.
--Jacob
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
@ 2026-05-28 15:17 ` Nathan Bossart <[email protected]>
1 sibling, 0 replies; 18+ messages in thread
From: Nathan Bossart @ 2026-05-28 15:17 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: pgsql-hackers
On Wed, May 27, 2026 at 02:39:53PM -0700, Jacob Champion wrote:
> On Tue, May 26, 2026 at 12:55 PM Nathan Bossart
> <[email protected]> wrote:
>> I wonder how difficult it would be to teach the protocol to advise clients
>> when prepared statements are deallocated...
>
> Probably not too difficult. But it seems like most, if not all, of the
> stuff in the DISCARD ALL umbrella is a target for a feature like
> that... This feels a lot like the perennial request for proxies to be
> able to separate their own context from the per-application/per-user
> contexts running on top of them.
I've been thinking through a bunch of options here, and a new protocol
message seems like the best choice, if for no other reason than it solves a
general problem for clients. I'm working on a patch that I'll post in a
new thread.
--
nathan
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
@ 2026-05-29 15:14 ` Nathan Bossart <[email protected]>
2026-05-29 15:43 ` Re: future of PQfn() Jacob Champion <[email protected]>
1 sibling, 1 reply; 18+ messages in thread
From: Nathan Bossart @ 2026-05-29 15:14 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: pgsql-hackers
On Wed, May 27, 2026 at 02:39:53PM -0700, Jacob Champion wrote:
> On Tue, May 26, 2026 at 12:55 PM Nathan Bossart
> <[email protected]> wrote:
>> I wonder how difficult it would be to teach the protocol to advise clients
>> when prepared statements are deallocated...
>
> Probably not too difficult. But it seems like most, if not all, of the
> stuff in the DISCARD ALL umbrella is a target for a feature like
> that... This feels a lot like the perennial request for proxies to be
> able to separate their own context from the per-application/per-user
> contexts running on top of them.
Here is a work-in-progress patch set that goes this direction. This
introduces a callback mechanism in libpq that is used to handle statement
deallocation notifications. Older servers/clients fall back to
PQexecParams(), which is slower, but the alternative is to leave PQnfn()
and related code around indefinitely.
I'm wondering whether this new message type is general enough. For
example, perhaps we could make an extensible message type for tracking
various things. And I want to ensure this is useful for other clients,
too.
--
nathan
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 15:14 ` Re: future of PQfn() Nathan Bossart <[email protected]>
@ 2026-05-29 15:43 ` Jacob Champion <[email protected]>
2026-05-29 16:10 ` Re: future of PQfn() Nathan Bossart <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2026-05-29 15:43 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: pgsql-hackers
On Fri, May 29, 2026 at 8:14 AM Nathan Bossart <[email protected]> wrote:
> Here is a work-in-progress patch set that goes this direction.
At a high level, I think advertising support for a single new message
needs to be done in a protocol extension rather than a minor version
bump.
> This
> introduces a callback mechanism in libpq that is used to handle statement
> deallocation notifications. Older servers/clients fall back to
> PQexecParams(), which is slower, but the alternative is to leave PQnfn()
> and related code around indefinitely.
IMO there's no hurry in getting rid of that path. If we decide to go
this direction, a fallback to PQnfn() seems like it'd fine for a few
releases; we could eventually swap to a PQexecParams() fallback and
get rid of the extra code once the older servers have aged out.
> I'm wondering whether this new message type is general enough. For
> example, perhaps we could make an extensible message type for tracking
> various things. And I want to ensure this is useful for other clients,
> too.
If it's just a general notification message, what does negotiating
"support" mean? Is best-effort notification okay, if the client has no
idea what a future message type means, or if the server doesn't send
the specific type of message the client is hoping for?
(In general, I'm kind of down on the "notify the client that X
happened" method of working around architectural issues. Maybe that's
what we need to move this specific part forward, but it doesn't feel
like a long-term solution and I don't know that we need to genericize
it without a solid set of use cases.)
--Jacob
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 15:14 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 15:43 ` Re: future of PQfn() Jacob Champion <[email protected]>
@ 2026-05-29 16:10 ` Nathan Bossart <[email protected]>
2026-05-29 16:33 ` Re: future of PQfn() Jacob Champion <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Nathan Bossart @ 2026-05-29 16:10 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: pgsql-hackers
On Fri, May 29, 2026 at 08:43:07AM -0700, Jacob Champion wrote:
> On Fri, May 29, 2026 at 8:14 AM Nathan Bossart <[email protected]> wrote:
>> Here is a work-in-progress patch set that goes this direction.
>
> At a high level, I think advertising support for a single new message
> needs to be done in a protocol extension rather than a minor version
> bump.
WFM
>> This
>> introduces a callback mechanism in libpq that is used to handle statement
>> deallocation notifications. Older servers/clients fall back to
>> PQexecParams(), which is slower, but the alternative is to leave PQnfn()
>> and related code around indefinitely.
>
> IMO there's no hurry in getting rid of that path. If we decide to go
> this direction, a fallback to PQnfn() seems like it'd fine for a few
> releases; we could eventually swap to a PQexecParams() fallback and
> get rid of the extra code once the older servers have aged out.
That's fine with me, too.
>> I'm wondering whether this new message type is general enough. For
>> example, perhaps we could make an extensible message type for tracking
>> various things. And I want to ensure this is useful for other clients,
>> too.
>
> If it's just a general notification message, what does negotiating
> "support" mean? Is best-effort notification okay, if the client has no
> idea what a future message type means, or if the server doesn't send
> the specific type of message the client is hoping for?
That's what I had in mind. But if we don't have anything specific in mind
that this mechanism could be extended to support, maybe we shouldn't
bother. Especially if we can just add protocol extensions as necessary.
> (In general, I'm kind of down on the "notify the client that X
> happened" method of working around architectural issues. Maybe that's
> what we need to move this specific part forward, but it doesn't feel
> like a long-term solution and I don't know that we need to genericize
> it without a solid set of use cases.)
I'm certainly open to other ideas, but I'm afraid this is the best I've
come up with in my admittedly limited time thinking about the problem.
--
nathan
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 15:14 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 15:43 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:10 ` Re: future of PQfn() Nathan Bossart <[email protected]>
@ 2026-05-29 16:33 ` Jacob Champion <[email protected]>
2026-05-29 16:42 ` Re: future of PQfn() Nathan Bossart <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2026-05-29 16:33 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: pgsql-hackers
On Fri, May 29, 2026 at 9:11 AM Nathan Bossart <[email protected]> wrote:
> I'm certainly open to other ideas, but I'm afraid this is the best I've
> come up with in my admittedly limited time thinking about the problem.
No worries -- I hadn't meant to block progress here on protocol
design. I think keeping PQnfn() for the immediate future is a good
plan. I just wanted to plant a seed for getting away from this problem
eventually.
(As for pie-in-the-sky alternative ideas, the ability for middleware
to separate contexts or streams of packets has come up before. libpq
could theoretically mark its own "context" of server-side allocations
that are not touched by an application-context DISCARD.)
--Jacob
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 15:14 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 15:43 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:10 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 16:33 ` Re: future of PQfn() Jacob Champion <[email protected]>
@ 2026-05-29 16:42 ` Nathan Bossart <[email protected]>
2026-05-29 17:04 ` Re: future of PQfn() Jacob Champion <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Nathan Bossart @ 2026-05-29 16:42 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: pgsql-hackers
For future reference in the archives, I'm moving the discussion about 0001
(the prepared statement deallocation notification mechanism) to a new
thread:
https://postgr.es/m/ahm_4eOKkkKJ3Gds%40nathan
On Fri, May 29, 2026 at 09:33:03AM -0700, Jacob Champion wrote:
> On Fri, May 29, 2026 at 9:11 AM Nathan Bossart <[email protected]> wrote:
>> I'm certainly open to other ideas, but I'm afraid this is the best I've
>> come up with in my admittedly limited time thinking about the problem.
>
> No worries -- I hadn't meant to block progress here on protocol
> design. I think keeping PQnfn() for the immediate future is a good
> plan. I just wanted to plant a seed for getting away from this problem
> eventually.
>
> (As for pie-in-the-sky alternative ideas, the ability for middleware
> to separate contexts or streams of packets has come up before. libpq
> could theoretically mark its own "context" of server-side allocations
> that are not touched by an application-context DISCARD.)
Along these lines, I did consider "pinning" statements or even having
"built-in" ones for libpq. I didn't like the "pinning" idea because that
seemed problematic for connection poolers. And the "built-in" idea seemed
too libpq-centric for what I'd argue is a general problem. The other ideas
involved guessing at what's happening based on the queries or somehow
trying to handle failures due to missing/wrong prepared statements, none of
which felt viable.
--
nathan
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 15:14 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 15:43 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:10 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 16:33 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:42 ` Re: future of PQfn() Nathan Bossart <[email protected]>
@ 2026-05-29 17:04 ` Jacob Champion <[email protected]>
2026-06-03 21:36 ` Re: future of PQfn() Nathan Bossart <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Jacob Champion @ 2026-05-29 17:04 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: pgsql-hackers
On Fri, May 29, 2026 at 9:42 AM Nathan Bossart <[email protected]> wrote:
> Along these lines, I did consider "pinning" statements or even having
> "built-in" ones for libpq. I didn't like the "pinning" idea because that
> seemed problematic for connection poolers.
Right -- whether a general context, or multiplexed streams, or
explicit pins, proxies would have to be intimately aware of them. They
can then layer their own on top, or make sure different client
requests don't conflict, or release them on client disconnection...
> And the "built-in" idea seemed
> too libpq-centric for what I'd argue is a general problem. The other ideas
> involved guessing at what's happening based on the queries or somehow
> trying to handle failures due to missing/wrong prepared statements, none of
> which felt viable.
Agreed. (Although, for that last point, I wondered whether we could
make this idempotent somehow. I think the answer is "not worth it".)
--Jacob
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 15:14 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 15:43 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:10 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 16:33 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:42 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 17:04 ` Re: future of PQfn() Jacob Champion <[email protected]>
@ 2026-06-03 21:36 ` Nathan Bossart <[email protected]>
2026-06-04 10:39 ` Re: future of PQfn() Dagfinn Ilmari Mannsåker <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Nathan Bossart @ 2026-06-03 21:36 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: pgsql-hackers
While the prepared statement deallocation idea bakes in the other thread, I
thought I'd post a new patch set here. I've reordered things a bit and
left out the conversion to prepared statements for now (although I do have
a patch for that). 0001 removes PQfn() and changes all internal uses to
PQnfn(). 0002 and 0003 are just some additional cleanup that I noticed
as I've been working on this stuff.
--
nathan
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 15:14 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 15:43 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:10 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 16:33 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:42 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 17:04 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-06-03 21:36 ` Re: future of PQfn() Nathan Bossart <[email protected]>
@ 2026-06-04 10:39 ` Dagfinn Ilmari Mannsåker <[email protected]>
2026-06-04 14:57 ` Re: future of PQfn() Nathan Bossart <[email protected]>
0 siblings, 1 reply; 18+ messages in thread
From: Dagfinn Ilmari Mannsåker @ 2026-06-04 10:39 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Jacob Champion <[email protected]>; pgsql-hackers
Nathan Bossart <[email protected]> writes:
> - <sect1 id="libpq-fastpath">
> - <title>The Fast-Path Interface</title>
> -
> - <indexterm zone="libpq-fastpath">
> - <primary>fast path</primary>
> - </indexterm>
Should we move this to the "Obsolete or Renamed Features" appendix
(e.g. appendix-obsolete-libpq-fastpath.sgml) with a description of why
it was removed?
- ilmari
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 17:42 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-26 19:55 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 15:14 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 15:43 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:10 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 16:33 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-05-29 16:42 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-05-29 17:04 ` Re: future of PQfn() Jacob Champion <[email protected]>
2026-06-03 21:36 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2026-06-04 10:39 ` Re: future of PQfn() Dagfinn Ilmari Mannsåker <[email protected]>
@ 2026-06-04 14:57 ` Nathan Bossart <[email protected]>
0 siblings, 0 replies; 18+ messages in thread
From: Nathan Bossart @ 2026-06-04 14:57 UTC (permalink / raw)
To: Dagfinn Ilmari Mannsåker <[email protected]>; +Cc: Jacob Champion <[email protected]>; pgsql-hackers
On Thu, Jun 04, 2026 at 11:39:02AM +0100, Dagfinn Ilmari Mannsåker wrote:
> Nathan Bossart <[email protected]> writes:
>
>> - <sect1 id="libpq-fastpath">
>> - <title>The Fast-Path Interface</title>
>> -
>> - <indexterm zone="libpq-fastpath">
>> - <primary>fast path</primary>
>> - </indexterm>
>
> Should we move this to the "Obsolete or Renamed Features" appendix
> (e.g. appendix-obsolete-libpq-fastpath.sgml) with a description of why
> it was removed?
Yes, I think we should. Done in the attached.
--
nathan
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
@ 2026-05-26 21:36 ` Jelte Fennema-Nio <[email protected]>
2026-05-27 18:23 ` Re: future of PQfn() Nathan Bossart <[email protected]>
2 siblings, 1 reply; 18+ messages in thread
From: Jelte Fennema-Nio @ 2026-05-26 21:36 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: pgsql-hackers
On Tue, 26 May 2026 at 18:05, Nathan Bossart <[email protected]>
wrote:
> Another approach we
> could take is to just send the query via PQexecParams(), but a simple test
> (creating and unlinking 10K LOs) showed a ~41% slowdown compared to HEAD.
> So, I guess we'll need to keep PQnfn() around for now...
>
> Thoughts?
I had a small WIP patch (fully AI generated and not yet vetted by me)
lying around for unrelated reasons to make the extended protocol perform
closer to the simple protocol with pgbench --select-only by using
CreateOneShotCachedPlan to create the plan for unnamed prepared
statements (see attached).
Could you share the simple LO test you were running here and/or rerun it
with this patch applied? I'd love to know if the patch reduces the
slowdown significantly, or if something else is the bottleneck.
Attachments:
[text/x-patch] v1-0001-Speed-up-extended-protocol-by-using-oneshot-cache.patch (4.8K, 2-v1-0001-Speed-up-extended-protocol-by-using-oneshot-cache.patch)
download | inline diff:
From 3586deb8bbcc25ac0c4de5c7ef6421ffddc93839 Mon Sep 17 00:00:00 2001
From: Jelte Fennema-Nio <[email protected]>
Date: Tue, 26 May 2026 16:07:57 +0200
Subject: [PATCH v1] Speed up extended protocol by using oneshot cached plan
DISCLAIMER: Fully written by AI. I still need to check if the code is
fully correct. The approach seems sensible though, and meson test
passes on my machine.
On my laptop on master I get much lower perf with the extended protocol
than the simple protocol in a basic pgbench --select-only test:
pgbench -i
pgbench --select-only -T 10 --protocol simple
pgbench --select-only -T 10 --protocol extended
For simple I get ~56k TPS and for extended I only get ~49k TPS
With this change I get ~53k TPS with extended so a ~8% improvement.
---
src/backend/tcop/postgres.c | 57 ++++++++++++++++++++++++++++++++-----
1 file changed, 50 insertions(+), 7 deletions(-)
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index dbef734a93f..3ab9d7ab3b2 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -1513,10 +1513,27 @@ exec_parse_message(const char *query_string, /* string to execute */
/*
* Create the CachedPlanSource before we do parse analysis, since it
- * needs to see the unmodified raw parse tree.
+ * needs to see the unmodified raw parse tree. For unnamed statements
+ * use a one-shot plan: this skips the deep copy of the raw parse
+ * tree, the deep copy of the rewritten query tree in BuildCachedPlan,
+ * and the SaveCachedPlan reparent into CacheMemoryContext. The
+ * trade-off is no invalidation support, which is acceptable for the
+ * common case where the unnamed statement is Parse+Bind+Execute'd in
+ * a single transaction.
*/
- psrc = CreateCachedPlan(raw_parse_tree, query_string,
- CreateCommandTag(raw_parse_tree->stmt));
+ if (is_named)
+ psrc = CreateCachedPlan(raw_parse_tree, query_string,
+ CreateCommandTag(raw_parse_tree->stmt));
+ else
+ {
+ /*
+ * CreateOneShotCachedPlan doesn't copy query_string, so we must
+ * stash it in unnamed_stmt_context (CurrentMemoryContext) so it
+ * outlives the MessageContext reset between Parse and Bind.
+ */
+ psrc = CreateOneShotCachedPlan(raw_parse_tree, pstrdup(query_string),
+ CreateCommandTag(raw_parse_tree->stmt));
+ }
/*
* Set up a snapshot if parse analysis will need one.
@@ -1546,8 +1563,12 @@ exec_parse_message(const char *query_string, /* string to execute */
{
/* Empty input string. This is legal. */
raw_parse_tree = NULL;
- psrc = CreateCachedPlan(raw_parse_tree, query_string,
- CMDTAG_UNKNOWN);
+ if (is_named)
+ psrc = CreateCachedPlan(raw_parse_tree, query_string,
+ CMDTAG_UNKNOWN);
+ else
+ psrc = CreateOneShotCachedPlan(raw_parse_tree, pstrdup(query_string),
+ CMDTAG_UNKNOWN);
querytree_list = NIL;
}
@@ -1584,9 +1605,13 @@ exec_parse_message(const char *query_string, /* string to execute */
else
{
/*
- * We just save the CachedPlanSource into unnamed_stmt_psrc.
+ * For unnamed statements we use a one-shot CachedPlanSource, so we
+ * can't SaveCachedPlan it. Instead reparent its context (which is
+ * unnamed_stmt_context) under CacheMemoryContext so that it survives
+ * the MessageContext reset between Parse and the following Bind.
+ * drop_unnamed_stmt() will MemoryContextDelete it explicitly.
*/
- SaveCachedPlan(psrc);
+ MemoryContextSetParent(psrc->context, CacheMemoryContext);
unnamed_stmt_psrc = psrc;
}
@@ -2028,8 +2053,18 @@ exec_bind_message(StringInfo input_message)
* Obtain a plan from the CachedPlanSource. Any cruft from (re)planning
* will be generated in MessageContext. The plan refcount will be
* assigned to the Portal, so it will be released at portal destruction.
+ *
+ * For one-shot plans (used for unnamed prepared statements),
+ * BuildCachedPlan does not allocate a dedicated plan_context; the plan
+ * ends up in CurrentMemoryContext. Switch to the portal's context for
+ * the call so the plan survives MessageContext resets between Bind and
+ * Execute (e.g. in pipelined extended-protocol traffic).
*/
+ if (psrc->is_oneshot)
+ MemoryContextSwitchTo(portal->portalContext);
cplan = GetCachedPlan(psrc, params, NULL, NULL);
+ if (psrc->is_oneshot)
+ MemoryContextSwitchTo(MessageContext);
/*
* Now we can define the portal.
@@ -2906,8 +2941,16 @@ drop_unnamed_stmt(void)
{
CachedPlanSource *psrc = unnamed_stmt_psrc;
+ /*
+ * For one-shot plans DropCachedPlan does not free the context (it's
+ * owned by the caller), so capture it and delete it ourselves.
+ */
+ MemoryContext oneshot_ctx = psrc->is_oneshot ? psrc->context : NULL;
+
unnamed_stmt_psrc = NULL;
DropCachedPlan(psrc);
+ if (oneshot_ctx)
+ MemoryContextDelete(oneshot_ctx);
}
}
base-commit: 6aa26be288fa811270dfc1e39c015c23a97688b4
--
2.54.0
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: future of PQfn()
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 21:36 ` Re: future of PQfn() Jelte Fennema-Nio <[email protected]>
@ 2026-05-27 18:23 ` Nathan Bossart <[email protected]>
0 siblings, 0 replies; 18+ messages in thread
From: Nathan Bossart @ 2026-05-27 18:23 UTC (permalink / raw)
To: Jelte Fennema-Nio <[email protected]>; +Cc: pgsql-hackers
On Tue, May 26, 2026 at 11:36:49PM +0200, Jelte Fennema-Nio wrote:
> Could you share the simple LO test you were running here and/or rerun it
> with this patch applied? I'd love to know if the patch reduces the
> slowdown significantly, or if something else is the bottleneck.
The test is just:
int main()
{
PCconn *conn = PQsetdb(NULL, NULL, NULL, NULL, "postgres");
for (int i = 0; i < 1000000; i++)
lo_create(conn, i);
for (int i = 0; i < 1000000; i++)
lo_unlink(conn, i);
}
Before applying your patch -> 0.457 seconds (best of ~10)
After applying your patch -> 0.444 seconds (best of ~10)
For reference, HEAD runs this test in 0.319 seconds.
I've attached my work-in-progress patches here, in case you're interested.
0001 switches the frontend LO interface to use prepared statements, and
0002 removes PQfn() entirely. 0003 applies on top of those two and
switches the frontend LO interface to use PQexecParams() instead.
--
nathan
^ permalink raw reply [nested|flat] 18+ messages in thread
end of thread, other threads:[~2026-06-04 14:57 UTC | newest]
Thread overview: 18+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-05-26 16:05 future of PQfn() Nathan Bossart <[email protected]>
2026-05-26 16:53 ` Christoph Berg <[email protected]>
2026-05-26 16:55 ` Christoph Berg <[email protected]>
2026-05-26 17:42 ` Jacob Champion <[email protected]>
2026-05-26 19:55 ` Nathan Bossart <[email protected]>
2026-05-27 21:39 ` Jacob Champion <[email protected]>
2026-05-28 15:17 ` Nathan Bossart <[email protected]>
2026-05-29 15:14 ` Nathan Bossart <[email protected]>
2026-05-29 15:43 ` Jacob Champion <[email protected]>
2026-05-29 16:10 ` Nathan Bossart <[email protected]>
2026-05-29 16:33 ` Jacob Champion <[email protected]>
2026-05-29 16:42 ` Nathan Bossart <[email protected]>
2026-05-29 17:04 ` Jacob Champion <[email protected]>
2026-06-03 21:36 ` Nathan Bossart <[email protected]>
2026-06-04 10:39 ` Dagfinn Ilmari Mannsåker <[email protected]>
2026-06-04 14:57 ` Nathan Bossart <[email protected]>
2026-05-26 21:36 ` Jelte Fennema-Nio <[email protected]>
2026-05-27 18:23 ` Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox