public inbox for [email protected]
help / color / mirror / Atom feedFrom: Jeroen Vermeulen <[email protected]>
To: Jelte Fennema <[email protected]>
Cc: [email protected]
Cc: Bharath Rupireddy <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: libpq: PQgetCopyData() and allocation overhead
Date: Fri, 3 Mar 2023 17:16:05 +0100
Message-ID: <CA+zULE5aADii2Ju9sZ=J4idQ7Ey=Z=r3zgAQN4jeKZ=VcNt8rw@mail.gmail.com> (raw)
In-Reply-To: <CAGECzQQYS4V5wR27DV8x8oQosMAkW1D6gZDUETpc1jkCw6vSGQ@mail.gmail.com>
References: <CA+zULE5AYd+78w7Uu7jN=5N3w-9vsKuK2WrpjkKwS65hyhWfbw@mail.gmail.com>
<CALj2ACXiC3Wy7o=81j8jpY=nQq3Hs4y+06gXDkOBrmoO+JDrrw@mail.gmail.com>
<CA+zULE6bV9yg9+=u0KAhjz-C_YpDr6bh5YknkffgqDHiKWugOg@mail.gmail.com>
<CA+zULE56RADYr4R=ss=bvrzA2r6PyzWE98Gkkwr_7nOzKsSd-A@mail.gmail.com>
<CALj2ACVaBEbFnBwJ44Uuy7xwr50M+wwTJoEz0C9dFOc=iZeTDw@mail.gmail.com>
<CA+zULE6aB9Y_CgT_rmGFPpXUuEANmVXi7_RbrZLH7EPhzy3-OA@mail.gmail.com>
<CA+zULE5BoZtkYMf9yPGH+BnPEPspXENVCZd_b4s=ayewU6DHZg@mail.gmail.com>
<[email protected]>
<CA+zULE5M8CujFba55g4NkGSSgRhxLeUhZbX+Q2oBivBcrmYyzQ@mail.gmail.com>
<CAGECzQQYS4V5wR27DV8x8oQosMAkW1D6gZDUETpc1jkCw6vSGQ@mail.gmail.com>
Thank you.
I meant to try PQExpBuffer — as you say, it fits much better with existing
libpq style. But then I hit on the callback idea which was even more
efficient, by a fair margin. It was also considerably simpler both inside
libpq and in the client code, eliminating all sorts of awkward questions
about who deallocates the buffer in which situations. So I ditched the
"dynamic buffer" concept and went with the callback.
One other possible alternative suggests itself: instead of taking a
callback and a context pointer, the function could probably just return a
struct: status/size, and buffer. Then the caller would have to figure out
whether there's a line in the buffer, and if so, process it. It seems like
more work for the client code, but it may make the compiler's optimisation
work easier.
Jeroen
On Fri, 3 Mar 2023 at 16:52, Jelte Fennema <[email protected]> wrote:
> On Thu, 2 Mar 2023 at 20:45, Jeroen Vermeulen <[email protected]> wrote:
> > I'm attaching a diff now. This is not a patch, it's just a discussion
> piece.
>
> Did you try with PQExpBuffer? I still think that probably fits better
> in the API design of libpq. Obviously if it's significantly slower
> than the callback approach in this patch then it's worth considering
> using the callback approach. Overall it definitely seems reasonable to
> me to have an API that doesn't do an allocation per row.
>
view thread (2+ messages)
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: Re: libpq: PQgetCopyData() and allocation overhead
In-Reply-To: <CA+zULE5aADii2Ju9sZ=J4idQ7Ey=Z=r3zgAQN4jeKZ=VcNt8rw@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox