public inbox for [email protected]
help / color / mirror / Atom feedFrom: Michael Paquier <[email protected]>
To: Masahiko Sawada <[email protected]>
Cc: Sutou Kouhei <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Subject: Re: Make COPY format extendable: Extract COPY TO format implementations
Date: Fri, 2 May 2025 08:03:50 +0900
Message-ID: <[email protected]> (raw)
In-Reply-To: <CAD21AoBuEqcz2_+dpA3WTiDUF=FgudPBKwM+nvH+qHT-k4p5mA@mail.gmail.com>
References: <CAD21AoDOcYah-nREv09BB3ZoB-k+Yf1XUfJcDMoq=LLvV1v75w@mail.gmail.com>
<[email protected]>
<CAD21AoAXzwPC7jjPMTcT80hnzmPa2SUJkiqdYHweEY8sZscEMA@mail.gmail.com>
<[email protected]>
<CAD21AoBuEqcz2_+dpA3WTiDUF=FgudPBKwM+nvH+qHT-k4p5mA@mail.gmail.com>
On Thu, May 01, 2025 at 12:15:30PM -0700, Masahiko Sawada wrote:
> In light of these concerns, I've been contemplating alternative
> interface designs. One promising approach would involve registering
> custom copy formats via a C function during module loading
> (specifically, in _PG_init()). This method would require extension
> authors to invoke a registration function, say
> RegisterCustomCopyFormat(), in _PG_init() as follows:
>
> JsonLinesFormatId = RegisterCustomCopyFormat("jsonlines",
> &JsonLinesCopyToRoutine,
> &JsonLinesCopyFromRoutine);
>
> The registration function would validate the format name and store it
> in TopMemoryContext. It would then return a unique identifier that can
> be used subsequently to reference the custom copy format extension.
Hmm. How much should we care about the observability of the COPY
format used by a given backend? Storing this information in a
backend's TopMemoryContext is OK to get the extensibility basics to
work, but could it make sense to use some shmem state to allocate a
uint32 ID that could be shared by all backends. Contrary to EXPLAIN,
COPY commands usually run for a very long time, so I am wondering if
these APIs should be designed so as it would be possible to monitor
the format used. One layer where the format information could be made
available is the progress reporting view for COPY, for example. I can
also imagine a pgstats kind where we do COPY stats aggregates, with a
per-format pgstats kind, and sharing a fixed ID across multiple
backends is relevant (when flushing the stats at shutdown, we would
use a name/ID mapping like replication slots).
I don't think that this needs to be relevant for the option part, just
for the format where, I suspect, we should store in a shmem array
based on the ID allocated the name of the format, the library of the
callback and the function name fed to load_external_function().
Note that custom LWLock and wait events use a shmem state for
monitoring purposes, where we are able to do ID->format name lookups
as much as format->ID lookups. Perhaps it's OK not to do that for
COPY, but I am wondering if we'd better design things from scratch
with states in shmem state knowing that COPY is a long-running
operation, and that if one mixes multiple formats they would most
likely want to know which formats are bottlenecks, through SQL. Cloud
providers would love that.
> This approach offers several advantages: it would eliminate the
> search_path issue, provide greater flexibility, and potentially
> simplify the overall interface for users and developers alike. We
> might be able to provide a view showing the registered custom COPY
> format in the future. Also, these interfaces align with other
> customizable functionalities such as custom rmgr, custom lwlock,
> custom waitevent, and custom EXPLAIN option etc.
Yeah, agreed with the search_path concerns. We are getting better at
making areas of Postgres more pluggable lately, having a loading path
where we don't have any of these potential issues by design matters.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../[email protected]/2-signature.asc)
download
view thread (25+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Make COPY format extendable: Extract COPY TO format implementations
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox