Date: Mon, 24 Feb 2020 10:41:21 -0800
From: Andres Freund <andres@anarazel.de>
To: Mark Dilger <mark.dilger@enterprisedb.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Thomas Munro <thomas.munro@gmail.com>,
	=?utf-8?B?UsOpbWk=?= Zara <remi_zara@mac.com>,
	Michael Paquier <michael@paquier.xyz>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: pgsql: Add kqueue(2) support to the WaitEventSet API.
Message-ID: <20200224184121.725gnltgycvnw3xu@alap3.anarazel.de>
References: 
 <CA+hUKG+yGQDPdwcH5euDb_pE7OcnbrvGDLoOi2JMaCwj4G+ttg@mail.gmail.com>
 <08364CCB-07F5-423E-8CE0-ABF0195C4CA2@mac.com>
 <17106.1582224047@sss.pgh.pa.us>
 <19710.1582227844@sss.pgh.pa.us>
 <20263.1582228617@sss.pgh.pa.us>
 <CA+hUKGLh5NSUhPmipF+3n2-y_ncuVWu3DSg7vKhvY2HwxEBiNQ@mail.gmail.com>
 <26174.1582236314@sss.pgh.pa.us>
 <6610.1582242943@sss.pgh.pa.us>
 <18091.1582259407@sss.pgh.pa.us>
 <13F92C31-0E60-45C9-9031-3B1ECFDD5261@enterprisedb.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <13F92C31-0E60-45C9-9031-3B1ECFDD5261@enterprisedb.com>
Precedence: bulk

Hi,

On 2020-02-24 10:29:51 -0800, Mark Dilger wrote:
> > On Feb 20, 2020, at 8:30 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > 
> > This idea doesn't fix every possible problem.  For instance, if you
> > have a plperlu function that wants to open a bunch of files, I don't
> > see any easy way to ensure we release VFDs to make that possible.
> > But it's sure an improvement on the status quo.
> 
> I understand that you were using plperlu just as an example, but it
> got me thinking.  Could we ship a wrapper using perl's tie() mechanism
> to call a new spi function to report when a file handle is opened and
> when it is closed?  Most plperlu functions would not participate,
> since developers will not necessarily know to use the wrapper, but at
> least they could learn about the wrapper and use it as a work-around
> if this becomes a problem for them.  Perhaps the same spi function
> could be used by other procedural languages.

While we're thinking a bit outside of the box: We could just dup() a
bunch of fds within fd.c to protect fd.c's fd "quota". And then close
them when actually needing the fds.

Not really suggesting that we go for that, but it does have some appeal.


> I can't see this solution working unless the backend can cleanup properly under exceptional conditions, and decrement the counter of used file handles appropriately.  But that's the same requirement that postgres_fdw would also have, right?  Would the same mechanism work for both?

We can just throw an error, and all fdw state should get cleaned up. We
can't generally rely on that for plperl, as it IIRC can global state. So
I don't think they're in the same boat.

Greetings,

Andres Freund