Message-ID: <1adf8fee-5a9a-69c2-057a-87bbddebab9d@gmail.com>
Date: Fri, 24 Nov 2023 07:00:01 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.2
Subject: Re: BUG #17828: postgres_fdw leaks file descriptors on error and
 aborts aborted transaction in lack of fds
Content-Language: en-US
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-bugs@lists.postgresql.org, Etsuro Fujita <etsuro.fujita@gmail.com>
References: <17828-122da8cba23236be@postgresql.org>
 <423731.1678381075@sss.pgh.pa.us>
From: Alexander Lakhin <exclusion@gmail.com>
In-Reply-To: <423731.1678381075@sss.pgh.pa.us>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: 
 <https://www.postgresql.org/message-id/1adf8fee-5a9a-69c2-057a-87bbddebab9d%40gmail.com>
Precedence: bulk

Hello,

09.03.2023 19:57, Tom Lane wrote:
> PG Bug reporting form <noreply@postgresql.org> writes:
>> The following script:
>> [ leaks a file descriptor per error ]
> Yeah, at least on platforms where WaitEventSets own kernel file
> descriptors.  I don't think it's postgres_fdw's fault though,
> but that of ExecAppendAsyncEventWait, which is ignoring the
> possibility of failing partway through.  It looks like it'd be
> sufficient to add a PG_CATCH or PG_FINALLY block there to make
> sure the WaitEventSet is disposed of properly --- fortunately,
> it doesn't need to have any longer lifespan than that one
> function.

Now that the leakage eliminated by 50c67c201/481d7d1c0 we still can observe
the assert-triggering half of the bug with something like that:
echo "
CREATE EXTENSION postgres_fdw;
SELECT id INTO t FROM generate_series(1,100000) id;
" | psql

ns=322 # assuming "ulimit -n" = 1024 and max_safe_fds / 3 = 329
for ((i=1;i<=$ns;i++)); do
echo "
DO \$d\$
   BEGIN
     EXECUTE \$\$CREATE SERVER loopback_$i FOREIGN DATA WRAPPER postgres_fdw
       OPTIONS (
         dbname '\$\$||current_database()||\$\$',
         port '\$\$||current_setting('port')||\$\$'
       )\$\$;
   END;
\$d\$;

CREATE USER MAPPING FOR CURRENT_USER SERVER loopback_$i;
CREATE FOREIGN TABLE ft_$i (id int) SERVER loopback_$i OPTIONS (table_name 't');
"
done | psql >psql-1.log

(echo "BEGIN;";
for ((i=1;i<=$ns;i++)); do echo "DECLARE cursor_$i CURSOR FOR SELECT * FROM ft_$i;"; done
echo "SELECT * INTO nt FROM ft_1 UNION SELECT * FROM ft_2;"
) | psql >psql-2.log

That leads to:
2023-11-24 03:03:50.281 UTC [1053419] ERROR:  epoll_create1 failed: Too many open files
2023-11-24 03:03:50.281 UTC [1053419] STATEMENT:  SELECT * INTO nt FROM ft_1 UNION SELECT * FROM ft_2;
2023-11-24 03:03:50.282 UTC [1053419] ERROR:  epoll_create1 failed: Too many open files
2023-11-24 03:03:50.282 UTC [1053419] WARNING:  AbortTransaction while in ABORT state
TRAP: failed Assert("TransactionIdIsValid(proc->xid)"), File: "procarray.c", Line: 677, PID: 1053419

Call chains for the errors are the same as in the initial report [1].

[1] https://www.postgresql.org/message-id/17828-122da8cba23236be%40postgresql.org

Best regards,
Alexander