From: Tom Lane <tgl@sss.pgh.pa.us>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: logical decoding / rewrite map vs. maxAllocatedDescs
In-reply-to: <738a590a-2ce5-9394-2bef-7b1caad89b37@2ndquadrant.com>
References: <738a590a-2ce5-9394-2bef-7b1caad89b37@2ndquadrant.com>
Comments: In-reply-to Tomas Vondra <tomas.vondra@2ndquadrant.com>
	message dated "Thu, 09 Aug 2018 19:31:00 +0200"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <12169.1533836537.1@sss.pgh.pa.us>
Date: Thu, 09 Aug 2018 13:42:17 -0400
Message-ID: <12170.1533836537@sss.pgh.pa.us>
Precedence: bulk

Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
> While investigating an issue with rewrite maps in logical decoding, I 
> found it's pretty darn trivial to hit this:
>      ERROR:  53000: exceeded maxAllocatedDescs (492) while trying to open
>      file "pg_logical/mappings/map-4000-4eb-1_60DE1E08-5376b5-537c6b"

> This happens because we seem to open the mapping files and keep them 
> open until the next checkpoint, at which point we fsync + close.

It sounds like whoever wrote that code was completely impervious to the
API spec for AllocateFile():

 * Note that files that will be open for any significant length of time
 * should NOT be handled this way, since they cannot share kernel file
 * descriptors with other files; there is grave risk of running out of FDs
 * if anyone locks down too many FDs.

Perhaps these files should be accessed through fd.c's VFD infrastructure
rather than ignoring it.  Reinventing it is especially not the way to go.

> Furthermore, the situation is made worse by opening the same file 
> repeatedly, under different file descriptor.

That's just stupid :-(

Note that the limit on maxAllocatedDescs is based on the kernel limit
on how many files a process can have open, so assuming you can increase
it is not going to end well.

			regards, tom lane