From: Tom Lane <tgl@sss.pgh.pa.us>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: logical decoding / rewrite map vs. maxAllocatedDescs
Date: Thu, 09 Aug 2018 13:42:17 -0400
Message-ID: <12170.1533836537@sss.pgh.pa.us> (raw)
In-Reply-To: <738a590a-2ce5-9394-2bef-7b1caad89b37@2ndquadrant.com>
References: <738a590a-2ce5-9394-2bef-7b1caad89b37@2ndquadrant.com>
Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
> While investigating an issue with rewrite maps in logical decoding, I
> found it's pretty darn trivial to hit this:
> ERROR: 53000: exceeded maxAllocatedDescs (492) while trying to open
> file "pg_logical/mappings/map-4000-4eb-1_60DE1E08-5376b5-537c6b"
> This happens because we seem to open the mapping files and keep them
> open until the next checkpoint, at which point we fsync + close.
It sounds like whoever wrote that code was completely impervious to the
API spec for AllocateFile():
* Note that files that will be open for any significant length of time
* should NOT be handled this way, since they cannot share kernel file
* descriptors with other files; there is grave risk of running out of FDs
* if anyone locks down too many FDs.
Perhaps these files should be accessed through fd.c's VFD infrastructure
rather than ignoring it. Reinventing it is especially not the way to go.
> Furthermore, the situation is made worse by opening the same file
> repeatedly, under different file descriptor.
That's just stupid :-(
Note that the limit on maxAllocatedDescs is based on the kernel limit
on how many files a process can have open, so assuming you can increase
it is not going to end well.
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tgl@sss.pgh.pa.us, tomas.vondra@2ndquadrant.com
Subject: Re: logical decoding / rewrite map vs. maxAllocatedDescs
In-Reply-To: <12170.1533836537@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox