From: Alvaro Herrera from 2ndQuadrant <alvherre@alvh.no-ip.org>
To: Amit Khandekar <amitdkhan.pg@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: logical decoding : exceeded maxAllocatedDescs for .spill files
Date: Wed, 11 Sep 2019 09:51:40 -0300
Message-ID: <20190911125140.GA21034@alvherre.pgsql> (raw)
In-Reply-To: <CAJ3gD9c-sECEn79zXw4yBnBdOttacoE-6gAyP0oy60nfs_sabQ@mail.gmail.com>
On 2019-Sep-11, Amit Khandekar wrote:
> I reproduced the error "exceeded maxAllocatedDescs (492) while trying
> to open file ...", which was also discussed about in the thread [1].
> This issue is similar but not exactly the same as [1]. In [1], the
> file for which this error used to show up was
> "pg_logical/mappings/map...." , while here it's the .spill file. And
> here the issue , in short, seems to be : The .spill file does not get
> closed there and then, unlike in [1] where there was a file descriptor
> leak.
Uh-oh :-( Thanks for the reproducer -- I confirm it breaks things.
> Looking at the code, what might be happening is,
> ReorderBufferIterTXNInit()=>ReorderBufferRestoreChanges() opens the
> files, but leaves them open if end of file is not reached. Eventually
> if end of file is reached, it gets closed. The function returns back
> without closing the file descriptor if reorder buffer changes being
> restored are more than max_changes_in_memory. Probably later on, the
> rest of the changes get restored in another
> ReorderBufferRestoreChanges() call. But meanwhile, if there are a lot
> of such files opened, we can run out of the max files that PG decides
> to keep open (it has some logic that takes into account system files
> allowed to be open, and already opened).
Makes sense.
> Offhand, what I am thinking is, we need to close the file descriptor
> before returning from ReorderBufferRestoreChanges(), and keep track of
> the file offset and file path, so that next time we can resume reading
> from there.
I think doing this all the time would make restore very slow -- there's
a reason we keep the files open, after all. It would be better if we
can keep the descriptors open as much as possible, and only close them
if there's trouble. I was under the impression that using
OpenTransientFile was already taking care of that, but that's evidently
not the case.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: alvherre@alvh.no-ip.org, amitdkhan.pg@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: logical decoding : exceeded maxAllocatedDescs for .spill files
In-Reply-To: <20190911125140.GA21034@alvherre.pgsql>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox