Subject: Re: logical decoding / rewrite map vs. maxAllocatedDescs
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>,
 Alvaro Herrera <alvherre@2ndquadrant.com>,
 PostgreSQL-development <pgsql-hackers@postgresql.org>
References: <20180809174746.aa6rbpsu4zjipzif@alvherre.pgsql>
 <b346c3eb-0e65-29ec-47db-0d9a46feaa2c@2ndquadrant.com>
 <20180810211340.nlgyfa7jlmkjcs5m@alap3.anarazel.de>
 <79a0c7f4-89fb-cf8d-82eb-5a7ca8e08d56@2ndquadrant.com>
 <6899.1533996956@sss.pgh.pa.us>
 <53047b69-9af7-1399-d190-6d7db60a7dd8@2ndquadrant.com>
Message-ID: <82d51b43-3b98-38e5-2e37-a8959b13ef49@2ndquadrant.com>
Date: Thu, 16 Aug 2018 17:07:17 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.5.2
MIME-Version: 1.0
In-Reply-To: <53047b69-9af7-1399-d190-6d7db60a7dd8@2ndquadrant.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 08/11/2018 04:18 PM, Tomas Vondra wrote:
> On 08/11/2018 04:15 PM, Tom Lane wrote:
>> Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
>>>>> On 08/09/2018 07:47 PM, Alvaro Herrera wrote:
>>>>>> Actually, it seems to me that ApplyLogicalMappingFile is just leaking
>>>>>> the file descriptor for no good reason.
>>
>>> I think the fix can be as simple as attached ... I'm mostly afk for the
>>> weekend, so I'll commit & backpatch on Monday or so.
>>
>> LGTM.  While you're at it, would you fix the misspelling three lines
>> below this?
>>
>>  * Check whether the TransactionOId 'xid' is in the pre-sorted array 'xip'.
>>                                 ^
> 
> Sure.
> 

I've pushed this, and backpatched it all the way back to 9.4 where
logical decoding was introduced.

While this resolves the way to run out of file descriptors, I wonder if
there are other trivial ways to trigger it (say, long-running
transaction spanning many 'vacuum full pg_class' runs). Not sure, will
try later.

The other question is whether errors are handled correctly - as reported
initially, I've seen this to trigger

ERROR:  XX000: subtransaction logged without previous top-level txn record

I assume just fixing the error did not really fix that, so that if
something else fails we might end up in the same state.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services