Subject: Re: logical decoding bug: segfault in ReorderBufferToastReplace()
To: Amit Kapila <amit.kapila16@gmail.com>
CC: Jeremy Schneider <schnjere@amazon.com>, Alvaro Herrera
	<alvherre@alvh.no-ip.org>, Andres Freund <andres@anarazel.de>, Tomas Vondra
	<tomas.vondra@2ndquadrant.com>, PostgreSQL-development
	<pgsql-hackers@postgresql.org>
References: <c41e5883-aedf-7944-010e-0b48b6c677ba@amazon.com>
 <202106042335.o64q6dui5hu2@alvherre.pgsql>
 <CAA4eK1+LG8rpwTt0F8M+JPEWMET0mP31YiGaMuEC_VCm1=8RrQ@mail.gmail.com>
 <444215b4-8fb5-6a82-a534-645abafbffb4@amazon.com>
 <CAA4eK1+deBBnMVPxfLuv1Aa7tuh-7e3FvnSvWTaCy4-_HPcBLg@mail.gmail.com>
 <8d45ed72-1706-8b21-75a6-a3420bd15b04@amazon.com>
 <CAA4eK1KcUPwwhDVhJmdQExc09AzEBZMGbOa-u3DYaJs1zzfEnA@mail.gmail.com>
From: "Drouvot, Bertrand" <bdrouvot@amazon.com>
Message-ID: <5f9a118e-86c5-d4f1-b584-199a33757cd4@amazon.com>
Date: Wed, 9 Jun 2021 08:17:28 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0)
 Gecko/20100101 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: 
 <CAA4eK1KcUPwwhDVhJmdQExc09AzEBZMGbOa-u3DYaJs1zzfEnA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Archived-At: 
 <https://www.postgresql.org/message-id/5f9a118e-86c5-d4f1-b584-199a33757cd4%40amazon.com>
Precedence: bulk

Hi,

On 6/9/21 8:10 AM, Amit Kapila wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> On Wed, Jun 9, 2021 at 11:37 AM Drouvot, Bertrand <bdrouvot@amazon.com> wrote:
>> On 6/9/21 5:33 AM, Amit Kapila wrote:
>>> On Wed, Jun 9, 2021 at 12:06 AM Jeremy Schneider <schnjere@amazon.com> wrote:
>>>> On 6/4/21 23:42, Amit Kapila wrote:
>>>>
>>>> On 2021-Jun-04, Jeremy Schneider wrote:
>>>>
>>>> ERROR: XX000: could not open relation with OID 0
>>>> LOCATION: ReorderBufferToastReplace, reorderbuffer.c:305
>>>>
>>>> Even, if this fixes the issue, I guess it is better to find why this
>>>> happens? I think the reason why the code is giving an error is that
>>>> after toast insertions we always expect the insert on the main table
>>>> of toast table, but if there happens to be a case where after toast
>>>> insertion, instead of getting the insertion on the main table we get
>>>> an insert in some other table then you will see this error. I think
>>>> this can happen for speculative insertions where insertions lead to a
>>>> toast table insert, then we get a speculative abort record, and then
>>>> insertion on some other table. The main thing is currently decoding
>>>> code ignores speculative aborts due to which such a problem can occur.
>>>> Now, there could be other cases where such a problem can happen but if
>>>> my theory is correct then the patch we are discussing in the thread
>>>> [1] should solve this problem.
>>>>
>>>> Jeremy, is this problem reproducible? Can we get a testcase or
>>>> pg_waldump output of previous WAL records?
>>>>
>>>> [1] - https://www.postgresql.org/message-id/CAExHW5sPKF-Oovx_qZe4p5oM6Dvof7_P%2BXgsNAViug15Fm99jA%40mail.gmail.com
>>>>
>>>>
>>>> It's unclear to me whether or not we'll be able to catch the repro on the actual production system. It seems that we are hitting this somewhat consistently, but at irregular and infrequent intervals. If we are able to catch it and walk the WAL records then I'll post back here.
>>>>
>>> Okay, one thing you can check is if there is a usage of Insert .. On
>>> Conflict .. statement in the actual production system?
>> Yes that's the case, so that a speculative abort record followed by an
>> insert on some other table looks a perfect valid scenario regarding this
>> current issue.
>>
> Okay, thanks for the confirmation. So the patch being discussed in
> that thread will fix your problem.

Yes, thanks a lot!

Bertrand