Subject: Re: logical decoding bug: segfault in ReorderBufferToastReplace()
To: Amit Kapila <amit.kapila16@gmail.com>, Alvaro Herrera
	<alvherre@alvh.no-ip.org>
CC: Andres Freund <andres@anarazel.de>, "Drouvot, Bertrand"
	<bdrouvot@amazon.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>,
	PostgreSQL-development <pgsql-hackers@postgresql.org>
References: <c41e5883-aedf-7944-010e-0b48b6c677ba@amazon.com>
 <202106042335.o64q6dui5hu2@alvherre.pgsql>
 <CAA4eK1+LG8rpwTt0F8M+JPEWMET0mP31YiGaMuEC_VCm1=8RrQ@mail.gmail.com>
From: Jeremy Schneider <schnjere@amazon.com>
Message-ID: <444215b4-8fb5-6a82-a534-645abafbffb4@amazon.com>
Date: Tue, 8 Jun 2021 11:35:57 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0)
 Gecko/20100101 Thunderbird/78.10.2
MIME-Version: 1.0
In-Reply-To: 
 <CAA4eK1+LG8rpwTt0F8M+JPEWMET0mP31YiGaMuEC_VCm1=8RrQ@mail.gmail.com>
Content-Type: multipart/alternative;
	boundary="------------C824EC3AE6E8CA5E272A6A48"
Content-Language: en-US
Archived-At: 
 <https://www.postgresql.org/message-id/444215b4-8fb5-6a82-a534-645abafbffb4%40amazon.com>
Precedence: bulk

--------------C824EC3AE6E8CA5E272A6A48
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

On 6/4/21 23:42, Amit Kapila wrote:
> On 2021-Jun-04, Jeremy Schneider wrote:
>>> ERROR: XX000: could not open relation with OID 0
>>> LOCATION: ReorderBufferToastReplace, reorderbuffer.c:305
> Even, if this fixes the issue, I guess it is better to find why this
> happens? I think the reason why the code is giving an error is that
> after toast insertions we always expect the insert on the main table
> of toast table, but if there happens to be a case where after toast
> insertion, instead of getting the insertion on the main table we get
> an insert in some other table then you will see this error. I think
> this can happen for speculative insertions where insertions lead to a
> toast table insert, then we get a speculative abort record, and then
> insertion on some other table. The main thing is currently decoding
> code ignores speculative aborts due to which such a problem can occur.
> Now, there could be other cases where such a problem can happen but if
> my theory is correct then the patch we are discussing in the thread
> [1] should solve this problem.
>
> Jeremy, is this problem reproducible? Can we get a testcase or
> pg_waldump output of previous WAL records?
>
> [1] - https://www.postgresql.org/message-id/CAExHW5sPKF-Oovx_qZe4p5oM6Dvof7_P%2BXgsNAViug15Fm99jA%40mail.gmail.com

It's unclear to me whether or not we'll be able to catch the repro on
the actual production system. It seems that we are hitting this somewhat
consistently, but at irregular and infrequent intervals. If we are able
to catch it and walk the WAL records then I'll post back here. FYI,
Bertrand was able to replicate the exact error message with pretty much
the same repro that's in the other email thread which is linked above.

Separately, would there be any harm in adding the relation OID to the
error message? Personally, I just think the error message is generally
more useful if it shows the main relation OID (since we know that the
toast OID can be 0). Not a big deal though.

-Jeremy


-- 
Jeremy Schneider
Database Engineer
Amazon Web Services


--------------C824EC3AE6E8CA5E272A6A48
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 6/4/21 23:42, Amit Kapila wrote:
    </div>
    <blockquote type="cite"
cite="mid:CAA4eK1+LG8rpwTt0F8M+JPEWMET0mP31YiGaMuEC_VCm1=8RrQ@mail.gmail.com">
      <pre class="moz-quote-pre" wrap="">On 2021-Jun-04, Jeremy Schneider wrote:
</pre>
      <blockquote type="cite">
        <blockquote type="cite">
          <pre class="moz-quote-pre" wrap="">ERROR: XX000: could not open relation with OID 0
LOCATION: ReorderBufferToastReplace, reorderbuffer.c:305
</pre>
        </blockquote>
      </blockquote>
      <pre class="moz-quote-pre" wrap="">
Even, if this fixes the issue, I guess it is better to find why this
happens? I think the reason why the code is giving an error is that
after toast insertions we always expect the insert on the main table
of toast table, but if there happens to be a case where after toast
insertion, instead of getting the insertion on the main table we get
an insert in some other table then you will see this error. I think
this can happen for speculative insertions where insertions lead to a
toast table insert, then we get a speculative abort record, and then
insertion on some other table. The main thing is currently decoding
code ignores speculative aborts due to which such a problem can occur.
Now, there could be other cases where such a problem can happen but if
my theory is correct then the patch we are discussing in the thread
[1] should solve this problem.

Jeremy, is this problem reproducible? Can we get a testcase or
pg_waldump output of previous WAL records?

[1] - <a class="moz-txt-link-freetext" href="https://www.postgresql.org/message-id/CAExHW5sPKF-Oovx_qZe4p5oM6Dvof7_P%2BXgsNAViug15Fm99jA%40mail.gmail.com">https://www.postgresql.org/message-id/CAExHW5sPKF-Oovx_qZe4p5oM6Dvof7_P%2BXgsNAViug15Fm99jA%40mail.gmail.com</a>
</pre>
    </blockquote>
    <br>
    It's unclear to me whether or not we'll be able to catch the repro
    on the actual production system. It seems that we are hitting this
    somewhat consistently, but at irregular and infrequent intervals. If
    we are able to catch it and walk the WAL records then I'll post back
    here. FYI, Bertrand was able to replicate the exact error message
    with pretty much the same repro that's in the other email thread
    which is linked above.<br>
    <br>
    Separately, would there be any harm in adding the relation OID to
    the error message? Personally, I just think the error message is
    generally more useful if it shows the main relation OID (since we
    know that the toast OID can be 0). Not a big deal though.<br>
    <br>
    -Jeremy<br>
    <br>
    <br>
    <pre class="moz-signature" cols="72">-- 
Jeremy Schneider
Database Engineer
Amazon Web Services</pre>
  </body>
</html>

--------------C824EC3AE6E8CA5E272A6A48--