IronPort-SDR: 
 ZeLYngJ8ODW3hGgumRG+l6B4rjLAlXkQKqGoM5p4/8FGkLALKC4nFFEatyHziP8mEDeYBe57rv
 gHZvl48jGk8A==
Subject: Re: logical decoding bug: segfault in ReorderBufferToastReplace()
To: Andres Freund <andres@anarazel.de>, "Drouvot, Bertrand"
	<bdrouvot@amazon.com>
CC: Tomas Vondra <tomas.vondra@2ndquadrant.com>, "pgsql-bugs@postgresql.org"
	<pgsql-bugs@postgresql.org>, PostgreSQL-development
	<pgsql-hackers@postgresql.org>
References: <E1i74JE-00024V-9J@gemulon.postgresql.org>
 <b1da9abc-bad7-efd6-2f1c-503d508c3877@amazon.com>
 <20191205133836.lulisfo2fphdrjp3@development>
 <EEB686D3-F8A7-4371-9A96-5DF3B72A7734@amazon.com>
 <20191211163522.t2ljtydarlmusoix@alap3.anarazel.de>
From: Jeremy Schneider <schnjere@amazon.com>
Message-ID: <81182626-e836-061f-8f19-204edac18922@amazon.com>
Date: Fri, 13 Dec 2019 16:13:35 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:68.0)
 Gecko/20100101 Thunderbird/68.2.2
MIME-Version: 1.0
In-Reply-To: <20191211163522.t2ljtydarlmusoix@alap3.anarazel.de>
Content-Type: multipart/alternative;
	boundary="------------4CC362CE056568EFE0D522B1"
Content-Language: en-US
Precedence: bulk

--------------4CC362CE056568EFE0D522B1
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit

On 12/11/19 08:35, Andres Freund wrote:
> I think we need to see pg_waldump output for the preceding records. That
> might allow us to see why there's a toast record that's being associated
> with this table, despite there not being a toast table.
Unfortunately the WAL logs are no longer available at this time.  :(

I did a little poking around in the core file and searching source code
but didn't find anything yet.  Is there any memory structure that would
have the preceding/following records cached in memory?  If so then I
might be able to extract this from the core dumps.

> Seems like we clearly should add an elog(ERROR) here, so we error out,
> rather than crash.
done - in the commit that I replied to when I started this thread :)

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=69f883fef14a3fc5849126799278abcc43f40f56

> Has there been DDL to this table?
I'm not sure that we will be able to find out at this point. 

> Could you print out *change?

This was also in the original email - here it is:

(gdb) print *change
$1 = {lsn = 9430473343416, action = REORDER_BUFFER_CHANGE_INSERT,
origin_id = 0, data = {tp = {relnode = {spcNode = 1663, dbNode = 16401,
        relNode = 16428}, clear_toast_afterwards = true, oldtuple = 0x0,
newtuple = 0x2b79313f9c68}, truncate = {
      nrelids = 70441758623359, cascade = 44, restart_seqs = 64, relids
= 0x0}, msg = {
      prefix = 0x40110000067f <Address 0x40110000067f out of bounds>,
message_size = 4294983724, message = 0x0},
    snapshot = 0x40110000067f, command_id = 1663, tuplecid = {node =
{spcNode = 1663, dbNode = 16401, relNode = 16428}, tid = {
        ip_blkid = {bi_hi = 1, bi_lo = 0}, ip_posid = 0}, cmin = 0, cmax
= 826252392, combocid = 11129}}, node = {prev = 0x30ac918,
    next = 0x30ac9b8}}

> Is this version of postgres effectively unmodified in any potentially
> relevant region (snapshot computations, generation of WAL records, ...)?
It's not changed from community code in any relevant regions.  (Also,
FYI, this is not Aurora.)

-Jeremy

-- 
Jeremy Schneider
Database Engineer
Amazon Web Services


--------------4CC362CE056568EFE0D522B1
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 12/11/19 08:35, Andres Freund wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:20191211163522.t2ljtydarlmusoix@alap3.anarazel.de">
      <pre class="moz-quote-pre" wrap="">I think we need to see pg_waldump output for the preceding records. That
might allow us to see why there's a toast record that's being associated
with this table, despite there not being a toast table.</pre>
    </blockquote>
    Unfortunately the WAL logs are no longer available at this time.  :(<br>
    <br>
    I did a little poking around in the core file and searching source
    code but didn't find anything yet.  Is there any memory structure
    that would have the preceding/following records cached in memory? 
    If so then I might be able to extract this from the core dumps.<br>
    <br>
    <blockquote type="cite"
      cite="mid:20191211163522.t2ljtydarlmusoix@alap3.anarazel.de">
      <pre class="moz-quote-pre" wrap="">Seems like we clearly should add an elog(ERROR) here, so we error out,
rather than crash.</pre>
    </blockquote>
    done - in the commit that I replied to when I started this thread :)<br>
    <br>
<a class="moz-txt-link-freetext" href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=69f883fef14a3fc5849126799278abcc43f40f56">https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=69f883fef14a3fc5849126799278abcc43f40f56</a><br>
    <br>
    <blockquote type="cite"
      cite="mid:20191211163522.t2ljtydarlmusoix@alap3.anarazel.de">
      <pre class="moz-quote-pre" wrap="">Has there been DDL to this table?</pre>
    </blockquote>
    I'm not sure that we will be able to find out at this point.  <br>
    <br>
    <blockquote type="cite"
      cite="mid:20191211163522.t2ljtydarlmusoix@alap3.anarazel.de">
      <pre class="moz-quote-pre" wrap="">Could you print out *change?</pre>
    </blockquote>
    <br>
    This was also in the original email - here it is:<br>
    <br>
    (gdb) print *change<br>
    $1 = {lsn = 9430473343416, action = REORDER_BUFFER_CHANGE_INSERT,
    origin_id = 0, data = {tp = {relnode = {spcNode = 1663, dbNode =
    16401,<br>
            relNode = 16428}, clear_toast_afterwards = true, oldtuple =
    0x0, newtuple = 0x2b79313f9c68}, truncate = {<br>
          nrelids = 70441758623359, cascade = 44, restart_seqs = 64,
    relids = 0x0}, msg = {<br>
          prefix = 0x40110000067f &lt;Address 0x40110000067f out of
    bounds&gt;, message_size = 4294983724, message = 0x0},<br>
        snapshot = 0x40110000067f, command_id = 1663, tuplecid = {node =
    {spcNode = 1663, dbNode = 16401, relNode = 16428}, tid = {<br>
            ip_blkid = {bi_hi = 1, bi_lo = 0}, ip_posid = 0}, cmin = 0,
    cmax = 826252392, combocid = 11129}}, node = {prev = 0x30ac918,<br>
        next = 0x30ac9b8}}<br>
    <br>
    <blockquote type="cite"
      cite="mid:20191211163522.t2ljtydarlmusoix@alap3.anarazel.de">
      <pre class="moz-quote-pre" wrap="">Is this version of postgres effectively unmodified in any potentially
relevant region (snapshot computations, generation of WAL records, ...)?
</pre>
    </blockquote>
    It's not changed from community code in any relevant regions. 
    (Also, FYI, this is not Aurora.)<br>
    <br>
    -Jeremy<br>
    <br>
    <pre class="moz-signature" cols="72">-- 
Jeremy Schneider
Database Engineer
Amazon Web Services</pre>
  </body>
</html>

--------------4CC362CE056568EFE0D522B1--