pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Andres Freund <andres@anarazel.de>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: andrey.salnikov@dataegret.com
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16125: Crash of PostgreSQL's wal sender during logical replication
Date: Mon, 18 Nov 2019 14:24:16 -0800
Message-ID: <20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de> (raw)
In-Reply-To: <20191118205816.5wne6hxedrm7zd5p@development>
References: <16125-a78dafa032d53f7e@postgresql.org>
	<20191118205816.5wne6hxedrm7zd5p@development>

Hi,

On 2019-11-18 21:58:16 +0100, Tomas Vondra wrote:
> and the ReorderBufferToastReplace does this:
> 
>     newtup = change->data.tp.newtuple;
> 
>     heap_deform_tuple(&newtup->tuple, desc, attrs, isnull);
> 
> but that fails, because the tuple pointer happens to be 0x8, which is
> clearly bogus. Not sure where that comes from, I don't recognize that as
> a typical patter.

It indicates that change->data.tp.newtuple is NULL,
afaict. newtup->tuple boils down to
((char *) newtup->tuple) + offsetof(ReorderBufferTupleBuf, tuple)
and offsetof(ReorderBufferTupleBuf, tuple) is 0x8.


> Can you create a core dump (see [1]), and print 'change' and 'txn' in
> frame #2? I wonder if some the other fields are bogus too (but it can't
> be entirely true ...), and if the transaction got serialized.

Please print change and *change, both, please.

I suspect what's happening is that somehow a change that shouldn't have
toast changes - e.g. a DELETE - somehow has toast changes. Which then
triggers a failure in ReorderBufferToastReplace(), which expects
newtuple to be valid.

It's probably worthwhile to add an elog(ERROR) check for this, even if
this does not turn out to be the case.



> > This behaviour does not depends on defined data in tables, because we see it
> > in different database with different sets of tables in publications.
> 
> I'm not sure I really believe that. Surely there has to be something
> special about your schema, or possibly access patter that triggers this
> bug in your environment and not elsewhere.

Yea.  Are there any C triggers present? Any unusual extensions? Users of
the transaction hook, for example?


> > Looks like a real issue in logical replication.
> > I will happy to provide an additional information about that issue, but i
> > should know what else to need to collect for helping to solve this
> > problem.
> > 
> 
> Well, if you can create a reproducer, that'd be the best option, because
> then we can investigate locally instead of the ping-ping here.
> 
> But if that's not possible, let's start with the schema and the
> additional information from the core file.
> 
> I'd also like to see the contents of the WAL, particularly for the XID
> triggering this issue. Please run pg_waldump and see how much data is
> there for XID 1667601527. It does commit at 25EE/D6DE6EB8, not sure
> where it starts. It may have subtransactions, so don't do just grep.

Yea, that'd be helpful.

Greetings,

Andres Freund





view thread (17+ messages)  latest in thread

Message-ID: <20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de>
Permalink:  ../20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de/
Also on:    postgresql.org/message-id/20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-bugs@postgresql.org
  Cc: andres@anarazel.de, tomas.vondra@2ndquadrant.com, andrey.salnikov@dataegret.com
  Subject: Re: BUG #16125: Crash of PostgreSQL's wal sender during logical replication
  In-Reply-To: <20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de>
  References: <16125-a78dafa032d53f7e@postgresql.org> <20191118205816.5wne6hxedrm7zd5p@development> <20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox