From: Andres Freund <andres@anarazel.de>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: andrey.salnikov@dataegret.com
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16125: Crash of PostgreSQL's wal sender during logical replication
Date: Mon, 18 Nov 2019 14:24:16 -0800
Message-ID: <20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de> (raw)
In-Reply-To: <20191118205816.5wne6hxedrm7zd5p@development>
References: <16125-a78dafa032d53f7e@postgresql.org>
<20191118205816.5wne6hxedrm7zd5p@development>
Hi,
On 2019-11-18 21:58:16 +0100, Tomas Vondra wrote:
> and the ReorderBufferToastReplace does this:
>
> newtup = change->data.tp.newtuple;
>
> heap_deform_tuple(&newtup->tuple, desc, attrs, isnull);
>
> but that fails, because the tuple pointer happens to be 0x8, which is
> clearly bogus. Not sure where that comes from, I don't recognize that as
> a typical patter.
It indicates that change->data.tp.newtuple is NULL,
afaict. newtup->tuple boils down to
((char *) newtup->tuple) + offsetof(ReorderBufferTupleBuf, tuple)
and offsetof(ReorderBufferTupleBuf, tuple) is 0x8.
> Can you create a core dump (see [1]), and print 'change' and 'txn' in
> frame #2? I wonder if some the other fields are bogus too (but it can't
> be entirely true ...), and if the transaction got serialized.
Please print change and *change, both, please.
I suspect what's happening is that somehow a change that shouldn't have
toast changes - e.g. a DELETE - somehow has toast changes. Which then
triggers a failure in ReorderBufferToastReplace(), which expects
newtuple to be valid.
It's probably worthwhile to add an elog(ERROR) check for this, even if
this does not turn out to be the case.
> > This behaviour does not depends on defined data in tables, because we see it
> > in different database with different sets of tables in publications.
>
> I'm not sure I really believe that. Surely there has to be something
> special about your schema, or possibly access patter that triggers this
> bug in your environment and not elsewhere.
Yea. Are there any C triggers present? Any unusual extensions? Users of
the transaction hook, for example?
> > Looks like a real issue in logical replication.
> > I will happy to provide an additional information about that issue, but i
> > should know what else to need to collect for helping to solve this
> > problem.
> >
>
> Well, if you can create a reproducer, that'd be the best option, because
> then we can investigate locally instead of the ping-ping here.
>
> But if that's not possible, let's start with the schema and the
> additional information from the core file.
>
> I'd also like to see the contents of the WAL, particularly for the XID
> triggering this issue. Please run pg_waldump and see how much data is
> there for XID 1667601527. It does commit at 25EE/D6DE6EB8, not sure
> where it starts. It may have subtransactions, so don't do just grep.
Yea, that'd be helpful.
Greetings,
Andres Freund
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-bugs@postgresql.org
Cc: andres@anarazel.de, tomas.vondra@2ndquadrant.com, andrey.salnikov@dataegret.com
Subject: Re: BUG #16125: Crash of PostgreSQL's wal sender during logical replication
In-Reply-To: <20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de>
References: <16125-a78dafa032d53f7e@postgresql.org> <20191118205816.5wne6hxedrm7zd5p@development> <20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox