From: Drouvot, Bertrand <bdrouvot@amazon.com>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: pgsql-bugs@postgresql.org <pgsql-bugs@postgresql.org>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Cc: Schneider (AWS), Jeremy <schnjere@amazon.com>
Subject: Re: logical decoding bug: segfault in ReorderBufferToastReplace()
Date: Wed, 11 Dec 2019 08:17:01 +0000
Message-ID: <EEB686D3-F8A7-4371-9A96-5DF3B72A7734@amazon.com> (raw)
In-Reply-To: <20191205133836.lulisfo2fphdrjp3@development>
References: <E1i74JE-00024V-9J@gemulon.postgresql.org>
<b1da9abc-bad7-efd6-2f1c-503d508c3877@amazon.com>
<20191205133836.lulisfo2fphdrjp3@development>
On 12/9/19, 10:10 AM, "Tomas Vondra" <tomas.vondra@2ndquadrant.com> wrote:
>On Wed, Dec 04, 2019 at 05:36:16PM -0800, Jeremy Schneider wrote:
>>On 9/8/19 14:01, Tom Lane wrote:
>>> Fix RelationIdGetRelation calls that weren't bothering with error checks.
>>>
>>> ...
>>>
>>> Details
>>> -------
>>> https://git.postgresql.org/pg/commitdiff/69f883fef14a3fc5849126799278abcc43f40f56
>>
>>We had two different databases this week (with the same schema) both
>>independently hit the condition of this recent commit from Tom. It's on
>>11.5 so we're actually segfaulting and restarting rather than just
>>causing the walsender process to ERROR, but regardless there's still
>>some underlying bug here.
>>
>>We have core files and we're still working to see if we can figure out
>>what's going on, but I thought I'd report now in case anyone has extra
>>ideas or suggestions. The segfault is on line 3034 of reorderbuffer.c.
>>
>>https://github.com/postgres/postgres/blob/REL_11_5/src/backend/replication/logical/reorderbuffer.c#L...
>>
>>3033 toast_rel = RelationIdGetRelation(relation->rd_rel->reltoastrelid);
>>3034 toast_desc = RelationGetDescr(toast_rel);
>>
>>We'll keep looking; let me know any feedback! Would love to track down
>>whatever bug is in the logical decoding code, if that's what it is.
>>
>>==========
>>
>>backtrace showing the call stack...
>>
>>Core was generated by `postgres: walsender <NAME-REDACTED>
>><DNS-REDACTED>(31712)'.
>>Program terminated with signal 11, Segmentation fault.
>>#0 ReorderBufferToastReplace (rb=0x3086af0, txn=0x3094a78,
>>relation=0x2b79177249c8, relation=0x2b79177249c8, change=0x30ac938)
>> at reorderbuffer.c:3034
>>3034 reorderbuffer.c: No such file or directory.
>>...
>>(gdb) #0 ReorderBufferToastReplace (rb=0x3086af0, txn=0x3094a78,
>>relation=0x2b79177249c8, relation=0x2b79177249c8, change=0x30ac938)
>> at reorderbuffer.c:3034
>>#1 ReorderBufferCommit (rb=0x3086af0, xid=xid@entry=1358809,
>>commit_lsn=9430473346032, end_lsn=<optimized out>,
>> commit_time=commit_time@entry=628712466364268,
>>origin_id=origin_id@entry=0, origin_lsn=origin_lsn@entry=0) at
>>reorderbuffer.c:1584
>>#2 0x0000000000716248 in DecodeCommit (xid=1358809,
>>parsed=0x7ffc4ce123f0, buf=0x7ffc4ce125b0, ctx=0x3068f70) at decode.c:637
>>#3 DecodeXactOp (ctx=0x3068f70, buf=buf@entry=0x7ffc4ce125b0) at
>>decode.c:245
>>#4 0x000000000071655a in LogicalDecodingProcessRecord (ctx=0x3068f70,
>>record=0x3069208) at decode.c:117
>>#5 0x0000000000727150 in XLogSendLogical () at walsender.c:2886
>>#6 0x0000000000729192 in WalSndLoop (send_data=send_data@entry=0x7270f0
>><XLogSendLogical>) at walsender.c:2249
>>#7 0x0000000000729f91 in StartLogicalReplication (cmd=0x30485a0) at
>>walsender.c:1111
>>#8 exec_replication_command (
>> cmd_string=cmd_string@entry=0x2f968b0 "START_REPLICATION SLOT
>>\"<NAME-REDACTED>\" LOGICAL 893/38002B98 (proto_version '1',
>>publication_names '\"<NAME-REDACTED>\"')") at walsender.c:1628
>>#9 0x000000000076e939 in PostgresMain (argc=<optimized out>,
>>argv=argv@entry=0x2fea168, dbname=0x2fea020 "<NAME-REDACTED>",
>> username=<optimized out>) at postgres.c:4182
>>#10 0x00000000004bdcb5 in BackendRun (port=0x2fdec50) at postmaster.c:4410
>>#11 BackendStartup (port=0x2fdec50) at postmaster.c:4082
>>#12 ServerLoop () at postmaster.c:1759
>>#13 0x00000000007062f9 in PostmasterMain (argc=argc@entry=7,
>>argv=argv@entry=0x2f92540) at postmaster.c:1432
>>#14 0x00000000004be73b in main (argc=7, argv=0x2f92540) at main.c:228
>>
>>==========
>>
>>Some additional context...
>>
>># select * from pg_publication_rel;
>> prpubid | prrelid
>>---------+---------
>> 71417 | 16453
>> 71417 | 54949
>>(2 rows)
>>
>>(gdb) print toast_rel
>>$4 = (struct RelationData *) 0x0
>>
>>(gdb) print *relation->rd_rel
>>$11 = {relname = {data = "<NAME-REDACTED>", '\000' <repeats 44 times>},
>>relnamespace = 16402, reltype = 16430, reloftype = 0,
>>relowner = 16393, relam = 0, relfilenode = 16428, reltablespace = 0,
>>relpages = 0, reltuples = 0, relallvisible = 0, reltoastrelid = 0,
>Hmmm, so reltoastrelid = 0, i.e. the relation does not have a TOAST
>relation. Yet we're calling ReorderBufferToastReplace on the decoded
>record ... interesting.
>
>Can you share structure of the relation causing the issue?
Here it is:
\d+ rel_having_issue
Table "public.rel_having_issue"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
----------------+--------------------------+-----------+----------+-------------------------------------------------+----------+--------------+-------------
id | integer | | not null | nextval('rel_having_issue_id_seq'::regclass) | plain | |
field1 | character varying(255) | | | | extended | |
field2 | integer | | | | plain | |
field3 | timestamp with time zone | | | | plain | |
Indexes:
"rel_having_issue_pkey" PRIMARY KEY, btree (id)
select relname,relfilenode,reltoastrelid from pg_class where relname='rel_having_issue';
relname | relfilenode | reltoastrelid
---------------------+-------------+---------------
rel_having_issue | 16428 | 0
Bertrand
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: bdrouvot@amazon.com, tomas.vondra@2ndquadrant.com, pgsql-hackers@postgresql.org, schnjere@amazon.com
Subject: Re: logical decoding bug: segfault in ReorderBufferToastReplace()
In-Reply-To: <EEB686D3-F8A7-4371-9A96-5DF3B72A7734@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox