Date: Tue, 16 Jul 2019 10:22:04 +0200
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Jerry Sievers <gsievers19@comcast.net>
Cc: pgsql-hackers@postgresql.org
Subject: Re: SegFault on 9.6.14
Message-ID: <20190716082204.iecyfh6mhu6khdhz@development>
References: <87ims2amh6.fsf@jsievers.enova.com>
 <20190716001529.uvpfngvxtlu5h2fr@development>
 <87blxuakv4.fsf@jsievers.enova.com>
 <20190716003449.fjegtxinhrqubysu@development>
 <87ftn6ojwf.fsf@jsievers.enova.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <87ftn6ojwf.fsf@jsievers.enova.com>
User-Agent: NeoMutt/20180716-1444-295967
Precedence: bulk

On Mon, Jul 15, 2019 at 08:20:00PM -0500, Jerry Sievers wrote:
>Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
>
>> On Mon, Jul 15, 2019 at 07:22:55PM -0500, Jerry Sievers wrote:
>>
>>>Tomas Vondra <tomas.vondra@2ndquadrant.com> writes:
>>>
>>>> On Mon, Jul 15, 2019 at 06:48:05PM -0500, Jerry Sievers wrote:
>>>>
>>>>>Greetings Hackers.
>>>>>
>>>>>We have a reproduceable case of $subject that issues a backtrace such as
>>>>>seen below.
>>>>>
>>>>>The query that I'd prefer to sanitize before sending is <30 lines of at
>>>>>a glance, not terribly complex logic.
>>>>>
>>>>>It nonetheless dies hard after a few seconds of running and as expected,
>>>>>results in an automatic all-backend restart.
>>>>>
>>>>>Please advise on how to proceed.  Thanks!
>>>>>
>>>>>bt
>>>>>#0  initscan (scan=scan@entry=0x55d7a7daa0b0, key=0x0, keep_startblock=keep_startblock@entry=1 '\001')
>>>>>    at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:233
>>>>>#1  0x000055d7a72fa8d0 in heap_rescan (scan=0x55d7a7daa0b0, key=key@entry=0x0) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/access/heap/heapam.c:1529
>>>>>#2  0x000055d7a7451fef in ExecReScanSeqScan (node=node@entry=0x55d7a7d85100) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeSeqscan.c:280
>>>>>#3  0x000055d7a742d36e in ExecReScan (node=0x55d7a7d85100) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:158
>>>>>#4  0x000055d7a7445d38 in ExecReScanGather (node=node@entry=0x55d7a7d84d30) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeGather.c:475
>>>>>#5  0x000055d7a742d255 in ExecReScan (node=0x55d7a7d84d30) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:166
>>>>>#6  0x000055d7a7448673 in ExecReScanHashJoin (node=node@entry=0x55d7a7d84110) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/nodeHashjoin.c:1019
>>>>>#7  0x000055d7a742d29e in ExecReScan (node=node@entry=0x55d7a7d84110) at /build/postgresql-9.6-5O8OLM/postgresql-9.6-9.6.14/build/../src/backend/executor/execAmi.c:226
>>>>><about 30 lines omitted>
>>>>>
>>>>
>>>> Hmmm, that means it's crashing here:
>>>>
>>>>    if (scan->rs_parallel != NULL)
>>>>        scan->rs_nblocks = scan->rs_parallel->phs_nblocks;     <--- here
>>>>    else
>>>>        scan->rs_nblocks = RelationGetNumberOfBlocks(scan->rs_rd);
>>>>
>>>> But clearly, scan is valid (otherwise it'd crash on the if condition),
>>>> and scan->rs_parallel must me non-NULL. Which probably means the pointer
>>>> is (no longer) valid.
>>>>
>>>> Could it be that the rs_parallel DSM disappears on rescan, or something
>>>> like that?
>>>
>>>No clue but something I just tried was to disable parallelism by setting
>>>max_parallel_workers_per_gather to 0 and however the query has not
>>>finished after a few minutes, there is no crash.
>>>
>>
>> That might be a hint my rough analysis was somewhat correct. The
>> question is whether the non-parallel plan does the same thing. Maybe it
>> picks a plan that does not require rescans, or something like that.
>>
>>>Please advise.
>>>
>>
>> It would be useful to see (a) exacution plan of the query, (b) full
>> backtrace and (c) a bit of context for the place where it crashed.
>>
>> Something like (in gdb):
>>
>>    bt full
>>    list
>>    p *scan
>
>The p *scan did nothing unless I ran it first however my gdb $foo isn't
>strong presently.

Hmm, the rs_parallel pointer looks sane (it's not obvious garbage). Can
you try this?

   p *scan->rs_parallel

Another question - are you sure this is not an OOM issue? That might
sometimes look like SIGSEGV due to overcommit. What's the memory
consumption / is there anything in dmesg?

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services