pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Random crashes - segmentation fault
6+ messages / 4 participants
[nested] [flat]

* Random crashes - segmentation fault
@ 2019-11-13 08:12  kjonca@poczta.onet.pl
  0 siblings, 1 reply; 6+ messages in thread

From: kjonca@poczta.onet.pl @ 2019-11-13 08:12 UTC (permalink / raw)
  To: pgsql-bugs@lists.postgresql.org

This is a copy of message sent to debian-user mailing list:


select version();
                                                     version                                                       
------------------------------------------------------------------------------------------------------------------
PostgreSQL 12.0 (Debian 12.0-1+b1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 9.2.1-8) 9.2.1 20190909, 64-bit


kjonca@poczta.onet.pl (Kamil Jońca) writes:

> It is home PC box with debian sid.
> Recently my postgres was upgraded from version 11 to 12.
> I migrate databases, and during last few days I have had 2 server
> crashes.
> Crashes were during different statements. And after crash these
> statements executed successfully.
> In log I have:
> ===============================================================
> 2019-11-04 00:07:38 CET LOG:  server process (PID 19244) was terminated by signal 11: Segmentation fault
> 2019-11-04 00:07:38 CET DETAIL:  Failed process was running: update queue set priority = -3 ;
> 2019-11-04 00:07:38 CET LOG:  terminating any other active server processes
> [...]
> 2019-11-04 00:07:39 CET LOG:  all server processes terminated; reinitializing
> 2019-11-04 00:07:39 CET DEBUG:  mmap(150994944) with MAP_HUGETLB failed, huge pages disabled: Cannot allocate memory
> 2019-11-04 00:07:39 CET LOG:  database system was interrupted; last known up at 2019-11-04 00:02:24 CET
> ===============================================================
> 2019-11-05 21:43:56 CET LOG:  server process (PID 23233) was terminated by signal 11: Segmentation fault
> 2019-11-05 21:43:56 CET DETAIL:  Failed process was running: SELECT po_nr FROM get_free_numbers(999);
> 2019-11-05 21:43:56 CET LOG:  terminating any other active server processes
> [...]
> 2019-11-05 21:43:57 CET LOG:  all server processes terminated; reinitializing
> 2019-11-05 21:43:57 CET DEBUG:  mmap(150994944) with MAP_HUGETLB failed, huge pages disabled: Cannot allocate memory
> 2019-11-05 21:43:58 CET LOG:  database system was interrupted; last known up at 2019-11-05 21:43:49 CET
> ===============================================================
>
> any hints?
>
> KJ


Today (13.XI.2019 - KJ)  was another crash.
Another piece of a puzzle: There is (unlogged) table with 70M+
rows. After crash this table is empty (but table itself exists.)

KJ
-- 
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
1.79 x 10^12 furlongs per fortnight -- it's not just a good idea, it's
the law!





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Random crashes - segmentation fault
@ 2019-11-13 12:58  Michael Paquier <michael@paquier.xyz>
  parent: kjonca@poczta.onet.pl
  0 siblings, 1 reply; 6+ messages in thread

From: Michael Paquier @ 2019-11-13 12:58 UTC (permalink / raw)
  To: Kamil Jońca <kjonca@poczta.onet.pl>; +Cc: pgsql-bugs@lists.postgresql.org

On Wed, Nov 13, 2019 at 09:12:02AM +0100, Kamil Jońca wrote:
> Today (13.XI.2019 - KJ)  was another crash.
> Another piece of a puzzle: There is (unlogged) table with 70M+
> rows. After crash this table is empty (but table itself exists.)

Could it be possible to see a backtrace of the crash?  There is
nothing we can really do without any hint.  If you can send a
self-contained test case which allows to reproduce the problem, that's
even better.  Please note that unlogged tables are reinitialized after
a crash at the beginning of recovery.  That's their design.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Random crashes - segmentation fault
@ 2019-11-13 13:16  kjonca@fastmail.com
  parent: Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 6+ messages in thread

From: kjonca@fastmail.com @ 2019-11-13 13:16 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-bugs@lists.postgresql.org

Michael Paquier <michael@paquier.xyz> writes:

> On Wed, Nov 13, 2019 at 09:12:02AM +0100, Kamil Jońca wrote:
>> Today (13.XI.2019 - KJ)  was another crash.
>> Another piece of a puzzle: There is (unlogged) table with 70M+
>> rows. After crash this table is empty (but table itself exists.)
>
> Could it be possible to see a backtrace of the crash?  There is

Erm. Only thing I can do is to configure for debug future crashes. Could you
tell me what and how to configure to get backtrace?
This is debian sid machine with postgres packaged by debian folks.
I guess I should enable core dumps. Anything else?

> nothing we can really do without any hint.  If you can send a
> self-contained test case which allows to reproduce the problem, that's
> even better.
I am afraid I cannot. These crasheas are rather rare and unpredictable.

> Please note that unlogged tables are reinitialized after
> a crash at the beginning of recovery.  That's their design.
I see. Thanks for explanation.

KJ


-- 
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
If I have trouble installing Linux, something is wrong. Very wrong.
		-- Linus Torvalds





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Random crashes - segmentation fault
@ 2019-11-13 15:07  Tom Lane <tgl@sss.pgh.pa.us>
  parent: kjonca@fastmail.com
  0 siblings, 1 reply; 6+ messages in thread

From: Tom Lane @ 2019-11-13 15:07 UTC (permalink / raw)
  To: kjonca@fastmail.com; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org

kjonca@fastmail.com (Kamil =?iso-8859-2?Q?Jo=F1ca?=) writes:
> Michael Paquier <michael@paquier.xyz> writes:
>> Could it be possible to see a backtrace of the crash?  There is

> Erm. Only thing I can do is to configure for debug future crashes. Could you
> tell me what and how to configure to get backtrace?

There's some accumulated wisdom at

https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend

I'd suggest that before worrying about that, you should install 12.1
(official release is tomorrow) and see if the crashes go away.  There's
not much info in this report, but it could match any of several
already-fixed bugs.

			regards, tom lane





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Random crashes - segmentation fault
@ 2019-12-13 15:05  kjonca@fastmail.com
  parent: Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 6+ messages in thread

From: kjonca@fastmail.com @ 2019-12-13 15:05 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org

Tom Lane <tgl@sss.pgh.pa.us> writes:

> kjonca@fastmail.com (Kamil =?iso-8859-2?Q?Jo=F1ca?=) writes:
>> Michael Paquier <michael@paquier.xyz> writes:
>>> Could it be possible to see a backtrace of the crash?  There is
>
>> Erm. Only thing I can do is to configure for debug future crashes. Could you
>> tell me what and how to configure to get backtrace?
>
> There's some accumulated wisdom at
>
> https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend
>
> I'd suggest that before worrying about that, you should install 12.1
> (official release is tomorrow) and see if the crashes go away.  There's
> not much info in this report, but it could match any of several
> already-fixed bugs.
>


Today server crashed again.
--8<---------------cut here---------------start------------->8---
2019-12-13 15:41:38 CET DEBUG:  starting background worker process "parallel worker for PID 364516"
TRAP: FailedAssertion("!(result->tdrefcount == -1)", File: "/build/postgresql-12-123lrX/postgresql-12-12.1/build/../src/backend/utils/cache/typcache.c", Line: 2621)
--8<---------------cut here---------------end--------------->8---

I have core dump.

KJ

> 			regards, tom lane
>

-- 
http://wolnelektury.pl/wesprzyj/teraz/





^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* Re: Random crashes - segmentation fault
@ 2019-12-17 04:59  Michael Paquier <michael@paquier.xyz>
  parent: kjonca@fastmail.com
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2019-12-17 04:59 UTC (permalink / raw)
  To: Kamil Jońca <kjonca@fastmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-bugs@lists.postgresql.org

On Fri, Dec 13, 2019 at 04:05:34PM +0100, Kamil Jońca wrote:
> Today server crashed again.
> --8<---------------cut here---------------start------------->8---
> 2019-12-13 15:41:38 CET DEBUG:  starting background worker process
> "parallel worker for PID 364516" 
> TRAP: FailedAssertion("!(result->tdrefcount == -1)", File:
> /build/postgresql-12-123lrX/postgresql-12-12.1/build/../src/backend/utils/cache/typcache.c",
> Line: 2621) 
> --8<---------------cut here---------------end--------------->8---
> 
> I have core dump.

Could it be possible to see a full backtrace then?  Using gdb that
would mean to use "bt" to print the full stack.  What kind of function
is get_free_numbers() actually and what does it do?
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 6+ messages in thread


end of thread, other threads:[~2019-12-17 04:59 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-11-13 08:12 Random crashes - segmentation fault kjonca@poczta.onet.pl
2019-11-13 12:58 ` Michael Paquier <michael@paquier.xyz>
2019-11-13 13:16   ` kjonca@fastmail.com
2019-11-13 15:07     ` Tom Lane <tgl@sss.pgh.pa.us>
2019-12-13 15:05       ` kjonca@fastmail.com
2019-12-17 04:59         ` Michael Paquier <michael@paquier.xyz>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox