pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
12+ messages / 5 participants
[nested] [flat]

* BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
@ 2020-01-15 02:00 PG Bug reporting form <noreply@postgresql.org>
  2020-01-15 02:58 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Michael Paquier <michael@paquier.xyz>
  0 siblings, 1 reply; 12+ messages in thread

From: PG Bug reporting form @ 2020-01-15 02:00 UTC (permalink / raw)
  To: pgsql-bugs@lists.postgresql.org; +Cc: pendekar.senja@outlook.com

The following bug has been logged on the website:

Bug reference:      16205
Logged by:          Firdaus Firdaus
Email address:      pendekar.senja@outlook.com
PostgreSQL version: 12.1
Operating system:   Ubuntu 16.04.6 LTS
Description:        

Hi there,
I got error when replicate with native logical method from PostgreSQL 11.6
to PostgreSQL 12.1, I will connecting table partition to table partition
from Version 11.6 to Version 12.1, but after all tables already replicated
around more than 2 hours, server can't access and recovery mode. please help
to advice, below log.
2020-01-15 08:31:40 WIB [24943] LOG:  background worker "logical replication
worker" (PID 25218) was terminated by signal 11: Segmentation fault
2020-01-15 08:31:40 WIB [24943] LOG:  terminating any other active server
processes
2020-01-15 08:31:40 WIB [25214] WARNING:  terminating connection because of
crash of another server process
2020-01-15 08:31:40 WIB [25214] DETAIL:  The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared
memory.
2020-01-15 08:31:40 WIB [25214] HINT:  In a moment you should be able to
reconnect to the database and repeat your command.
2020-01-15 08:31:40 WIB [24943] LOG:  all server processes terminated;
reinitializing
2020-01-15 08:31:41 WIB [25222] LOG:  database system was interrupted; last
known up at 2020-01-15 08:31:40 WIB
2020-01-15 08:31:41 WIB [24943] LOG:  received fast shutdown request
2020-01-15 08:31:41 WIB [25222] LOG:  recovered replication state of node 1
to 2EA/B70B8DC8
2020-01-15 08:31:41 WIB [25222] LOG:  database system was not properly shut
down; automatic recovery in progress
2020-01-15 08:31:41 WIB [25222] LOG:  invalid record length at 132/DF032918:
wanted 24, got 0
2020-01-15 08:31:41 WIB [25222] LOG:  redo is not required
2020-01-15 08:31:41 WIB [25222] LOG:  checkpoint starting: end-of-recovery
immediate
2020-01-15 08:31:41 WIB [25222] LOG:  checkpoint complete: wrote 0 buffers
(0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.051 s,
sync=0.000 s, total=0.062 s; sync files=0, longest=0.000 s, average=0.000 s;
distance=0 kB, estimate=0 kB
2020-01-15 08:31:41 WIB [24943] LOG:  abnormal database system shutdown
2020-01-15 08:31:41 WIB [24943] LOG:  database system is shut down



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 02:00 BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation PG Bug reporting form <noreply@postgresql.org>
@ 2020-01-15 02:58 ` Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2020-01-15 02:58 UTC (permalink / raw)
  To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> I got error when replicate with native logical method from PostgreSQL 11.6
> to PostgreSQL 12.1, I will connecting table partition to table partition
> from Version 11.6 to Version 12.1, but after all tables already replicated
> around more than 2 hours, server can't access and recovery mode. please help
> to advice, below log.

Could you provide a backtrace?  Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:
https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
@ 2020-01-15 04:02 Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-15 04:42 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Michael Paquier <michael@paquier.xyz>
  2020-01-15 10:02 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
  0 siblings, 2 replies; 12+ messages in thread

From: Pendekar Dikala Senja @ 2020-01-15 04:02 UTC (permalink / raw)
  To: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>


Here are the logs and our configuration.
Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu

LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

Can you help us to find out the solution and configuration?
Thanks for your help

________________________________
From: Michael Paquier
Sent: Tuesday, January 14, 2020 6:58 PM
To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> I got error when replicate with native logical method from PostgreSQL 11.6
> to PostgreSQL 12.1, I will connecting table partition to table partition
> from Version 11.6 to Version 12.1, but after all tables already replicated
> around more than 2 hours, server can't access and recovery mode. please help
> to advice, below log.

Could you provide a backtrace?  Looking at the recent commit history,
we had one fix which could be relevant to you after 12.1 was released:
https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org

And here is the commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e

The problem may be different though, so a backtrace would be good to
check anyway.
--
Michael

Attachments:

postgres-config.txt (text/plain, 1.0K) download data_directory = '/data/postgres/data/' listen_addresses = '*' port = 5432 max_connections = 1024 ssl = off shared_buffers = 16GB work_mem = 8MB maintenance_work_mem = 1GB shared_preload_libraries = 'pg_stat_statements,pg_partman_bgw' pg_stat_statements.track = all wal_level = logical checkpoint_timeout = 15min max_wal_size = 1GB min_wal_size = 80MB checkpoint_completion_target = 0.7 archive_mode = on archive_command = '/bin/true' max_wal_senders = 64 wal_keep_segments = 100 max_replication_slots = 20 hot_standby = on hot_standby_feedback = on log_destination = 'stderr' logging_collector = on log_directory = 'pg_log' log_file_mode = 0600 log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' log_truncate_on_rotation = on log_rotation_age = 1d log_rotation_size = 50MB log_min_duration_statement = 100 log_checkpoints = on log_line_prefix = '%t [%p] %q%u@%d %h ' log_lock_waits = on log_statement = 'none' log_timezone = 'Asia/Jakarta' timezone = 'Asia/Jakarta' pg_partman_bgw.interval = 3600 pg_partman_bgw.dbname = 'dborder'
postgresql-log.txt (text/plain, 2.3K) download 2020-01-15 09:34:31 WIB [14515] LOG: database system is ready to accept connections 2020-01-15 09:34:31 WIB [14929] LOG: pg_partman master background worker master process initialized with role postgres 2020-01-15 09:34:31 WIB [14931] LOG: logical replication apply worker for subscription "tkpordersubs" has started 2020-01-15 09:34:31 WIB [14933] LOG: logical replication table synchronization worker for subscription "tkpordersubs", table "ws_order_p2017_12" has started 2020-01-15 09:34:31 WIB [14934] LOG: logical replication table synchronization worker for subscription "tkpordersubs", table "ws_order_p2017_11" has started 2020-01-15 09:34:31 WIB [14515] LOG: background worker "logical replication worker" (PID 14931) was terminated by signal 11: Segmentation fault 2020-01-15 09:34:31 WIB [14515] LOG: terminating any other active server processes 2020-01-15 09:34:31 WIB [14927] WARNING: terminating connection because of crash of another server process 2020-01-15 09:34:31 WIB [14927] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2020-01-15 09:34:31 WIB [14927] HINT: In a moment you should be able to reconnect to the database and repeat your command. 2020-01-15 09:34:31 WIB [14515] LOG: all server processes terminated; reinitializing 2020-01-15 09:34:31 WIB [14515] LOG: received fast shutdown request 2020-01-15 09:34:31 WIB [14940] LOG: database system was interrupted; last known up at 2020-01-15 09:34:31 WIB 2020-01-15 09:34:31 WIB [14941] replication_pg@dborder 172.31.000.000 FATAL: the database system is shutting down 2020-01-15 09:34:31 WIB [14942] replication_pg@dborder 172.31.000.000 FATAL: the database system is shutting down 2020-01-15 09:34:31 WIB [14943] replication_pg@dborder 172.31.000.000 FATAL: the database system is shutting down 2020-01-15 09:34:32 WIB [14940] LOG: recovered replication state of node 1 to 2EA/B70B8DC8 2020-01-15 09:34:32 WIB [14940] LOG: database system was not properly shut down; automatic recovery in progress 2020-01-15 09:34:32 WIB [14940] LOG: redo starts at 132/E02E97E8 2020-01-15 09:34:32 WIB [14515] LOG: abnormal database system shutdown 2020-01-15 09:34:32 WIB [14515] LOG: database system is shut down
syslog.txt (text/plain, 879B) download Jan 15 10:34:15 localhost kernel: [ 1846.610895] postgres[14526]: segfault at 56085f796128 ip 00007f2860e34168 sp 00007ffe59e14ca8 error 4 in libc-2.23.so[7f2860ce6000+1c0000] Jan 15 10:34:15 localhost kernel: [ 1847.171272] postgres[14538]: segfault at 56085f796c48 ip 00007f2860e34168 sp 00007ffe59e14ca8 error 4 in libc-2.23.so[7f2860ce6000+1c0000] Jan 15 10:34:16 localhost kernel: [ 1847.725079] postgres[14554]: segfault at 56085f796c48 ip 00007f2860e34168 sp 00007ffe59e14ca8 error 4 in libc-2.23.so[7f2860ce6000+1c0000] Jan 15 10:34:16 localhost kernel: [ 1848.281706] postgres[14569]: segfault at 56085f79cdc8 ip 00007f2860e34168 sp 00007ffe59e14ca8 error 4 in libc-2.23.so[7f2860ce6000+1c0000] Jan 15 10:34:17 localhost kernel: [ 1848.889465] postgres[14586]: segfault at 56085f79cdc8 ip 00007f2860e34168 sp 00007ffe59e14ca8 error 4 in libc-2.23.so[7f2860ce6000+1c0000]
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
@ 2020-01-15 04:42 ` Michael Paquier <michael@paquier.xyz>
  1 sibling, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2020-01-15 04:42 UTC (permalink / raw)
  To: Pendekar Dikala Senja <pendekar.senja@outlook.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>

On Wed, Jan 15, 2020 at 04:02:38AM +0000, Pendekar Dikala Senja wrote:
> Can you help us to find out the solution and configuration?

A more detailed backtrace would be nice.  Here are some guidelines:
https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
@ 2020-01-15 10:02 ` Mahendra Singh Thalor <mahi6run@gmail.com>
  2020-01-15 10:08   ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-16 18:46   ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Christophe Pettus <xof@thebuild.com>
  1 sibling, 2 replies; 12+ messages in thread

From: Mahendra Singh Thalor @ 2020-01-15 10:02 UTC (permalink / raw)
  To: Pendekar Dikala Senja <pendekar.senja@outlook.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <
pendekar.senja@outlook.com> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6
to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version:
core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
> Subject: Re: BUG #16205: background worker "logical replication worker"
(PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL
11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already
replicated
> > around more than 2 hours, server can't access and recovery mode. please
help
> > to advice, below log.
>
> Could you provide a backtrace?  Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
>
https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

*How to get stack trace:*
After crash, you can see core.PID file in your data directory. Attach core
file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

-- 
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-15 10:02 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
@ 2020-01-15 10:08   ` Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-16 03:20     ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  1 sibling, 1 reply; 12+ messages in thread

From: Pendekar Dikala Senja @ 2020-01-15 10:08 UTC (permalink / raw)
  To: Mahendra Singh Thalor <mahi6run@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>

Hi,
right now we're still trying to setup again, and will provide log later.


Kind Regards

your good friend
________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>
> Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL 11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already replicated
> > around more than 2 hours, server can't access and recovery mode. please help
> > to advice, below log.
>
> Could you provide a backtrace?  Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-15 10:02 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
  2020-01-15 10:08   ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
@ 2020-01-16 03:20     ` Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-16 03:30       ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Pendekar Dikala Senja @ 2020-01-16 03:20 UTC (permalink / raw)
  To: Mahendra Singh Thalor <mahi6run@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>

following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.


sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/;.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.
(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)


thanks


________________________________
From: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.


Kind Regards

your good friend
________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>
> Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL 11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already replicated
> > around more than 2 hours, server can't access and recovery mode. please help
> > to advice, below log.
>
> Could you provide a backtrace?  Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--
Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com

Attachments:

log.txt (text/plain, 5.9K) download 2020-01-16 10:13:00 WIB [15803] LOG: background worker "logical replication worker" (PID 28135) was terminated by signal 11: Segmentation fault 2020-01-16 10:13:00 WIB [15803] LOG: terminating any other active server processes 2020-01-16 10:13:00 WIB [28131] WARNING: terminating connection because of crash of another server process 2020-01-16 10:13:00 WIB [28131] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2020-01-16 10:13:00 WIB [28131] HINT: In a moment you should be able to reconnect to the database and repeat your command. 2020-01-16 10:13:00 WIB [15803] LOG: all server processes terminated; reinitializing 2020-01-16 10:13:01 WIB [28137] LOG: database system was interrupted; last known up at 2020-01-16 10:13:00 WIB 2020-01-16 10:13:01 WIB [28138] replication_pg@dborder 172.21.000.000 FATAL: the database system is in recovery mode 2020-01-16 10:13:01 WIB [28140] replication_pg@dborder 172.21.000.000 FATAL: the database system is in recovery mode 2020-01-16 10:13:01 WIB [28139] replication_pg@dborder 172.21.000.000 FATAL: the database system is in recovery mode 2020-01-16 10:13:01 WIB [28141] gundala@[unknown] 172.21.000.000 FATAL: the database system is in recovery mode 2020-01-16 10:13:01 WIB [28142] gundala@[unknown] 172.21.000.000 FATAL: the database system is in recovery mode 2020-01-16 10:13:01 WIB [28143] gundala@[unknown] 172.21.000.000 FATAL: the database system is in recovery mode 2020-01-16 10:13:01 WIB [28137] LOG: recovered replication state of node 1 to 2F2/26BE7810 2020-01-16 10:13:01 WIB [28137] LOG: database system was not properly shut down; automatic recovery in progress 2020-01-16 10:13:01 WIB [28137] LOG: invalid record length at 13F/D1B03F78: wanted 24, got 0 2020-01-16 10:13:01 WIB [28137] LOG: redo is not required 2020-01-16 10:13:01 WIB [28137] LOG: checkpoint starting: end-of-recovery immediate 2020-01-16 10:13:01 WIB [28137] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.058 s, sync=0.000 s, total=0.070 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=0 kB 2020-01-16 10:13:01 WIB [15803] LOG: database system is ready to accept connections 2020-01-16 10:13:01 WIB [28149] LOG: pg_partman master background worker master process initialized with role postgres 2020-01-16 10:13:01 WIB [28151] LOG: logical replication apply worker for subscription "ordersubs" has started 2020-01-16 10:13:01 WIB [15803] LOG: background worker "logical replication worker" (PID 28151) was terminated by signal 11: Segmentation fault 2020-01-16 10:13:01 WIB [15803] LOG: terminating any other active server processes 2020-01-16 10:13:01 WIB [28147] WARNING: terminating connection because of crash of another server process 2020-01-16 10:13:01 WIB [28147] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2020-01-16 10:13:01 WIB [28147] HINT: In a moment you should be able to reconnect to the database and repeat your command. 2020-01-16 10:13:01 WIB [15803] LOG: all server processes terminated; reinitializing 2020-01-16 10:13:01 WIB [28156] LOG: database system was interrupted; last known up at 2020-01-16 10:13:01 WIB 2020-01-16 10:13:01 WIB [28156] LOG: recovered replication state of node 1 to 2F2/26BE7810 2020-01-16 10:13:01 WIB [28156] LOG: database system was not properly shut down; automatic recovery in progress 2020-01-16 10:13:01 WIB [28156] LOG: invalid record length at 13F/D1B03FF0: wanted 24, got 0 2020-01-16 10:13:01 WIB [28156] LOG: redo is not required 2020-01-16 10:13:01 WIB [28156] LOG: checkpoint starting: end-of-recovery immediate 2020-01-16 10:13:01 WIB [28156] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.058 s, sync=0.000 s, total=0.069 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=0 kB 2020-01-16 10:13:01 WIB [15803] LOG: database system is ready to accept connections 2020-01-16 10:13:01 WIB [28162] LOG: pg_partman master background worker master process initialized with role postgres 2020-01-16 10:13:01 WIB [28164] LOG: logical replication apply worker for subscription "ordersubs" has started 2020-01-16 10:13:01 WIB [15803] LOG: background worker "logical replication worker" (PID 28164) was terminated by signal 11: Segmentation fault 2020-01-16 10:13:01 WIB [15803] LOG: terminating any other active server processes 2020-01-16 10:13:01 WIB [28160] WARNING: terminating connection because of crash of another server process 2020-01-16 10:13:01 WIB [28160] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2020-01-16 10:13:01 WIB [28160] HINT: In a moment you should be able to reconnect to the database and repeat your command. 2020-01-16 10:13:01 WIB [15803] LOG: all server processes terminated; reinitializing 2020-01-16 10:13:02 WIB [28166] LOG: database system was interrupted; last known up at 2020-01-16 10:13:01 WIB 2020-01-16 10:13:02 WIB [28166] LOG: recovered replication state of node 1 to 2F2/26BE7810 2020-01-16 10:13:02 WIB [28166] LOG: database system was not properly shut down; automatic recovery in progress 2020-01-16 10:13:02 WIB [28166] LOG: invalid record length at 13F/D1B04080: wanted 24, got 0 2020-01-16 10:13:02 WIB [28166] LOG: redo is not required 2020-01-16 10:13:02 WIB [28166] LOG: checkpoint starting: end-of-recovery immediate 2020-01-16 10:13:02 WIB [28166] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.058 s, sync=0.000 s, total=0.074 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=0 kB
^ permalink raw reply [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-15 10:02 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
  2020-01-15 10:08   ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-16 03:20     ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
@ 2020-01-16 03:30       ` Mahendra Singh Thalor <mahi6run@gmail.com>
  2020-01-16 03:44         ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Mahendra Singh Thalor @ 2020-01-16 03:30 UTC (permalink / raw)
  To: Pendekar Dikala Senja <pendekar.senja@outlook.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <
pendekar.senja@outlook.com> wrote:

> following your intruction. please find the information from my server
> below after running stact trace. I can't capture the PID because PID is
> always changing and not always the same.
>
>
> sudo gdb -p 28164
> GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
> Copyright (C) 2016 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html;
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/;.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/;.
> For help, type "help".
> Type "apropos word" to search for commands related to "word".
> Attaching to process 28164
> ptrace: No such process.
>

Yes, every time pid will change. You have to give '-c" option while stating
the server to get call stack.
You have to give "gdb postgres data/core.pid".
This command should be fired from your bin(binray) folder.
Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com


(gdb) set pagination off
> (gdb) set logging file debuglog.txt
> (gdb) set logging on
> Copying output to debuglog.txt.
> (gdb) cont
> The program is not being run.
> (gdb) detach
> The program is not being run.
> (gdb) b errfinish
> No symbol table is loaded.  Use the "file" command.
> Make breakpoint pending on future shared library load? (y or [n]) n
> (gdb)
>
>
> thanks
>
>
> ------------------------------
> *From:* Pendekar Dikala Senja <pendekar.senja@outlook.com>
> *Sent:* Wednesday, January 15, 2020 2:08 AM
> *To:* Mahendra Singh Thalor <mahi6run@gmail.com>
> *Cc:* Michael Paquier <michael@paquier.xyz>;
> pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
> *Subject:* Re: BUG #16205: background worker "logical replication worker"
> (PID 25218) was terminated by signal 11: Segmentation
>
> Hi,
> right now we're still trying to setup again, and will provide log later.
>
>
> Kind Regards
>
> your good friend
> ------------------------------
> *From:* Mahendra Singh Thalor <mahi6run@gmail.com>
> *Sent:* Wednesday, January 15, 2020 5:02:57 PM
> *To:* Pendekar Dikala Senja <pendekar.senja@outlook.com>
> *Cc:* Michael Paquier <michael@paquier.xyz>;
> pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
> *Subject:* Re: BUG #16205: background worker "logical replication worker"
> (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <
> pendekar.senja@outlook.com> wrote:
> >
> >
> > Here are the logs and our configuration.
> > Actually we are facing problem when replicating partition table from
> 11.6 to 12.1, but if there is no partition table the replication works so
> well.
> > As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
> >
> > LSB Version:
> core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> > Distributor ID: Ubuntu
> > Description: Ubuntu 16.04.6 LTS
> > Release: 16.04
> > Codename: xenial
> >
> > Can you help us to find out the solution and configuration?
> > Thanks for your help
> >
> > ________________________________ From: Michael Paquier
> > Sent: Tuesday, January 14, 2020 6:58 PM
> > To: pendekar.senja@outlook.com; pgsql-bugs@lists.postgresql.org
> > Subject: Re: BUG #16205: background worker "logical replication worker"
> (PID 25218) was terminated by signal 11: Segmentation
> >
> > On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > > I got error when replicate with native logical method from PostgreSQL
> 11.6
> > > to PostgreSQL 12.1, I will connecting table partition to table
> partition
> > > from Version 11.6 to Version 12.1, but after all tables already
> replicated
> > > around more than 2 hours, server can't access and recovery mode.
> please help
> > > to advice, below log.
> >
> > Could you provide a backtrace?  Looking at the recent commit history,
> > we had one fix which could be relevant to you after 12.1 was released:
> >
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
> >
> > And here is the commit:
> > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
> >
> > The problem may be different though, so a backtrace would be good to
> > check anyway.
> > --
> > Michael
>
> Hi,
> It looks like that server is crashed due to segmentation fault.
> Can you provide call stack(stack trace) of Pid 25218.
> To get call stack, you should start server with "-c" option.
> Ex: ./pg_ctl -D data -l logfile -c start
>
> *How to get stack trace:*
> After crash, you can see core.PID file in your data directory. Attach core
> file to gdb and fire "bt" command.
> Ex: gdb postgres data/core.25218
> bt
>
> --
>
>

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-15 10:02 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
  2020-01-15 10:08   ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-16 03:20     ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-16 03:30       ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
@ 2020-01-16 03:44         ` Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-16 03:47           ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  0 siblings, 1 reply; 12+ messages in thread

From: Pendekar Dikala Senja @ 2020-01-16 03:44 UTC (permalink / raw)
  To: Mahendra Singh Thalor <mahi6run@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>

I didn't see where the path pg_ctl

$./pg_ctl -D /data/postgres/data -l logfile -c start
-su: ./pg_ctl: No such file or directory
$ pg_ctl -D /data/postgres/data -l logfile -c start
pg_ctl: command not found

on datadir I did not see core.PID, let me know where the dir.
$ ls -lrth
total 144K
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_twophase
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_tblspc
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_snapshots
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_serial
drwx------ 4 postgres postgres 4.0K Jan  2 16:34 pg_multixact
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_dynshmem
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_commit_ts
-rw------- 1 postgres postgres    3 Jan  2 16:34 PG_VERSION
-rw------- 1 postgres postgres   88 Jan  2 16:34 postgresql.auto.conf
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_xact
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_subtrans
drwx------ 6 postgres postgres 4.0K Jan  2 16:44 base
drwx------ 5 postgres postgres 4.0K Jan  3 18:29 pg_replslot
-rw------- 1 postgres postgres  127 Jan 15 12:38 postmaster.opts
drwx------ 2 postgres postgres 4.0K Jan 16 01:00 pg_log
-rw------- 1 postgres postgres   47 Jan 16 01:00 current_logfiles
drwx------ 3 postgres postgres  52K Jan 16 11:00 pg_wal
drwx------ 4 postgres postgres 4.0K Jan 16 11:40 pg_logical
-rw------- 1 postgres postgres   93 Jan 16 11:40 postmaster.pid
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat_tmp
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_notify
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 global
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat
$ cd postmaster.pid
-su: cd: postmaster.pid: Not a directory
$ more postmaster.pid
15803
/data/postgres/data
1579063112
5432
/var/run/postgresql
*
  5432001 169312256
ready

________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 7:30 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:
following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.


sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/;.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.

Yes, every time pid will change. You have to give '-c" option while stating the server to get call stack.
You have to give "gdb postgres data/core.pid".
This command should be fired from your bin(binray) folder.
Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com<http://www.enterprisedb.com/;


(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)


thanks


________________________________
From: Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com<mailto:mahi6run@gmail.com>>
Cc: Michael Paquier <michael@paquier.xyz<mailto:michael@paquier.xyz>>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org> <pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.


Kind Regards

your good friend
________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com<mailto:mahi6run@gmail.com>>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>>
Cc: Michael Paquier <michael@paquier.xyz<mailto:michael@paquier.xyz>>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org> <pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>
> Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL 11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already replicated
> > around more than 2 hours, server can't access and recovery mode. please help
> > to advice, below log.
>
> Could you provide a backtrace?  Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-15 10:02 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
  2020-01-15 10:08   ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-16 03:20     ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-16 03:30       ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
  2020-01-16 03:44         ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
@ 2020-01-16 03:47           ` Pendekar Dikala Senja <pendekar.senja@outlook.com>
  0 siblings, 0 replies; 12+ messages in thread

From: Pendekar Dikala Senja @ 2020-01-16 03:47 UTC (permalink / raw)
  To: Mahendra Singh Thalor <mahi6run@gmail.com>; +Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>

please find below :

sudo gdb -p 15803
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/;.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 15803
Reading symbols from /usr/lib/postgresql/12/bin/postgres...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/.build-id/b1/7c21299099640a6d863e423d99265824e7bb16.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from /usr/lib/x86_64-linux-gnu/libxml2.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.23.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libicui18n.so.55...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libicuuc.so.55...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libsystemd.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.23.so...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.23.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/liblzma.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.3...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libk5crypto.so.3...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcom_err.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5support.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libresolv.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libresolv-2.23.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libsasl2.so.2...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgssapi.so.3...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgnutls.so.30...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libicudata.so.55...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libselinux.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcrypt.so.20...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libkeyutils.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libheimntlm.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libkrb5.so.26...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libasn1.so.8...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libhcrypto.so.4...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libroken.so.18...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libp11-kit.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libidn.so.11...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libtasn1.so.6...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.6...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.4...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libpcre.so.3...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgpg-error.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libwind.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libheimbase.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libhx509.so.5...(no debugging symbols found)...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libsqlite3.so.0...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.23.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libffi.so.6...(no debugging symbols found)...done.
Reading symbols from /usr/lib/postgresql/12/lib/pg_stat_statements.so...(no debugging symbols found)...done.
Reading symbols from /usr/lib/postgresql/12/lib/pg_partman_bgw.so...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.23.so...done.
done.
0x00007f5ce4bf85b3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.

________________________________
From: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Sent: Wednesday, January 15, 2020 7:44 PM
To: Mahendra Singh Thalor <mahi6run@gmail.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

I didn't see where the path pg_ctl

$./pg_ctl -D /data/postgres/data -l logfile -c start
-su: ./pg_ctl: No such file or directory
$ pg_ctl -D /data/postgres/data -l logfile -c start
pg_ctl: command not found

on datadir I did not see core.PID, let me know where the dir.
$ ls -lrth
total 144K
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_twophase
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_tblspc
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_snapshots
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_serial
drwx------ 4 postgres postgres 4.0K Jan  2 16:34 pg_multixact
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_dynshmem
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_commit_ts
-rw------- 1 postgres postgres    3 Jan  2 16:34 PG_VERSION
-rw------- 1 postgres postgres   88 Jan  2 16:34 postgresql.auto.conf
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_xact
drwx------ 2 postgres postgres 4.0K Jan  2 16:34 pg_subtrans
drwx------ 6 postgres postgres 4.0K Jan  2 16:44 base
drwx------ 5 postgres postgres 4.0K Jan  3 18:29 pg_replslot
-rw------- 1 postgres postgres  127 Jan 15 12:38 postmaster.opts
drwx------ 2 postgres postgres 4.0K Jan 16 01:00 pg_log
-rw------- 1 postgres postgres   47 Jan 16 01:00 current_logfiles
drwx------ 3 postgres postgres  52K Jan 16 11:00 pg_wal
drwx------ 4 postgres postgres 4.0K Jan 16 11:40 pg_logical
-rw------- 1 postgres postgres   93 Jan 16 11:40 postmaster.pid
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat_tmp
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_notify
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 global
drwx------ 2 postgres postgres 4.0K Jan 16 11:40 pg_stat
$ cd postmaster.pid
-su: cd: postmaster.pid: Not a directory
$ more postmaster.pid
15803
/data/postgres/data
1579063112
5432
/var/run/postgresql
*
  5432001 169312256
ready

________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com>
Sent: Wednesday, January 15, 2020 7:30 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com>
Cc: Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Thu, Jan 16, 2020, 08:50 Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:
following your intruction. please find the information from my server below after running stact trace. I can't capture the PID because PID is always changing and not always the same.


sudo gdb -p 28164
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/;.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/;.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 28164
ptrace: No such process.

Yes, every time pid will change. You have to give '-c" option while stating the server to get call stack.
You have to give "gdb postgres data/core.pid".
This command should be fired from your bin(binray) folder.
Only running process can be attached like "gdb -p pid'.

Thanks and Regards
Mahendra Singh Thalor
EnterpriseDB: http://www.enterprisedb.com<http://www.enterprisedb.com/;


(gdb) set pagination off
(gdb) set logging file debuglog.txt
(gdb) set logging on
Copying output to debuglog.txt.
(gdb) cont
The program is not being run.
(gdb) detach
The program is not being run.
(gdb) b errfinish
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb)


thanks


________________________________
From: Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>>
Sent: Wednesday, January 15, 2020 2:08 AM
To: Mahendra Singh Thalor <mahi6run@gmail.com<mailto:mahi6run@gmail.com>>
Cc: Michael Paquier <michael@paquier.xyz<mailto:michael@paquier.xyz>>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org> <pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

Hi,
right now we're still trying to setup again, and will provide log later.


Kind Regards

your good friend
________________________________
From: Mahendra Singh Thalor <mahi6run@gmail.com<mailto:mahi6run@gmail.com>>
Sent: Wednesday, January 15, 2020 5:02:57 PM
To: Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>>
Cc: Michael Paquier <michael@paquier.xyz<mailto:michael@paquier.xyz>>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org> <pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>>
Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation

On Wed, 15 Jan 2020 at 09:32, Pendekar Dikala Senja <pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>> wrote:
>
>
> Here are the logs and our configuration.
> Actually we are facing problem when replicating partition table from 11.6 to 12.1, but if there is no partition table the replication works so well.
> As FYI, our server spec is 16 core and 64GB. And we are using this ubuntu
>
> LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.6 LTS
> Release: 16.04
> Codename: xenial
>
> Can you help us to find out the solution and configuration?
> Thanks for your help
>
> ________________________________ From: Michael Paquier
> Sent: Tuesday, January 14, 2020 6:58 PM
> To: pendekar.senja@outlook.com<mailto:pendekar.senja@outlook.com>; pgsql-bugs@lists.postgresql.org<mailto:pgsql-bugs@lists.postgresql.org>
> Subject: Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
>
> On Wed, Jan 15, 2020 at 02:00:48AM +0000, PG Bug reporting form wrote:
> > I got error when replicate with native logical method from PostgreSQL 11.6
> > to PostgreSQL 12.1, I will connecting table partition to table partition
> > from Version 11.6 to Version 12.1, but after all tables already replicated
> > around more than 2 hours, server can't access and recovery mode. please help
> > to advice, below log.
>
> Could you provide a backtrace?  Looking at the recent commit history,
> we had one fix which could be relevant to you after 12.1 was released:
> https://www.postgresql.org/message-id//16129-a0c0f48e71741e5f@postgresql.org
>
> And here is the commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a2aa224e
>
> The problem may be different though, so a backtrace would be good to
> check anyway.
> --
> Michael

Hi,
It looks like that server is crashed due to segmentation fault.
Can you provide call stack(stack trace) of Pid 25218.
To get call stack, you should start server with "-c" option.
Ex: ./pg_ctl -D data -l logfile -c start

How to get stack trace:
After crash, you can see core.PID file in your data directory. Attach core file to gdb and fire "bt" command.
Ex: gdb postgres data/core.25218
bt

--

^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-15 10:02 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
@ 2020-01-16 18:46   ` Christophe Pettus <xof@thebuild.com>
  2020-01-16 23:45     ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Michael Paquier <michael@paquier.xyz>
  1 sibling, 1 reply; 12+ messages in thread

From: Christophe Pettus @ 2020-01-16 18:46 UTC (permalink / raw)
  To: Mahendra Singh Thalor <mahi6run@gmail.com>; +Cc: Pendekar Dikala Senja <pendekar.senja@outlook.com>; Michael Paquier <michael@paquier.xyz>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>

We've been encountering the same bug; here's a stack trace from the crashed process:

(gdb) bt
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:427
#1  0x00005582bda1fb57 in memcpy (__len=425318492, __src=<optimized out>, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#2  tts_virtual_materialize (slot=0x5582bebca418) at ./build/../src/backend/executor/execTuples.c:235
#3  0x00005582bda21179 in ExecFetchSlotHeapTuple (slot=0x5582bebca418, materialize=<optimized out>, shouldFree=0x7ffeccfccc37) at ./build/../src/backend/executor/execTuples.c:1624
#4  0x00005582bd8b4e9c in heapam_tuple_update (relation=0x7f0bf1a92430, otid=0x5582bebca6a8, slot=0x5582bebca418, cid=943, snapshot=<optimized out>, crosscheck=0x0, wait=true,
    tmfd=0x7ffeccfcccc0, lockmode=0x7ffeccfcccbc, update_indexes=0x7ffeccfccd37) at ./build/../src/backend/access/heap/heapam_handler.c:325
#5  0x00005582bd8e4306 in table_tuple_update (update_indexes=0x7ffeccfccd37, lockmode=0x7ffeccfcccbc, tmfd=0x7ffeccfcccc0, wait=true, crosscheck=0x0, snapshot=0x0,
    cid=<optimized out>, slot=0x5582bebca418, otid=0x5582bebca6a8, rel=0x7f0bf1a92430) at ./build/../src/include/access/tableam.h:1261
#6  simple_table_tuple_update (rel=0x7f0bf1a92430, otid=otid@entry=0x5582bebca6a8, slot=slot@entry=0x5582bebca418, snapshot=0x0, update_indexes=update_indexes@entry=0x7ffeccfccd37)
    at ./build/../src/backend/access/table/tableam.c:325
#7  0x00005582bda1d462 in ExecSimpleRelationUpdate (estate=estate@entry=0x5582bebd1970, epqstate=epqstate@entry=0x7ffeccfcce10, searchslot=searchslot@entry=0x5582bebca678,
    slot=slot@entry=0x5582bebca418) at ./build/../src/backend/executor/execReplication.c:493
#8  0x00005582bdb1a188 in apply_handle_update (s=s@entry=0x7ffeccfd4420) at ./build/../src/backend/replication/logical/worker.c:750
#9  0x00005582bdb1affe in apply_dispatch (s=0x7ffeccfd4420) at ./build/../src/backend/replication/logical/worker.c:968
#10 LogicalRepApplyLoop (last_received=8839008698464) at ./build/../src/backend/replication/logical/worker.c:1176
#11 ApplyWorkerMain (main_arg=<optimized out>) at ./build/../src/backend/replication/logical/worker.c:1734
#12 0x00005582bdae972e in StartBackgroundWorker () at ./build/../src/backend/postmaster/bgworker.c:834
#13 0x00005582bdaf579c in do_start_bgworker (rw=0x5582beafcca0) at ./build/../src/backend/postmaster/postmaster.c:5770
#14 maybe_start_bgworkers () at ./build/../src/backend/postmaster/postmaster.c:5996
#15 0x00005582bdaf6235 in sigusr1_handler (postgres_signal_arg=<optimized out>) at ./build/../src/backend/postmaster/postmaster.c:5167
#16 <signal handler called>
#17 0x00007f1013bfaff7 in __GI___select (nfds=nfds@entry=8, readfds=readfds@entry=0x7ffeccfd5470, writefds=writefds@entry=0x0, exceptfds=exceptfds@entry=0x0,
    timeout=timeout@entry=0x7ffeccfd53d0) at ../sysdeps/unix/sysv/linux/select.c:41
#18 0x00005582bdaf67db in ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1668
#19 0x00005582bdaf8513 in PostmasterMain (argc=5, argv=0x5582beaaa090) at ./build/../src/backend/postmaster/postmaster.c:1377
#20 0x00005582bd86ecf6 in main (argc=5, argv=0x5582beaaa090) at ./build/../src/backend/main/main.c:228
(gdb)
--
-- Christophe Pettus
   xof@thebuild.com






^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation
  2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
  2020-01-15 10:02 ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Mahendra Singh Thalor <mahi6run@gmail.com>
  2020-01-16 18:46   ` Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Christophe Pettus <xof@thebuild.com>
@ 2020-01-16 23:45     ` Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2020-01-16 23:45 UTC (permalink / raw)
  To: Christophe Pettus <xof@thebuild.com>; +Cc: Mahendra Singh Thalor <mahi6run@gmail.com>; Pendekar Dikala Senja <pendekar.senja@outlook.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>

On Thu, Jan 16, 2020 at 10:46:55AM -0800, Christophe Pettus wrote:
> We've been encountering the same bug; here's a stack trace from the
> crashed process:
>
> [stack trace]

I cannot say if Pendekar's bug is the same issue, but yours looks very
familiar to that:
https://www.postgresql.org/message-id/16129-a0c0f48e71741e5f@postgresql.org

And this has already been fixed, in time for 12.2.
--
Michael

Attachments:

  signature.asc (application/pgp-signature, 832B) download

^ permalink  raw  reply  [nested|flat] 12+ messages in thread


end of thread, other threads:[~2020-01-16 23:45 UTC | newest]

Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-01-15 02:00 BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation PG Bug reporting form <noreply@postgresql.org>
2020-01-15 02:58 ` Michael Paquier <michael@paquier.xyz>
2020-01-15 04:02 Re: BUG #16205: background worker "logical replication worker" (PID 25218) was terminated by signal 11: Segmentation Pendekar Dikala Senja <pendekar.senja@outlook.com>
2020-01-15 04:42 ` Michael Paquier <michael@paquier.xyz>
2020-01-15 10:02 ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-01-15 10:08   ` Pendekar Dikala Senja <pendekar.senja@outlook.com>
2020-01-16 03:20     ` Pendekar Dikala Senja <pendekar.senja@outlook.com>
2020-01-16 03:30       ` Mahendra Singh Thalor <mahi6run@gmail.com>
2020-01-16 03:44         ` Pendekar Dikala Senja <pendekar.senja@outlook.com>
2020-01-16 03:47           ` Pendekar Dikala Senja <pendekar.senja@outlook.com>
2020-01-16 18:46   ` Christophe Pettus <xof@thebuild.com>
2020-01-16 23:45     ` Michael Paquier <michael@paquier.xyz>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox