Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jJZJ3-0007HU-T1 for pgsql-bugs@arkaria.postgresql.org; Wed, 01 Apr 2020 09:05:05 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jJZJ2-0003XH-Di for pgsql-bugs@arkaria.postgresql.org; Wed, 01 Apr 2020 09:05:04 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jJZJ2-0003X9-42 for pgsql-bugs@lists.postgresql.org; Wed, 01 Apr 2020 09:05:04 +0000 Received: from mail-lf1-x144.google.com ([2a00:1450:4864:20::144]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1jJZIz-0002nX-Sv for pgsql-bugs@lists.postgresql.org; Wed, 01 Apr 2020 09:05:03 +0000 Received: by mail-lf1-x144.google.com with SMTP id h6so14013677lfp.6 for ; Wed, 01 Apr 2020 02:05:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=SOwzASc+Jp1SmrOtN62LvPVUYCT27/OFQCSZiNztEXw=; b=ppTQT2w3hAQ2p+Eo8Nom/v5RT4FZp1AQmnNA2SevkskNn0Geq9Rt69E8OxNetyGkwH qyD88yB99NwzdknHgwL4MfiK8BTvFEPtit8epFGTR6rGtEdD03hEGDtaIoBndFGqbo5G jYW+GRRJ5rs6DzDIwNVH+ImELtQY6YKoUfpOi/FfgACTSlEJYsNZpjetBIY6GRlbKVW2 FiUAtv6Un4csRrz9jTFmAbZ4WuWfhP0mc9HJfzO9/dmMn/gpE1JP9ZFwinq5iiR+9p6z TNB32QVgD+bz2oTciI2K5rZ4ZSWqBW4QL++jCVFpKtADioPWn2EcZFVaBaM9nv41sTqa u8PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=SOwzASc+Jp1SmrOtN62LvPVUYCT27/OFQCSZiNztEXw=; b=Fp2I9eWzaS7RWsS2Y1H9nvyeeu4GjLqUmvJySS9bN1WBGyTc1b+tE28SZFpdl1WYhD OZvmqgrK+Ptnl0SuPU3hOA+Slhr8j+gwDaF7zPURhvy9H4VuWPeFqh3NwsAEsTFUy5Fh RQayeMiCCegG/LZ5BQoDHLmEi4R3OvpGKz+6pCTgtRa35n8qcNb8fgf6vphCjzmhe+9y jMrOCQOquQBUf/bYaEJkkBsu6/yXyKtjuJlZQgTKwUiMx/VTUuraqcn4PRUNIlhts6rf ggVYpyy119FxK4RY7uQygbLcYefUcD9gB4RVCqU0dso4OLK3JiQyfJ9CDO/tWcNGF6pu p7Xw== X-Gm-Message-State: AGi0PuZplBh7k+mIdINXq7O9hSvxOUD78sgxWCORwdgzVULA0AxKSLWk /diUSIGDQEQhFayRxIBWOow= X-Google-Smtp-Source: APiQypIRX4jD/1n6XkdJ+rad+8PvGMfZspjCV6m60AqWfnej7sSxjnNrYsw+bRAdSMy6BK5B2sDxqw== X-Received: by 2002:a19:c210:: with SMTP id l16mr12153930lfc.85.1585731900082; Wed, 01 Apr 2020 02:05:00 -0700 (PDT) Received: from nol (82-64-124-11.subs.proxad.net. [82.64.124.11]) by smtp.gmail.com with ESMTPSA id s6sm805753ljm.58.2020.04.01.02.04.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Apr 2020 02:04:59 -0700 (PDT) Date: Wed, 1 Apr 2020 11:04:55 +0200 From: Julien Rouhaud To: jmlich83@gmail.com, pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16331: segfault in checkpointer with full disk Message-ID: <20200401090455.GB82418@nol> References: <16331-16d6e966536b6390@postgresql.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <16331-16d6e966536b6390@postgresql.org> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Hi, On Wed, Apr 01, 2020 at 08:51:56AM +0000, PG Bug reporting form wrote: > The following bug has been logged on the website: > > Bug reference: 16331 > Logged by: Jozef Mlich > Email address: jmlich83@gmail.com > PostgreSQL version: 12.2 > Operating system: CentOS > Description: > > I can see segfaults on CentOS 7 with postgresql 12.2-2PGDG.rhel7 (from > yum.postgresql.org). I am using multiple extensions (cstore, postgres_fdw, > pgcrypto,dblink, etc.). It seems crash is related to disk run out of space > (I am using separate partion for / and for /var/lib/pgsql). It occurs few > times a day. According to backtrace it seems to be related to checkpointer. > Replication is not configured. > > > [New LWP 26290] > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `postgres: checkpointer > '. > Program terminated with signal 6, Aborted. > #0 0x00007fe4604c1207 in __GI_raise (sig=sig@entry=6) at > ../nptl/sysdeps/unix/sysv/linux/raise.c:55 > 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); > > Thread 1 (Thread 0x7fe462e148c0 (LWP 26290)): > #0 0x00007fe4604c1207 in __GI_raise (sig=sig@entry=6) at > ../nptl/sysdeps/unix/sysv/linux/raise.c:55 > resultvar = 0 > pid = 26290 > selftid = 26290 > #1 0x00007fe4604c28f8 in __GI_abort () at abort.c:90 > save_stage = 2 > act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, > sa_mask = {__val = {0, 0, 0, 0, 0, 9268713, 70403103920717, > 39808819211026438, 20126216749056, 70394513997832, 9268713, 70403103920719, > 17316096998686159616, 20134806683648, 140618848608704, 140618848592800}}, > sa_flags = 1615828275, sa_restorer = 0x0} > sigs = {__val = {32, 0 }} > #2 0x000000000087840a in errfinish (dummy=) at elog.c:552 > edata = 0xd47040 > elevel = 22 > oldcontext = 0x171a6d0 > econtext = 0x0 > __func__ = "errfinish" > #3 0x0000000000706b24 in CheckPointReplicationOrigin () at origin.c:562 > tmppath = 0x9e6fa8 "pg_logical/replorigin_checkpoint.tmp" > path = 0x9e6fd0 "pg_logical/replorigin_checkpoint" > tmpfd = > i = > magic = 307747550 > crc = 4294967295 > __func__ = "CheckPointReplicationOrigin" That's not a bug (nor a segfault) but the expected behavior if the checkpointer is not able to do its work. As data durability can't be guaranteed in such case, the checkpointer raises a PANIC level message, which raises an abort so that the whole instance do an emergency restart cycle. Do you have monitoring for this filesystem? Do you see spikes in disk usage or other strange behavior?