Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mkpy1-00038K-FZ for pgsql-hackers@arkaria.postgresql.org; Wed, 10 Nov 2021 15:56:53 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mkpy0-0000s3-5e for pgsql-hackers@arkaria.postgresql.org; Wed, 10 Nov 2021 15:56:52 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mkpxz-0000rt-Ng for pgsql-hackers@lists.postgresql.org; Wed, 10 Nov 2021 15:56:51 +0000 Received: from mail-io1-xd2d.google.com ([2607:f8b0:4864:20::d2d]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1mkpxw-0005e7-3g for pgsql-hackers@postgresql.org; Wed, 10 Nov 2021 15:56:51 +0000 Received: by mail-io1-xd2d.google.com with SMTP id y16so3321392ioc.8 for ; Wed, 10 Nov 2021 07:56:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=a/OFX9vwpdT/O8RU6GvwsIrR+9jGUCz43Rth675fv0A=; b=Dqakxo1M77JnK5xrrfWushsCyJU5+51yg5/2icMIG2we4Ljm0pFevaYELW+HXGUnk2 CzwCfK/X9PgylYaug6FN3W1O0uJnXAzdk6BdJD6k6jOu8CTZ4UQqkJSIWbFc/20vIIkX 6+Wgw0bKFTyZ57ClnhN9Mc+AyHx50XQbCAtKH7qhqdSykiq5aqKYlCXDLXo3Dbpd6G0G AY2sqMA4OSuUaJ6telwz3cU5prNZUbOmQCACa7/w5yFCGZDTAYs14jkd6SwlGvFjTn+T utMVBgv5nneTK5ksB5Adrq10KV6CJlvYYTKSlK+DXvA7lTEuihg5jwx9yuF2hn8nDq1H Trkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=a/OFX9vwpdT/O8RU6GvwsIrR+9jGUCz43Rth675fv0A=; b=HY4ljJ2oPb9NBthmQDel5oZXicHZY4kY/qPtMFnmNJyc62/HAD6N+ZHZy1mIfz7xn1 xW17yZRJqirkJuHNv+MOBSuydai01fDqaTRsOtS5JPabHuhsn1HrYJzJr1Z7tOtWvazV 198QjWg43x6rqbt+hIGF75Y59nozSV7lQ0EkR041I8ILGlYBDOs1obdk7LMFlfmzHjgR ZqkD/M7wBxWenI6n10XxaGbDTDb+UuWiGLA/JTkBrRKNLc8OjKPoueYvkV3IosJxaBIJ AwixuwIEKwY2VtfAWHQ/DDu03V1rC6qCdMnP5L/Z5KiIGkVM4EdIt5oYay6/f2HcwvCp 9nsA== X-Gm-Message-State: AOAM532C/5hvw/2SnLpOp+Jfb2/X1U3kO8coopkt+4ApWWG6hxGruok1 fQ3oPrHE1a4ti/WqryfEjAj0gQ== X-Google-Smtp-Source: ABdhPJyKYouBb0JBxdo+ZFVHDo68xsGWezOv+P7/gCeV/Im1fBYZRyNyPM6uZBSzagxunjUschzxgA== X-Received: by 2002:a05:6602:15d0:: with SMTP id f16mr97484iow.196.1636559806069; Wed, 10 Nov 2021 07:56:46 -0800 (PST) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id p14sm146092ilo.39.2021.11.10.07.56.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Nov 2021 07:56:45 -0800 (PST) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id DA3058005D3; Wed, 10 Nov 2021 09:56:44 -0600 (CST) Date: Wed, 10 Nov 2021 09:56:44 -0600 From: Justin Pryzby To: Andres Freund Cc: pgsql-hackers@postgresql.org, Tom Lane Subject: Re: terminate called after throwing an instance of 'std::bad_alloc' Message-ID: <20211110155644.GA17618@telsasoft.com> References: <20201001021609.GC8476@telsasoft.com> <20201003220236.GL8476@telsasoft.com> <20201003230149.mtd7fjsjwgii3jv7@alap3.anarazel.de> <20201003233045.GM8476@telsasoft.com> <20201004020127.zbrol2ypu6tg2bei@alap3.anarazel.de> <20201007230242.rnwsbnxywvthef3w@alap3.anarazel.de> <20201218235607.GC30237@telsasoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20201218235607.GC30237@telsasoft.com> User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Thread starting here: https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com On Fri, Dec 18, 2020 at 05:56:07PM -0600, Justin Pryzby wrote: > I'm 99% sure the "bad_alloc" is from LLVM. It happened multiple times on > different servers (running a similar report) after setting jit=on during pg13 > upgrade, and never happened since re-setting jit=off. Since this recurred a few times recently (now running pg14.0), and I finally managed to get a non-truncated corefile... Core was generated by `postgres: telsasoft ts 192.168.122.13(40808) SELECT '. Program terminated with signal 6, Aborted. #0 0x00007f2497880337 in raise () from /lib64/libc.so.6 #1 0x00007f2497881a28 in abort () from /lib64/libc.so.6 #2 0x00007f2487cbf265 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/llvm5.0/lib/libLLVM-5.0.so #3 0x00007f2487c66696 in __cxxabiv1::__terminate(void (*)()) () from /usr/lib64/llvm5.0/lib/libLLVM-5.0.so #4 0x00007f2487c666c3 in std::terminate() () from /usr/lib64/llvm5.0/lib/libLLVM-5.0.so #5 0x00007f2487c687d3 in __cxa_throw () from /usr/lib64/llvm5.0/lib/libLLVM-5.0.so #6 0x00007f2487c686cd in operator new(unsigned long) () from /usr/lib64/llvm5.0/lib/libLLVM-5.0.so #7 0x00007f2486477b9c in llvm::DenseMap >, llvm::DenseMapAPIntKeyInfo, llvm::detail::DenseMapPair > > >::grow(unsigned int) () from /usr/lib64/llvm5.0/lib/libLLVM-5.0.so #8 0x00007f2486477f29 in llvm::ConstantInt::get(llvm::LLVMContext&, llvm::APInt const&) () from /usr/lib64/llvm5.0/lib/libLLVM-5.0.so #9 0x00007f2486478263 in llvm::ConstantInt::get(llvm::IntegerType*, unsigned long, bool) () from /usr/lib64/llvm5.0/lib/libLLVM-5.0.so #10 0x00007f2488f66c18 in l_ptr_const (type=0x3000650, ptr=) at ../../../../src/include/jit/llvmjit_emit.h:29 #11 llvm_compile_expr (state=) at llvmjit_expr.c:246 #12 0x0000000000635499 in ExecReadyExpr (state=state@entry=0x1a5317178) at execExpr.c:885 #13 0x00000000006388a5 in ExecInitQual (qual=0x1a6d07de8, parent=parent@entry=0x1a4c4ade0) at execExpr.c:280 #14 0x0000000000661d9d in ExecInitHashJoin (node=node@entry=0x1a6c935b0, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeHashjoin.c:733 #15 0x0000000000648f6e in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:307 #16 0x000000000065e8fd in ExecInitHash (node=, estate=, eflags=) at nodeHash.c:381 #17 0x0000000000648e87 in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:365 #18 0x0000000000661cd5 in ExecInitHashJoin (node=node@entry=0x1a6c93ce0, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeHashjoin.c:663 #19 0x0000000000648f6e in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:307 #20 0x00000000006723e4 in ExecInitSort (node=node@entry=0x1a6c94a50, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeSort.c:210 #21 0x0000000000648f2f in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:320 #22 0x0000000000655ef9 in ExecInitAgg (node=node@entry=0x1a6c95100, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeAgg.c:3382 #23 0x0000000000648ef0 in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:340 #24 0x0000000000674628 in ExecInitSubqueryScan (node=node@entry=0x1a6c94020, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeSubqueryscan.c:126 #25 0x0000000000649055 in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:249 #26 0x0000000000658e61 in ExecInitAppend (node=node@entry=0x1a6c60008, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeAppend.c:232 #27 0x0000000000649166 in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:181 #28 0x0000000000670e1a in ExecInitResult (node=node@entry=0x1a6c95c40, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeResult.c:210 #29 0x00000000006491a5 in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:166 #30 0x0000000000658e61 in ExecInitAppend (node=node@entry=0x1a6c5d548, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeAppend.c:232 #31 0x0000000000649166 in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:181 #32 0x0000000000674628 in ExecInitSubqueryScan (node=node@entry=0x1a6ca5368, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeSubqueryscan.c:126 #33 0x0000000000649055 in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:249 #34 0x00000000006723e4 in ExecInitSort (node=node@entry=0x1a6ca5a28, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeSort.c:210 #35 0x0000000000648f2f in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:320 #36 0x00000000006793eb in ExecInitWindowAgg (node=node@entry=0x1a6ca6ab0, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeWindowAgg.c:2314 #37 0x0000000000648edb in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:345 #38 0x0000000000674628 in ExecInitSubqueryScan (node=node@entry=0x1a6ca5628, estate=estate@entry=0x1a4da87b8, eflags=eflags@entry=16) at nodeSubqueryscan.c:126 #39 0x0000000000649055 in ExecInitNode (node=, estate=, eflags=) at execProcnode.c:249 #40 0x0000000000642ab1 in InitPlan (eflags=16, queryDesc=0x344dc88) at execMain.c:936 #41 standard_ExecutorStart (queryDesc=0x344dc88, eflags=16) at execMain.c:263 #42 0x00007f248b45656d in pgss_ExecutorStart (queryDesc=0x344dc88, eflags=) at pg_stat_statements.c:965 #43 0x00007f248b03a861 in explain_ExecutorStart (queryDesc=0x344dc88, eflags=) at auto_explain.c:301 #44 0x00000000007b8d1b in PortalStart (portal=0x1be2168, params=0x0, eflags=0, snapshot=0x0) at pquery.c:512 -- Justin