Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oOff4-0007Lq-D7 for pgsql-hackers@arkaria.postgresql.org; Thu, 18 Aug 2022 13:34:14 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1oOff3-0007cO-B0 for pgsql-hackers@arkaria.postgresql.org; Thu, 18 Aug 2022 13:34:13 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oOff2-0007cC-Om for pgsql-hackers@lists.postgresql.org; Thu, 18 Aug 2022 13:34:13 +0000 Received: from mail-io1-xd2c.google.com ([2607:f8b0:4864:20::d2c]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1oOfez-0000Hk-Ci for pgsql-hackers@postgresql.org; Thu, 18 Aug 2022 13:34:11 +0000 Received: by mail-io1-xd2c.google.com with SMTP id h78so1059379iof.13 for ; Thu, 18 Aug 2022 06:34:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20210112.gappssmtp.com; s=20210112; h=user-agent:content-transfer-encoding:content-disposition :mime-version:message-id:subject:cc:to:from:date:from:to:cc; bh=PNOaAsQeoMsqcaFOPmx5dY9x4kNFMn0/t1VhOqPEgw8=; b=bgYnz+w4tCo0/27vWFb34tYL5SXDU3nqsG5S7SCyK1RVH3xGyw+LglS4i1H7/H+zuE rTVrGAQyqytP2t6SzEj84dxY+NwW6m3qqB1yoz5YLd88XieDR8bvFLG7ugJxjQnUesgh E62/QT88UAnfIRZkgqig8WRwngd40HhAnzisRElpJfmuw3McV6r1d7+6gvbQLVierArp 9yny0OFThTr+7HcmA/O2+kMILvHkv95pZdRGo3xYUDuKX7Dt0zaojvHBikJkto4NoffF XN99i6jhYEq6ViActWMBUqZV0RxAf+pV5jRaV81393wtY7k6djGp86OPw3gE0nGy64Yo H3bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=user-agent:content-transfer-encoding:content-disposition :mime-version:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc; bh=PNOaAsQeoMsqcaFOPmx5dY9x4kNFMn0/t1VhOqPEgw8=; b=S7W3khVxwrW8S1AUIXVf9e0gNeEqO+Z7sb5MRiEio+3t4se13jHscJAoOeaDhHMPSR z0S8Sbkrt/l95cRotig79aL8hrzc3PPQUoWwa5USXtpoOCCN8CyzUnr2LA1UM4ggUi2S G/re2G7n8vXCWLBuQqiwr5pYhUkURT6O9zU/Nqfks2Cv55GOUuEfTUKPaFTePTd2+X6q bQcdgHQez95bG1AHi7hW4l6k/7zroAXmADVNUEbMy/oL3z2MomXpY2a5TetlbY+MHodd 2u8wRPcXPKFWUWIqBDCNTfSsHe2rhlgok2z01dHCoAqtIyy6hdsId39Ba+tkQ9rESwNc iJ9g== X-Gm-Message-State: ACgBeo0LBxGS76k3YDxUXbl1UhHHl3iakbLvDDLX5R3EMHuMeIzDl3KT lAr3+eGCyjtBKtXnAgUOotvTvg== X-Google-Smtp-Source: AA6agR6A2M8w6v++HQpleu6IjgSDKParG/4Vou9b2R9ZQVCco5956Cz9BIHNQZO8obhPWWSXIiB0OQ== X-Received: by 2002:a02:5b4a:0:b0:343:5a53:6937 with SMTP id g71-20020a025b4a000000b003435a536937mr1404236jab.79.1660829648154; Thu, 18 Aug 2022 06:34:08 -0700 (PDT) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id e89-20020a028662000000b00346bda133a6sm596864jai.116.2022.08.18.06.34.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Aug 2022 06:34:07 -0700 (PDT) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id 059C580071F; Thu, 18 Aug 2022 08:34:06 -0500 (CDT) Date: Thu, 18 Aug 2022 08:34:06 -0500 From: Justin Pryzby To: pgsql-hackers@postgresql.org Cc: Amit Kapila , Masahiko Sawada Subject: pg15b3: crash in paralell vacuum Message-ID: <20220818133406.GL26426@telsasoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Immediately after upgrading an internal instance, a loop around "vacuum" did this: TRAP: FailedAssertion("indstats->status =3D=3D PARALLEL_INDVAC_STATUS_INITI= AL", File: "vacuumparallel.c", Line: 611, PID: 27635) postgres: postgres pryzbyj [local] VACUUM(ExceptionalCondition+0x8d)[0x99d9= fd] postgres: postgres pryzbyj [local] VACUUM[0x6915db] postgres: postgres pryzbyj [local] VACUUM(heap_vacuum_rel+0x12b6)[0x5083e6] postgres: postgres pryzbyj [local] VACUUM[0x68e97a] postgres: postgres pryzbyj [local] VACUUM(vacuum+0x48e)[0x68fe9e] postgres: postgres pryzbyj [local] VACUUM(ExecVacuum+0x2ae)[0x69065e] postgres: postgres pryzbyj [local] VACUUM(standard_ProcessUtility+0x530)[0x= 8567b0] /usr/pgsql-15/lib/pg_stat_statements.so(+0x5450)[0x7f52b891c450] postgres: postgres pryzbyj [local] VACUUM[0x85490a] postgres: postgres pryzbyj [local] VACUUM[0x854a53] postgres: postgres pryzbyj [local] VACUUM(PortalRun+0x179)[0x855029] postgres: postgres pryzbyj [local] VACUUM[0x85099b] postgres: postgres pryzbyj [local] VACUUM(PostgresMain+0x199a)[0x85268a] postgres: postgres pryzbyj [local] VACUUM[0x496a21] postgres: postgres pryzbyj [local] VACUUM(PostmasterMain+0x11c0)[0x7b3980] postgres: postgres pryzbyj [local] VACUUM(main+0x1c6)[0x4986a6] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f52c4b893d5] postgres: postgres pryzbyj [local] VACUUM[0x498c59] < 2022-08-18 07:56:51.963 CDT >LOG: server process (PID 27635) was termin= ated by signal 6: Aborted < 2022-08-18 07:56:51.963 CDT >DETAIL: Failed process was running: VACUUM= ANALYZE alarms Unfortunately, it looks like the RPM packages are compiled with -O2, so thi= s is of limited use. Core was generated by `postgres: postgres pryzbyj [local] VACUUM = '. Program terminated with signal 6, Aborted. #0 0x00007f52c4b9d207 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install audit-libs-2.8.4-4.el7.= x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-23.el7.x86_64 e= lfutils-libelf-0.176-5.el7.x86_64 elfutils-libs-0.176-5.el7.x86_64 glibc-2.= 17-260.el7_6.3.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-51.= el7_9.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-9.el7.x86_64 libcap-n= g-0.7.5-4.el7.x86_64 libcom_err-1.42.9-19.el7.x86_64 libgcc-4.8.5-39.el7.x8= 6_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50= =2E1.2-17.el7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-39.el7.x8= 6_64 libxml2-2.9.1-6.el7_9.6.x86_64 libzstd-1.5.2-1.el7.x86_64 lz4-1.7.5-2.= el7.x86_64 nspr-4.19.0-1.el7_5.x86_64 nss-3.36.0-7.1.el7_6.x86_64 nss-softo= kn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.1.el7_6.x86_64 openldap-2= =2E4.44-21.el7_6.x86_64 openssl-libs-1.0.2k-22.el7_9.x86_64 pam-1.1.8-22.el= 7.x86_64 pcre-8.32-17.el7.x86_64 systemd-libs-219-62.el7_6.5.x86_64 xz-libs= -5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64 (gdb) bt #0 0x00007f52c4b9d207 in raise () from /lib64/libc.so.6 #1 0x00007f52c4b9e8f8 in abort () from /lib64/libc.so.6 #2 0x000000000099da1e in ExceptionalCondition (conditionName=3DconditionNa= me@entry=3D0xafae40 "indstats->status =3D=3D PARALLEL_INDVAC_STATUS_INITIAL= ", errorType=3DerrorType@entry=3D0x9fb4b7 "FailedAssertion",=20 fileName=3DfileName@entry=3D0xafb0c0 "vacuumparallel.c", lineNumber=3Dl= ineNumber@entry=3D611) at assert.c:69 #3 0x00000000006915db in parallel_vacuum_process_all_indexes (pvs=3D0x2e85= f80, num_index_scans=3D, vacuum=3D) at vacuum= parallel.c:611 #4 0x00000000005083e6 in heap_vacuum_rel (rel=3D, params=3D= , bstrategy=3D) at vacuumlazy.c:2679 #5 0x000000000068e97a in table_relation_vacuum (bstrategy=3D, params=3D0x7fff46de9a80, rel=3D0x7f52c7bc2c10) at ../../../src/include/a= ccess/tableam.h:1680 #6 vacuum_rel (relid=3D52187497, relation=3D, params=3D0x7f= ff46de9a80) at vacuum.c:2092 #7 0x000000000068fe9e in vacuum (relations=3D0x2dbeee8, params=3Dparams@en= try=3D0x7fff46de9a80, bstrategy=3D, bstrategy@entry=3D0x0, i= sTopLevel=3DisTopLevel@entry=3Dtrue) at vacuum.c:475 #8 0x000000000069065e in ExecVacuum (pstate=3Dpstate@entry=3D0x2dc38d0, va= cstmt=3Dvacstmt@entry=3D0x2d9f3a0, isTopLevel=3DisTopLevel@entry=3Dtrue) at= vacuum.c:275 #9 0x00000000008567b0 in standard_ProcessUtility (pstmt=3Dpstmt@entry=3D0x= 2d9f7a0, queryString=3DqueryString@entry=3D0x2d9e8a0 "VACUUM ANALYZE alarms= ", readOnlyTree=3D, context=3Dcontext@entry=3DPROCESS_UTILIT= Y_TOPLEVEL,=20 params=3Dparams@entry=3D0x0, queryEnv=3DqueryEnv@entry=3D0x0, dest=3Dde= st@entry=3D0x2d9f890, qc=3Dqc@entry=3D0x7fff46dea0c0) at utility.c:866 #10 0x00007f52b891c450 in pgss_ProcessUtility (pstmt=3D0x2d9f7a0, queryStri= ng=3D0x2d9e8a0 "VACUUM ANALYZE alarms", readOnlyTree=3D, con= text=3DPROCESS_UTILITY_TOPLEVEL, params=3D0x0, queryEnv=3D0x0, dest=3D0x2d9= f890,=20 qc=3D0x7fff46dea0c0) at pg_stat_statements.c:1143 #11 0x000000000085490a in PortalRunUtility (portal=3Dportal@entry=3D0x2e20f= c0, pstmt=3D0x2d9f7a0, isTopLevel=3DisTopLevel@entry=3Dtrue, setHoldSnapsho= t=3DsetHoldSnapshot@entry=3Dfalse, dest=3D0x2d9f890, qc=3D0x7fff46dea0c0) a= t pquery.c:1158 #12 0x0000000000854a53 in PortalRunMulti (portal=3Dportal@entry=3D0x2e20fc0= , isTopLevel=3DisTopLevel@entry=3Dtrue, setHoldSnapshot=3DsetHoldSnapshot@e= ntry=3Dfalse, dest=3Ddest@entry=3D0x2d9f890, altdest=3Daltdest@entry=3D0x2d= 9f890,=20 qc=3Dqc@entry=3D0x7fff46dea0c0) at pquery.c:1322 #13 0x0000000000855029 in PortalRun (portal=3D0x2e20fc0, count=3D9223372036= 854775807, isTopLevel=3D, run_once=3D, dest= =3D0x2d9f890, altdest=3D0x2d9f890, qc=3D0x7fff46dea0c0) at pquery.c:791 #14 0x000000000085099b in exec_simple_query (query_string=3D0x2d9e8a0 "VACU= UM ANALYZE alarms") at postgres.c:1250 #15 0x000000000085268a in PostgresMain (dbname=3D, username= =3D) at postgres.c:4581 #16 0x0000000000496a21 in BackendRun (port=3D, port=3D) at postmaster.c:4504 #17 BackendStartup (port=3D0x2dbe9c0) at postmaster.c:4232 #18 ServerLoop () at postmaster.c:1806 #19 0x00000000007b3980 in PostmasterMain (argc=3Dargc@entry=3D3, argv=3Darg= v@entry=3D0x2d99280) at postmaster.c:1478 #20 0x00000000004986a6 in main (argc=3D3, argv=3D0x2d99280) at main.c:202 (gdb) p *pvs $2 =3D {pcxt =3D 0x2e84490, indrels =3D 0x2e84220, nindexes =3D 8, shared = =3D 0x2aaaaf142380, indstats =3D 0x2aaaaf1423c0, dead_items =3D 0x2aaaab142= 380, buffer_usage =3D 0x2aaaab142260, wal_usage =3D 0x2aaaab142220,=20 will_parallel_vacuum =3D 0x2e8f750, nindexes_parallel_bulkdel =3D 5, nind= exes_parallel_cleanup =3D 0, nindexes_parallel_condcleanup =3D 5, bstrategy= =3D 0x2dbed40, relnamespace =3D 0x0, relname =3D 0x0, indname =3D 0x0,=20 status =3D PARALLEL_INDVAC_STATUS_INITIAL} (gdb) info locals=20 indstats =3D i =3D nworkers =3D 2 (gdb) p *pvs $4 =3D {pcxt =3D 0x2e84490, indrels =3D 0x2e84220, nindexes =3D 8, shared = =3D 0x2aaaaf142380, indstats =3D 0x2aaaaf1423c0, dead_items =3D 0x2aaaab142= 380, buffer_usage =3D 0x2aaaab142260, wal_usage =3D 0x2aaaab142220,=20 will_parallel_vacuum =3D 0x2e8f750, nindexes_parallel_bulkdel =3D 5, nind= exes_parallel_cleanup =3D 0, nindexes_parallel_condcleanup =3D 5, bstrategy= =3D 0x2dbed40, relnamespace =3D 0x0, relname =3D 0x0, indname =3D 0x0,=20 status =3D PARALLEL_INDVAC_STATUS_INITIAL} I reproduced it like this: pryzbyj=3D# VACUUM (PARALLEL 2,VERBOSE,INDEX_CLEANUP on) alarms; -- DISABLE= _PAGE_SKIPPING true INFO: vacuuming "pryzbyj.public.alarms" server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. The connection to the server was lost. Attempting reset: Failed. The connection to the server was lost. Attempting reset: Failed. So I'll be back shortly with more... --=20 Justin