Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from ) id 1jA43d-0003vE-5e for pgsql-hackers@arkaria.postgresql.org; Fri, 06 Mar 2020 03:53:53 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1jA43a-0000bQ-I8 for pgsql-hackers@arkaria.postgresql.org; Fri, 06 Mar 2020 03:53:50 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1jA43a-0000bI-3b for pgsql-hackers@lists.postgresql.org; Fri, 06 Mar 2020 03:53:50 +0000 Received: from sender4-op-o15.zoho.com ([136.143.188.15]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from ) id 1jA43W-0005uE-HN for pgsql-hackers@lists.postgresql.org; Fri, 06 Mar 2020 03:53:48 +0000 ARC-Seal: i=1; a=rsa-sha256; t=1583466819; cv=none; d=zohomail.com; s=zohoarc; b=U8ET+cUqcY3Lz0j7Y1aEZNg6wDLE87Cr5oUGIrpkOAJWKmA/UupBKIexEg63fzNNuyu/skJdaZ80m7JP5sqBuqEVpmpV74LMCIASIxh0fwIR3srSK5FCsJ7OILHJr3HeENj2i1363Iz5u4+VlzPxm+veazxrDZYbbtO1n0SbBxU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583466819; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=e6/bXiVlhdE+u6nBBKerzrnc+fKOcKHkEz4O1t9BYz4=; b=GK2fw1fkoxUKzYpBYKj74dCorF6yewQQQBByjGvdqCZEK3K4cst5pvw6E1P8js6GbCN2EigLHCkIJhbSYUewHc9KQh65klnHglzBpTFakXju9ikA+uKKa/dn9ABB8Z7ymYuvr3WDzChllky62q5Tk05LQiktxchH2BQsDyUHvL8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=highgo.ca; spf=pass smtp.mailfrom=david.zhang@highgo.ca; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1583466819; s=zoho; d=highgo.ca; i=david.zhang@highgo.ca; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=e6/bXiVlhdE+u6nBBKerzrnc+fKOcKHkEz4O1t9BYz4=; b=bViUh1QZPaXP4XCZHKm4ejirndNNKnW2yYuP6HE7fUIu93qFeGCeDQ9d7CJUDzm7 mFj6FoGsYHDTspnRAkNWp1IlIYXqB8XQG06nkxbBYepjIdrAc4Po8KOCN2pcLym0dCx WkPxrYnR1PJetPhabzhwB2VnFkI3XD4QEePJ3QJ0= Received: from david.highgo.ca (S01061056118cdb19.vf.shawcable.net [70.71.187.22]) by mx.zohomail.com with SMTPS id 158346681662048.06726161217932; Thu, 5 Mar 2020 19:53:36 -0800 (PST) Subject: Re: BUG #16147: postgresql 12.1 (from homebrew) - pg_restore -h localhost --jobs=2 crashes To: Tom Lane , PostgreSQL Hackers References: <16147-34dfe783aba141b0@postgresql.org> <15364.1575468206@sss.pgh.pa.us> From: David Zhang Message-ID: <2a1b69ea-6a33-cacb-8b3e-32886cb84368@highgo.ca> Date: Thu, 5 Mar 2020 19:53:35 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <15364.1575468206@sss.pgh.pa.us> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-ZohoMailClient: External List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Hi, I can reproduce this pg_restore crash issue (pg_dump crash too when=20 running with multiple jobs) on MacOS 10.14 Mojave and MacOS 10.15=20 Catalina using following steps. 1. build pg_resotre from 12.2 with "--with-gssapi" enabled, or use the=20 release from https://www.postgresql.org/download/macosx/ 2. start pg server and generate some load, =C2=A0=C2=A0=C2=A0 pgbench -i -p 5432 -d postgres -s 10" 3. backup database, =C2=A0=C2=A0=C2=A0 pg_dump -h localhost -Fc --no-acl --no-owner postgres >= /tmp/128m 4. drop the tables, =C2=A0=C2=A0=C2=A0 psql -d postgres -c "drop table pgbench_accounts; drop = table=20 pgbench_branches; drop table pgbench_history; drop table pgbench_tellers;" 5. restore database, =C2=A0=C2=A0=C2=A0 pg_restore -d postgres -h localhost -Fc /tmp/128m --job= s=3D2 =C2=A0=C2=A0=C2=A0 Password: =C2=A0=C2=A0=C2=A0 pg_restore: error: a worker process died unexpectedly 6. check tables, all display size 0 bytes. =C2=A0=C2=A0=C2=A0 postgres=3D# \d+ =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 List of relations =C2=A0=C2=A0=C2=A0 =C2=A0Schema |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Name= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | Type=C2=A0 |=C2=A0 Owner=C2=A0=C2=A0= |=C2=A0 Size=C2=A0=C2=A0 | Description --------+------------------+-------+----------+---------+------------- =C2=A0=C2=A0=C2=A0 =C2=A0public | pgbench_accounts | table | postgres | 0 = bytes | =C2=A0=C2=A0=C2=A0 =C2=A0public | pgbench_branches | table | postgres | 0 = bytes | =C2=A0=C2=A0=C2=A0 =C2=A0public | pgbench_history=C2=A0 | table | postgres= | 0 bytes | =C2=A0=C2=A0=C2=A0 =C2=A0public | pgbench_tellers=C2=A0 | table | postgres= | 0 bytes | =C2=A0=C2=A0=C2=A0 (4 rows) 7. core dump, about 2G, (lldb) bt all * thread #1, stop reason =3D signal SIGSTOP =C2=A0 * frame #0: 0x00007fff6c29c44e=20 libdispatch.dylib`_dispatch_mgr_queue_push + 41 =C2=A0=C2=A0=C2=A0 frame #1: 0x00007fff41475a74=20 Security`___ZN8Security12KeychainCore14StorageManager14tickleKeychainEPNS0_= 12KeychainImplE_block_invoke_2=20 + 76 =C2=A0=C2=A0=C2=A0 frame #2: 0x00007fff6c29250e=20 libdispatch.dylib`_dispatch_client_callout + 8 =C2=A0=C2=A0=C2=A0 frame #3: 0x00007fff6c29e567=20 libdispatch.dylib`_dispatch_lane_barrier_sync_invoke_and_complete + 60 =C2=A0=C2=A0=C2=A0 frame #4: 0x00007fff41475935=20 Security`Security::KeychainCore::StorageManager::tickleKeychain(Security::K= eychainCore::KeychainImpl*)=20 + 485 =C2=A0=C2=A0=C2=A0 frame #5: 0x00007fff412400d8=20 Security`Security::KeychainCore::KCCursorImpl::next(Security::KeychainCore:= :Item&)=20 + 352 =C2=A0=C2=A0=C2=A0 frame #6: 0x00007fff41417975=20 Security`Security::KeychainCore::IdentityCursor::next(Security::SecPointer<= Security::KeychainCore::Identity>&)=20 + 217 =C2=A0=C2=A0=C2=A0 frame #7: 0x00007fff4143c4c3 Security`SecIdentitySearch= CopyNext + 155 =C2=A0=C2=A0=C2=A0 frame #8: 0x00007fff414477d8=20 Security`SecItemCopyMatching_osx(__CFDictionary const*, void const**) + 261 =C2=A0=C2=A0=C2=A0 frame #9: 0x00007fff4144b024 Security`SecItemCopyMatchi= ng + 338 =C2=A0=C2=A0=C2=A0 frame #10: 0x00007fff56dab303 Heimdal`keychain_query + = 531 =C2=A0=C2=A0=C2=A0 frame #11: 0x00007fff56da8f4c Heimdal`hx509_certs_find = + 92 =C2=A0=C2=A0=C2=A0 frame #12: 0x00007fff56d67b52 Heimdal`_krb5_pk_find_cer= t + 466 =C2=A0=C2=A0=C2=A0 frame #13: 0x00007fff376da9bb GSS`_gsspku2u_acquire_cre= d + 619 =C2=A0=C2=A0=C2=A0 frame #14: 0x00007fff376bfc1c GSS`gss_acquire_cred + 94= 0 =C2=A0=C2=A0=C2=A0 frame #15: 0x000000010016e6e1=20 libpq.5.dylib`pg_GSS_have_cred_cache(cred_out=3D0x0000000100505688) at=20 fe-gssapi-common.c:67:10 =C2=A0=C2=A0=C2=A0 frame #16: 0x000000010014f769=20 libpq.5.dylib`PQconnectPoll(conn=3D0x0000000100505310) at fe-connect.c:2785= :22 =C2=A0=C2=A0=C2=A0 frame #17: 0x000000010014be9f=20 libpq.5.dylib`connectDBComplete(conn=3D0x0000000100505310) at=20 fe-connect.c:2095:10 =C2=A0=C2=A0=C2=A0 frame #18: 0x000000010014bb0c=20 libpq.5.dylib`PQconnectdbParams(keywords=3D0x00007ffeefbfeee0,=20 values=3D0x00007ffeefbfeea0, expand_dbname=3D1) at fe-connect.c:625:10 =C2=A0=C2=A0=C2=A0 frame #19: 0x000000010000ec20=20 pg_restore`ConnectDatabase(AHX=3D0x0000000100505070, dbname=3D"postgres",= =20 pghost=3D"david.highgo.ca", pgport=3D0x0000000000000000, username=3D"david"= ,=20 prompt_password=3DTRI_DEFAULT) at pg_backup_db.c:287:20 =C2=A0=C2=A0=C2=A0 frame #20: 0x000000010000a75a=20 pg_restore`CloneArchive(AH=3D0x00000001002020f0) at=20 pg_backup_archiver.c:4850:3 =C2=A0=C2=A0=C2=A0 frame #21: 0x0000000100017b4b=20 pg_restore`RunWorker(AH=3D0x00000001002020f0, slot=3D0x0000000100221718) at= =20 parallel.c:866:7 =C2=A0=C2=A0=C2=A0 frame #22: 0x00000001000179f5=20 pg_restore`ParallelBackupStart(AH=3D0x00000001002020f0) at parallel.c:1028:= 4 =C2=A0=C2=A0=C2=A0 frame #23: 0x0000000100004473=20 pg_restore`RestoreArchive(AHX=3D0x00000001002020f0) at=20 pg_backup_archiver.c:662:12 =C2=A0=C2=A0=C2=A0 frame #24: 0x0000000100001be4 pg_restore`main(argc=3D10= ,=20 argv=3D0x00007ffeefbff8f0) at pg_restore.c:447:3 =C2=A0=C2=A0=C2=A0 frame #25: 0x00007fff6c2eb7fd libdyld.dylib`start + 1 (lldb) 8. however it works with either, =C2=A0=C2=A0=C2=A0 PGGSSENCMODE=3Ddisable pg_restore -d postgres -h localh= ost -Fc=20 /tmp/128m --jobs=3D2 or, =C2=A0=C2=A0=C2=A0 pg_restore -d "dbname=3Dpostgres gssencmode=3Ddisable" = -h localhost -Fc=20 /tmp/128m --jobs=3D2 9. pg_config output and versions, no SSL configured, $ pg_config BINDIR =3D /Users/david/sandbox/pg122/app/bin DOCDIR =3D /Users/david/sandbox/pg122/app/share/doc/postgresql HTMLDIR =3D /Users/david/sandbox/pg122/app/share/doc/postgresql INCLUDEDIR =3D /Users/david/sandbox/pg122/app/include PKGINCLUDEDIR =3D /Users/david/sandbox/pg122/app/include/postgresql INCLUDEDIR-SERVER =3D /Users/david/sandbox/pg122/app/include/postgresql/ser= ver LIBDIR =3D /Users/david/sandbox/pg122/app/lib PKGLIBDIR =3D /Users/david/sandbox/pg122/app/lib/postgresql LOCALEDIR =3D /Users/david/sandbox/pg122/app/share/locale MANDIR =3D /Users/david/sandbox/pg122/app/share/man SHAREDIR =3D /Users/david/sandbox/pg122/app/share/postgresql SYSCONFDIR =3D /Users/david/sandbox/pg122/app/etc/postgresql PGXS =3D=20 /Users/david/sandbox/pg122/app/lib/postgresql/pgxs/src/makefiles/pgxs.mk CONFIGURE =3D '--with-gssapi' '--prefix=3D/Users/david/sandbox/pg122/app'= =20 '--enable-debug' 'CFLAGS=3D-ggdb -O0 -fno-omit-frame-pointer' CC =3D gcc CPPFLAGS =3D -isysroot=20 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Develo= per/SDKs/MacOSX10.15.sdk CFLAGS =3D -Wall -Wmissing-prototypes -Wpointer-arith=20 -Wdeclaration-after-statement -Werror=3Dvla -Wendif-labels=20 -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing=20 -fwrapv -Wno-unused-command-line-argument -g -ggdb -O0=20 -fno-omit-frame-pointer CFLAGS_SL =3D LDFLAGS =3D -Wl,-dead_strip_dylibs LDFLAGS_EX =3D LDFLAGS_SL =3D LIBS =3D -lpgcommon -lpgport -lgssapi_krb5 -lz -lreadline -lm VERSION =3D PostgreSQL 12.2 $ lldb --version lldb-1100.0.30.12 Apple Swift version 5.1.3 (swiftlang-1100.0.282.1 clang-1100.0.33.15) $ klist --version klist (Heimdal 1.5.1apple1) Copyright 1995-2011 Kungliga Tekniska H=C3=B6gskolan Send bug-reports to heimdal-bugs@h5l.org Hopefully the above information can help. On 2019-12-04 6:03 a.m., Tom Lane wrote: > PG Bug reporting form writes: >> The following bug has been logged on the website: >> Bug reference: 16147 >> Logged by: Bill Tihen >> Email address: btihen@gmail.com >> PostgreSQL version: 12.1 >> Operating system: MacOS 10.15.1 >> Description: >> The following command crashes with any database I've tried (both large a= nd >> small) DBs: >> `pg_restore -U wti0405 -d stage3 -h localhost --jobs=3D8 -Fc >> database_12_04-01-00.bak -x` > I failed to reproduce this on my own 10.15.1 laptop, using manual > builds of either HEAD or the v12 branch. Plausible reasons for > the difference in results might include: > > * There's something different about the homebrew build (could we > see the output of pg_config?) > > * There's something unusual about your configuration (one thought > that comes to mind: do you have SSL turned on for localhost > connections?) > > * There's something about the data in this specific database > (your report that it happens for multiple databases puts a crimp > in this idea, though maybe they all share a common feature) > > Anyway, we need more info to investigate. You might try looking > into the server log to see what the failure looks like from that > side --- is there a query error, or just the worker disconnecting > unexpectedly? > > =09=09=09regards, tom lane > > > > --=20 David Software Engineer Highgo Software Inc. (Canada) www.highgo.ca