Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iT2Hj-00048x-AK for pgsql-bugs@arkaria.postgresql.org; Fri, 08 Nov 2019 11:18:35 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iT2Hi-0005O1-2V for pgsql-bugs@arkaria.postgresql.org; Fri, 08 Nov 2019 11:18:34 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iT2Hh-0005NW-JB for pgsql-bugs@lists.postgresql.org; Fri, 08 Nov 2019 11:18:33 +0000 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1iT2He-00060W-JQ for pgsql-bugs@postgresql.org; Fri, 08 Nov 2019 11:18:32 +0000 Received: by mail-wm1-x342.google.com with SMTP id c17so5774093wmk.2 for ; Fri, 08 Nov 2019 03:18:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=qV315g3TewjB/+xkqC8wQ8jJihVx6M04eQ6j8bNfs5s=; b=te12C7pz054VYIXiOXLyVRXwPt/SNFiO7vOiIym7FV8oVlxFG/OX6O6VjONW6J6UcW Lz4z1YOoNQltipuBwaNMK8uTo36k4WzQcrUpFD6Ls/eDh6zXBSAK8mMZCipY0sD03zhE Cpob1tUsCecpSVxN5funpkCVFiRVhbuTPu5wBI9KGLoXdJGTZonfLd+OevZjogGt/gf5 +wGF1QaPB7mE77t0jd852WtRiVRNflVdBPtkKI3vNTBHBmsxW5VCuUtu5erRa/w3VM9f 4OTSqC6bXbcy8E9A1jCtp6DEDD2eFuFPqW3aAqXMNLBNKYnzSmr9to6ui2+E+8M2Ur1B koEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=qV315g3TewjB/+xkqC8wQ8jJihVx6M04eQ6j8bNfs5s=; b=JuB13DwFbvF+I3DGeuwhTGulMAXqMqw3eGBYL/ffSOvuMtaoEr9Vo9BuQ8vkcz2HUX BSBCdAFTh3NVXE0C+fPgoxSal1nwEbfcKxsL4QkRp2l1n78PL/mGeLSYXE76B9G1+72e wKxwpbZ2Gxl9Amd6XDPz07/JFT8aWSj1/F9dxzYunwTMkzNB9HlKqoyeSZn9QVOKwZ+P TNi8f8Eo56RYCINMhDNhOfgNIasUMHffWDFwWNVcZasHZZBVJHzZVwXgqZSTcGLh8J2X 6jbNPD2kWl6154So//8LacgG/UUQ6Fn5UqZ0P3JQhcyf9MMVkPCojEeWv/+FG353AErn nE8Q== X-Gm-Message-State: APjAAAVQYNvvnmZttuwMpKRciPJu1Vc94I6yX3LDEJQA0wmqPiCUsin9 1okjr8u4YpsPoGHmf1Z5GZytJA== X-Google-Smtp-Source: APXvYqwT/Hhj8pVw/N6P5n3R18zpemJwm+2f8lGxBFeHFlo5a5G+0YS6khdChw2bP0HjgwI7UOSThQ== X-Received: by 2002:a7b:c10c:: with SMTP id w12mr8068295wmi.114.1573211909147; Fri, 08 Nov 2019 03:18:29 -0800 (PST) Received: from localhost ([217.8.174.100]) by smtp.gmail.com with ESMTPSA id w12sm5326141wmi.17.2019.11.08.03.18.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Nov 2019 03:18:28 -0800 (PST) Date: Fri, 8 Nov 2019 12:18:27 +0100 From: Tomas Vondra To: nemo Cc: pgsql-bugs@postgresql.org Subject: Re: pg_dump: error: schema with OID 7956828 does not exist Message-ID: <20191108111827.eu6uw7vm3syv4y55@development> References: <1573196364683-0.post@n3.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <1573196364683-0.post@n3.nabble.com> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Thu, Nov 07, 2019 at 11:59:24PM -0700, nemo wrote: >Hello, I have a problem on PG12 >os:CentOS Linux release 7.4.1708 (Core) >db:PostgreSQL 12.0 > >the db log is >2019-11-08 11:59:40.385 CST,,,19005,,5dc4e7d2.4a3d,1,,2019-11-08 11:58:10 >CST,1/0,0,WARNING,57P02,"terminating connection because of crash of another >server process","The postmaster has commanded this server process to roll >back the current transaction and exit, because another server process exited >abnormally and possibly corrupted shared memory.","In a moment you should be >able to reconnect to the database and repeat your command.",,,,,,"quickdie, >postgres.c:2717"," > >os message ,i find is not oom >Nov 6 10:00:02 xx-xx crond: /usr/sbin/sendmail: Cannot allocate memory >Nov 8 16:23:59 xx-xx kernel: Reserving 161MB of memory at 688MB for >crashkernel (System RAM: 8191MB) >Nov 8 16:23:59 xx-xx kernel: Early memory node ranges >Nov 8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem >0x0009f000-0x0009ffff] >Nov 8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem >0x000a0000-0x000effff] >Nov 8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem >0x000f0000-0x000fffff] >Nov 8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem >0xbffe0000-0xbfffffff] >Nov 8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem >0xc0000000-0xfeffbfff] >Nov 8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem >0xfeffc000-0xfeffffff] >Nov 8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem >0xff000000-0xfffbffff] >Nov 8 16:23:59 xx-xx kernel: PM: Registered nosave memory: [mem >0xfffc0000-0xffffffff] >Nov 8 16:23:59 xx-xx kernel: please try 'cgroup_disable=memory' option if >you don't want memory cgroups >Nov 8 16:23:59 xx-xx kernel: Initializing cgroup subsys memory >Nov 8 16:23:59 xx-xx kernel: Freeing initrd memory: 17248k freed >Nov 8 16:23:59 xx-xx kernel: Non-volatile memory driver v1.3 >Nov 8 16:23:59 xx-xx kernel: crash memory driver: version 1.1 >Nov 8 16:23:59 xx-xx kernel: Freeing unused kernel memory: 1764k freed >Nov 8 16:24:00 xx-xx kernel: [TTM] Zone kernel: Available graphics memory: >4005228 kiB >Nov 8 16:24:00 xx-xx kernel: [TTM] Zone dma32: Available graphics memory: >2097152 kiB >Nov 8 17:36:48 xx-xx kernel: Reserving 161MB of memory at 688MB for >crashkernel (System RAM: 8191MB) >Nov 8 17:36:48 xx-xx kernel: Early memory node ranges >Nov 8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem >0x0009f000-0x0009ffff] >Nov 8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem >0x000a0000-0x000effff] >Nov 8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem >0x000f0000-0x000fffff] >Nov 8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem >0xbffe0000-0xbfffffff] >Nov 8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem >0xc0000000-0xfeffbfff] >Nov 8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem >0xfeffc000-0xfeffffff] >Nov 8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem >0xff000000-0xfffbffff] >Nov 8 17:36:48 xx-xx kernel: PM: Registered nosave memory: [mem >0xfffc0000-0xffffffff] >Nov 8 17:36:48 xx-xx kernel: please try 'cgroup_disable=memory' option if >you don't want memory cgroups >Nov 8 17:36:48 xx-xx kernel: Initializing cgroup subsys memory >Nov 8 17:36:48 xx-xx kernel: Freeing initrd memory: 17824k freed >Nov 8 17:36:48 xx-xx kernel: Non-volatile memory driver v1.3 >Nov 8 17:36:48 xx-xx kernel: crash memory driver: version 1.1 >Nov 8 17:36:48 xx-xx kernel: Freeing unused kernel memory: 1984k freed >Nov 8 17:36:48 xx-xx kernel: Freeing unused kernel memory: 424k freed >Nov 8 17:36:48 xx-xx kernel: Freeing unused kernel memory: 556k freed >Nov 8 17:36:49 xx-xx kernel: [TTM] Zone kernel: Available graphics memory: >4004518 kiB >Nov 8 17:36:49 xx-xx kernel: [TTM] Zone dma32: Available graphics memory: >2097152 kiB >Nov 8 17:38:12 xx-xx kernel: Reserving 161MB of memory at 688MB for >crashkernel (System RAM: 8191MB) >Nov 8 17:38:12 xx-xx kernel: Early memory node ranges >Nov 8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem >0x0009f000-0x0009ffff] >Nov 8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem >0x000a0000-0x000effff] >Nov 8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem >0x000f0000-0x000fffff] >Nov 8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xbffe0000-0xbfffffff] >Nov 8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xc0000000-0xfeffbfff] >Nov 8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xfeffc000-0xfeffffff] >Nov 8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xff000000-0xfffbffff] >Nov 8 17:38:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xfffc0000-0xffffffff] >Nov 8 17:38:12 xx-xx kernel: please try 'cgroup_disable=memory' option if >you don't want memory cgroups >Nov 8 17:38:12 xx-xx kernel: Initializing cgroup subsys memory >Nov 8 17:38:12 xx-xx kernel: Freeing initrd memory: 17824k freed >Nov 8 17:38:12 xx-xx kernel: Non-volatile memory driver v1.3 >Nov 8 17:38:12 xx-xx kernel: crash memory driver: version 1.1 >Nov 8 17:38:12 xx-xx kernel: Freeing unused kernel memory: 1984k freed >Nov 8 17:38:12 xx-xx kernel: Freeing unused kernel memory: 424k freed >Nov 8 17:38:12 xx-xx kernel: Freeing unused kernel memory: 556k freed >Nov 8 17:38:13 xx-xx kernel: [TTM] Zone kernel: Available graphics memory: >4004514 kiB >Nov 8 17:38:13 xx-xx kernel: [TTM] Zone dma32: Available graphics memory: >2097152 kiB >Nov 8 18:35:12 xx-xx kernel: Reserving 161MB of memory at 688MB for >crashkernel (System RAM: 8191MB) >Nov 8 18:35:12 xx-xx kernel: Early memory node ranges >Nov 8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem >0x0009f000-0x0009ffff] >Nov 8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem >0x000a0000-0x000effff] >Nov 8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem >0x000f0000-0x000fffff] >Nov 8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xbffe0000-0xbfffffff] >Nov 8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xc0000000-0xfeffbfff] >Nov 8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xfeffc000-0xfeffffff] >Nov 8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xff000000-0xfffbffff] >Nov 8 18:35:12 xx-xx kernel: PM: Registered nosave memory: [mem >0xfffc0000-0xffffffff] >Nov 8 18:35:12 xx-xx kernel: please try 'cgroup_disable=memory' option if >you don't want memory cgroups >Nov 8 18:35:12 xx-xx kernel: Initializing cgroup subsys memory >Nov 8 18:35:12 xx-xx kernel: Freeing initrd memory: 17824k freed >Nov 8 18:35:12 xx-xx kernel: Non-volatile memory driver v1.3 >Nov 8 18:35:12 xx-xx kernel: crash memory driver: version 1.1 >Nov 8 18:35:12 xx-xx kernel: Freeing unused kernel memory: 1984k freed >Nov 8 18:35:12 xx-xx kernel: Freeing unused kernel memory: 424k freed >Nov 8 18:35:12 xx-xx kernel: Freeing unused kernel memory: 556k freed >Nov 8 18:35:13 xx-xx kernel: [TTM] Zone kernel: Available graphics memory: >4004514 kiB >Nov 8 18:35:13 xx-xx kernel: [TTM] Zone dma32: Available graphics memory: >2097152 kiB >Nov 8 18:47:14 xx-xx kernel: Reserving 161MB of memory at 688MB for >crashkernel (System RAM: 8191MB) >Nov 8 18:47:14 xx-xx kernel: Early memory node ranges >Nov 8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem >0x0009f000-0x0009ffff] >Nov 8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem >0x000a0000-0x000effff] >Nov 8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem >0x000f0000-0x000fffff] >Nov 8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem >0xbffe0000-0xbfffffff] >Nov 8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem >0xc0000000-0xfeffbfff] >Nov 8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem >0xfeffc000-0xfeffffff] >Nov 8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem >0xff000000-0xfffbffff] >Nov 8 18:47:14 xx-xx kernel: PM: Registered nosave memory: [mem >0xfffc0000-0xffffffff] >Nov 8 18:47:14 xx-xx kernel: please try 'cgroup_disable=memory' option if >you don't want memory cgroups >Nov 8 18:47:14 xx-xx kernel: Initializing cgroup subsys memory >Nov 8 18:47:14 xx-xx kernel: Freeing initrd memory: 17824k freed >Nov 8 18:47:14 xx-xx kernel: Non-volatile memory driver v1.3 >Nov 8 18:47:14 xx-xx kernel: crash memory driver: version 1.1 >Nov 8 18:47:14 xx-xx kernel: Freeing unused kernel memory: 1984k freed >Nov 8 18:47:14 xx-xx kernel: Freeing unused kernel memory: 424k freed >Nov 8 18:47:14 xx-xx kernel: Freeing unused kernel memory: 556k freed >Nov 8 18:47:15 xx-xx kernel: [TTM] Zone kernel: Available graphics memory: >4004518 kiB >Nov 8 18:47:15 xx-xx kernel: [TTM] Zone dma32: Available graphics memory: >2097152 kiB > >os error message: >Nov 8 14:30:56 xx-xx kernel: postgres[26427]: segfault at 0 ip >00007f0e0a9e8641 sp 00007fff971a9598 error 4 in >libc-2.17.so[7f0e0a879000+1c3000] >Nov 8 14:31:41 xx-xx kernel: postgres[26453]: segfault at 0 ip >00007f0e0a9e8641 sp 00007fff971a9598 error 4 in >libc-2.17.so[7f0e0a879000+1c3000] >Nov 8 14:32:27 xx-xx kernel: postgres[26479]: segfault at 0 ip >00007f0e0a9e8641 sp 00007fff971a9598 error 4 in >libc-2.17.so[7f0e0a879000+1c3000] >Nov 8 14:33:12 xx-xx kernel: postgres[26523]: segfault at 0 ip >00007f0e0a9e8641 sp 00007fff971a9598 error 4 in >libc-2.17.so[7f0e0a879000+1c3000] >Nov 8 14:33:58 xx-xx kernel: postgres[26556]: segfault at 0 ip >00007f0e0a9e8641 sp 00007fff971a9598 error 4 in >libc-2.17.so[7f0e0a879000+1c3000] >Nov 8 14:40:43 xx-xx kernel: postgres[26571]: segfault at 0 ip >00007f0e0a9e8641 sp 00007fff971a9598 error 4 in >libc-2.17.so[7f0e0a879000+1c3000] > >then i run the pg_dump >$ pg_dump -Fc -d testdb -f test-20191108.dump >pg_dump: error: schema with OID 7956828 does not exist > ># select * from pg_namespace where oid = 7956828 ; > oid | nspname | nspowner | nspacl >-----+---------+----------+-------- >(0 rows) > >please help me!!! > Clearly the postgres process is crashing because of segfault. Please see [1] (there's a section about core dumps) and give us backtrace and other details. Other than that, it'd be interesting to know where does the schema OID come from, so maybe investigate a bit in the server log and catalogs and try to figure out which object is referring to the OID. [1] https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services