Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1g5W7N-0000qv-RK for pgsql-admin@arkaria.postgresql.org; Thu, 27 Sep 2018 13:14:09 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1g5W7L-0005Ru-CE for pgsql-admin@arkaria.postgresql.org; Thu, 27 Sep 2018 13:14:07 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1g5W7L-0005Rn-4n for pgsql-admin@lists.postgresql.org; Thu, 27 Sep 2018 13:14:07 +0000 Received: from hermes.gatewaynet.com ([193.92.121.98]) by magus.postgresql.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1g5W7I-0006J1-Jf for pgsql-admin@lists.postgresql.org; Thu, 27 Sep 2018 13:14:06 +0000 Received: from smadev.internal.net (pc131.internal.net [10.9.200.131]) by hermes.gatewaynet.com (8.14.3/8.14.3/Debian-9.4) with ESMTP id w8RDDusk022550 for ; Thu, 27 Sep 2018 16:14:01 +0300 Received: from smadev.internal.net (smadev [10.9.200.131]) by smadev.internal.net (8.15.2/8.15.2) with ESMTP id w8RDDtc1042223 for ; Thu, 27 Sep 2018 16:13:56 +0300 (EEST) (envelope-from achill@matrix.gatewaynet.com) Subject: Re: Data Corruption Errors To: pgsql-admin@lists.postgresql.org References: <64b2d16e-6c7c-002a-1e16-209a3ae43bef@2ndquadrant.com> <1538051973729.67553@sdc-world.com> From: Achilleas Mantzios Message-ID: <78ea6e8f-a55b-5724-c86e-723a4e7b9e19@matrix.gatewaynet.com> Date: Thu, 27 Sep 2018 16:13:55 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0.1 MIME-Version: 1.0 In-Reply-To: <1538051973729.67553@sdc-world.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Scanned: clamav-milter 0.98.7 at hermes X-Virus-Status: Clean X-Spam-Status: No, score=-29.0 required=5.0 tests=ALL_TRUSTED,DYN_HAM_COORD, DYN_HAM_IT,DYN_HAM_NAUTICAL autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on hermes.gatewaynet.com List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On 27/9/18 3:39 μ.μ., Ferrell, Denise SDC wrote: > Using Linux/PostgreSQL 9.5 & setup on a VM. > > I'm receiving the following errors during an insert using a function but this does not happen every time. When this does occur the database/system is unavailable until the processes are killed or the service is restarted. > > "LOG: server process (PID xxxxx) was terminated by signal 9: Killed > DETAIL: Failed process was running > select insert_data ($1, $2,....) > WARNING: terminating connection because of crash of another server process > DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. > HINT: In a moment you should be able to reconnect .... > LOG: connection received > FATAL: the database system is in recovery mode" Did your DB eventually recover? Have you ruled reasons like OOM, non-optimal vm.swapiness , etc ? > > The last 2 lines are repeated and logging is not available. > > How can I determine data/table/shared memory corruption? > > Thank you in advance for your responses. > > D. Ferrell -- Achilleas Mantzios IT DEV Lead IT DEPT Dynacom Tankers Mgmt