pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedData Corruption Errors
5+ messages / 4 participants
[nested] [flat]
* Data Corruption Errors
@ 2018-09-27 12:39 Ferrell, Denise SDC <denise.ferrell@sdc-world.com>
0 siblings, 3 replies; 5+ messages in thread
From: Ferrell, Denise SDC @ 2018-09-27 12:39 UTC (permalink / raw)
To: pgsql-admin
Using Linux/PostgreSQL 9.5 & setup on a VM.
I'm receiving the following errors during an insert using a function but this does not happen every time. When this does occur the database/system is unavailable until the processes are killed or the service is restarted.
"LOG: server process (PID xxxxx) was terminated by signal 9: Killed
DETAIL: Failed process was running
select insert_data ($1, $2,....)
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect ....
LOG: connection received
FATAL: the database system is in recovery mode"
The last 2 lines are repeated and logging is not available.
How can I determine data/table/shared memory corruption?
Thank you in advance for your responses.
D. Ferrell
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Data Corruption Errors
@ 2018-09-27 13:13 Achilleas Mantzios <achill@matrix.gatewaynet.com>
parent: Ferrell, Denise SDC <denise.ferrell@sdc-world.com>
2 siblings, 0 replies; 5+ messages in thread
From: Achilleas Mantzios @ 2018-09-27 13:13 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
On 27/9/18 3:39 μ.μ., Ferrell, Denise SDC wrote:
> Using Linux/PostgreSQL 9.5 & setup on a VM.
>
> I'm receiving the following errors during an insert using a function but this does not happen every time. When this does occur the database/system is unavailable until the processes are killed or the service is restarted.
>
> "LOG: server process (PID xxxxx) was terminated by signal 9: Killed
> DETAIL: Failed process was running
> select insert_data ($1, $2,....)
> WARNING: terminating connection because of crash of another server process
> DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
> HINT: In a moment you should be able to reconnect ....
> LOG: connection received
> FATAL: the database system is in recovery mode"
Did your DB eventually recover? Have you ruled reasons like OOM, non-optimal vm.swapiness , etc ?
>
> The last 2 lines are repeated and logging is not available.
>
> How can I determine data/table/shared memory corruption?
>
> Thank you in advance for your responses.
>
> D. Ferrell
--
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Data Corruption Errors
@ 2018-09-27 13:24 Kevin Grittner <kgrittn@gmail.com>
parent: Ferrell, Denise SDC <denise.ferrell@sdc-world.com>
2 siblings, 1 reply; 5+ messages in thread
From: Kevin Grittner @ 2018-09-27 13:24 UTC (permalink / raw)
To: denise.ferrell@sdc-world.com; +Cc: pgsql-admin
On Thu, Sep 27, 2018 at 7:40 AM Ferrell, Denise SDC
<denise.ferrell@sdc-world.com> wrote:
> Using Linux/PostgreSQL 9.5 & setup on a VM.
Please always show the entire version number -- it can matter a lot.
One way to see it is to run `select version();` on a database
connection. From the shell you can run one of the executables with
the --version option, which will show the version and do nothing else.
If you are running in a VM, please say which product, and show the
exact version of that.
> I'm receiving the following errors during an insert using a function but this does not happen every time. When this does occur the database/system is unavailable until the processes are killed or the service is restarted.
>
> "LOG: server process (PID xxxxx) was terminated by signal 9: Killed
> DETAIL: Failed process was running
> select insert_data ($1, $2,....)
> WARNING: terminating connection because of crash of another server process
> DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
> HINT: In a moment you should be able to reconnect ....
> LOG: connection received
> FATAL: the database system is in recovery mode"
>
> The last 2 lines are repeated and logging is not available.
>
> How can I determine data/table/shared memory corruption?
You have not shown any evidence of corruption. What happened was that
some database service process was killed with SIGKILL (kill -9).
Never do that. It may have been done by the Out of Memory Killer
process. You should disable that by setting vm.overcommit_memory=2
and vm.overcommit_ratio=80 *or maybe as high as 90).
When a database service process is killed with SIGKILL the service
cannot continue without *RISK* of corruption, so it does a crash and
restart. These messages:
FATAL: the database system is in recovery mode
Just mean that an attempt was made to the database while the recovery
was in progress, so the connection attempt was rejected. Just ignore
those messages. Just wait for the service to restart.
--
Kevin Grittner
VMware vCenter Server
https://www.vmware.com/
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Data Corruption Errors
@ 2018-09-27 13:27 Samed YILDIRIM <samed@reddoc.net>
parent: Ferrell, Denise SDC <denise.ferrell@sdc-world.com>
2 siblings, 0 replies; 5+ messages in thread
From: Samed YILDIRIM @ 2018-09-27 13:27 UTC (permalink / raw)
To: Ferrell, Denise SDC <denise.ferrell@sdc-world.com>; pgsql-admin
<div xmlns="http://www.w3.org/1999/xhtml">Dear Ferrell,</div><div xmlns="http://www.w3.org/1999/xhtml"> </div><div xmlns="http://www.w3.org/1999/xhtml">Could you please check the system logs? It looks like it is related with OOM. Or, do you use any extension in the insert function? Maybe a memory leak exists caused by an extension.</div><div xmlns="http://www.w3.org/1999/xhtml"> </div><div xmlns="http://www.w3.org/1999/xhtml">Best regards.</div><div xmlns="http://www.w3.org/1999/xhtml">Samed YILDIRIM</div><div xmlns="http://www.w3.org/1999/xhtml"> </div><div xmlns="http://www.w3.org/1999/xhtml">27.09.2018, 15:40, "Ferrell, Denise SDC" <denise.ferrell@sdc-world.com>:</div><blockquote xmlns="http://www.w3.org/1999/xhtml"; type="cite"><p>Using Linux/PostgreSQL 9.5 & setup on a VM.<br /><br />I'm receiving the following errors during an insert using a function but this does not happen every time. When this does occur the database/system is unavailable until the processes are killed or the service is restarted.<br /><br />"LOG: server process (PID xxxxx) was terminated by signal 9: Killed<br />DETAIL: Failed process was running<br /> select insert_data ($1, $2,....)<br />WARNING: terminating connection because of crash of another server process<br />DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.<br />HINT: In a moment you should be able to reconnect ....<br />LOG: connection received<br />FATAL: the database system is in recovery mode"<br /><br />The last 2 lines are repeated and logging is not available.<br /><br />How can I determine data/table/shared memory corruption?<br /><br />Thank you in advance for your responses.<br /><br />D. Ferrell</p></blockquote>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Data Corruption Errors
@ 2018-09-27 14:01 Ferrell, Denise SDC <denise.ferrell@sdc-world.com>
parent: Kevin Grittner <kgrittn@gmail.com>
0 siblings, 0 replies; 5+ messages in thread
From: Ferrell, Denise SDC @ 2018-09-27 14:01 UTC (permalink / raw)
To: Kevin Grittner <kgrittn@gmail.com>; +Cc: pgsql-admin
This seems to have been the issue. Thank you for the responses.
________________________________________
From: Kevin Grittner <kgrittn@gmail.com>
Sent: Thursday, September 27, 2018 9:24 AM
To: Ferrell, Denise SDC
Cc: pgsql-admin@postgresql.org
Subject: Re: Data Corruption Errors
On Thu, Sep 27, 2018 at 7:40 AM Ferrell, Denise SDC
<denise.ferrell@sdc-world.com> wrote:
> Using Linux/PostgreSQL 9.5 & setup on a VM.
Please always show the entire version number -- it can matter a lot.
One way to see it is to run `select version();` on a database
connection. From the shell you can run one of the executables with
the --version option, which will show the version and do nothing else.
If you are running in a VM, please say which product, and show the
exact version of that.
> I'm receiving the following errors during an insert using a function but this does not happen every time. When this does occur the database/system is unavailable until the processes are killed or the service is restarted.
>
> "LOG: server process (PID xxxxx) was terminated by signal 9: Killed
> DETAIL: Failed process was running
> select insert_data ($1, $2,....)
> WARNING: terminating connection because of crash of another server process
> DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
> HINT: In a moment you should be able to reconnect ....
> LOG: connection received
> FATAL: the database system is in recovery mode"
>
> The last 2 lines are repeated and logging is not available.
>
> How can I determine data/table/shared memory corruption?
You have not shown any evidence of corruption. What happened was that
some database service process was killed with SIGKILL (kill -9).
Never do that. It may have been done by the Out of Memory Killer
process. You should disable that by setting vm.overcommit_memory=2
and vm.overcommit_ratio=80 *or maybe as high as 90).
When a database service process is killed with SIGKILL the service
cannot continue without *RISK* of corruption, so it does a crash and
restart. These messages:
FATAL: the database system is in recovery mode
Just mean that an attempt was made to the database while the recovery
was in progress, so the connection attempt was rejected. Just ignore
those messages. Just wait for the service to restart.
--
Kevin Grittner
VMware vCenter Server
https://www.vmware.com/
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2018-09-27 14:01 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-09-27 12:39 Data Corruption Errors Ferrell, Denise SDC <denise.ferrell@sdc-world.com>
2018-09-27 13:13 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2018-09-27 13:24 ` Kevin Grittner <kgrittn@gmail.com>
2018-09-27 14:01 ` Ferrell, Denise SDC <denise.ferrell@sdc-world.com>
2018-09-27 13:27 ` Samed YILDIRIM <samed@reddoc.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox