pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedserver crash
17+ messages / 9 participants
[nested] [flat]
* server crash
@ 2004-11-09 16:01 Medora Schauer <mschauer@fairfield.com>
2004-11-09 16:53 ` Re: server crash Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 17+ messages in thread
From: Medora Schauer @ 2004-11-09 16:01 UTC (permalink / raw)
To: pgsql-admin
OS: linux 2.4.18
h/w: moto 5100 SBC
PG: 7.3
We have a recurring problem with the pg server crashing. It doesn't
happen often (once every couple of months or so) but we require 24/7 up
time. Attached is the pg.log from the last time it crashed. The stuff
towards the end seems to indicate some kind of serious problem but
beyond that it means little to me. I'm hoping it will be more
meaningful to someone on this list. Suggestions about how to obtain
more information would also be welcome.
Is there any way to get timestamps in the log file? It would be helpful
to know which of the messages are related in a time sense at least.
Is there a description of the various postmaster debug levels? I might
try upping the level but since the crash doesn't happen very often I
would be helpful if I knew which level would yield the most useful
information without clogging the file with unneeded detail.
Any help will be greatly appreciated,
Medora Schauer
Attachments:
pg.log.sav (application/octet-stream, 20.2K) download
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: server crash
2004-11-09 16:01 server crash Medora Schauer <mschauer@fairfield.com>
@ 2004-11-09 16:53 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 17+ messages in thread
From: Tom Lane @ 2004-11-09 16:53 UTC (permalink / raw)
To: Medora Schauer <mschauer@fairfield.com>; +Cc: pgsql-admin
"Medora Schauer" <mschauer@fairfield.com> writes:
> PG: 7.3
I hope you meant "7.3.8", or at least something later than original 7.3 ;-)
> PANIC: link from /data/database/pg_xlog/00000001000000D9 to /data/database/pg_xlog/00000001000000E1 (initialization of log file 1, segment 225) failed: No such file or directory
AFAICS, the only way that that message could come out is if
a readdir() scan of /data/database/pg_xlog found a file named
00000001000000D9 but then an immediately following link() call
didn't find it. This suggests filesystem corruption to me;
it might be worth running fsck. Also check whether your kernel
is reasonably up to date, as it could be a kernel bug.
The fact that the subsequent startup process failed again in just the
same way seems to eliminate most of the theories that would suggest
it's a Postgres bug. In particular, it isn't a race condition such as
two processes trying to delete the same file at the same time, because
the recovery startup sequence does no parallel processing.
regards, tom lane
^ permalink raw reply [nested|flat] 17+ messages in thread
* Server Crash
@ 2008-04-22 14:14 Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 15:13 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 17+ messages in thread
From: Hajek, Nick @ 2008-04-22 14:14 UTC (permalink / raw)
To: pgsql-admin
All,
We experienced a crash of a Postgresql server which from the log appears
to have began with this entry:
Log: background writer process (PID 3457) was terminated by signal 9
After the db was restarted, it operated apparently normally for about 15
minutes and then crashed again with the log recording the same message
at the beginning of the second event. After that crash, I rebooted the
server and it has ran normally since that time - although that's been
less than one hour.
System Details - Postgresql 8.2.4, Suse 10.1 Linux (2.6.16), HP DL380
w/ RAID drives.
Anyone have any thoughts?
thanks,
Nick
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2008-04-22 14:14 Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
@ 2008-04-22 15:13 ` Scott Marlowe <scott.marlowe@gmail.com>
2008-04-22 15:17 ` Re: Server Crash Ray Stell <stellr@cns.vt.edu>
2008-04-22 15:39 ` Re: Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
0 siblings, 2 replies; 17+ messages in thread
From: Scott Marlowe @ 2008-04-22 15:13 UTC (permalink / raw)
To: Hajek, Nick <Nick.Hajek@vishay.com>; +Cc: pgsql-admin
On Tue, Apr 22, 2008 at 8:14 AM, Hajek, Nick <Nick.Hajek@vishay.com> wrote:
>
>
> All,
> We experienced a crash of a Postgresql server which from the log appears to
> have began with this entry:
>
> Log: background writer process (PID 3457) was terminated by signal 9
Kill -9 is the "shoot it in the head" signal. It is not generated by
postgresql in normal operation. It can be generated by "pg_ctl -m
immediate stop" . At least I think that's what signal it sends.
Anyway, the most common cause of kill -9s randomly showing up in linux
is the OOM killer.
It's quite possible you're running your machine out of memory / swap
somehow and linux is killing the biggest, fattest process it can find,
which is pgsql.
you might wanna run vmstat 1 to see what's happening during these times.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2008-04-22 14:14 Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 15:13 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
@ 2008-04-22 15:17 ` Ray Stell <stellr@cns.vt.edu>
1 sibling, 0 replies; 17+ messages in thread
From: Ray Stell @ 2008-04-22 15:17 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: Hajek, Nick <Nick.Hajek@vishay.com>; pgsql-admin
On Tue, Apr 22, 2008 at 09:13:09AM -0600, Scott Marlowe wrote:
> On Tue, Apr 22, 2008 at 8:14 AM, Hajek, Nick <Nick.Hajek@vishay.com> wrote:
> It's quite possible you're running your machine out of memory / swap
> somehow and linux is killing the biggest, fattest process it can find,
> which is pgsql.
syslog would have something to say about that, also.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2008-04-22 14:14 Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 15:13 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
@ 2008-04-22 15:39 ` Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 16:06 ` Re: Server Crash Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 17+ messages in thread
From: Hajek, Nick @ 2008-04-22 15:39 UTC (permalink / raw)
To: pgsql-admin
> -----Original Message-----
> From: Scott Marlowe [mailto:scott.marlowe@gmail.com]
> Sent: Tuesday, April 22, 2008 10:13 AM
> To: Hajek, Nick
> Cc: pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] Server Crash
>
> On Tue, Apr 22, 2008 at 8:14 AM, Hajek, Nick
> <Nick.Hajek@vishay.com> wrote:
> >
> >
> > All,
> > We experienced a crash of a Postgresql server which from the log
> > appears to have began with this entry:
> >
> > Log: background writer process (PID 3457) was terminated
> by signal 9
>
> Kill -9 is the "shoot it in the head" signal. It is not
> generated by postgresql in normal operation. It can be
> generated by "pg_ctl -m immediate stop" . At least I think
> that's what signal it sends.
>
> Anyway, the most common cause of kill -9s randomly showing up
> in linux is the OOM killer.
>
> It's quite possible you're running your machine out of memory
> / swap somehow and linux is killing the biggest, fattest
> process it can find, which is pgsql.
>
> you might wanna run vmstat 1 to see what's happening during
> these times.
>
Bingo. I checked the syslog and found the OOM killer and indications
that the free swap space was zero. Now I just need to find what's
eating memory. Thanks for the help.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2008-04-22 14:14 Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 15:13 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
2008-04-22 15:39 ` Re: Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
@ 2008-04-22 16:06 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-04-22 16:21 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 17+ messages in thread
From: Tom Lane @ 2008-04-22 16:06 UTC (permalink / raw)
To: pgsql-admin
> From: Scott Marlowe [mailto:scott.marlowe@gmail.com]
>> Kill -9 is the "shoot it in the head" signal. It is not
>> generated by postgresql in normal operation. It can be
>> generated by "pg_ctl -m immediate stop" . At least I think
>> that's what signal it sends.
Just for the archives: Postgres never generates kill -9 at all.
(Immediate stop uses SIGQUIT, instead.) When you see that in
the log, you can be sure it was a manual action or the OOM killer.
regards, tom lane
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2008-04-22 14:14 Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 15:13 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
2008-04-22 15:39 ` Re: Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 16:06 ` Re: Server Crash Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-04-22 16:21 ` Scott Marlowe <scott.marlowe@gmail.com>
2008-04-22 16:32 ` Re: Server Crash Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 17+ messages in thread
From: Scott Marlowe @ 2008-04-22 16:21 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin
On Tue, Apr 22, 2008 at 10:06 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > From: Scott Marlowe [mailto:scott.marlowe@gmail.com]
>
> >> Kill -9 is the "shoot it in the head" signal. It is not
> >> generated by postgresql in normal operation. It can be
> >> generated by "pg_ctl -m immediate stop" . At least I think
> >> that's what signal it sends.
>
> Just for the archives: Postgres never generates kill -9 at all.
> (Immediate stop uses SIGQUIT, instead.) When you see that in
> the log, you can be sure it was a manual action or the OOM killer.
Thanks. Just wondering, what's the difference in behavior from
pgsql's perspective from sigquit and siqkill? Is sigkill more
dangerous than sigquit?
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2008-04-22 14:14 Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 15:13 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
2008-04-22 15:39 ` Re: Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 16:06 ` Re: Server Crash Tom Lane <tgl@sss.pgh.pa.us>
2008-04-22 16:21 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
@ 2008-04-22 16:32 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 17+ messages in thread
From: Tom Lane @ 2008-04-22 16:32 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: pgsql-admin
"Scott Marlowe" <scott.marlowe@gmail.com> writes:
> Thanks. Just wondering, what's the difference in behavior from
> pgsql's perspective from sigquit and siqkill? Is sigkill more
> dangerous than sigquit?
Yes it is, because sigkill can't be trapped --- it causes instant
process death with no chance to clean up. Not that we have backends
do a lot of cleanup after sigquit either, but at least the option
exists. The real difference is in the postmaster: kill -9 on the
postmaster is a seriously bad idea, because it gets no chance to shut
down its children.
regards, tom lane
^ permalink raw reply [nested|flat] 17+ messages in thread
* Server Crash
@ 2018-06-26 12:47 Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 08:51 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 09:31 ` Re: Server Crash Sergei Kornilov <sk@zsrv.org>
0 siblings, 2 replies; 17+ messages in thread
From: Anjul Tyagi @ 2018-06-26 12:47 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Hi All,
We have recently deployed couple of new plperl SPROC in our postgres
production server and after that server start throwing error. Currently
we are using postgres 9.1 and planning to upgrade with PG 10.
But we are not sure if that will cause the same error in PG 10 as well.
We really need help.
Error Message:
2018-06-21 13:43:29 EDT [22212]: [4-1] user=cmsuser,db=forte
,host=10.10.1.3,port=10.10.1.3(56601),id=5b2bdea5.56c4,line=4DETAIL:
The postmaster has commanded this server process to roll back the
current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
2018-06-21 13:43:29 EDT [3673]: [5-1] user=cmsuser,db=forte
,host=10.10.1.3,port=10.10.1.3(55649),id=5b2bd568.e59,line=5DETAIL: The
postmaster has commanded this server process to roll back the current
transaction and exit, because another server process exited abnormally
and possibly corrupted shared memory.
Server Details:
Linux - Red Hat Enterprise Linux Server release 6.3 (Santiago)
Postgres - Postgres 9.1.13
below are the sample SPROC code:
CREATE OR REPLACE FUNCTION getauthcode(
claim character varying,
oflag character varying,
pacode character varying,
dispos character varying)
RETURNS text AS
$BODY$
use strict;
use warnings;
use SOAP::Lite;
use JSON;
try
{
my $host = `hostname`;
my $rv = spi_exec_query("select * from
getsprocurl('getauthcode','".$host."')");
my $url = $rv->{rows}[0]->{ret_url};
elog(NOTICE, 'Host Name ' . $host . ' URL '. $url );
my $soap = SOAP::Lite->new();
my $service = $soap->service($url);
my %params = ("claim" => $_[0], "oflag" => $_[1], "pacode" =>
$_[2], "dispos" => $_[3]);
my $response = $service->getAuthCode(%params);
my $json_array = decode_json($response);
for my $report ( $json_array) {
my $status = $report->{status}, '\n';
if ($status eq 'success')
{
return $report->{PriorApprovalCode};
}
else{
return $report->{message};
}
}
}
catch Exception with
{
my $ex = shift;
return 'SOAPFAULT: ' . date('H:i:s') . ' ' . exception($ex);
}
$BODY$
LANGUAGE plperlu VOLATILE STRICT
COST 100;
Appreciate for your help in advance.
Regards,
Anjul TYAGI
ü Go Green
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2018-06-26 12:47 Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
@ 2018-06-27 08:51 ` Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 09:13 ` Re: Server Crash Fabio Pardi <f.pardi@portavita.eu>
1 sibling, 1 reply; 17+ messages in thread
From: Anjul Tyagi @ 2018-06-27 08:51 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; pgsql-admin
Hi All,
can you please suggest on the issue we are facing?
Regards,
Anjul TYAGI
ü Go Green
------ Original Message ------
From: "Anjul Tyagi" <anjul@ibosstech-us.com>
To: "pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org>
Sent: 26-06-2018 18:17:26
Subject: Server Crash
>Hi All,
>
>We have recently deployed couple of new plperl SPROC in our postgres
>production server and after that server start throwing error. Currently
>we are using postgres 9.1 and planning to upgrade with PG 10.
>
>But we are not sure if that will cause the same error in PG 10 as well.
>We really need help.
>
>Error Message:
>
>2018-06-21 13:43:29 EDT [22212]: [4-1] user=cmsuser,db=forte
>,host=10.10.1.3,port=10.10.1.3(56601),id=5b2bdea5.56c4,line=4DETAIL:
>The postmaster has commanded this server process to roll back the
>current transaction and exit, because another server process exited
>abnormally and possibly corrupted shared memory.
>
>
>
>2018-06-21 13:43:29 EDT [3673]: [5-1] user=cmsuser,db=forte
>,host=10.10.1.3,port=10.10.1.3(55649),id=5b2bd568.e59,line=5DETAIL:
>The postmaster has commanded this server process to roll back the
>current transaction and exit, because another server process exited
>abnormally and possibly corrupted shared memory.
>
>
>Server Details:
>Linux - Red Hat Enterprise Linux Server release 6.3 (Santiago)
>Postgres - Postgres 9.1.13
>
>below are the sample SPROC code:
>
>
>CREATE OR REPLACE FUNCTION getauthcode(
> claim character varying,
> oflag character varying,
> pacode character varying,
> dispos character varying)
> RETURNS text AS
>$BODY$
>use strict;
>use warnings;
>use SOAP::Lite;
>use JSON;
>
>
>try
>{
> my $host = `hostname`;
> my $rv = spi_exec_query("select * from
>getsprocurl('getauthcode','".$host."')");
> my $url = $rv->{rows}[0]->{ret_url};
> elog(NOTICE, 'Host Name ' . $host . ' URL '. $url );
>
> my $soap = SOAP::Lite->new();
> my $service = $soap->service($url);
> my %params = ("claim" => $_[0], "oflag" => $_[1], "pacode" =>
>$_[2], "dispos" => $_[3]);
> my $response = $service->getAuthCode(%params);
> my $json_array = decode_json($response);
>
> for my $report ( $json_array) {
> my $status = $report->{status}, '\n';
> if ($status eq 'success')
> {
> return $report->{PriorApprovalCode};
> }
> else{
> return $report->{message};
> }
> }
>}
>catch Exception with
>{
> my $ex = shift;
> return 'SOAPFAULT: ' . date('H:i:s') . ' ' . exception($ex);
>}
>
>$BODY$
> LANGUAGE plperlu VOLATILE STRICT
> COST 100;
>
>Appreciate for your help in advance.
>
>Regards,
>
>Anjul TYAGI
>
>
>
>ü Go Green
>
>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2018-06-26 12:47 Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 08:51 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
@ 2018-06-27 09:13 ` Fabio Pardi <f.pardi@portavita.eu>
2018-06-27 10:40 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
0 siblings, 1 reply; 17+ messages in thread
From: Fabio Pardi @ 2018-06-27 09:13 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hi Anjul,
please avoid cross posting over multiple mailing lists.
Also asking again does not help, and usually is counterproductive: members see a reply and they might put efforts on helping somebody who did not get a reply yet.
Please take good note of it.
About your problem:
I would suggest you to upgrade to a newer version, since Postgres 9.1 is too old and does not get updates any longer.
Besides that, not being aperl expert i cannot help with your procedure. I think anyway the problem might be somewhere else. I suspectthat your server crashes and what you read there is only a consequence, not the cause.
Could you post any relevant logentry in /var/log/messages and maybe anything else appearing in the postgres logfile?
What kind of machine are you running on? can we have more specs?
How is your server configured?
regards,
fabio pardi
On 27/06/18 10:51, Anjul Tyagi wrote:
> Hi All,
>
> can you please suggest on the issue we are facing?
>
>
>
>
> Regards,
>
> Anjul *TYAGI*
>
> * *
>
> *ü** **Go Green*
>
>
> ------ Original Message ------
> From: "Anjul Tyagi" <anjul@ibosstech-us.com <mailto:anjul@ibosstech-us.com>>
> To: "pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org <mailto:pgsql-bugs@lists.postgresql.org>>
> Sent: 26-06-2018 18:17:26
> Subject: Server Crash
>
>> Hi All,
>>
>> We have recently deployed couple of new plperl SPROC in our postgres production server and after that server start throwing error. Currently we are using postgres 9.1 and planning to upgrade with PG 10.
>>
>> But we are not sure if that will cause the same error in PG 10 as well. We really need help.
>>
>> Error Message:
>>
>> 2018-06-21 13:43:29 EDT [22212]: [4-1] user=cmsuser,db=forte ,host=10.10.1.3,port=10.10.1.3(56601),id=5b2bdea5.56c4,line=4DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
>>
>>
>>
>> 2018-06-21 13:43:29 EDT [3673]: [5-1] user=cmsuser,db=forte ,host=10.10.1.3,port=10.10.1.3(55649),id=5b2bd568.e59,line=5DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
>>
>>
>> Server Details:
>> Linux - Red Hat Enterprise Linux Server release 6.3 (Santiago)
>> Postgres - Postgres 9.1.13
>>
>> below are the sample SPROC code:
>>
>>
>> CREATE OR REPLACE FUNCTION getauthcode(
>> claim character varying,
>> oflag character varying,
>> pacode character varying,
>> dispos character varying)
>> RETURNS text AS
>> $BODY$
>> use strict;
>> use warnings;
>> use SOAP::Lite;
>> use JSON;
>>
>>
>> try
>> {
>> my $host = `hostname`;
>> my $rv = spi_exec_query("select * from getsprocurl('getauthcode','".$host."')");
>> my $url = $rv->{rows}[0]->{ret_url};
>> elog(NOTICE, 'Host Name ' . $host . ' URL '. $url );
>>
>> my $soap = SOAP::Lite->new();
>> my $service = $soap->service($url);
>> my %params = ("claim" => $_[0], "oflag" => $_[1], "pacode" => $_[2], "dispos" => $_[3]);
>> my $response = $service->getAuthCode(%params);
>> my $json_array = decode_json($response);
>>
>> for my $report ( $json_array) {
>> my $status = $report->{status}, '\n';
>> if ($status eq 'success')
>> {
>> return $report->{PriorApprovalCode};
>> }
>> else{
>> return $report->{message};
>> }
>> }
>> }
>> catch Exception with
>> {
>> my $ex = shift;
>> return 'SOAPFAULT: ' . date('H:i:s') . ' ' . exception($ex);
>> }
>>
>> $BODY$
>> LANGUAGE plperlu VOLATILE STRICT
>> COST 100;
>>
>> Appreciate for your help in advance.
>>
>>
>> Regards,
>>
>> Anjul *TYAGI*
>>
>> * *
>>
>> *ü** **Go Green*
>>
>>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2018-06-26 12:47 Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 08:51 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 09:13 ` Re: Server Crash Fabio Pardi <f.pardi@portavita.eu>
@ 2018-06-27 10:40 ` Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 11:30 ` Re: Server Crash Fabio Pardi <f.pardi@portavita.eu>
0 siblings, 1 reply; 17+ messages in thread
From: Anjul Tyagi @ 2018-06-27 10:40 UTC (permalink / raw)
To: Fabio Pardi <f.pardi@portavita.eu>; pgsql-admin@lists.postgresql.org
Hi Fabio,
I understand your point, however i am in tremendous pressure to provide
the solution.
I have attached the log file for review.
We have Red Hat machine and Postgres 9.1.13 version.
Appreciate for your response and help.
Regards,
Anjul TYAGI
ü Go Green
------ Original Message ------
From: "Fabio Pardi" <f.pardi@portavita.eu>
To: pgsql-admin@lists.postgresql.org
Sent: 27-06-2018 14:43:30
Subject: Re: Server Crash
>Hi Anjul,
>
>please avoid cross posting over multiple mailing lists.
>
>Also asking again does not help, and usually is counterproductive:
>members see a reply and they might put efforts on helping somebody who
>did not get a reply yet.
>
>Please take good note of it.
>
>About your problem:
>
>I would suggest you to upgrade to a newer version, since Postgres 9.1
>is too old and does not get updates any longer.
>
>Besides that, not being a perl expert i cannot help with your
>procedure. I think anyway the problem might be somewhere else. I
>suspect that your server crashes and what you read there is only a
>consequence, not the cause.
>
>Could you post any relevant log entry in /var/log/messages and maybe
>anything else appearing in the postgres logfile?
>
>
>
>What kind of machine are you running on? can we have more specs?
>
>How is your server configured?
>
>
>
>regards,
>
>fabio pardi
>
>
>
>
>
>
>On 27/06/18 10:51, Anjul Tyagi wrote:
>>Hi All,
>>
>>can you please suggest on the issue we are facing?
>>
>>
>>
>>Regards,
>>
>>Anjul TYAGI
>>
>>
>>
>>ü Go Green
>>
>>
>>------ Original Message ------
>>From: "Anjul Tyagi" <anjul@ibosstech-us.com>
>>To: "pgsql-bugs@lists.postgresql.org"
>><mailto:pgsql-bugs@lists.postgresql.org>
>><pgsql-bugs@lists.postgresql.org>
>>Sent: 26-06-2018 18:17:26
>>Subject: Server Crash
>>
>>>Hi All,
>>>
>>>We have recently deployed couple of new plperl SPROC in our postgres
>>>production server and after that server start throwing error.
>>>Currently we are using postgres 9.1 and planning to upgrade with PG
>>>10.
>>>
>>>But we are not sure if that will cause the same error in PG 10 as
>>>well. We really need help.
>>>
>>>Error Message:
>>>
>>>2018-06-21 13:43:29 EDT [22212]: [4-1] user=cmsuser,db=forte
>>>,host=10.10.1.3,port=10.10.1.3(56601),id=5b2bdea5.56c4,line=4DETAIL:
>>>The postmaster has commanded this server process to roll back the
>>>current transaction and exit, because another server process exited
>>>abnormally and possibly corrupted shared memory.
>>>
>>>
>>>
>>>2018-06-21 13:43:29 EDT [3673]: [5-1] user=cmsuser,db=forte
>>>,host=10.10.1.3,port=10.10.1.3(55649),id=5b2bd568.e59,line=5DETAIL:
>>>The postmaster has commanded this server process to roll back the
>>>current transaction and exit, because another server process exited
>>>abnormally and possibly corrupted shared memory.
>>>
>>>
>>>Server Details:
>>>Linux - Red Hat Enterprise Linux Server release 6.3 (Santiago)
>>>Postgres - Postgres 9.1.13
>>>
>>>below are the sample SPROC code:
>>>
>>>
>>>CREATE OR REPLACE FUNCTION getauthcode(
>>> claim character varying,
>>> oflag character varying,
>>> pacode character varying,
>>> dispos character varying)
>>> RETURNS text AS
>>>$BODY$
>>>use strict;
>>>use warnings;
>>>use SOAP::Lite;
>>>use JSON;
>>>
>>>
>>>try
>>>{
>>> my $host = `hostname`;
>>> my $rv = spi_exec_query("select * from
>>>getsprocurl('getauthcode','".$host."')");
>>> my $url = $rv->{rows}[0]->{ret_url};
>>> elog(NOTICE, 'Host Name ' . $host . ' URL '. $url );
>>>
>>> my $soap = SOAP::Lite->new();
>>> my $service = $soap->service($url);
>>> my %params = ("claim" => $_[0], "oflag" => $_[1], "pacode" =>
>>>$_[2], "dispos" => $_[3]);
>>> my $response = $service->getAuthCode(%params);
>>> my $json_array = decode_json($response);
>>>
>>> for my $report ( $json_array) {
>>> my $status = $report->{status}, '\n';
>>> if ($status eq 'success')
>>> {
>>> return $report->{PriorApprovalCode};
>>> }
>>> else{
>>> return $report->{message};
>>> }
>>> }
>>>}
>>>catch Exception with
>>>{
>>> my $ex = shift;
>>> return 'SOAPFAULT: ' . date('H:i:s') . ' ' . exception($ex);
>>>}
>>>
>>>$BODY$
>>> LANGUAGE plperlu VOLATILE STRICT
>>> COST 100;
>>>
>>>Appreciate for your help in advance.
>>>
>>>Regards,
>>>
>>>Anjul TYAGI
>>>
>>>
>>>
>>>ü Go Green
>>>
>>>
>
Attachments:
PG_error.txt (text/plain, 1016.4K) download
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2018-06-26 12:47 Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 08:51 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 09:13 ` Re: Server Crash Fabio Pardi <f.pardi@portavita.eu>
2018-06-27 10:40 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
@ 2018-06-27 11:30 ` Fabio Pardi <f.pardi@portavita.eu>
0 siblings, 0 replies; 17+ messages in thread
From: Fabio Pardi @ 2018-06-27 11:30 UTC (permalink / raw)
To: Anjul Tyagi <anjul@ibosstech-us.com>; pgsql-admin@lists.postgresql.org
Hi,
what was running as PID 7471?
what happened before the first message at 10:29:52?
what is 'alarm clock' appearing at first line in the logfile?
still missing answers from my previous message:
/var/log/messages ?
how much ram on the machine?
what are the memory settings of postgres?
regards,
fabio pardi
On 27/06/18 12:40, Anjul Tyagi wrote:
> Hi Fabio,
>
> I understand your point, however i am in tremendous pressure to provide the solution.
>
> I have attached the log file for review.
>
> We have Red Hat machine and Postgres 9.1.13 version.
>
> Appreciate for your response and help.
>
>
>
>
> Regards,
>
> Anjul *TYAGI*
>
> * *
>
> *ü** **Go Green*
>
>
> ------ Original Message ------
> From: "Fabio Pardi" <f.pardi@portavita.eu <mailto:f.pardi@portavita.eu>>
> To: pgsql-admin@lists.postgresql.org <mailto:pgsql-admin@lists.postgresql.org>
> Sent: 27-06-2018 14:43:30
> Subject: Re: Server Crash
>
>> Hi Anjul,
>>
>> please avoid cross posting over multiple mailing lists.
>>
>> Also asking again does not help, and usually is counterproductive: members see a reply and they might put efforts on helping somebody who did not get a reply yet.
>>
>> Please take good note of it.
>>
>> About your problem:
>>
>> I would suggest you to upgrade to a newer version, since Postgres 9.1 is too old and does not get updates any longer.
>>
>> Besides that, not being aperl expert i cannot help with your procedure. I think anyway the problem might be somewhere else. I suspectthat your server crashes and what you read there is only a consequence, not the cause.
>>
>> Could you post any relevant logentry in /var/log/messages and maybe anything else appearing in the postgres logfile?
>>
>>
>> What kind of machine are you running on? can we have more specs?
>>
>> How is your server configured?
>>
>>
>> regards,
>>
>> fabio pardi
>>
>>
>>
>>
>> On 27/06/18 10:51, Anjul Tyagi wrote:
>>> Hi All,
>>>
>>> can you please suggest on the issue we are facing?
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Anjul *TYAGI*
>>>
>>> * *
>>>
>>> *ü** **Go Green*
>>>
>>>
>>> ------ Original Message ------
>>> From: "Anjul Tyagi" <anjul@ibosstech-us.com <mailto:anjul@ibosstech-us.com>>
>>> To: "pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org <mailto:pgsql-bugs@lists.postgresql.org>>
>>> Sent: 26-06-2018 18:17:26
>>> Subject: Server Crash
>>>
>>>> Hi All,
>>>>
>>>> We have recently deployed couple of new plperl SPROC in our postgres production server and after that server start throwing error. Currently we are using postgres 9.1 and planning to upgrade with PG 10.
>>>>
>>>> But we are not sure if that will cause the same error in PG 10 as well. We really need help.
>>>>
>>>> Error Message:
>>>>
>>>> 2018-06-21 13:43:29 EDT [22212]: [4-1] user=cmsuser,db=forte ,host=10.10.1.3,port=10.10.1.3(56601),id=5b2bdea5.56c4,line=4DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
>>>>
>>>>
>>>>
>>>> 2018-06-21 13:43:29 EDT [3673]: [5-1] user=cmsuser,db=forte ,host=10.10.1.3,port=10.10.1.3(55649),id=5b2bd568.e59,line=5DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
>>>>
>>>>
>>>> Server Details:
>>>> Linux - Red Hat Enterprise Linux Server release 6.3 (Santiago)
>>>> Postgres - Postgres 9.1.13
>>>>
>>>> below are the sample SPROC code:
>>>>
>>>>
>>>> CREATE OR REPLACE FUNCTION getauthcode(
>>>> claim character varying,
>>>> oflag character varying,
>>>> pacode character varying,
>>>> dispos character varying)
>>>> RETURNS text AS
>>>> $BODY$
>>>> use strict;
>>>> use warnings;
>>>> use SOAP::Lite;
>>>> use JSON;
>>>>
>>>>
>>>> try
>>>> {
>>>> my $host = `hostname`;
>>>> my $rv = spi_exec_query("select * from getsprocurl('getauthcode','".$host."')");
>>>> my $url = $rv->{rows}[0]->{ret_url};
>>>> elog(NOTICE, 'Host Name ' . $host . ' URL '. $url );
>>>>
>>>> my $soap = SOAP::Lite->new();
>>>> my $service = $soap->service($url);
>>>> my %params = ("claim" => $_[0], "oflag" => $_[1], "pacode" => $_[2], "dispos" => $_[3]);
>>>> my $response = $service->getAuthCode(%params);
>>>> my $json_array = decode_json($response);
>>>>
>>>> for my $report ( $json_array) {
>>>> my $status = $report->{status}, '\n';
>>>> if ($status eq 'success')
>>>> {
>>>> return $report->{PriorApprovalCode};
>>>> }
>>>> else{
>>>> return $report->{message};
>>>> }
>>>> }
>>>> }
>>>> catch Exception with
>>>> {
>>>> my $ex = shift;
>>>> return 'SOAPFAULT: ' . date('H:i:s') . ' ' . exception($ex);
>>>> }
>>>>
>>>> $BODY$
>>>> LANGUAGE plperlu VOLATILE STRICT
>>>> COST 100;
>>>>
>>>> Appreciate for your help in advance.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Anjul *TYAGI*
>>>>
>>>> * *
>>>>
>>>> *ü** **Go Green*
>>>>
>>>>
>>
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2018-06-26 12:47 Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
@ 2018-06-27 09:31 ` Sergei Kornilov <sk@zsrv.org>
2018-06-27 10:58 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
1 sibling, 1 reply; 17+ messages in thread
From: Sergei Kornilov @ 2018-06-27 09:31 UTC (permalink / raw)
To: Anjul Tyagi <anjul@ibosstech-us.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Hello
> 2018-06-21 13:43:29 EDT [22212]: [4-1] user=cmsuser,db=forte ,host=10.10.1.3,port=10.10.1.3(56601),id=5b2bdea5.56c4,line=4DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
This is just consequence of error. Similar message is reported by every client connect process. Real error was before all of these messages.
> Postgres - Postgres 9.1.13
You are using unsupported major version and even very old minor. Last released 9.1 was 9.1.24 (Release Date: 2016-10-27). You missed bug fixes for 2.5 years.
According versioning policy [1] bug fixing ceases for EOL versions.
I suggest upgrade to one of supported version. Postgresql 10 is good choice.
If error still occurs on supported version - we need exact error message from failed backend.
regards, Sergei
[1] https://www.postgresql.org/support/versioning/
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2018-06-26 12:47 Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 09:31 ` Re: Server Crash Sergei Kornilov <sk@zsrv.org>
@ 2018-06-27 10:58 ` Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-28 13:10 ` Re: Server Crash Alexey Bashtanov <bashtanov@imap.cc>
0 siblings, 1 reply; 17+ messages in thread
From: Anjul Tyagi @ 2018-06-27 10:58 UTC (permalink / raw)
To: Sergei Kornilov <sk@zsrv.org>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Hi Sergei,
thanks for your response. We have already plan to upgrade the same in PG
10, but i am scared if we faced the same issue with upgraded version as
well.
Regards,
Anjul TYAGI
ü Go Green
------ Original Message ------
From: "Sergei Kornilov" <sk@zsrv.org>
To: "Anjul Tyagi" <anjul@ibosstech-us.com>;
"pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org>
Sent: 27-06-2018 15:01:15
Subject: Re: Server Crash
>Hello
>
>>2018-06-21 13:43:29 EDT [22212]: [4-1] user=cmsuser,db=forte
>>,host=10.10.1.3,port=10.10.1.3(56601),id=5b2bdea5.56c4,line=4DETAIL:
>>The postmaster has commanded this server process to roll back the
>>current transaction and exit, because another server process exited
>>abnormally and possibly corrupted shared memory.
>This is just consequence of error. Similar message is reported by every
>client connect process. Real error was before all of these messages.
>
>>Postgres - Postgres 9.1.13
>You are using unsupported major version and even very old minor. Last
>released 9.1 was 9.1.24 (Release Date: 2016-10-27). You missed bug
>fixes for 2.5 years.
>According versioning policy [1] bug fixing ceases for EOL versions.
>
>I suggest upgrade to one of supported version. Postgresql 10 is good
>choice.
>If error still occurs on supported version - we need exact error
>message from failed backend.
>
>regards, Sergei
>
>[1] https://www.postgresql.org/support/versioning/
>
Attachments:
PG_error.txt (text/plain, 1016.4K) download
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Server Crash
2018-06-26 12:47 Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 09:31 ` Re: Server Crash Sergei Kornilov <sk@zsrv.org>
2018-06-27 10:58 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
@ 2018-06-28 13:10 ` Alexey Bashtanov <bashtanov@imap.cc>
0 siblings, 0 replies; 17+ messages in thread
From: Alexey Bashtanov @ 2018-06-28 13:10 UTC (permalink / raw)
To: Anjul Tyagi <anjul@ibosstech-us.com>; Sergei Kornilov <sk@zsrv.org>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Hello Anjul,
The first line of the log looks relevant.
My only guess is a stored procedure written in an untrusted language may
send a backend signal 14.
Do you have any of these, like pl/perl?
Best,
Alex
On 27/06/18 11:58, Anjul Tyagi wrote:
> Hi Sergei,
>
> thanks for your response. We have already plan to upgrade the same in
> PG 10, but i am scared if we faced the same issue with upgraded
> version as well.
>
> Regards,
>
> Anjul *TYAGI*
>
> **
>
> *ü****Go Green*
>
>
> ------ Original Message ------
> From: "Sergei Kornilov" <sk@zsrv.org <mailto:sk@zsrv.org>>
> To: "Anjul Tyagi" <anjul@ibosstech-us.com
> <mailto:anjul@ibosstech-us.com>>; "pgsql-bugs@lists.postgresql.org"
> <pgsql-bugs@lists.postgresql.org <mailto:pgsql-bugs@lists.postgresql.org>>
> Sent: 27-06-2018 15:01:15
> Subject: Re: Server Crash
>
>> Hello
>>> 2018-06-21 13:43:29 EDT [22212]: [4-1] user=cmsuser,db=forte
>>> ,host=10.10.1.3,port=10.10.1.3(56601),id=5b2bdea5.56c4,line=4DETAIL:
>>> The postmaster has commanded this server process to roll back the
>>> current transaction and exit, because another server process exited
>>> abnormally and possibly corrupted shared memory.
>> This is just consequence of error. Similar message is reported by
>> every client connect process. Real error was before all of these
>> messages.
>>> Postgres - Postgres 9.1.13
>> You are using unsupported major version and even very old minor. Last
>> released 9.1 was 9.1.24 (Release Date: 2016-10-27). You missed bug
>> fixes for 2.5 years.
>> According versioning policy [1] bug fixing ceases for EOL versions.
>> I suggest upgrade to one of supported version. Postgresql 10 is good
>> choice.
>> If error still occurs on supported version - we need exact error
>> message from failed backend.
>> regards, Sergei
>> [1] https://www.postgresql.org/support/versioning/
^ permalink raw reply [nested|flat] 17+ messages in thread
end of thread, other threads:[~2018-06-28 13:10 UTC | newest]
Thread overview: 17+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-11-09 16:01 server crash Medora Schauer <mschauer@fairfield.com>
2004-11-09 16:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-04-22 14:14 Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 15:13 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
2008-04-22 15:17 ` Re: Server Crash Ray Stell <stellr@cns.vt.edu>
2008-04-22 15:39 ` Re: Server Crash Hajek, Nick <Nick.Hajek@Vishay.com>
2008-04-22 16:06 ` Re: Server Crash Tom Lane <tgl@sss.pgh.pa.us>
2008-04-22 16:21 ` Re: Server Crash Scott Marlowe <scott.marlowe@gmail.com>
2008-04-22 16:32 ` Re: Server Crash Tom Lane <tgl@sss.pgh.pa.us>
2018-06-26 12:47 Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 08:51 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 09:13 ` Re: Server Crash Fabio Pardi <f.pardi@portavita.eu>
2018-06-27 10:40 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-27 11:30 ` Re: Server Crash Fabio Pardi <f.pardi@portavita.eu>
2018-06-27 09:31 ` Re: Server Crash Sergei Kornilov <sk@zsrv.org>
2018-06-27 10:58 ` Re: Server Crash Anjul Tyagi <anjul@ibosstech-us.com>
2018-06-28 13:10 ` Re: Server Crash Alexey Bashtanov <bashtanov@imap.cc>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox