pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedBus error in postgres 8.3
6+ messages / 5 participants
[nested] [flat]
* Bus error in postgres 8.3
@ 2008-04-29 05:05 Peter Koczan <pjkoczan@gmail.com>
2008-04-29 06:01 ` Re: Bus error in postgres 8.3 Tom Lane <tgl@sss.pgh.pa.us>
2008-04-29 06:03 ` Re: Bus error in postgres 8.3 Pavan Deolasee <pavan.deolasee@gmail.com>
0 siblings, 2 replies; 6+ messages in thread
From: Peter Koczan @ 2008-04-29 05:05 UTC (permalink / raw)
To: pgsql-admin
Hi all,
I'm looking to push 8.3 out this week, but I'm running into a
particularly nasty bus error. I'm not sure what's causing it as it
appears to be transient (or at least somewhat random), but I do know
that it bites on connection time and takes down the entire server with
it.
I'm going to try to force a crash on my test server to see if I can
find out anything more. Does anyone know how to get useful debugging
info at the time of the crash? I don't think I have things set to dump
core anywhere, but that and/or stack traces would be nice (especially
for child processes).
If anyone has any idea what's going on and could clue me in, that
would be excellent.
Peter
P.S. Here's the syslog transcript of the crash.
Apr 28 23:02:02 mitchell postgres[3442]: [3-1] LOG: connection
received: host=mitchell.cs.wisc.edu port=37588
Apr 28 23:02:02 mitchell postgres[3442]: [4-1] LOG: connection
authorized: user=postgres database=template1
Apr 28 23:02:02 mitchell postgres[3442]: [5-1] LOG: disconnection:
session time: 0:00:00.239 user=postgres database=template1
host=mitchell.cs.wisc.edu port=37588
Apr 28 23:02:02 mitchell postgres[3444]: [3-1] LOG: connection
received: host=mitchell.cs.wisc.edu port=37589
Apr 28 23:02:02 mitchell postgres[3444]: [4-1] LOG: connection
authorized: user=postgres database=564testdb
Apr 28 23:02:02 mitchell postgres[461]: [3-1] LOG: server process
(PID 3444) was terminated by signal 7: Bus error
Apr 28 23:02:02 mitchell postgres[461]: [4-1] LOG: terminating any
other active server processes
Apr 28 23:02:02 mitchell postgres[461]: [5-1] LOG: all server
processes terminated; reinitializing
Apr 28 23:02:02 mitchell postgres[3447]: [6-1] LOG: connection
received: host=mitchell.cs.wisc.edu port=37590
Apr 28 23:02:02 mitchell postgres[3447]: [7-1] FATAL: the database
system is in recovery mode
Apr 28 23:02:02 mitchell postgres[3448]: [6-1] LOG: connection
received: host=mitchell.cs.wisc.edu port=37591
Apr 28 23:02:02 mitchell postgres[3448]: [7-1] FATAL: the database
system is in recovery mode
Apr 28 23:02:23 mitchell postgres[461]: [6-1] LOG: startup process
(PID 3446) was terminated by signal 7: Bus error
Apr 28 23:02:23 mitchell postgres[461]: [7-1] LOG: aborting startup
due to startup process failure
Apr 28 23:22:15 mitchell postgres[3702]: [1-1] LOG: could not load
root certificate file "root.crt": No such file or directory
Apr 28 23:22:15 mitchell postgres[3702]: [1-2] DETAIL: Will not
verify client certificates.
Apr 28 23:22:15 mitchell postgres[3703]: [2-1] LOG: database system
was interrupted; last known up at 2008-04-28 07:02:32 CDT
Apr 28 23:22:15 mitchell postgres[3703]: [3-1] LOG: database system
was not properly shut down; automatic recovery in progress
Apr 28 23:22:15 mitchell postgres[3703]: [4-1] LOG: record with zero
length at 6/915320A0
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Bus error in postgres 8.3
2008-04-29 05:05 Bus error in postgres 8.3 Peter Koczan <pjkoczan@gmail.com>
@ 2008-04-29 06:01 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-04-30 20:38 ` Re: Bus error in postgres 8.3 Peter Koczan <pjkoczan@gmail.com>
2008-05-01 08:22 ` Re: Bus error in postgres 8.3 Filip Svendsen <fs@basepointmedia.com>
1 sibling, 2 replies; 6+ messages in thread
From: Tom Lane @ 2008-04-29 06:01 UTC (permalink / raw)
To: Peter Koczan <pjkoczan@gmail.com>; +Cc: pgsql-admin
"Peter Koczan" <pjkoczan@gmail.com> writes:
> I'm going to try to force a crash on my test server to see if I can
> find out anything more. Does anyone know how to get useful debugging
> info at the time of the crash? I don't think I have things set to dump
> core anywhere, but that and/or stack traces would be nice (especially
> for child processes).
Make sure the postmaster is started under ulimit -c unlimited.
On a depressingly large fraction of modern platforms, daemons are
started with ulimit -c 0 by default :-(. Try putting "ulimit -c unlimited"
into your PG init script and restarting.
regards, tom lane
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Bus error in postgres 8.3
2008-04-29 05:05 Bus error in postgres 8.3 Peter Koczan <pjkoczan@gmail.com>
2008-04-29 06:01 ` Re: Bus error in postgres 8.3 Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-04-30 20:38 ` Peter Koczan <pjkoczan@gmail.com>
1 sibling, 0 replies; 6+ messages in thread
From: Peter Koczan @ 2008-04-30 20:38 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Pavan Deolasee <pavan.deolasee@gmail.com>; +Cc: pgsql-admin
On Tue, Apr 29, 2008 at 1:01 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Make sure the postmaster is started under ulimit -c unlimited.
> On a depressingly large fraction of modern platforms, daemons are
> started with ulimit -c 0 by default :-(. Try putting "ulimit -c unlimited"
> into your PG init script and restarting.
> On Tue, Apr 29, 2008 at 10:35 AM, Peter Koczan <pjkoczan@gmail.com> wrote:
> Yeah, a stack trace and if possible, a self contained test case to
> reproduce the bug would help. If you are using a custom build, then
> using a debug build would help a lot too.
So far, this problem hasn't reoccurred since I first saw it, no matter
how much I pummel my test server.
I was testing some patches about the time that this happened
(including one in the libpq backend), so there may have been an ABI
mismatch which led to the bus error.
In any case, I've set the postmaster to have ulimit -c unlimited in
case this problem reoccurs. I'll also look into using a debug build in
case this happens again. But until then, I'm confident enough to keep
pushing out 8.3.
Thanks again.
Peter
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Bus error in postgres 8.3
2008-04-29 05:05 Bus error in postgres 8.3 Peter Koczan <pjkoczan@gmail.com>
2008-04-29 06:01 ` Re: Bus error in postgres 8.3 Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-05-01 08:22 ` Filip Svendsen <fs@basepointmedia.com>
2008-05-02 08:59 ` Re: Bus error in postgres 8.3 Darren Reed <darrenr@fastmail.net>
1 sibling, 1 reply; 6+ messages in thread
From: Filip Svendsen @ 2008-05-01 08:22 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Peter Koczan <pjkoczan@gmail.com>; pgsql-admin
On Tue, Apr 29, 2008 at 8:01 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> "Peter Koczan" <pjkoczan@gmail.com> writes:
> > I'm going to try to force a crash on my test server to see if I can
> > find out anything more. Does anyone know how to get useful debugging
> > info at the time of the crash? I don't think I have things set to dump
> > core anywhere, but that and/or stack traces would be nice (especially
> > for child processes).
>
> Make sure the postmaster is started under ulimit -c unlimited.
> On a depressingly large fraction of modern platforms, daemons are
> started with ulimit -c 0 by default :-(. Try putting "ulimit -c unlimited"
> into your PG init script and restarting.
>
> regards, tom lane
Do you think a too low limit on user processes could be causing the problem?
We have had similar crashes (server process terminated with Bus
Error). This is running 8.2.5 on OS X (Tiger, intel), with about
130-140 connections, but the process count limit is 1000. Any reason
why the process limit should be vastly higher than the number of
connections?
--
Venlig Hilsen / Kind Regards
Filip Svendsen
fs@basepointmedia.com
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Bus error in postgres 8.3
2008-04-29 05:05 Bus error in postgres 8.3 Peter Koczan <pjkoczan@gmail.com>
2008-04-29 06:01 ` Re: Bus error in postgres 8.3 Tom Lane <tgl@sss.pgh.pa.us>
2008-05-01 08:22 ` Re: Bus error in postgres 8.3 Filip Svendsen <fs@basepointmedia.com>
@ 2008-05-02 08:59 ` Darren Reed <darrenr@fastmail.net>
0 siblings, 0 replies; 6+ messages in thread
From: Darren Reed @ 2008-05-02 08:59 UTC (permalink / raw)
To: Filip Svendsen <fs@basepointmedia.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Peter Koczan <pjkoczan@gmail.com>; pgsql-admin
Filip Svendsen wrote:
> On Tue, Apr 29, 2008 at 8:01 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
>> "Peter Koczan" <pjkoczan@gmail.com> writes:
>> > I'm going to try to force a crash on my test server to see if I can
>> > find out anything more. Does anyone know how to get useful debugging
>> > info at the time of the crash? I don't think I have things set to dump
>> > core anywhere, but that and/or stack traces would be nice (especially
>> > for child processes).
>>
>> Make sure the postmaster is started under ulimit -c unlimited.
>> On a depressingly large fraction of modern platforms, daemons are
>> started with ulimit -c 0 by default :-(. Try putting "ulimit -c unlimited"
>> into your PG init script and restarting.
>>
>> regards, tom lane
>>
>
> Do you think a too low limit on user processes could be causing the problem?
>
> We have had similar crashes (server process terminated with Bus
> Error). This is running 8.2.5 on OS X (Tiger, intel), with about
> 130-140 connections, but the process count limit is 1000. Any reason
> why the process limit should be vastly higher than the number of
> connections?
>
"ulimit -c unlimited" sets the core dump size to be unlimited, not the
connection count.
That would be related to "ulimit -n" (descriptor count.)
Darren
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Bus error in postgres 8.3
2008-04-29 05:05 Bus error in postgres 8.3 Peter Koczan <pjkoczan@gmail.com>
@ 2008-04-29 06:03 ` Pavan Deolasee <pavan.deolasee@gmail.com>
1 sibling, 0 replies; 6+ messages in thread
From: Pavan Deolasee @ 2008-04-29 06:03 UTC (permalink / raw)
To: Peter Koczan <pjkoczan@gmail.com>; +Cc: pgsql-admin
On Tue, Apr 29, 2008 at 10:35 AM, Peter Koczan <pjkoczan@gmail.com> wrote:
>
> I'm going to try to force a crash on my test server to see if I can
> find out anything more. Does anyone know how to get useful debugging
> info at the time of the crash? I don't think I have things set to dump
> core anywhere, but that and/or stack traces would be nice (especially
> for child processes).
>
Yeah, a stack trace and if possible, a self contained test case to
reproduce the bug would help. If you are using a custom build, then
using a debug build would help a lot too.
Thanks,
Pavan
--
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2008-05-02 08:59 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-04-29 05:05 Bus error in postgres 8.3 Peter Koczan <pjkoczan@gmail.com>
2008-04-29 06:01 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-04-30 20:38 ` Peter Koczan <pjkoczan@gmail.com>
2008-05-01 08:22 ` Filip Svendsen <fs@basepointmedia.com>
2008-05-02 08:59 ` Darren Reed <darrenr@fastmail.net>
2008-04-29 06:03 ` Pavan Deolasee <pavan.deolasee@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox