pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Postgres database is not logging any messages after the reboot
7+ messages / 3 participants
[nested] [flat]

* Postgres database is not logging any messages after the reboot
@ 2007-10-10 17:43 Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
  0 siblings, 0 replies; 7+ messages in thread

From: Bhella Paramjeet-PFCW67 @ 2007-10-10 17:43 UTC (permalink / raw)
  To: pgsql-admin; +Cc: Subbiah Stalin-XCGF84 <SSubbiah@Motorola.com>

Hi 

The server that hosts our production postgres databases had to be
rebooted last night as the file systems got remounted as read only.
After the reboot we re-started the databases but now nothing is getting
logged into the pglogs. Can you please assist us in figuring out the
problem and help resolve it. Below listed are the error messages in the
/var/log/messages file due to which we had to reboot the server. This is
a linux x86-64 server. Your prompt response will be highly appreciated.

 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931168
Oct 10 06:45:02 sandbox kernel: Aborting journal on device sdb1.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931169
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931170
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931171
Oct 10 06:45:02 sandbox kernel: ext3_abort called.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_journal_start_sb: Detected aborted journa
l
Oct 10 06:45:02 sandbox kernel: Remounting filesystem read-only
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_free_blocks_sb: Journal has aborted
Oct 10 06:45:02 sandbox last message repeated 70 times
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_reserve_inode_write: Journal has aborte
d
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_truncate: Journal has aborted
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_reserve_inode_write: Journal has aborte
d
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_orphan_del: Journal has aborted
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_reserve_inode_write: Journal has aborte
d
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_delete_inode: Journal has aborted
Oct 10 06:45:02 sandbox kernel: __journal_remove_journal_head: freeing
b_committed_data

Thanks
Paramjeet Bhella
 

---------------------------(end of broadcast)---------------------------
TIP 7: You can help support the PostgreSQL project by donating at

                http://www.postgresql.org/about/donate



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Postgres database is not logging any messages after the reboot
@ 2007-10-10 17:43 Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
  2007-10-19 14:51 ` Re: Postgres database is not logging any messages after the reboot Alvaro Herrera <alvherre@commandprompt.com>
  0 siblings, 1 reply; 7+ messages in thread

From: Bhella Paramjeet-PFCW67 @ 2007-10-10 17:43 UTC (permalink / raw)
  To: pgsql-admin; +Cc: Subbiah Stalin-XCGF84 <SSubbiah@Motorola.com>

Hi 

The server that hosts our production postgres databases had to be
rebooted last night as the file systems got remounted as read only.
After the reboot we re-started the databases but now nothing is getting
logged into the pglogs. Can you please assist us in figuring out the
problem and help resolve it. Below listed are the error messages in the
/var/log/messages file due to which we had to reboot the server. This is
a linux x86-64 server. Your prompt response will be highly appreciated.

 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931168
Oct 10 06:45:02 sandbox kernel: Aborting journal on device sdb1.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931169
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931170
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b
lock 3931171
Oct 10 06:45:02 sandbox kernel: ext3_abort called.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_journal_start_sb: Detected aborted journa
l
Oct 10 06:45:02 sandbox kernel: Remounting filesystem read-only
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_free_blocks_sb: Journal has aborted
Oct 10 06:45:02 sandbox last message repeated 70 times
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_reserve_inode_write: Journal has aborte
d
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_truncate: Journal has aborted
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_reserve_inode_write: Journal has aborte
d
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_orphan_del: Journal has aborted
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_reserve_inode_write: Journal has aborte
d
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1) in
ext3_delete_inode: Journal has aborted
Oct 10 06:45:02 sandbox kernel: __journal_remove_journal_head: freeing
b_committed_data

Thanks
Paramjeet Bhella
 



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Postgres database is not logging any messages after the reboot
  2007-10-10 17:43 Postgres database is not logging any messages after the reboot Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
@ 2007-10-19 14:51 ` Alvaro Herrera <alvherre@commandprompt.com>
  0 siblings, 0 replies; 7+ messages in thread

From: Alvaro Herrera @ 2007-10-19 14:51 UTC (permalink / raw)
  To: Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>; +Cc: pgsql-admin@postgresql.org, Subbiah Stalin-XCGF84 <SSubbiah@Motorola.com>

Bhella Paramjeet-PFCW67 wrote:
> Hi 
> 
> The server that hosts our production postgres databases had to be
> rebooted last night as the file systems got remounted as read only.
> After the reboot we re-started the databases but now nothing is getting
> logged into the pglogs. Can you please assist us in figuring out the
> problem and help resolve it. Below listed are the error messages in the
> /var/log/messages file due to which we had to reboot the server. This is
> a linux x86-64 server. Your prompt response will be highly appreciated.

When a filesystem is automatically remounted read-only by the operating
system, it's because serious trouble is ahead.  Rebooting to remount
read-write seems a recipe for disaster unless you have corrected the
problem.

This may be a Linux kernel bug, a disk/cable/controller failure, or
something else.

-- 
Alvaro Herrera       Valdivia, Chile   ICBM: S 39º 49' 18.1", W 73º 13' 56.4"
"The ability to monopolize a planet is insignificant
next to the power of the source"



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Postgres database is not logging any messages after the reboot
@ 2007-10-10 18:05 Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
  2007-10-11 14:29 ` Re: Postgres database is not logging any messages after the reboot Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 7+ messages in thread

From: Bhella Paramjeet-PFCW67 @ 2007-10-10 18:05 UTC (permalink / raw)
  To: Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>; pgsql-admin; +Cc: Subbiah Stalin-XCGF84 <SSubbiah@Motorola.com>

Here are the messages from the pg_log before and after server reboot.

2007-10-10 06:22:50 CDT  LOG:  archived transaction log file
"000000010000007000000090"
 2007-10-10 06:29:04 CDT  LOG:  autovacuum: processing database "nsc"
 2007-10-10 06:34:58 CDT  LOG:  archived transaction log file
"000000010000007000000091"
 2007-10-10 07:11:48 CDT  LOG:  database system was interrupted at
2007-10-10 06:40:05 CDT
 2007-10-10 07:11:48 CDT  LOG:  checkpoint record is at 70/925E7B68
 2007-10-10 07:11:48 CDT  LOG:  redo record is at 70/925E7B68; undo
record is at 0/0; shutdown FALSE
 2007-10-10 07:11:48 CDT  LOG:  next transaction ID: 656073787; next
OID: 2915367
 2007-10-10 07:11:48 CDT  LOG:  next MultiXactId: 3599; next
MultiXactOffset: 7197
 2007-10-10 07:11:48 CDT  LOG:  database system was not properly shut
down; automatic recovery in progress
 2007-10-10 07:11:48 CDT  LOG:  redo starts at 70/925E7BB8
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl:  the database system is
starting up
:
:
:
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
 2007-10-10 07:11:53 CDT  LOG:  record with zero length at 70/92BBF1C8
 2007-10-10 07:11:53 CDT  LOG:  redo done at 70/92BBF198
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl:  the database system is
starting up
 2007-10-10 07:11:53 CDT  LOG:  database system is ready
 2007-10-10 07:11:53 CDT  LOG:  transaction ID wrap limit is 2147484146,
limited by database "postgres"
 2007-10-10 07:11:59 CDT  LOG:  archived transaction log file
"000000010000007000000092"
nsc 2007-10-10 07:19:37 CDT attbase WARNING:  there is no transaction in
progress
nsc 2007-10-10 07:20:48 CDT attbase WARNING:  there is no transaction in
progress
nsc 2007-10-10 07:23:53 CDT attbase WARNING:  there is no transaction in
progress
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG:  unexpected EOF on client
connection

Thanks
Paramjeet Kaur

-----Original Message-----
From: Bhella Paramjeet-PFCW67 
Sent: Wednesday, October 10, 2007 10:43 AM
To: 'pgsql-admin@postgresql.org'
Cc: Subbiah Stalin-XCGF84
Subject: Postgres database is not logging any messages after the reboot

Hi 

The server that hosts our production postgres databases had to be
rebooted last night as the file systems got remounted as read only.
After the reboot we re-started the databases but now nothing is getting
logged into the pglogs. Can you please assist us in figuring out the
problem and help resolve it. Below listed are the error messages in the
/var/log/messages file due to which we had to reboot the server. This is
a linux x86-64 server. Your prompt response will be highly appreciated.

 Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931168 Oct 10
06:45:02 sandbox kernel: Aborting journal on device sdb1.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931169 Oct 10
06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931170 Oct 10
06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931171 Oct 10
06:45:02 sandbox kernel: ext3_abort called.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_journal_start_sb: Detected aborted journa l Oct 10 06:45:02 sandbox
kernel: Remounting filesystem read-only Oct 10 06:45:02 sandbox kernel:
EXT3-fs error (device sdb1) in ext3_free_blocks_sb: Journal has aborted
Oct 10 06:45:02 sandbox last message repeated 70 times Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_truncate: Journal has aborted Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_orphan_del: Journal has aborted Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_delete_inode: Journal has aborted Oct 10 06:45:02
sandbox kernel: __journal_remove_journal_head: freeing b_committed_data

Thanks
Paramjeet Bhella
 



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Postgres database is not logging any messages after the reboot
  2007-10-10 18:05 Re: Postgres database is not logging any messages after the reboot Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
@ 2007-10-11 14:29 ` Tom Lane <tgl@sss.pgh.pa.us>
  2007-10-11 22:34   ` Re: Postgres database is not logging any messages after the reboot Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
  0 siblings, 1 reply; 7+ messages in thread

From: Tom Lane @ 2007-10-11 14:29 UTC (permalink / raw)
  To: Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>; +Cc: pgsql-admin@postgresql.org, "Subbiah Stalin-XCGF84" <SSubbiah@Motorola.com>

"Bhella Paramjeet-PFCW67" <PBhella@Motorola.com> writes:
> Here are the messages from the pg_log before and after server reboot.

Uh, I don't see the problem?  That looks like a perfectly normal
crash-recovery process, and it certainly doesn't seem to be the case
that "nothing is getting logged".  What were you expecting to see?

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Postgres database is not logging any messages after the reboot
  2007-10-10 18:05 Re: Postgres database is not logging any messages after the reboot Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
  2007-10-11 14:29 ` Re: Postgres database is not logging any messages after the reboot Tom Lane <tgl@sss.pgh.pa.us>
@ 2007-10-11 22:34   ` Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
  2007-10-11 22:38     ` Re: Postgres database is not logging any messages after the reboot Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 7+ messages in thread

From: Bhella Paramjeet-PFCW67 @ 2007-10-11 22:34 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin; Subbiah Stalin-XCGF84 <SSubbiah@Motorola.com>

Hi Tom

After the server reboot nothing is getting logged by the postgres
database into pg_log like messages about archived log, autovacuum,
panic, alerts etc. The alerts from the pglogs get forwarded to our cell
phones and since no messages are getting logged we will not get notified
in case of a problem with database. This is the first time we have seen
this strange behaviour. Your help in resolving the issue will be highly
appreciated. 

Thanks
Paramjeet Kaur 

-----Original Message-----
From: Tom Lane [mailto:tgl@sss.pgh.pa.us] 
Sent: Thursday, October 11, 2007 7:30 AM
To: Bhella Paramjeet-PFCW67
Cc: pgsql-admin@postgresql.org; Subbiah Stalin-XCGF84
Subject: Re: [ADMIN] Postgres database is not logging any messages after
the reboot 

"Bhella Paramjeet-PFCW67" <PBhella@Motorola.com> writes:
> Here are the messages from the pg_log before and after server reboot.

Uh, I don't see the problem?  That looks like a perfectly normal
crash-recovery process, and it certainly doesn't seem to be the case
that "nothing is getting logged".  What were you expecting to see?

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Postgres database is not logging any messages after the reboot
  2007-10-10 18:05 Re: Postgres database is not logging any messages after the reboot Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
  2007-10-11 14:29 ` Re: Postgres database is not logging any messages after the reboot Tom Lane <tgl@sss.pgh.pa.us>
  2007-10-11 22:34   ` Re: Postgres database is not logging any messages after the reboot Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
@ 2007-10-11 22:38     ` Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 7+ messages in thread

From: Tom Lane @ 2007-10-11 22:38 UTC (permalink / raw)
  To: Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>; +Cc: pgsql-admin@postgresql.org, "Subbiah Stalin-XCGF84" <SSubbiah@Motorola.com>

"Bhella Paramjeet-PFCW67" <PBhella@Motorola.com> writes:
> After the server reboot nothing is getting logged by the postgres
> database into pg_log like messages about archived log, autovacuum,
> panic, alerts etc.

Sounds to me like you changed your logging settings...

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 7+ messages in thread


end of thread, other threads:[~2007-10-19 14:51 UTC | newest]

Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2007-10-10 17:43 Postgres database is not logging any messages after the reboot Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
2007-10-10 17:43 Postgres database is not logging any messages after the reboot Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
2007-10-19 14:51 ` Alvaro Herrera <alvherre@commandprompt.com>
2007-10-10 18:05 Re: Postgres database is not logging any messages after the reboot Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
2007-10-11 14:29 ` Tom Lane <tgl@sss.pgh.pa.us>
2007-10-11 22:34   ` Bhella Paramjeet-PFCW67 <PBhella@Motorola.com>
2007-10-11 22:38     ` Tom Lane <tgl@sss.pgh.pa.us>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox