Received: from makus.postgresql.org (makus.postgresql.org [98.129.198.125]) by mail.postgresql.org (Postfix) with ESMTP id E8F5CC3830A for ; Thu, 14 Jun 2012 04:34:01 -0300 (ADT) Received: from snt0-omc1-s32.snt0.hotmail.com ([65.55.90.43]) by makus.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1Sf4Z5-0005lC-Gd for pgsql-admin@postgresql.org; Thu, 14 Jun 2012 07:34:01 +0000 Received: from SNT139-W28 ([65.55.90.7]) by snt0-omc1-s32.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 14 Jun 2012 00:33:45 -0700 Message-ID: Content-Type: multipart/alternative; boundary="_f3431e55-8f37-434e-98af-ce43a89fbd2b_" X-Originating-IP: [189.174.146.236] From: Fabricio To: , Subject: Re: (new thread) could not rename temporary statistics file "pg_stat_tmp/pgstat.tmp" to "pg_stat_tmp/pgstat.stat": No such file or directory Date: Thu, 14 Jun 2012 00:33:45 -0700 Importance: Normal In-Reply-To: <4FD19416.4000700@ringerc.id.au> References: <4FD19416.4000700@ringerc.id.au> MIME-Version: 1.0 X-OriginalArrivalTime: 14 Jun 2012 07:33:45.0994 (UTC) FILETIME=[08193EA0:01CD4A00] X-Pg-Spam-Score: -1.7 (-) X-Archive-Number: 201206/74 X-Sequence-Number: 39665 --_f3431e55-8f37-434e-98af-ce43a89fbd2b_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable > Date: Fri=2C 8 Jun 2012 13:56:38 +0800 > From: ringerc@ringerc.id.au > To: pgsql-admin@postgresql.org=3B fabrixio1@hotmail.com > Subject: [ADMIN] (new thread) could not rename temporary statistics file = "pg_stat_tmp/pgstat.tmp" to "pg_stat_tmp/pgstat.stat": No such file or dire= ctory >=20 > > Hi. > > > > I have this problem: > > > > I have PostgreSQL 9.1.3 and the last night crash it. > > > > This was the first error after an autovacuum (the night before last): > > > > <2012-06-06 00:59:07 MDT 814 4fceffbb.32e >LOG: autovacuum: found=20 > > orphan temp table "(null)"."tmpmuestadistica" in database "dbRX" > > <2012-06-06 01:05:26 MDT 1854 4fc7d1eb.73e >LOG: could not rename=20 > > temporary statistics file "pg_stat_tmp/pgstat.tmp" to=20 > > "pg_stat_tmp/pgstat.stat": No such file or directory > > <2012-06-06 01:05:28 MDT 1383 4fcf0136.567 >ERROR: tuple=20 > > concurrently updated > > <2012-06-06 01:05:28 MDT 1383 4fcf0136.567 >CONTEXT: automatic=20 > > vacuum of table "global.pg_catalog.pg_attrdef" > > <2012-06-06 01:06:09 MDT 1851 4fc7d1eb.73b >ERROR: xlog flush=20 > > request 4/E29EE490 is not satisfied --- flushed only to 3/13527A10 > > <2012-06-06 01:06:09 MDT 1851 4fc7d1eb.73b >CONTEXT: writing block= =20 > > 0 of relation base/311360/12244_vm > > <2012-06-06 01:06:10 MDT 1851 4fc7d1eb.73b >ERROR: xlog flush=20 > > request 4/E29EE490 is not satisfied --- flushed only to 3/13527A10 > > <2012-06-06 01:06:10 MDT 1851 4fc7d1eb.73b >CONTEXT: writing block= =20 > > 0 of relation base/311360/12244_vm > > <2012-06-06 01:06:10 MDT 1851 4fc7d1eb.73b >WARNING: could not=20 > > write block 0 of base/311360/12244_vm > > <2012-06-06 01:06:10 MDT 1851 4fc7d1eb.73b >DETAIL: Multiple=20 > > failures --- write error might be permanent. > > > > > > Last night it was terminated by signal 6. > > > > <2012-06-07 01:36:44 MDT 2509 4fd05a0c.9cd >LOG: startup process=20 > > (PID 2525) was terminated by signal 6: Aborted > > <2012-06-07 01:36:44 MDT 2509 4fd05a0c.9cd >LOG: aborting startup=20 > > due to startup process failure > > <2012-06-07 01:37:37 MDT 2680 4fd05a41.a78 >LOG: database system=20 > > shutdown was interrupted=3B last known up at 2012-06-07 01:29:40 MDT > > <2012-06-07 01:37:37 MDT 2680 4fd05a41.a78 >LOG: could not open=20 > > file "pg_xlog/000000010000000300000013" (log file 3=2C segment 19): No= =20 > > such file or directory > > <2012-06-07 01:37:37 MDT 2680 4fd05a41.a78 >LOG: invalid primary=20 > > checkpoint record > > > > And the only option was pg_resetxlog. > > > > After this a lot of querys showed me this error: > > <2012-06-07 09:24:22 MDT 1306 4fd0c7a6.51a >ERROR: missing chunk=20 > > number 0 for toast value 393330 in pg_toast_2619 > > <2012-06-07 09:24:31 MDT 1306 4fd0c7a6.51a >ERROR: missing chunk=20 > > number 0 for toast value 393332 in pg_toast_2619 > > > > I lost some databases. > > > > I restarted the cluster again with initdb and then I restored the=20 > > databases that I could backup (for the other I restored an old backup) > > > > no space or permissions problem. No filesystem or disk error. > > > > Can you help me to know what happened? > > >=20 > Did you take a copy of the PostgreSQL data directory and error logs=20 > before you tried to fix the problem=2C as per the advice here: >=20 Sorry=2C I didn't. Maybe was filesystem error: This was the S.O log. ext4 filesyetem Jun 7 01:32:55 SERVIDOR kernel: IRQ 71/cciss0: IRQF_DISABLED is not guara= nteed on shared IRQs Jun 7 01:32:55 SERVIDOR kernel: cciss/c0d0: p2 size 1127850720 exceeds de= vice capacity=2C limited to end of disk Jun 7 01:32:55 SERVIDOR kernel: JBD: barrier-based sync failed on cciss!c= 0d0p2-8 - disabling barriers Jun 7 01:32:55 SERVIDOR kernel: JBD: barrier-based sync failed on cciss!c= 0d0p2-8 - disabling b > http://wiki.postgresql.org/wiki/Corruption >=20 > If you did=2C it might be possible to tell what happened. If you didn't=20 > then you've probably destroyed the evidence needed to determine what=20 > went wrong (and maybe recover some lost data). >=20 > -- > Craig Ringer = --_f3431e55-8f37-434e-98af-ce43a89fbd2b_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable


>=3B = Date: Fri=2C 8 Jun 2012 13:56:38 +0800
>=3B From: ringerc@ringerc.id.a= u
>=3B To: pgsql-admin@postgresql.org=3B fabrixio1@hotmail.com
>= =3B Subject: [ADMIN] (new thread) could not rename temporary statistics fil= e "pg_stat_tmp/pgstat.tmp" to "pg_stat_tmp/pgstat.stat": No such file or di= rectory
>=3B
>=3B >=3B Hi.
>=3B >=3B
>=3B >=3B I= have this problem:
>=3B >=3B
>=3B >=3B I have PostgreSQL 9.1= .3 and the last night crash it.
>=3B >=3B
>=3B >=3B This was = the first error after an autovacuum (the night before last):
>=3B >= =3B
>=3B >=3B <=3B2012-06-06 00:59:07 MDT 814 4fceffbb.32e >= =3BLOG: autovacuum: found
>=3B >=3B orphan temp table "(null)"."tm= pmuestadistica" in database "dbRX"
>=3B >=3B <=3B2012-06-06 01:05:= 26 MDT 1854 4fc7d1eb.73e >=3BLOG: could not rename
>=3B >=3B = temporary statistics file "pg_stat_tmp/pgstat.tmp" to
>=3B >=3B "pg= _stat_tmp/pgstat.stat": No such file or directory
>=3B >=3B <=3B20= 12-06-06 01:05:28 MDT 1383 4fcf0136.567 >=3BERROR: tuple
>=3B &= gt=3B concurrently updated
>=3B >=3B <=3B2012-06-06 01:05:28 MDT = 1383 4fcf0136.567 >=3BCONTEXT: automatic
>=3B >=3B vacuum of t= able "global.pg_catalog.pg_attrdef"
>=3B >=3B <=3B2012-06-06 01:06= :09 MDT 1851 4fc7d1eb.73b >=3BERROR: xlog flush
>=3B >=3B req= uest 4/E29EE490 is not satisfied --- flushed only to 3/13527A10
>=3B &= gt=3B <=3B2012-06-06 01:06:09 MDT 1851 4fc7d1eb.73b >=3BCONTEXT: wr= iting block
>=3B >=3B 0 of relation base/311360/12244_vm
>=3B = >=3B <=3B2012-06-06 01:06:10 MDT 1851 4fc7d1eb.73b >=3BERROR: xlo= g flush
>=3B >=3B request 4/E29EE490 is not satisfied --- flushed o= nly to 3/13527A10
>=3B >=3B <=3B2012-06-06 01:06:10 MDT 1851 4f= c7d1eb.73b >=3BCONTEXT: writing block
>=3B >=3B 0 of relation ba= se/311360/12244_vm
>=3B >=3B <=3B2012-06-06 01:06:10 MDT 1851 4= fc7d1eb.73b >=3BWARNING: could not
>=3B >=3B write block 0 of ba= se/311360/12244_vm
>=3B >=3B <=3B2012-06-06 01:06:10 MDT 1851 4= fc7d1eb.73b >=3BDETAIL: Multiple
>=3B >=3B failures --- write er= ror might be permanent.
>=3B >=3B
>=3B >=3B
>=3B >=3B = Last night it was terminated by signal 6.
>=3B >=3B
>=3B >=3B= <=3B2012-06-07 01:36:44 MDT 2509 4fd05a0c.9cd >=3BLOG: startup pro= cess
>=3B >=3B (PID 2525) was terminated by signal 6: Aborted
&g= t=3B >=3B <=3B2012-06-07 01:36:44 MDT 2509 4fd05a0c.9cd >=3BLOG: = aborting startup
>=3B >=3B due to startup process failure
>=3B= >=3B <=3B2012-06-07 01:37:37 MDT 2680 4fd05a41.a78 >=3BLOG: data= base system
>=3B >=3B shutdown was interrupted=3B last known up at = 2012-06-07 01:29:40 MDT
>=3B >=3B <=3B2012-06-07 01:37:37 MDT 2= 680 4fd05a41.a78 >=3BLOG: could not open
>=3B >=3B file "pg_xlog= /000000010000000300000013" (log file 3=2C segment 19): No
>=3B >=3B= such file or directory
>=3B >=3B <=3B2012-06-07 01:37:37 MDT 2= 680 4fd05a41.a78 >=3BLOG: invalid primary
>=3B >=3B checkpoint r= ecord
>=3B >=3B
>=3B >=3B And the only option was pg_resetxlo= g.
>=3B >=3B
>=3B >=3B After this a lot of querys showed me t= his error:
>=3B >=3B <=3B2012-06-07 09:24:22 MDT 1306 4fd0c7a6.51a= >=3BERROR: missing chunk
>=3B >=3B number 0 for toast value 3933= 30 in pg_toast_2619
>=3B >=3B <=3B2012-06-07 09:24:31 MDT 1306 4fd= 0c7a6.51a >=3BERROR: missing chunk
>=3B >=3B number 0 for toast v= alue 393332 in pg_toast_2619
>=3B >=3B
>=3B >=3B I lost some = databases.
>=3B >=3B
>=3B >=3B I restarted the cluster again = with initdb and then I restored the
>=3B >=3B databases that I cou= ld backup (for the other I restored an old backup)
>=3B >=3B
>= =3B >=3B no space or permissions problem. No filesystem or disk error.>=3B >=3B
>=3B >=3B Can you help me to know what happened?
&= gt=3B >=3B
>=3B
>=3B Did you take a copy of the PostgreSQL dat= a directory and error logs
>=3B before you tried to fix the problem= =2C as per the advice here:
>=3B

Sorry=2C I didn't.

May= be was filesystem error:

This was the S.O log.

ext4 filesyete= m
 Jun  7 01:32:55 SERVIDOR kernel: IRQ 71/cciss0: IRQF_DISABLED is=
 not guaranteed on shared IRQs
Jun 7 01:32:55 SERVIDOR kernel: cciss/c= 0d0: p2 size 1127850720 exceeds device capacity=2C limited to end of disk Jun 7 01:32:55 SERVIDOR kernel: JBD: barrier-based sync failed on cciss= !c0d0p2-8 - disabling barriers
Jun 7 01:32:55 SERVIDOR kernel: JBD: ba= rrier-based sync failed on cciss!c0d0p2-8 - disabling b


>=3B= http://wiki.postgresql.org/wiki/Corruption
>=3B
>=3B If you did= =2C it might be possible to tell what happened. If you didn't
>=3B th= en you've probably destroyed the evidence needed to determine what
>= =3B went wrong (and maybe recover some lost data).
>=3B
>=3B --<= br>>=3B Craig Ringer
= --_f3431e55-8f37-434e-98af-ce43a89fbd2b_--