Date: Mon, 23 Feb 2015 15:04:10 -0800
From: Cheyne Wallace <cheyne.wallace@gmail.com>
To: pgsql-admin@postgresql.org
Message-ID: <etPan.54ebb1ea.6b68079a.341@cheyne.local>
Subject: Postgres won't start after disk space issue
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Precedence: bulk
Sender: pgsql-admin-owner@postgresql.org

=C2=A0
Hey guys,=20=20

I have a problem i=E2=80=99m stuck on. Our Postgres server ran out of disk =
space last night and it fell over. We=E2=80=99ve resolved the disk space is=
sue and there is plenty now although Postgres will not start.=20=20

When I view the main log it reads:=20=20

2015-02-23 17:49:35 UTC LOG: =C2=A0redo starts at 75/202674D8
2015-02-23 17:49:35 UTC LOG: =C2=A0incomplete startup packet
2015-02-23 17:49:36 UTC FATAL: =C2=A0the database system is starting up
2015-02-23 17:49:36 UTC FATAL: =C2=A0the database system is starting up
2015-02-23 17:49:41 UTC FATAL: =C2=A0the database system is starting up
2015-02-23 17:49:41 UTC LOG: =C2=A0incomplete startup packet

When I look at ps , I can see that postgres is trying to perform a recovery=
, although it never finishes.=C2=A0

- postgres: startup process =C2=A0 recovering 00000001000000750000002D =C2=
=A0=C2=A0

I ran an strace on this process to try and see what it=E2=80=99s doing and =
I see these lines repeating over and over again:

read(7, 0x7fff6c82fadf, 1) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
=3D -1 EAGAIN (Resource temporarily unavailable)
open("pg_clog/01ED", O_RDWR|O_CREAT, 0600) =3D 27
lseek(27, 212992, SEEK_SET) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D 2=
12992
write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0=
"..., 8192) =3D 8192
fsync(27) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =3D 0
close(27)

When I look under pg_clog I can see that the 01ED file it=E2=80=99s opening=
 here is the most recent one and has a different file size to the others in=
 the dir.=C2=A0
I=E2=80=99m assuming this is the file it was writing too when the disk spac=
e ran out and now it=E2=80=99s corrupt or something along those lines.

Can anyone please help me figure out how to fix this? =C2=A0I was told that=
 deleting any of these pg_clog files would be bad, how ever I had no idea h=
ow to repair it.

p.s I also asked this question here:=C2=A0http://dba.stackexchange.com/ques=
tions/93594/postgres-wont-start-incomplete-startup-packet

Cheers
Cheyne


--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin