pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedPostgres won't start after disk space issue
2+ messages / 2 participants
[nested] [flat]
* Postgres won't start after disk space issue
@ 2015-02-23 23:04 Cheyne Wallace <cheyne.wallace@gmail.com>
0 siblings, 1 reply; 2+ messages in thread
From: Cheyne Wallace @ 2015-02-23 23:04 UTC (permalink / raw)
To: pgsql-admin
Hey guys,
I have a problem i’m stuck on. Our Postgres server ran out of disk space last night and it fell over. We’ve resolved the disk space issue and there is plenty now although Postgres will not start.
When I view the main log it reads:
2015-02-23 17:49:35 UTC LOG: redo starts at 75/202674D8
2015-02-23 17:49:35 UTC LOG: incomplete startup packet
2015-02-23 17:49:36 UTC FATAL: the database system is starting up
2015-02-23 17:49:36 UTC FATAL: the database system is starting up
2015-02-23 17:49:41 UTC FATAL: the database system is starting up
2015-02-23 17:49:41 UTC LOG: incomplete startup packet
When I look at ps , I can see that postgres is trying to perform a recovery, although it never finishes.
- postgres: startup process recovering 00000001000000750000002D
I ran an strace on this process to try and see what it’s doing and I see these lines repeating over and over again:
read(7, 0x7fff6c82fadf, 1) = -1 EAGAIN (Resource temporarily unavailable)
open("pg_clog/01ED", O_RDWR|O_CREAT, 0600) = 27
lseek(27, 212992, SEEK_SET) = 212992
write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192
fsync(27) = 0
close(27)
When I look under pg_clog I can see that the 01ED file it’s opening here is the most recent one and has a different file size to the others in the dir.
I’m assuming this is the file it was writing too when the disk space ran out and now it’s corrupt or something along those lines.
Can anyone please help me figure out how to fix this? I was told that deleting any of these pg_clog files would be bad, how ever I had no idea how to repair it.
p.s I also asked this question here: http://dba.stackexchange.com/questions/93594/postgres-wont-start-incomplete-startup-packet
Cheers
Cheyne
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: Postgres won't start after disk space issue
@ 2015-02-24 09:32 Albe Laurenz <laurenz.albe@wien.gv.at>
parent: Cheyne Wallace <cheyne.wallace@gmail.com>
0 siblings, 0 replies; 2+ messages in thread
From: Albe Laurenz @ 2015-02-24 09:32 UTC (permalink / raw)
To: Cheyne Wallace *EXTERN* <cheyne.wallace@gmail.com>; pgsql-admin
Cheyne Wallace wrote:
> I have a problem i’m stuck on. Our Postgres server ran out of disk space last night and it fell over.
> We’ve resolved the disk space issue and there is plenty now although Postgres will not start.
>
> When I view the main log it reads:
>
> 2015-02-23 17:49:35 UTC LOG: redo starts at 75/202674D8
> 2015-02-23 17:49:35 UTC LOG: incomplete startup packet
> 2015-02-23 17:49:36 UTC FATAL: the database system is starting up
> 2015-02-23 17:49:36 UTC FATAL: the database system is starting up
> 2015-02-23 17:49:41 UTC FATAL: the database system is starting up
> 2015-02-23 17:49:41 UTC LOG: incomplete startup packet
>
> When I look at ps , I can see that postgres is trying to perform a recovery, although it never
> finishes.
>
> - postgres: startup process recovering 00000001000000750000002D
>
> I ran an strace on this process to try and see what it’s doing and I see these lines repeating over
> and over again:
>
> read(7, 0x7fff6c82fadf, 1) = -1 EAGAIN (Resource temporarily unavailable)
> open("pg_clog/01ED", O_RDWR|O_CREAT, 0600) = 27
> lseek(27, 212992, SEEK_SET) = 212992
> write(27, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192) = 8192
> fsync(27) = 0
> close(27)
>
> When I look under pg_clog I can see that the 01ED file it’s opening here is the most recent one and
> has a different file size to the others in the dir.
> I’m assuming this is the file it was writing too when the disk space ran out and now it’s corrupt or
> something along those lines.
>
> Can anyone please help me figure out how to fix this? I was told that deleting any of these pg_clog
> files would be bad, how ever I had no idea how to repair it.
>
> p.s I also asked this question here: http://dba.stackexchange.com/questions/93594/postgres-wont-start-
> incomplete-startup-packet
Can you get a stack trace from the startup process?
Yours,
Laurenz Albe
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2015-02-24 09:32 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2015-02-23 23:04 Postgres won't start after disk space issue Cheyne Wallace <cheyne.wallace@gmail.com>
2015-02-24 09:32 ` Albe Laurenz <laurenz.albe@wien.gv.at>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox