pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedERROR: could not open file "base/125542/12631" Corruption?
6+ messages / 3 participants
[nested] [flat]
* ERROR: could not open file "base/125542/12631" Corruption?
@ 2013-10-04 14:40 Mike Broers <mbroers@gmail.com>
0 siblings, 2 replies; 6+ messages in thread
From: Mike Broers @ 2013-10-04 14:40 UTC (permalink / raw)
To: pgsql-admin
Hello, our postgresql 9.2.4 qa database (thankfully its just qa) seems to
be hosed.
Starting at around 3:39am last night I started seeing errors about missing
files and now I cannot run a pgdump or a vacuum without it complaining
about files that it cannot find with errors like this: ERROR: could not
open file "base/125542/12631". When I check the filesystem the files are
indeed not there. The 1am regular vacuum completed and its log is clean.
The postgres log is clean before these errors occurred.
Since this is qa we do not perform backups, and the solution if we cannot
repair the problem will be to create a fresh qa server but I am intrigued
about how to determine the source of the problem and the extent of the
problem.
Is there a way to force vacuum to continue on errors or an alternate way to
help determine all the missing files?
It might be totally unrelated, but yesterday morning on this qa server I
stopped postgres, and created a symlink to pg_xlog so that it was writing
to a different volume, and restarted. This was working fine all day so its
possibly a red herring but I thought I should mention it.
Any advice is appreciated, thanks!
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: ERROR: could not open file "base/125542/12631" Corruption?
@ 2013-10-04 16:53 Alejandro Brust <alejandrob@pasteleros.org.ar>
parent: Mike Broers <mbroers@gmail.com>
1 sibling, 1 reply; 6+ messages in thread
From: Alejandro Brust @ 2013-10-04 16:53 UTC (permalink / raw)
To: pgsql-admin
El 04/10/2013 11:40, Mike Broers escribió:
> Hello, our postgresql 9.2.4 qa database (thankfully its just qa) seems
> to be hosed.
>
> Starting at around 3:39am last night I started seeing errors about
> missing files and now I cannot run a pgdump or a vacuum without it
> complaining about files that it cannot find with errors like this:
> ERROR: could not open file "base/125542/12631". When I check the
> filesystem the files are indeed not there. The 1am regular vacuum
> completed and its log is clean. The postgres log is clean before
> these errors occurred.
>
> Since this is qa we do not perform backups, and the solution if we
> cannot repair the problem will be to create a fresh qa server but I am
> intrigued about how to determine the source of the problem and the
> extent of the problem.
>
> Is there a way to force vacuum to continue on errors or an alternate
> way to help determine all the missing files?
>
> It might be totally unrelated, but yesterday morning on this qa server
> I stopped postgres, and created a symlink to pg_xlog so that it was
> writing to a different volume, and restarted. This was working fine
> all day so its possibly a red herring but I thought I should mention it.
>
> Any advice is appreciated, thanks!
>
>
/Hello
U could try some like "SET/ zero_damaged_pages = on" and perform a
vacuum-db and may be pg_dump
hope it help!
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: ERROR: could not open file "base/125542/12631" Corruption?
@ 2013-10-04 17:10 Mike Broers <mbroers@gmail.com>
parent: Mike Broers <mbroers@gmail.com>
1 sibling, 2 replies; 6+ messages in thread
From: Mike Broers @ 2013-10-04 17:10 UTC (permalink / raw)
To: pgsql-admin
Strange, this is happening in a totally different environment now too. The
only thing these two environments share is a SAN, but I wouldnt think
something going on at the SAN level would make files disappear. Any
suggestions are greatly appreciated.
On Fri, Oct 4, 2013 at 9:40 AM, Mike Broers <mbroers@gmail.com> wrote:
> Hello, our postgresql 9.2.4 qa database (thankfully its just qa) seems to
> be hosed.
>
> Starting at around 3:39am last night I started seeing errors about missing
> files and now I cannot run a pgdump or a vacuum without it complaining
> about files that it cannot find with errors like this: ERROR: could not
> open file "base/125542/12631". When I check the filesystem the files are
> indeed not there. The 1am regular vacuum completed and its log is clean.
> The postgres log is clean before these errors occurred.
>
> Since this is qa we do not perform backups, and the solution if we cannot
> repair the problem will be to create a fresh qa server but I am intrigued
> about how to determine the source of the problem and the extent of the
> problem.
>
> Is there a way to force vacuum to continue on errors or an alternate way
> to help determine all the missing files?
>
> It might be totally unrelated, but yesterday morning on this qa server I
> stopped postgres, and created a symlink to pg_xlog so that it was writing
> to a different volume, and restarted. This was working fine all day so its
> possibly a red herring but I thought I should mention it.
>
> Any advice is appreciated, thanks!
>
>
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: ERROR: could not open file "base/125542/12631" Corruption?
@ 2013-10-04 17:16 Alejandro Brust <alejandrob@pasteleros.org.ar>
parent: Mike Broers <mbroers@gmail.com>
1 sibling, 0 replies; 6+ messages in thread
From: Alejandro Brust @ 2013-10-04 17:16 UTC (permalink / raw)
To: pgsql-admin
El 04/10/2013 14:10, Mike Broers escribió:
> Strange, this is happening in a totally different environment now too.
> The only thing these two environments share is a SAN, but I wouldnt
> think something going on at the SAN level would make files disappear.
> Any suggestions are greatly appreciated.
>
>
> On Fri, Oct 4, 2013 at 9:40 AM, Mike Broers <mbroers@gmail.com
> <mailto:mbroers@gmail.com>> wrote:
>
> Hello, our postgresql 9.2.4 qa database (thankfully its just qa)
> seems to be hosed.
>
> Starting at around 3:39am last night I started seeing errors about
> missing files and now I cannot run a pgdump or a vacuum without it
> complaining about files that it cannot find with errors like this:
> ERROR: could not open file "base/125542/12631". When I check the
> filesystem the files are indeed not there. The 1am regular vacuum
> completed and its log is clean. The postgres log is clean before
> these errors occurred.
>
> Since this is qa we do not perform backups, and the solution if we
> cannot repair the problem will be to create a fresh qa server but
> I am intrigued about how to determine the source of the problem
> and the extent of the problem.
>
> Is there a way to force vacuum to continue on errors or an
> alternate way to help determine all the missing files?
>
> It might be totally unrelated, but yesterday morning on this qa
> server I stopped postgres, and created a symlink to pg_xlog so
> that it was writing to a different volume, and restarted. This
> was working fine all day so its possibly a red herring but I
> thought I should mention it.
>
> Any advice is appreciated, thanks!
>
>
>
Hello, U should try to put the data directory in a local disk (SAN may
be falling). I don't like SAN volumes for databases.
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: ERROR: could not open file "base/125542/12631" Corruption?
@ 2013-10-04 17:51 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Alejandro Brust <alejandrob@pasteleros.org.ar>
0 siblings, 0 replies; 6+ messages in thread
From: Alvaro Herrera @ 2013-10-04 17:51 UTC (permalink / raw)
To: Alejandro Brust <alejandrob@pasteleros.org.ar>; +Cc: pgsql-admin
Alejandro Brust escribió:
> U could try some like "SET/ zero_damaged_pages = on" and perform a
> vacuum-db and may be pg_dump
I don't think this is a good idea. It might cause data loss. In any
case it's unlikely to fix the reported problem.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: ERROR: could not open file "base/125542/12631" Corruption?
@ 2013-10-04 19:14 Mike Broers <mbroers@gmail.com>
parent: Mike Broers <mbroers@gmail.com>
1 sibling, 0 replies; 6+ messages in thread
From: Mike Broers @ 2013-10-04 19:14 UTC (permalink / raw)
To: pgsql-admin
Update - someone unleashed a 'cleanup script' yesterday via puppet to
multiple hosts and greedily deleted files that had not been modified in 15
days. This is the most likely culprit so mystery basically solved.
Thankfully this is in QA, whew! It would be interesting to still know if
there are ways of having postgres check and verify that files it expects to
find are there, and to get an idea on the extent of the damage.
On Fri, Oct 4, 2013 at 12:10 PM, Mike Broers <mbroers@gmail.com> wrote:
> Strange, this is happening in a totally different environment now too.
> The only thing these two environments share is a SAN, but I wouldnt think
> something going on at the SAN level would make files disappear. Any
> suggestions are greatly appreciated.
>
>
> On Fri, Oct 4, 2013 at 9:40 AM, Mike Broers <mbroers@gmail.com> wrote:
>
>> Hello, our postgresql 9.2.4 qa database (thankfully its just qa) seems to
>> be hosed.
>>
>> Starting at around 3:39am last night I started seeing errors about
>> missing files and now I cannot run a pgdump or a vacuum without it
>> complaining about files that it cannot find with errors like this: ERROR:
>> could not open file "base/125542/12631". When I check the filesystem the
>> files are indeed not there. The 1am regular vacuum completed and its log
>> is clean. The postgres log is clean before these errors occurred.
>>
>> Since this is qa we do not perform backups, and the solution if we cannot
>> repair the problem will be to create a fresh qa server but I am intrigued
>> about how to determine the source of the problem and the extent of the
>> problem.
>>
>> Is there a way to force vacuum to continue on errors or an alternate way
>> to help determine all the missing files?
>>
>> It might be totally unrelated, but yesterday morning on this qa server I
>> stopped postgres, and created a symlink to pg_xlog so that it was writing
>> to a different volume, and restarted. This was working fine all day so its
>> possibly a red herring but I thought I should mention it.
>>
>> Any advice is appreciated, thanks!
>>
>>
>>
>
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2013-10-04 19:14 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2013-10-04 14:40 ERROR: could not open file "base/125542/12631" Corruption? Mike Broers <mbroers@gmail.com>
2013-10-04 16:53 ` Alejandro Brust <alejandrob@pasteleros.org.ar>
2013-10-04 17:51 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2013-10-04 17:10 ` Mike Broers <mbroers@gmail.com>
2013-10-04 17:16 ` Alejandro Brust <alejandrob@pasteleros.org.ar>
2013-10-04 19:14 ` Mike Broers <mbroers@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox