Message-ID: <524EF7EA.5010905@pasteleros.org.ar>
Date: Fri, 04 Oct 2013 14:16:26 -0300
From: Alejandro Brust <alejandrob@pasteleros.org.ar>
Reply-To: alejandrob@pasteleros.org.ar
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:17.0) Gecko/20130503 Thunderbird/17.0.5
MIME-Version: 1.0
To: pgsql-admin@postgresql.org
Subject: Re: ERROR: could not open file "base/125542/12631" Corruption?
References: 
 <CAB9893jR-_zaRxeRc0b4rapWLa-M-=Z-vSKt-vxD5uOhFs-z9w@mail.gmail.com>
 <CAB9893hALtLpUaz6tCCO3C0uj1icXcDZt0oBdRFFYzbwV1iGSQ@mail.gmail.com>
In-Reply-To: 
 <CAB9893hALtLpUaz6tCCO3C0uj1icXcDZt0oBdRFFYzbwV1iGSQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------040907010003020705080206"
Precedence: bulk
Sender: pgsql-admin-owner@postgresql.org

This is a multi-part message in MIME format.
--------------040907010003020705080206
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit


El 04/10/2013 14:10, Mike Broers escribió:
> Strange, this is happening in a totally different environment now too.
>  The only thing these two environments share is a SAN, but I wouldnt
> think something going on at the SAN level would make files disappear.
>  Any suggestions are greatly appreciated.
>
>
> On Fri, Oct 4, 2013 at 9:40 AM, Mike Broers <mbroers@gmail.com
> <mailto:mbroers@gmail.com>> wrote:
>
>     Hello, our postgresql 9.2.4 qa database (thankfully its just qa)
>     seems to be hosed.  
>
>     Starting at around 3:39am last night I started seeing errors about
>     missing files and now I cannot run a pgdump or a vacuum without it
>     complaining about files that it cannot find with errors like this:
>     ERROR: could not open file "base/125542/12631". When I check the
>     filesystem the files are indeed not there.  The 1am regular vacuum
>     completed and its log is clean.  The postgres log is clean before
>     these errors occurred.
>
>     Since this is qa we do not perform backups, and the solution if we
>     cannot repair the problem will be to create a fresh qa server but
>     I am intrigued about how to determine the source of the problem
>     and the extent of the problem.  
>
>     Is there a way to force vacuum to continue on errors or an
>     alternate way to help determine all the missing files?  
>
>     It might be totally unrelated, but yesterday morning on this qa
>     server I stopped postgres, and created a symlink to pg_xlog so
>     that it was writing to a different volume, and restarted.  This
>     was working fine all day so its possibly a red herring but I
>     thought I should mention it.
>
>     Any advice is appreciated, thanks!  
>
>
>
Hello, U should try to put the data directory in a local disk (SAN may
be falling). I don't like SAN volumes for databases.


--------------040907010003020705080206
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <br>
    <div class="moz-cite-prefix">El 04/10/2013 14:10, Mike Broers
      escribi&oacute;:<br>
    </div>
    <blockquote
cite="mid:CAB9893hALtLpUaz6tCCO3C0uj1icXcDZt0oBdRFFYzbwV1iGSQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">Strange, this is happening in a totally different
        environment now too. &nbsp;The only thing these two environments
        share is a SAN, but I wouldnt think something going on at the
        SAN level would make files disappear. &nbsp;Any suggestions are
        greatly appreciated.</div>
      <div class="gmail_extra"><br>
        <br>
        <div class="gmail_quote">On Fri, Oct 4, 2013 at 9:40 AM, Mike
          Broers <span dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:mbroers@gmail.com" target="_blank">mbroers@gmail.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">Hello, our postgresql 9.2.4 qa database
              (thankfully its just qa) seems to be hosed. &nbsp;
              <div><br>
              </div>
              <div>Starting at around 3:39am last night I started seeing
                errors about missing files and now I cannot run a pgdump
                or a vacuum without it complaining about files that it
                cannot find with errors like this: ERROR: could not open
                file "base/125542/12631". When I check the filesystem
                the files are indeed not there. &nbsp;The 1am regular vacuum
                completed and its log is clean. &nbsp;The postgres log is
                clean before these errors occurred.
                <div>
                  <br>
                </div>
                <div>Since this is qa we do not perform backups, and the
                  solution if we cannot repair the problem will be to
                  create a fresh qa server but I am intrigued about how
                  to determine the source of the problem and the extent
                  of the problem. &nbsp;</div>
                <div><br>
                </div>
                <div>Is there a way to force vacuum to continue on
                  errors or an alternate way to help determine all the
                  missing files? &nbsp;</div>
                <div><br>
                </div>
                <div>It might be totally unrelated, but yesterday
                  morning on this qa server I stopped postgres, and
                  created a symlink to pg_xlog so that it was writing to
                  a different volume, and restarted. &nbsp;This was working
                  fine all day so its possibly a red herring but I
                  thought I should mention it.</div>
                <div><br>
                </div>
                <div>Any advice is appreciated, thanks! &nbsp;</div>
                <div><br>
                </div>
                <div><br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    Hello, U should try to put the data directory in a local disk (SAN
    may be falling). I don't like SAN volumes for databases.<br>
    <br>
    <br>
  </body>
</html>

--------------040907010003020705080206--