From: Steve Holdoway <steve.holdoway@firetrust.com>
To: pgsql-admin@postgresql.org
Subject: Re: what happend to my database
Date: Thu, 29 May 2008 13:52:35 +1200
Message-ID: <20080529135235.d6fcad87.steve.holdoway@firetrust.com> (raw)
In-Reply-To: <8078a1730805281837y1ed331bey4e7db16a63423cfa@mail.gmail.com>
References: <8078a1730805281837y1ed331bey4e7db16a63423cfa@mail.gmail.com>
On Wed, 28 May 2008 18:37:06 -0700
"Medi Montaseri" <montaseri@gmail.com> wrote:
> Hi,
>
> I am faced with a database disapperance and seeking some explanations
> outside of gremlins.
> I had a database running at
>
> cat /etc/sysconfig/pgsl/postmaster
> PGDATA=/qmsvol/pg_8.1.9/data
> PGLOG=/var/log/pgsql/pgstartup.log
>
> Where /qmsvol is an iSCSI block device
> A couple of days ago, my server was rebooted and by the time I got to it my
> database was deleted, gone, zapped, not there any more.
>
> I looked at my pgstartup.log where I see the following....
>
> postmaster cannot access the server configuration file
> "/qmsvol/pg_8.1.9/postgresql.conf": Permission denied
> over 17 times and then following by...
> The database cluster will be initialized with locale en_US.UTF-8.
>
> I think the following happend...
> Since my PGDATA was on an iSCSI device, by the time /etc/rc3.d/S64postgresql
> was executed, the device below it was not available.....question...why the
> error says "permission denied" vs "file not found". In the meantime, pg_ctl
> kept trying and finally concluded that the data directory is blank, and
> hence this must be a out-of-box case and he is good to initdb the PGDATA and
> as it called initdb to do the job... the iSCSI volume below it came online
> and by then the bomb had already been dropped.
>
> Now I need to find some facts to support this...
When you mount a partition on linux, it does this by overlaying it's root directory with the existing one on the parent volume. Ownerships and permissions are also replaced. I expect that the /qmsvol directory will be owned by root, with fairly restrictive access rights. This will not be the case the root ( . ) directory on the external device, which will be postgres-friendly.
> Where else can I look for forensics
I don't think you need any more! To fix this, I'd do 2 things. First, start postgres much later in the boot sequence:
cd /etc/rc3.d ; mv S64postgresql S99postgresql
( and the same in rc5.d if you're using a gui at all ).
and do the converse to whichever script mounts your external devices. Also add in a test that the device is mounted in the start) block of /etc/init.d/postgresql... something simple like
while [ ! -d /qmsvol/pg_8.1.9/data ]
do
sleep 5
done
( well, something that can't hang forever would be preferable! ).
>
> Thanks
> Medi
>
hth,
Steve
--
Steve Holdoway <steve.holdoway@firetrust.com>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: steve.holdoway@firetrust.com
Subject: Re: what happend to my database
In-Reply-To: <20080529135235.d6fcad87.steve.holdoway@firetrust.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox