pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Chris Hoover <revoohc@sermonaudio.com>
To: pgsql-admin@postgresql.org
Subject: URGENT - HELP WITH 7.3.4 Strangeness
Date: Mon, 22 Nov 2004 13:48:31 -0500
Message-ID: <41A2347F.8030606@sermonaudio.com> (raw)

I am in need of some help.

We are experiencing a lot of very weird strangeness with our Postgresql 
databases.  It appears that we are suddenly suffering from a lot of 
possible index corruption or other db problems.

Here is what is happening:

1.  Last Sunday, ran a reindex on all databases across our 3 postgresql 
servers.  This was done using the reindexdb script from the contrib 
directory.  The reindex was prompted due to strangeness on  db server 1.

2.  Wednesday, our application suddenly starts having data issues.  It 
issues selects to db and does not display the rows.  Nothing on the 
application has changed.  This is happening across all 3 database 
servers.  (App runs on Windows and connects via a linux "app" server 
using linux odbc (from gborg.postgresql.org) to connect to postgres.

3.  Pulling queries from db logs and running them via psql returns rows.

4.  Wednesday night.  Moved all db's from initial problem database 
server to new db server.  Moved another problem db to new server.  New 
db's run fine. 

5.  Wednesday night.  Dumped and reloaded several problem db's on 
problem servers 2 and 3.

6.  Thursday was relatively quiet.

7.  Friday, problems reappeared on servers 2 and 3.

8.  Friday night.  Shutdown, powered off db servers 2 and 3.  Ran fsck 
on all drives.  Ran reindexdb across all db's and vacuum analyzed.

9.  Saturday morning - life is good.

10.  Sunday night, check db's to verify things are good.  DB Server 2 is 
exhibiting problems, db server 3 is fine.

11.  Sunday night - reboot db server 2.  Everything is fine.

12.  Monday (today). - Apparent index corruption on new db server 1.  
Row in db table but query using index will not pull it.  Small database 
so reindexed it.

In synopsis:

System runs fine after reboot, but appears to start having index 
issues.  We are able to query the db and see the data if we force a seq 
scan, but an index scan returns random results.  Nothing has changed 
application or db wise.

All server Dell Poweredge 2650's
RedHat ES 2.1
Postgres 7.3.4  (with perl)
DB's on Dell PowerVaults using raid5

I really need some help solving this.  I have never seen anything like 
this before.  Does anyone have any ideas? 

Thanks,

Chris




view thread (4+ messages)  latest in thread

Message-ID: <41A2347F.8030606@sermonaudio.com>
Permalink:  ../41A2347F.8030606@sermonaudio.com/
Also on:    postgresql.org/message-id/41A2347F.8030606@sermonaudio.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-admin@postgresql.org
  Cc: revoohc@sermonaudio.com
  Subject: Re: URGENT - HELP WITH 7.3.4 Strangeness
  In-Reply-To: <41A2347F.8030606@sermonaudio.com>
  References: <41A2347F.8030606@sermonaudio.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox