pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
destroyed db/index (corruption)
4+ messages / 2 participants
[nested] [flat]

* destroyed db/index (corruption)
@ 2007-02-05 14:00  Michael Monnerie <michael.monnerie@it-management.at>
  0 siblings, 1 reply; 4+ messages in thread

From: Michael Monnerie @ 2007-02-05 14:00 UTC (permalink / raw)
  To: pgsql-admin

dear list,

I've found database corruptions occuring with openSUSE 10.1, postgreSQL 
version # rpm -qa|grep postg
postgresql-libs-8.1.4-1.2
postgresql-server-8.1.4-1.2
postgresql-8.1.4-1.2

I found this in database logs:
2007-02-05 13:15:51 CET DB=bayes_pg3 HOST=195.202.170.130(54789) 
SESSTRT=2007-02-04 05:01:15 CET FEHLER:  duplizierter Schlüssel 
verletzt Unique-Constraint »bayes_seen_pkey«

translated it says "duplicate key violates unique constraint", and it's 
true, I have 2 rows with the same Primary Key in that table (make dump, 
then reload fails on it).

The DB is a bayes DB for spamassassin. I've had this error before, and 
dumped/edited dump/reloaded the db. But it keeps coming again. So there 
might be hardware or software problems.

The hardware is a server running one AMD Opteron dual core 2210, with 
8GB RAM, and VMware. We have www, db, mailserver and mailrelays running 
within a VM, and the only problem I see is with the db, so it doesn't 
look like a hardware problem.

I could believe it's a software problem. But which? Could a bad config 
cause such problems? I've set this in postgresql.conf:

listen_addresses = '*'
port = 5432
max_connections = 200
shared_buffers = 50000                  
max_prepared_transactions = 0           
work_mem = 128000         
maintenance_work_mem = 128000   
max_fsm_pages = 60000                   
max_fsm_relations = 1000                
max_files_per_process = 5000            
vacuum_cost_delay = 100                 
vacuum_cost_page_hit = 1                
vacuum_cost_page_miss = 10              
vacuum_cost_page_dirty = 20             
vacuum_cost_limit = 10000               
bgwriter_delay = 1000                   
bgwriter_lru_percent = 5.0              
bgwriter_lru_maxpages = 100     
bgwriter_all_percent = 1.333    
bgwriter_all_maxpages = 100     
fsync = off                             
wal_buffers = 8                 
commit_delay = 3000                     
commit_siblings = 25                    
checkpoint_segments = 4         
checkpoint_timeout = 300                
checkpoint_warning = 30         
enable_bitmapscan = on
enable_hashagg = on
enable_hashjoin = on
enable_indexscan = on
enable_mergejoin = on
enable_nestloop = on
enable_seqscan = on
enable_sort = on
enable_tidscan = on
effective_cache_size = 64000            
geqo = on
log_destination = 'stderr'              
redirect_stderr = on                    
log_directory = '/var/log/'             
log_filename = 'postgresql.%Y' 
log_truncate_on_rotation = off 
log_rotation_age = 0                    
log_rotation_size = 0           
log_min_duration_statement = 5000       
log_connections = off
log_disconnections = off
log_duration = off
log_line_prefix = '%t DB=%d HOST=%r SESSTRT=%s '                        
log_statement = 'ddl'                   
stats_start_collector = on
stats_block_level = on
stats_row_level = on
autovacuum = on                         
autovacuum_naptime = 3600               
autovacuum_vacuum_threshold = 5000      
autovacuum_analyze_threshold = 2500     
autovacuum_vacuum_scale_factor = 0.4    
autovacuum_analyze_scale_factor = 0.2   
autovacuum_vacuum_cost_delay = -1       
autovacuum_vacuum_cost_limit = -1       
lc_messages = 'de_DE.UTF-8'                     
lc_monetary = 'de_DE.UTF-8'                     
lc_numeric = 'de_DE.UTF-8'                      
lc_time = 'de_DE.UTF-8'                   

log_filename = 'postgresql.%Y' 

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:        "curl -s http://zmi.at/zmi4.asc | gpg --import"
// Fingerprint: EA39 8918 EDFF 0A68 ACFB  11B7 BA2D 060F 1C6F E6B0
// Keyserver: www.keyserver.net                   Key-ID: 1C6FE6B0

^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: destroyed db/index (corruption)
@ 2007-02-05 14:22  Achilleas Mantzios <achill@matrix.gatewaynet.com>
  parent: Michael Monnerie <michael.monnerie@it-management.at>
  0 siblings, 1 reply; 4+ messages in thread

From: Achilleas Mantzios @ 2007-02-05 14:22 UTC (permalink / raw)
  To: pgsql-admin

Στις Δευτέρα 05 Φεβρουάριος 2007 16:00, ο/η Michael Monnerie έγραψε:
> dear list,
>
> I've found database corruptions occuring with openSUSE 10.1, postgreSQL
> version # rpm -qa|grep postg
> postgresql-libs-8.1.4-1.2
> postgresql-server-8.1.4-1.2
> postgresql-8.1.4-1.2
>
> I found this in database logs:
> 2007-02-05 13:15:51 CET DB=bayes_pg3 HOST=195.202.170.130(54789)
> SESSTRT=2007-02-04 05:01:15 CET FEHLER:  duplizierter Schlüssel
> verletzt Unique-Constraint »bayes_seen_pkey«
>
> translated it says "duplicate key violates unique constraint", and it's
> true, I have 2 rows with the same Primary Key in that table (make dump,
> then reload fails on it).

This ERROR is normally thrown when you *try* to violate a unique constraint.
Can you demonstrate the presence of the UNIQUE constraint on your table
and the pair of identical key values rows?
Post a description (\d) of your table, and then the select stmt which return
the 2 bad rows.

I might be wrong, but i would bet the whole issue began because
some rows were inserted by a client (other than the reloading psql) during 
reload but before the UNIQUE KEY DDL
was executed, or smth like that, or simply because this bayes_seen value is 
attempted to be inserted again
(for reasons beyond our scope)
>
> The DB is a bayes DB for spamassassin. I've had this error before, and
> dumped/edited dump/reloaded the db. But it keeps coming again. So there
> might be hardware or software problems.
>
> The hardware is a server running one AMD Opteron dual core 2210, with
> 8GB RAM, and VMware. We have www, db, mailserver and mailrelays running
> within a VM, and the only problem I see is with the db, so it doesn't
> look like a hardware problem.
>
> I could believe it's a software problem. But which? Could a bad config
> cause such problems? I've set this in postgresql.conf:
>
> listen_addresses = '*'
> port = 5432
> max_connections = 200
> shared_buffers = 50000
> max_prepared_transactions = 0
> work_mem = 128000
> maintenance_work_mem = 128000
> max_fsm_pages = 60000
> max_fsm_relations = 1000
> max_files_per_process = 5000
> vacuum_cost_delay = 100
> vacuum_cost_page_hit = 1
> vacuum_cost_page_miss = 10
> vacuum_cost_page_dirty = 20
> vacuum_cost_limit = 10000
> bgwriter_delay = 1000
> bgwriter_lru_percent = 5.0
> bgwriter_lru_maxpages = 100
> bgwriter_all_percent = 1.333
> bgwriter_all_maxpages = 100
> fsync = off
> wal_buffers = 8
> commit_delay = 3000
> commit_siblings = 25
> checkpoint_segments = 4
> checkpoint_timeout = 300
> checkpoint_warning = 30
> enable_bitmapscan = on
> enable_hashagg = on
> enable_hashjoin = on
> enable_indexscan = on
> enable_mergejoin = on
> enable_nestloop = on
> enable_seqscan = on
> enable_sort = on
> enable_tidscan = on
> effective_cache_size = 64000
> geqo = on
> log_destination = 'stderr'
> redirect_stderr = on
> log_directory = '/var/log/'
> log_filename = 'postgresql.%Y'
> log_truncate_on_rotation = off
> log_rotation_age = 0
> log_rotation_size = 0
> log_min_duration_statement = 5000
> log_connections = off
> log_disconnections = off
> log_duration = off
> log_line_prefix = '%t DB=%d HOST=%r SESSTRT=%s '
> log_statement = 'ddl'
> stats_start_collector = on
> stats_block_level = on
> stats_row_level = on
> autovacuum = on
> autovacuum_naptime = 3600
> autovacuum_vacuum_threshold = 5000
> autovacuum_analyze_threshold = 2500
> autovacuum_vacuum_scale_factor = 0.4
> autovacuum_analyze_scale_factor = 0.2
> autovacuum_vacuum_cost_delay = -1
> autovacuum_vacuum_cost_limit = -1
> lc_messages = 'de_DE.UTF-8'
> lc_monetary = 'de_DE.UTF-8'
> lc_numeric = 'de_DE.UTF-8'
> lc_time = 'de_DE.UTF-8'
>
> log_filename = 'postgresql.%Y'
>
> mfg zmi

-- 
Achilleas Mantzios



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: destroyed db/index (corruption)
@ 2007-02-05 21:36  Michael Monnerie <michael.monnerie@it-management.at>
  parent: Achilleas Mantzios <achill@matrix.gatewaynet.com>
  0 siblings, 1 reply; 4+ messages in thread

From: Michael Monnerie @ 2007-02-05 21:36 UTC (permalink / raw)
  To: pgsql-admin

On Montag, 5. Februar 2007 15:22 Achilleas Mantzios wrote:
> This ERROR is normally thrown when you *try* to violate a unique
> constraint. Can you demonstrate the presence of the UNIQUE constraint
> on your table and the pair of identical key values rows?
> Post a description (\d) of your table, and then the select stmt which
> return the 2 bad rows.

Kalispera Achilleas,

I will do this soon. Just now I'm upgrading that VM from SUSE 10.1 to 
10.2, so later this night I'll reload the databases. 

> I might be wrong, but i would bet the whole issue began because
> some rows were inserted by a client (other than the reloading psql)
> during reload but before the UNIQUE KEY DDL
> was executed, or smth like that, or simply because this bayes_seen
> value is attempted to be inserted again
> (for reasons beyond our scope)

No, I started those DBs fresh with the command

pg_restore -v pg.fulldump.bayes_power2u.sqlz | \
psql -U postgres bayes_pg3 -f -

Is there an easy way to find out which key is duplicated while I have 
that DB running? The log doesn't show it, and the last time I did 
reload with trial-and-error:
- start reload
- it stops somewhere complaining about line 34587328 or similiar
- make some tail & head foo, or vi, to delete the one duplicate line
- start again...

It would be nice to have a simpler process. Could it be that the stored 
procedure could cause a problem? I took all that from the SpamAssassin 
Wiki, without modification.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0676/846 914 666                      .network.your.ideas.
// PGP Key:        "curl -s http://zmi.at/zmi4.asc | gpg --import"
// Fingerprint: EA39 8918 EDFF 0A68 ACFB  11B7 BA2D 060F 1C6F E6B0
// Keyserver: www.keyserver.net                   Key-ID: 1C6FE6B0

^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: destroyed db/index (corruption)
@ 2007-02-06 07:46  Achilleas Mantzios <achill@matrix.gatewaynet.com>
  parent: Michael Monnerie <michael.monnerie@it-management.at>
  0 siblings, 0 replies; 4+ messages in thread

From: Achilleas Mantzios @ 2007-02-06 07:46 UTC (permalink / raw)
  To: pgsql-admin

Στις Δευτέρα 05 Φεβρουάριος 2007 23:36, ο/η Michael Monnerie έγραψε:
> On Montag, 5. Februar 2007 15:22 Achilleas Mantzios wrote:
> > This ERROR is normally thrown when you *try* to violate a unique
> > constraint. Can you demonstrate the presence of the UNIQUE constraint
> > on your table and the pair of identical key values rows?
> > Post a description (\d) of your table, and then the select stmt which
> > return the 2 bad rows.
>
> Kalispera Achilleas,
:)
>
> I will do this soon. Just now I'm upgrading that VM from SUSE 10.1 to
> 10.2, so later this night I'll reload the databases.
>
> > I might be wrong, but i would bet the whole issue began because
> > some rows were inserted by a client (other than the reloading psql)
> > during reload but before the UNIQUE KEY DDL
> > was executed, or smth like that, or simply because this bayes_seen
> > value is attempted to be inserted again
> > (for reasons beyond our scope)
>
> No, I started those DBs fresh with the command
>
> pg_restore -v pg.fulldump.bayes_power2u.sqlz | \
> psql -U postgres bayes_pg3 -f -
>

Thats what i mean, postmaster is alive, and accepting connections
during the above command, so what would prevent a 2nd client (from your mail 
server setup most likely)
from inserting a value that is later present in the dump,
which is later attempted to be COPY'ed to the table
and ofcource that violates the PRIMARY KEY CONSTRAINT?
In other words, do you have full control of your DB when that happens?
If some 2nd client, not controled by you, inserts a value that is later
found in the dump, it is completely normal that this ERROR will be thrown.
Have you tried, for instance, disabling all unwanted access to postmaster,
e.g. through data/pg_hba.conf ?

> Is there an easy way to find out which key is duplicated while I have
> that DB running? The log doesn't show it, and the last time I did
> reload with trial-and-error:
> - start reload
> - it stops somewhere complaining about line 34587328 or similiar
> - make some tail & head foo, or vi, to delete the one duplicate line
> - start again...
>

The "steps" you mentioned earlier are not what the average admin whould do.
For the average admin that would trigger some kind of panic mode!
It is simply not normal to have this ERROR produced by the dump alone.
Try to isolate your DB, by controling access to it, and try to replicate
the problem (if any).

> It would be nice to have a simpler process. Could it be that the stored
> procedure could cause a problem? I took all that from the SpamAssassin
> Wiki, without modification.
>
> mfg zmi

-- 
Achilleas Mantzios



^ permalink  raw  reply  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2007-02-06 07:46 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2007-02-05 14:00 destroyed db/index (corruption) Michael Monnerie <michael.monnerie@it-management.at>
2007-02-05 14:22 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2007-02-05 21:36   ` Michael Monnerie <michael.monnerie@it-management.at>
2007-02-06 07:46     ` Achilleas Mantzios <achill@matrix.gatewaynet.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox