Date: Wed, 25 May 2011 22:07:16 +0000
From: John Rouillard <rouilj@renesys.com>
To: pgsql-performance@postgresql.org
Subject: Re: "error with invalid page header" while vacuuming
 pgbench data
Message-ID: <20110525220716.GF31823@renesys.com>
References: <20110523221603.GA31823@renesys.com>
	<4DDA9780020000250003DB62@gw.wicourts.gov>
	<20110525171239.GC31823@renesys.com>
	<4DDD1E1F020000250003DCDA@gw.wicourts.gov>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4DDD1E1F020000250003DCDA@gw.wicourts.gov>
User-Agent: Mutt/1.5.20 (2009-06-14)

On Wed, May 25, 2011 at 03:19:59PM -0500, Kevin Grittner wrote:
> John Rouillard <rouilj@renesys.com> wrote:
> > On Mon, May 23, 2011 at 05:21:04PM -0500, Kevin Grittner wrote:
> >> John Rouillard <rouilj@renesys.com> wrote:
> >>  
> >> >  I seem to be able to provoke this error:
> >> > 
> >> >    vacuum...ERROR:  invalid page header in
> >> >             block 2128910 of relation base/16385/21476
> >>  
> >> What version of PostgreSQL?
> > 
> > Hmm, I thought I replied to this, but I haven't seen it come back
> > to me on list.  It's postgres version: 8.4.5.
> > 
> > rpm -q shows
> > 
> >    postgresql84-server-8.4.5-1.el5_5.1
>  
> I was hoping someone else would jump in, but I see that your
> previous post didn't copy the list, which solves *that* mystery.
>  
> I'm curious whether you might have enabled one of the "it's OK to
> trash my database integrity to boost performance" options.  (People
> with enough replication often feel that this *is* OK.)  Please run
> the query on this page and post the results:
>  
> http://wiki.postgresql.org/wiki/Server_Configuration
>  
> Basically, if fsync or full_page_writes is turned off and there was
> a crash, that explains it.  If not, it provides more information to
> proceed.

Nope. Neither is turned off. I can't run the query at the moment since
the system is in the middle of a memtest86+ check of 96GB of
memory. The relevent parts from the config file from the Configuration
Management system are:

  #fsync = on                             # turns forced synchronization
                                          # on or off
  #synchronous_commit = on                # immediate fsync at commit
  #wal_sync_method = fsync                # the default is the first option 

  #full_page_writes = on                  # recover from partial page writes

this is the same setup I use on all my data warehouse systems (with
minor pgtune type changes based on amount of memory). Running the
query on another system (using ext3, centos 5.5) shows:

 version                        | PostgreSQL 8.4.5 on
x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 20080704 (Red
Hat 4.1.2-48), 64-bit
 archive_command                | if test ! -e
/var/lib/pgsql/data/ARCHIVE_ENABLED; then exit 0; fi; test ! -f
/var/bak/pgsql/%f && cp %p /var/bak/p
gsql/%f
 archive_mode                   | on
 checkpoint_completion_target   | 0.9
 checkpoint_segments            | 64
 constraint_exclusion           | on
 custom_variable_classes        | pg_stat_statements
 default_statistics_target      | 100
 effective_cache_size           | 8GB
 lc_collate                     | en_US.UTF-8
 lc_ctype                       | en_US.UTF-8
 listen_addresses               | *
 log_checkpoints                | on
 log_connections                | on
 log_destination                | stderr,syslog
 log_directory                  | pg_log
 log_filename                   | postgresql-%a.log
 log_line_prefix                | %t %u@%d(%p)i: 
 log_lock_waits                 | on
 log_min_duration_statement     | 2s
 log_min_error_statement        | warning
 log_min_messages               | notice
 log_rotation_age               | 1d
 log_rotation_size              | 0
 log_temp_files                 | 0
 log_truncate_on_rotation       | on
 logging_collector              | on
 maintenance_work_mem           | 1GB
 max_connections                | 300
 max_locks_per_transaction      | 128
 max_stack_depth                | 2MB
 port                           | 5432
 server_encoding                | UTF8
 shared_buffers                 | 4GB
 shared_preload_libraries       | pg_stat_statements
 superuser_reserved_connections | 3
 tcp_keepalives_count           | 0
 tcp_keepalives_idle            | 0
 tcp_keepalives_interval        | 0
 TimeZone                       | UTC
 wal_buffers                    | 32MB
 work_mem                       | 16MB

> You might want to re-start the thread on pgsql-general, though.  Not
> everybody who might be able to help with a problem like this follows
> the performance list.  Or, if you didn't set any of the dangerous
> configuration options, this sounds like a bug -- so pgsql-bugs might
> be even better.

Well I am also managing to panic the kernel on some runs as well.  So
my guess is this is not only a postgres bug (if it's a postgres issue
at all).

As gregg mentioned in another followup ext4 under centos 5.x may be an
issue. I'll drop back to ext3 and see if I can replicate the
corruption or crashes one I rule out some potential hardware issues.

If I can replicate with ext3, then I'll follow up on -general or
-bugs.

Ext4 pgbench results complete faster, but if it's not reliable ....

Thanks for your help.

--
				-- rouilj

John Rouillard       System Administrator
Renesys Corporation  603-244-9084 (cell)  603-643-9300 x 111