Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id 8231E9FB2FD for ; Thu, 5 Oct 2006 13:24:31 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 06980-01-2 for ; Thu, 5 Oct 2006 16:24:16 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey- Received: from mmp3.nems.noaa.gov (mmp3.nems.noaa.gov [140.90.121.158]) by postgresql.org (Postfix) with ESMTP id 698AB9FA3B0 for ; Thu, 5 Oct 2006 13:24:16 -0300 (ADT) Received: from [10.201.6.73] by mmp3.nems.noaa.gov (Sun Java System Messaging Server 6.2-2.05 (built Apr 28 2005)) with ESMTPSA id <0J6O007NM88BYF80@mmp3.nems.noaa.gov> for pgsql-admin@postgresql.org; Thu, 05 Oct 2006 12:24:11 -0400 (EDT) Date: Thu, 05 Oct 2006 12:24:11 -0400 From: Kevin Johnson Subject: Re: Disk space consumed by pk not returned after vacuum or reindex In-reply-to: <20060922170557.GB571@wolff.to> To: Bruno Wolff III , pgsql-admin@postgresql.org, Sean Webb Message-id: <452531AB.2050709@noaa.gov> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_4ubwVCNf2JQju27/p/SZJg)" References: <45080D4C.2020709@noaa.gov> <20060922170557.GB571@wolff.to> User-Agent: Thunderbird 1.5.0.7 (Windows/20060909) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.001 tagged_above=0 required=5 tests=HTML_MESSAGE X-Spam-Level: X-Archive-Number: 200610/30 X-Sequence-Number: 23333 This is a multi-part message in MIME format. --Boundary_(ID_4ubwVCNf2JQju27/p/SZJg) Content-type: text/plain; charset=us-ascii; format=flowed Content-transfer-encoding: 7BIT Thank you for the suggestion, Bruno. The clustering did the trick in reducing the current disk usage, however eventually the disk space get consumed once more. I fear that we may just need to update the version of Postgres to help alleviate index bloat! Bruno Wolff III wrote: > On Wed, Sep 13, 2006 at 09:53:16 -0400, > Kevin Johnson wrote: > >> We have a database, which consistently consumes more and more of the >> disk space in it's lvol until it reaches 100%. So far, we have tried to >> run a full vacuum on the database, with limited success. Eventually, we >> had to drop and reload the database with the same data inside. It >> brought the disk usage down to 73%. It then began to creep once more >> toward 100%. After some research, I was able to use the pg_class >> catalog to find that the items which are expanding the quickest are >> primary key (btree) indexes. I attempted to run a REINDEX on one of the >> tables with the pk taking up the largest amount of space. The usage >> according to pg_class dropped dramatically, however the disk space was >> not returned to the system. So I attempted another full vacuum >> afterwards, and still nothing was returned to the system. These tables >> are updated extremely frequently (although their total number of rows is >> close to constant), which is my guess as to why the pk indexes increase >> so rapidly in terms of their disk usage. Unfortunately, PostgreSQL >> knowledge is limited, and I was wondering if anyone had experienced >> something similar / knows what else we can do to return this disk space >> back to the system? >> > > This is possibly index bloat due to new keys always being larger than existing > keys. This was fixed in later releases. There is still some potential for > bloat due to fragmentation, but I believe that has a constant bound. > You might try using the cluster command. I think that will both clean up > the indexes and remove dead rows and do it faster than using a vacuum full > and reindexing. The downside is that the table will be unavailable during > the cluster which might be a deal breaker for you. > -- Kevin Johnson Raytheon AWIPS Sr. Systems Engineer NWS Network Control Facility p: 301.713.9362x325 f: 301.713.1905 --Boundary_(ID_4ubwVCNf2JQju27/p/SZJg) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT Thank you for the suggestion, Bruno.  The clustering did the trick in reducing the current disk usage, however eventually the disk space get consumed once more.  I fear that we may just need to update the version of Postgres to help alleviate index bloat!


Bruno Wolff III wrote:
On Wed, Sep 13, 2006 at 09:53:16 -0400,
  Kevin Johnson <Kevin.Johnson@noaa.gov> wrote:
  
We have a database, which consistently consumes more and more of the 
disk space in it's lvol until it reaches 100%.  So far, we have tried to 
run a full vacuum on the database, with limited success.  Eventually, we 
had to drop and reload the database with the same data inside.  It 
brought the disk usage down to 73%.  It then began to creep once more 
toward 100%.  After some research, I was able to use the pg_class 
catalog to find that the items which are expanding the quickest are 
primary key (btree) indexes.  I attempted to run a REINDEX on one of the 
tables with the pk taking up the largest amount of space.  The usage 
according to pg_class dropped dramatically, however the disk space was 
not returned to the system.  So I attempted another full vacuum 
afterwards, and still nothing was returned to the system.  These tables 
are updated extremely frequently (although their total number of rows is 
close to constant), which is my guess as to why the pk indexes increase 
so rapidly in terms of their disk usage.  Unfortunately, PostgreSQL 
knowledge is limited, and I was wondering if anyone had experienced 
something similar / knows what else we can do to return this disk space 
back to the system?
    

This is possibly index bloat due to new keys always being larger than existing
keys. This was fixed in later releases. There is still some potential for
bloat due to fragmentation, but I believe that has a constant bound.
You might try using the cluster command. I think that will both clean up
the indexes and remove dead rows and do it faster than using a vacuum full
and reindexing. The downside is that the table will be unavailable during
the cluster which might be a deal breaker for you.
  

-- 
Kevin Johnson
Raytheon
AWIPS Sr. Systems Engineer
NWS Network Control Facility
p: 301.713.9362x325
f: 301.713.1905
--Boundary_(ID_4ubwVCNf2JQju27/p/SZJg)--