From Kevin.Johnson@noaa.gov Wed Sep 13 13:53:28 2006 X-Original-To: pgsql-admin-postgresql.org@postgresql.org Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id 62FF09FA4B4 for ; Wed, 13 Sep 2006 10:53:28 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 07616-08 for ; Wed, 13 Sep 2006 13:53:24 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey- Received: from mmp2.nems.noaa.gov (mmp2.nems.noaa.gov [140.90.121.157]) by postgresql.org (Postfix) with ESMTP id 80B589FA380 for ; Wed, 13 Sep 2006 10:53:24 -0300 (ADT) Received: from [192.168.2.2] (ip68-227-201-228.dc.dc.cox.net [68.227.201.228]) by mmp2.nems.noaa.gov (iPlanet Messaging Server 5.2 HotFix 2.01 (built Aug 26 2004)) with ESMTPSA id <0J5J007GUAKVJZ@mmp2.nems.noaa.gov> for pgsql-admin@postgresql.org; Wed, 13 Sep 2006 09:53:19 -0400 (EDT) Date: Wed, 13 Sep 2006 09:53:16 -0400 From: Kevin Johnson Subject: Disk space consumed by pk not returned after vacuum or reindex To: pgsql-admin@postgresql.org Cc: Sean Webb Message-id: <45080D4C.2020709@noaa.gov> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_BMCzPKqpq5COdRjZiAiLRQ)" User-Agent: Thunderbird 1.5.0.5 (Windows/20060719) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.001 tagged_above=0 required=5 tests=HTML_MESSAGE X-Spam-Level: X-Archive-Number: 200609/107 X-Sequence-Number: 23021 This is a multi-part message in MIME format. --Boundary_(ID_BMCzPKqpq5COdRjZiAiLRQ) Content-type: multipart/alternative; boundary="Boundary_(ID_779RKC+wyBai/V0W0cpm+A)" --Boundary_(ID_779RKC+wyBai/V0W0cpm+A) Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT Hello, I am running into a problem on a RHEL3 systems, running PostgreSQL 7.4 We have a database, which consistently consumes more and more of the disk space in it's lvol until it reaches 100%. So far, we have tried to run a full vacuum on the database, with limited success. Eventually, we had to drop and reload the database with the same data inside. It brought the disk usage down to 73%. It then began to creep once more toward 100%. After some research, I was able to use the pg_class catalog to find that the items which are expanding the quickest are primary key (btree) indexes. I attempted to run a REINDEX on one of the tables with the pk taking up the largest amount of space. The usage according to pg_class dropped dramatically, however the disk space was not returned to the system. So I attempted another full vacuum afterwards, and still nothing was returned to the system. These tables are updated extremely frequently (although their total number of rows is close to constant), which is my guess as to why the pk indexes increase so rapidly in terms of their disk usage. Unfortunately, PostgreSQL knowledge is limited, and I was wondering if anyone had experienced something similar / knows what else we can do to return this disk space back to the system? Thank you in advance for any/all help! Kevin --Boundary_(ID_779RKC+wyBai/V0W0cpm+A) Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: 7BIT Hello,

I am running into a problem on a RHEL3 systems, running PostgreSQL 7.4

We have a database, which consistently consumes more and more of the disk space in it's lvol until it reaches 100%.  So far, we have tried to run a full vacuum on the database, with limited success.  Eventually, we had to drop and reload the database with the same data inside.  It brought the disk usage down to 73%.  It then began to creep once more toward 100%.  After some research, I was able to use the pg_class catalog to find that the items which are expanding the quickest are primary key (btree) indexes.  I attempted to run a REINDEX on one of the tables with the pk taking up the largest amount of space.  The usage according to pg_class dropped dramatically, however the disk space was not returned to the system.  So I attempted another full vacuum afterwards, and still nothing was returned to the system.  These tables are updated extremely frequently (although their total number of rows is close to constant), which is my guess as to why the pk indexes increase so rapidly in terms of their disk usage.  Unfortunately, PostgreSQL knowledge is limited, and I was wondering if anyone had experienced something similar / knows what else we can do to return this disk space back to the system?

Thank you in advance for any/all help!

Kevin

--Boundary_(ID_779RKC+wyBai/V0W0cpm+A)-- --Boundary_(ID_BMCzPKqpq5COdRjZiAiLRQ) Content-type: text/x-vcard; charset=utf-8; name=Kevin.Johnson.vcf Content-transfer-encoding: 7BIT Content-disposition: attachment; filename=Kevin.Johnson.vcf begin:vcard fn:Kevin Johnson n:Johnson;Kevin org:Raytheon AWIPS Team adr:SMCC II;;;1325 East West Highway;Silver Spring;MD;20910 email;internet:Kevin.Johnson@noaa.gov title:Sr. Systems Engineer tel;work:301.713.9362 x325 tel;cell:301.787.0648 x-mozilla-html:TRUE version:2.1 end:vcard --Boundary_(ID_BMCzPKqpq5COdRjZiAiLRQ)-- From bruno@wolff.to Fri Sep 22 17:23:08 2006 X-Original-To: pgsql-admin-postgresql.org@postgresql.org Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id 55F2F9FB5C4 for ; Fri, 22 Sep 2006 14:23:08 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 43154-05-2 for ; Fri, 22 Sep 2006 17:21:11 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey- Received: from wolff.to (wolff.to [66.93.197.194]) by postgresql.org (Postfix) with SMTP id B05F79FB4A0 for ; Fri, 22 Sep 2006 13:56:19 -0300 (ADT) Received: (qmail 2963 invoked by uid 500); 22 Sep 2006 17:05:58 -0000 Date: Fri, 22 Sep 2006 12:05:57 -0500 From: Bruno Wolff III To: Kevin Johnson Cc: pgsql-admin@postgresql.org, Sean Webb Subject: Re: Disk space consumed by pk not returned after vacuum or reindex Message-ID: <20060922170557.GB571@wolff.to> Mail-Followup-To: Bruno Wolff III , Kevin Johnson , pgsql-admin@postgresql.org, Sean Webb References: <45080D4C.2020709@noaa.gov> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <45080D4C.2020709@noaa.gov> User-Agent: Mutt/1.4.2.1i X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests= X-Spam-Level: X-Archive-Number: 200609/246 X-Sequence-Number: 23160 On Wed, Sep 13, 2006 at 09:53:16 -0400, Kevin Johnson wrote: > > We have a database, which consistently consumes more and more of the > disk space in it's lvol until it reaches 100%. So far, we have tried to > run a full vacuum on the database, with limited success. Eventually, we > had to drop and reload the database with the same data inside. It > brought the disk usage down to 73%. It then began to creep once more > toward 100%. After some research, I was able to use the pg_class > catalog to find that the items which are expanding the quickest are > primary key (btree) indexes. I attempted to run a REINDEX on one of the > tables with the pk taking up the largest amount of space. The usage > according to pg_class dropped dramatically, however the disk space was > not returned to the system. So I attempted another full vacuum > afterwards, and still nothing was returned to the system. These tables > are updated extremely frequently (although their total number of rows is > close to constant), which is my guess as to why the pk indexes increase > so rapidly in terms of their disk usage. Unfortunately, PostgreSQL > knowledge is limited, and I was wondering if anyone had experienced > something similar / knows what else we can do to return this disk space > back to the system? This is possibly index bloat due to new keys always being larger than existing keys. This was fixed in later releases. There is still some potential for bloat due to fragmentation, but I believe that has a constant bound. You might try using the cluster command. I think that will both clean up the indexes and remove dead rows and do it faster than using a vacuum full and reindexing. The downside is that the table will be unavailable during the cluster which might be a deal breaker for you. From Kevin.Johnson@noaa.gov Thu Oct 5 16:24:31 2006 Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id 8231E9FB2FD for ; Thu, 5 Oct 2006 13:24:31 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 06980-01-2 for ; Thu, 5 Oct 2006 16:24:16 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey- Received: from mmp3.nems.noaa.gov (mmp3.nems.noaa.gov [140.90.121.158]) by postgresql.org (Postfix) with ESMTP id 698AB9FA3B0 for ; Thu, 5 Oct 2006 13:24:16 -0300 (ADT) Received: from [10.201.6.73] by mmp3.nems.noaa.gov (Sun Java System Messaging Server 6.2-2.05 (built Apr 28 2005)) with ESMTPSA id <0J6O007NM88BYF80@mmp3.nems.noaa.gov> for pgsql-admin@postgresql.org; Thu, 05 Oct 2006 12:24:11 -0400 (EDT) Date: Thu, 05 Oct 2006 12:24:11 -0400 From: Kevin Johnson Subject: Re: Disk space consumed by pk not returned after vacuum or reindex In-reply-to: <20060922170557.GB571@wolff.to> To: Bruno Wolff III , pgsql-admin@postgresql.org, Sean Webb Message-id: <452531AB.2050709@noaa.gov> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_4ubwVCNf2JQju27/p/SZJg)" References: <45080D4C.2020709@noaa.gov> <20060922170557.GB571@wolff.to> User-Agent: Thunderbird 1.5.0.7 (Windows/20060909) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.001 tagged_above=0 required=5 tests=HTML_MESSAGE X-Spam-Level: X-Archive-Number: 200610/30 X-Sequence-Number: 23333 This is a multi-part message in MIME format. --Boundary_(ID_4ubwVCNf2JQju27/p/SZJg) Content-type: text/plain; charset=us-ascii; format=flowed Content-transfer-encoding: 7BIT Thank you for the suggestion, Bruno. The clustering did the trick in reducing the current disk usage, however eventually the disk space get consumed once more. I fear that we may just need to update the version of Postgres to help alleviate index bloat! Bruno Wolff III wrote: > On Wed, Sep 13, 2006 at 09:53:16 -0400, > Kevin Johnson wrote: > >> We have a database, which consistently consumes more and more of the >> disk space in it's lvol until it reaches 100%. So far, we have tried to >> run a full vacuum on the database, with limited success. Eventually, we >> had to drop and reload the database with the same data inside. It >> brought the disk usage down to 73%. It then began to creep once more >> toward 100%. After some research, I was able to use the pg_class >> catalog to find that the items which are expanding the quickest are >> primary key (btree) indexes. I attempted to run a REINDEX on one of the >> tables with the pk taking up the largest amount of space. The usage >> according to pg_class dropped dramatically, however the disk space was >> not returned to the system. So I attempted another full vacuum >> afterwards, and still nothing was returned to the system. These tables >> are updated extremely frequently (although their total number of rows is >> close to constant), which is my guess as to why the pk indexes increase >> so rapidly in terms of their disk usage. Unfortunately, PostgreSQL >> knowledge is limited, and I was wondering if anyone had experienced >> something similar / knows what else we can do to return this disk space >> back to the system? >> > > This is possibly index bloat due to new keys always being larger than existing > keys. This was fixed in later releases. There is still some potential for > bloat due to fragmentation, but I believe that has a constant bound. > You might try using the cluster command. I think that will both clean up > the indexes and remove dead rows and do it faster than using a vacuum full > and reindexing. The downside is that the table will be unavailable during > the cluster which might be a deal breaker for you. > -- Kevin Johnson Raytheon AWIPS Sr. Systems Engineer NWS Network Control Facility p: 301.713.9362x325 f: 301.713.1905 --Boundary_(ID_4ubwVCNf2JQju27/p/SZJg) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT Thank you for the suggestion, Bruno.  The clustering did the trick in reducing the current disk usage, however eventually the disk space get consumed once more.  I fear that we may just need to update the version of Postgres to help alleviate index bloat!


Bruno Wolff III wrote:
On Wed, Sep 13, 2006 at 09:53:16 -0400,
  Kevin Johnson <Kevin.Johnson@noaa.gov> wrote:
  
We have a database, which consistently consumes more and more of the 
disk space in it's lvol until it reaches 100%.  So far, we have tried to 
run a full vacuum on the database, with limited success.  Eventually, we 
had to drop and reload the database with the same data inside.  It 
brought the disk usage down to 73%.  It then began to creep once more 
toward 100%.  After some research, I was able to use the pg_class 
catalog to find that the items which are expanding the quickest are 
primary key (btree) indexes.  I attempted to run a REINDEX on one of the 
tables with the pk taking up the largest amount of space.  The usage 
according to pg_class dropped dramatically, however the disk space was 
not returned to the system.  So I attempted another full vacuum 
afterwards, and still nothing was returned to the system.  These tables 
are updated extremely frequently (although their total number of rows is 
close to constant), which is my guess as to why the pk indexes increase 
so rapidly in terms of their disk usage.  Unfortunately, PostgreSQL 
knowledge is limited, and I was wondering if anyone had experienced 
something similar / knows what else we can do to return this disk space 
back to the system?
    

This is possibly index bloat due to new keys always being larger than existing
keys. This was fixed in later releases. There is still some potential for
bloat due to fragmentation, but I believe that has a constant bound.
You might try using the cluster command. I think that will both clean up
the indexes and remove dead rows and do it faster than using a vacuum full
and reindexing. The downside is that the table will be unavailable during
the cluster which might be a deal breaker for you.
  

-- 
Kevin Johnson
Raytheon
AWIPS Sr. Systems Engineer
NWS Network Control Facility
p: 301.713.9362x325
f: 301.713.1905
--Boundary_(ID_4ubwVCNf2JQju27/p/SZJg)-- From jimn@enterprisedb.com Fri Oct 6 05:23:24 2006 Received: from localhost (mx1.hub.org [200.46.208.251]) by postgresql.org (Postfix) with ESMTP id 13CA09FB269 for ; Fri, 6 Oct 2006 02:23:24 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.208.251]) (amavisd-new, port 10024) with ESMTP id 86301-05 for ; Fri, 6 Oct 2006 02:23:17 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey- Received: from svr4.postgresql.org (svr4.postgresql.org [66.98.251.159]) by postgresql.org (Postfix) with ESMTP id 77CF69FB1E2 for ; Fri, 6 Oct 2006 02:23:14 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.3 Received: from webmail.enterprisedb.com (webmail.enterprisedb.com [63.246.7.176]) by svr4.postgresql.org (Postfix) with ESMTP id 106E95AF87E for ; Fri, 6 Oct 2006 05:23:10 +0000 (GMT) Received: from [192.168.42.113] (natpool.bovine.net [67.100.216.14]) by webmail.enterprisedb.com (Postfix) with ESMTP id 41E67A24E18; Fri, 6 Oct 2006 00:23:08 -0500 (CDT) In-Reply-To: <452531AB.2050709@noaa.gov> References: <45080D4C.2020709@noaa.gov> <20060922170557.GB571@wolff.to> <452531AB.2050709@noaa.gov> Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Cc: Bruno Wolff III , pgsql-admin@postgresql.org, Sean Webb Content-Transfer-Encoding: 7bit From: Jim Nasby Subject: Re: Disk space consumed by pk not returned after vacuum or reindex Date: Thu, 5 Oct 2006 20:29:44 -0500 To: Kevin Johnson X-Mailer: Apple Mail (2.752.2) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.281 tagged_above=0 required=5 tests=AWL, DATE_IN_PAST_03_06 X-Spam-Level: X-Archive-Number: 200610/36 X-Sequence-Number: 23339 I didn't see you mention what version you're running; index bloat shouldn't be a big issue in 7.4 and above. You also didn't mention how often you're vacuuming the table. If you don't vacuum the table frequently enough, you're going to get bloat, plain and simple. On Oct 5, 2006, at 11:24 AM, Kevin Johnson wrote: > Thank you for the suggestion, Bruno. The clustering did the trick > in reducing the current disk usage, however eventually the disk > space get consumed once more. I fear that we may just need to > update the version of Postgres to help alleviate index bloat! > > Bruno Wolff III wrote: >> On Wed, Sep 13, 2006 at 09:53:16 -0400, Kevin Johnson >> wrote: >>> We have a database, which consistently consumes more and more of >>> the disk space in it's lvol until it reaches 100%. So far, we >>> have tried to run a full vacuum on the database, with limited >>> success. Eventually, we had to drop and reload the database with >>> the same data inside. It brought the disk usage down to 73%. It >>> then began to creep once more toward 100%. After some research, I >>> was able to use the pg_class catalog to find that the items which >>> are expanding the quickest are primary key (btree) indexes. I >>> attempted to run a REINDEX on one of the tables with the pk >>> taking up the largest amount of space. The usage according to >>> pg_class dropped dramatically, however the disk space was not >>> returned to the system. So I attempted another full vacuum >>> afterwards, and still nothing was returned to the system. These >>> tables are updated extremely frequently (although their total >>> number of rows is close to constant), which is my guess as to why >>> the pk indexes increase so rapidly in terms of their disk usage. >>> Unfortunately, PostgreSQL knowledge is limited, and I was >>> wondering if anyone had experienced something similar / knows >>> what else we can do to return this disk space back to the system? >> This is possibly index bloat due to new keys always being larger >> than existing keys. This was fixed in later releases. There is >> still some potential for bloat due to fragmentation, but I believe >> that has a constant bound. You might try using the cluster >> command. I think that will both clean up the indexes and remove >> dead rows and do it faster than using a vacuum full and >> reindexing. The downside is that the table will be unavailable >> during the cluster which might be a deal breaker for you. -- Jim Nasby jim@nasby.net EnterpriseDB http://enterprisedb.com 512.569.9461 (cell) -- Jim Nasby jimn@enterprisedb.com EnterpriseDB http://enterprisedb.com 512.569.9461 (cell) From Kevin.Johnson@noaa.gov Fri Oct 6 10:39:57 2006 Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id DEAE99FB3A3 for ; Fri, 6 Oct 2006 07:39:57 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 38404-09 for ; Fri, 6 Oct 2006 10:39:51 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey- Received: from mmp1.nems.noaa.gov (mmp1.nems.noaa.gov [140.90.121.156]) by postgresql.org (Postfix) with ESMTP id 573DE9FB39F for ; Fri, 6 Oct 2006 07:39:51 -0300 (ADT) Received: from [192.168.2.4] (ip68-227-201-228.dc.dc.cox.net [68.227.201.228]) by mmp1.nems.noaa.gov (iPlanet Messaging Server 5.2 HotFix 2.01 (built Aug 26 2004)) with ESMTPSA id <0J6P00AHFMYDZ0@mmp1.nems.noaa.gov> for pgsql-admin@postgresql.org; Fri, 06 Oct 2006 06:39:50 -0400 (EDT) Date: Fri, 06 Oct 2006 06:39:40 -0400 From: Kevin Johnson Subject: Re: Disk space consumed by pk not returned after vacuum or In-reply-to: To: Jim Nasby Cc: Bruno Wolff III , pgsql-admin@postgresql.org, Sean Webb Message-id: <4526326C.2000009@noaa.gov> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_MIN/6XN8ThxE3ec0zodQQg)" User-Agent: Thunderbird 1.5.0.7 (Windows/20060909) References: <45080D4C.2020709@noaa.gov> <20060922170557.GB571@wolff.to> <452531AB.2050709@noaa.gov> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.589 tagged_above=0 required=5 tests=HTML_30_40, HTML_MESSAGE, HTML_TITLE_EMPTY X-Spam-Level: X-Archive-Number: 200610/41 X-Sequence-Number: 23344 This is a multi-part message in MIME format. --Boundary_(ID_MIN/6XN8ThxE3ec0zodQQg) Content-type: multipart/alternative; boundary="Boundary_(ID_Hyr6W926Ab/8lD/wXPAHOA)" --Boundary_(ID_Hyr6W926Ab/8lD/wXPAHOA) Content-type: text/plain; charset=ISO-8859-1; format=flowed Content-transfer-encoding: 7BIT You are correct, it is 7.4 we are running on a RHEL3 system. The database itself is vacuumed via cron 6 times a day. Jim Nasby wrote: > I didn't see you mention what version you're running; index bloat > shouldn't be a big issue in 7.4 and above. You also didn't mention how > often you're vacuuming the table. If you don't vacuum the table > frequently enough, you're going to get bloat, plain and simple. > > On Oct 5, 2006, at 11:24 AM, Kevin Johnson wrote: >> Thank you for the suggestion, Bruno. The clustering did the trick in >> reducing the current disk usage, however eventually the disk space >> get consumed once more. I fear that we may just need to update the >> version of Postgres to help alleviate index bloat! >> >> Bruno Wolff III wrote: >>> On Wed, Sep 13, 2006 at 09:53:16 -0400, Kevin Johnson >>> wrote: >>>> We have a database, which consistently consumes more and more of >>>> the disk space in it's lvol until it reaches 100%. So far, we have >>>> tried to run a full vacuum on the database, with limited success. >>>> Eventually, we had to drop and reload the database with the same >>>> data inside. It brought the disk usage down to 73%. It then began >>>> to creep once more toward 100%. After some research, I was able to >>>> use the pg_class catalog to find that the items which are expanding >>>> the quickest are primary key (btree) indexes. I attempted to run a >>>> REINDEX on one of the tables with the pk taking up the largest >>>> amount of space. The usage according to pg_class dropped >>>> dramatically, however the disk space was not returned to the >>>> system. So I attempted another full vacuum afterwards, and still >>>> nothing was returned to the system. These tables are updated >>>> extremely frequently (although their total number of rows is close >>>> to constant), which is my guess as to why the pk indexes increase >>>> so rapidly in terms of their disk usage. Unfortunately, PostgreSQL >>>> knowledge is limited, and I was wondering if anyone had experienced >>>> something similar / knows what else we can do to return this disk >>>> space back to the system? >>> This is possibly index bloat due to new keys always being larger >>> than existing keys. This was fixed in later releases. There is still >>> some potential for bloat due to fragmentation, but I believe that >>> has a constant bound. You might try using the cluster command. I >>> think that will both clean up the indexes and remove dead rows and >>> do it faster than using a vacuum full and reindexing. The downside >>> is that the table will be unavailable during the cluster which might >>> be a deal breaker for you. > -- > Jim Nasby jim@nasby.net > EnterpriseDB http://enterprisedb.com 512.569.9461 (cell) > > > -- > Jim Nasby jimn@enterprisedb.com > EnterpriseDB http://enterprisedb.com 512.569.9461 (cell) > > -- Kevin Johnson Raytheon AWIPS Sr. Systems Engineer NWS Network Control Facility p: 301.713.9362x325 f: 301.713.1905 --Boundary_(ID_Hyr6W926Ab/8lD/wXPAHOA) Content-type: text/html; charset=ISO-8859-1 Content-transfer-encoding: 7BIT You are correct, it is 7.4 we are running on a RHEL3 system.
The database itself is vacuumed via cron 6 times a day. 

Jim Nasby wrote:
I didn't see you mention what version you're running; index bloat shouldn't be a big issue in 7.4 and above. You also didn't mention how often you're vacuuming the table. If you don't vacuum the table frequently enough, you're going to get bloat, plain and simple.

On Oct 5, 2006, at 11:24 AM, Kevin Johnson wrote:
Thank you for the suggestion, Bruno.  The clustering did the trick in reducing the current disk usage, however eventually the disk space get consumed once more.  I fear that we may just need to update the version of Postgres to help alleviate index bloat!

Bruno Wolff III wrote:
On Wed, Sep 13, 2006 at 09:53:16 -0400, Kevin Johnson <Kevin.Johnson@noaa.gov> wrote:
We have a database, which consistently consumes more and more of the disk space in it's lvol until it reaches 100%. So far, we have tried to run a full vacuum on the database, with limited success. Eventually, we had to drop and reload the database with the same data inside. It brought the disk usage down to 73%. It then began to creep once more toward 100%. After some research, I was able to use the pg_class catalog to find that the items which are expanding the quickest are  primary key (btree) indexes. I attempted to run a REINDEX on one of the tables with the pk taking up the largest amount of space. The usage according to pg_class dropped dramatically, however the disk space was not returned to the system. So I attempted another full vacuum afterwards, and still nothing was returned to the system. These tables are updated extremely frequently (although their total number of rows is close to constant), which is my guess as to why the pk indexes increase so rapidly in terms of their disk usage. Unfortunately, PostgreSQL knowledge is limited, and I was wondering if anyone had experienced something similar / knows what else we can do to return this disk space back to the system?
This is possibly index bloat due to new keys always being larger than existing keys. This was fixed in later releases. There is still some potential for bloat due to fragmentation, but I believe that has a constant bound. You might try using the cluster command. I think that will both clean up the indexes and remove dead rows and do it faster than using a vacuum full and reindexing. The downside is that the table will be unavailable during the cluster which might be a deal breaker for you.
--
Jim Nasby                                            jim@nasby.net
EnterpriseDB      http://enterprisedb.com      512.569.9461 (cell)


--
Jim Nasby                                    jimn@enterprisedb.com
EnterpriseDB      http://enterprisedb.com      512.569.9461 (cell)



-- 
Kevin Johnson
Raytheon
AWIPS Sr. Systems Engineer
NWS Network Control Facility
p: 301.713.9362x325
f: 301.713.1905
--Boundary_(ID_Hyr6W926Ab/8lD/wXPAHOA)-- --Boundary_(ID_MIN/6XN8ThxE3ec0zodQQg) Content-type: text/x-vcard; charset=utf-8; name=Kevin.Johnson.vcf Content-transfer-encoding: 7BIT Content-disposition: attachment; filename=Kevin.Johnson.vcf begin:vcard fn:Kevin Johnson n:Johnson;Kevin org:Raytheon AWIPS Team adr:SMCC II;;;1325 East West Highway;Silver Spring;MD;20910 email;internet:Kevin.Johnson@noaa.gov title:Sr. Systems Engineer tel;work:301.713.9362 x325 tel;cell:301.787.0648 x-mozilla-html:TRUE version:2.1 end:vcard --Boundary_(ID_MIN/6XN8ThxE3ec0zodQQg)-- From jimn@enterprisedb.com Fri Oct 6 15:38:01 2006 Received: from localhost (wm.hub.org [200.46.204.128]) by postgresql.org (Postfix) with ESMTP id 0E1D39FB279 for ; Fri, 6 Oct 2006 12:38:01 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.128]) (amavisd-new, port 10024) with ESMTP id 15406-03 for ; Fri, 6 Oct 2006 15:37:55 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey- Received: from webmail.enterprisedb.com (webmail.enterprisedb.com [63.246.7.176]) by postgresql.org (Postfix) with ESMTP id CDEA59FB28B for ; Fri, 6 Oct 2006 12:37:54 -0300 (ADT) Received: from [192.168.42.113] (natpool.bovine.net [67.100.216.14]) by webmail.enterprisedb.com (Postfix) with ESMTP id 786A7A24DCE; Fri, 6 Oct 2006 10:37:53 -0500 (CDT) In-Reply-To: <4526326C.2000009@noaa.gov> References: <45080D4C.2020709@noaa.gov> <20060922170557.GB571@wolff.to> <452531AB.2050709@noaa.gov> <4526326C.2000009@noaa.gov> Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <8A3D37C4-8DA1-4ABE-A41E-A02FF03C2149@enterprisedb.com> Cc: Bruno Wolff III , pgsql-admin@postgresql.org, Sean Webb Content-Transfer-Encoding: 7bit From: Jim Nasby Subject: Re: Disk space consumed by pk not returned after vacuum or reindex Date: Fri, 6 Oct 2006 10:37:51 -0500 To: Kevin Johnson X-Mailer: Apple Mail (2.752.2) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests= X-Spam-Level: X-Archive-Number: 200610/46 X-Sequence-Number: 23349 On Oct 6, 2006, at 5:39 AM, Kevin Johnson wrote: > You are correct, it is 7.4 we are running on a RHEL3 system. > The database itself is vacuumed via cron 6 times a day. > > Jim Nasby wrote: >> I didn't see you mention what version you're running; index bloat >> shouldn't be a big issue in 7.4 and above. You also didn't mention >> how often you're vacuuming the table. If you don't vacuum the >> table frequently enough, you're going to get bloat, plain and simple. For a heavy update load, even 6 times a day may not be enough. Also, remember that the size you get after a REINDEX will be unreasonably low, because there will be very little free space in the index. That's fine if you won't be updating the table much, but otherwise the index will have no choice but to grow. -- Jim Nasby jimn@enterprisedb.com EnterpriseDB http://enterprisedb.com 512.569.9461 (cell) From bruno@wolff.to Fri Oct 6 19:13:28 2006 Received: from localhost (mx1.hub.org [200.46.208.251]) by postgresql.org (Postfix) with ESMTP id E1F639FA0B9 for ; Fri, 6 Oct 2006 16:13:28 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.208.251]) (amavisd-new, port 10024) with ESMTP id 12570-01-3 for ; Fri, 6 Oct 2006 16:13:25 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey- Received: from wolff.to (wolff.to [66.93.197.194]) by postgresql.org (Postfix) with SMTP id 842CA9FB55A for ; Fri, 6 Oct 2006 15:51:05 -0300 (ADT) Received: (qmail 8918 invoked by uid 500); 6 Oct 2006 19:01:34 -0000 Date: Fri, 6 Oct 2006 14:01:34 -0500 From: Bruno Wolff III To: Kevin Johnson Cc: Jim Nasby , pgsql-admin@postgresql.org, Sean Webb Subject: Re: Disk space consumed by pk not returned after vacuum or Message-ID: <20061006190134.GA6084@wolff.to> Mail-Followup-To: Bruno Wolff III , Kevin Johnson , Jim Nasby , pgsql-admin@postgresql.org, Sean Webb References: <45080D4C.2020709@noaa.gov> <20060922170557.GB571@wolff.to> <452531AB.2050709@noaa.gov> <4526326C.2000009@noaa.gov> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4526326C.2000009@noaa.gov> User-Agent: Mutt/1.4.2.1i X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.047 tagged_above=0 required=5 tests=AWL X-Spam-Level: X-Archive-Number: 200610/50 X-Sequence-Number: 23353 On Fri, Oct 06, 2006 at 06:39:40 -0400, Kevin Johnson wrote: > You are correct, it is 7.4 we are running on a RHEL3 system. > The database itself is vacuumed via cron 6 times a day. The other thing you need to watch out for is that your FSM setting is large enough to recover all of the dead space. If your vacuum runs aren't recovering as much dead space as is being created on average, then you are going to have dead space accumalate.