pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
TOAST tables keeps growing!
5+ messages / 3 participants
[nested] [flat]

* TOAST tables keeps growing!
@ 2004-08-12 14:46 Thomas Madsen <tm@softcom.dk>
  2004-08-13 17:36 ` Re: TOAST tables keeps growing! Stephan Szabo <sszabo@megazone.bigpanda.com>
  2004-08-13 19:02 ` Re: TOAST tables keeps growing! Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 2 replies; 5+ messages in thread

From: Thomas Madsen @ 2004-08-12 14:46 UTC (permalink / raw)
  To: pgsql-admin

Hi.
 
I'm running a production server with PostgreSQL version 7.2.1, that does
email scans.
That means up to tens of thousands of entries in the database (log and
quarantine) each day, and an equal amount removed every night.
 
The problem is that TOAST tables keeps springing forth and consume disk
space. 
We do a VACUUM every night, but it does not reduce or stop the TOAST
table growth.
 
I searched the mailing list for answers to this problem, but found only
other people describing the same problem.
Namely, that their max_fsm_pages were too low and that postgresql now
has lost track of the surplus TOAST data pages.
 
In our postgresql.conf we ourselves had the default value of 10000 for
max_fsm_pages, which I have now raised to around a million. 
I restarted psql and did a vacuum to see if that would reduce the disk
usage, but no change was seen.
 
So the question(s) is: 
    What can I do to reclaim the wasted TOAST diskspace?
    Can I find and eliminate the lost TOAST tuples somehow?
 
 
Thanks!
 
Best regards,
Thomas M. Madsen.
 
 
 

^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: TOAST tables keeps growing!
  2004-08-12 14:46 TOAST tables keeps growing! Thomas Madsen <tm@softcom.dk>
@ 2004-08-13 17:36 ` Stephan Szabo <sszabo@megazone.bigpanda.com>
  1 sibling, 0 replies; 5+ messages in thread

From: Stephan Szabo @ 2004-08-13 17:36 UTC (permalink / raw)
  To: Thomas Madsen <tm@softcom.dk>; +Cc: pgsql-admin

On Thu, 12 Aug 2004, Thomas Madsen wrote:

> I'm running a production server with PostgreSQL version 7.2.1, that does

You really should upgrade.  If not to 7.4.x, at least to the last 7.2
release.

> The problem is that TOAST tables keeps springing forth and consume disk
> space.
> We do a VACUUM every night, but it does not reduce or stop the TOAST
> table growth.
>
> I searched the mailing list for answers to this problem, but found only
> other people describing the same problem.
> Namely, that their max_fsm_pages were too low and that postgresql now
> has lost track of the surplus TOAST data pages.
>
> In our postgresql.conf we ourselves had the default value of 10000 for
> max_fsm_pages, which I have now raised to around a million.
> I restarted psql and did a vacuum to see if that would reduce the disk
> usage, but no change was seen.

Generally, raising fsm and revacuuming (without full) will mean that more
of the space will get re-used so that it should stop growing from day to
day, but it's not really going to remove space already taken (excepting
blank pages at the end if it can get appropriate locks I think).

Vacuum Full should remove the blank space at the cost of an exclusive lock
on the table.



^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: TOAST tables keeps growing!
  2004-08-12 14:46 TOAST tables keeps growing! Thomas Madsen <tm@softcom.dk>
@ 2004-08-13 19:02 ` Tom Lane <tgl@sss.pgh.pa.us>
  1 sibling, 0 replies; 5+ messages in thread

From: Tom Lane @ 2004-08-13 19:02 UTC (permalink / raw)
  To: Thomas Madsen <tm@softcom.dk>; +Cc: pgsql-admin

"Thomas Madsen" <tm@softcom.dk> writes:
>     What can I do to reclaim the wasted TOAST diskspace?

vacuum full might help.

>     Can I find and eliminate the lost TOAST tuples somehow?

Well, one thing to ask is whether it's the toast *table* that's
bloating, or the *index* on the toast table.  (VACUUM VERBOSE
would help you find out.)

If it's the table, then more frequent plain vacuums and/or increasing
FSM should fix it.  If the index is growing while the table stays
about the same, you could try periodic REINDEXes, but the only real
solution is to move up to PG 7.4.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: TOAST tables keeps growing!
@ 2004-08-16 09:27 Thomas Madsen <tm@softcom.dk>
  0 siblings, 0 replies; 5+ messages in thread

From: Thomas Madsen @ 2004-08-16 09:27 UTC (permalink / raw)
  To: pgsql-admin



> -----Original Message-----
> From: pgsql-admin-owner@postgresql.org 
> [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Tom Lane
> Sent: 13. august 2004 21:02
> To: Thomas Madsen
> Cc: pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] TOAST tables keeps growing! 
> 
> 
> "Thomas Madsen" <tm@softcom.dk> writes:
> >     What can I do to reclaim the wasted TOAST diskspace?
> 
> vacuum full might help.

Yes, vacuum full did the trick!
 
> >     Can I find and eliminate the lost TOAST tuples somehow?
> 
> Well, one thing to ask is whether it's the toast *table* 
> that's bloating, or the *index* on the toast table.  (VACUUM 
> VERBOSE would help you find out.)
>
> If it's the table, then more frequent plain vacuums and/or 
> increasing FSM should fix it.  If the index is growing while 
> the table stays about the same, you could try periodic 
> REINDEXes, but the only real solution is to move up to PG 7.4.

The bloating was primarily caused by the TOAST tables (several gigs
surplus), but the TOAST indexes are also slowly bloating as we have
indexes on a UNIX timestamp column. I will have to choose between
reindexing and upgrading...

For now I will keep an eye on the TOAST tables to see if the new FSM
value (1,000,000) keeps the TOAST tables at a reasonable size.

Thank you for the input!

Cheers,
Thomas.



^ permalink  raw  reply  [nested|flat] 5+ messages in thread

* Re: TOAST tables keeps growing!
@ 2004-08-16 09:46 Thomas Madsen <tm@softcom.dk>
  0 siblings, 0 replies; 5+ messages in thread

From: Thomas Madsen @ 2004-08-16 09:46 UTC (permalink / raw)
  To: pgsql-admin

> -----Original Message-----
> From: pgsql-admin-owner@postgresql.org
> [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Stephan Szabo
> Sent: 13. august 2004 19:36
> To: Thomas Madsen
> Cc: pgsql-admin@postgresql.org
> Subject: Re: [ADMIN] TOAST tables keeps growing!
> 
> 
> On Thu, 12 Aug 2004, Thomas Madsen wrote:
>
> > I'm running a production server with PostgreSQL version 7.2.1, that
> > does
> 
> You really should upgrade.  If not to 7.4.x, at least to the
> last 7.2 release.

Yes I know, I'm painfully aware of the need to upgrade :))

> > The problem is that TOAST tables keeps springing forth and consume
> > disk space. We do a VACUUM every night, but it does not 
> reduce or stop
> > the TOAST table growth.
> >
> > I searched the mailing list for answers to this problem, but found
> > only other people describing the same problem. Namely, that their 
> > max_fsm_pages were too low and that postgresql now has lost 
> track of
> > the surplus TOAST data pages.
> >
> > In our postgresql.conf we ourselves had the default value
> of 10000 for
> > max_fsm_pages, which I have now raised to around a million. I
> > restarted psql and did a vacuum to see if that would reduce 
> the disk
> > usage, but no change was seen.
> 
> Generally, raising fsm and revacuuming (without full) will
> mean that more of the space will get re-used so that it 
> should stop growing from day to day, but it's not really 
> going to remove space already taken (excepting blank pages at 
> the end if it can get appropriate locks I think).
>
> Vacuum Full should remove the blank space at the cost of an
> exclusive lock on the table.

Thanks, vacuum full on the table itself worked just perfect! ;)

Cheers,
Thomas.



^ permalink  raw  reply  [nested|flat] 5+ messages in thread


end of thread, other threads:[~2004-08-16 09:46 UTC | newest]

Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-08-12 14:46 TOAST tables keeps growing! Thomas Madsen <tm@softcom.dk>
2004-08-13 17:36 ` Stephan Szabo <sszabo@megazone.bigpanda.com>
2004-08-13 19:02 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-08-16 09:27 Re: TOAST tables keeps growing! Thomas Madsen <tm@softcom.dk>
2004-08-16 09:46 Re: TOAST tables keeps growing! Thomas Madsen <tm@softcom.dk>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox