pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedDatabase-wide vacuum can take a long time, during which tables are not being analyzed
5+ messages / 4 participants
[nested] [flat]
* Database-wide vacuum can take a long time, during which tables are not being analyzed
@ 2006-11-01 19:15 Steven Flatt <steven.flatt@gmail.com>
0 siblings, 2 replies; 5+ messages in thread
From: Steven Flatt @ 2006-11-01 19:15 UTC (permalink / raw)
To: pgsql-performance
Here is a potential problem with the auto-vacuum daemon, and I'm wondering
if anyone has considered this. To avoid transaction ID wraparound, the
auto-vacuum daemon will periodically determine that it needs to do a DB-wide
vacuum, which takes a long time. On our system, it is on the order of a
couple of weeks. (The system is very busy and there is a lot of I/O going
on pretty much 24/7). During this period of time, there is nothing to
automatically analyze any of the tables, leading to further performance
problems. What are your thoughts on having the DB-wide vacuum running on a
separate thread so that the daemon can concurrently wake up and take care of
analyzing tables?
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Database-wide vacuum can take a long time, during which
@ 2006-11-01 21:56 Matthew O'Connor <matthew@zeut.net>
parent: Steven Flatt <steven.flatt@gmail.com>
1 sibling, 0 replies; 5+ messages in thread
From: Matthew O'Connor @ 2006-11-01 21:56 UTC (permalink / raw)
To: Steven Flatt <steven.flatt@gmail.com>; +Cc: pgsql-performance
Steven Flatt wrote:
> Here is a potential problem with the auto-vacuum daemon, and I'm
> wondering if anyone has considered this. To avoid transaction ID
> wraparound, the auto-vacuum daemon will periodically determine that it
> needs to do a DB-wide vacuum, which takes a long time. On our system,
> it is on the order of a couple of weeks. (The system is very busy and
> there is a lot of I/O going on pretty much 24/7). During this period of
> time, there is nothing to automatically analyze any of the tables,
> leading to further performance problems. What are your thoughts on
> having the DB-wide vacuum running on a separate thread so that the
> daemon can concurrently wake up and take care of analyzing tables?
Two issues here:
1)XID Wraparound: There has been work done on this already, and in 8.2
I believe there will no longer be a requirement that a database wide
vacuum be issued, rather, XID wraparound will be managed on a per table
basis rather than per database, so that will solve this problem.
2)Concurrent Vacuuming: There has been a lot of talk about
multiple-concurrent vacuums and I believe that this is required in the
long run, but it's not here yet, and won't be in 8.2, hopefully it will
get done for 8.3.
Matt
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Database-wide vacuum can take a long time, duringwhich tables are not being analyzed
@ 2006-11-02 10:49 Simon Riggs <simon@2ndquadrant.com>
parent: Steven Flatt <steven.flatt@gmail.com>
1 sibling, 1 reply; 5+ messages in thread
From: Simon Riggs @ 2006-11-02 10:49 UTC (permalink / raw)
To: Steven Flatt <steven.flatt@gmail.com>; +Cc: pgsql-performance
On Wed, 2006-11-01 at 14:15 -0500, Steven Flatt wrote:
> Here is a potential problem with the auto-vacuum daemon, and I'm
> wondering if anyone has considered this. To avoid transaction ID
> wraparound, the auto-vacuum daemon will periodically determine that it
> needs to do a DB-wide vacuum, which takes a long time. On our system,
> it is on the order of a couple of weeks. (The system is very busy and
> there is a lot of I/O going on pretty much 24/7). During this period
> of time, there is nothing to automatically analyze any of the tables,
> leading to further performance problems. What are your thoughts on
> having the DB-wide vacuum running on a separate thread so that the
> daemon can concurrently wake up and take care of analyzing tables?
Yes, do it.
Every couple of weeks implies a transaction rate of ~~500tps, so I'd be
interested to hear more about your system.
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Database-wide vacuum can take a long time, duringwhich tables are not being analyzed
@ 2006-11-02 15:15 Steven Flatt <steven.flatt@gmail.com>
parent: Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 5+ messages in thread
From: Steven Flatt @ 2006-11-02 15:15 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: pgsql-performance
Sorry, I think there's a misunderstanding here. Our system is not doing
near that number of transactions per second. I meant that the duration of a
single DB-wide vacuum takes on the order of a couple of weeks. The time
between DB-wide vacuums is a little over a year, I believe.
> Every couple of weeks implies a transaction rate of ~~500tps, so I'd be
> interested to hear more about your system.
>
> --
> Simon Riggs
> EnterpriseDB http://www.enterprisedb.com
>
>
>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Database-wide vacuum can take a long time, duringwhich tables are not being analyzed
@ 2006-11-02 15:53 Alvaro Herrera <alvherre@commandprompt.com>
parent: Steven Flatt <steven.flatt@gmail.com>
0 siblings, 0 replies; 5+ messages in thread
From: Alvaro Herrera @ 2006-11-02 15:53 UTC (permalink / raw)
To: Steven Flatt <steven.flatt@gmail.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-performance
Steven Flatt wrote:
> Sorry, I think there's a misunderstanding here. Our system is not doing
> near that number of transactions per second. I meant that the duration of a
> single DB-wide vacuum takes on the order of a couple of weeks. The time
> between DB-wide vacuums is a little over a year, I believe.
I wonder if this is using some vacuum delay setting?
If that's the case, I think you could manually run a database-wide
vacuum with a zero vacuum delay setting, so that said vacuum takes less
time to finish (say, once every 8 months).
(8.2 pretty much solves this issue BTW, by not requiring database-wide
vacuums).
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2006-11-02 15:53 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-11-01 19:15 Database-wide vacuum can take a long time, during which tables are not being analyzed Steven Flatt <steven.flatt@gmail.com>
2006-11-01 21:56 ` Re: Database-wide vacuum can take a long time, during which Matthew O'Connor <matthew@zeut.net>
2006-11-02 10:49 ` Re: Database-wide vacuum can take a long time, duringwhich tables are not being analyzed Simon Riggs <simon@2ndquadrant.com>
2006-11-02 15:15 ` Re: Database-wide vacuum can take a long time, duringwhich tables are not being analyzed Steven Flatt <steven.flatt@gmail.com>
2006-11-02 15:53 ` Re: Database-wide vacuum can take a long time, duringwhich tables are not being analyzed Alvaro Herrera <alvherre@commandprompt.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox