Date: Thu, 14 Jul 2005 20:28:24 -0400
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: Alison Winters <alison@mirrabooka.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: lots of updates on small table
Message-ID: <20050715002824.GA23728@alvh.no-ip.org>
References: <42D5F34E.nail9QE1195IE@pluto.mirrabooka.com>
	<20050714162114.GA19778@alvh.no-ip.org>
	<42D6F854.nail3K81VIICH@pluto.mirrabooka.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <42D6F854.nail3K81VIICH@pluto.mirrabooka.com>
User-Agent: Mutt/1.5.9i

On Fri, Jul 15, 2005 at 09:42:12AM +1000, Alison Winters wrote:

> > > Our application requires a number of processes to select and update rows
> > > from a very small (<10 rows) Postgres table on a regular and frequent
> > > basis.  These processes often run for weeks at a time, but over the
> > > space of a few days we find that updates start getting painfully slow.
> > > We are running a full vacuum/analyze and reindex on the table every day,
> > Full vacuum, eh?  I wonder if what you really need is very frequent
> > non-full vacuum.  Say, once in 15 minutes (exact rate depending on dead
> > tuple rate.)
> >
> Is there a difference between vacuum and vacuum full?

Yes.  Vacuum full is more aggresive in compacting the table.  Though it
really works the same in the presence of long-running transactions:
tuples just can't be removed.

> The most recent output was this:
> 
> INFO:  --Relation public.plc_fldio--
> INFO:  Pages 1221: Changed 3, reaped 256, Empty 0, New 0; Tup 108137: Vac 4176, Keep/VTL 108133/108133, UnUsed 19, MinLen 84, MaxLen 84; Re-using: Free/Avail. Space 445176/371836; EndEmpty/Avail. Pages 0/256.
>         CPU 0.04s/0.14u sec elapsed 0.18 sec.
> INFO:  Index plcpage_idx: Pages 315; Tuples 108137: Deleted 4176.
>         CPU 0.03s/0.04u sec elapsed 0.14 sec.
> INFO:  Rel plc_fldio: Pages: 1221 --> 1221; Tuple(s) moved: 0.
>         CPU 0.03s/0.04u sec elapsed 0.36 sec.
> INFO:  Analyzing public.plc_fldio

Hmm, so it seems your hourly vacuum is enough.  I think the bloat theory
can be trashed.  Unless I'm reading this output wrong; I don't remember
the details of this vacuum output.

> We'll up it to every 15 minutes, but i don't know if that'll help
> because even with the current vacuuming the updates are still getting
> slower and slower over the course of several days.  What really puzzles
> me is why restarting the processes fixes it.

I wonder if the problem may be plan caching.  I didn't pay full
attention to the description of your problem, so I don't remember if it
could be an issue, but it's something to consider.

> Does PostgreSQL keep some kind of backlog of transactions all for one
> database connection?

No.  There could be a problem if you had very long transactions, but
apparently this isn't your problem.

> Isn't it normal to have processes that keep a single database
> connection open for days at a time?

I guess it depends on exactly what you do with it.  I know of at least
one case where an app keeps a connection open for months, without a
problem.  (It's been running for four or five years, and monthly
"uptime" for that particular daemon is not unheard of.)

-- 
Alvaro Herrera (<alvherre[a]alvh.no-ip.org>)
"Everybody understands Mickey Mouse. Few understand Hermann Hesse.
Hardly anybody understands Einstein. And nobody understands Emperor Norton."