pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedSlow Inserts on large tables
5+ messages / 4 participants
[nested] [flat]
* Slow Inserts on large tables
@ 2008-10-03 07:25 Peter Childs <peterachilds@gmail.com>
2008-10-03 08:22 ` Re: Slow Inserts on large tables Peter Eisentraut <peter_e@gmx.net>
0 siblings, 1 reply; 5+ messages in thread
From: Peter Childs @ 2008-10-03 07:25 UTC (permalink / raw)
To: pgsql-performance
I have a problem where by an insert on a "large" table will sometimes
take longer than usual.
Usually the inserts are quick then from time to time they will take a
long time sometimes as much as 10seconds or longer. (But usually under
500ms which is when I start logging them)
The queries are slow drip fed so bulk loading really is not an option,
Its logging data. Used in analysis and for historical purposes mostly.
I think the problem might have something to do with checkpoints, I'm
relatively sure its not when the table expands as I've run a vacuum
verbose straight away after a longer insert and not found loads of
space in the fsm.
I'm using 8.3.1 (I thought I'd upgraded to 8.3.3 but it does not look
like the upgrade worked) I'm more than happy to upgrade just have to
find the down time (even a few seconds can be difficult)
Any help would be appreciated.
Regards
Peter Childs
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Slow Inserts on large tables
2008-10-03 07:25 Slow Inserts on large tables Peter Childs <peterachilds@gmail.com>
@ 2008-10-03 08:22 ` Peter Eisentraut <peter_e@gmx.net>
2008-10-03 08:47 ` Re: Slow Inserts on large tables Peter Childs <peterachilds@gmail.com>
0 siblings, 1 reply; 5+ messages in thread
From: Peter Eisentraut @ 2008-10-03 08:22 UTC (permalink / raw)
To: Peter Childs <peterachilds@gmail.com>; +Cc: pgsql-performance
Peter Childs wrote:
> I have a problem where by an insert on a "large" table will sometimes
> take longer than usual.
> I think the problem might have something to do with checkpoints,
Then show us your checkpointing-related parameters. Or try to set them
to a lot higher values so checkpoints happen more rarely and see if that
makes a difference.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Slow Inserts on large tables
2008-10-03 07:25 Slow Inserts on large tables Peter Childs <peterachilds@gmail.com>
2008-10-03 08:22 ` Re: Slow Inserts on large tables Peter Eisentraut <peter_e@gmx.net>
@ 2008-10-03 08:47 ` Peter Childs <peterachilds@gmail.com>
2008-10-03 10:09 ` Re: Slow Inserts on large tables Iulian Dragan <iulian_dragan@yahoo.com>
2008-10-03 12:19 ` Re: Slow Inserts on large tables Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 5+ messages in thread
From: Peter Childs @ 2008-10-03 08:47 UTC (permalink / raw)
To: ; +Cc: pgsql-performance
2008/10/3 Peter Eisentraut <peter_e@gmx.net>:
> Peter Childs wrote:
>>
>> I have a problem where by an insert on a "large" table will sometimes
>> take longer than usual.
>
>> I think the problem might have something to do with checkpoints,
>
> Then show us your checkpointing-related parameters. Or try to set them to a
> lot higher values so checkpoints happen more rarely and see if that makes a
> difference.
>
>
More often or less often?
I've currently got them set to
checkpoint_segments = 3
checkpoint_timeout = 180s
checkpoint_completion_target = 0.5
after reading that doing more smaller checkpoints might make each
checkpoint work quicker and hence less of a performance hit when they
actually happen.
Regards
Peter
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Slow Inserts on large tables
2008-10-03 07:25 Slow Inserts on large tables Peter Childs <peterachilds@gmail.com>
2008-10-03 08:22 ` Re: Slow Inserts on large tables Peter Eisentraut <peter_e@gmx.net>
2008-10-03 08:47 ` Re: Slow Inserts on large tables Peter Childs <peterachilds@gmail.com>
@ 2008-10-03 10:09 ` Iulian Dragan <iulian_dragan@yahoo.com>
1 sibling, 0 replies; 5+ messages in thread
From: Iulian Dragan @ 2008-10-03 10:09 UTC (permalink / raw)
To: Peter Childs <peterachilds@gmail.com>; +Cc: pgsql-performance
Peter,
(please take this with a pinch of salt as I am no expert)
Here is a possible scenario:
Each of your checkpoints takes 90 seconds or more (you told it so with the checkpoint_completion_target).
If your insert fills 3 checkpoint segments (48 megs ) in less than 90 seconds then a new checkpoint request is issued. And maybe a third one, and so on. I imagine that this can flood the disk cache with write requests at some point although I can't explain how.
Have a look at the log, see the interval between the checkpoint requests and try to make this (a lot) larger than the checkpoint duration.
Start by increasing your checkpoint_segments (to, say, 16). If this doesn't work, maybe the timeout is too short, or the 90 seconds target to generous.
Regards,
Iulian
--- On Fri, 10/3/08, Peter Childs <peterachilds@gmail.com> wrote:
From: Peter Childs <peterachilds@gmail.com>
Subject: Re: [PERFORM] Slow Inserts on large tables
To:
Cc: "Postgresql Performance" <pgsql-performance@postgresql.org>
Date: Friday, October 3, 2008, 9:47 AM
2008/10/3 Peter Eisentraut <peter_e@gmx.net>:
> Peter Childs wrote:
>>
>> I have a problem where by an insert on a "large" table will
sometimes
>> take longer than usual.
>
>> I think the problem might have something to do with checkpoints,
>
> Then show us your checkpointing-related parameters. Or try to set them to
a
> lot higher values so checkpoints happen more rarely and see if that makes
a
> difference.
>
>
More often or less often?
I've currently got them set to
checkpoint_segments = 3
checkpoint_timeout = 180s
checkpoint_completion_target = 0.5
after reading that doing more smaller checkpoints might make each
checkpoint work quicker and hence less of a performance hit when they
actually happen.
Regards
Peter
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Slow Inserts on large tables
2008-10-03 07:25 Slow Inserts on large tables Peter Childs <peterachilds@gmail.com>
2008-10-03 08:22 ` Re: Slow Inserts on large tables Peter Eisentraut <peter_e@gmx.net>
2008-10-03 08:47 ` Re: Slow Inserts on large tables Peter Childs <peterachilds@gmail.com>
@ 2008-10-03 12:19 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 5+ messages in thread
From: Tom Lane @ 2008-10-03 12:19 UTC (permalink / raw)
To: Peter Childs <peterachilds@gmail.com>; +Cc: pgsql-performance
"Peter Childs" <peterachilds@gmail.com> writes:
> 2008/10/3 Peter Eisentraut <peter_e@gmx.net>:
>> Then show us your checkpointing-related parameters.
> I've currently got them set to
> checkpoint_segments = 3
> checkpoint_timeout = 180s
> checkpoint_completion_target = 0.5
> after reading that doing more smaller checkpoints might make each
> checkpoint work quicker and hence less of a performance hit when they
> actually happen.
That concept is actually pretty obsolete in 8.3: with spread-out
checkpoints it basically shouldn't hurt to increase the checkpoint
interval, and could actually help because the bgwriter doesn't have
such a tight deadline to finish the checkpoint. In any case you
*definitely* need to increase checkpoint_segments --- the value
you've got could be forcing a checkpoint every few seconds not
every few minutes.
What I would suggest is turning on log_checkpoints and then seeing
if there's any correlation between your slow insert commands and the
checkpoints. I'm suspicious that the problem is somewhere else.
(For instance, have you got anything that might take a lock on the
table? Maybe enabling log_lock_waits would be a good idea too.)
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2008-10-03 12:19 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-10-03 07:25 Slow Inserts on large tables Peter Childs <peterachilds@gmail.com>
2008-10-03 08:22 ` Peter Eisentraut <peter_e@gmx.net>
2008-10-03 08:47 ` Peter Childs <peterachilds@gmail.com>
2008-10-03 10:09 ` Iulian Dragan <iulian_dragan@yahoo.com>
2008-10-03 12:19 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox