pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedDeteriorating performance when loading large objects
7+ messages / 4 participants
[nested] [flat]
* Deteriorating performance when loading large objects
@ 2008-11-25 12:54 Vegard Bønes <vegard.bones@met.no>
0 siblings, 2 replies; 7+ messages in thread
From: Vegard Bønes @ 2008-11-25 12:54 UTC (permalink / raw)
To: pgsql-performance
Hi,
I have a problem with large objects in postgresql 8.1: The performance of
loading large objects into a database goes way down after a few days of
operation.
I have a cron job kicking in twice a day, which generates and loads around
6000 large objects of 3.7MB each. Each night, old data is deleted, so
there is never more than 24000 large object in the database.
If I start loading on a freshly installed database, load times for this is
around 13 minutes, including generating the data to be stored. If I let
the database run for a few days, this takes much longer. After one or two
days, this goes down to almost an hour, with logs indicating that this
extra time is solely spent transferring the large objects from file to
database.
Turning autovacuum on or off seems to have no effect on this.
I have only made the following changes to the default postgresql.conf file:
max_fsm_pages = 25000000
vacuum_cost_delay = 10
checkpoint_segments = 256
So, my question for you is: Why does this happen, and what can I do about it?
Regards,
Vegard Bønes
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Deteriorating performance when loading large objects
@ 2008-11-25 13:17 Ivan Voras <ivoras@freebsd.org>
parent: Vegard Bønes <vegard.bones@met.no>
1 sibling, 0 replies; 7+ messages in thread
From: Ivan Voras @ 2008-11-25 13:17 UTC (permalink / raw)
To: pgsql-performance
Vegard Bønes wrote:
> Hi,
>
> I have a problem with large objects in postgresql 8.1: The performance of
> loading large objects into a database goes way down after a few days of
> operation.
>
> I have a cron job kicking in twice a day, which generates and loads around
> 6000 large objects of 3.7MB each. Each night, old data is deleted, so
> there is never more than 24000 large object in the database.
> So, my question for you is: Why does this happen, and what can I do about it?
Try putting a "vacuumdb -zf" command as a cron job after the data is
deleted.
Attachments:
signature.asc (application/pgp-signature, 251B) download
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Deteriorating performance when loading large objects
@ 2008-11-25 13:36 Tom Lane <tgl@sss.pgh.pa.us>
parent: Vegard Bønes <vegard.bones@met.no>
1 sibling, 1 reply; 7+ messages in thread
From: Tom Lane @ 2008-11-25 13:36 UTC (permalink / raw)
To: vegard.bones@met.no; +Cc: pgsql-performance
"=?iso-8859-1?Q?Vegard_B=F8nes?=" <vegard.bones@met.no> writes:
> I have a problem with large objects in postgresql 8.1: The performance of
> loading large objects into a database goes way down after a few days of
> operation.
> I have a cron job kicking in twice a day, which generates and loads around
> 6000 large objects of 3.7MB each. Each night, old data is deleted, so
> there is never more than 24000 large object in the database.
Are you sure you're deleting the large objects themselves (ie,
lo_unlink), and not just deleting some references to them?
A manual "vacuum verbose" on pg_largeobject might be informative.
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Deteriorating performance when loading large objects
@ 2008-11-27 19:04 Vegard Bønes <vegard.bones@met.no>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 7+ messages in thread
From: Vegard Bønes @ 2008-11-27 19:04 UTC (permalink / raw)
To: tgl@sss.pgh.pa.us; +Cc: pgsql-performance
> "=?iso-8859-1?Q?Vegard_B=F8nes?=" <vegard.bones@met.no> writes:
>> I have a problem with large objects in postgresql 8.1: The performance
>> of loading large objects into a database goes way down after a few
>> days of operation.
>
>> I have a cron job kicking in twice a day, which generates and loads
>> around 6000 large objects of 3.7MB each. Each night, old data is
>> deleted, so there is never more than 24000 large object in the
>> database.
>
> Are you sure you're deleting the large objects themselves (ie,
> lo_unlink), and not just deleting some references to them?
>
> A manual "vacuum verbose" on pg_largeobject might be informative.
I do call lo_unlink via a trigger function. Also, a SELECT count(distinct
loid) FROM pg_largeobject yields the same result as a similar call to the
table which references the large objects.
Running VACUUM VERBOSE pg_largeobject took quite some time. Here's the
output:
INFO: vacuuming "pg_catalog.pg_largeobject"
INFO: index "pg_largeobject_loid_pn_index" now contains 11060658 row
versions in 230587 pages
DETAIL: 178683 index pages have been deleted, 80875 are currently reusable.
CPU 0.92s/0.10u sec elapsed 199.38 sec.
INFO: "pg_largeobject": found 0 removable, 11060658 nonremovable row
versions in 6849398 pages
DETAIL: 0 dead row versions cannot be removed yet.
There were 84508215 unused item pointers.
0 pages are entirely empty.
CPU 0.98s/0.10u sec elapsed 4421.17 sec.
VACUUM
I will try to run VACUUM ANALYZE FULL after the next delete tonight, as
suggested by Ivan Voras in another post. But as I understand it, this will
put an exclusive lock on whatever table is being vacuumed, so it is not
really an option for the database in question, as it needs to be
accessitble 24 hours a day.
Is there any other possible solution to this?
As a side note, I have noticed that loading times seem to have stabilized
at just above an hour.
Regards,
Vegard Bønes
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Deteriorating performance when loading large objects
@ 2008-11-27 21:29 Tom Lane <tgl@sss.pgh.pa.us>
parent: Vegard Bønes <vegard.bones@met.no>
0 siblings, 2 replies; 7+ messages in thread
From: Tom Lane @ 2008-11-27 21:29 UTC (permalink / raw)
To: vegard.bones@met.no; +Cc: pgsql-performance
"=?iso-8859-1?Q?Vegard_B=F8nes?=" <vegard.bones@met.no> writes:
> Running VACUUM VERBOSE pg_largeobject took quite some time. Here's the
> output:
> INFO: vacuuming "pg_catalog.pg_largeobject"
> INFO: index "pg_largeobject_loid_pn_index" now contains 11060658 row
> versions in 230587 pages
> DETAIL: 178683 index pages have been deleted, 80875 are currently reusable.
> CPU 0.92s/0.10u sec elapsed 199.38 sec.
> INFO: "pg_largeobject": found 0 removable, 11060658 nonremovable row
> versions in 6849398 pages
> DETAIL: 0 dead row versions cannot be removed yet.
> There were 84508215 unused item pointers.
> 0 pages are entirely empty.
> CPU 0.98s/0.10u sec elapsed 4421.17 sec.
> VACUUM
Hmm ... although you have no dead rows now, the very large number of
unused item pointers suggests that there were times in the past when
pg_largeobject didn't get vacuumed often enough. You need to look at
your vacuuming policy. If you're using autovacuum, it might need to have
its parameters adjusted. Otherwise, how often are you vacuuming, and
are you doing it as superuser?
> I will try to run VACUUM ANALYZE FULL after the next delete tonight, as
> suggested by Ivan Voras in another post.
Actually, a CLUSTER might be more effective.
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Deteriorating performance when loading large objects
@ 2008-11-28 08:30 Mario Weilguni <mweilguni@sime.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 7+ messages in thread
From: Mario Weilguni @ 2008-11-28 08:30 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: vegard.bones@met.no; pgsql-performance
Tom Lane schrieb:
> "=?iso-8859-1?Q?Vegard_B=F8nes?=" <vegard.bones@met.no> writes:
>
>> Running VACUUM VERBOSE pg_largeobject took quite some time. Here's the
>> output:
>>
>
>
>> INFO: vacuuming "pg_catalog.pg_largeobject"
>> INFO: index "pg_largeobject_loid_pn_index" now contains 11060658 row
>> versions in 230587 pages
>> DETAIL: 178683 index pages have been deleted, 80875 are currently reusable.
>> CPU 0.92s/0.10u sec elapsed 199.38 sec.
>> INFO: "pg_largeobject": found 0 removable, 11060658 nonremovable row
>> versions in 6849398 pages
>> DETAIL: 0 dead row versions cannot be removed yet.
>> There were 84508215 unused item pointers.
>> 0 pages are entirely empty.
>> CPU 0.98s/0.10u sec elapsed 4421.17 sec.
>> VACUUM
>>
>
> Hmm ... although you have no dead rows now, the very large number of
> unused item pointers suggests that there were times in the past when
> pg_largeobject didn't get vacuumed often enough. You need to look at
> your vacuuming policy. If you're using autovacuum, it might need to have
> its parameters adjusted. Otherwise, how often are you vacuuming, and
> are you doing it as superuser?
>
>
>> I will try to run VACUUM ANALYZE FULL after the next delete tonight, as
>> suggested by Ivan Voras in another post.
>>
>
> Actually, a CLUSTER might be more effective.
>
> regards, tom lane
>
>
Does CLUSTER really help here? On my 8.2 database, I get:
CLUSTER pg_largeobject_loid_pn_index on pg_largeobject ;
ERROR: "pg_largeobject" is a system catalog
Has this changed in >= 8.3?
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Deteriorating performance when loading large objects
@ 2008-11-28 15:32 Vegard Bønes <vegard.bones@met.no>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 7+ messages in thread
From: Vegard Bønes @ 2008-11-28 15:32 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-performance
Tom Lane wrote:
>> INFO: vacuuming "pg_catalog.pg_largeobject"
>> INFO: index "pg_largeobject_loid_pn_index" now contains 11060658 row
>> versions in 230587 pages
>> DETAIL: 178683 index pages have been deleted, 80875 are currently reusable.
>> CPU 0.92s/0.10u sec elapsed 199.38 sec.
>> INFO: "pg_largeobject": found 0 removable, 11060658 nonremovable row
>> versions in 6849398 pages
>> DETAIL: 0 dead row versions cannot be removed yet.
>> There were 84508215 unused item pointers.
>> 0 pages are entirely empty.
>> CPU 0.98s/0.10u sec elapsed 4421.17 sec.
>> VACUUM
>
> Hmm ... although you have no dead rows now, the very large number of
> unused item pointers suggests that there were times in the past when
> pg_largeobject didn't get vacuumed often enough. You need to look at
> your vacuuming policy. If you're using autovacuum, it might need to have
> its parameters adjusted. Otherwise, how often are you vacuuming, and
> are you doing it as superuser?
>
>> I will try to run VACUUM ANALYZE FULL after the next delete tonight, as
>> suggested by Ivan Voras in another post.
>
> Actually, a CLUSTER might be more effective.
>
> regards, tom lane
>
I have autovacuum turned on, but the high number of unused item pointers
may be related to some experimentation I did earlier on the same
database with only vacuuming after the nightly delete. I have, however,
seen the same performance degradation on a database that had only ever
been vacuumed by autovacuum.
I am a bit unsure about what parameters to adjust in order to maintain a
good loading performance for bulk loading. Do you have any suggestions?
Also, VACUUM ANALYZE FULL has been running for 15 hours now, blocking
the loading of today's data. It will be interesting to see how the
database will work once it is completed. Is there any point in trying to
use CLUSTER instead if this does not give any result?
Regards,
Vegard Bønes
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2008-11-28 15:32 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-11-25 12:54 Deteriorating performance when loading large objects Vegard Bønes <vegard.bones@met.no>
2008-11-25 13:17 ` Ivan Voras <ivoras@freebsd.org>
2008-11-25 13:36 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-11-27 19:04 ` Vegard Bønes <vegard.bones@met.no>
2008-11-27 21:29 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-11-28 08:30 ` Mario Weilguni <mweilguni@sime.com>
2008-11-28 15:32 ` Vegard Bønes <vegard.bones@met.no>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox