pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedpoor performance with Context Switch Storm at TPC-W.
8+ messages / 5 participants
[nested] [flat]
* poor performance with Context Switch Storm at TPC-W.
@ 2006-07-11 11:09 Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
0 siblings, 3 replies; 8+ messages in thread
From: Katsuhiko Okano @ 2006-07-11 11:09 UTC (permalink / raw)
To: pgsql-hackers
Hi,All.
The problem has occurred in my customer.
poor performance with Context Switch Storm occurred
with the following composition.
Usually, CS is about 5000, WIPS=360.
when CSStorm occurrence, CS is about 100000, WIPS=60 or less.
(WIPS = number of web interactions per second)
It is under investigation using the patch
which collects a LWLock.
I suspected conflict of BufMappingLock.
but, collected results are seen,
occurrence of CSStorm and the increase of BufMappingLock counts
seem not to correspond.
Instead, SubtransControlLock and SubTrans were increasing.
I do not understand what in the cause of CSStorm.
[DB server]*1
Intel Xeon 3.0GHz*4(2CPU * H/T ON)
4GB Memory
Red Hat Enterprise Linux ES release 4(Nahant Update 3)
Linux version 2.6.9-34.ELsmp
PostgreSQL8.1.3 (The version 8.2(head-6/15) was also occurred)
shared_buffers=131072
temp_buffers=1000
max_connections=300
[AP server]*2
200 connection pooling.
TPC-W model workload
[Clinet]*4
TPC-W model workload
(1)
The following discussion were read.
http://archives.postgresql.org/pgsql-hackers/2006-05/msg01003.php
From: Tom Lane <tgl ( at ) sss ( dot ) pgh ( dot ) pa ( dot ) us>
To: josh ( at ) agliodbs ( dot ) com
Subject: Re: Further reduction of bufmgr lock contention
Date: Wed, 24 May 2006 15:25:26 -0400
If there is a patch for investigation or a technique,
would someone show it to me?
(2)
It seems that much sequential scan has occurred at CSStorm.
When reading a tuple, do the visible satisfy check.
it seems to generate the subtransaction for every transaction.
How much is a possibility that
the LWLock to a subtransaction cause CSStorm?
best regards.
--------
Katsuhiko Okano
okano katsuhiko _at_ oss ntt co jp
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: poor performance with Context Switch Storm at TPC-W.
@ 2006-07-11 13:06 Qingqing Zhou <zhouqq@cs.toronto.edu>
parent: Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
2 siblings, 1 reply; 8+ messages in thread
From: Qingqing Zhou @ 2006-07-11 13:06 UTC (permalink / raw)
To: pgsql-hackers
"Katsuhiko Okano" <okano.katsuhiko@oss.ntt.co.jp> wrote
>
> The problem has occurred in my customer.
> poor performance with Context Switch Storm occurred
> with the following composition.
> Usually, CS is about 5000, WIPS=360.
> when CSStorm occurrence, CS is about 100000, WIPS=60 or less.
>
> Intel Xeon 3.0GHz*4(2CPU * H/T ON)
> 4GB Memory
Do you have bgwriter on and what's the parameters? I read a theory somewhere
that bgwriter scan a large portion of memory and cause L1/L2 thrushing, so
with HT on, the other backends sharing the physical processor with it also
get thrashed ... So try to turn bgwriter off or turn HT off see what's the
difference.
Regards,
Qingqing
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: poor performance with Context Switch Storm at TPC-W.
@ 2006-07-11 13:59 Alvaro Herrera <alvherre@commandprompt.com>
parent: Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
2 siblings, 1 reply; 8+ messages in thread
From: Alvaro Herrera @ 2006-07-11 13:59 UTC (permalink / raw)
To: Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>; +Cc: pgsql-hackers
Katsuhiko Okano wrote:
> I suspected conflict of BufMappingLock.
> but, collected results are seen,
> occurrence of CSStorm and the increase of BufMappingLock counts
> seem not to correspond.
> Instead, SubtransControlLock and SubTrans were increasing.
> I do not understand what in the cause of CSStorm.
Please see this thread:
http://archives.postgresql.org/pgsql-hackers/2005-11/msg01547.php
(actually it's a single message AFAICT)
This was applied on the 8.2dev code, so I'm surprised that 8.2dev
behaves the same as 8.1.
Does your problem have any relationship to what's described there?
I also wondered whether the problem may be that the number of SLRU
buffers we use for subtrans is too low. But the number was increased
from the default 8 to 32 in 8.2dev as well. Maybe you could try
increasing that even further; say 128 and see if the problem is still
there. (src/include/access/subtrans.h, NUM_SUBTRANS_BUFFERS).
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: poor performance with Context Switch Storm at TPC-W.
@ 2006-07-11 14:25 Josh Berkus <josh@agliodbs.com>
parent: Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
2 siblings, 0 replies; 8+ messages in thread
From: Josh Berkus @ 2006-07-11 14:25 UTC (permalink / raw)
To: Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>; +Cc: pgsql-hackers
Katsuhiko,
Have you tried turning HT off? HT is not generally considered (even by
Intel) a good idea for database appplications.
--Josh
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: poor performance with Context Switch Storm at TPC-W.
@ 2006-07-12 00:15 Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
parent: Qingqing Zhou <zhouqq@cs.toronto.edu>
0 siblings, 0 replies; 8+ messages in thread
From: Katsuhiko Okano @ 2006-07-12 00:15 UTC (permalink / raw)
To: pgsql-hackers
hello.
> Do you have bgwriter on and what's the parameters? I read a theory somewhere
> that bgwriter scan a large portion of memory and cause L1/L2 thrushing, so
> with HT on, the other backends sharing the physical processor with it also
> get thrashed ... So try to turn bgwriter off or turn HT off see what's the
> difference.
bgwriter is ON.
at postgresql.conf:
> # - Background writer -
>
> bgwriter_delay = 200 # 10-10000 milliseconds between rounds
> bgwriter_lru_percent = 1.0 # 0-100% of LRU buffers scanned/round
> bgwriter_lru_maxpages = 5 # 0-1000 buffers max written/round
> bgwriter_all_percent = 0.333 # 0-100% of all buffers scanned/round
> bgwriter_all_maxpages = 5 # 0-1000 buffers max written/round
I tried turn H/T OFF, but CSStorm occurred.
Usually, CS is about 5000.
when CSStrom occurrence, CS is about 70000.
(CS is a value smaller than the case where H/T is ON.
I think that it is because the performance of CPU fell.)
Regards
--------
Katsuhiko Okano
okano katsuhiko _at_ oss ntt co jp
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: poor performance with Context Switch Storm at TPC-W.
@ 2006-07-14 05:58 Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
parent: Alvaro Herrera <alvherre@commandprompt.com>
0 siblings, 1 reply; 8+ messages in thread
From: Katsuhiko Okano @ 2006-07-14 05:58 UTC (permalink / raw)
To: pgsql-hackers
Hi.
Alvaro Herrera wrote:
> Katsuhiko Okano wrote:
>
> > I suspected conflict of BufMappingLock.
> > but, collected results are seen,
> > occurrence of CSStorm and the increase of BufMappingLock counts
> > seem not to correspond.
> > Instead, SubtransControlLock and SubTrans were increasing.
> > I do not understand what in the cause of CSStorm.
>
> Please see this thread:
> http://archives.postgresql.org/pgsql-hackers/2005-11/msg01547.php
> (actually it's a single message AFAICT)
>
> This was applied on the 8.2dev code, so I'm surprised that 8.2dev
> behaves the same as 8.1.
>
> Does your problem have any relationship to what's described there?
>
Probably it is related.
There is no telling are a thing with the bad method of a lock and
whether it is bad that the number of LRU buffers is simply small.
> I also wondered whether the problem may be that the number of SLRU
> buffers we use for subtrans is too low. But the number was increased
> from the default 8 to 32 in 8.2dev as well. Maybe you could try
> increasing that even further; say 128 and see if the problem is still
> there. (src/include/access/subtrans.h, NUM_SUBTRANS_BUFFERS).
By PostgreSQL8.2, NUM_SUBTRANS_BUFFERS was changed into 128
and recompile and measured again.
NOT occurrence of CSStorm. The value of WIPS was about 400.
(but the value of WIPS fell about to 320 at intervals of 4 to 6 minutes.)
If the number of SLRU buffers is too low,
also in PostgreSQL8.1.4, if the number of buffers is increased
I think that the same result is brought.
(Although the buffer of CLOG or a multi-transaction also increases,
I think that effect is small)
Now, NUM_SLRU_BUFFERS is changed into 128 in PostgreSQL8.1.4
and is under measurement.
regards,
--------
Katsuhiko Okano
okano katsuhiko _at_ oss ntt co jp
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: poor performance with Context Switch Storm at TPC-W.
@ 2006-07-19 15:35 Jim C. Nasby <jnasby@pervasive.com>
parent: Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
0 siblings, 1 reply; 8+ messages in thread
From: Jim C. Nasby @ 2006-07-19 15:35 UTC (permalink / raw)
To: Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>; +Cc: pgsql-hackers
On Fri, Jul 14, 2006 at 02:58:36PM +0900, Katsuhiko Okano wrote:
> NOT occurrence of CSStorm. The value of WIPS was about 400.
> (but the value of WIPS fell about to 320 at intervals of 4 to 6 minutes.)
If you haven't changed checkpoint timeout, this drop-off every 4-6
minutes indicates that you need to make the bgwriter more aggressive.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: poor performance with Context Switch Storm at TPC-W.
@ 2006-07-20 10:07 Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
parent: Jim C. Nasby <jnasby@pervasive.com>
0 siblings, 0 replies; 8+ messages in thread
From: Katsuhiko Okano @ 2006-07-20 10:07 UTC (permalink / raw)
To: pgsql-hackers
"Jim C. Nasby" wrote:
> If you haven't changed checkpoint timeout, this drop-off every 4-6
> minutes indicates that you need to make the bgwriter more aggressive.
I'll say to a customer when proposing and explaining.
Thank you for the information.
Regards,
--------
Katsuhiko Okano
okano katsuhiko _at_ oss ntt co jp
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2006-07-20 10:07 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-07-11 11:09 poor performance with Context Switch Storm at TPC-W. Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
2006-07-11 13:06 ` Qingqing Zhou <zhouqq@cs.toronto.edu>
2006-07-12 00:15 ` Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
2006-07-11 13:59 ` Alvaro Herrera <alvherre@commandprompt.com>
2006-07-14 05:58 ` Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
2006-07-19 15:35 ` Jim C. Nasby <jnasby@pervasive.com>
2006-07-20 10:07 ` Katsuhiko Okano <okano.katsuhiko@oss.ntt.co.jp>
2006-07-11 14:25 ` Josh Berkus <josh@agliodbs.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox