pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: Strange behavior: pgbench and new Linux kernels
5+ messages / 3 participants
[nested] [flat]
* Re: Strange behavior: pgbench and new Linux kernels
@ 2009-03-31 21:35 Kevin Grittner <Kevin.Grittner@wicourts.gov>
0 siblings, 2 replies; 5+ messages in thread
From: Kevin Grittner @ 2009-03-31 21:35 UTC (permalink / raw)
To: Greg Smith <gsmith@gregsmith.com>; pgsql-performance
>>> On Thu, Apr 17, 2008 at 7:26 PM, Greg Smith wrote:
> On this benchmark 2.6.25 is the worst kernel yet:
> It seems I have a lot of work ahead of me here
> to nail down and report what's going on here.
I don't remember seeing a follow-up on this issue from last year.
Are there still any particular kernels to avoid based on this?
Thanks,
-Kevin
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Strange behavior: pgbench and new Linux kernels
@ 2009-03-31 23:35 Greg Smith <gsmith@gregsmith.com>
parent: Kevin Grittner <Kevin.Grittner@wicourts.gov>
1 sibling, 0 replies; 5+ messages in thread
From: Greg Smith @ 2009-03-31 23:35 UTC (permalink / raw)
To: Kevin Grittner <Kevin.Grittner@wicourts.gov>; +Cc: pgsql-performance
On Tue, 31 Mar 2009, Kevin Grittner wrote:
>>>> On Thu, Apr 17, 2008 at 7:26 PM, Greg Smith wrote:
>
>> On this benchmark 2.6.25 is the worst kernel yet:
>
>> It seems I have a lot of work ahead of me here
>> to nail down and report what's going on here.
>
> I don't remember seeing a follow-up on this issue from last year.
> Are there still any particular kernels to avoid based on this?
I never got any confirmation that the patches that came out of my
discussions with the kernel developers were ever merged. I'm in the
middle of a bunch of pgbench tests this week, and one of the things I
planned to try was seeing if the behavior has changed in 2.6.28 or 2.6.29.
I'm speaking about pgbench at the PostgreSQL East conference this weekend
and will have an update by then (along with a new toolchain for automating
large quantities of pgbench tests).
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Strange behavior: pgbench and new Linux kernels
@ 2009-04-04 16:07 Greg Smith <gsmith@gregsmith.com>
parent: Kevin Grittner <Kevin.Grittner@wicourts.gov>
1 sibling, 1 reply; 5+ messages in thread
From: Greg Smith @ 2009-04-04 16:07 UTC (permalink / raw)
To: Kevin Grittner <Kevin.Grittner@wicourts.gov>; +Cc: pgsql-performance
On Tue, 31 Mar 2009, Kevin Grittner wrote:
>>>> On Thu, Apr 17, 2008 at 7:26 PM, Greg Smith wrote:
>
>> On this benchmark 2.6.25 is the worst kernel yet:
>
> I don't remember seeing a follow-up on this issue from last year.
> Are there still any particular kernels to avoid based on this?
I just discovered something really fascinating here. The problem is
strictly limited to when you're connecting via Unix-domain sockets; use
TCP/IP instead, and it goes away.
To refresh everyone's memory here, I reported a problem to the LKML here:
http://lkml.org/lkml/2008/5/21/292 Got some patches and some kernel tweaks
for the scheduler but never a clear resolution for the cause, which kept
anybody from getting too excited about merging anything. Test results
comparing various tweaks on the hardware I'm still using now are at
http://lkml.org/lkml/2008/5/26/288
For example, here's kernel 2.6.25 running pgbench with 50 clients with a
Q6000 processor, demonstrating poor performance--I'd get >20K TPS here
with a pre-CFS kernel:
$ pgbench -S -t 4000 -c 50 -n pgbench
transaction type: SELECT only
scaling factor: 10
query mode: simple
number of clients: 50
number of transactions per client: 4000
number of transactions actually processed: 200000/200000
tps = 8288.047442 (including connections establishing)
tps = 8319.702195 (excluding connections establishing)
If I now execute exactly the same test, but using localhost, performance
returns to normal:
$ pgbench -S -t 4000 -c 50 -n -h localhost pgbench
transaction type: SELECT only
scaling factor: 10
query mode: simple
number of clients: 50
number of transactions per client: 4000
number of transactions actually processed: 200000/200000
tps = 17575.277771 (including connections establishing)
tps = 17724.651090 (excluding connections establishing)
That's 100% repeatable, I ran each test several times each way.
So the new summary here of what I've found is that if:
1) You're running Linux 2.6.23 or greater (confirmed in up to 2.6.26)
2) You connect over a Unix-domain socket
3) Your client count is relatively high (>8 clients/core)
You can expect your pgbench results to tank. Switch to connecting over
TCP/IP to localhost, and everything is fine; it's not quite as fast as the
pre-CFS kernels in some cases, in others it's faster though.
I haven't gotten to testing kernels newer than 2.6.26 yet, when I saw a
17K TPS result during one of my tests on 2.6.25 I screeched to a halt to
isolate this instead.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Strange behavior: pgbench and new Linux kernels
@ 2009-04-04 18:55 Josh Berkus <josh@agliodbs.com>
parent: Greg Smith <gsmith@gregsmith.com>
0 siblings, 1 reply; 5+ messages in thread
From: Josh Berkus @ 2009-04-04 18:55 UTC (permalink / raw)
To: Greg Smith <gsmith@gregsmith.com>; +Cc: Kevin Grittner <Kevin.Grittner@wicourts.gov>; pgsql-performance
On 4/4/09 9:07 AM, Greg Smith wrote:
> On Tue, 31 Mar 2009, Kevin Grittner wrote:
>
>>>>> On Thu, Apr 17, 2008 at 7:26 PM, Greg Smith wrote:
>>
>>> On this benchmark 2.6.25 is the worst kernel yet:
>>
>> I don't remember seeing a follow-up on this issue from last year.
>> Are there still any particular kernels to avoid based on this?
>
> I just discovered something really fascinating here. The problem is
> strictly limited to when you're connecting via Unix-domain sockets; use
> TCP/IP instead, and it goes away.
Have you sent this to any Linux kernel engineers? My experience is that
they're fairly responsive to this sort of thing.
--Josh
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Strange behavior: pgbench and new Linux kernels
@ 2009-04-04 20:11 Greg Smith <gsmith@gregsmith.com>
parent: Josh Berkus <josh@agliodbs.com>
0 siblings, 0 replies; 5+ messages in thread
From: Greg Smith @ 2009-04-04 20:11 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; +Cc: Kevin Grittner <Kevin.Grittner@wicourts.gov>; pgsql-performance
On Sat, 4 Apr 2009, Josh Berkus wrote:
> Have you sent this to any Linux kernel engineers? My experience is that
> they're fairly responsive to this sort of thing.
I'm going to submit an updated report to LKML once I get back from East, I
want to test against the latest kernel first.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2009-04-04 20:11 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2009-03-31 21:35 Re: Strange behavior: pgbench and new Linux kernels Kevin Grittner <Kevin.Grittner@wicourts.gov>
2009-03-31 23:35 ` Greg Smith <gsmith@gregsmith.com>
2009-04-04 16:07 ` Greg Smith <gsmith@gregsmith.com>
2009-04-04 18:55 ` Josh Berkus <josh@agliodbs.com>
2009-04-04 20:11 ` Greg Smith <gsmith@gregsmith.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox