pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedHyperthreading (was: Two identical systems, radically different performance)
11+ messages / 7 participants
[nested] [flat]
* Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-09 20:12 Craig James <cjames@emolecules.com>
0 siblings, 4 replies; 11+ messages in thread
From: Craig James @ 2012-10-09 20:12 UTC (permalink / raw)
To: pgsql-performance
I've confirmed that hyperthreading causes a huge drop in performance on a
2x4-core Intel Xeon E5620 2.40GHz system. The bottom line is:
~3200 TPS max with hyperthreading
~9000 TPS max without hyprethreading
Here are my results.
"Hyprethreads" (Run1) is "out of the box" with hyperthreads enabled. Only
one column of hyperthread results are shown, but prior to today's work I
ran this a dozen or so times, and "Run1" is very representative of all
those runs. I also rebooted and confirmed that rebooting had no effect.
"+NoHyperthreads" (Run2-5) involved rebooting into the BIOS and disabling
hyperthreads.
"+NoAutoVacuum" (Run6-8) means I turned off autovacuum in the
postgresql.conf file and restarted Postgres (with hyperthreads still
disabled).
Hyperthreads +NoHyperthreads +NoAutoVacuum
------------ ---------------------- ----------------
-c -t Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8
5 20000 2733 2152 2352 2398 2769 2767 2777 4463
10 10000 2783 2404 3529 3365 4397 4457 4217 4172
20 5000 3241 3128 3728 5170 5253 5252 4832 8123
30 3333 2987 5699 6180 8173 8259 6435 8225 8123
40 2500 2739 7133 6507 9298 7845 9133 9298 9230
50 2000 2119 5420 5020 8411 5670 9344 7624 8304
I'll be upgrading to 8.4.14 and making more changes to postgres.conf based
on feedback. The server is available for a day or so for more tests if
anyone has suggestions.
Here's how I got these results:
su postgres
unset LANG
export LD_LIBRARY_PATH=/usr/local/pgsql/lib
/usr/local/pgsql/bin/initdb --pgdata=/data/postgres/main \
--xlogdir=/postgres_xlog/xlog --username=postgres
Edit config file:
max_connections = 500
shared_buffers = 1000MB
work_mem = 128MB
synchronous_commit = off
full_page_writes = off
wal_buffers = 256kB
checkpoint_segments = 30
effective_cache_size = 4GB
track_activities = on
track_counts = on
track_functions = none
autovacuum = on
autovacuum_naptime = 5min
escape_string_warning = off
createuser -U postgres test
createdb -U postgres -O test test
pgbench -i -s 100 -U test
for p in "5 20000" "10 10000" "20 5000" "30 3333" "40 2500" "50 2000" ; do
echo
c=`echo $p | cut -d' ' -f1`
t=`echo $p | cut -d' ' -f2`
cmd="pgbench -U test -c $c -t $t"
echo "--------- $cmd ---------"
$cmd
done
The hardware:
CPU: 2x4-core Intex Xeon E5620 2.40 GHz
Memory: 12 GB DDR EC
Disks: 12x500GB disks (Western Digital 7200RPM SATA)
XLOG: 2 disks, RAID1 ext2
PGDATA: 8 disks, RAID10
3WARE 9650SE-12ML with battery-backed cache. The admin tool (tw_cli)
indicates that the battery is charged and the cache is working on both
units.
Linux: 2.6.32-41-server #94-Ubuntu SMP (new server's disk was
actually cloned from old server).
Postgres: 8.4.4
Craig
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-09 20:53 Tom Lane <tgl@sss.pgh.pa.us>
parent: Craig James <cjames@emolecules.com>
3 siblings, 0 replies; 11+ messages in thread
From: Tom Lane @ 2012-10-09 20:53 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
Craig James <cjames@emolecules.com> writes:
> I've confirmed that hyperthreading causes a huge drop in performance on a
> 2x4-core Intel Xeon E5620 2.40GHz system. The bottom line is:
Interesting.
> I'll be upgrading to 8.4.14 and making more changes to postgres.conf based
> on feedback. The server is available for a day or so for more tests if
> anyone has suggestions.
It would be nice to see similar tests done with 9.2. 8.4 is kind of old
news as far as server performance is concerned.
regards, tom lane
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-09 20:56 Shaun Thomas <sthomas@optionshouse.com>
parent: Craig James <cjames@emolecules.com>
3 siblings, 1 reply; 11+ messages in thread
From: Shaun Thomas @ 2012-10-09 20:56 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On 10/09/2012 03:12 PM, Craig James wrote:
> ~3200 TPS max with hyperthreading
> ~9000 TPS max without hyprethreading
That's really odd. We got almost the opposite effect on our X5645's.
Also, there's no way your RAID is sustaining 9000TPS. Something here
sounds fishy.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-09 20:56 Josh Berkus <josh@agliodbs.com>
parent: Craig James <cjames@emolecules.com>
3 siblings, 0 replies; 11+ messages in thread
From: Josh Berkus @ 2012-10-09 20:56 UTC (permalink / raw)
To: pgsql-performance
On 10/9/12 1:12 PM, Craig James wrote:
> I've confirmed that hyperthreading causes a huge drop in performance on a
> 2x4-core Intel Xeon E5620 2.40GHz system. The bottom line is:
>
> ~3200 TPS max with hyperthreading
> ~9000 TPS max without hyprethreading
Oh, interesting! This is the first time I've seen results like this on
a later Linux kernel and processor. I thought Intel had licked the
"hyperthreading penalty".
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-09 21:24 Jeff Janes <jeff.janes@gmail.com>
parent: Shaun Thomas <sthomas@optionshouse.com>
0 siblings, 0 replies; 11+ messages in thread
From: Jeff Janes @ 2012-10-09 21:24 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On Tue, Oct 9, 2012 at 1:56 PM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> On 10/09/2012 03:12 PM, Craig James wrote:
>
>> ~3200 TPS max with hyperthreading
>> ~9000 TPS max without hyprethreading
>
>
> That's really odd. We got almost the opposite effect on our X5645's.
>
> Also, there's no way your RAID is sustaining 9000TPS. Something here sounds
> fishy.
synchronous-commit=off
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-09 23:30 Craig James <cjames@emolecules.com>
parent: Craig James <cjames@emolecules.com>
3 siblings, 1 reply; 11+ messages in thread
From: Craig James @ 2012-10-09 23:30 UTC (permalink / raw)
To: pgsql-performance
For your amusement ... I upgraded from 8.4.4 to 9.2.1 results. Threw away
the DB completely and did a new init. Same hardware, postgres.conf and
Linux as before.
ra is "blockdev --getra" (both PGDATA and XLOG disks)
walb is postgres.conf "wal_buffers"
ra:8192 walb:1M ra:256 walb:1M ra:256 walb:256kB
---------------- ---------------- -----------------
-c -t Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9
5 20000 1963 2103 2145 2292 2312 2353 2296 2175 2294
10 10000 2587 2749 2762 3252 3265 3276 3267 3228 3263
20 5000 3449 3578 3438 4910 4958 4949 4949 4927 4943
30 3333 4222 3731 3992 6929 6924 6562 6754 6995 6869
40 2500 4261 3722 4243 9286 9240 5712 9310 8530 8872
50 2000 4138 4399 3865 9213 9351 9578 8011 7651 8362
Unfortunately, distance prevents me from going to the co-location facility
and trying this with hyperthreading turned back on.
Craig
On Tue, Oct 9, 2012 at 1:12 PM, Craig James <cjames@emolecules.com> wrote:
> I've confirmed that hyperthreading causes a huge drop in performance on a
> 2x4-core Intel Xeon E5620 2.40GHz system. The bottom line is:
>
> ~3200 TPS max with hyperthreading
> ~9000 TPS max without hyprethreading
>
> Here are my results.
>
> "Hyprethreads" (Run1) is "out of the box" with hyperthreads enabled. Only
> one column of hyperthread results are shown, but prior to today's work I
> ran this a dozen or so times, and "Run1" is very representative of all
> those runs. I also rebooted and confirmed that rebooting had no effect.
>
> "+NoHyperthreads" (Run2-5) involved rebooting into the BIOS and disabling
> hyperthreads.
>
> "+NoAutoVacuum" (Run6-8) means I turned off autovacuum in the
> postgresql.conf file and restarted Postgres (with hyperthreads still
> disabled).
>
> Hyperthreads +NoHyperthreads +NoAutoVacuum
> ------------ ---------------------- ----------------
> -c -t Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8
> 5 20000 2733 2152 2352 2398 2769 2767 2777 4463
> 10 10000 2783 2404 3529 3365 4397 4457 4217 4172
> 20 5000 3241 3128 3728 5170 5253 5252 4832 8123
> 30 3333 2987 5699 6180 8173 8259 6435 8225 8123
> 40 2500 2739 7133 6507 9298 7845 9133 9298 9230
> 50 2000 2119 5420 5020 8411 5670 9344 7624 8304
>
> I'll be upgrading to 8.4.14 and making more changes to postgres.conf based
> on feedback. The server is available for a day or so for more tests if
> anyone has suggestions.
>
> Here's how I got these results:
>
> su postgres
> unset LANG
> export LD_LIBRARY_PATH=/usr/local/pgsql/lib
> /usr/local/pgsql/bin/initdb --pgdata=/data/postgres/main \
> --xlogdir=/postgres_xlog/xlog --username=postgres
>
> Edit config file:
>
> max_connections = 500
> shared_buffers = 1000MB
> work_mem = 128MB
> synchronous_commit = off
> full_page_writes = off
> wal_buffers = 256kB
> checkpoint_segments = 30
> effective_cache_size = 4GB
> track_activities = on
> track_counts = on
> track_functions = none
> autovacuum = on
> autovacuum_naptime = 5min
> escape_string_warning = off
>
> createuser -U postgres test
> createdb -U postgres -O test test
>
> pgbench -i -s 100 -U test
> for p in "5 20000" "10 10000" "20 5000" "30 3333" "40 2500" "50 2000" ; do
> echo
> c=`echo $p | cut -d' ' -f1`
> t=`echo $p | cut -d' ' -f2`
> cmd="pgbench -U test -c $c -t $t"
> echo "--------- $cmd ---------"
> $cmd
> done
>
> The hardware:
>
> CPU: 2x4-core Intex Xeon E5620 2.40 GHz
>
> Memory: 12 GB DDR EC
>
> Disks: 12x500GB disks (Western Digital 7200RPM SATA)
> XLOG: 2 disks, RAID1 ext2
> PGDATA: 8 disks, RAID10
>
> 3WARE 9650SE-12ML with battery-backed cache. The admin tool (tw_cli)
> indicates that the battery is charged and the cache is working on both
> units.
>
> Linux: 2.6.32-41-server #94-Ubuntu SMP (new server's disk was
> actually cloned from old server).
>
> Postgres: 8.4.4
>
> Craig
>
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-10 12:52 Shaun Thomas <sthomas@optionshouse.com>
parent: Craig James <cjames@emolecules.com>
0 siblings, 1 reply; 11+ messages in thread
From: Shaun Thomas @ 2012-10-10 12:52 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On 10/09/2012 06:30 PM, Craig James wrote:
> ra:8192 walb:1M ra:256 walb:1M ra:256 walb:256kB
> ---------------- ---------------- -----------------
> -c -t Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9
> 40 2500 4261 3722 4243 9286 9240 5712 9310 8530 8872
> 50 2000 4138 4399 3865 9213 9351 9578 8011 7651 8362
I think I speak for more than a few people here when I say: wat.
About the only thing I can ask, is: did you make these tests fair? And
by fair, I mean:
echo 3 > /proc/sys/vm/drop_caches
pg_ctl -D /your/pg/dir restart
Between every test to make sure shared buffers and the OS inode cache
was empty before the start of each test? If you're using that bash-style
for-loop you attached earlier, probably not. Still though, I don't think
that would account for this much variance between having read-ahead at
8M as opposed to 256kb.
My head hurts.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-10 14:44 Claudio Freire <klaussfreire@gmail.com>
parent: Shaun Thomas <sthomas@optionshouse.com>
0 siblings, 3 replies; 11+ messages in thread
From: Claudio Freire @ 2012-10-10 14:44 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On Wed, Oct 10, 2012 at 9:52 AM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> On 10/09/2012 06:30 PM, Craig James wrote:
>
>> ra:8192 walb:1M ra:256 walb:1M ra:256 walb:256kB
>> ---------------- ---------------- -----------------
>> -c -t Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9
>> 40 2500 4261 3722 4243 9286 9240 5712 9310 8530 8872
>> 50 2000 4138 4399 3865 9213 9351 9578 8011 7651 8362
>
>
> I think I speak for more than a few people here when I say: wat.
>
> About the only thing I can ask, is: did you make these tests fair? And by
> fair, I mean:
>
> echo 3 > /proc/sys/vm/drop_caches
> pg_ctl -D /your/pg/dir restart
Yes, I was thinking the same. Especially if you check the tendency to
perform better in higher-numbered runs. But, as you said, that doesn't
explain that jump to twice the TPS. I was thinking, and I'm not
pgbench expert, could it be that the database grows from run to run,
changing performance characteristics?
> My head hurts.
I'm just confused. No headache yet.
But really interesting numbers in any case. It these results are on
the level, then maybe the kernel's read-ahead algorithm isn't as
fool-proof as we thought? Gotta read the source. BRB
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-10 15:46 Bruce Momjian <bruce@momjian.us>
parent: Claudio Freire <klaussfreire@gmail.com>
2 siblings, 0 replies; 11+ messages in thread
From: Bruce Momjian @ 2012-10-10 15:46 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: sthomas@optionshouse.com, Craig James <cjames@emolecules.com>; pgsql-performance
On Wed, Oct 10, 2012 at 11:44:50AM -0300, Claudio Freire wrote:
> On Wed, Oct 10, 2012 at 9:52 AM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> > On 10/09/2012 06:30 PM, Craig James wrote:
> >
> >> ra:8192 walb:1M ra:256 walb:1M ra:256 walb:256kB
> >> ---------------- ---------------- -----------------
> >> -c -t Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9
> >> 40 2500 4261 3722 4243 9286 9240 5712 9310 8530 8872
> >> 50 2000 4138 4399 3865 9213 9351 9578 8011 7651 8362
> >
> >
> > I think I speak for more than a few people here when I say: wat.
> >
> > About the only thing I can ask, is: did you make these tests fair? And by
> > fair, I mean:
> >
> > echo 3 > /proc/sys/vm/drop_caches
> > pg_ctl -D /your/pg/dir restart
>
> Yes, I was thinking the same. Especially if you check the tendency to
> perform better in higher-numbered runs. But, as you said, that doesn't
> explain that jump to twice the TPS. I was thinking, and I'm not
> pgbench expert, could it be that the database grows from run to run,
> changing performance characteristics?
>
> > My head hurts.
>
> I'm just confused. No headache yet.
>
> But really interesting numbers in any case. It these results are on
> the level, then maybe the kernel's read-ahead algorithm isn't as
> fool-proof as we thought? Gotta read the source. BRB
Well, I have exactly the same setup here:
new: 2x4-core Intex Xeon E5620 2.40 GHz
Let me know if you want any tests run, on SSDs or magnetic disk. I do
have hyperthreading enabled, and Greg Smith benchmarked my server and
said it was good.
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-10 17:28 Craig James <cjames@emolecules.com>
parent: Claudio Freire <klaussfreire@gmail.com>
2 siblings, 0 replies; 11+ messages in thread
From: Craig James @ 2012-10-10 17:28 UTC (permalink / raw)
To: pgsql-performance
Sent this to Claudio rather than the whole list ... here it is.
On Wed, Oct 10, 2012 at 7:44 AM, Claudio Freire <klaussfreire@gmail.com>wrote:
> On Wed, Oct 10, 2012 at 9:52 AM, Shaun Thomas <sthomas@optionshouse.com>
> wrote:
> > On 10/09/2012 06:30 PM, Craig James wrote:
> >
> >> ra:8192 walb:1M ra:256 walb:1M ra:256 walb:256kB
> >> ---------------- ---------------- -----------------
> >> -c -t Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9
> >> 40 2500 4261 3722 4243 9286 9240 5712 9310 8530 8872
> >> 50 2000 4138 4399 3865 9213 9351 9578 8011 7651 8362
> >
> >
> > I think I speak for more than a few people here when I say: wat.
> >
> > About the only thing I can ask, is: did you make these tests fair? And by
> > fair, I mean:
> >
> > echo 3 > /proc/sys/vm/drop_caches
> > pg_ctl -D /your/pg/dir restart
>
>
I showed the exact commands I used -- if it's not there, I didn't do it.
So the answer is no, I didn't drop caches.
On the other hand, I wanted to know what happened on cold start and after
running for a while. Running pgbench once isn't as interesting as running
it three times.
> Yes, I was thinking the same. Especially if you check the tendency to
> perform better in higher-numbered runs. But, as you said, that doesn't
> explain that jump to twice the TPS. I was thinking, and I'm not
> pgbench expert, could it be that the database grows from run to run,
> changing performance characteristics?
>
> > My head hurts.
>
> I'm just confused. No headache yet.
>
> But really interesting numbers in any case. It these results are on
> the level, then maybe the kernel's read-ahead algorithm isn't as
> fool-proof as we thought? Gotta read the source. BRB
>
Big numbers, little numbers ... I'm just reporting what pgbench tells me
and how I got them. I'm good at chemical databases, you guys are the
Postgres performance experts.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: Hyperthreading (was: Two identical systems, radically different performance)
@ 2012-10-10 18:08 Claudio Freire <klaussfreire@gmail.com>
parent: Claudio Freire <klaussfreire@gmail.com>
2 siblings, 0 replies; 11+ messages in thread
From: Claudio Freire @ 2012-10-10 18:08 UTC (permalink / raw)
To: sthomas@optionshouse.com; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On Wed, Oct 10, 2012 at 11:44 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
> On Wed, Oct 10, 2012 at 9:52 AM, Shaun Thomas <sthomas@optionshouse.com> wrote:
>> On 10/09/2012 06:30 PM, Craig James wrote:
>>
>>> ra:8192 walb:1M ra:256 walb:1M ra:256 walb:256kB
>>> ---------------- ---------------- -----------------
>>> -c -t Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9
>>> 40 2500 4261 3722 4243 9286 9240 5712 9310 8530 8872
>>> 50 2000 4138 4399 3865 9213 9351 9578 8011 7651 8362
>>
...
> But really interesting numbers in any case. It these results are on
> the level, then maybe the kernel's read-ahead algorithm isn't as
> fool-proof as we thought? Gotta read the source. BRB
So, I've been digging.
Newer kernels (above 2.6.23) have a new read-ahead algorithm, called
the "on-demand" read ahead. Benchmarks on this new code[0] suggest
there is a penalty for random writes that wasn't there before. This
penalty is small in the benchmarks (less than 10% in all cases), but I
would imagine the issue would be amplified in the case of a kernel
with a misconfigured bg_writer (this case, IIRC).
This makes sense, in fact. Back when I tested the 8MB read-ahead on my
server, it was using 2.6.18. Now, 2.6.23 is in fact rather old, so
those benchmarks may no longer be relevant. There are tons of commits
since then[1], though only one pertaining writes from what I can tell.
However, I'm inclined to blame the bg_writer. How about tweaking the
dirty_background_ratio and dirty_ratio and re-trying?
[0] http://lwn.net/Articles/235181/
[1] https://github.com/torvalds/linux/commits/f5a246eab9a268f51ba8189ea5b098a1bfff200e/mm/readahead.c?pa...
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2012-10-10 18:08 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-10-09 20:12 Hyperthreading (was: Two identical systems, radically different performance) Craig James <cjames@emolecules.com>
2012-10-09 20:53 ` Tom Lane <tgl@sss.pgh.pa.us>
2012-10-09 20:56 ` Shaun Thomas <sthomas@optionshouse.com>
2012-10-09 21:24 ` Jeff Janes <jeff.janes@gmail.com>
2012-10-09 20:56 ` Josh Berkus <josh@agliodbs.com>
2012-10-09 23:30 ` Craig James <cjames@emolecules.com>
2012-10-10 12:52 ` Shaun Thomas <sthomas@optionshouse.com>
2012-10-10 14:44 ` Claudio Freire <klaussfreire@gmail.com>
2012-10-10 15:46 ` Bruce Momjian <bruce@momjian.us>
2012-10-10 17:28 ` Craig James <cjames@emolecules.com>
2012-10-10 18:08 ` Claudio Freire <klaussfreire@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox