pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedTwo identical systems, radically different performance
47+ messages / 15 participants
[nested] [flat]
* Two identical systems, radically different performance
@ 2012-10-08 21:45 Craig James <cjames@emolecules.com>
0 siblings, 6 replies; 47+ messages in thread
From: Craig James @ 2012-10-08 21:45 UTC (permalink / raw)
To: pgsql-performance
This is driving me crazy. A new server, virtually identical to an old one,
has 50% of the performance with pgbench. I've checked everything I can
think of.
The setups (call the servers "old" and "new"):
old: 2 x 4-core Intel Xeon E5620
new: 4 x 4-core Intel Xeon E5606
both:
memory: 12 GB DDR EC
Disks: 12x500GB disks (Western Digital 7200RPM SATA)
2 disks, RAID1: OS (ext4) and postgres xlog (ext2)
8 disks, RAID10: $PGDATA
3WARE 9650SE-12ML with battery-backed cache. The admin tool (tw_cli)
indicates that the battery is charged and the cache is working on both
units.
Linux: 2.6.32-41-server #94-Ubuntu SMP (new server's disk was
actually cloned from old server).
Postgres: 8.4.4 (yes, I should update. But both are identical.)
The postgres.conf files are identical; diffs from the original are:
max_connections = 500
shared_buffers = 1000MB
work_mem = 128MB
synchronous_commit = off
full_page_writes = off
wal_buffers = 256kB
checkpoint_segments = 30
effective_cache_size = 4GB
track_activities = on
track_counts = on
track_functions = none
autovacuum = on
autovacuum_naptime = 5min
escape_string_warning = off
Note that the old server is in production and was serving a light load
while this test was running, so in theory it should be slower, not faster,
than the new server.
pgbench: Old server
pgbench -i -s 100 -U test
pgbench -U test -c ... -t ...
-c -t TPS
5 20000 3777
10 10000 2622
20 5000 3759
30 3333 5712
40 2500 5953
50 2000 6141
New server
-c -t TPS
5 20000 2733
10 10000 2783
20 5000 3241
30 3333 2987
40 2500 2739
50 2000 2119
As you can see, the new server is dramatically slower than the old one.
I tested both the RAID10 data disk and the RAID1 xlog disk with bonnie++.
The xlog disks were almost identical in performance. The RAID10 pg-data
disks looked like this:
Old server:
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
xenon 24064M 687 99 203098 26 81904 16 3889 96 403747 31
737.6 31
Latency 20512us 469ms 394ms 21402us 396ms
112ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
xenon -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 15953 27 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++
Latency 43291us 857us 519us 1588us 37us
178us
1.96,1.96,xenon,1,1349726125,24064M,,687,99,203098,26,81904,16,3889,96,403747,31,737.6,31,16,,,,,15953,27,+++++,+++,+++++,++\
+,+++++,+++,+++++,+++,+++++,+++,20512us,469ms,394ms,21402us,396ms,112ms,43291us,857us,519us,1588us,37us,178us
New server:
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
zinc 24064M 862 99 212143 54 96008 14 4921 99 279239 17
752.0 23
Latency 15613us 598ms 597ms 2764us 398ms
215ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
zinc -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 20380 26 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++
Latency 487us 627us 407us 972us 29us
262us
1.96,1.96,zinc,1,1349722017,24064M,,862,99,212143,54,96008,14,4921,99,279239,17,752.0,23,16,,,,,20380,26,+++++,+++,+++++,+++\
,+++++,+++,+++++,+++,+++++,+++,15613us,598ms,597ms,2764us,398ms,215ms,487us,627us,407us,972us,29us,262us
I don't know enough about bonnie++ to know if these differences are
interesting.
One dramatic difference I noted via vmstat. On the old server, the I/O
load during the bonnie++ run was steady, like this:
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
r b swpd free buff cache si so bi bo in cs us sy id
wa
0 2 71800 2117612 17940 9375660 0 0 82948 81944 1992 1341 1 3
86 10
0 2 71800 2113328 17948 9383896 0 0 76288 75806 1751 1167 0 2
86 11
0 1 71800 2111004 17948 9386540 92 0 93324 94232 2230 1510 0 4
86 10
0 1 71800 2106796 17948 9387436 114 0 67698 67588 1572 1088 0 2
87 11
0 1 71800 2106724 17956 9387968 50 0 81970 85710 1918 1287 0 3
86 10
1 1 71800 2103304 17956 9390700 0 0 92096 92160 1970 1194 0 4
86 10
0 2 71800 2103196 17976 9389204 0 0 70722 69680 1655 1116 1 3
86 10
1 1 71800 2099064 17980 9390824 0 0 57346 57348 1357 949 0 2
87 11
0 1 71800 2095596 17980 9392720 0 0 57344 57348 1379 987 0 2
86 12
But the new server varied wildly during bonnie++:
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
0 1 0 4518352 12004 7167000 0 0 118894 120838 2613 1539 0 2
93 5
0 1 0 4517252 12004 7167824 0 0 52116 53248 1179 793 0 1
94 5
0 1 0 4515864 12004 7169088 0 0 46764 49152 1104 733 0 1
91 7
0 1 0 4515180 12012 7169764 0 0 32924 30724 750 542 0 1
93 6
0 1 0 4514328 12016 7170780 0 0 42188 45056 1019 664 0 1
90 9
0 1 0 4513072 12016 7171856 0 0 67528 65540 1487 993 0 1
96 4
0 1 0 4510852 12016 7173160 0 0 56876 57344 1358 942 0 1
94 5
0 1 0 4500280 12044 7179924 0 0 91564 94220 2505 2504 1 2
91 6
0 1 0 4495564 12052 7183492 0 0 102660 104452 2289 1473 0 2
92 6
0 1 0 4492092 12052 7187720 0 0 98498 96274 2140 1385 0 2
93 5
0 1 0 4488608 12060 7190772 0 0 97628 100358 2176 1398 0 1
94 4
1 0 0 4485880 12052 7192600 0 0 112406 114686 2461 1509 0 3
90 7
1 0 0 4483424 12052 7195612 0 0 64678 65536 1449 948 0 1
91 8
0 1 0 4480252 12052 7199404 0 0 99608 100356 2217 1452 0 1
96 3
Any ideas where to look next would be greatly appreciated.
Craig
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 21:57 Evgeny Shishkin <itparanoia@gmail.com>
parent: Craig James <cjames@emolecules.com>
5 siblings, 1 reply; 47+ messages in thread
From: Evgeny Shishkin @ 2012-10-08 21:57 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On Oct 9, 2012, at 1:45 AM, Craig James <cjames@emolecules.com> wrote:
> This is driving me crazy. A new server, virtually identical to an old one, has 50% of the performance with pgbench. I've checked everything I can think of.
>
> The setups (call the servers "old" and "new"):
>
> old: 2 x 4-core Intel Xeon E5620
> new: 4 x 4-core Intel Xeon E5606
>
> both:
>
> memory: 12 GB DDR EC
> Disks: 12x500GB disks (Western Digital 7200RPM SATA)
> 2 disks, RAID1: OS (ext4) and postgres xlog (ext2)
> 8 disks, RAID10: $PGDATA
>
> 3WARE 9650SE-12ML with battery-backed cache. The admin tool (tw_cli)
> indicates that the battery is charged and the cache is working on both units.
>
> Linux: 2.6.32-41-server #94-Ubuntu SMP (new server's disk was
> actually cloned from old server).
>
> Postgres: 8.4.4 (yes, I should update. But both are identical.)
>
> The postgres.conf files are identical; diffs from the original are:
>
> max_connections = 500
> shared_buffers = 1000MB
> work_mem = 128MB
> synchronous_commit = off
> full_page_writes = off
> wal_buffers = 256kB
wal buffers seems very small. Simon suggests to set them at least to 16MB.
> checkpoint_segments = 30
> effective_cache_size = 4GB
You have 12Gb RAM.
> track_activities = on
> track_counts = on
> track_functions = none
> autovacuum = on
> autovacuum_naptime = 5min
> escape_string_warning = off
>
> Note that the old server is in production and was serving a light load while this test was running, so in theory it should be slower, not faster, than the new server.
>
> pgbench: Old server
>
> pgbench -i -s 100 -U test
> pgbench -U test -c ... -t ...
>
> -c -t TPS
> 5 20000 3777
> 10 10000 2622
> 20 5000 3759
> 30 3333 5712
> 40 2500 5953
> 50 2000 6141
>
> New server
> -c -t TPS
> 5 20000 2733
> 10 10000 2783
> 20 5000 3241
> 30 3333 2987
> 40 2500 2739
> 50 2000 2119
>
> As you can see, the new server is dramatically slower than the old one.
>
> I tested both the RAID10 data disk and the RAID1 xlog disk with bonnie++. The xlog disks were almost identical in performance. The RAID10 pg-data disks looked like this:
>
> Old server:
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> xenon 24064M 687 99 203098 26 81904 16 3889 96 403747 31 737.6 31
> Latency 20512us 469ms 394ms 21402us 396ms 112ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> xenon -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 15953 27 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
> Latency 43291us 857us 519us 1588us 37us 178us
> 1.96,1.96,xenon,1,1349726125,24064M,,687,99,203098,26,81904,16,3889,96,403747,31,737.6,31,16,,,,,15953,27,+++++,+++,+++++,++\
> +,+++++,+++,+++++,+++,+++++,+++,20512us,469ms,394ms,21402us,396ms,112ms,43291us,857us,519us,1588us,37us,178us
>
>
> New server:
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> zinc 24064M 862 99 212143 54 96008 14 4921 99 279239 17 752.0 23
> Latency 15613us 598ms 597ms 2764us 398ms 215ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> zinc -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 20380 26 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
> Latency 487us 627us 407us 972us 29us 262us
> 1.96,1.96,zinc,1,1349722017,24064M,,862,99,212143,54,96008,14,4921,99,279239,17,752.0,23,16,,,,,20380,26,+++++,+++,+++++,+++\
> ,+++++,+++,+++++,+++,+++++,+++,15613us,598ms,597ms,2764us,398ms,215ms,487us,627us,407us,972us,29us,262us
Sequential Input on the new one is 279MB/s, on the old 400MB/s.
> I don't know enough about bonnie++ to know if these differences are interesting.
>
> One dramatic difference I noted via vmstat. On the old server, the I/O load during the bonnie++ run was steady, like this:
>
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 2 71800 2117612 17940 9375660 0 0 82948 81944 1992 1341 1 3 86 10
> 0 2 71800 2113328 17948 9383896 0 0 76288 75806 1751 1167 0 2 86 11
> 0 1 71800 2111004 17948 9386540 92 0 93324 94232 2230 1510 0 4 86 10
> 0 1 71800 2106796 17948 9387436 114 0 67698 67588 1572 1088 0 2 87 11
> 0 1 71800 2106724 17956 9387968 50 0 81970 85710 1918 1287 0 3 86 10
> 1 1 71800 2103304 17956 9390700 0 0 92096 92160 1970 1194 0 4 86 10
> 0 2 71800 2103196 17976 9389204 0 0 70722 69680 1655 1116 1 3 86 10
> 1 1 71800 2099064 17980 9390824 0 0 57346 57348 1357 949 0 2 87 11
> 0 1 71800 2095596 17980 9392720 0 0 57344 57348 1379 987 0 2 86 12
>
> But the new server varied wildly during bonnie++:
>
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 1 0 4518352 12004 7167000 0 0 118894 120838 2613 1539 0 2 93 5
> 0 1 0 4517252 12004 7167824 0 0 52116 53248 1179 793 0 1 94 5
> 0 1 0 4515864 12004 7169088 0 0 46764 49152 1104 733 0 1 91 7
> 0 1 0 4515180 12012 7169764 0 0 32924 30724 750 542 0 1 93 6
> 0 1 0 4514328 12016 7170780 0 0 42188 45056 1019 664 0 1 90 9
> 0 1 0 4513072 12016 7171856 0 0 67528 65540 1487 993 0 1 96 4
> 0 1 0 4510852 12016 7173160 0 0 56876 57344 1358 942 0 1 94 5
> 0 1 0 4500280 12044 7179924 0 0 91564 94220 2505 2504 1 2 91 6
> 0 1 0 4495564 12052 7183492 0 0 102660 104452 2289 1473 0 2 92 6
> 0 1 0 4492092 12052 7187720 0 0 98498 96274 2140 1385 0 2 93 5
> 0 1 0 4488608 12060 7190772 0 0 97628 100358 2176 1398 0 1 94 4
> 1 0 0 4485880 12052 7192600 0 0 112406 114686 2461 1509 0 3 90 7
> 1 0 0 4483424 12052 7195612 0 0 64678 65536 1449 948 0 1 91 8
> 0 1 0 4480252 12052 7199404 0 0 99608 100356 2217 1452 0 1 96 3
>
> Any ideas where to look next would be greatly appreciated.
>
> Craig
>
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:06 Craig James <cjames@emolecules.com>
parent: Evgeny Shishkin <itparanoia@gmail.com>
0 siblings, 2 replies; 47+ messages in thread
From: Craig James @ 2012-10-08 22:06 UTC (permalink / raw)
To: Evgeny Shishkin <itparanoia@gmail.com>; +Cc: pgsql-performance
On Mon, Oct 8, 2012 at 2:57 PM, Evgeny Shishkin <itparanoia@gmail.com>wrote:
>
> On Oct 9, 2012, at 1:45 AM, Craig James <cjames@emolecules.com> wrote:
>
> I tested both the RAID10 data disk and the RAID1 xlog disk with bonnie++.
> The xlog disks were almost identical in performance. The RAID10 pg-data
> disks looked like this:
>
> Old server:
> Version 1.96 ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> xenon 24064M 687 99 203098 26 81904 16 3889 96 403747 31
> 737.6 31
> Latency 20512us 469ms 394ms 21402us 396ms
> 112ms
> Version 1.96 ------Sequential Create------ --------Random
> Create--------
> xenon -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 15953 27 +++++ +++ +++++ +++ +++++ +++ +++++ +++
> +++++ +++
> Latency 43291us 857us 519us 1588us 37us
> 178us
>
> 1.96,1.96,xenon,1,1349726125,24064M,,687,99,203098,26,81904,16,3889,96,403747,31,737.6,31,16,,,,,15953,27,+++++,+++,+++++,++\
>
> +,+++++,+++,+++++,+++,+++++,+++,20512us,469ms,394ms,21402us,396ms,112ms,43291us,857us,519us,1588us,37us,178us
>
>
> New server:
> Version 1.96 ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> zinc 24064M 862 99 212143 54 96008 14 4921 99 279239 17
> 752.0 23
> Latency 15613us 598ms 597ms 2764us 398ms
> 215ms
> Version 1.96 ------Sequential Create------ --------Random
> Create--------
> zinc -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 20380 26 +++++ +++ +++++ +++ +++++ +++ +++++ +++
> +++++ +++
> Latency 487us 627us 407us 972us 29us
> 262us
>
> 1.96,1.96,zinc,1,1349722017,24064M,,862,99,212143,54,96008,14,4921,99,279239,17,752.0,23,16,,,,,20380,26,+++++,+++,+++++,+++\
>
> ,+++++,+++,+++++,+++,+++++,+++,15613us,598ms,597ms,2764us,398ms,215ms,487us,627us,407us,972us,29us,262us
>
>
> Sequential Input on the new one is 279MB/s, on the old 400MB/s.
>
>
But why? What have I overlooked?
Thanks,
Craig
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:08 Evgeny Shishkin <itparanoia@gmail.com>
parent: Craig James <cjames@emolecules.com>
1 sibling, 0 replies; 47+ messages in thread
From: Evgeny Shishkin @ 2012-10-08 22:08 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On Oct 9, 2012, at 2:06 AM, Craig James <cjames@emolecules.com> wrote:
>
>
> On Mon, Oct 8, 2012 at 2:57 PM, Evgeny Shishkin <itparanoia@gmail.com> wrote:
>
> On Oct 9, 2012, at 1:45 AM, Craig James <cjames@emolecules.com> wrote:
>
>> I tested both the RAID10 data disk and the RAID1 xlog disk with bonnie++. The xlog disks were almost identical in performance. The RAID10 pg-data disks looked like this:
>>
>> Old server:
>> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
>> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
>> xenon 24064M 687 99 203098 26 81904 16 3889 96 403747 31 737.6 31
>> Latency 20512us 469ms 394ms 21402us 396ms 112ms
>> Version 1.96 ------Sequential Create------ --------Random Create--------
>> xenon -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
>> 16 15953 27 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
>> Latency 43291us 857us 519us 1588us 37us 178us
>> 1.96,1.96,xenon,1,1349726125,24064M,,687,99,203098,26,81904,16,3889,96,403747,31,737.6,31,16,,,,,15953,27,+++++,+++,+++++,++\
>> +,+++++,+++,+++++,+++,+++++,+++,20512us,469ms,394ms,21402us,396ms,112ms,43291us,857us,519us,1588us,37us,178us
>>
>>
>> New server:
>> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
>> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
>> zinc 24064M 862 99 212143 54 96008 14 4921 99 279239 17 752.0 23
>> Latency 15613us 598ms 597ms 2764us 398ms 215ms
>> Version 1.96 ------Sequential Create------ --------Random Create--------
>> zinc -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
>> 16 20380 26 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
>> Latency 487us 627us 407us 972us 29us 262us
>> 1.96,1.96,zinc,1,1349722017,24064M,,862,99,212143,54,96008,14,4921,99,279239,17,752.0,23,16,,,,,20380,26,+++++,+++,+++++,+++\
>> ,+++++,+++,+++++,+++,+++++,+++,15613us,598ms,597ms,2764us,398ms,215ms,487us,627us,407us,972us,29us,262us
>
> Sequential Input on the new one is 279MB/s, on the old 400MB/s.
>
>
> But why? What have I overlooked?
blockdev --setra 32000 ?
Also you benchmarked volume for pgdata? Can you provide benchmarks for wal volume?
>
> Thanks,
> Craig
>
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:09 Claudio Freire <klaussfreire@gmail.com>
parent: Craig James <cjames@emolecules.com>
1 sibling, 1 reply; 47+ messages in thread
From: Claudio Freire @ 2012-10-08 22:09 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: Evgeny Shishkin <itparanoia@gmail.com>; pgsql-performance
On Mon, Oct 8, 2012 at 7:06 PM, Craig James <cjames@emolecules.com> wrote:
>> Sequential Input on the new one is 279MB/s, on the old 400MB/s.
>>
>
> But why? What have I overlooked?
Do you have readahead properly set up on the new one?
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:16 Steve Crawford <scrawford@pinpointresearch.com>
parent: Craig James <cjames@emolecules.com>
5 siblings, 0 replies; 47+ messages in thread
From: Steve Crawford @ 2012-10-08 22:16 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On 10/08/2012 02:45 PM, Craig James wrote:
> This is driving me crazy. A new server, virtually identical to an old
> one, has 50% of the performance with pgbench. I've checked everything
> I can think of.
>
> The setups (call the servers "old" and "new"):
>
> old: 2 x 4-core Intel Xeon E5620
> new: 4 x 4-core Intel Xeon E5606
>
> both:
>
> memory: 12 GB DDR EC
> Disks: 12x500GB disks (Western Digital 7200RPM SATA)
> 2 disks, RAID1: OS (ext4) and postgres xlog (ext2)
> 8 disks, RAID10: $PGDATA
Exact same model of disk, same on-board cache, same RAID-card RAM size,
same RAID strip-size, etc.??
Cheers,
Steve
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:25 Craig James <cjames@emolecules.com>
parent: Claudio Freire <klaussfreire@gmail.com>
0 siblings, 1 reply; 47+ messages in thread
From: Craig James @ 2012-10-08 22:25 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Evgeny Shishkin <itparanoia@gmail.com>; pgsql-performance
On Mon, Oct 8, 2012 at 3:09 PM, Claudio Freire <klaussfreire@gmail.com>wrote:
> On Mon, Oct 8, 2012 at 7:06 PM, Craig James <cjames@emolecules.com> wrote:
> >> Sequential Input on the new one is 279MB/s, on the old 400MB/s.
> >>
> >
> > But why? What have I overlooked?
>
> Do you have readahead properly set up on the new one?
>
# blockdev --getra /dev/sdb1
256
Same on both servers.
Thanks,
Craig
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:28 Imre Samu <pella.samu@gmail.com>
parent: Craig James <cjames@emolecules.com>
5 siblings, 0 replies; 47+ messages in thread
From: Imre Samu @ 2012-10-08 22:28 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
>old: 2 x 4-core Intel Xeon E5620
>new: 4 x 4-core Intel Xeon E5606
http://ark.intel.com/compare/47925,52583
old: Xeon E5620 : 4 cores ; 8 Threads ; *Clock Speed : 2.40 GHz
; Max Turbo Frequency: 2.66 GHz*
new: Xeon E5606 : 4 cores ; 4 Threads ; Clock Speed : 2.13 GHz ;
Max Turbo Frequency: -
the older processor maybe faster ;
Imre
2012/10/8 Craig James <cjames@emolecules.com>
> This is driving me crazy. A new server, virtually identical to an old
> one, has 50% of the performance with pgbench. I've checked everything I
> can think of.
>
> The setups (call the servers "old" and "new"):
>
> old: 2 x 4-core Intel Xeon E5620
> new: 4 x 4-core Intel Xeon E5606
>
> both:
>
> memory: 12 GB DDR EC
> Disks: 12x500GB disks (Western Digital 7200RPM SATA)
> 2 disks, RAID1: OS (ext4) and postgres xlog (ext2)
> 8 disks, RAID10: $PGDATA
>
> 3WARE 9650SE-12ML with battery-backed cache. The admin tool (tw_cli)
> indicates that the battery is charged and the cache is working on both
> units.
>
> Linux: 2.6.32-41-server #94-Ubuntu SMP (new server's disk was
> actually cloned from old server).
>
> Postgres: 8.4.4 (yes, I should update. But both are identical.)
>
> The postgres.conf files are identical; diffs from the original are:
>
> max_connections = 500
> shared_buffers = 1000MB
> work_mem = 128MB
> synchronous_commit = off
> full_page_writes = off
> wal_buffers = 256kB
> checkpoint_segments = 30
> effective_cache_size = 4GB
> track_activities = on
> track_counts = on
> track_functions = none
> autovacuum = on
> autovacuum_naptime = 5min
> escape_string_warning = off
>
> Note that the old server is in production and was serving a light load
> while this test was running, so in theory it should be slower, not faster,
> than the new server.
>
> pgbench: Old server
>
> pgbench -i -s 100 -U test
> pgbench -U test -c ... -t ...
>
> -c -t TPS
> 5 20000 3777
> 10 10000 2622
> 20 5000 3759
> 30 3333 5712
> 40 2500 5953
> 50 2000 6141
>
> New server
> -c -t TPS
> 5 20000 2733
> 10 10000 2783
> 20 5000 3241
> 30 3333 2987
> 40 2500 2739
> 50 2000 2119
>
> As you can see, the new server is dramatically slower than the old one.
>
> I tested both the RAID10 data disk and the RAID1 xlog disk with bonnie++.
> The xlog disks were almost identical in performance. The RAID10 pg-data
> disks looked like this:
>
> Old server:
> Version 1.96 ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> xenon 24064M 687 99 203098 26 81904 16 3889 96 403747 31
> 737.6 31
> Latency 20512us 469ms 394ms 21402us 396ms
> 112ms
> Version 1.96 ------Sequential Create------ --------Random
> Create--------
> xenon -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 15953 27 +++++ +++ +++++ +++ +++++ +++ +++++ +++
> +++++ +++
> Latency 43291us 857us 519us 1588us 37us
> 178us
>
> 1.96,1.96,xenon,1,1349726125,24064M,,687,99,203098,26,81904,16,3889,96,403747,31,737.6,31,16,,,,,15953,27,+++++,+++,+++++,++\
>
> +,+++++,+++,+++++,+++,+++++,+++,20512us,469ms,394ms,21402us,396ms,112ms,43291us,857us,519us,1588us,37us,178us
>
>
> New server:
> Version 1.96 ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> zinc 24064M 862 99 212143 54 96008 14 4921 99 279239 17
> 752.0 23
> Latency 15613us 598ms 597ms 2764us 398ms
> 215ms
> Version 1.96 ------Sequential Create------ --------Random
> Create--------
> zinc -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 20380 26 +++++ +++ +++++ +++ +++++ +++ +++++ +++
> +++++ +++
> Latency 487us 627us 407us 972us 29us
> 262us
>
> 1.96,1.96,zinc,1,1349722017,24064M,,862,99,212143,54,96008,14,4921,99,279239,17,752.0,23,16,,,,,20380,26,+++++,+++,+++++,+++\
>
> ,+++++,+++,+++++,+++,+++++,+++,15613us,598ms,597ms,2764us,398ms,215ms,487us,627us,407us,972us,29us,262us
>
> I don't know enough about bonnie++ to know if these differences are
> interesting.
>
> One dramatic difference I noted via vmstat. On the old server, the I/O
> load during the bonnie++ run was steady, like this:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 0 2 71800 2117612 17940 9375660 0 0 82948 81944 1992 1341 1 3
> 86 10
> 0 2 71800 2113328 17948 9383896 0 0 76288 75806 1751 1167 0 2
> 86 11
> 0 1 71800 2111004 17948 9386540 92 0 93324 94232 2230 1510 0 4
> 86 10
> 0 1 71800 2106796 17948 9387436 114 0 67698 67588 1572 1088 0 2
> 87 11
> 0 1 71800 2106724 17956 9387968 50 0 81970 85710 1918 1287 0 3
> 86 10
> 1 1 71800 2103304 17956 9390700 0 0 92096 92160 1970 1194 0 4
> 86 10
> 0 2 71800 2103196 17976 9389204 0 0 70722 69680 1655 1116 1 3
> 86 10
> 1 1 71800 2099064 17980 9390824 0 0 57346 57348 1357 949 0 2
> 87 11
> 0 1 71800 2095596 17980 9392720 0 0 57344 57348 1379 987 0 2
> 86 12
>
> But the new server varied wildly during bonnie++:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 0 1 0 4518352 12004 7167000 0 0 118894 120838 2613 1539 0
> 2 93 5
> 0 1 0 4517252 12004 7167824 0 0 52116 53248 1179 793 0
> 1 94 5
> 0 1 0 4515864 12004 7169088 0 0 46764 49152 1104 733 0
> 1 91 7
> 0 1 0 4515180 12012 7169764 0 0 32924 30724 750 542 0
> 1 93 6
> 0 1 0 4514328 12016 7170780 0 0 42188 45056 1019 664 0
> 1 90 9
> 0 1 0 4513072 12016 7171856 0 0 67528 65540 1487 993 0
> 1 96 4
> 0 1 0 4510852 12016 7173160 0 0 56876 57344 1358 942 0
> 1 94 5
> 0 1 0 4500280 12044 7179924 0 0 91564 94220 2505 2504 1
> 2 91 6
> 0 1 0 4495564 12052 7183492 0 0 102660 104452 2289 1473 0
> 2 92 6
> 0 1 0 4492092 12052 7187720 0 0 98498 96274 2140 1385 0
> 2 93 5
> 0 1 0 4488608 12060 7190772 0 0 97628 100358 2176 1398 0
> 1 94 4
> 1 0 0 4485880 12052 7192600 0 0 112406 114686 2461 1509 0
> 3 90 7
> 1 0 0 4483424 12052 7195612 0 0 64678 65536 1449 948 0
> 1 91 8
> 0 1 0 4480252 12052 7199404 0 0 99608 100356 2217 1452 0
> 1 96 3
>
> Any ideas where to look next would be greatly appreciated.
>
> Craig
>
>
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:29 Craig James <cjames@emolecules.com>
parent: Craig James <cjames@emolecules.com>
5 siblings, 1 reply; 47+ messages in thread
From: Craig James @ 2012-10-08 22:29 UTC (permalink / raw)
To: pgsql-performance
One mistake in my descriptions...
On Mon, Oct 8, 2012 at 2:45 PM, Craig James <cjames@emolecules.com> wrote:
> This is driving me crazy. A new server, virtually identical to an old
> one, has 50% of the performance with pgbench. I've checked everything I
> can think of.
>
> The setups (call the servers "old" and "new"):
>
> old: 2 x 4-core Intel Xeon E5620
> new: 4 x 4-core Intel Xeon E5606
>
Actually it's not 16 cores. It's 8 cores, hyperthreaded. Hyperthreading
is disabled on the old system.
Is that enough to make this radical difference? (The server is at a
co-location site, so I have to go down there to boot into the BIOS and
disable hyperthreading.)
Craig
>
> both:
>
> memory: 12 GB DDR EC
> Disks: 12x500GB disks (Western Digital 7200RPM SATA)
> 2 disks, RAID1: OS (ext4) and postgres xlog (ext2)
> 8 disks, RAID10: $PGDATA
>
> 3WARE 9650SE-12ML with battery-backed cache. The admin tool (tw_cli)
> indicates that the battery is charged and the cache is working on both
> units.
>
> Linux: 2.6.32-41-server #94-Ubuntu SMP (new server's disk was
> actually cloned from old server).
>
> Postgres: 8.4.4 (yes, I should update. But both are identical.)
>
> The postgres.conf files are identical; diffs from the original are:
>
> max_connections = 500
> shared_buffers = 1000MB
> work_mem = 128MB
> synchronous_commit = off
> full_page_writes = off
> wal_buffers = 256kB
> checkpoint_segments = 30
> effective_cache_size = 4GB
> track_activities = on
> track_counts = on
> track_functions = none
> autovacuum = on
> autovacuum_naptime = 5min
> escape_string_warning = off
>
> Note that the old server is in production and was serving a light load
> while this test was running, so in theory it should be slower, not faster,
> than the new server.
>
> pgbench: Old server
>
> pgbench -i -s 100 -U test
> pgbench -U test -c ... -t ...
>
> -c -t TPS
> 5 20000 3777
> 10 10000 2622
> 20 5000 3759
> 30 3333 5712
> 40 2500 5953
> 50 2000 6141
>
> New server
> -c -t TPS
> 5 20000 2733
> 10 10000 2783
> 20 5000 3241
> 30 3333 2987
> 40 2500 2739
> 50 2000 2119
>
> As you can see, the new server is dramatically slower than the old one.
>
> I tested both the RAID10 data disk and the RAID1 xlog disk with bonnie++.
> The xlog disks were almost identical in performance. The RAID10 pg-data
> disks looked like this:
>
> Old server:
> Version 1.96 ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> xenon 24064M 687 99 203098 26 81904 16 3889 96 403747 31
> 737.6 31
> Latency 20512us 469ms 394ms 21402us 396ms
> 112ms
> Version 1.96 ------Sequential Create------ --------Random
> Create--------
> xenon -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 15953 27 +++++ +++ +++++ +++ +++++ +++ +++++ +++
> +++++ +++
> Latency 43291us 857us 519us 1588us 37us
> 178us
>
> 1.96,1.96,xenon,1,1349726125,24064M,,687,99,203098,26,81904,16,3889,96,403747,31,737.6,31,16,,,,,15953,27,+++++,+++,+++++,++\
>
> +,+++++,+++,+++++,+++,+++++,+++,20512us,469ms,394ms,21402us,396ms,112ms,43291us,857us,519us,1588us,37us,178us
>
>
> New server:
> Version 1.96 ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> zinc 24064M 862 99 212143 54 96008 14 4921 99 279239 17
> 752.0 23
> Latency 15613us 598ms 597ms 2764us 398ms
> 215ms
> Version 1.96 ------Sequential Create------ --------Random
> Create--------
> zinc -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 20380 26 +++++ +++ +++++ +++ +++++ +++ +++++ +++
> +++++ +++
> Latency 487us 627us 407us 972us 29us
> 262us
>
> 1.96,1.96,zinc,1,1349722017,24064M,,862,99,212143,54,96008,14,4921,99,279239,17,752.0,23,16,,,,,20380,26,+++++,+++,+++++,+++\
>
> ,+++++,+++,+++++,+++,+++++,+++,15613us,598ms,597ms,2764us,398ms,215ms,487us,627us,407us,972us,29us,262us
>
> I don't know enough about bonnie++ to know if these differences are
> interesting.
>
> One dramatic difference I noted via vmstat. On the old server, the I/O
> load during the bonnie++ run was steady, like this:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 0 2 71800 2117612 17940 9375660 0 0 82948 81944 1992 1341 1 3
> 86 10
> 0 2 71800 2113328 17948 9383896 0 0 76288 75806 1751 1167 0 2
> 86 11
> 0 1 71800 2111004 17948 9386540 92 0 93324 94232 2230 1510 0 4
> 86 10
> 0 1 71800 2106796 17948 9387436 114 0 67698 67588 1572 1088 0 2
> 87 11
> 0 1 71800 2106724 17956 9387968 50 0 81970 85710 1918 1287 0 3
> 86 10
> 1 1 71800 2103304 17956 9390700 0 0 92096 92160 1970 1194 0 4
> 86 10
> 0 2 71800 2103196 17976 9389204 0 0 70722 69680 1655 1116 1 3
> 86 10
> 1 1 71800 2099064 17980 9390824 0 0 57346 57348 1357 949 0 2
> 87 11
> 0 1 71800 2095596 17980 9392720 0 0 57344 57348 1379 987 0 2
> 86 12
>
> But the new server varied wildly during bonnie++:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 0 1 0 4518352 12004 7167000 0 0 118894 120838 2613 1539 0
> 2 93 5
> 0 1 0 4517252 12004 7167824 0 0 52116 53248 1179 793 0
> 1 94 5
> 0 1 0 4515864 12004 7169088 0 0 46764 49152 1104 733 0
> 1 91 7
> 0 1 0 4515180 12012 7169764 0 0 32924 30724 750 542 0
> 1 93 6
> 0 1 0 4514328 12016 7170780 0 0 42188 45056 1019 664 0
> 1 90 9
> 0 1 0 4513072 12016 7171856 0 0 67528 65540 1487 993 0
> 1 96 4
> 0 1 0 4510852 12016 7173160 0 0 56876 57344 1358 942 0
> 1 94 5
> 0 1 0 4500280 12044 7179924 0 0 91564 94220 2505 2504 1
> 2 91 6
> 0 1 0 4495564 12052 7183492 0 0 102660 104452 2289 1473 0
> 2 92 6
> 0 1 0 4492092 12052 7187720 0 0 98498 96274 2140 1385 0
> 2 93 5
> 0 1 0 4488608 12060 7190772 0 0 97628 100358 2176 1398 0
> 1 94 4
> 1 0 0 4485880 12052 7192600 0 0 112406 114686 2461 1509 0
> 3 90 7
> 1 0 0 4483424 12052 7195612 0 0 64678 65536 1449 948 0
> 1 91 8
> 0 1 0 4480252 12052 7199404 0 0 99608 100356 2217 1452 0
> 1 96 3
>
> Any ideas where to look next would be greatly appreciated.
>
> Craig
>
>
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:33 Evgeny Shishkin <itparanoia@gmail.com>
parent: Craig James <cjames@emolecules.com>
5 siblings, 2 replies; 47+ messages in thread
From: Evgeny Shishkin @ 2012-10-08 22:33 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On Oct 9, 2012, at 1:45 AM, Craig James <cjames@emolecules.com> wrote:
> This is driving me crazy. A new server, virtually identical to an old one, has 50% of the performance with pgbench. I've checked everything I can think of.
>
> The setups (call the servers "old" and "new"):
>
> old: 2 x 4-core Intel Xeon E5620
> new: 4 x 4-core Intel Xeon E5606
>
> both:
>
> memory: 12 GB DDR EC
> Disks: 12x500GB disks (Western Digital 7200RPM SATA)
> 2 disks, RAID1: OS (ext4) and postgres xlog (ext2)
> 8 disks, RAID10: $PGDATA
>
> 3WARE 9650SE-12ML with battery-backed cache. The admin tool (tw_cli)
> indicates that the battery is charged and the cache is working on both units.
>
> Linux: 2.6.32-41-server #94-Ubuntu SMP (new server's disk was
> actually cloned from old server).
>
> Postgres: 8.4.4 (yes, I should update. But both are identical.)
>
> The postgres.conf files are identical; diffs from the original are:
>
> max_connections = 500
> shared_buffers = 1000MB
> work_mem = 128MB
> synchronous_commit = off
> full_page_writes = off
> wal_buffers = 256kB
> checkpoint_segments = 30
> effective_cache_size = 4GB
> track_activities = on
> track_counts = on
> track_functions = none
> autovacuum = on
> autovacuum_naptime = 5min
> escape_string_warning = off
>
> Note that the old server is in production and was serving a light load while this test was running, so in theory it should be slower, not faster, than the new server.
>
> pgbench: Old server
>
> pgbench -i -s 100 -U test
> pgbench -U test -c ... -t ...
>
> -c -t TPS
> 5 20000 3777
> 10 10000 2622
> 20 5000 3759
> 30 3333 5712
> 40 2500 5953
> 50 2000 6141
>
> New server
> -c -t TPS
> 5 20000 2733
> 10 10000 2783
> 20 5000 3241
> 30 3333 2987
> 40 2500 2739
> 50 2000 2119
On new server postgresql do not scale at all. Looks like contention.
>
> As you can see, the new server is dramatically slower than the old one.
>
> I tested both the RAID10 data disk and the RAID1 xlog disk with bonnie++. The xlog disks were almost identical in performance. The RAID10 pg-data disks looked like this:
>
> Old server:
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> xenon 24064M 687 99 203098 26 81904 16 3889 96 403747 31 737.6 31
> Latency 20512us 469ms 394ms 21402us 396ms 112ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> xenon -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 15953 27 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
> Latency 43291us 857us 519us 1588us 37us 178us
> 1.96,1.96,xenon,1,1349726125,24064M,,687,99,203098,26,81904,16,3889,96,403747,31,737.6,31,16,,,,,15953,27,+++++,+++,+++++,++\
> +,+++++,+++,+++++,+++,+++++,+++,20512us,469ms,394ms,21402us,396ms,112ms,43291us,857us,519us,1588us,37us,178us
>
>
> New server:
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> zinc 24064M 862 99 212143 54 96008 14 4921 99 279239 17 752.0 23
> Latency 15613us 598ms 597ms 2764us 398ms 215ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> zinc -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 20380 26 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
> Latency 487us 627us 407us 972us 29us 262us
> 1.96,1.96,zinc,1,1349722017,24064M,,862,99,212143,54,96008,14,4921,99,279239,17,752.0,23,16,,,,,20380,26,+++++,+++,+++++,+++\
> ,+++++,+++,+++++,+++,+++++,+++,15613us,598ms,597ms,2764us,398ms,215ms,487us,627us,407us,972us,29us,262us
>
> I don't know enough about bonnie++ to know if these differences are interesting.
>
> One dramatic difference I noted via vmstat. On the old server, the I/O load during the bonnie++ run was steady, like this:
>
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 2 71800 2117612 17940 9375660 0 0 82948 81944 1992 1341 1 3 86 10
> 0 2 71800 2113328 17948 9383896 0 0 76288 75806 1751 1167 0 2 86 11
> 0 1 71800 2111004 17948 9386540 92 0 93324 94232 2230 1510 0 4 86 10
> 0 1 71800 2106796 17948 9387436 114 0 67698 67588 1572 1088 0 2 87 11
> 0 1 71800 2106724 17956 9387968 50 0 81970 85710 1918 1287 0 3 86 10
> 1 1 71800 2103304 17956 9390700 0 0 92096 92160 1970 1194 0 4 86 10
> 0 2 71800 2103196 17976 9389204 0 0 70722 69680 1655 1116 1 3 86 10
> 1 1 71800 2099064 17980 9390824 0 0 57346 57348 1357 949 0 2 87 11
> 0 1 71800 2095596 17980 9392720 0 0 57344 57348 1379 987 0 2 86 12
>
> But the new server varied wildly during bonnie++:
>
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 1 0 4518352 12004 7167000 0 0 118894 120838 2613 1539 0 2 93 5
> 0 1 0 4517252 12004 7167824 0 0 52116 53248 1179 793 0 1 94 5
> 0 1 0 4515864 12004 7169088 0 0 46764 49152 1104 733 0 1 91 7
> 0 1 0 4515180 12012 7169764 0 0 32924 30724 750 542 0 1 93 6
> 0 1 0 4514328 12016 7170780 0 0 42188 45056 1019 664 0 1 90 9
> 0 1 0 4513072 12016 7171856 0 0 67528 65540 1487 993 0 1 96 4
> 0 1 0 4510852 12016 7173160 0 0 56876 57344 1358 942 0 1 94 5
> 0 1 0 4500280 12044 7179924 0 0 91564 94220 2505 2504 1 2 91 6
> 0 1 0 4495564 12052 7183492 0 0 102660 104452 2289 1473 0 2 92 6
> 0 1 0 4492092 12052 7187720 0 0 98498 96274 2140 1385 0 2 93 5
> 0 1 0 4488608 12060 7190772 0 0 97628 100358 2176 1398 0 1 94 4
> 1 0 0 4485880 12052 7192600 0 0 112406 114686 2461 1509 0 3 90 7
> 1 0 0 4483424 12052 7195612 0 0 64678 65536 1449 948 0 1 91 8
> 0 1 0 4480252 12052 7199404 0 0 99608 100356 2217 1452 0 1 96 3
>
Also note the difference in free/cache distribution. Unless you took these numbers in completely different stages of bonnie++.
> Any ideas where to look next would be greatly appreciated.
>
> Craig
>
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:42 Craig James <cjames@emolecules.com>
parent: Evgeny Shishkin <itparanoia@gmail.com>
1 sibling, 0 replies; 47+ messages in thread
From: Craig James @ 2012-10-08 22:42 UTC (permalink / raw)
To: Evgeny Shishkin <itparanoia@gmail.com>; +Cc: pgsql-performance
On Mon, Oct 8, 2012 at 3:33 PM, Evgeny Shishkin <itparanoia@gmail.com>wrote:
>
> On Oct 9, 2012, at 1:45 AM, Craig James <cjames@emolecules.com> wrote:
>
> One dramatic difference I noted via vmstat. On the old server, the I/O
> load during the bonnie++ run was steady, like this:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 0 2 71800 2117612 17940 9375660 0 0 82948 81944 1992 1341 1 3
> 86 10
> 0 2 71800 2113328 17948 9383896 0 0 76288 75806 1751 1167 0 2
> 86 11
> 0 1 71800 2111004 17948 9386540 92 0 93324 94232 2230 1510 0 4
> 86 10
> 0 1 71800 2106796 17948 9387436 114 0 67698 67588 1572 1088 0 2
> 87 11
> 0 1 71800 2106724 17956 9387968 50 0 81970 85710 1918 1287 0 3
> 86 10
> 1 1 71800 2103304 17956 9390700 0 0 92096 92160 1970 1194 0 4
> 86 10
> 0 2 71800 2103196 17976 9389204 0 0 70722 69680 1655 1116 1 3
> 86 10
> 1 1 71800 2099064 17980 9390824 0 0 57346 57348 1357 949 0 2
> 87 11
> 0 1 71800 2095596 17980 9392720 0 0 57344 57348 1379 987 0 2
> 86 12
>
> But the new server varied wildly during bonnie++:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 0 1 0 4518352 12004 7167000 0 0 118894 120838 2613 1539 0
> 2 93 5
> 0 1 0 4517252 12004 7167824 0 0 52116 53248 1179 793 0
> 1 94 5
> 0 1 0 4515864 12004 7169088 0 0 46764 49152 1104 733 0
> 1 91 7
> 0 1 0 4515180 12012 7169764 0 0 32924 30724 750 542 0
> 1 93 6
> 0 1 0 4514328 12016 7170780 0 0 42188 45056 1019 664 0
> 1 90 9
> 0 1 0 4513072 12016 7171856 0 0 67528 65540 1487 993 0
> 1 96 4
> 0 1 0 4510852 12016 7173160 0 0 56876 57344 1358 942 0
> 1 94 5
> 0 1 0 4500280 12044 7179924 0 0 91564 94220 2505 2504 1
> 2 91 6
> 0 1 0 4495564 12052 7183492 0 0 102660 104452 2289 1473 0
> 2 92 6
> 0 1 0 4492092 12052 7187720 0 0 98498 96274 2140 1385 0
> 2 93 5
> 0 1 0 4488608 12060 7190772 0 0 97628 100358 2176 1398 0
> 1 94 4
> 1 0 0 4485880 12052 7192600 0 0 112406 114686 2461 1509 0
> 3 90 7
> 1 0 0 4483424 12052 7195612 0 0 64678 65536 1449 948 0
> 1 91 8
> 0 1 0 4480252 12052 7199404 0 0 99608 100356 2217 1452 0
> 1 96 3
>
>
> Also note the difference in free/cache distribution. Unless you took these
> numbers in completely different stages of bonnie++.
>
>
The old server is in production and is running Apache/Postgres requests.
Craig
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:44 Claudio Freire <klaussfreire@gmail.com>
parent: Craig James <cjames@emolecules.com>
0 siblings, 2 replies; 47+ messages in thread
From: Claudio Freire @ 2012-10-08 22:44 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: Evgeny Shishkin <itparanoia@gmail.com>; pgsql-performance
On Mon, Oct 8, 2012 at 7:25 PM, Craig James <cjames@emolecules.com> wrote:
>> > But why? What have I overlooked?
>>
>> Do you have readahead properly set up on the new one?
>
>
> # blockdev --getra /dev/sdb1
> 256
It's probably this. 256 is way too low to saturate your I/O system.
Pump it up. I've found 8192 works nice for a system I have, 32000 I
guess could work too.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:46 Evgeny Shishkin <itparanoia@gmail.com>
parent: Claudio Freire <klaussfreire@gmail.com>
1 sibling, 0 replies; 47+ messages in thread
From: Evgeny Shishkin @ 2012-10-08 22:46 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On Oct 9, 2012, at 2:44 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
> On Mon, Oct 8, 2012 at 7:25 PM, Craig James <cjames@emolecules.com> wrote:
>>>> But why? What have I overlooked?
>>>
>>> Do you have readahead properly set up on the new one?
>>
>>
>> # blockdev --getra /dev/sdb1
>> 256
>
>
> It's probably this. 256 is way too low to saturate your I/O system.
> Pump it up. I've found 8192 works nice for a system I have, 32000 I
> guess could work too.
This, i also suggest to rebenchmark with increased wal_buffers. May be that downscale comes from wal mutex contention.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:48 Craig James <cjames@emolecules.com>
parent: Claudio Freire <klaussfreire@gmail.com>
1 sibling, 2 replies; 47+ messages in thread
From: Craig James @ 2012-10-08 22:48 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Evgeny Shishkin <itparanoia@gmail.com>; pgsql-performance
On Mon, Oct 8, 2012 at 3:44 PM, Claudio Freire <klaussfreire@gmail.com>wrote:
> On Mon, Oct 8, 2012 at 7:25 PM, Craig James <cjames@emolecules.com> wrote:
> >> > But why? What have I overlooked?
> >>
> >> Do you have readahead properly set up on the new one?
> >
> >
> > # blockdev --getra /dev/sdb1
> > 256
>
>
> It's probably this. 256 is way too low to saturate your I/O system.
> Pump it up. I've found 8192 works nice for a system I have, 32000 I
> guess could work too.
>
But again ... the two systems are identical. This can't explain it.
Thanks,
Craig
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 22:50 Claudio Freire <klaussfreire@gmail.com>
parent: Craig James <cjames@emolecules.com>
1 sibling, 1 reply; 47+ messages in thread
From: Claudio Freire @ 2012-10-08 22:50 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: Evgeny Shishkin <itparanoia@gmail.com>; pgsql-performance
On Mon, Oct 8, 2012 at 7:48 PM, Craig James <cjames@emolecules.com> wrote:
>> > # blockdev --getra /dev/sdb1
>> > 256
>>
>>
>> It's probably this. 256 is way too low to saturate your I/O system.
>> Pump it up. I've found 8192 works nice for a system I have, 32000 I
>> guess could work too.
>
>
> But again ... the two systems are identical. This can't explain it.
Is the read-ahead the same in both systems?
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 23:03 Craig James <cjames@emolecules.com>
parent: Claudio Freire <klaussfreire@gmail.com>
0 siblings, 2 replies; 47+ messages in thread
From: Craig James @ 2012-10-08 23:03 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Evgeny Shishkin <itparanoia@gmail.com>; pgsql-performance
On Mon, Oct 8, 2012 at 3:50 PM, Claudio Freire <klaussfreire@gmail.com>wrote:
> On Mon, Oct 8, 2012 at 7:48 PM, Craig James <cjames@emolecules.com> wrote:
> >> > # blockdev --getra /dev/sdb1
> >> > 256
> >>
> >>
> >> It's probably this. 256 is way too low to saturate your I/O system.
> >> Pump it up. I've found 8192 works nice for a system I have, 32000 I
> >> guess could work too.
> >
> >
> > But again ... the two systems are identical. This can't explain it.
>
> Is the read-ahead the same in both systems?
>
Yes, as I said in the original reply (it got cut off from your reply):
"Same on both servers."
Craig
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 23:10 Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
parent: Craig James <cjames@emolecules.com>
1 sibling, 0 replies; 47+ messages in thread
From: Mark Kirkwood @ 2012-10-08 23:10 UTC (permalink / raw)
To: pgsql-performance
On 09/10/12 11:48, Craig James wrote:
> On Mon, Oct 8, 2012 at 3:44 PM, Claudio Freire <klaussfreire@gmail.com>wrote:
>
>> On Mon, Oct 8, 2012 at 7:25 PM, Craig James <cjames@emolecules.com> wrote:
>>>>> But why? What have I overlooked?
>>>> Do you have readahead properly set up on the new one?
>>>
>>> # blockdev --getra /dev/sdb1
>>> 256
>>
>> It's probably this. 256 is way too low to saturate your I/O system.
>> Pump it up. I've found 8192 works nice for a system I have, 32000 I
>> guess could work too.
>>
> But again ... the two systems are identical. This can't explain it.
>
Maybe check all sysctl's are the same - in particular:
vm.zone_reclaim_mode
has a tendency to set itself to 1 on newer hardware, which will reduce
performance of database style workloads.
Cheers
Mark
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 23:12 Claudio Freire <klaussfreire@gmail.com>
parent: Craig James <cjames@emolecules.com>
1 sibling, 0 replies; 47+ messages in thread
From: Claudio Freire @ 2012-10-08 23:12 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: Evgeny Shishkin <itparanoia@gmail.com>; pgsql-performance
On Mon, Oct 8, 2012 at 8:03 PM, Craig James <cjames@emolecules.com> wrote:
>> > But again ... the two systems are identical. This can't explain it.
>>
>> Is the read-ahead the same in both systems?
>
>
> Yes, as I said in the original reply (it got cut off from your reply): "Same
> on both servers."
Oh, yes. Google collapsed it. Wierd.
Anyway, sequential I/O isn't the same in both servers, and usually you
don't get full sequential performance unless you bump up the
read-ahead. I'm still betting on that for the difference in sequential
performance.
As for pgbench, I'm not sure, but I think pgbench doesn't really
stress sequential performance. You seem to be getting bad queueing
performance. Did you check NCQ status on the RAID controller? Is it on
on both servers?
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 23:16 Tomas Vondra <tv@fuzzy.cz>
parent: Craig James <cjames@emolecules.com>
1 sibling, 0 replies; 47+ messages in thread
From: Tomas Vondra @ 2012-10-08 23:16 UTC (permalink / raw)
To: pgsql-performance
On 9.10.2012 01:03, Craig James wrote:
>
>
> On Mon, Oct 8, 2012 at 3:50 PM, Claudio Freire <klaussfreire@gmail.com
> <mailto:klaussfreire@gmail.com>> wrote:
>
> On Mon, Oct 8, 2012 at 7:48 PM, Craig James <cjames@emolecules.com
> <mailto:cjames@emolecules.com>> wrote:
> >> > # blockdev --getra /dev/sdb1
> >> > 256
> >>
> >>
> >> It's probably this. 256 is way too low to saturate your I/O system.
> >> Pump it up. I've found 8192 works nice for a system I have, 32000 I
> >> guess could work too.
> >
> >
> > But again ... the two systems are identical. This can't explain it.
>
> Is the read-ahead the same in both systems?
>
>
> Yes, as I said in the original reply (it got cut off from your reply):
> "Same on both servers."
And what about read-ahead settings on the controller? 3WARE used to have
a read-ahead settings on their own (usually there are three options -
read-ahead, no read-ahead and adaptive). Is this set to the same value
on both machines?
Tomas
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 23:24 Tomas Vondra <tv@fuzzy.cz>
parent: Evgeny Shishkin <itparanoia@gmail.com>
1 sibling, 1 reply; 47+ messages in thread
From: Tomas Vondra @ 2012-10-08 23:24 UTC (permalink / raw)
To: pgsql-performance
On 9.10.2012 00:33, Evgeny Shishkin wrote:
>>
>> pgbench: Old server
>>
>> pgbench -i -s 100 -U test
>> pgbench -U test -c ... -t ...
>>
>> -c -t TPS
>> 5 20000 3777
>> 10 10000 2622
>> 20 5000 3759
>> 30 3333 5712
>> 40 2500 5953
>> 50 2000 6141
>>
>> New server
>> -c -t TPS
>> 5 20000 2733
>> 10 10000 2783
>> 20 5000 3241
>> 30 3333 2987
>> 40 2500 2739
>> 50 2000 2119
>
> On new server postgresql do not scale at all. Looks like contention.
Why? The evidence we've seen so far IMHO suggests a poorly performing
I/O subsystem. Post a few lines of "vmstat 1" / "iostat -x -k 1"
collected when the pgbench is running, that might tell us more.
Try a few very basic I/O tests that are easy to understand rather than
running bonnie++ which is quite complex. For example try this:
time sh -c "dd if=/dev/zero of=myfile.tmp bs=8192 count=4194304 && sync"
dd if=myfile.tmp of=/dev/null bs=8192
The former measures sequential write speed, the latter measures
sequential read speed in a very primitive way. Watch vmstat/iostat and
don't bother running pgbench until you get a reasonable performance on
both systems.
Tomas
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 23:30 Evgeny Shishkin <itparanoia@gmail.com>
parent: Tomas Vondra <tv@fuzzy.cz>
0 siblings, 0 replies; 47+ messages in thread
From: Evgeny Shishkin @ 2012-10-08 23:30 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; +Cc: pgsql-performance
On Oct 9, 2012, at 3:24 AM, Tomas Vondra <tv@fuzzy.cz> wrote:
> On 9.10.2012 00:33, Evgeny Shishkin wrote:
>>>
>>> pgbench: Old server
>>>
>>> pgbench -i -s 100 -U test
>>> pgbench -U test -c ... -t ...
>>>
>>> -c -t TPS
>>> 5 20000 3777
>>> 10 10000 2622
>>> 20 5000 3759
>>> 30 3333 5712
>>> 40 2500 5953
>>> 50 2000 6141
>>>
>>> New server
>>> -c -t TPS
>>> 5 20000 2733
>>> 10 10000 2783
>>> 20 5000 3241
>>> 30 3333 2987
>>> 40 2500 2739
>>> 50 2000 2119
>>
>> On new server postgresql do not scale at all. Looks like contention.
>
> Why? The evidence we've seen so far IMHO suggests a poorly performing
> I/O subsystem. Post a few lines of "vmstat 1" / "iostat -x -k 1"
> collected when the pgbench is running, that might tell us more.
>
Because 50 clients can push io even with small read ahead. And hear we see nice parabola. Just guessing anyway.
> Try a few very basic I/O tests that are easy to understand rather than
> running bonnie++ which is quite complex. For example try this:
>
> time sh -c "dd if=/dev/zero of=myfile.tmp bs=8192 count=4194304 && sync"
>
> dd if=myfile.tmp of=/dev/null bs=8192
>
> The former measures sequential write speed, the latter measures
> sequential read speed in a very primitive way. Watch vmstat/iostat and
> don't bother running pgbench until you get a reasonable performance on
> both systems.
>
>
> Tomas
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 23:40 Craig James <cjames@emolecules.com>
parent: Craig James <cjames@emolecules.com>
0 siblings, 5 replies; 47+ messages in thread
From: Craig James @ 2012-10-08 23:40 UTC (permalink / raw)
To: pgsql-performance
Nobody has commented on the hyperthreading question yet ... does it really
matter? The old (fast) server has hyperthreading disabled, and the new
(slower) server has hyperthreads enabled.
If hyperthreading is definitely NOT an issue, it will save me a trip to the
co-lo facility.
Thanks,
Craig
On Mon, Oct 8, 2012 at 3:29 PM, Craig James <cjames@emolecules.com> wrote:
> One mistake in my descriptions...
>
> On Mon, Oct 8, 2012 at 2:45 PM, Craig James <cjames@emolecules.com> wrote:
>
>> This is driving me crazy. A new server, virtually identical to an old
>> one, has 50% of the performance with pgbench. I've checked everything I
>> can think of.
>>
>> The setups (call the servers "old" and "new"):
>>
>> old: 2 x 4-core Intel Xeon E5620
>> new: 4 x 4-core Intel Xeon E5606
>>
>
> Actually it's not 16 cores. It's 8 cores, hyperthreaded. Hyperthreading
> is disabled on the old system.
>
> Is that enough to make this radical difference? (The server is at a
> co-location site, so I have to go down there to boot into the BIOS and
> disable hyperthreading.)
>
> Craig
>
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-08 23:52 Gavin Flower <GavinFlower@archidevsys.co.nz>
parent: Craig James <cjames@emolecules.com>
4 siblings, 0 replies; 47+ messages in thread
From: Gavin Flower @ 2012-10-08 23:52 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On 09/10/12 12:40, Craig James wrote:
> Nobody has commented on the hyperthreading question yet ... does it
> really matter? The old (fast) server has hyperthreading disabled, and
> the new (slower) server has hyperthreads enabled.
>
> If hyperthreading is definitely NOT an issue, it will save me a trip
> to the co-lo facility.
>
> Thanks,
> Craig
>
> On Mon, Oct 8, 2012 at 3:29 PM, Craig James <cjames@emolecules.com
> <mailto:cjames@emolecules.com>> wrote:
>
> One mistake in my descriptions...
>
> On Mon, Oct 8, 2012 at 2:45 PM, Craig James <cjames@emolecules.com
> <mailto:cjames@emolecules.com>> wrote:
>
> This is driving me crazy. A new server, virtually identical
> to an old one, has 50% of the performance with pgbench. I've
> checked everything I can think of.
>
> The setups (call the servers "old" and "new"):
>
> old: 2 x 4-core Intel Xeon E5620
> new: 4 x 4-core Intel Xeon E5606
>
>
> Actually it's not 16 cores. It's 8 cores, hyperthreaded.
> Hyperthreading is disabled on the old system.
>
> Is that enough to make this radical difference? (The server is at
> a co-location site, so I have to go down there to boot into the
> BIOS and disable hyperthreading.)
>
> Craig
>
>
My latest development box (Intel Latest Core i7 3770K Ivy Bridge Quad
Core with HT 3.4GHz) has hyperthreading - and it *_does_* make a
significant difference.
Cheers,
Gavin
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-09 00:00 Ants Aasma <ants@cybertec.at>
parent: Craig James <cjames@emolecules.com>
4 siblings, 0 replies; 47+ messages in thread
From: Ants Aasma @ 2012-10-09 00:00 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On Tue, Oct 9, 2012 at 2:40 AM, Craig James <cjames@emolecules.com> wrote:
> Nobody has commented on the hyperthreading question yet ... does it really
> matter? The old (fast) server has hyperthreading disabled, and the new
> (slower) server has hyperthreads enabled.
>
> If hyperthreading is definitely NOT an issue, it will save me a trip to the
> co-lo facility.
Hyperthreading will make lock contention issues worse by having more
threads fighting. Test the new box with postgres 9.2, if the newer
version exhibits much better scaling behavior it strongly suggest lock
contention rather than IO being the root cause.
Ants Aasma
--
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-09 11:20 Yeb Havinga <yebhavinga@gmail.com>
parent: Craig James <cjames@emolecules.com>
5 siblings, 0 replies; 47+ messages in thread
From: Yeb Havinga @ 2012-10-09 11:20 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On 2012-10-08 23:45, Craig James wrote:
> This is driving me crazy. A new server, virtually identical to an old
> one, has 50% of the performance with pgbench. I've checked everything
> I can think of.
>
> The setups (call the servers "old" and "new"):
>
> old: 2 x 4-core Intel Xeon E5620
> new: 4 x 4-core Intel Xeon E5606
How are the filesystems formatted and mounted (-o nobarrier?)
regards
Yeb
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-09 16:02 Shaun Thomas <sthomas@optionshouse.com>
parent: Craig James <cjames@emolecules.com>
4 siblings, 1 reply; 47+ messages in thread
From: Shaun Thomas @ 2012-10-09 16:02 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On 10/08/2012 06:40 PM, Craig James wrote:
> Nobody has commented on the hyperthreading question yet ... does it
> really matter? The old (fast) server has hyperthreading disabled, and
> the new (slower) server has hyperthreads enabled.
I doubt it's this. With the newer post-Nehalem processors,
hyperthreading is actually much better than it was before. But you also
have this:
CPU Speed L3 Cache DDR3 Speed
E5606 2.13Ghz 8MB 800Mhz
E5620 2.4Ghz 12MB 1066Mhz
Even with "equal" threads, the CPUs you have in the new server, as
opposed to the old, are much worse. The E5606 doesn't even have
hyper-threading, so it's not an issue here. In fact, if you enabled it
on the old server, it would likely get *much faster*.
We saw a 40% improvement by enabling hyper-threading. Sure, it's not
100%, but it's not negative or zero, either.
Basically we can see, at the very least, that your servers are not
"identical." Little things like this can make a massive difference. The
old server has a much better CPU. Even crippled without hyperthreading,
I could see it beating the new server.
One thing you might want to check in the BIOS of the new server, is to
make sure that power saving mode is disabled everywhere you can find it.
Some servers come with that set by default, and that puts the CPU to
sleep occasionally, and the spin-up necessary to re-engage it is
punishing and inconsistent. We saw 20-40% drops in pgbench pretty much
at random, when CPU power saving was enabled.
This doesn't cover why your IO subsystem is slower on the new system,
but I suspect it might have something to do with the memory speed. It
suggests a slower PCI bus, which could choke your RAID card.
--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com
______________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-09 16:14 David Thomas <david@digitaldogma.org>
parent: Craig James <cjames@emolecules.com>
4 siblings, 1 reply; 47+ messages in thread
From: David Thomas @ 2012-10-09 16:14 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On Mon, Oct 08, 2012 at 04:40:31PM -0700, Craig James wrote:
> Nobody has commented on the hyperthreading question yet ... does it
> really matter? The old (fast) server has hyperthreading disabled, and
> the new (slower) server has hyperthreads enabled.
> If hyperthreading is definitely NOT an issue, it will save me a trip to
> the co-lo facility.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-09 16:41 Craig James <cjames@emolecules.com>
parent: Shaun Thomas <sthomas@optionshouse.com>
0 siblings, 0 replies; 47+ messages in thread
From: Craig James @ 2012-10-09 16:41 UTC (permalink / raw)
To: pgsql-performance
On Tue, Oct 9, 2012 at 9:02 AM, Shaun Thomas <sthomas@optionshouse.com>wrote:
> On 10/08/2012 06:40 PM, Craig James wrote:
>
> Nobody has commented on the hyperthreading question yet ... does it
>> really matter? The old (fast) server has hyperthreading disabled, and
>> the new (slower) server has hyperthreads enabled.
>>
>
> I doubt it's this. With the newer post-Nehalem processors, hyperthreading
> is actually much better than it was before. But you also have this:
>
> CPU Speed L3 Cache DDR3 Speed
> E5606 2.13Ghz 8MB 800Mhz
> E5620 2.4Ghz 12MB 1066Mhz
>
> Even with "equal" threads, the CPUs you have in the new server, as
> opposed to the old, are much worse. The E5606 doesn't even have
> hyper-threading, so it's not an issue here. In fact, if you enabled it on
> the old server, it would likely get *much faster*.
>
Even more mysterious, because it turns out it's backwards. I
copy-and-pasted the CPU information wrong. I wrote:
> old: 2 x 4-core Intel Xeon E5620
> new: 4 x 4-core Intel Xeon E5606
The correct configuration is:
old: 2x4-core Intel Xeon E2606 2.133 GHz
new: 2x4-core Intex Xeon E5620 2.40 GHz
So that makes the poor performance of the new system even more mystifying.
I'm going down there right now to disable hyperthreading and see if that's
the answer. So far, that's the only concrete thing that I've been able to
discover that's different between the two systems.
>
> We saw a 40% improvement by enabling hyper-threading. Sure, it's not 100%,
> but it's not negative or zero, either.
>
> Basically we can see, at the very least, that your servers are not
> "identical." Little things like this can make a massive difference. The old
> server has a much better CPU. Even crippled without hyperthreading, I could
> see it beating the new server.
>
> One thing you might want to check in the BIOS of the new server, is to
> make sure that power saving mode is disabled everywhere you can find it.
> Some servers come with that set by default, and that puts the CPU to sleep
> occasionally, and the spin-up necessary to re-engage it is punishing and
> inconsistent. We saw 20-40% drops in pgbench pretty much at random, when
> CPU power saving was enabled.
>
Thanks, I'll double check that too. That's a good suspect.
>
> This doesn't cover why your IO subsystem is slower on the new system, but
> I suspect it might have something to do with the memory speed. It suggests
> a slower PCI bus, which could choke your RAID card.
>
The motherboards are supposed to be identical. But I'll double check that
too.
Craig
>
> --
> Shaun Thomas
> OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
> 312-444-8534
> sthomas@optionshouse.com
>
> ______________________________________________
>
> See http://www.peak6.com/email_disclaimer/ for terms and conditions
> related to this email
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-09 16:43 Craig James <cjames@emolecules.com>
parent: David Thomas <david@digitaldogma.org>
0 siblings, 0 replies; 47+ messages in thread
From: Craig James @ 2012-10-09 16:43 UTC (permalink / raw)
To: david@digitaldogma.org; +Cc: pgsql-performance
On Tue, Oct 9, 2012 at 9:14 AM, David Thomas <david@digitaldogma.org> wrote:
> On Mon, Oct 08, 2012 at 04:40:31PM -0700, Craig James wrote:
> > Nobody has commented on the hyperthreading question yet ... does it
> > really matter? The old (fast) server has hyperthreading disabled, and
> > the new (slower) server has hyperthreads enabled.
> > If hyperthreading is definitely NOT an issue, it will save me a trip
> to
> > the co-lo facility.
>
> From my reading it seems that hyperthreading hasn't been a major issue
> for quite sometime on modern kernels.
> http://archives.postgresql.org/pgsql-performance/2004-10/msg00052.php
>
> I doubt it would hurt much, but I wouldn't make a special trip to the
> co-lo to change it.
>
At this point I've discovered no other options, so down to the co-lo I go.
I'm also going to check power-save options and the RAID controller's
built-in configuration to see if I overlooked something there (readahead,
blocksize, whatever).
Craig
> --
> DavidT
>
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-11 14:14 Andrea Suisani <sickpig@opinioni.net>
parent: Craig James <cjames@emolecules.com>
4 siblings, 1 reply; 47+ messages in thread
From: Andrea Suisani @ 2012-10-11 14:14 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: pgsql-performance
On 10/09/2012 01:40 AM, Craig James wrote:
> Nobody has commented on the hyperthreading question yet ... does it really matter? The old (fast) server has hyperthreading disabled, and the new (slower) server has hyperthreads enabled.
>
> If hyperthreading is definitely NOT an issue, it will save me a trip to the co-lo facility.
sorry to come late to the party, but being in a similar condition
I've googled a bit and I've found a way to disable hyperthreading without
the need to reboot the system and entering the bios:
echo 0 >/sys/devices/system/node/node0/cpuX/online
where X belongs to 1..(#cores * 2) if hyperthreading is enabled
(cpu0 can't be switched off).
didn't try myself on live system, but I definitely will
as soon as I have a new machine to test.
Andrea
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-11 14:19 Claudio Freire <klaussfreire@gmail.com>
parent: Andrea Suisani <sickpig@opinioni.net>
0 siblings, 1 reply; 47+ messages in thread
From: Claudio Freire @ 2012-10-11 14:19 UTC (permalink / raw)
To: Andrea Suisani <sickpig@opinioni.net>; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On Thu, Oct 11, 2012 at 11:14 AM, Andrea Suisani <sickpig@opinioni.net> wrote:
> sorry to come late to the party, but being in a similar condition
> I've googled a bit and I've found a way to disable hyperthreading without
> the need to reboot the system and entering the bios:
>
> echo 0 >/sys/devices/system/node/node0/cpuX/online
>
> where X belongs to 1..(#cores * 2) if hyperthreading is enabled
> (cpu0 can't be switched off).
>
> didn't try myself on live system, but I definitely will
> as soon as I have a new machine to test.
Question is... will that remove the performance penalty of HyperThreading?
I don't think so, because a big one is the register file split (half
the hardware registers go to a CPU, half to the other). If that action
doesn't tell the CPU to "unsplit", some shared components may become
unbogged, like the decode stage probably, but I'm not sure it's the
same as disabling it from the BIOS.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-11 14:40 Andrea Suisani <sickpig@opinioni.net>
parent: Claudio Freire <klaussfreire@gmail.com>
0 siblings, 1 reply; 47+ messages in thread
From: Andrea Suisani @ 2012-10-11 14:40 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On 10/11/2012 04:19 PM, Claudio Freire wrote:
> On Thu, Oct 11, 2012 at 11:14 AM, Andrea Suisani <sickpig@opinioni.net> wrote:
>> sorry to come late to the party, but being in a similar condition
>> I've googled a bit and I've found a way to disable hyperthreading without
>> the need to reboot the system and entering the bios:
>>
>> echo 0 >/sys/devices/system/node/node0/cpuX/online
>>
>> where X belongs to 1..(#cores * 2) if hyperthreading is enabled
>> (cpu0 can't be switched off).
>>
>> didn't try myself on live system, but I definitely will
>> as soon as I have a new machine to test.
>
> Question is... will that remove the performance penalty of HyperThreading?
So I've added to my todo list to perform a test to verify this claim :)
> I don't think so, because a big one is the register file split (half
> the hardware registers go to a CPU, half to the other). If that action
> doesn't tell the CPU to "unsplit", some shared components may become
> unbogged, like the decode stage probably, but I'm not sure it's the
> same as disabling it from the BIOS.
Although I think that you're probably right to assume that disabling HT
through the syfs interface won't remove the performance penalty for real.
thanks
Andrea
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 08:27 Andrea Suisani <sickpig@opinioni.net>
parent: Andrea Suisani <sickpig@opinioni.net>
0 siblings, 2 replies; 47+ messages in thread
From: Andrea Suisani @ 2012-10-15 08:27 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On 10/11/2012 04:40 PM, Andrea Suisani wrote:
> On 10/11/2012 04:19 PM, Claudio Freire wrote:
>> On Thu, Oct 11, 2012 at 11:14 AM, Andrea Suisani <sickpig@opinioni.net> wrote:
>>> sorry to come late to the party, but being in a similar condition
>>> I've googled a bit and I've found a way to disable hyperthreading without
>>> the need to reboot the system and entering the bios:
>>>
>>> echo 0 >/sys/devices/system/node/node0/cpuX/online
>>>
>>> where X belongs to 1..(#cores * 2) if hyperthreading is enabled
>>> (cpu0 can't be switched off).
>>>
>>> didn't try myself on live system, but I definitely will
>>> as soon as I have a new machine to test.
>>
>> Question is... will that remove the performance penalty of HyperThreading?
>
> So I've added to my todo list to perform a test to verify this claim :)
done.
in a brief: the box is dell a PowerEdge r720 with 16GB of RAM,
the cpu is a Xeon 5620 with 6 core, the OS is installed on a raid
(sata disk 7.2k rpm) and the PGDATA is on separate RAID 1 array
(sas 15K rpm) and the controller is a PERC H710 (bbwc with a cache
of 512 MB).
Postgres ver 9.2.1 (sorry for not having benchmarked 9.1,
but this what we plan to deploy in production). Both the OS
(Ubuntu 12.04.1) and Postgres had been briefly tuned according
to the usal standards while trying to mimic Craig's configuration
(see specific settings at the bottom).
TPS including connection establishing, pgbench run in a single
thread mode, connection made through unix socket, OS cache dropped
and Postgres restarted for every run.
those are the results:
HT HT SYSFS DIS HT BIOS DISABLE
-c -t r1 r2 r3 r1 r2 r3 r1 r2 r3
5 20K 1641 1831 1496 2020 1974 2033 2005 1988 1967
10 10K 2161 2134 2136 2277 2252 2216 1854 1824 1810
20 5k 2550 2508 2558 2417 2388 2357 1924 1928 1954
30 3333 2216 2272 2250 2333 2493 2496 1993 2009 2008
40 2.5K 2179 2221 2250 2568 2535 2500 2025 2048 2018
50 2K 2217 2213 2213 2487 2449 2604 2112 2016 2023
Despite the fact the results don't match my expectation
(I suspect that there's something wrong with the PERC
because, having the controller cache enabled make no
difference in terms of TPS), it seems strange that disabling
HT from the bios will give lesser TPS that HT disable through
sysfs interface.
OS conf:
vm.swappiness=0
vm.overcommit_memory=2
vm.dirty_ratio=2
vm.dirty_background_ratio=1
kernel.shmmax=3454820352
kernel.shmall=2048341
/sbin/blockdev --setra 8192 /dev/sdb
$PGDATA is on ext4 (rw,noatime)
Linux cloud 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
sdb scheduler is [cfq]
DB conf:
max_connections = 100
shared_buffers = 3200MB
work_mem = 30MB
maintenance_work_mem = 800MB
synchronous_commit = off
full_page_writes = off
checkpoint_segments = 40
checkpoint_timeout = 5min
checkpoint_completion_target = 0.9
random_page_cost = 3.5
effective_cache_size = 10GB
log_autovacuum_min_duration = 0
autovacuum_naptime = 5min
Andrea
p.s. as last try in the process of increasing TPS
I've change the scheduler from cfq to deadline
and for -c 5 t 20K I've got r1=3007, r2=2930 and r3=2985.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 15:01 Craig James <cjames@emolecules.com>
parent: Andrea Suisani <sickpig@opinioni.net>
1 sibling, 3 replies; 47+ messages in thread
From: Craig James @ 2012-10-15 15:01 UTC (permalink / raw)
To: Andrea Suisani <sickpig@opinioni.net>; +Cc: Claudio Freire <klaussfreire@gmail.com>; pgsql-performance
On Mon, Oct 15, 2012 at 1:27 AM, Andrea Suisani <sickpig@opinioni.net> wrote:
> On 10/11/2012 04:40 PM, Andrea Suisani wrote:
>>
>> On 10/11/2012 04:19 PM, Claudio Freire wrote:
>>>
>>> On Thu, Oct 11, 2012 at 11:14 AM, Andrea Suisani <sickpig@opinioni.net>
>>> wrote:
>>>>
>>>> sorry to come late to the party, but being in a similar condition
>>>> I've googled a bit and I've found a way to disable hyperthreading
>>>> without
>>>> the need to reboot the system and entering the bios:
>>>>
>>>> echo 0 >/sys/devices/system/node/node0/cpuX/online
>>>>
>>>> where X belongs to 1..(#cores * 2) if hyperthreading is enabled
>>>> (cpu0 can't be switched off).
>>>>
>>>> didn't try myself on live system, but I definitely will
>>>> as soon as I have a new machine to test.
>>>
>>>
>>> Question is... will that remove the performance penalty of
>>> HyperThreading?
>>
>>
>> So I've added to my todo list to perform a test to verify this claim :)
>
>
> done.
>
> in a brief: the box is dell a PowerEdge r720 with 16GB of RAM,
> the cpu is a Xeon 5620 with 6 core, the OS is installed on a raid
> (sata disk 7.2k rpm) and the PGDATA is on separate RAID 1 array
> (sas 15K rpm) and the controller is a PERC H710 (bbwc with a cache
> of 512 MB).
>
> Postgres ver 9.2.1 (sorry for not having benchmarked 9.1,
> but this what we plan to deploy in production). Both the OS
> (Ubuntu 12.04.1) and Postgres had been briefly tuned according
> to the usal standards while trying to mimic Craig's configuration
> (see specific settings at the bottom).
>
> TPS including connection establishing, pgbench run in a single
> thread mode, connection made through unix socket, OS cache dropped
> and Postgres restarted for every run.
>
> those are the results:
>
> HT HT SYSFS DIS HT BIOS DISABLE
> -c -t r1 r2 r3 r1 r2 r3 r1 r2 r3
> 5 20K 1641 1831 1496 2020 1974 2033 2005 1988 1967
> 10 10K 2161 2134 2136 2277 2252 2216 1854 1824 1810
> 20 5k 2550 2508 2558 2417 2388 2357 1924 1928 1954
> 30 3333 2216 2272 2250 2333 2493 2496 1993 2009 2008
> 40 2.5K 2179 2221 2250 2568 2535 2500 2025 2048 2018
> 50 2K 2217 2213 2213 2487 2449 2604 2112 2016 2023
>
> Despite the fact the results don't match my expectation
You have a RAID1 with 15K SAS disks. I have a RAID10 with 8 7200 SATA
disks plus another RAID1 for the XLOG file system. Ten 7K SATA disks
on two file systems should be quite a bit faster than two 15K SAS
disks, right?
> (I suspect that there's something wrong with the PERC
> because, having the controller cache enabled make no
> difference in terms of TPS), it seems strange that disabling
> HT from the bios will give lesser TPS that HT disable through
> sysfs interface.
Well, all I can say is that I like my 3WARE controllers, and it's the
secondary reason why I moved away from Dell (the primary reason is
price).
Craig
>
> OS conf:
>
> vm.swappiness=0
> vm.overcommit_memory=2
> vm.dirty_ratio=2
> vm.dirty_background_ratio=1
> kernel.shmmax=3454820352
> kernel.shmall=2048341
> /sbin/blockdev --setra 8192 /dev/sdb
> $PGDATA is on ext4 (rw,noatime)
> Linux cloud 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012
> x86_64 x86_64 x86_64 GNU/Linux
> sdb scheduler is [cfq]
>
> DB conf:
>
> max_connections = 100
> shared_buffers = 3200MB
> work_mem = 30MB
> maintenance_work_mem = 800MB
> synchronous_commit = off
> full_page_writes = off
> checkpoint_segments = 40
> checkpoint_timeout = 5min
> checkpoint_completion_target = 0.9
> random_page_cost = 3.5
> effective_cache_size = 10GB
> log_autovacuum_min_duration = 0
> autovacuum_naptime = 5min
>
>
> Andrea
>
> p.s. as last try in the process of increasing TPS
> I've change the scheduler from cfq to deadline
> and for -c 5 t 20K I've got r1=3007, r2=2930 and r3=2985.
>
>
>
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 15:01 Claudio Freire <klaussfreire@gmail.com>
parent: Andrea Suisani <sickpig@opinioni.net>
1 sibling, 1 reply; 47+ messages in thread
From: Claudio Freire @ 2012-10-15 15:01 UTC (permalink / raw)
To: Andrea Suisani <sickpig@opinioni.net>; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On Mon, Oct 15, 2012 at 5:27 AM, Andrea Suisani <sickpig@opinioni.net> wrote:
> it seems strange that disabling
> HT from the bios will give lesser TPS that HT disable through
> sysfs interface.
It does prove they're not equivalent though.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 15:24 Andrea Suisani <sickpig@opinioni.net>
parent: Claudio Freire <klaussfreire@gmail.com>
0 siblings, 1 reply; 47+ messages in thread
From: Andrea Suisani @ 2012-10-15 15:24 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On 10/15/2012 05:01 PM, Claudio Freire wrote:
> On Mon, Oct 15, 2012 at 5:27 AM, Andrea Suisani <sickpig@opinioni.net> wrote:
>> it seems strange that disabling
>> HT from the bios will give lesser TPS that HT disable through
>> sysfs interface.
>
> It does prove they're not equivalent though.
>
sure you're right.
It's just that my bet was on a higher throughput
when HT was isabled from the BIOS (as you stated
previously in this thread).
Andrea
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 15:28 Claudio Freire <klaussfreire@gmail.com>
parent: Andrea Suisani <sickpig@opinioni.net>
0 siblings, 2 replies; 47+ messages in thread
From: Claudio Freire @ 2012-10-15 15:28 UTC (permalink / raw)
To: Andrea Suisani <sickpig@opinioni.net>; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On Mon, Oct 15, 2012 at 12:24 PM, Andrea Suisani <sickpig@opinioni.net> wrote:
>> It does prove they're not equivalent though.
>>
>
> sure you're right.
>
> It's just that my bet was on a higher throughput
> when HT was isabled from the BIOS (as you stated
> previously in this thread).
Yes, mine too. It's bizarre. If I were you, I'd look into it more
deeply. It may be a flaw in your test methodology (maybe you disabled
the wrong cores?). If not, it would be good to know why the extra TPS
to replicate elsewhere.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 15:32 Scott Marlowe <scott.marlowe@gmail.com>
parent: Craig James <cjames@emolecules.com>
2 siblings, 0 replies; 47+ messages in thread
From: Scott Marlowe @ 2012-10-15 15:32 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: Andrea Suisani <sickpig@opinioni.net>; Claudio Freire <klaussfreire@gmail.com>; pgsql-performance
On Mon, Oct 15, 2012 at 9:01 AM, Craig James <cjames@emolecules.com> wrote:
> On Mon, Oct 15, 2012 at 1:27 AM, Andrea Suisani <sickpig@opinioni.net> wrote:
>> (I suspect that there's something wrong with the PERC
>> because, having the controller cache enabled make no
>> difference in terms of TPS), it seems strange that disabling
>> HT from the bios will give lesser TPS that HT disable through
>> sysfs interface.
>
> Well, all I can say is that I like my 3WARE controllers, and it's the
> secondary reason why I moved away from Dell (the primary reason is
> price).
Mediocre performance, random lockups, and Dell's refusal to address
said lockups are the reasons I abandoned Dell's PERC controllers. My
preference is Areca 1680/1880, then 3Ware 96xx, then LSI, then
Adaptec. Areca's web interface on a dedicated ethernet port make them
super easy to configure while the machine is running with no need for
specialized software for a given OS, and they're performance and
reliability are great. The 3Wares are very solid with later model
BIOS on board. LSI gets a rasberry for MegaCLI, the 2nd klunkiest
interface ever, the worst being their horrible horrible BIOS boot
setup screen.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 15:34 Scott Marlowe <scott.marlowe@gmail.com>
parent: Claudio Freire <klaussfreire@gmail.com>
1 sibling, 2 replies; 47+ messages in thread
From: Scott Marlowe @ 2012-10-15 15:34 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Andrea Suisani <sickpig@opinioni.net>; Craig James <cjames@emolecules.com>; pgsql-performance
On Mon, Oct 15, 2012 at 9:28 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
> On Mon, Oct 15, 2012 at 12:24 PM, Andrea Suisani <sickpig@opinioni.net> wrote:
>> sure you're right.
>>
>> It's just that my bet was on a higher throughput
>> when HT was isabled from the BIOS (as you stated
>> previously in this thread).
>
> Yes, mine too. It's bizarre. If I were you, I'd look into it more
> deeply. It may be a flaw in your test methodology (maybe you disabled
> the wrong cores?). If not, it would be good to know why the extra TPS
> to replicate elsewhere.
I'd recommend more synthetic benchmarks when trying to compare systems
like this. bonnie++, the memory stream test that Greg Smith was
working on, and so on. Get an idea what core differences the machines
display under such testing.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 15:45 Andrea Suisani <sickpig@opinioni.net>
parent: Craig James <cjames@emolecules.com>
2 siblings, 0 replies; 47+ messages in thread
From: Andrea Suisani @ 2012-10-15 15:45 UTC (permalink / raw)
To: Craig James <cjames@emolecules.com>; +Cc: Claudio Freire <klaussfreire@gmail.com>; pgsql-performance
[cut]
>> TPS including connection establishing, pgbench run in a single
>> thread mode, connection made through unix socket, OS cache dropped
>> and Postgres restarted for every run.
>>
>> those are the results:
>>
>> HT HT SYSFS DIS HT BIOS DISABLE
>> -c -t r1 r2 r3 r1 r2 r3 r1 r2 r3
>> 5 20K 1641 1831 1496 2020 1974 2033 2005 1988 1967
>> 10 10K 2161 2134 2136 2277 2252 2216 1854 1824 1810
>> 20 5k 2550 2508 2558 2417 2388 2357 1924 1928 1954
>> 30 3333 2216 2272 2250 2333 2493 2496 1993 2009 2008
>> 40 2.5K 2179 2221 2250 2568 2535 2500 2025 2048 2018
>> 50 2K 2217 2213 2213 2487 2449 2604 2112 2016 2023
>>
>> Despite the fact the results don't match my expectation
>
> You have a RAID1 with 15K SAS disks. I have a RAID10 with 8 7200 SATA
> disks plus another RAID1 for the XLOG file system. Ten 7K SATA disks
> on two file systems should be quite a bit faster than two 15K SAS
> disks, right?
I think you're right. But I never have the chance to try such
a configuration in first person. But, yes, spreading I/O on two
different subsystems (xlog and pgdata) and having pgdata on
a RAID10 should surely outperform my RAID1 with 15K SAS disks.
>> (I suspect that there's something wrong with the PERC
>> because, having the controller cache enabled make no
>> difference in terms of TPS), it seems strange that disabling
>> HT from the bios will give lesser TPS that HT disable through
>> sysfs interface.
>
> Well, all I can say is that I like my 3WARE controllers, and it's the
> secondary reason why I moved away from Dell (the primary reason is
> price).
Something I surely will take into account the next time
I will buy a new server.
Andrea
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 15:56 Andrea Suisani <sickpig@opinioni.net>
parent: Claudio Freire <klaussfreire@gmail.com>
1 sibling, 0 replies; 47+ messages in thread
From: Andrea Suisani @ 2012-10-15 15:56 UTC (permalink / raw)
To: Claudio Freire <klaussfreire@gmail.com>; +Cc: Craig James <cjames@emolecules.com>; pgsql-performance
On 10/15/2012 05:28 PM, Claudio Freire wrote:
> On Mon, Oct 15, 2012 at 12:24 PM, Andrea Suisani <sickpig@opinioni.net> wrote:
>>> It does prove they're not equivalent though.
>>>
>>
>> sure you're right.
>>
>> It's just that my bet was on a higher throughput
>> when HT was isabled from the BIOS (as you stated
>> previously in this thread).
>
> Yes, mine too. It's bizarre. If I were you, I'd look into it more
> deeply. It may be a flaw in your test methodology (maybe you disabled
> the wrong cores?).
this is the first thing I thought after looking at the results
but I've double-checked cores topology (core_id, core_siblings_list end
friends under /sys/devices/system/cpu/cpu0/topology) and I seems
to me that I've disabled the right ones.
It could be that I've messed up with something else...
> If not, it would be good to know why the extra TPS
> to replicate elsewhere.
definitely I will try to understand the
probable causes performing other tests...
any hints are welcome :)
>
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-15 15:56 Andrea Suisani <sickpig@opinioni.net>
parent: Scott Marlowe <scott.marlowe@gmail.com>
1 sibling, 0 replies; 47+ messages in thread
From: Andrea Suisani @ 2012-10-15 15:56 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: Claudio Freire <klaussfreire@gmail.com>; Craig James <cjames@emolecules.com>; pgsql-performance
On 10/15/2012 05:34 PM, Scott Marlowe wrote:
> On Mon, Oct 15, 2012 at 9:28 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
>> On Mon, Oct 15, 2012 at 12:24 PM, Andrea Suisani <sickpig@opinioni.net> wrote:
>>> sure you're right.
>>>
>>> It's just that my bet was on a higher throughput
>>> when HT was isabled from the BIOS (as you stated
>>> previously in this thread).
>>
>> Yes, mine too. It's bizarre. If I were you, I'd look into it more
>> deeply. It may be a flaw in your test methodology (maybe you disabled
>> the wrong cores?). If not, it would be good to know why the extra TPS
>> to replicate elsewhere.
>
> I'd recommend more synthetic benchmarks when trying to compare systems
> like this. bonnie++, the memory stream test that Greg Smith was
> working on, and so on. Get an idea what core differences the machines
> display under such testing.
>
Will try tomorrow
thanks for the hint
Andrea
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-16 05:07 Marinos Yannikos <mjy@geizhals.at>
parent: Craig James <cjames@emolecules.com>
2 siblings, 0 replies; 47+ messages in thread
From: Marinos Yannikos @ 2012-10-16 05:07 UTC (permalink / raw)
To: pgsql-performance
On 15.10.2012 17:01, Craig James wrote:
>>>> On Thu, Oct 11, 2012 at 11:14 AM, Andrea Suisani <sickpig@opinioni.net>
>>>> wrote:
>>>>> I've googled a bit and I've found a way to disable hyperthreading
>>>>> without
>>>>> the need to reboot the system and entering the bios:
>>>>>
>>>>> echo 0 >/sys/devices/system/node/node0/cpuX/online
A safer method is probably to just add the "noht" kernel boot option and
reboot.
Did you set the same stride / stripe-width values on your FS when you
initialized them? Are both really freshly-made ext4 FS and not e.g. the
old one an ext3 mounted as ext4? Do all the disks have the same cache,
link speed and NCQ settings (for their own caches, not the controller;
try /c0/p0 show all etc. with tw_cli)?
-mjy
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-17 15:45 Andrea Suisani <sickpig@opinioni.net>
parent: Scott Marlowe <scott.marlowe@gmail.com>
1 sibling, 1 reply; 47+ messages in thread
From: Andrea Suisani @ 2012-10-17 15:45 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: Claudio Freire <klaussfreire@gmail.com>; Craig James <cjames@emolecules.com>; pgsql-performance
On 10/15/2012 05:34 PM, Scott Marlowe wrote:
> On Mon, Oct 15, 2012 at 9:28 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
>> On Mon, Oct 15, 2012 at 12:24 PM, Andrea Suisani <sickpig@opinioni.net> wrote:
>>> sure you're right.
>>>
>>> It's just that my bet was on a higher throughput
>>> when HT was isabled from the BIOS (as you stated
>>> previously in this thread).
>>
>> Yes, mine too. It's bizarre. If I were you, I'd look into it more
>> deeply. It may be a flaw in your test methodology (maybe you disabled
>> the wrong cores?). If not, it would be good to know why the extra TPS
>> to replicate elsewhere.
>
> I'd recommend more synthetic benchmarks when trying to compare systems
> like this. bonnie++,
you were right. bonnie++ (-f -n 0 -c 4) show that there's very little (if any)
difference in terms of sequential input whether or not cache is enabled on the
RAID1 (SAS 15K, sdb).
I've run 2 bonnie++ test with both cache enabled and disabled and what I get
(see attachments for more details) it's a 400MB/s sequential input (cache) vs
390MBs (nocache).
I dunno why but I would have expected a higher delta (due to the 512MB cache)
not a mere 10MB/s, but this is only based on my gut feeling.
I've also tried to test RAID1 array where the OS is installed (2 SATA 7.2Krpm, sda)
just to verify if cache effect is comparable with the one I get from SAS disks.
Well it seems that there's no cache effects or if it's is there is so small as to be
confused with the noise.
Both array are configured with this params
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Stripe Element Size : 64 KB
Disk Cache Policy : Disabled
those tests are performed with HT disable from the BIOS, but without
using noht kernel boot param. the scheduler for sdb was setted to deadline
while the default cfq for sda.
> the memory stream test that Greg Smith was
> working on, and so on.
this one https://github.com/gregs1104/stream-scaling, right?
I've executed the test with HT enabled, HT disabled from the BIOS
and HT disable using sys interface. Attached 3 graphs and related
text files
> Get an idea what core differences the machines
> display under such testing.
I'm trying... hard :)
Andrea
Attachments:
bonnie_sdb_cache_wo_pgsql (text/plain, 532B) download
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
cloud 32000M 180829 23 112581 15 401611 19 917.7 10
Latency 445ms 954ms 360ms 64788us
1.96,1.96,cloud,4,1350463530,32000M,,,,180829,23,112581,15,,,401611,19,917.7,10,,,,,,,,,,,,,,,,,,,445ms,954ms,,360ms,64788us,,,,,,
bonnie_sdb_cache_wo_pgsql_2 (text/plain, 534B) download
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
cloud 32000M 179259 23 112093 15 400021 18 600.9 26
Latency 670ms 514ms 99640us 86081us
1.96,1.96,cloud,4,1350465025,32000M,,,,179259,23,112093,15,,,400021,18,600.9,26,,,,,,,,,,,,,,,,,,,670ms,514ms,,99640us,86081us,,,,,,
bonnie_sdb_nocache_wo_pgsql (text/plain, 533B) download
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
cloud 32000M 174212 22 108909 15 387598 18 928.9 11
Latency 960ms 1397ms 967ms 71495us
1.96,1.96,cloud,4,1350465002,32000M,,,,174212,22,108909,15,,,387598,18,928.9,11,,,,,,,,,,,,,,,,,,,960ms,1397ms,,967ms,71495us,,,,,,
bonnie_sdb_nocache_wo_pgsql_2 (text/plain, 535B) download
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
cloud 32000M 175185 22 109229 15 391409 18 912.9 10
Latency 1002ms 1225ms 1059ms 91373us
1.96,1.96,cloud,4,1350466614,32000M,,,,175185,22,109229,15,,,391409,18,912.9,10,,,,,,,,,,,,,,,,,,,1002ms,1225ms,,1059ms,91373us,,,,,,
stream-ht_disabled_bios.png (image/png, 3.3K) download
view image
stream-ht_disabled_bios.txt (text/plain, 4.5K) download
=== CPU cache information === [288/809]
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu0 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu0 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu1 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu2 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu3 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu4 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu5 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 3 Cache: 15360K (Unified)
Total CPU system cache: 96141312 bytes
Suggested minimum array elements needed: 43700596
Array elements used: 43700596
=== CPU Core Summary ===
processor : 5
model name : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
cpu MHz : 1999.842
siblings : 6
=== Check and build stream ===
=== Testing up to 6 cores ===
-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 43700596, Offset = 0
Total memory required = 1000.2 MB.
Each test is run 10 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 1
-------------------------------------------------------------
Printing one line per active thread....
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 42747 microseconds.
(= 42747 clock ticks)
Increase the size of the arrays if this shows that
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Rate (MB/s) Avg time Min time Max time
Copy: 11189.6576 0.0625 0.0625 0.0626
Scale: 11226.3233 0.0623 0.0623 0.0624
Add: 12419.6669 0.0845 0.0844 0.0847
Triad: 12282.1773 0.0855 0.0854 0.0856
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------
Number of Threads requested = 2
Function Rate (MB/s) Avg time Min time Max time
Triad: 21066.0085 0.0500 0.0498 0.0502
Number of Threads requested = 3
Function Rate (MB/s) Avg time Min time Max time
Triad: 23206.8603 0.0453 0.0452 0.0455
Number of Threads requested = 4
Function Rate (MB/s) Avg time Min time Max time
Triad: 23555.5498 0.0446 0.0445 0.0446
Number of Threads requested = 5
Function Rate (MB/s) Avg time Min time Max time
Triad: 23424.7239 0.0448 0.0448 0.0449
Number of Threads requested = 6
Function Rate (MB/s) Avg time Min time Max time
Triad: 23298.1809 0.0451 0.0450 0.0452
stream-ht_disabled_sysfs.png (image/png, 3.3K) download
view image
stream-ht_disabled_sysfs.txt (text/plain, 1.8K) download
-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 43700596, Offset = 0
Total memory required = 1000.2 MB.
Each test is run 10 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 6
-------------------------------------------------------------
Printing one line per active thread....
Printing one line per active thread....
Printing one line per active thread....
Printing one line per active thread....
Printing one line per active thread....
Printing one line per active thread....
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 26278 microseconds.
(= 26278 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Rate (MB/s) Avg time Min time Max time
Copy: 20539.5415 0.0347 0.0340 0.0392
Scale: 20521.5758 0.0347 0.0341 0.0390
Add: 23627.7925 0.0455 0.0444 0.0526
Triad: 23673.6951 0.0450 0.0443 0.0499
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------
stream-ht_enabled.png (image/png, 3.6K) download
view image
stream-ht_enabled.txt (text/plain, 6.6K) download
=== CPU cache information ===
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu0 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu0 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu1 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu10 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu10 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu10 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu10 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu11 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu11 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu11 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu11 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu2 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu3 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu4 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu5 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu6 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu6 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu6 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu6 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu7 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu7 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu7 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu7 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu8 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu8 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu8 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu8 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu9 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu9 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu9 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu9 Level 3 Cache: 15360K (Unified)
Total CPU system cache: 192282624 bytes
Suggested minimum array elements needed: 87401192
Array elements used: 87401192
=== CPU Core Summary ===
processor : 11
model name : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
cpu MHz : 1999.946
siblings : 12
=== Check and build stream ===
=== Testing up to 12 cores ===
-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 87401192, Offset = 0
Total memory required = 2000.5 MB.
Each test is run 10 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 1
-------------------------------------------------------------
Printing one line per active thread....
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 87534 microseconds.
(= 87534 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Rate (MB/s) Avg time Min time Max time
Copy: 10611.6239 0.1319 0.1318 0.1320
Scale: 10975.4789 0.1275 0.1274 0.1277
Add: 11859.7763 0.1769 0.1769 0.1771
Triad: 12273.3535 0.1710 0.1709 0.1712
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------
Number of Threads requested = 2
Function Rate (MB/s) Avg time Min time Max time
Triad: 21378.2018 0.0983 0.0981 0.0985
Number of Threads requested = 3
Function Rate (MB/s) Avg time Min time Max time
Triad: 24124.2561 0.0875 0.0870 0.0883
Number of Threads requested = 4
Function Rate (MB/s) Avg time Min time Max time
Triad: 24631.0616 0.0855 0.0852 0.0858
Number of Threads requested = 5
Function Rate (MB/s) Avg time Min time Max time
Triad: 24246.1186 0.0868 0.0865 0.0872
Number of Threads requested = 6
Function Rate (MB/s) Avg time Min time Max time
Triad: 23878.2058 0.0880 0.0878 0.0884
Number of Threads requested = 7
Function Rate (MB/s) Avg time Min time Max time
Triad: 22625.2297 0.0952 0.0927 0.0990
Number of Threads requested = 8
Function Rate (MB/s) Avg time Min time Max time
Triad: 23126.5826 0.0982 0.0907 0.1056
Number of Threads requested = 9
Function Rate (MB/s) Avg time Min time Max time
Triad: 23425.1605 0.0950 0.0895 0.1040
Number of Threads requested = 10
Function Rate (MB/s) Avg time Min time Max time
Triad: 22919.8752 0.0937 0.0915 0.0954
Number of Threads requested = 11
Function Rate (MB/s) Avg time Min time Max time
Triad: 23267.4353 0.0947 0.0902 0.1027
Number of Threads requested = 12
Function Rate (MB/s) Avg time Min time Max time
Triad: 23229.3473 0.0905 0.0903 0.0907
* Re: Two identical systems, radically different performance
@ 2012-10-17 16:35 Scott Marlowe <scott.marlowe@gmail.com>
parent: Andrea Suisani <sickpig@opinioni.net>
0 siblings, 1 reply; 47+ messages in thread
From: Scott Marlowe @ 2012-10-17 16:35 UTC (permalink / raw)
To: Andrea Suisani <sickpig@opinioni.net>; +Cc: Claudio Freire <klaussfreire@gmail.com>; Craig James <cjames@emolecules.com>; pgsql-performance
On Wed, Oct 17, 2012 at 9:45 AM, Andrea Suisani <sickpig@opinioni.net> wrote:
> On 10/15/2012 05:34 PM, Scott Marlowe wrote:
>> I'd recommend more synthetic benchmarks when trying to compare systems
>> like this. bonnie++,
>
>
> you were right. bonnie++ (-f -n 0 -c 4) show that there's very little (if
> any)
> difference in terms of sequential input whether or not cache is enabled on
> the
> RAID1 (SAS 15K, sdb).
I'm mainly wanting to know the difference between the two systems, so
if you can run it on the old and new machine and compare that that's
the real test.
> I've run 2 bonnie++ test with both cache enabled and disabled and what I get
> (see attachments for more details) it's a 400MB/s sequential input (cache)
> vs
> 390MBs (nocache).
>
> I dunno why but I would have expected a higher delta (due to the 512MB
> cache)
> not a mere 10MB/s, but this is only based on my gut feeling.
Well the sequential throughput doesn't really rely on caching. It's
the random writes that benefit from caching, and the other things
(random reads and seq read/write) that indirectly benefit because the
random writes are so much faster that they no longer get in the way.
So mostly compare random access between the old and new machines and
look for differences there.
>> the memory stream test that Greg Smith was
>> working on, and so on.
>
>
> this one https://github.com/gregs1104/stream-scaling, right?
Yep.
> I've executed the test with HT enabled, HT disabled from the BIOS
> and HT disable using sys interface. Attached 3 graphs and related
> text files
Well it's pretty meh. I'd like to see the older machine compared to
the newer one here tho.
> I'm trying... hard :)
You're doing great. These problems take effort to sort out.
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-18 06:57 Andrea Suisani <sickpig@opinioni.net>
parent: Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 47+ messages in thread
From: Andrea Suisani @ 2012-10-18 06:57 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: Claudio Freire <klaussfreire@gmail.com>; Craig James <cjames@emolecules.com>; pgsql-performance
On 10/17/2012 06:35 PM, Scott Marlowe wrote:
> On Wed, Oct 17, 2012 at 9:45 AM, Andrea Suisani <sickpig@opinioni.net> wrote:
>> On 10/15/2012 05:34 PM, Scott Marlowe wrote:
>>> I'd recommend more synthetic benchmarks when trying to compare systems
>>> like this. bonnie++,
>>
>>
>> you were right. bonnie++ (-f -n 0 -c 4) show that there's very little (if
>> any)
>> difference in terms of sequential input whether or not cache is enabled on
>> the
>> RAID1 (SAS 15K, sdb).
Maybe there's a misunderstanding here.. :) Craig (James) is the one
the had started this thread. I've joined later suggesting a way to
disable HT without rebooting (using sysfs interface), trying to avoid
a trip to the data-center to Craig.
At that point Claudio Freire wondering if disabling HT from sysfs
would have removed the performance penalty that Craig has experienced.
So I decided to test this on a brand new box that I've just bought.
When performing this test I've discovered by chance that
the raid controller (PERC H710) behave in an unexpected way,
cause the hw cache has almost no effect in terms of TPS in
a pgbench session.
> I'm mainly wanting to know the difference between the two systems, so
> if you can run it on the old and new machine and compare that that's
> the real test.
This is something that Craig can do.
[cut]
>> I dunno why but I would have expected a higher delta (due to the 512MB
>> cache)
>> not a mere 10MB/s, but this is only based on my gut feeling.
>
> Well the sequential throughput doesn't really rely on caching. It's
> the random writes that benefit from caching, and the other things
> (random reads and seq read/write) that indirectly benefit because the
> random writes are so much faster that they no longer get in the way.
> So mostly compare random access between the old and new machines and
> look for differences there.
make sense.
I will focus on tests that measure random path access.
>>> the memory stream test that Greg Smith was
>>> working on, and so on.
>>
>>
>> this one https://github.com/gregs1104/stream-scaling, right?
>
> Yep.
>
>> I've executed the test with HT enabled, HT disabled from the BIOS
>> and HT disable using sys interface. Attached 3 graphs and related
>> text files
>
> Well it's pretty meh.
:/
do you think that Xeon Xeon 5620 perform poorly ?
> I'd like to see the older machine compared to
> the newer one here tho.
also this one is on Craig side.
>> I'm trying... hard :)
>
> You're doing great. These problems take effort to sort out.
thanks
^ permalink raw reply [nested|flat] 47+ messages in thread
* Re: Two identical systems, radically different performance
@ 2012-10-18 16:39 Craig James <cjames@emolecules.com>
parent: Andrea Suisani <sickpig@opinioni.net>
0 siblings, 0 replies; 47+ messages in thread
From: Craig James @ 2012-10-18 16:39 UTC (permalink / raw)
To: Andrea Suisani <sickpig@opinioni.net>; +Cc: Scott Marlowe <scott.marlowe@gmail.com>; Claudio Freire <klaussfreire@gmail.com>; pgsql-performance
On Wed, Oct 17, 2012 at 11:57 PM, Andrea Suisani <sickpig@opinioni.net> wrote:
> On 10/17/2012 06:35 PM, Scott Marlowe wrote:
>>
>> On Wed, Oct 17, 2012 at 9:45 AM, Andrea Suisani <sickpig@opinioni.net>
>> wrote:
>>>
>>> On 10/15/2012 05:34 PM, Scott Marlowe wrote:
>>>>
>>>> I'd recommend more synthetic benchmarks when trying to compare systems
>>>> like this. bonnie++,
>>>
>>>
>>>
>>> you were right. bonnie++ (-f -n 0 -c 4) show that there's very little (if
>>> any)
>>> difference in terms of sequential input whether or not cache is enabled
>>> on
>>> the
>>> RAID1 (SAS 15K, sdb).
>
>
> Maybe there's a misunderstanding here.. :) Craig (James) is the one
> the had started this thread. I've joined later suggesting a way to
> disable HT without rebooting (using sysfs interface), trying to avoid
> a trip to the data-center to Craig.
>
> At that point Claudio Freire wondering if disabling HT from sysfs
> would have removed the performance penalty that Craig has experienced.
>
> So I decided to test this on a brand new box that I've just bought.
>
> When performing this test I've discovered by chance that
> the raid controller (PERC H710) behave in an unexpected way,
> cause the hw cache has almost no effect in terms of TPS in
> a pgbench session.
>
>> I'm mainly wanting to know the difference between the two systems, so
>> if you can run it on the old and new machine and compare that that's
>> the real test.
>
>
> This is something that Craig can do.
Too late ... the new machine is in production.
Craig
>
> [cut]
>
>>> I dunno why but I would have expected a higher delta (due to the 512MB
>>> cache)
>>> not a mere 10MB/s, but this is only based on my gut feeling.
>
>>
>>
>> Well the sequential throughput doesn't really rely on caching. It's
>> the random writes that benefit from caching, and the other things
>> (random reads and seq read/write) that indirectly benefit because the
>> random writes are so much faster that they no longer get in the way.
>> So mostly compare random access between the old and new machines and
>> look for differences there.
>
>
> make sense.
>
> I will focus on tests that measure random path access.
>
>>>> the memory stream test that Greg Smith was
>>>> working on, and so on.
>>>
>>>
>>>
>>> this one https://github.com/gregs1104/stream-scaling, right?
>>
>>
>> Yep.
>>
>>> I've executed the test with HT enabled, HT disabled from the BIOS
>>> and HT disable using sys interface. Attached 3 graphs and related
>>> text files
>>
>>
>> Well it's pretty meh.
>
>
> :/
>
> do you think that Xeon Xeon 5620 perform poorly ?
>
>> I'd like to see the older machine compared to
>> the newer one here tho.
>
>
> also this one is on Craig side.
>
>>> I'm trying... hard :)
>>
>>
>> You're doing great. These problems take effort to sort out.
>
>
> thanks
>
>
^ permalink raw reply [nested|flat] 47+ messages in thread
end of thread, other threads:[~2012-10-18 16:39 UTC | newest]
Thread overview: 47+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-10-08 21:45 Two identical systems, radically different performance Craig James <cjames@emolecules.com>
2012-10-08 21:57 ` Evgeny Shishkin <itparanoia@gmail.com>
2012-10-08 22:06 ` Craig James <cjames@emolecules.com>
2012-10-08 22:08 ` Evgeny Shishkin <itparanoia@gmail.com>
2012-10-08 22:09 ` Claudio Freire <klaussfreire@gmail.com>
2012-10-08 22:25 ` Craig James <cjames@emolecules.com>
2012-10-08 22:44 ` Claudio Freire <klaussfreire@gmail.com>
2012-10-08 22:46 ` Evgeny Shishkin <itparanoia@gmail.com>
2012-10-08 22:48 ` Craig James <cjames@emolecules.com>
2012-10-08 22:50 ` Claudio Freire <klaussfreire@gmail.com>
2012-10-08 23:03 ` Craig James <cjames@emolecules.com>
2012-10-08 23:12 ` Claudio Freire <klaussfreire@gmail.com>
2012-10-08 23:16 ` Tomas Vondra <tv@fuzzy.cz>
2012-10-08 23:10 ` Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
2012-10-08 22:16 ` Steve Crawford <scrawford@pinpointresearch.com>
2012-10-08 22:28 ` Imre Samu <pella.samu@gmail.com>
2012-10-08 22:29 ` Craig James <cjames@emolecules.com>
2012-10-08 23:40 ` Craig James <cjames@emolecules.com>
2012-10-08 23:52 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
2012-10-09 00:00 ` Ants Aasma <ants@cybertec.at>
2012-10-09 16:02 ` Shaun Thomas <sthomas@optionshouse.com>
2012-10-09 16:41 ` Craig James <cjames@emolecules.com>
2012-10-09 16:14 ` David Thomas <david@digitaldogma.org>
2012-10-09 16:43 ` Craig James <cjames@emolecules.com>
2012-10-11 14:14 ` Andrea Suisani <sickpig@opinioni.net>
2012-10-11 14:19 ` Claudio Freire <klaussfreire@gmail.com>
2012-10-11 14:40 ` Andrea Suisani <sickpig@opinioni.net>
2012-10-15 08:27 ` Andrea Suisani <sickpig@opinioni.net>
2012-10-15 15:01 ` Craig James <cjames@emolecules.com>
2012-10-15 15:32 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-10-15 15:45 ` Andrea Suisani <sickpig@opinioni.net>
2012-10-16 05:07 ` Marinos Yannikos <mjy@geizhals.at>
2012-10-15 15:01 ` Claudio Freire <klaussfreire@gmail.com>
2012-10-15 15:24 ` Andrea Suisani <sickpig@opinioni.net>
2012-10-15 15:28 ` Claudio Freire <klaussfreire@gmail.com>
2012-10-15 15:34 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-10-15 15:56 ` Andrea Suisani <sickpig@opinioni.net>
2012-10-17 15:45 ` Andrea Suisani <sickpig@opinioni.net>
2012-10-17 16:35 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-10-18 06:57 ` Andrea Suisani <sickpig@opinioni.net>
2012-10-18 16:39 ` Craig James <cjames@emolecules.com>
2012-10-15 15:56 ` Andrea Suisani <sickpig@opinioni.net>
2012-10-08 22:33 ` Evgeny Shishkin <itparanoia@gmail.com>
2012-10-08 22:42 ` Craig James <cjames@emolecules.com>
2012-10-08 23:24 ` Tomas Vondra <tv@fuzzy.cz>
2012-10-08 23:30 ` Evgeny Shishkin <itparanoia@gmail.com>
2012-10-09 11:20 ` Yeb Havinga <yebhavinga@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox