From: Andrea Suisani <sickpig@opinioni.net>
To: Scott Marlowe <scott.marlowe@gmail.com>
Cc: Claudio Freire <klaussfreire@gmail.com>
Cc: Craig James <cjames@emolecules.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Two identical systems, radically different performance
Date: Wed, 17 Oct 2012 17:45:23 +0200
Message-ID: <507ED293.6010803@opinioni.net> (raw)
In-Reply-To: <CAOR=d=02vBk24Mt7cJd9SBxjAcL26Ao=N_8o7ZxB-qe0VW1T8A@mail.gmail.com>
References: <CAFwQ8rcvw4mSoJ9q+ouONCeB9dg1T9gAUV5J27MpVZrehRHayQ@mail.gmail.com>
<CAFwQ8rfm2AcuoFHDfBLA_hg7ffbsYNsrsaJPDqcOeGqaVW02OQ@mail.gmail.com>
<CAFwQ8rcyCKcoeEn62_QHMRSmJM03ULERtW6_R=3TaO0twOEV9A@mail.gmail.com>
<5076D433.7090605@opinioni.net>
<CAGTBQpYGyq70q_S_dUngSwB7yLbbCxetnfXpaMC9LEr7zL6iWA@mail.gmail.com>
<5076DA4E.80504@opinioni.net>
<507BC8DE.9050109@opinioni.net>
<CAGTBQpaHGxo9iFWoTJhuU+eTOK13Q-YkQk+9QC7xuT5HDmbv6w@mail.gmail.com>
<507C2ACA.3020503@opinioni.net>
<CAGTBQpbt6sFkbGqELBzesTuz=v=iY3NQPdf+_AppfN7KNzkJEQ@mail.gmail.com>
<CAOR=d=02vBk24Mt7cJd9SBxjAcL26Ao=N_8o7ZxB-qe0VW1T8A@mail.gmail.com>
On 10/15/2012 05:34 PM, Scott Marlowe wrote:
> On Mon, Oct 15, 2012 at 9:28 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
>> On Mon, Oct 15, 2012 at 12:24 PM, Andrea Suisani <sickpig@opinioni.net> wrote:
>>> sure you're right.
>>>
>>> It's just that my bet was on a higher throughput
>>> when HT was isabled from the BIOS (as you stated
>>> previously in this thread).
>>
>> Yes, mine too. It's bizarre. If I were you, I'd look into it more
>> deeply. It may be a flaw in your test methodology (maybe you disabled
>> the wrong cores?). If not, it would be good to know why the extra TPS
>> to replicate elsewhere.
>
> I'd recommend more synthetic benchmarks when trying to compare systems
> like this. bonnie++,
you were right. bonnie++ (-f -n 0 -c 4) show that there's very little (if any)
difference in terms of sequential input whether or not cache is enabled on the
RAID1 (SAS 15K, sdb).
I've run 2 bonnie++ test with both cache enabled and disabled and what I get
(see attachments for more details) it's a 400MB/s sequential input (cache) vs
390MBs (nocache).
I dunno why but I would have expected a higher delta (due to the 512MB cache)
not a mere 10MB/s, but this is only based on my gut feeling.
I've also tried to test RAID1 array where the OS is installed (2 SATA 7.2Krpm, sda)
just to verify if cache effect is comparable with the one I get from SAS disks.
Well it seems that there's no cache effects or if it's is there is so small as to be
confused with the noise.
Both array are configured with this params
Read Policy : Adaptive Read Ahead
Write Policy : Write Back
Stripe Element Size : 64 KB
Disk Cache Policy : Disabled
those tests are performed with HT disable from the BIOS, but without
using noht kernel boot param. the scheduler for sdb was setted to deadline
while the default cfq for sda.
> the memory stream test that Greg Smith was
> working on, and so on.
this one https://github.com/gregs1104/stream-scaling, right?
I've executed the test with HT enabled, HT disabled from the BIOS
and HT disable using sys interface. Attached 3 graphs and related
text files
> Get an idea what core differences the machines
> display under such testing.
I'm trying... hard :)
Andrea
Attachments:bonnie_sdb_cache_wo_pgsql (text/plain, 532B) download
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
cloud 32000M 180829 23 112581 15 401611 19 917.7 10
Latency 445ms 954ms 360ms 64788us
1.96,1.96,cloud,4,1350463530,32000M,,,,180829,23,112581,15,,,401611,19,917.7,10,,,,,,,,,,,,,,,,,,,445ms,954ms,,360ms,64788us,,,,,,
bonnie_sdb_cache_wo_pgsql_2 (text/plain, 534B) download
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
cloud 32000M 179259 23 112093 15 400021 18 600.9 26
Latency 670ms 514ms 99640us 86081us
1.96,1.96,cloud,4,1350465025,32000M,,,,179259,23,112093,15,,,400021,18,600.9,26,,,,,,,,,,,,,,,,,,,670ms,514ms,,99640us,86081us,,,,,,
bonnie_sdb_nocache_wo_pgsql (text/plain, 533B) download
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
cloud 32000M 174212 22 108909 15 387598 18 928.9 11
Latency 960ms 1397ms 967ms 71495us
1.96,1.96,cloud,4,1350465002,32000M,,,,174212,22,108909,15,,,387598,18,928.9,11,,,,,,,,,,,,,,,,,,,960ms,1397ms,,967ms,71495us,,,,,,
bonnie_sdb_nocache_wo_pgsql_2 (text/plain, 535B) download
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
cloud 32000M 175185 22 109229 15 391409 18 912.9 10
Latency 1002ms 1225ms 1059ms 91373us
1.96,1.96,cloud,4,1350466614,32000M,,,,175185,22,109229,15,,,391409,18,912.9,10,,,,,,,,,,,,,,,,,,,1002ms,1225ms,,1059ms,91373us,,,,,,
stream-ht_disabled_bios.png (image/png, 3.3K) downloadview imagestream-ht_disabled_bios.txt (text/plain, 4.5K) download
=== CPU cache information === [288/809]
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu0 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu0 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu1 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu2 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu3 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu4 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu5 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 3 Cache: 15360K (Unified)
Total CPU system cache: 96141312 bytes
Suggested minimum array elements needed: 43700596
Array elements used: 43700596
=== CPU Core Summary ===
processor : 5
model name : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
cpu MHz : 1999.842
siblings : 6
=== Check and build stream ===
=== Testing up to 6 cores ===
-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 43700596, Offset = 0
Total memory required = 1000.2 MB.
Each test is run 10 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 1
-------------------------------------------------------------
Printing one line per active thread....
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 42747 microseconds.
(= 42747 clock ticks)
Increase the size of the arrays if this shows that
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Rate (MB/s) Avg time Min time Max time
Copy: 11189.6576 0.0625 0.0625 0.0626
Scale: 11226.3233 0.0623 0.0623 0.0624
Add: 12419.6669 0.0845 0.0844 0.0847
Triad: 12282.1773 0.0855 0.0854 0.0856
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------
Number of Threads requested = 2
Function Rate (MB/s) Avg time Min time Max time
Triad: 21066.0085 0.0500 0.0498 0.0502
Number of Threads requested = 3
Function Rate (MB/s) Avg time Min time Max time
Triad: 23206.8603 0.0453 0.0452 0.0455
Number of Threads requested = 4
Function Rate (MB/s) Avg time Min time Max time
Triad: 23555.5498 0.0446 0.0445 0.0446
Number of Threads requested = 5
Function Rate (MB/s) Avg time Min time Max time
Triad: 23424.7239 0.0448 0.0448 0.0449
Number of Threads requested = 6
Function Rate (MB/s) Avg time Min time Max time
Triad: 23298.1809 0.0451 0.0450 0.0452
stream-ht_disabled_sysfs.png (image/png, 3.3K) downloadview imagestream-ht_disabled_sysfs.txt (text/plain, 1.8K) download
-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 43700596, Offset = 0
Total memory required = 1000.2 MB.
Each test is run 10 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 6
-------------------------------------------------------------
Printing one line per active thread....
Printing one line per active thread....
Printing one line per active thread....
Printing one line per active thread....
Printing one line per active thread....
Printing one line per active thread....
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 26278 microseconds.
(= 26278 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Rate (MB/s) Avg time Min time Max time
Copy: 20539.5415 0.0347 0.0340 0.0392
Scale: 20521.5758 0.0347 0.0341 0.0390
Add: 23627.7925 0.0455 0.0444 0.0526
Triad: 23673.6951 0.0450 0.0443 0.0499
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------
stream-ht_enabled.png (image/png, 3.6K) downloadview imagestream-ht_enabled.txt (text/plain, 6.6K) download
=== CPU cache information ===
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu0 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu0 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu1 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu10 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu10 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu10 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu10 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu11 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu11 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu11 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu11 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu2 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu3 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu4 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu5 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu6 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu6 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu6 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu6 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu7 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu7 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu7 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu7 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu8 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu8 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu8 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu8 Level 3 Cache: 15360K (Unified)
CPU /sys/devices/system/cpu/cpu9 Level 1 Cache: 32K (Data)
CPU /sys/devices/system/cpu/cpu9 Level 1 Cache: 32K (Instruction)
CPU /sys/devices/system/cpu/cpu9 Level 2 Cache: 256K (Unified)
CPU /sys/devices/system/cpu/cpu9 Level 3 Cache: 15360K (Unified)
Total CPU system cache: 192282624 bytes
Suggested minimum array elements needed: 87401192
Array elements used: 87401192
=== CPU Core Summary ===
processor : 11
model name : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
cpu MHz : 1999.946
siblings : 12
=== Check and build stream ===
=== Testing up to 12 cores ===
-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 87401192, Offset = 0
Total memory required = 2000.5 MB.
Each test is run 10 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 1
-------------------------------------------------------------
Printing one line per active thread....
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 87534 microseconds.
(= 87534 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Rate (MB/s) Avg time Min time Max time
Copy: 10611.6239 0.1319 0.1318 0.1320
Scale: 10975.4789 0.1275 0.1274 0.1277
Add: 11859.7763 0.1769 0.1769 0.1771
Triad: 12273.3535 0.1710 0.1709 0.1712
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------
Number of Threads requested = 2
Function Rate (MB/s) Avg time Min time Max time
Triad: 21378.2018 0.0983 0.0981 0.0985
Number of Threads requested = 3
Function Rate (MB/s) Avg time Min time Max time
Triad: 24124.2561 0.0875 0.0870 0.0883
Number of Threads requested = 4
Function Rate (MB/s) Avg time Min time Max time
Triad: 24631.0616 0.0855 0.0852 0.0858
Number of Threads requested = 5
Function Rate (MB/s) Avg time Min time Max time
Triad: 24246.1186 0.0868 0.0865 0.0872
Number of Threads requested = 6
Function Rate (MB/s) Avg time Min time Max time
Triad: 23878.2058 0.0880 0.0878 0.0884
Number of Threads requested = 7
Function Rate (MB/s) Avg time Min time Max time
Triad: 22625.2297 0.0952 0.0927 0.0990
Number of Threads requested = 8
Function Rate (MB/s) Avg time Min time Max time
Triad: 23126.5826 0.0982 0.0907 0.1056
Number of Threads requested = 9
Function Rate (MB/s) Avg time Min time Max time
Triad: 23425.1605 0.0950 0.0895 0.1040
Number of Threads requested = 10
Function Rate (MB/s) Avg time Min time Max time
Triad: 22919.8752 0.0937 0.0915 0.0954
Number of Threads requested = 11
Function Rate (MB/s) Avg time Min time Max time
Triad: 23267.4353 0.0947 0.0902 0.1027
Number of Threads requested = 12
Function Rate (MB/s) Avg time Min time Max time
Triad: 23229.3473 0.0905 0.0903 0.0907
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: sickpig@opinioni.net, scott.marlowe@gmail.com, klaussfreire@gmail.com, cjames@emolecules.com
Subject: Re: Two identical systems, radically different performance
In-Reply-To: <507ED293.6010803@opinioni.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox