pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedRAID 10 Benchmark with different I/O schedulers (was: Performance increase with elevator=deadline)
10+ messages / 6 participants
[nested] [flat]
* RAID 10 Benchmark with different I/O schedulers (was: Performance increase with elevator=deadline)
@ 2008-05-05 23:33 Craig James <craig_james@emolecules.com>
0 siblings, 3 replies; 10+ messages in thread
From: Craig James @ 2008-05-05 23:33 UTC (permalink / raw)
To: pgsql-performance
I had the opportunity to do more testing on another new server to see whether the kernel's I/O scheduling makes any difference. Conclusion: On a battery-backed RAID 10 system, the kernel's I/O scheduling algorithm has no effect. This makes sense, since a battery-backed cache will supercede any I/O rescheduling that the kernel tries to do.
Hardware:
Dell 2950
8 CPU (Intel 2GHz Xeon)
8 GB memory
Dell Perc 6i with battery-backed cache
RAID 10 of 8x 146GB SAS 10K 2.5" disks
Software:
Linux 2.6.24, 64-bit
XFS file system
Postgres 8.3.0
max_connections = 1000
shared_buffers = 2000MB
work_mem = 256MB
max_fsm_pages = 1000000
max_fsm_relations = 5000
synchronous_commit = off
wal_buffers = 256kB
checkpoint_segments = 30
effective_cache_size = 4GB
Each test was run 5 times:
drop database test
create database test
pgbench -i -s 20 -U test
pgbench -c 10 -t 50000 -v -U test
The I/O scheduler was changed on-the-fly using (for example) "echo cfq >/sys/block/sda/queue/scheduler".
Autovacuum was turned off during the test.
Here are the results. The numbers are those reported as "tps = xxxx (including connections establishing)" (which were almost identical to the "excluding..." tps number).
I/O Sched AVG Test1 Test2 Test3 Test4 Test5
--------- ----- ----- ----- ----- ----- -----
cfq 3355 3646 3207 3132 3204 3584
noop 3163 2901 3190 3293 3124 3308
deadline 3547 3923 3722 3351 3484 3254
anticipatory 3384 3453 3916 2944 3451 3156
As you can see, the averages are very close -- closer than the "noise" between runs. As far as I can tell, there is no significant advantage, or even any significant difference, between the various I/O scheduler algorithms.
(It also reinforces what the pgbench man page says: Short runs aren't useful. Even these two-minute runs have a lot of variability. Before I turned off AutoVacuum, the variability was more like 50% between runs.)
Craig
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: RAID 10 Benchmark with different I/O schedulers (was: Performance increase with elevator=deadline)
@ 2008-05-06 03:15 Scott Marlowe <scott.marlowe@gmail.com>
parent: Craig James <craig_james@emolecules.com>
2 siblings, 0 replies; 10+ messages in thread
From: Scott Marlowe @ 2008-05-06 03:15 UTC (permalink / raw)
To: Craig James <craig_james@emolecules.com>; +Cc: pgsql-performance
On Mon, May 5, 2008 at 5:33 PM, Craig James <craig_james@emolecules.com> wrote:
>
> (It also reinforces what the pgbench man page says: Short runs aren't
> useful. Even these two-minute runs have a lot of variability. Before I
> turned off AutoVacuum, the variability was more like 50% between runs.)
I'd suggest a couple things for more realistic tests. Run the tests
much longer, say 30 minutes to an hour. Crank up your scaling factor
until your test db is larger than memory. Turn on autovacuum, maybe
raising the cost / delay factors so it doesn't affect performance too
negatively. And lastly tuning the bgwriter so that checkpoints are
short and don't interfere too much.
My guess is if you let it run for a while, you'll get a much more
reliable number.
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: RAID 10 Benchmark with different I/O schedulers (was: Performance increase with elevator=deadline)
@ 2008-05-06 03:59 Greg Smith <gsmith@gregsmith.com>
parent: Craig James <craig_james@emolecules.com>
2 siblings, 1 reply; 10+ messages in thread
From: Greg Smith @ 2008-05-06 03:59 UTC (permalink / raw)
To: Craig James <craig_james@emolecules.com>; +Cc: pgsql-performance
On Mon, 5 May 2008, Craig James wrote:
> pgbench -i -s 20 -U test
That's way too low to expect you'll see a difference in I/O schedulers.
A scale of 20 is giving you a 320MB database, you can fit the whole thing
in RAM and almost all of it on your controller cache. What's there to
schedule? You're just moving between buffers that are generally large
enough to hold most of what they need.
> pgbench -c 10 -t 50000 -v -U test
This is OK, because when you increase the size you're not going to be
pushing 3500 TPS anymore and this test will take quite a while.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: RAID 10 Benchmark with different I/O schedulers (was: Performance increase with elevator=deadline)
@ 2008-05-06 12:26 Jeff <threshar@torgo.978.org>
parent: Craig James <craig_james@emolecules.com>
2 siblings, 0 replies; 10+ messages in thread
From: Jeff @ 2008-05-06 12:26 UTC (permalink / raw)
To: Craig James <craig_james@emolecules.com>; +Cc: pgsql-performance
On May 5, 2008, at 7:33 PM, Craig James wrote:
> I had the opportunity to do more testing on another new server to
> see whether the kernel's I/O scheduling makes any difference.
> Conclusion: On a battery-backed RAID 10 system, the kernel's I/O
> scheduling algorithm has no effect. This makes sense, since a
> battery-backed cache will supercede any I/O rescheduling that the
> kernel tries to do.
>
this goes against my real world experience here.
> pgbench -i -s 20 -U test
> pgbench -c 10 -t 50000 -v -U test
>
You should use a sample size of 2x ram to get a more realistic number,
or try out my pgiosim tool on pgfoundry which "sort of" simulates an
index scan. I posted numbers from that a month or two ago here.
--
Jeff Trout <jeff@jefftrout.com>
http://www.stuarthamm.net/
http://www.dellsmartexitin.com/
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: RAID 10 Benchmark with different I/O schedulers
@ 2008-05-06 18:30 Craig James <craig_james@emolecules.com>
parent: Greg Smith <gsmith@gregsmith.com>
0 siblings, 2 replies; 10+ messages in thread
From: Craig James @ 2008-05-06 18:30 UTC (permalink / raw)
To: Greg Smith <gsmith@gregsmith.com>; +Cc: pgsql-performance
Greg Smith wrote:
> On Mon, 5 May 2008, Craig James wrote:
>
>> pgbench -i -s 20 -U test
>
> That's way too low to expect you'll see a difference in I/O schedulers.
> A scale of 20 is giving you a 320MB database, you can fit the whole
> thing in RAM and almost all of it on your controller cache. What's
> there to schedule? You're just moving between buffers that are
> generally large enough to hold most of what they need.
Test repeated with:
autovacuum enabled
database destroyed and recreated between runs
pgbench -i -s 600 ...
pgbench -c 10 -t 50000 -n ...
I/O Sched AVG Test1 Test2
--------- ----- ----- -----
cfq 705 695 715
noop 758 769 747
deadline 741 705 775
anticipatory 494 477 511
I only did two runs of each, which took about 24 minutes. Like the first round of tests, the "noise" in the measurements (about 10%) exceeds the difference between scheduler-algorithm performance, except that "anticipatory" seems to be measurably slower.
So it still looks like cfq, noop and deadline are more or less equivalent when used with a battery-backed RAID.
Craig
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: RAID 10 Benchmark with different I/O schedulers
@ 2008-05-06 20:21 Greg Smith <gsmith@gregsmith.com>
parent: Craig James <craig_james@emolecules.com>
1 sibling, 1 reply; 10+ messages in thread
From: Greg Smith @ 2008-05-06 20:21 UTC (permalink / raw)
To: Craig James <craig_james@emolecules.com>; +Cc: pgsql-performance
On Tue, 6 May 2008, Craig James wrote:
> I only did two runs of each, which took about 24 minutes. Like the first
> round of tests, the "noise" in the measurements (about 10%) exceeds the
> difference between scheduler-algorithm performance, except that
> "anticipatory" seems to be measurably slower.
Those are much better results. Any test that says anticipatory is
anything other than useless for database system use with a good controller
I presume is broken, so that's how I know you're in the right ballpark now
but weren't before.
In order to actually get some useful data out of the noise that is
pgbench, you need a lot more measurements of longer runs. As perspective,
the last time I did something in this area, in order to get enough data to
get a clear picture I ran tests for 12 hours. I'm hoping to repeat that
soon with some more common hardware that gives useful results I can give
out.
> So it still looks like cfq, noop and deadline are more or less equivalent
> when used with a battery-backed RAID.
I think it's fair to say they're within 10% of one another on raw
throughput. The thing you're not measuring here is worst-case latency,
and that's where there might be a more interesting difference. Most tests
I've seen suggest deadline is the best in that regard, cfq the worst, and
where noop fits in depends on the underlying controller.
pgbench produces log files with latency measurements if you pass it "-l".
Here's a snippet of shell that runs pgbench then looks at the resulting
latency results for the worst 5 numbers:
pgbench ... -l &
p=$!
wait $p
mv pgbench_log.${p} pgbench.log
echo Worst latency results:
cat pgbench.log | cut -f 3 -d " " | sort -n | tail -n 5
However, that may not give you much useful info either--in most cases
checkpoint issues kind of swamp the worst-base behavior in PostgreSQL,
and to quantify I/O schedulers you need to look more complicated
statistics on latency.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: RAID 10 Benchmark with different I/O schedulers
@ 2008-05-06 20:43 Craig James <craig_james@emolecules.com>
parent: Greg Smith <gsmith@gregsmith.com>
0 siblings, 1 reply; 10+ messages in thread
From: Craig James @ 2008-05-06 20:43 UTC (permalink / raw)
To: Greg Smith <gsmith@gregsmith.com>; +Cc: pgsql-performance
Greg Smith wrote:
> On Tue, 6 May 2008, Craig James wrote:
>
>> I only did two runs of each, which took about 24 minutes. Like the
>> first round of tests, the "noise" in the measurements (about 10%)
>> exceeds the difference between scheduler-algorithm performance, except
>> that "anticipatory" seems to be measurably slower.
>
> Those are much better results. Any test that says anticipatory is
> anything other than useless for database system use with a good
> controller I presume is broken, so that's how I know you're in the right
> ballpark now but weren't before.
>
> In order to actually get some useful data out of the noise that is
> pgbench, you need a lot more measurements of longer runs. As
> perspective, the last time I did something in this area, in order to get
> enough data to get a clear picture I ran tests for 12 hours. I'm hoping
> to repeat that soon with some more common hardware that gives useful
> results I can give out.
This data is good enough for what I'm doing. There were reports from non-RAID users that the I/O scheduling could make as much as a 4x difference in performance (which makes sense for non-RAID), but these tests show me that three of the four I/O schedulers are within 10% of each other. Since this matches my intuition of how battery-backed RAID will work, I'm satisfied. If our servers get overloaded to the point where 10% matters, then I need a much more dramatic solution, like faster machines or more machines.
Craig
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: RAID 10 Benchmark with different I/O schedulers
@ 2008-05-07 07:29 Albe Laurenz *EXTERN* <laurenz.albe@wien.gv.at>
parent: Craig James <craig_james@emolecules.com>
0 siblings, 1 reply; 10+ messages in thread
From: Albe Laurenz *EXTERN* @ 2008-05-07 07:29 UTC (permalink / raw)
To: Craig James *EXTERN* <craig_james@emolecules.com>; Greg Smith <gsmith@gregsmith.com>; +Cc: pgsql-performance
Craig James wrote:
> This data is good enough for what I'm doing. There were
> reports from non-RAID users that the I/O scheduling could
> make as much as a 4x difference in performance (which makes
> sense for non-RAID), but these tests show me that three of
> the four I/O schedulers are within 10% of each other. Since
> this matches my intuition of how battery-backed RAID will
> work, I'm satisfied. If our servers get overloaded to the
> point where 10% matters, then I need a much more dramatic
> solution, like faster machines or more machines.
I should comment on this as I am the one who reported the
big performance increase with the deadline scheduler.
I was very surprised at this increase myself as I had not seen
any similar reports, so I thought I should share it for whatever
it is worth.
Our SAN *is* a RAID-5 with lots of cache, so there must be a flaw
in your intuition.
Performance measures depend a lot on your hardware and
software setup (e.g. kernel version in this case) and on the
specific load. The load we used was a real life load, collected
over seveal hours and extracted from the log files.
My opinion is that performance observations can rarely be
generalized - I am not surprised that with a different system
and a different load you observe hardly any difference between
"cfq" and "deadline".
For the record, in our test case "noop" performed practically
as good as "deadline", while the other two did way worse.
Like yourself, I have wondered why different I/O scheduling
algorithms should make so much difference.
Here is my home-spun theory of what may happen; tear it apart
and replace it with a better one at your convenience:
Our SAN probably (we're investigating) has its own brains to
optimize I/O, and I guess that any optimization that the kernel
does can only deteriorate performance because the two algorithms
might "step on each other's toes". This is backed by "noop"
performing well.
I believe that caching will not make much difference, because the
cache is way smaller than the database, and whatever is neither in
the shared buffer nor in the kernel filesystem cache is also not
likely to be in the storage system's cache. Remember that our load
was read-only.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: RAID 10 Benchmark with different I/O schedulers
@ 2008-05-07 10:46 Matthew Wakeling <matthew@flymine.org>
parent: Craig James <craig_james@emolecules.com>
1 sibling, 0 replies; 10+ messages in thread
From: Matthew Wakeling @ 2008-05-07 10:46 UTC (permalink / raw)
To: pgsql-performance
On Tue, 6 May 2008, Craig James wrote:
> I/O Sched AVG Test1 Test2
> --------- ----- ----- -----
> cfq 705 695 715
> noop 758 769 747
> deadline 741 705 775
> anticipatory 494 477 511
Interesting. That contrasts with some tests I did a while back on a
16-disc RAID-0, where noop, deadline, and anticipatory were all identical
in performance, with cfq being significantly slower. Admittedly, the disc
test was single-process, which is probably why the anticipatory behaviour
didn't kick in. You are seeing a little bit of degradation with cfq - I
guess it's worse the bigger the disc subsystem you have.
Matthew
--
Matthew: That's one of things about Cambridge - all the roads keep changing
names as you walk along them, like Hills Road in particular.
Sagar: Yes, Sidney Street is a bit like that too.
Matthew: Sidney Street *is* Hills Road.
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: RAID 10 Benchmark with different I/O schedulers
@ 2008-06-26 01:52 Craig James <craig_james@emolecules.com>
parent: Albe Laurenz *EXTERN* <laurenz.albe@wien.gv.at>
0 siblings, 0 replies; 10+ messages in thread
From: Craig James @ 2008-06-26 01:52 UTC (permalink / raw)
To: pgsql-performance
This seems like a bug to me, but it shows up as a performance problem. Since the column being queried is an integer, the second query (see below) can't possibly match, yet Postgres uses a typecast, forcing a full table scan for a value that can't possibly be in the table.
The application could intercept these bogus queries, but that requires building schema-specific and postgres-specific knowledge into the application (i.e. "What is the maximum legal integer for this column?").
Craig
explain analyze select version_id, parent_id from version where version_id = 99999;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------
Index Scan using version_pkey on version (cost=0.00..9.89 rows=1 width=8) (actual time=0.054..0.054 rows=0 loops=1)
Index Cond: (version_id = 99999)
Total runtime: 0.130 ms
(3 rows)
emol_warehouse_1=> explain analyze select version_id, parent_id from version where version_id = 999999999999999999999999999;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------
Seq Scan on version (cost=0.00..253431.77 rows=48393 width=8) (actual time=3135.530..3135.530 rows=0 loops=1)
Filter: ((version_id)::numeric = 999999999999999999999999999::numeric)
Total runtime: 3135.557 ms
(3 rows)
\d version
Table "emol_warehouse_1.version"
Column | Type | Modifiers
------------+---------+-----------
version_id | integer | not null
parent_id | integer | not null
... more columns
Indexes:
"version_pkey" PRIMARY KEY, btree (version_id)
^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2008-06-26 01:52 UTC | newest]
Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-05-05 23:33 RAID 10 Benchmark with different I/O schedulers (was: Performance increase with elevator=deadline) Craig James <craig_james@emolecules.com>
2008-05-06 03:15 ` Scott Marlowe <scott.marlowe@gmail.com>
2008-05-06 03:59 ` Greg Smith <gsmith@gregsmith.com>
2008-05-06 18:30 ` Re: RAID 10 Benchmark with different I/O schedulers Craig James <craig_james@emolecules.com>
2008-05-06 20:21 ` Re: RAID 10 Benchmark with different I/O schedulers Greg Smith <gsmith@gregsmith.com>
2008-05-06 20:43 ` Re: RAID 10 Benchmark with different I/O schedulers Craig James <craig_james@emolecules.com>
2008-05-07 07:29 ` Re: RAID 10 Benchmark with different I/O schedulers Albe Laurenz *EXTERN* <laurenz.albe@wien.gv.at>
2008-06-26 01:52 ` Re: RAID 10 Benchmark with different I/O schedulers Craig James <craig_james@emolecules.com>
2008-05-07 10:46 ` Re: RAID 10 Benchmark with different I/O schedulers Matthew Wakeling <matthew@flymine.org>
2008-05-06 12:26 ` Jeff <threshar@torgo.978.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox