MIME-Version: 1.0
In-Reply-To: 
 <CAMkU=1z92pimpoDV1oTpd4AvSHS0CrLzQDFSxw1Z5cpmmjEOkw@mail.gmail.com>
References: 
 <CADFyZw7aGoD0AaStxdyHByR5Qta=M5wx0v=iptKLhPUp+EOKvA@mail.gmail.com>
 <CAMkU=1z92pimpoDV1oTpd4AvSHS0CrLzQDFSxw1Z5cpmmjEOkw@mail.gmail.com>
From: Charles Nadeau <charles.nadeau@gmail.com>
Date: Tue, 11 Jul 2017 13:02:05 +0200
Message-ID: 
 <CADFyZw7Am=mv-6RXOr_C2oxEhPyKjnbvFgaEPq+-aKw+j9oDrw@mail.gmail.com>
Subject: Re: Very poor read performance, query independent
To: Jeff Janes <jeff.janes@gmail.com>
Cc: "pgsql-performance@postgresql.org" <pgsql-performance@postgresql.org>
Content-Type: multipart/alternative; boundary="94eb2c1252aa768cd3055408a250"
Precedence: bulk
Sender: pgsql-performance-owner@postgresql.org

--94eb2c1252aa768cd3055408a250
Content-Type: text/plain; charset="UTF-8"

Jeff,

I used fio in a quick benchmarking script inspired by
https://smcleod.net/benchmarking-io/:

#!/bin/bash
#Random throughput
echo "Random throughput"
sync
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=test --bs=4M --iodepth=256 --size=10G --readwrite=randread
--ramp_time=4
#Random IOPS
echo "Random IOPS"
sync
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=test --bs=4k --iodepth=256 --size=4G --readwrite=randread
--ramp_time=4
#Sequential throughput
echo "Sequential throughput"
sync
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=test --bs=4M --iodepth=256 --size=10G --readwrite=read
--ramp_time=4
#Sequential IOPS
echo "Sequential IOPS"
sync
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=test --bs=4k --iodepth=256 --size=4G --readwrite=read
--ramp_time=4

Performing the test you suggested, I get 128.5MB/s. Monitoring the test, I
find that the throughput is constant from start to finish and that the
iowait is also constant at 5%:

charles@hpdl380g6:~$ sudo sh -c 'time cat /mnt/data/postgresql/base/16385/*
| wc -c'
[sudo] password for charles:
1.62user 179.94system 29:50.79elapsed 10%CPU (0avgtext+0avgdata
1920maxresident)k
448026264inputs+0outputs (0major+117minor)pagefaults 0swaps
241297594904


After making the changes to HugePage suggested by Rick Otten (above), I
found slightly better results (135.7MB/s):

charles@hpdl380g6:~$ sudo sh -c 'time cat /mnt/data/postgresql/base/16385/*
| wc -c'
[sudo] password for charles:
0.86user 130.84system 28:15.78elapsed 7%CPU (0avgtext+0avgdata
1820maxresident)k
471286792inputs+0outputs (1major+118minor)pagefaults 0swaps
241297594904


Could you suggest another way to benchmark random reads?

Thanks for your help!

Charles

On Mon, Jul 10, 2017 at 9:24 PM, Jeff Janes <jeff.janes@gmail.com> wrote:

> On Mon, Jul 10, 2017 at 7:03 AM, Charles Nadeau <charles.nadeau@gmail.com>
> wrote:
>
>>
>> The problem I have is very poor read. When I benchmark my array with fio
>> I get random reads of about 200MB/s and 1100IOPS and sequential reads of
>> about 286MB/s and 21000IPS.
>>
>
>
> That doesn't seem right.  Sequential is only 43% faster?  What job file
> are giving to fio?
>
> What do you get if you do something simpler, like:
>
> time cat ~/$PGDATA/base/16402/*|wc -c
>
> replacing 16402 with whatever your biggest database is.
>
> Cheers,
>
> Jeff
>


-- 
Charles Nadeau Ph.D.
http://charlesnadeau.blogspot.com/

--94eb2c1252aa768cd3055408a250
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Jeff,</div><div><br></div><div>I used fio in a quick =
benchmarking script inspired by <a href=3D"https://smcleod.net/benchmarking=
-io/">https://smcleod.net/benchmarking-io/</a>:</div><blockquote style=3D"m=
argin:0 0 0 40px;border:none;padding:0px"><div>#!/bin/bash</div><div>#Rando=
m throughput</div><div>echo &quot;Random throughput&quot;</div><div>sync</d=
iv><div>fio --randrepeat=3D1 --ioengine=3Dlibaio --direct=3D1 --gtod_reduce=
=3D1 --name=3Dtest --filename=3Dtest --bs=3D4M --iodepth=3D256 --size=3D10G=
 --readwrite=3Drandread --ramp_time=3D4</div><div>#Random IOPS</div><div>ec=
ho &quot;Random IOPS&quot;</div><div>sync</div><div>fio --randrepeat=3D1 --=
ioengine=3Dlibaio --direct=3D1 --gtod_reduce=3D1 --name=3Dtest --filename=
=3Dtest --bs=3D4k --iodepth=3D256 --size=3D4G --readwrite=3Drandread --ramp=
_time=3D4</div><div>#Sequential throughput</div><div>echo &quot;Sequential =
throughput&quot;</div><div>sync</div><div>fio --randrepeat=3D1 --ioengine=
=3Dlibaio --direct=3D1 --gtod_reduce=3D1 --name=3Dtest --filename=3Dtest --=
bs=3D4M --iodepth=3D256 --size=3D10G --readwrite=3Dread --ramp_time=3D4</di=
v><div>#Sequential IOPS</div><div>echo &quot;Sequential IOPS&quot;</div><di=
v>sync</div><div>fio --randrepeat=3D1 --ioengine=3Dlibaio --direct=3D1 --gt=
od_reduce=3D1 --name=3Dtest --filename=3Dtest --bs=3D4k --iodepth=3D256 --s=
ize=3D4G --readwrite=3Dread --ramp_time=3D4</div><div><br></div></blockquot=
e><div>Performing the test you suggested, I get 128.5MB/s. Monitoring the t=
est, I find that the throughput is constant from start to finish and that t=
he iowait is also constant at 5%:</div><blockquote style=3D"margin:0 0 0 40=
px;border:none;padding:0px"><div>charles@hpdl380g6:~$ sudo sh -c &#39;time =
cat /mnt/data/postgresql/base/16385/* | wc -c&#39;</div><div>[sudo] passwor=
d for charles:=C2=A0</div><div>1.62user 179.94system 29:50.79elapsed 10%CPU=
 (0avgtext+0avgdata 1920maxresident)k</div><div>448026264inputs+0outputs (0=
major+117minor)pagefaults 0swaps</div><div>241297594904</div></blockquote><=
div><br></div><div>After making the changes to HugePage suggested by Rick O=
tten (above), I found slightly better results (135.7MB/s):</div><blockquote=
 style=3D"margin:0 0 0 40px;border:none;padding:0px"><div>charles@hpdl380g6=
:~$ sudo sh -c &#39;time cat /mnt/data/postgresql/base/16385/* | wc -c&#39;=
</div><div>[sudo] password for charles:=C2=A0</div><div>0.86user 130.84syst=
em 28:15.78elapsed 7%CPU (0avgtext+0avgdata 1820maxresident)k</div><div>471=
286792inputs+0outputs (1major+118minor)pagefaults 0swaps</div><div>24129759=
4904</div></blockquote><div><br></div><div>Could you suggest another way to=
 benchmark random reads?</div><div><br></div><div>Thanks for your help!</di=
v><div><br></div><div>Charles</div></div><div class=3D"gmail_extra"><br><di=
v class=3D"gmail_quote">On Mon, Jul 10, 2017 at 9:24 PM, Jeff Janes <span d=
ir=3D"ltr">&lt;<a href=3D"mailto:jeff.janes@gmail.com" target=3D"_blank">je=
ff.janes@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quot=
e" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">=
<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><spa=
n class=3D"">On Mon, Jul 10, 2017 at 7:03 AM, Charles Nadeau <span dir=3D"l=
tr">&lt;<a href=3D"mailto:charles.nadeau@gmail.com" target=3D"_blank">charl=
es.nadeau@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quo=
te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204=
);padding-left:1ex"><div dir=3D"ltr">
=09
=09
=09


<p style=3D"margin-bottom:0in;line-height:100%"><br></p>
<p style=3D"margin-bottom:0in;line-height:100%">The problem I have
is very poor read. When I benchmark my array with fio I get random
reads of about 200MB/s and 1100IOPS and sequential reads of about
286MB/s and 21000IPS. </p></div></blockquote><div><br></div><div><br></div>=
</span><div>That doesn&#39;t seem right.=C2=A0 Sequential is only 43% faste=
r?=C2=A0 What job file are giving to fio?</div><div><br></div><div>What do =
you get if you do something simpler, like:=C2=A0</div><div><br></div><div>t=
ime cat ~/$PGDATA/base/16402/*|wc -c<br></div><div><br></div><div>replacing=
 16402 with whatever your biggest database is.</div><div><br></div><div>Che=
ers,</div><div><br></div><div>Jeff</div></div></div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div class=
=3D"gmail_signature" data-smartmail=3D"gmail_signature">Charles Nadeau Ph.D=
.<br><a href=3D"http://charlesnadeau.blogspot.com/" target=3D"_blank">http:=
//charlesnadeau.blogspot.com/</a></div>
</div>

--94eb2c1252aa768cd3055408a250--