Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1av3c7-0003wJ-KM for pgsql-performance@arkaria.postgresql.org; Tue, 26 Apr 2016 14:05:19 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1av3c7-0006lZ-79 for pgsql-performance@arkaria.postgresql.org; Tue, 26 Apr 2016 14:05:19 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1av3aQ-0004gU-3I for pgsql-performance@postgresql.org; Tue, 26 Apr 2016 14:03:34 +0000 Received: from mail-lf0-x22e.google.com ([2a00:1450:4010:c07::22e]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1av3aM-0005RQ-Px for pgsql-performance@postgresql.org; Tue, 26 Apr 2016 14:03:32 +0000 Received: by mail-lf0-x22e.google.com with SMTP id y84so17184730lfc.0 for ; Tue, 26 Apr 2016 07:03:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leboutique.com; s=google; h=mime-version:from:date:message-id:subject:to; bh=Tzo2YQAfnYYck6PvCotlqv4dLIydbtGfrl1pEdyh9Go=; b=MePqZl94+Jzvh1mDoSc5PXwYDOqNSHVDs2tPV9rmStGyexDh949cew3GiIvcWNlbQx biJuamF2p1QRtpfLnkztFenu0QOW2Hhcb6hTAK1ZKCCpuVPGsv/olMGk9yg7ePcPDqrR +R05SmaDtQA6R17Yn99fFr6XbxmVYbZB0OBhA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Tzo2YQAfnYYck6PvCotlqv4dLIydbtGfrl1pEdyh9Go=; b=d3F/XRytwx6MOdzIGZYR97Kf+nFKTdFo15f745BXcC00lYfdmXa1DhiT/1/juvik5o DlhDdW8KkdeJSqYP6iI69Lr/fO1fpO/5FeZkbkYhe7ccJDGossqJoNfCfGYEplXwAIx0 ZZCzG9y4XLKLQgrZPGsXE8Hv/vWBUa9NdT7yqfH2nX7LMXUB5nKRWTHMwfnxkv/4PooV /Pj/5NmWt92/vof7YbsqW0Xy/Rn03cr2L/KKCraVQ1ThpVw7/uS9swNXd3ggSZD8Lzh4 7Q3GAlXEeVqYIaILtXX1qR2k9phtCD9p4iS90oUZFitVYvoSn0BUId3mAQ4FpuqnB9aT bJgg== X-Gm-Message-State: AOPr4FXhv3/SwVh8KHhHSqi/wUxnEL3YjUyboEClNu024YHBvMFZujqt9KCEONv2GyRO5Hdi5KgiHFU87/lV611q X-Received: by 10.25.15.36 with SMTP id e36mr1415841lfi.148.1461679408393; Tue, 26 Apr 2016 07:03:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.17.79 with HTTP; Tue, 26 Apr 2016 07:03:08 -0700 (PDT) From: Artem Tomyuk Date: Tue, 26 Apr 2016 17:03:08 +0300 Message-ID: Subject: Poor disk (virtio) Performance Inside KVM virt-machine vs host machine To: pgsql-performance@postgresql.org Content-Type: multipart/alternative; boundary=001a113fb8f6e9ca82053163c149 X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org --001a113fb8f6e9ca82053163c149 Content-Type: text/plain; charset=UTF-8 Hi All. I've noticed that there is a huge (more than ~3x slower) performance difference between KVM guest and host machine. Host machine: dell r720xd RAID10 with 12 SAS 15 k drives and RAID0 with 2*128 GB INTEL SSD drives in Dell CacheCade mode. *On the KVM guest:* /usr/pgsql-9.4/bin/pg_test_fsync -f test.sync 5 seconds per test O_DIRECT supported on this platform for open_datasync and open_sync. Compare file sync methods using one 8kB write: (in wal_sync_method preference order, except fdatasync is Linux's default) open_datasync 5190.279 ops/sec 193 usecs/op fdatasync 4022.553 ops/sec 249 usecs/op fsync 3069.069 ops/sec 326 usecs/op fsync_writethrough n/a open_sync 4892.348 ops/sec 204 usecs/op Compare file sync methods using two 8kB writes: (in wal_sync_method preference order, except fdatasync is Linux's default) open_datasync 2406.577 ops/sec 416 usecs/op fdatasync 4309.413 ops/sec 232 usecs/op fsync 3518.844 ops/sec 284 usecs/op fsync_writethrough n/a open_sync 1159.604 ops/sec 862 usecs/op Compare open_sync with different write sizes: (This is designed to compare the cost of writing 16kB in different write open_sync sizes.) 1 * 16kB open_sync write 3700.689 ops/sec 270 usecs/op 2 * 8kB open_sync writes 2581.405 ops/sec 387 usecs/op 4 * 4kB open_sync writes 1318.871 ops/sec 758 usecs/op 8 * 2kB open_sync writes 698.640 ops/sec 1431 usecs/op 16 * 1kB open_sync writes 262.506 ops/sec 3809 usecs/op Test if fsync on non-write file descriptor is honored: (If the times are similar, fsync() can sync data written on a different descriptor.) write, fsync, close 3071.141 ops/sec 326 usecs/op write, close, fsync 3303.946 ops/sec 303 usecs/op Non-Sync'ed 8kB writes: write 251321.188 ops/sec 4 usecs/op *On the host machine:* /usr/pgsql-9.4/bin/pg_test_fsync -f test.sync 5 seconds per test O_DIRECT supported on this platform for open_datasync and open_sync. Compare file sync methods using one 8kB write: (in wal_sync_method preference order, except fdatasync is Linux's default) open_datasync 11364.136 ops/sec 88 usecs/op fdatasync 12352.160 ops/sec 81 usecs/op fsync 9833.745 ops/sec 102 usecs/op fsync_writethrough n/a open_sync 14938.531 ops/sec 67 usecs/op Compare file sync methods using two 8kB writes: (in wal_sync_method preference order, except fdatasync is Linux's default) open_datasync 7703.471 ops/sec 130 usecs/op fdatasync 11494.492 ops/sec 87 usecs/op fsync 9029.837 ops/sec 111 usecs/op fsync_writethrough n/a open_sync 6504.138 ops/sec 154 usecs/op Compare open_sync with different write sizes: (This is designed to compare the cost of writing 16kB in different write open_sync sizes.) 1 * 16kB open_sync write 14113.912 ops/sec 71 usecs/op 2 * 8kB open_sync writes 7843.234 ops/sec 127 usecs/op 4 * 4kB open_sync writes 3995.702 ops/sec 250 usecs/op 8 * 2kB open_sync writes 1788.979 ops/sec 559 usecs/op 16 * 1kB open_sync writes 937.177 ops/sec 1067 usecs/op Test if fsync on non-write file descriptor is honored: (If the times are similar, fsync() can sync data written on a different descriptor.) write, fsync, close 10144.280 ops/sec 99 usecs/op write, close, fsync 8378.558 ops/sec 119 usecs/op Non-Sync'ed 8kB writes: write 159176.122 ops/sec 6 usecs/op The file system "inside" and "outside" the same - ext4 on LVM. Disk scheduler "inside" and "outside" set to "noop". Fstab options same to, setted to rw,noatime,nodiratime,barrier=0. OS on host and guest the same CentOS release 6.5 (Final). Guest volume options: Disk bus: Virtio Cache mode: none IO mode: native Any ideas? --001a113fb8f6e9ca82053163c149 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi All.=C2=A0

I've noticed that the= re is a huge (more than ~3x slower) performance difference between KVM gues= t and host machine.=C2=A0
Host machine:=C2=A0
dell r720= xd=C2=A0
RAID10 with 12 SAS 15 k drives and RAID0 =C2=A0with 2*12= 8 GB INTEL SSD drives in Dell CacheCade mode.

O= n the KVM guest:

=C2=A0/usr/pgsql-9.4/bin/pg_test_fsync -f te= st.sync

5 seconds per test

O_DIRECT supported on this platform for open= _datasync and open_sync.


Compare file sync methods using one 8kB writ= e:

(in wal_sync_method preference order, except= fdatasync

is Linux's default)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 open_datasync=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 5= 190.279 ops/sec =C2=A0 =C2=A0 193 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 fdatasync=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 4022.553 ops/sec =C2=A0 =C2=A0 249 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 fsync=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 3069.069 ops/sec =C2=A0 =C2=A0 326 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 fsync_writethrou= gh=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 n/a

=C2=A0 =C2=A0 =C2=A0 =C2=A0 open_sync=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 4892.348 ops/sec =C2=A0 =C2=A0 204 usecs/op


Compare file sync methods using two 8kB writ= es:

(in wal_sync_method preference order, except= fdatasync

is Linux's default)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 open_datasync=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 2= 406.577 ops/sec =C2=A0 =C2=A0 416 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 fdatasync=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 4309.413 ops/sec =C2=A0 =C2=A0 232 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 fsync=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 3518.844 ops/sec =C2=A0 =C2=A0 284 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 fsync_writethrou= gh=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 n/a

=C2=A0 =C2=A0 =C2=A0 =C2=A0 open_sync=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 1159.604 ops/sec =C2=A0 =C2=A0 862 usecs/op


Compare open_sync with different write sizes= :

(This is designed to compare the cost of wri= ting 16kB

in different write open_sync sizes.)<= /p>

=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 1 * 16kB o= pen_sync write=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3700.689 ops/sec =C2=A0 = =C2=A0 270 usecs/op

=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 2 *=C2=A0 = 8kB open_sync writes =C2=A0 =C2=A0 =C2=A0 =C2=A0 2581.405 ops/sec =C2=A0 = =C2=A0 387 usecs/op

=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 4 *=C2=A0 = 4kB open_sync writes =C2=A0 =C2=A0 =C2=A0 =C2=A0 1318.871 ops/sec =C2=A0 = =C2=A0 758 usecs/op

=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 8 *=C2=A0 = 2kB open_sync writes=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 698.640 ops/sec=C2= =A0 =C2=A0 1431 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 16 *=C2=A0 1kB o= pen_sync writes=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 262.506 ops/sec=C2=A0 =C2= =A0 3809 usecs/op


Test if fsync on non-write file descriptor i= s honored:

(If the times are similar, fsync() can sync = data written

on a different descriptor.)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 write, fsync, cl= ose=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3071.141 ops/sec= =C2=A0 =C2=A0 326 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 write, close, fs= ync=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 3303.946 ops/sec= =C2=A0 =C2=A0 303 usecs/op


Non-Sync'ed 8kB writes:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 write=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 251321.188 ops/sec =C2=A0 =C2=A0 =C2=A0 4 usecs/op

=


On the host machine:

/usr= /pgsql-9.4/bin/pg_test_fsync -f test.sync

5 seconds per test

O_DIREC= T supported on this platform for open_datasync and open_sync.


C= ompare file sync methods using one 8kB write:

(in wal_sync_method preference order, except fdatasync

is Linux's default)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 open_datasync =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 11364.136 ops/s= ec=C2=A0 =C2=A0 =C2=A0 88 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 fdatasync =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 12352.160 ops/sec=C2=A0 = =C2=A0 =C2=A0 81 usecs/op

=C2=A0 = =C2=A0 =C2=A0 =C2=A0 fsync=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 9833.745 ops/sec = =C2=A0 =C2=A0 102 usecs/op

=C2=A0 = =C2=A0 =C2=A0 =C2=A0 fsync_writethrough=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 n/a

=C2=A0 =C2=A0 =C2=A0 =C2=A0 open_sy= nc =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 14938.531 ops/sec=C2=A0 =C2=A0 =C2=A0 67 usecs/op

<= p class=3D"">

Compare file sync methods using two 8kB writes:

(in wal_sync_method preference order, except fdatasync

is Linux's default)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 open_datasync=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 7703.= 471 ops/sec =C2=A0 =C2=A0 130 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 fdatasync =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 11494.492 ops/sec=C2= =A0 =C2=A0 =C2=A0 87 usecs/op

=C2= =A0 =C2=A0 =C2=A0 =C2=A0 fsync=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 9029.837 ops/se= c =C2=A0 =C2=A0 111 usecs/op

=C2= =A0 =C2=A0 =C2=A0 =C2=A0 fsync_writethrough=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 n= /a

=C2=A0 =C2=A0 =C2=A0 =C2=A0 ope= n_sync=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 6504.138 ops/sec =C2=A0 =C2=A0 154 usecs/op


Compare open_sync with different write sizes:

(This is designed to compare the cost of writing 16kB=

in different write open_sync sizes.)

=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 1 *= 16kB open_sync write =C2=A0 =C2=A0 =C2=A0 =C2=A0 14113.912 ops/sec=C2=A0 = =C2=A0 =C2=A0 71 usecs/op

=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 2 *=C2=A0 8kB open_sync writes =C2=A0 =C2=A0 = =C2=A0 =C2=A0 7843.234 ops/sec =C2=A0 =C2=A0 127 usecs/op

=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 4 *=C2=A0 4kB ope= n_sync writes =C2=A0 =C2=A0 =C2=A0 =C2=A0 3995.702 ops/sec =C2=A0 =C2=A0 25= 0 usecs/op

=C2=A0=C2=A0 =C2=A0 =C2= =A0 =C2=A0 8 *=C2=A0 2kB open_sync writes =C2=A0 =C2=A0 =C2=A0 =C2=A0 1788.= 979 ops/sec =C2=A0 =C2=A0 559 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 16 *=C2=A0 1kB open_sync writes=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 937.177 ops/sec=C2=A0 =C2=A0 1067 usecs/op<= /p>


Test if fsync on non-write file descriptor is honored:

(If the times are similar, fsync() can sync data= written

on a different descriptor= .)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 wri= te, fsync, close =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 10144.280= ops/sec=C2=A0 =C2=A0 =C2=A0 99 usecs/op

=C2=A0 =C2=A0 =C2=A0 =C2=A0 write, close, fsync=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 8378.558 ops/sec =C2=A0 =C2=A0 119 usecs= /op


<= span class=3D"">Non-Sync'ed 8kB writes:

=C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0write= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 159176.122 ops/sec =C2=A0 =C2=A0 =C2=A0 6 usecs/op=


The file system "inside" and "outside" the= same - ext4 on LVM. Disk scheduler =C2=A0"inside" and &qu= ot;outside" set to "noop". Fstab options same to, setted to = rw,noatime,nodiratime,barrier=3D0. OS on host and guest the same CentOS rel= ease 6.5 (Final).

Guest volume options:

D= isk bus: Virtio

Cache mode: none

IO mode:= native=C2=A0


Any ideas?=C2=A0


=




=C2=A0



--001a113fb8f6e9ca82053163c149--