Message-ID: <4F2649BD.3000009@gmail.com>
Date: Mon, 30 Jan 2012 08:41:49 +0100
From: Ron Arts <ron.arts@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
	rv:9.0) Gecko/20111222 Thunderbird/9.0.1
MIME-Version: 1.0
To: Jose Ildefonso Camargo Tolosa <ildefonso.camargo@gmail.com>
CC: pgsql-performance@postgresql.org
Subject: Re: Having I/O problems in simple virtualized environment
References: <4F25CCD4.8080009@gmail.com>
	<CAETJ_S-VOr=2ReeVx=eC0HxLh7xCY5dxkN8Xg6+EYXHP=jJH-w@mail.gmail.com>
In-Reply-To: 
 <CAETJ_S-VOr=2ReeVx=eC0HxLh7xCY5dxkN8Xg6+EYXHP=jJH-w@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Op 30-01-12 02:52, Jose Ildefonso Camargo Tolosa schreef:
> On Sun, Jan 29, 2012 at 6:18 PM, Ron Arts <ron.arts@gmail.com> wrote:
>> Hi list,
>>
>> I am running PostgreSQL 8.1 (CentOS 5.7) on a VM on a single XCP (Xenserver) host.
>> This is a HP server with 8GB, Dual Quad Core, and 2 SATA in RAID-1.
>>
>> The problem is: it's running very slow compared to running it on bare metal, and
>> the VM is starving for I/O bandwidht, so other processes (slow to a crawl.
>> This does not happen on bare metal.
>>
>> I had to replace the server with a bare-metal one, I could not troubleshoot in production.
>> Also it was hard to emulte the workload for that VM in a test environment, so I
>> concentrated on PostgreSQLand why it apparently generated so much I/O.
>>
>> Before I start I should confess having only spotty experience with Xen and PostgreSQL
>> performance testing.
>>
>> I setup a test Xen server created a CentOS5.7 VM with out-of-the-box PostgreSQL and ran:
>> pgbench -i  pgbench ; time pgbench -t 100000 pgbench
>> This ran for 3:28. Then I replaced the SATA HD with an SSD disk, and reran the test.
>> It ran for 2:46. This seemed strange as I expected the run to finish much faster.
>>
>> I reran the first test on the SATA, and looked at CPU and I/O use. The CPU was not used
>> too much in both the VM (30%) and in dom0 (10%). The I/O use was not much as well,
>> around 8MB/sec in the VM. (Couldn't use iotop in dom0, because of missing kernel support
>> in XCP 1.1).
>>
>> It reran the second test on SSD, and experienced almost the same CPU, and I/O load.
>>
>> (I now probably need to run the same test on bare metal, but didn't get to that yet,
>> all this already ruined my weekend.)
>>
>> Now I came this far, can anybody give me some pointers? Why doesn't pgbench saturate
>> either the CPU or the I/O? Why does using SSD only change the performance this much?
> 
> Ok, one point: Which IO scheduler are you using?  (on dom0 and on the VM).

Ok, first dom0:

For the SSD (hda):
# cat /sys/block/sda/queue/scheduler
[noop] anticipatory deadline cfq

For the SATA:
# cat /sys/block/sdb/queue/scheduler
noop anticipatory deadline [cfq]

Then in the VM:

# cat /sys/block/xvda/queue/scheduler
[noop] anticipatory deadline cfq

Ron