To: pgsql-performance@postgresql.org
From: Ivan Voras <ivoras@freebsd.org>
Subject: Re: How to achieve sustained disk performance of 1.25 GB write for
	5 mins
Date: Thu, 18 Nov 2010 00:27:35 +0100
Lines: 32
Message-ID: <ic1ode$mt8$1@dough.gmane.org>
References: <ic0onf$3fi$1@news.hub.org>
	<AANLkTiky7UGdCbpdrNMAiwo69WFgv0sEah=_ov0nCN0J@mail.gmail.com>
	<35FABCF85D99464FB00BC5123DC2A70A0B7EFC38@s228130hz1ew09.apptix-01.savvis.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US;
	rv:1.9.2.12) Gecko/20101102 Thunderbird/3.1.6
In-Reply-To: 
 <35FABCF85D99464FB00BC5123DC2A70A0B7EFC38@s228130hz1ew09.apptix-01.savvis.net>

On 11/17/10 22:11, Eric Comeau wrote:

>> *) what kind of data do you expect to be writing out at this speed?
>>
> Large Video files ... our s/w is used to displace FTP.
>>
>> *) how many transactions per second will you expect to have?
>>
> Ideally 1 large file, but it may have to be multiple. We find that if we
> send multiple files it just causes the disk to thrash more so we get
> better throughput by sending one large file.
 >
>> *) what is the architecture of the client? how many connections will
>> be open to postgres writing?
>>
> Our s/w can do multiple streams, but I believe we get better performance
> with 1 stream handling one large file, you could have 4 streams with 4
> files in flight, but the disk thrashes more... postgres is not be
> writing the file data, our agent reports back to postgres stats on the
> transfer rate being achieved ... postgres transactions is not the issue.
> The client and server are written in C and use UDP (with our own error
> correction) to achieve high network throughput as opposed to TCP.

I hope you know what you are doing, there is a large list of tricks used 
by modern high performance FTP and web servers to get maximum 
performance from hardware and the operating system while minimizing CPU 
usage - and most of them don't work with UDP.

Before you test with real hardware, try simply sending dummy data or 
/dev/null data (i.e. not from disks, not from file systems) and see how 
it goes.