From: Josh Berkus <josh@agliodbs.com>
Reply-To: josh@agliodbs.com
Organization: Aglio Database Solutions
To: "Jeffrey W. Baker" <jwbaker@acm.org>
Subject: Re: [HACKERS] A Better External Sort?
Date: Mon, 3 Oct 2005 15:03:11 -0700
User-Agent: KMail/1.8
Cc: pgsql-hackers@postgresql.org, Michael Stone <mstone+postgres@mathom.us>,
	pgsql-performance@postgresql.org
References: 
 <31833713.1128111650174.JavaMail.root@elwamui-polski.atl.sa.earthlink.net>
	<200510031416.15620.josh@agliodbs.com>
	<1128375146.29080.9.camel@toonses.gghcwest.com>
In-Reply-To: <1128375146.29080.9.camel@toonses.gghcwest.com>
MIME-Version: 1.0
Content-Type: text/plain;
  charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200510031503.12158.josh@agliodbs.com>

Jeffrey,

> I guess database reads are different, but I remain unconvinced that they
> are *fundamentally* different.  After all, a tab-delimited file (my sort
> workload) is a kind of database.

Unfortunately, they are ... because of CPU overheads.   I'm basing what's 
"reasonable" for data writes on the rates which other high-end DBs can 
make.   From that, 25mb/s or even 40mb/s for sorts should be achievable 
but doing 120mb/s would require some kind of breakthrough.

> On a single disk you wouldn't notice, but XFS scales much better when
> you throw disks at it.  I get a 50MB/sec boost from the 24th disk,
> whereas ext3 stops scaling after 16 disks.  For writes both XFS and ext3
> top out around 8 disks, but in this case XFS tops out at 500MB/sec while
> ext3 can't break 350MB/sec.

That would explain it.  I seldom get more than 6 disks (and 2 channels) to 
test with.

-- 
--Josh

Josh Berkus
Aglio Database Solutions
San Francisco