Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 27D862E4704 for ; Wed, 6 Feb 2008 12:19:01 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 36346-01-2 for ; Wed, 6 Feb 2008 12:19:00 -0400 (AST) Received: from exprod6og103.obsmtp.com (exprod6og103.obsmtp.com [64.18.1.185]) by postgresql.org (Postfix) with SMTP id 434ED2E474C for ; Wed, 6 Feb 2008 12:17:44 -0400 (AST) Received: from source ([63.240.6.3]) (using TLSv1) by exprod6ob103.postini.com ([64.18.5.12]) with SMTP; Wed, 06 Feb 2008 08:17:43 PST Received: from D01SMTP04.Mi8.com ([172.16.1.243]) by Outbound01.Mi8.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 6 Feb 2008 11:19:41 -0500 Received: from MI8NYCMAIL08.Mi8.com ([172.16.1.165]) by D01SMTP04.Mi8.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 6 Feb 2008 11:19:41 -0500 Received: from 69.181.101.67 ([69.181.101.67]) by MI8NYCMAIL08.Mi8.com ([172.16.1.190]) via Exchange Front-End Server mi8owa.mi8.com ([172.16.1.104]) with Microsoft Exchange Server HTTP-DAV ; Wed, 6 Feb 2008 16:19:40 +0000 User-Agent: Microsoft-Entourage/12.0.0.071130 Date: Wed, 06 Feb 2008 08:17:42 -0800 Subject: Re: Benchmark Data requested --- pgloader CE design ideas From: Luke Lonergan To: Greg Smith , Message-ID: Thread-Topic: [PERFORM] Benchmark Data requested --- pgloader CE design ideas Thread-Index: Acho28yc8K6oPW2VwkSFvCUwbklJ3A== In-Reply-To: Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit X-OriginalArrivalTime: 06 Feb 2008 16:19:41.0298 (UTC) FILETIME=[13B80520:01C868DC] X-TM-AS-Product-Ver: SMEX-7.0.0.1345-5.0.1023-15714.000 X-TM-AS-Result: No--18.370000-8.000000-31 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200802/74 X-Sequence-Number: 28616 Hi Greg, On 2/6/08 7:56 AM, "Greg Smith" wrote: > If I'm loading a TB file, odds are good I can split that into 4 or more > vertical pieces (say rows 1-25%, 25-50%, 50-75%, 75-100%), start 4 loaders > at once, and get way more than 1 disk worth of throughput reading. You > have to play with the exact number because if you push the split too far > you introduce seek slowdown instead of improvements, but that's the basic > design I'd like to see one day. It's not parallel loading that's useful > for the cases I'm thinking about until something like this comes around. Just load 4 relfiles. You have to be able to handle partial relfiles, which changes the storage mgmt a bit, but the benefits are easier to achieve. - Luke