X-Original-To: pgsql-sql-postgresql.org@localhost.postgresql.org Received: from localhost (av.hub.org [200.46.204.144]) by postgresql.org (Postfix) with ESMTP id AA8FE9DCA3E for ; Wed, 4 Jan 2006 05:24:33 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 70067-07 for ; Wed, 4 Jan 2006 05:24:31 -0400 (AST) X-Greylist: from auto-whitelisted by SQLgrey- Received: from mail.metronet.co.uk (mail.metronet.co.uk [213.162.97.75]) by postgresql.org (Postfix) with ESMTP id 1A50C9DC9BC for ; Wed, 4 Jan 2006 05:24:27 -0400 (AST) Received: from mainbox.archonet.com (84-51-143-99.archon037.adsl.metronet.co.uk [84.51.143.99]) by smtp.metronet.co.uk (MetroNet Mail) with ESMTP id 88BA4408A45; Wed, 4 Jan 2006 09:21:18 +0000 (GMT) Received: from [192.168.1.17] (client17.office.archonet.com [192.168.1.17]) by mainbox.archonet.com (Postfix) with ESMTP id D0AFF15EE9; Wed, 4 Jan 2006 09:21:22 +0000 (GMT) Message-ID: <43BB9392.5080805@archonet.com> Date: Wed, 04 Jan 2006 09:21:22 +0000 From: Richard Huxton User-Agent: Thunderbird 1.5 (X11/20051025) MIME-Version: 1.0 To: frank church Cc: pgsql-sql@postgresql.org Subject: Re: Loading lots of data in a SQL command References: <1136338331.43bb259b9334f@webmail.adontendev.net> In-Reply-To: <1136338331.43bb259b9334f@webmail.adontendev.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, score=0.04 required=5 tests=[AWL=0.040] X-Spam-Score: 0.04 X-Spam-Level: X-Archive-Number: 200601/13 X-Sequence-Number: 23802 frank church wrote: > I am load lots of data via SQL into a database and wrapping it into transactions > speeds it up. > > However this fails a number of times. The queries results are logged so it is > easy for me to find problem records. > > However a single failure causes the whole transaction to fail. > > Is there a setting or feature that allows which allows the same performance as > transactions, without causing the whole process to fail, like a delayed updates > or write mechanism of some sort. Not as it stands. I tend to use a small perl wrapper myself that loads in batches of e.g. 10000 rows and if there is an error deal with it separately. I seem to recall it being discussed as a built-in feature recently though, so there might be someone working on it for a future version. > It is something I would like to set in that particular data looad. You might find the "pgloader" project meets your needs exactly: http://pgfoundry.org/projects/pgloader/ -- Richard Huxton Archonet Ltd