X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 49EAC52863 for ; Thu, 23 Jun 2005 18:48:37 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 55708-10 for ; Thu, 23 Jun 2005 21:48:29 +0000 (GMT) Received: from area51.givex.com (tor149-99-39-156.dedicated.sprintdsl.ca [149.99.39.156]) by svr1.postgresql.org (Postfix) with ESMTP id C42615295E for ; Thu, 23 Jun 2005 18:48:27 -0300 (ADT) Received: by area51.givex.com (Postfix, from userid 66) id E105119115; Thu, 23 Jun 2005 17:48:24 -0400 (EDT) Received: from vlad.givex.com (vlad.givex.com [192.168.7.51]) by area51.givex.com (Postfix) with ESMTP id 03BF01910D; Thu, 23 Jun 2005 17:48:23 -0400 (EDT) From: Vladimir Yevdokimov Organization: Givex Canada Corp. To: David.Bear@asu.edu Subject: Re: psql copy errors Date: Thu, 23 Jun 2005 17:48:19 -0400 User-Agent: KMail/1.6.2 Cc: pgsql-admin@postgresql.org References: <20050623192744.GC398@asu.edu> In-Reply-To: <20050623192744.GC398@asu.edu> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200506231748.19051.vladimir@givex.com> X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.05 tagged_above=0 required=5 tests=FORGED_RCVD_HELO X-Spam-Level: X-Archive-Number: 200506/180 X-Sequence-Number: 17935 On June 23, 2005 03:27 pm, David Bear wrote: > I'm finding the \copy is very brittle. It seems to stop for everyone > little reason. Is there a way to tell it to be more forgiving -- for > example, to ignore extra data fields that might exists on a line? > > Or, to have it just skip that offending record but continue on to the > next. > > I've got a tab delimited file, but if \copy sees any extra tabs in the > file it just stops at that record. I want to be able to control what > pg does when it hits an exception. > > I'm curious what others do for bulk data migration. Since copy seems > so brittle, there must be a better way... > You may use '-d' option of pg_dump in which case it dumps data into INSERT statements. In this case when you load the damped data it will process tabs properly and will fail any invalid records but finish the process itself. If you redirect output into a separate file you can analyze later how many records failed. May be that's what you need in your case. The only problem with this method I know is that it takes longer to load the data as it does full validation for each record. -- Vladimir Yevdokimov