Date: Wed, 21 Mar 2007 10:01:15 -0700
To: pgsql-sql@postgresql.org
From: Steve Midgley <public@misuse.org>
Subject: Re: Issue with copying data from a text file.
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Message-Id: <20070321170127.D94E39FBD45@postgresql.org>

Hi,

I think I had the exact same problem as you do a while back and I 
solved it by removing the header row and the "CSV HEADER" clause of the 
statement. For the large files I had, it was easier (for me) to remove 
the header row than it was to escape out all the quotes (or regen the 
file):

COPY deal_lines_temp_load FROM
'c:/temp/autodrs_deal_lines.txt'
WITH DELIMITER AS '^';

I think the parser doesn't look for nor generate quoted rows except 
when CSV is specified.. It would be nice if there was a way to specify 
a "HEADER" row without invoking CSV parsing rules (friendly hint to 
core devs!) :)

Let us all know if that works!

Steve

At 03:14 AM 3/20/2007, you wrote:
>Date: Tue, 20 Mar 2007 11:25:38 +0900
>From: Paul Lambert <paul.lambert@autoledgers.com.au>
>To: pgsql-sql@postgresql.org
>Subject: Issue with copying data from a text file.
>Message-ID: <45FF4622.2010404@autoledgers.com.au>
>
>I have a procedure in place that copies data from a caret delimited 
>text
>file into a table storing some information.
>
>One of the fields in the table contains an item description which may
>contain item dimensions such as - 17" alloy wheels
>
>The problem I am getting when I do my load is I believe due to the
>presence of the double quotation marks giving the copy the impression
>that it is to include the information following as a single text 
>string
>until it gets to the next set of double quotes. As a result, I get the 
>
>following:
>
>AutoDRS=#       COPY deal_lines_temp_load FROM
>'c:/temp/autodrs_deal_lines.txt'
>WITH DELIMITER AS '^' CSV HEADER;
>ERROR:  value too long for type character varying(30)
>CONTEXT:  COPY deal_lines_temp_load, line 87, column order_desc: "17 5 
>
>spoke alloy wheels.^1291.18^117.38^983.69^1291.18^^C^^
>
>The column as you can see is defined as a 30 character field, the load 
>
>contains in this column ^17" 5 spoke alloy wheels.^
>
>I note an option in the COPY command to specify the quote character,
>defaulting to double quote. The problem being a single quote will also 
>
>be used in the data, as will other characters. Is there any way to get 
>a
>copy to have no quote character? I.e. read the file and put whatever 
>is
>between the caret characters straight into the appropriate field 
>exactly
>as is.
>
>TIA,
>Paul.
>
>--
>Paul Lambert
>Database Administrator
>AutoLedgers