pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedIssue with copying data from a text file.
9+ messages / 5 participants
[nested] [flat]
* Issue with copying data from a text file.
@ 2007-03-20 02:25 Paul Lambert <paul.lambert@autoledgers.com.au>
0 siblings, 2 replies; 9+ messages in thread
From: Paul Lambert @ 2007-03-20 02:25 UTC (permalink / raw)
To: pgsql-sql
I have a procedure in place that copies data from a caret delimited text
file into a table storing some information.
One of the fields in the table contains an item description which may
contain item dimensions such as - 17" alloy wheels
The problem I am getting when I do my load is I believe due to the
presence of the double quotation marks giving the copy the impression
that it is to include the information following as a single text string
until it gets to the next set of double quotes. As a result, I get the
following:
AutoDRS=# COPY deal_lines_temp_load FROM
'c:/temp/autodrs_deal_lines.txt'
WITH DELIMITER AS '^' CSV HEADER;
ERROR: value too long for type character varying(30)
CONTEXT: COPY deal_lines_temp_load, line 87, column order_desc: "17 5
spoke alloy wheels.^1291.18^117.38^983.69^1291.18^^C^^
The column as you can see is defined as a 30 character field, the load
contains in this column ^17" 5 spoke alloy wheels.^
I note an option in the COPY command to specify the quote character,
defaulting to double quote. The problem being a single quote will also
be used in the data, as will other characters. Is there any way to get a
copy to have no quote character? I.e. read the file and put whatever is
between the caret characters straight into the appropriate field exactly
as is.
TIA,
Paul.
--
Paul Lambert
Database Administrator
AutoLedgers
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Issue with copying data from a text file.
@ 2007-03-20 02:37 Phillip Smith <phillip.smith@weatherbeeta.com.au>
parent: Paul Lambert <paul.lambert@autoledgers.com.au>
1 sibling, 1 reply; 9+ messages in thread
From: Phillip Smith @ 2007-03-20 02:37 UTC (permalink / raw)
To: Paul Lambert <paul.lambert@autoledgers.com.au>; +Cc: pgsql-sql
Can you not export the source file with escape characters? ie,
^17\" Alloy Wheels^
~p
On Tue, 2007-03-20 at 11:25 +0900, Paul Lambert wrote:
> I have a procedure in place that copies data from a caret delimited text
> file into a table storing some information.
>
> One of the fields in the table contains an item description which may
> contain item dimensions such as - 17" alloy wheels
>
> The problem I am getting when I do my load is I believe due to the
> presence of the double quotation marks giving the copy the impression
> that it is to include the information following as a single text string
> until it gets to the next set of double quotes. As a result, I get the
> following:
>
> AutoDRS=# COPY deal_lines_temp_load FROM
> 'c:/temp/autodrs_deal_lines.txt'
> WITH DELIMITER AS '^' CSV HEADER;
> ERROR: value too long for type character varying(30)
> CONTEXT: COPY deal_lines_temp_load, line 87, column order_desc: "17 5
> spoke alloy wheels.^1291.18^117.38^983.69^1291.18^^C^^
>
> The column as you can see is defined as a 30 character field, the load
> contains in this column ^17" 5 spoke alloy wheels.^
>
> I note an option in the COPY command to specify the quote character,
> defaulting to double quote. The problem being a single quote will also
> be used in the data, as will other characters. Is there any way to get a
> copy to have no quote character? I.e. read the file and put whatever is
> between the caret characters straight into the appropriate field exactly
> as is.
>
> TIA,
> Paul.
>
Phillip Smith
IT Coordinator
Weatherbeeta P/L
8 Moncrief Rd
Nunawading, Vic, 3131
AUSTRALIA
P. +613 9845 0600
F. +613 9845 0655
E. phillip.smith@weatherbeeta.com.au
Attachments:
smime.p7s (application/x-pkcs7-signature, 2.1K) download
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Issue with copying data from a text file.
@ 2007-03-20 02:44 Paul Lambert <paul.lambert@autoledgers.com.au>
parent: Phillip Smith <phillip.smith@weatherbeeta.com.au>
0 siblings, 1 reply; 9+ messages in thread
From: Paul Lambert @ 2007-03-20 02:44 UTC (permalink / raw)
To: Phillip Smith <phillip.smith@weatherbeeta.com.au>; pgsql-sql
Phillip Smith wrote:
> Can you not export the source file with escape characters? ie,
>
> ^17\" Alloy Wheels^
>
> ~p
>
The source file comes from extracts on our main application which sits
inside an in-house pretending-to-be-a-dbms file system. The content of
these extracts would be difficult to change - the extract program would
need to parse the data looking for quotes and preceed them with the
necessary escape character.
Not being a proper database dump it's not a simple matter of flicking a
switch to get it to include the escape character. The way the extracts
are written would require a few dozen lines of code to each extract, and
theres about 40ish extracts.
Plus I don't maintain that side of our code, and those that do can be a
bit lazy and I'd likely be waiting months to get it done - if they even
decide to do it.
--
Paul Lambert
Database Administrator
AutoLedgers
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Issue with copying data from a text file.
@ 2007-03-20 02:53 Andrej Ricnik-Bay <andrej.groups@gmail.com>
parent: Paul Lambert <paul.lambert@autoledgers.com.au>
0 siblings, 1 reply; 9+ messages in thread
From: Andrej Ricnik-Bay @ 2007-03-20 02:53 UTC (permalink / raw)
To: Paul Lambert <paul.lambert@autoledgers.com.au>; +Cc: Phillip Smith <phillip.smith@weatherbeeta.com.au>; pgsql-sql
On 3/20/07, Paul Lambert <paul.lambert@autoledgers.com.au> wrote:
> The source file comes from extracts on our main application which sits
> inside an in-house pretending-to-be-a-dbms file system. The content of
> these extracts would be difficult to change - the extract program would
> need to parse the data looking for quotes and preceed them with the
> necessary escape character.
>
> Not being a proper database dump it's not a simple matter of flicking a
> switch to get it to include the escape character. The way the extracts
> are written would require a few dozen lines of code to each extract, and
> theres about 40ish extracts.
>
> Plus I don't maintain that side of our code, and those that do can be a
> bit lazy and I'd likely be waiting months to get it done - if they even
> decide to do it.
Pipe it through sed and replace the Carets with TABS?
sed 's/^/\t/g' c:/temp/autodrs_deal_lines.txt > c:/temp/autodrs_deal_lines.tab
Then use copy like so:
\copy table from 'c:/temp/autodrs_deal_lines.tab' delimiter E'\t' null ''
Cheers,
Andrej
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Issue with copying data from a text file.
@ 2007-03-20 02:59 Paul Lambert <paul.lambert@autoledgers.com.au>
parent: Andrej Ricnik-Bay <andrej.groups@gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Paul Lambert @ 2007-03-20 02:59 UTC (permalink / raw)
To: Andrej Ricnik-Bay <andrej.groups@gmail.com>; +Cc: Phillip Smith <phillip.smith@weatherbeeta.com.au>; pgsql-sql
Andrej Ricnik-Bay wrote:
> On 3/20/07, Paul Lambert <paul.lambert@autoledgers.com.au> wrote:
>
>> The source file comes from extracts on our main application which sits
>> inside an in-house pretending-to-be-a-dbms file system. The content of
>> these extracts would be difficult to change - the extract program would
>> need to parse the data looking for quotes and preceed them with the
>> necessary escape character.
>>
>> Not being a proper database dump it's not a simple matter of flicking a
>> switch to get it to include the escape character. The way the extracts
>> are written would require a few dozen lines of code to each extract, and
>> theres about 40ish extracts.
>>
>> Plus I don't maintain that side of our code, and those that do can be a
>> bit lazy and I'd likely be waiting months to get it done - if they even
>> decide to do it.
> Pipe it through sed and replace the Carets with TABS?
> sed 's/^/\t/g' c:/temp/autodrs_deal_lines.txt >
> c:/temp/autodrs_deal_lines.tab
>
> Then use copy like so:
> \copy table from 'c:/temp/autodrs_deal_lines.tab' delimiter E'\t' null ''
>
>
> Cheers,
> Andrej
>
>
The data contains tabs... don't ask why... I don't have a clue :P
I'll do something along the lines of sed... but not with sed, I'll use
the command line interpreter on the OpenVMS systems where the extracts
run. I just thought there might have been a quicker way to switch it off
in the copy command, i.e. specifying "quote none" as one of the
parameters to the command. I guess not...
Thanks for the pointers.
P.
--
Paul Lambert
Database Administrator
AutoLedgers
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Issue with copying data from a text file.
@ 2007-03-20 03:05 Phillip Smith <phillip.smith@weatherbeeta.com.au>
parent: Paul Lambert <paul.lambert@autoledgers.com.au>
0 siblings, 1 reply; 9+ messages in thread
From: Phillip Smith @ 2007-03-20 03:05 UTC (permalink / raw)
To: pgsql-sql
Maybe use char 254 or 253 or something similar... Anything that isn't
going to be found in the file.
Have you tried using a string as a delimiter? QUOTE
'THIS.STRING.ISNT.IN.THE.TEXT.FILE'
~p
On Tue, 2007-03-20 at 11:59 +0900, Paul Lambert wrote:
> Andrej Ricnik-Bay wrote:
> > On 3/20/07, Paul Lambert <paul.lambert@autoledgers.com.au> wrote:
> >
> >> The source file comes from extracts on our main application which sits
> >> inside an in-house pretending-to-be-a-dbms file system. The content of
> >> these extracts would be difficult to change - the extract program would
> >> need to parse the data looking for quotes and preceed them with the
> >> necessary escape character.
> >>
> >> Not being a proper database dump it's not a simple matter of flicking a
> >> switch to get it to include the escape character. The way the extracts
> >> are written would require a few dozen lines of code to each extract, and
> >> theres about 40ish extracts.
> >>
> >> Plus I don't maintain that side of our code, and those that do can be a
> >> bit lazy and I'd likely be waiting months to get it done - if they even
> >> decide to do it.
> > Pipe it through sed and replace the Carets with TABS?
> > sed 's/^/\t/g' c:/temp/autodrs_deal_lines.txt >
> > c:/temp/autodrs_deal_lines.tab
> >
> > Then use copy like so:
> > \copy table from 'c:/temp/autodrs_deal_lines.tab' delimiter E'\t' null ''
> >
> >
> > Cheers,
> > Andrej
> >
> >
>
> The data contains tabs... don't ask why... I don't have a clue :P
>
> I'll do something along the lines of sed... but not with sed, I'll use
> the command line interpreter on the OpenVMS systems where the extracts
> run. I just thought there might have been a quicker way to switch it off
> in the copy command, i.e. specifying "quote none" as one of the
> parameters to the command. I guess not...
>
> Thanks for the pointers.
>
> P.
>
Phillip Smith
IT Coordinator
Weatherbeeta P/L
8 Moncrief Rd
Nunawading, Vic, 3131
AUSTRALIA
P. +613 9845 0600
F. +613 9845 0655
E. phillip.smith@weatherbeeta.com.au
Attachments:
smime.p7s (application/x-pkcs7-signature, 2.1K) download
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Issue with copying data from a text file.
@ 2007-03-20 03:17 Michael Fuhr <mike@fuhr.org>
parent: Paul Lambert <paul.lambert@autoledgers.com.au>
1 sibling, 0 replies; 9+ messages in thread
From: Michael Fuhr @ 2007-03-20 03:17 UTC (permalink / raw)
To: Paul Lambert <paul.lambert@autoledgers.com.au>; +Cc: pgsql-sql
On Tue, Mar 20, 2007 at 11:25:38AM +0900, Paul Lambert wrote:
> I note an option in the COPY command to specify the quote character,
> defaulting to double quote. The problem being a single quote will also
> be used in the data, as will other characters. Is there any way to get a
> copy to have no quote character?
Are there any control characters that won't appear in the data?
--
Michael Fuhr
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Issue with copying data from a text file.
@ 2007-03-20 03:21 Paul Lambert <paul.lambert@autoledgers.com.au>
parent: Phillip Smith <phillip.smith@weatherbeeta.com.au>
0 siblings, 0 replies; 9+ messages in thread
From: Paul Lambert @ 2007-03-20 03:21 UTC (permalink / raw)
To: pgsql-sql
Phillip Smith wrote:
> Maybe use char 254 or 253 or something similar... Anything that isn't
> going to be found in the file.
>
> Have you tried using a string as a delimiter? QUOTE
> 'THIS.STRING.ISNT.IN.THE.TEXT.FILE'
>
> ~p
>
I changed it to QUOTE '\f' to set the quote character to form-feed which
is not going to appear in the file and that appears to do the trick
without any changes to the source file.
Thanks,
P.
--
Paul Lambert
Database Administrator
AutoLedgers
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Issue with copying data from a text file.
@ 2007-03-21 17:01 Steve Midgley <public@misuse.org>
0 siblings, 0 replies; 9+ messages in thread
From: Steve Midgley @ 2007-03-21 17:01 UTC (permalink / raw)
To: pgsql-sql
Hi,
I think I had the exact same problem as you do a while back and I
solved it by removing the header row and the "CSV HEADER" clause of the
statement. For the large files I had, it was easier (for me) to remove
the header row than it was to escape out all the quotes (or regen the
file):
COPY deal_lines_temp_load FROM
'c:/temp/autodrs_deal_lines.txt'
WITH DELIMITER AS '^';
I think the parser doesn't look for nor generate quoted rows except
when CSV is specified.. It would be nice if there was a way to specify
a "HEADER" row without invoking CSV parsing rules (friendly hint to
core devs!) :)
Let us all know if that works!
Steve
At 03:14 AM 3/20/2007, you wrote:
>Date: Tue, 20 Mar 2007 11:25:38 +0900
>From: Paul Lambert <paul.lambert@autoledgers.com.au>
>To: pgsql-sql@postgresql.org
>Subject: Issue with copying data from a text file.
>Message-ID: <45FF4622.2010404@autoledgers.com.au>
>
>I have a procedure in place that copies data from a caret delimited
>text
>file into a table storing some information.
>
>One of the fields in the table contains an item description which may
>contain item dimensions such as - 17" alloy wheels
>
>The problem I am getting when I do my load is I believe due to the
>presence of the double quotation marks giving the copy the impression
>that it is to include the information following as a single text
>string
>until it gets to the next set of double quotes. As a result, I get the
>
>following:
>
>AutoDRS=# COPY deal_lines_temp_load FROM
>'c:/temp/autodrs_deal_lines.txt'
>WITH DELIMITER AS '^' CSV HEADER;
>ERROR: value too long for type character varying(30)
>CONTEXT: COPY deal_lines_temp_load, line 87, column order_desc: "17 5
>
>spoke alloy wheels.^1291.18^117.38^983.69^1291.18^^C^^
>
>The column as you can see is defined as a 30 character field, the load
>
>contains in this column ^17" 5 spoke alloy wheels.^
>
>I note an option in the COPY command to specify the quote character,
>defaulting to double quote. The problem being a single quote will also
>
>be used in the data, as will other characters. Is there any way to get
>a
>copy to have no quote character? I.e. read the file and put whatever
>is
>between the caret characters straight into the appropriate field
>exactly
>as is.
>
>TIA,
>Paul.
>
>--
>Paul Lambert
>Database Administrator
>AutoLedgers
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2007-03-21 17:01 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2007-03-20 02:25 Issue with copying data from a text file. Paul Lambert <paul.lambert@autoledgers.com.au>
2007-03-20 02:37 ` Phillip Smith <phillip.smith@weatherbeeta.com.au>
2007-03-20 02:44 ` Paul Lambert <paul.lambert@autoledgers.com.au>
2007-03-20 02:53 ` Andrej Ricnik-Bay <andrej.groups@gmail.com>
2007-03-20 02:59 ` Paul Lambert <paul.lambert@autoledgers.com.au>
2007-03-20 03:05 ` Phillip Smith <phillip.smith@weatherbeeta.com.au>
2007-03-20 03:21 ` Paul Lambert <paul.lambert@autoledgers.com.au>
2007-03-20 03:17 ` Michael Fuhr <mike@fuhr.org>
2007-03-21 17:01 Re: Issue with copying data from a text file. Steve Midgley <public@misuse.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox