pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedHow To: LARGE html text or csv file COPY FROM?
2+ messages / 2 participants
[nested] [flat]
* How To: LARGE html text or csv file COPY FROM?
@ 2010-09-15 13:03 Lou Picciano <loupicciano@comcast.net>
0 siblings, 1 reply; 2+ messages in thread
From: Lou Picciano @ 2010-09-15 13:03 UTC (permalink / raw)
To: pgsql-admin
Hello Friends,
We're trying something for the first time: A COPY into a database, from a TEXT (or CSV) file containing one really, really, big field of html.
The field happens to be content of complete webpages, which we then need to later analyze, slice, dice, etc. - so it's verbatim html, with all the carriage returns, spaces, linefeeds(?) and double quotes included!
Problem is: With the very first record, the COPY commands hiccups with: missing data from column error.
in CSV mode, it's 'extra data after last expected column' (yes, using different input files for test).
Both errors above make sense to me; COPY is running into either a cr or a tab character in each case.
Q: Is there way to handle this directly, as a PG import?
Meanwhile, we're off into using grep/gawk to remove all carriage returns in the field?
TIA for any help, inspiration, recipes (or time in the stocks). Lou
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: How To: LARGE html text or csv file COPY FROM?
@ 2010-09-16 01:26 Greg Smith <greg@2ndquadrant.com>
parent: Lou Picciano <loupicciano@comcast.net>
0 siblings, 0 replies; 2+ messages in thread
From: Greg Smith @ 2010-09-16 01:26 UTC (permalink / raw)
To: Lou Picciano <loupicciano@comcast.net>; +Cc: pgsql-admin
Lou Picciano wrote:p { margin: 0; }
> Both errors above make sense to me; COPY is running into either a cr
> or a tab character in each case.
Even though they might seem very similar at first, text mode and CSV
mode work very differently here. To quote TFM at
http://www.postgresql.org/docs/current/static/sql-copy.html , which
covers all this if you dig into it enough:
"CSV mode will both recognize and produce CSV files with quoted values
containing embedded carriage returns and line feeds. Thus the files are
not strictly one line per table row like text-mode files."
So as long as the field is quoted, you can have a CR or tab in the middle.
In text mode: "the following characters /must/ be preceded by a
backslash if they appear as part of a column value: backslash itself,
newline, carriage return, and the current delimiter character." Text
mode will therefore require escaping everything on that list.
So if this is close to being importable in CSV mode, you might just need
to wrap the HTML field with a pair of quotes, and escape any quote
characters that are inside of it. Text mode is going to require a whole
escaping run of all the reserved characters that are input.
--
Greg Smith, 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services and Support www.2ndQuadrant.us
Author, "PostgreSQL 9.0 High Performance" Pre-ordering at:
https://www.packtpub.com/postgresql-9-0-high-performance/book
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2010-09-16 01:26 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-09-15 13:03 How To: LARGE html text or csv file COPY FROM? Lou Picciano <loupicciano@comcast.net>
2010-09-16 01:26 ` Greg Smith <greg@2ndquadrant.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox