Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 3175C1338142 for ; Wed, 15 Sep 2010 10:03:58 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 06736-01-2 for ; Wed, 15 Sep 2010 13:03:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from qmta05.westchester.pa.mail.comcast.net (qmta05.westchester.pa.mail.comcast.net [76.96.62.48]) by mail.postgresql.org (Postfix) with ESMTP id 18A591337DD6 for ; Wed, 15 Sep 2010 10:03:14 -0300 (ADT) Received: from omta05.westchester.pa.mail.comcast.net ([76.96.62.43]) by qmta05.westchester.pa.mail.comcast.net with comcast id 6yUq1f0010vyq2s5513GyP; Wed, 15 Sep 2010 13:03:16 +0000 Received: from sz0093.wc.mail.comcast.net ([76.96.58.151]) by omta05.westchester.pa.mail.comcast.net with comcast id 713G1f0093FmDic3R13Gpe; Wed, 15 Sep 2010 13:03:16 +0000 Date: Wed, 15 Sep 2010 13:03:16 +0000 (UTC) From: Lou Picciano To: pgsql-admin Message-ID: <594168409.400001.1284555796001.JavaMail.root@sz0093a.westchester.pa.mail.comcast.net> In-Reply-To: <203410795.399786.1284555612401.JavaMail.root@sz0093a.westchester.pa.mail.comcast.net> Subject: How To: LARGE html text or csv file COPY FROM? MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_400000_1401323694.1284555796000" X-Originating-IP: [76.117.184.231] X-Mailer: Zimbra 6.0.5_GA_2429.RHEL4 (ZimbraWebClient - SAF3 (Mac)/6.0.5_GA_2427.RHEL4) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-0.509 tagged_above=-10 required=5 tests=BAYES_05=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201009/75 X-Sequence-Number: 35619 ------=_Part_400000_1401323694.1284555796000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hello Friends, We're trying something for the first time: A COPY into a database, from a TEXT (or CSV) file containing one really, really, big field of html. The field happens to be content of complete webpages, which we then need to later analyze, slice, dice, etc. - so it's verbatim html, with all the carriage returns, spaces, linefeeds(?) and double quotes included! Problem is: With the very first record, the COPY commands hiccups with: missing data from column error. in CSV mode, it's 'extra data after last expected column' (yes, using different input files for test). Both errors above make sense to me; COPY is running into either a cr or a tab character in each case. Q: Is there way to handle this directly, as a PG import? Meanwhile, we're off into using grep/gawk to remove all carriage returns in the field? TIA for any help, inspiration, recipes (or time in the stocks). Lou ------=_Part_400000_1401323694.1284555796000 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <= div style=3D'font-family: Verdana; font-size: 12pt; color: #000000'>Hello F= riends,

We're trying something for the first time: A COP= Y into a database, from a TEXT (or CSV) file containing one really, really,= big field of html.

The field happens to be conten= t of complete webpages, which we then need to later analyze, slice, dice, e= tc. - so it's verbatim html, with all the carriage returns, spaces, linefee= ds(?) and double quotes included!

Problem is: With= the very first record, the COPY commands hiccups with: missing data from c= olumn error.
in CSV mode, it's 'extra data after last expected co= lumn'  (yes, using different input files for test).

Both errors above make sense to me; COPY is running into either a cr= or a tab character in each case.

Q: Is there way = to handle this directly, as a PG import? 

Mea= nwhile, we're off into using grep/gawk to remove all carriage returns in th= e field?

TIA for any help, inspiration, recipes (o= r time in the stocks).     Lou
------=_Part_400000_1401323694.1284555796000--