Date: Tue, 29 May 2007 08:39:57 +0200
From: Johann Spies <jspies@sun.ac.za>
To: pgsql-admin@postgresql.org
Subject: Re: ERROR: invalid memory alloc request size
Message-ID: <20070529063956.GA20616@sun.ac.za>
Mail-Followup-To: Johann Spies <jspies@sun.ac.za>, pgsql-admin@postgresql.org
References: <5221.1179337804@sss.pgh.pa.us> <20070517132033.GA9482@sun.ac.za>
	<25608.1179412804@sss.pgh.pa.us> <20070518122942.GA28931@sun.ac.za>
	<27574.1179501021@sss.pgh.pa.us> <20070521140718.GC10742@sun.ac.za>
	<10687.1179758440@sss.pgh.pa.us> <20070528134559.GC5615@sun.ac.za>
	<20070528144651.GA7795@alvh.no-ip.org>
	<28065.1180365517@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <28065.1180365517@sss.pgh.pa.us>
User-Agent: Mutt/1.5.13 (2006-08-11)

On Mon, May 28, 2007 at 11:18:37AM -0400, Tom Lane wrote:
L> Alvaro Herrera <alvherre@commandprompt.com> writes:
> > Johann Spies wrote:
> >> #3  0x00000000005ff1c8 in perform_default_encoding_conversion (
> >> src=0x2b881e87f040 "\"2007-04-20\" \"18:07:06\" \"192.168.0.100\" \"73\" \"http://www.kerkbode.co.za/kerkbode/images/blank.gif\\\"\"\n\"2007-04-20\" \"18:07:06\" \"192.168.0.100\" \"69\" \"http://develop.christians.co.za/phpAdsNew/adx.js\"\n\"20"..., len=337739539, is_client_to_server=5 '\005') at mbutils.c:461
> 
> 
> > Interesting.  You are passing a single 300 MB string here, not
> > 90-something char lines.  The problem is probably in newlines.  Why are
> > they not getting used as line separators escapes me.
> 
> Stripped of gdb's own backslashing, the data seems to be 
> 
> 	...images/blank.gif\""
> 	"2007...
>

Thanks to you both.  I have grepped -C 5 for the particuler line(s) and the result
was:

"2007-04-20" "18:07:06" "146.232.182.90" "0" "http://static.ak.facebook.com/images/x_to_hide_hover.gif"
"2007-04-20" "18:07:06" "192.168.0.100" "73" "http://www.kerkbode.co.za/kerkbode/images/blank.gif\""
"2007-04-20" "18:07:06" "192.168.0.100" "69" "http://develop.christians.co.za/phpAdsNew/adx.js"
"2007-04-20" "18:07:06" "146.232.186.248" "873" "http://ads.cnn.com/html.ng/site=cnn&cnn_position=306x60_lft&cnn_rollup=homepage&params.styles=fs&tile=1177085201252&page.allowcompete=yes&domId=283029"
"2007-04-20" "18:07:06" "146.232.182.216" "13958" "http://us.js2.yimg.com/us.js.yimg.com/lib/s2/yschx_intl_20070330.css"
"2007-04-20" "18:07:06" "146.232.141.240" "5246" "http://photos-819.ak.facebook.com/photos-ak-sf2p/v75/18/87/46601522/a46601522_30445819_7495.jpg"
"2007-04-20" "18:07:06" "146.232.53.112" "30663" "http://spe.atdmt.com/ds/YCMEAOAKLO07/oakley_sportsperformance/mlt_RI_300x250.jpg?"


> which leads me to think that the dump was generated on the assumption
> that backslash is the escape character, but is being reloaded under a
> different assumption.  Once you got out-of-sync on that one data value,
> you'd have a mess --- all the intended line-ending newlines would be
> inside quotes, up till you hit another URL with an embedded quote
> (or newline?).  CSV is not exactly a robust representation :-(

What alternative is there to import large amounts of data?

 
> So the short answer seems to be that we were all misled by the large
> alloc-request number into thinking there was some data-corruption
> behavior involved, when we should have been quizzing Johann on exactly
> what COPY parameters he was giving.

I think I gave it in my first email.  Here it is:

\copy squidlogs from '/home/js/accesspg.csv' using delimiters ' ' csv quote as '"'

Regards
Johann
-- 
Johann Spies          Telefoon: 021-808 4036
Informasietegnologie, Universiteit van Stellenbosch

     "Let no man despise thy youth; but be thou an example 
      of the believers, in word, in conversation, in
      charity, in spirit, in faith, in purity."      
                                       I Timothy 4:12