From nikola@rs-pi.com Tue Dec  24 10:24:46 2002
Received: from localhost (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with ESMTP id 0BE78475843
	for <pgsql-sql@postgresql.org>; Tue, 24 Dec 2002 05:24:46 -0500 (EST)
Received: from rs-pi.com (unknown [193.95.196.91])
	by postgresql.org (Postfix) with ESMTP id 4C997474E61
	for <pgsql-sql@postgresql.org>; Tue, 24 Dec 2002 05:24:45 -0500 (EST)
Received: from spot (fw.rs-pi.com [193.95.196.154])
	by rs-pi.com (Postfix) with SMTP id 210258DE9
	for <pgsql-sql@postgresql.org>; Tue, 24 Dec 2002 11:29:42 +0100 (CET)
Message-ID: <007d01c2ab36$85d92760$1100a8c0@rspi>
From: "Nikola Ivacic" <nikola@rs-pi.com>
To: <pgsql-sql@postgresql.org>
Subject: COPY command problems
Date: Tue, 24 Dec 2002 11:23:38 +0100
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_007A_01C2AB3E.E79467A0"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-Virus-Scanned: by AMaViS new-20020517
X-Archive-Number: 200212/266
X-Sequence-Number: 10854

This is a multi-part message in MIME format.

------=_NextPart_000_007A_01C2AB3E.E79467A0
Content-Type: text/plain;
	charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable

I need help

Has somebody experienced difficulties using COPY command on large files.
I have a large (250MB) file and each time I insert records I've got one or =
more (< 30 of cca 1079000) corrupted
records. The number of corrupted records is not constant (i.e. 1, 30, 7, 23=
 etc..)

p.s. the file is tab-nl separated and corruption is done in the middle of r=
ecord field
(not the end or begining)

lp

Nikola


------=_NextPart_000_007A_01C2AB3E.E79467A0
Content-Type: text/html;
	charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Diso-8859-2">
<META content=3D"MSHTML 6.00.2600.0" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>I need help</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Has somebody experienced difficulties usin=
g COPY=20
command on large files.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>I have a large (250MB) file and each time =
I insert=20
records I've got&nbsp;one or more (&lt; 30 of cca 1079000)=20
corrupted</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>records. The number of corrupted records i=
s not=20
constant (i.e. 1, 30, 7, 23 etc..)</FONT></DIV>
<DIV>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>p.s. the file is tab-nl&nbsp;separated and=
=20
corruption is done in the middle&nbsp;of record field</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>(not the end or begining)</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>lp</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Nikola</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV></BODY></HTML>

------=_NextPart_000_007A_01C2AB3E.E79467A0--




From tgl@sss.pgh.pa.us Tue Dec  24 15:24:46 2002
Received: from localhost (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with ESMTP id 7128A4762FE
	for <pgsql-sql@postgresql.org>; Tue, 24 Dec 2002 10:24:46 -0500 (EST)
Received: from sss.pgh.pa.us (unknown [192.204.191.242])
	by postgresql.org (Postfix) with ESMTP id C468D4762D8
	for <pgsql-sql@postgresql.org>; Tue, 24 Dec 2002 10:24:45 -0500 (EST)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.12.6/8.12.6) with ESMTP id gBOFOnd0019756;
	Tue, 24 Dec 2002 10:24:50 -0500 (EST)
To: "Nikola Ivacic" <nikola@rs-pi.com>
Cc: pgsql-sql@postgresql.org
Subject: Re: COPY command problems 
In-reply-to: <007d01c2ab36$85d92760$1100a8c0@rspi> 
References: <007d01c2ab36$85d92760$1100a8c0@rspi>
Comments: In-reply-to "Nikola Ivacic" <nikola@rs-pi.com>
	message dated "Tue, 24 Dec 2002 11:23:38 +0100"
Date: Tue, 24 Dec 2002 10:24:49 -0500
Message-ID: <19755.1040743489@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
X-Virus-Scanned: by AMaViS new-20020517
X-Archive-Number: 200212/267
X-Sequence-Number: 10855

"Nikola Ivacic" <nikola@rs-pi.com> writes:
> Has somebody experienced difficulties using COPY command on large files.
> I have a large (250MB) file and each time I insert records I've got one or =
> more (< 30 of cca 1079000) corrupted
> records. The number of corrupted records is not constant (i.e. 1, 30, 7, 23=
>  etc..)

I'd bet on flaky hardware --- have you run memory and disk tests?  If
the COPY data is passing across a network, then network problems are
also worthy of suspicion.

			regards, tom lane



From nikola@rs-pi.com Tue Dec  24 15:56:12 2002
Received: from localhost (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with ESMTP id F03C5475A09
	for <pgsql-sql@postgresql.org>; Tue, 24 Dec 2002 10:56:12 -0500 (EST)
Received: from rs-pi.com (unknown [193.95.196.91])
	by postgresql.org (Postfix) with ESMTP id ADEF1475843
	for <pgsql-sql@postgresql.org>; Tue, 24 Dec 2002 10:56:11 -0500 (EST)
Received: from spot (fw.rs-pi.com [193.95.196.154])
	by rs-pi.com (Postfix) with SMTP
	id 60A5D8D8E; Tue, 24 Dec 2002 17:01:12 +0100 (CET)
Message-ID: <013101c2ab64$d368df60$1100a8c0@rspi>
From: "Nikola Ivacic" <nikola@rs-pi.com>
To: "Tom Lane" <tgl@sss.pgh.pa.us>
Cc: "Ales Kuclar" <ales.kuclar@rs-pi.com>, <pgsql-sql@postgresql.org>
References: <007d01c2ab36$85d92760$1100a8c0@rspi>
	<19755.1040743489@sss.pgh.pa.us>
Subject: Re: COPY command problems 
Date: Tue, 24 Dec 2002 16:55:05 +0100
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-2"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
X-Virus-Scanned: by AMaViS new-20020517
X-Archive-Number: 200212/268
X-Sequence-Number: 10856

It must be internal error:
two reasons:
1.) the original file is OK (I checked with grep + there is no network
envolved)
2.) Error has strange patern: it substitutes 0x31 with 0x21 (1 with !) also
0x34 with 0x24 (4 with $)
and 0x39 with 0x29 (9 with ) )

so I guess you are right.

can you suggest some tools for FreeBSD
to test RAM, because I think the hard disk is ok.

p.s. right now I am testing it with splited file

thanks

Nikola

----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Nikola Ivacic" <nikola@rs-pi.com>
Cc: <pgsql-sql@postgresql.org>
Sent: Tuesday, December 24, 2002 4:24 PM
Subject: Re: [SQL] COPY command problems


> "Nikola Ivacic" <nikola@rs-pi.com> writes:
> > Has somebody experienced difficulties using COPY command on large files.
> > I have a large (250MB) file and each time I insert records I've got one
or =
> > more (< 30 of cca 1079000) corrupted
> > records. The number of corrupted records is not constant (i.e. 1, 30, 7,
23=
> >  etc..)
>
> I'd bet on flaky hardware --- have you run memory and disk tests?  If
> the COPY data is passing across a network, then network problems are
> also worthy of suspicion.
>
> regards, tom lane
>




From mallah@trade-india.com Fri Dec  27 11:12:53 2002
Received: from localhost (postgresql.org [64.49.215.8])
	by postgresql.org (Postfix) with ESMTP id BD508476029
	for <pgsql-sql@postgresql.org>; Fri, 27 Dec 2002 06:12:53 -0500 (EST)
Received: from rtlocal.trade-india.com (mail-relay.trade-india.com
	[203.196.129.235])
	by postgresql.org (Postfix) with SMTP id AAF5A475FC4
	for <pgsql-sql@postgresql.org>; Fri, 27 Dec 2002 06:12:49 -0500 (EST)
Received: (qmail 1592 invoked from network); 27 Dec 2002 11:05:17 -0000
Received: from unknown (HELO system67.trade-india-local.com) (192.168.0.67)
	by infocom-236-129-del.trade-india.com with SMTP;
	27 Dec 2002 11:05:17 -0000
From: "Rajesh Kumar Mallah." <mallah@trade-india.com>
Organization: Infocom Network Limited.
To: "Nikola Ivacic" <nikola@rs-pi.com>,
	"Tom Lane" <tgl@sss.pgh.pa.us>
Subject: Re: COPY command problems
Date: Fri, 27 Dec 2002 16:47:34 +0530
User-Agent: KMail/1.5
Cc: "Ales Kuclar" <ales.kuclar@rs-pi.com>, <pgsql-sql@postgresql.org>
References: <007d01c2ab36$85d92760$1100a8c0@rspi>
	<19755.1040743489@sss.pgh.pa.us>
	<013101c2ab64$d368df60$1100a8c0@rspi>
In-Reply-To: <013101c2ab64$d368df60$1100a8c0@rspi>
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-2"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200212271647.34446.mallah@trade-india.com>
X-Virus-Scanned: by AMaViS new-20020517
X-Archive-Number: 200212/272
X-Sequence-Number: 10860



http://www.memtest86.com/
may be useful.


regds
mallah.

On Tuesday 24 December 2002 09:25 pm, Nikola Ivacic wrote:
> It must be internal error:
> two reasons:
> 1.) the original file is OK (I checked with grep + there is no network
> envolved)
> 2.) Error has strange patern: it substitutes 0x31 with 0x21 (1 with !) also
> 0x34 with 0x24 (4 with $)
> and 0x39 with 0x29 (9 with ) )
>
> so I guess you are right.
>
> can you suggest some tools for FreeBSD
> to test RAM, because I think the hard disk is ok.
>
> p.s. right now I am testing it with splited file
>
> thanks
>
> Nikola
>
> ----- Original Message -----
> From: "Tom Lane" <tgl@sss.pgh.pa.us>
> To: "Nikola Ivacic" <nikola@rs-pi.com>
> Cc: <pgsql-sql@postgresql.org>
> Sent: Tuesday, December 24, 2002 4:24 PM
> Subject: Re: [SQL] COPY command problems
>
> > "Nikola Ivacic" <nikola@rs-pi.com> writes:
> > > Has somebody experienced difficulties using COPY command on large
> > > files. I have a large (250MB) file and each time I insert records I've
> > > got one
>
> or =
>
> > > more (< 30 of cca 1079000) corrupted
> > > records. The number of corrupted records is not constant (i.e. 1, 30,
> > > 7,
>
> 23=
>
> > >  etc..)
> >
> > I'd bet on flaky hardware --- have you run memory and disk tests?  If
> > the COPY data is passing across a network, then network problems are
> > also worthy of suspicion.
> >
> > regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

-- 
Rajesh Kumar Mallah,
Project Manager (Development)
Infocom Network Limited, New Delhi
phone: +91(11)6152172 (221) (L) ,9811255597 (M)

Visit http://www.trade-india.com ,
India's Leading B2B eMarketplace.