pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Large file support needed?  Trying to identify root of error.
4+ messages / 3 participants
[nested] [flat]

* Large file support needed?  Trying to identify root of error.
@ 2004-07-19 19:28  Kris Kiger <kris@musicrebellion.com>
  0 siblings, 2 replies; 4+ messages in thread

From: Kris Kiger @ 2004-07-19 19:28 UTC (permalink / raw)
  To: pgsql-admin

I've got a database that is a single table with 5 integers, a timestamp 
with time zone, and a boolean.  The table is 170 million rows in length. 
 The contents of the tar'd dump file it produced using:
    pg_dump -U postgres -Ft test > test_backup.tar
is: 8.dat (approximately 8GB), a toc, and restore.sql.  

No errors are reported on dump, however, when a restore is attempted I get:

ERROR:  unexpected message type 0x58 during COPY from stdin
CONTEXT:  COPY test_table, line 86077128: ""
ERROR:  could not send data to client: Broken pipe
CONTEXT:  COPY test_table, line 86077128: ""

I am doing the dump & restore on the same machine.

Any ideas?  If the file is too large, is there anyway postgres could 
break it up into smaller chunks for the tar when backing up?  Thanks for 
the help!

Kris






^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Large file support needed? Trying to identify root of error.
@ 2004-07-19 20:00  Tom Lane <tgl@sss.pgh.pa.us>
  parent: Kris Kiger <kris@musicrebellion.com>
  1 sibling, 1 reply; 4+ messages in thread

From: Tom Lane @ 2004-07-19 20:00 UTC (permalink / raw)
  To: Kris Kiger <kris@musicrebellion.com>; +Cc: pgsql-admin

Kris Kiger <kris@musicrebellion.com> writes:
> I've got a database that is a single table with 5 integers, a timestamp 
> with time zone, and a boolean.  The table is 170 million rows in length. 
>  The contents of the tar'd dump file it produced using:
>     pg_dump -U postgres -Ft test > test_backup.tar
> is: 8.dat (approximately 8GB), a toc, and restore.sql.  

Try -Fc instead.  I have some recollection that tar format has a
hard-wired limit on the size of individual members.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Large file support needed?  Trying to identify root of
@ 2004-07-19 20:07  Scott Marlowe <smarlowe@qwest.net>
  parent: Kris Kiger <kris@musicrebellion.com>
  1 sibling, 0 replies; 4+ messages in thread

From: Scott Marlowe @ 2004-07-19 20:07 UTC (permalink / raw)
  To: Kris Kiger <kris@musicrebellion.com>; +Cc: pgsql-admin

On Mon, 2004-07-19 at 13:28, Kris Kiger wrote:
> I've got a database that is a single table with 5 integers, a timestamp 
> with time zone, and a boolean.  The table is 170 million rows in length. 
>  The contents of the tar'd dump file it produced using:
>     pg_dump -U postgres -Ft test > test_backup.tar
> is: 8.dat (approximately 8GB), a toc, and restore.sql.  
> 
> No errors are reported on dump, however, when a restore is attempted I get:
> 
> ERROR:  unexpected message type 0x58 during COPY from stdin
> CONTEXT:  COPY test_table, line 86077128: ""
> ERROR:  could not send data to client: Broken pipe
> CONTEXT:  COPY test_table, line 86077128: ""
> 
> I am doing the dump & restore on the same machine.
> 
> Any ideas?  If the file is too large, is there anyway postgres could 
> break it up into smaller chunks for the tar when backing up?  Thanks for 
> the help!

How, exactly, are you restoring?  Doing things like:

cat file|pg_restore ...

can cause problems because cat is often limited to 2 gigs on many OSes. 
Just use a redirect:

psql dbname <file




^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: Large file support needed? Trying to identify root of
@ 2004-07-20 15:09  Kris Kiger <kris@musicrebellion.com>
  parent: Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 4+ messages in thread

From: Kris Kiger @ 2004-07-20 15:09 UTC (permalink / raw)
  To: pgsql-admin

Thanks Tom, -Fc worked great!  I appreciate your help.

Kris

Tom Lane wrote:

>Kris Kiger <kris@musicrebellion.com> writes:
>  
>
>>I've got a database that is a single table with 5 integers, a timestamp 
>>with time zone, and a boolean.  The table is 170 million rows in length. 
>> The contents of the tar'd dump file it produced using:
>>    pg_dump -U postgres -Ft test > test_backup.tar
>>is: 8.dat (approximately 8GB), a toc, and restore.sql.  
>>    
>>
>
>Try -Fc instead.  I have some recollection that tar format has a
>hard-wired limit on the size of individual members.
>
>			regards, tom lane
>
>---------------------------(end of broadcast)---------------------------
>TIP 6: Have you searched our list archives?
>
>               http://archives.postgresql.org
>  
>




^ permalink  raw  reply  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2004-07-20 15:09 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-07-19 19:28 Large file support needed?  Trying to identify root of error. Kris Kiger <kris@musicrebellion.com>
2004-07-19 20:00 ` Re: Large file support needed? Trying to identify root of error. Tom Lane <tgl@sss.pgh.pa.us>
2004-07-20 15:09   ` Re: Large file support needed? Trying to identify root of Kris Kiger <kris@musicrebellion.com>
2004-07-19 20:07 ` Re: Large file support needed?  Trying to identify root of Scott Marlowe <smarlowe@qwest.net>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox