From: Ted Allen <tallen@blackducksoftware.com>
To: Merlin Moncure <mmoncure@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-performance@postgresql.org <pgsql-performance@postgresql.org>
Subject: Re: Troubles dumping a very large table.
Date: Sun, 28 Dec 2008 22:11:47 -0500
Message-ID: <49583FF3.8030105@blackducksoftware.com> (raw)
In-Reply-To: <b42b73150812261150k6b17b819i704b6b7720243eb8@mail.gmail.com>
References: <495271EB.80508@blackducksoftware.com>
<12778.1230140996@sss.pgh.pa.us>
<FD2D13C2E4A95C4499BF8AA8BAB85AEC20DD41D698@BDSEXCH2K7CLS.blackducksoftware.com>
<13141.1230313118@sss.pgh.pa.us>
<b42b73150812261150k6b17b819i704b6b7720243eb8@mail.gmail.com>
I was hoping use pg_dump and not to have to do a manual dump but if that
latest solution (moving rows >300mb elsewhere and dealing with them
later) does not work I'll try that.
Thanks everyone.
Merlin Moncure wrote:
> On Fri, Dec 26, 2008 at 12:38 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
>> Ted Allen <tallen@blackducksoftware.com> writes:
>>
>>> 600mb measured by get_octet_length on data. If there is a better way to measure the row/cell size, please let me know because we thought it was the >1Gb problem too. We thought we were being conservative by getting rid of the larger rows but I guess we need to get rid of even more.
>>>
>> Yeah, the average expansion of bytea data in COPY format is about 3X :-(
>> So you need to get the max row length down to around 300mb. I'm curious
>> how you got the data in to start with --- were the values assembled on
>> the server side?
>>
>
> Wouldn't binary style COPY be more forgiving in this regard? (if so,
> the OP might have better luck running COPY BINARY)...
>
> This also goes for libpq traffic..large (>1mb) bytea definately want
> to be passed using the binary switch in the protocol.
>
> merlin
>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: tallen@blackducksoftware.com, mmoncure@gmail.com, tgl@sss.pgh.pa.us
Subject: Re: Troubles dumping a very large table.
In-Reply-To: <49583FF3.8030105@blackducksoftware.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox