Subject: Re: Backup

From: Ron Johnson <[email-masked]>
Date: Wed, 16 Oct 2024 16:02:24 -0400

On Wed, Oct 16, 2024 at 4:00 PM Achilleas Mantzios <
[email-masked]> wrote:

> Στις 16/10/24 22:55, ο/η Ron Johnson έγραψε:
>
> On Wed, Oct 16, 2024 at 3:37 PM Andy Hartman <[email-masked]>
> wrote:
>
>> I am very new to Postgres and have always worked in the mssql world. I'm
>> looking for suggestions on DB backups. I currently have a DB used to store
>> Historical information that has images it's currently around 100gig.
>>
>> I'm looking to take a monthly backup as I archive a month of data at a
>> time. I am looking for it to be compressed and have a machine that has
>> multiple cpu's and ample memory.
>>
>> Suggestions on things I can try ?
>> I did a pg_dump using these parms
>> --format=t --blobs lobarch
>>
>> it ran my device out of storage:
>>
>> pg_dump: error: could not write to output file: No space left on device
>>
>> I have 150gig free on my backup drive... can obviously add more
>>
>> looking for the quickest and smallest backup file output...
>>
>> Thanks again for help\suggestions
>>
>
> Step 1: redesign your DB to *NOT* use large objects.  It's an old, slow
> and unmaintained data type.  The data type is what you should use.
>
> You mean bytea I guess. As a side note, (not a fan of LOs), I had the
> impression that certain drivers such as the JDBC support streaming for LOs
> but not for bytea? It's been a while I haven't hit the docs tho.
>

Our database is stuffed with images in bytea fields.  The Java application
uses JDBC and handles them just fine.

-- 
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> crustacean!

---

From: Achilleas Mantzios <[email-masked]>
Date: Wed, 16 Oct 2024 23:16:53 +0300

Στις 16/10/24 23:02, ο/η Ron Johnson έγραψε:
> On Wed, Oct 16, 2024 at 4:00 PM Achilleas Mantzios 
> <[email-masked]> wrote:
>
>>     On Wed, Oct 16, 2024 at 3:37 PM Andy Hartman
>>     <[email-masked]> wrote:
>>
>>         I am very new to Postgres and have always worked in the mssql
>>         world. I'm looking for suggestions on DB backups. I currently
>>         have a DB used to store Historical information that has
>>         images it's currently around 100gig.
>>
>>         I'm looking to take a monthly backup as I archive a month of
>>         data at a time. I am looking for it to be compressed and have
>>         a machine that has multiple cpu's and ample memory.
>>
>>     Step 1: redesign your DB to *NOT* use large objects.  It's an
>>     old, slow and unmaintained data type.  The data type is what you
>>     should use.
>     You mean bytea I guess. As a side note, (not a fan of LOs), I had
>     the impression that certain drivers such as the JDBC support
>     streaming for LOs but not for bytea? It's been a while I haven't
>     hit the docs tho.
>
>
> Our database is stuffed with images in bytea fields.  The Java 
> application uses JDBC and handles them just fine.
Likewise, but the "streaming" part is still not clear to me, unless one 
reads the newest JDBC source. Lots of problems due to image explosion, 
java app heap space exhaustion and the like.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Wed, 16 Oct 2024 23:48:47 +0200

>         On Wed, Oct 16, 2024 at 3:37 PM Andy Hartman <[email-masked]>
[...]
> 
>         Step 1: redesign your DB to NOT use large objects.  It's an old, slow
> Our database is stuffed with images in bytea fields.  The Java application uses
> JDBC and handles them just fine.

Images are usually small enough (a few MB) that they don't need to be
streamed.

I don't think bytea can be streamed in general. It's just like text, you
write and read the whole thing at once.

If you have data which is too large for that and want to store it in
bytea fields, you'll probably have to chunk it yourself (which you
probably have to anyway because for me "so large it has to be streamed"
implies "at least possibly larger than 1 GB").

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | [email-masked]         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

---

From: Andy Hartman <[email-masked]>
Date: Wed, 16 Oct 2024 18:59:32 -0400

I'm on Ver16 and yes Our database has  image in a bytea field.

Running on Win22 box...