pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedNot able to restore database - error: could not decompress data: Allocation error : not enough memory
4+ messages / 2 participants
[nested] [flat]
Commits from this thread:
2024-12-18 3f06324705 04b8601981 8cfff087b5 Fix memory leak in pg_restore with zstd-compressed data.
* Not able to restore database - error: could not decompress data: Allocation error : not enough memory
@ 2024-12-17 20:29 Tomasz Szypowski <tomasz.szypowski@asseco.pl>
2024-12-18 02:22 ` Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 4+ messages in thread
From: Tomasz Szypowski @ 2024-12-17 20:29 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Our database consists of 408032 blobs (pg_largeobject).
Backup was made using zstd and it weights 70GB.
While restoring the used memory increases constantly (from 5 MB to 5GB - i have got 8GB RAM, after reaching 5GB it uses 5GB for some time, maybe swaping) and then crashes with the error.
During this period it has restored about 48000 blobs.
pg_restore: restoring large object with OID 169685
pg_restore: restoring large object with OID 169686
pg_restore: restoring large object with OID 169687
pg_restore: restoring large object with OID 169688
pg_restore: restoring large object with OID 169689
pg_restore: restoring large object with OID 169690
pg_restore: restoring large object with OID 169691
pg_restore: restoring large object with OID 169692
pg_restore: restoring large object with OID 169693
pg_restore: restoring large object with OID 169694
pg_restore: restoring large object with OID 169695
pg_restore: restoring large object with OID 169696
pg_restore: restoring large object with OID 169697
pg_restore: restoring large object with OID 169698
pg_restore: restoring large object with OID 169699
pg_restore: error: could not decompress data: Allocation error : not enough memory
It was checked with the original pg_restore.exe and libzstd.dll as well as with compiled by mingw both 32 and 64 bits. Several libzstd.dll was used. Without success.
Dump: pg_dump.exe --format=custom --compress=zstd:4 --file=zstd.zstd dbname=TEST
Restore: pg_restore -d TEST zstd.zstd
Best regards
Thomas
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory
2024-12-17 20:29 Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tomasz Szypowski <tomasz.szypowski@asseco.pl>
@ 2024-12-18 02:22 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-12-18 07:12 ` RE: Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tomasz Szypowski <tomasz.szypowski@asseco.pl>
0 siblings, 1 reply; 4+ messages in thread
From: Tom Lane @ 2024-12-18 02:22 UTC (permalink / raw)
To: Tomasz Szypowski <tomasz.szypowski@asseco.pl>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Tomasz Szypowski <tomasz.szypowski@asseco.pl> writes:
> Our database consists of 408032 blobs (pg_largeobject).
> Backup was made using zstd and it weights 70GB.
> While restoring the used memory increases constantly (from 5 MB to 5GB - i have got 8GB RAM, after reaching 5GB it uses 5GB for some time, maybe swaping) and then crashes with the error.
Yeah, leak reproduced here. Apparently it's specific to the zstd
code path, because I don't see it with the default compression
method. Should be easy to fix (awaiting valgrind results),
but in the meantime just use default compression.
Thanks for the report!
regards, tom lane
^ permalink raw reply [nested|flat] 4+ messages in thread
* RE: Not able to restore database - error: could not decompress data: Allocation error : not enough memory
2024-12-17 20:29 Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tomasz Szypowski <tomasz.szypowski@asseco.pl>
2024-12-18 02:22 ` Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tom Lane <tgl@sss.pgh.pa.us>
@ 2024-12-18 07:12 ` Tomasz Szypowski <tomasz.szypowski@asseco.pl>
2024-12-18 15:13 ` Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 4+ messages in thread
From: Tomasz Szypowski @ 2024-12-18 07:12 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Thank you for the quick response.
Do you think it will be fixed within few months? Next year we plan to upgrade our clients to 17 and zstd compression is about two times faster than gzip and produces about 20% less backup.
Regards
Thomas
-----Original Message-----
From: Tom Lane <tgl@sss.pgh.pa.us>
Sent: Wednesday, December 18, 2024 3:22 AM
To: Tomasz Szypowski <tomasz.szypowski@asseco.pl>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory
[Nie otrzymujesz cz?sto wiadomo?ci e-mail z tgl@sss.pgh.pa.us. Dowiedz si?, dlaczego jest to wa?ne, na stronie https://aka.ms/LearnAboutSenderIdentification ]
Tomasz Szypowski <tomasz.szypowski@asseco.pl> writes:
> Our database consists of 408032 blobs (pg_largeobject).
> Backup was made using zstd and it weights 70GB.
> While restoring the used memory increases constantly (from 5 MB to 5GB - i have got 8GB RAM, after reaching 5GB it uses 5GB for some time, maybe swaping) and then crashes with the error.
Yeah, leak reproduced here. Apparently it's specific to the zstd code path, because I don't see it with the default compression method. Should be easy to fix (awaiting valgrind results), but in the meantime just use default compression.
Thanks for the report!
regards, tom lane
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory
2024-12-17 20:29 Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tomasz Szypowski <tomasz.szypowski@asseco.pl>
2024-12-18 02:22 ` Re: Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tom Lane <tgl@sss.pgh.pa.us>
2024-12-18 07:12 ` RE: Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tomasz Szypowski <tomasz.szypowski@asseco.pl>
@ 2024-12-18 15:13 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 4+ messages in thread
From: Tom Lane @ 2024-12-18 15:13 UTC (permalink / raw)
To: Tomasz Szypowski <tomasz.szypowski@asseco.pl>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Tomasz Szypowski <tomasz.szypowski@asseco.pl> writes:
> Do you think it will be fixed within few months? Next year we plan to upgrade our clients to 17 and zstd compression is about two times faster than gzip and produces about 20% less backup.
The fix will be in February's releases.
regards, tom lane
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2024-12-18 15:13 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-12-17 20:29 Not able to restore database - error: could not decompress data: Allocation error : not enough memory Tomasz Szypowski <tomasz.szypowski@asseco.pl>
2024-12-18 02:22 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-12-18 07:12 ` Tomasz Szypowski <tomasz.szypowski@asseco.pl>
2024-12-18 15:13 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox