From: Tom Lane <tgl@sss.pgh.pa.us>
To: Marc Rechté <marc4@rechte.fr>
Cc: pgsql-performance@lists.postgresql.org
Subject: Re: OOM killer while pg_restore
Date: Thu, 03 Mar 2022 13:43:32 -0500
Message-ID: <3305985.1646333012@sss.pgh.pa.us> (raw)
In-Reply-To: <51f8f5dc-9cb7-f3b7-1705-447e16d1ac6b@rechte.fr>
References: <866ba95f-86fb-fa9b-ecee-65281160cff9@rechte.fr>
<3161846.1646321519@sss.pgh.pa.us>
<51f8f5dc-9cb7-f3b7-1705-447e16d1ac6b@rechte.fr>
=?UTF-8?Q?Marc_Recht=c3=a9?= <marc4@rechte.fr> writes:
> Le 03/03/2022 à 16:31, Tom Lane a écrit :
>> Does memory consumption hold steady if you drop the FK constraints?
> Actually the number of rows is 232735712.
> Accordingly the RAM consumption would be x12 x3 = 7.8 GiB.
> This is close to the 8,1g I reported earlier (actually it was closer to
> 7.8 GB, due to GiB vs. GB confusion).
> So there is no memory leak.
> It took 16 hours on my box to reach that RAM consumption, and then the
> COPY failed when checking the first FK (as the referenced table was empty).
I'm guessing it was swapping like mad :-(
We've long recommended dropping FK constraints during bulk data loads,
and then re-establishing them later. That's a lot cheaper than retail
validity checks, even without the memory-consumption angle. Ideally
that sort of behavior would be automated, but nobody's gotten that
done yet. (pg_restore does do it like that during a full restore,
but not for a data-only restore, so I guess you were doing the latter.)
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: tgl@sss.pgh.pa.us, marc4@rechte.fr, pgsql-performance@lists.postgresql.org
Subject: Re: OOM killer while pg_restore
In-Reply-To: <3305985.1646333012@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox