Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r5stE-00Ffkv-0g for pgsql-general@arkaria.postgresql.org; Wed, 22 Nov 2023 19:28:00 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1r5stC-00C9kZ-H8 for pgsql-general@arkaria.postgresql.org; Wed, 22 Nov 2023 19:27:58 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r5stC-00C9kP-6U for pgsql-general@lists.postgresql.org; Wed, 22 Nov 2023 19:27:58 +0000 Received: from sss.pgh.pa.us ([68.162.161.243]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r5stA-007pNk-0D for pgsql-general@lists.postgresql.org; Wed, 22 Nov 2023 19:27:57 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 3AMJRqcv2298672; Wed, 22 Nov 2023 14:27:53 -0500 From: Tom Lane To: "Efrain J. Berdecia" cc: "pgsql-general@lists.postgresql.org" , Adrian Klaver Subject: Re: pg_restore enhancements In-reply-to: <383739960.7061560.1700672154915@mail.yahoo.com> References: <362032628.6955904.1700659540303.ref@mail.yahoo.com> <362032628.6955904.1700659540303@mail.yahoo.com> <383739960.7061560.1700672154915@mail.yahoo.com> Comments: In-reply-to "Efrain J. Berdecia" message dated "Wed, 22 Nov 2023 16:55:54 +0000" MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-ID: <2298670.1700681272.1@sss.pgh.pa.us> Content-Transfer-Encoding: quoted-printable Date: Wed, 22 Nov 2023 14:27:52 -0500 Message-ID: <2298671.1700681272@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk "Efrain J. Berdecia" writes: > Thanks, the issue we've run into, which I guess could be really a setup = issue, with running a COPY command while executing pg_restore,=C2=A0is tha= t if we are restoring a large table (bigger than 500GB) our WAL directory = can grow to be very large. > I would think that if the pg_restore or COPY command was able to support= a batch-size option,=C2=A0this should allow postgres to either archive or= remove wal files and prevent having to re-size the WAL directory for a on= e time refresh operation. > I'm trying to gage how feasible would be to start looking at contributin= g to add such a feature to either the COPY command or pg_restore. Given the shortage of other complaints, I tend to agree with Adrian that there's not likely to be much interest in adding complexity to pg_restore (or COPY) to address this. You should probably look harder at the idea that you have some configuration problem that's triggering your WAL bloat. If COPY can run you out of WAL space, then so could any future bulk insert or update. regards, tom lane