From: Tom Lane <tgl@sss.pgh.pa.us>
To: Boris Sagadin <boris@infosplet.com>
Cc: pgsql-generallists.postgresql.org <pgsql-general@lists.postgresql.org>
Subject: Re: pg_restore mostly idle on restoring a large number of tables
Date: Sat, 15 Jul 2023 13:56:13 -0400
Message-ID: <3614137.1689443773@sss.pgh.pa.us> (raw)
In-Reply-To: <CAEzn=HSPXi6OS-5KzGMcZeKzWKOOX1me2u2eCiGtMEZDz9Fqdg@mail.gmail.com>
References: <CAEzn=HSPXi6OS-5KzGMcZeKzWKOOX1me2u2eCiGtMEZDz9Fqdg@mail.gmail.com>
Boris Sagadin <boris@infosplet.com> writes:
> restoring a 1.5TB database with about 800k tables on i3.4xlarge AWS
> instace, PgSQL V12.15 on Ubuntu.
> Running pg_restore with -j 16, I noticed the pg_restore is busy for an hour
> or so with IO at 80%+ and then most of processes start idling and only a
> few doing some work, disk IO at 1-2%, pg_stat_activity is mostly idle, same
> goes for CPU, and this state proceeds for further 6 hours, disk space
> increases very slowly.
Yeah, this seems like a performance bug in pg_restore, per analysis at
[1]. Until somebody gets around to doing something about that, I'd
counsel not bothering with parallelized restore for this many tables.
If the tables are mostly small, it'd likely be more useful to use
--single-transaction mode; though you'd have to make sure that
max_locks_per_transaction is set high enough to allow creating all
those tables in one transaction.
regards, tom lane
[1] https://www.postgresql.org/message-id/3612876.1689443232%40sss.pgh.pa.us
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: tgl@sss.pgh.pa.us, boris@infosplet.com
Subject: Re: pg_restore mostly idle on restoring a large number of tables
In-Reply-To: <3614137.1689443773@sss.pgh.pa.us>
References: <CAEzn=HSPXi6OS-5KzGMcZeKzWKOOX1me2u2eCiGtMEZDz9Fqdg@mail.gmail.com> <3614137.1689443773@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox