pg.ddx.io pgsql-general@postgresql.org mailing list archive
help / color / mirror / Atom feedpg_restore mostly idle on restoring a large number of tables
3+ messages / 3 participants
[nested] [flat]
* pg_restore mostly idle on restoring a large number of tables
@ 2023-07-13 07:41 Boris Sagadin <boris@infosplet.com>
0 siblings, 2 replies; 3+ messages in thread
From: Boris Sagadin @ 2023-07-13 07:41 UTC (permalink / raw)
To: pgsql-generallists.postgresql.org <pgsql-general@lists.postgresql.org>
Hi,
restoring a 1.5TB database with about 800k tables on i3.4xlarge AWS
instace, PgSQL V12.15 on Ubuntu.
Running pg_restore with -j 16, I noticed the pg_restore is busy for an hour
or so with IO at 80%+ and then most of processes start idling and only a
few doing some work, disk IO at 1-2%, pg_stat_activity is mostly idle, same
goes for CPU, and this state proceeds for further 6 hours, disk space
increases very slowly.
I thought because of a lot of small tables, number of workers should be
increased to increase parallel efficiency, so I tried with -j 128. The
situation was somewhat better, but most of the workers start idling, again
disk IO lowers to about 4% util, CPU util goes to about 4%, too.
Stracing workers produces the perpetual read call on most pg_restore
workers:
# strace -p 59567
strace: Process 59567 attached
read(3,
With only about 10 or so (out of 128) workers doing some actual work:
strace -p 59367 -e sendto
strace: Process 59367 attached
sendto(4, "Q\0\0\0\263ALTER TABLE ONLY raw.space0"..., 180, MSG_NOSIGNAL,
NULL, 0) = 180
sendto(4, "Q\0\0\0.TRUNCATE TABLE ONLY raw.spa"..., 47, MSG_NOSIGNAL, NULL,
0) = 47
sendto(4, "Q\0\0\0\205COPY raw.space000448117 (da"..., 134, MSG_NOSIGNAL,
NULL, 0) = sendto(4, "Q\0\0\0\vCOMMIT\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
sendto(4, "Q\0\0\0pCREATE INDEX idx_space00044"..., 113, MSG_NOSIGNAL,
NULL, 0) = 113
sendto(4, "Q\0\0\0\263ALTER TABLE ONLY raw.space0"..., 180, MSG_NOSIGNAL,
NULL, 0) = 180
...
.. some lines omitted
I would think that all workers would proceed with creating indexes and
doing some useful work until the restore is finished completely?
Most of the tables are very small, 2 indexes per table and without any
foreign references etc., we have a multi tenant environment.
Thanks,
Boris
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: pg_restore mostly idle on restoring a large number of tables
@ 2023-07-15 14:51 Ron <ronljohnsonjr@gmail.com>
parent: Boris Sagadin <boris@infosplet.com>
1 sibling, 0 replies; 3+ messages in thread
From: Ron @ 2023-07-15 14:51 UTC (permalink / raw)
To: pgsql-general@lists.postgresql.org
On 7/13/23 02:41, Boris Sagadin wrote:
> Hi,
>
> restoring a 1.5TB database with about 800k tables on i3.4xlarge AWS
> instace, PgSQL V12.15 on Ubuntu.
>
> Running pg_restore with -j 16, I noticed the pg_restore is busy for an
> hour or so with IO at 80%+ and then most of processes start idling and
> only a few doing some work, disk IO at 1-2%, pg_stat_activity is mostly
> idle, same goes for CPU, and this state proceeds for further 6 hours, disk
> space increases very slowly.
>
> I thought because of a lot of small tables, number of workers should be
> increased to increase parallel efficiency, so I tried with -j 128. The
> situation was somewhat better, but most of the workers start idling, again
> disk IO lowers to about 4% util, CPU util goes to about 4%, too.
>
> Stracing workers produces the perpetual read call on most pg_restore workers:
>
> # strace -p 59567
> strace: Process 59567 attached
> read(3,
>
> With only about 10 or so (out of 128) workers doing some actual work:
>
> strace -p 59367 -e sendto
> strace: Process 59367 attached
> sendto(4, "Q\0\0\0\263ALTER TABLE ONLY raw.space0"..., 180, MSG_NOSIGNAL,
> NULL, 0) = 180
> sendto(4, "Q\0\0\0.TRUNCATE TABLE ONLY raw.spa"..., 47, MSG_NOSIGNAL,
> NULL, 0) = 47
> sendto(4, "Q\0\0\0\205COPY raw.space000448117 (da"..., 134, MSG_NOSIGNAL,
> NULL, 0) = sendto(4, "Q\0\0\0\vCOMMIT\0", 12, MSG_NOSIGNAL, NULL, 0) = 12
> sendto(4, "Q\0\0\0pCREATE INDEX idx_space00044"..., 113, MSG_NOSIGNAL,
> NULL, 0) = 113
> sendto(4, "Q\0\0\0\263ALTER TABLE ONLY raw.space0"..., 180, MSG_NOSIGNAL,
> NULL, 0) = 180
> ...
> .. some lines omitted
>
> I would think that all workers would proceed with creating indexes and
> doing some useful work until the restore is finished completely?
>
> Most of the tables are very small, 2 indexes per table and without any
> foreign references etc., we have a multi tenant environment.
I always run pg_dump and pg_restore with "--verbose", and redirect stdout &
stderr to a log file. Then I "tail -f" that log file. It'll tell you what's
happening.
Also, iotop is quite useful.
--
Born in Arizona, moved to Babylonia.
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: pg_restore mostly idle on restoring a large number of tables
@ 2023-07-15 17:56 Tom Lane <tgl@sss.pgh.pa.us>
parent: Boris Sagadin <boris@infosplet.com>
1 sibling, 0 replies; 3+ messages in thread
From: Tom Lane @ 2023-07-15 17:56 UTC (permalink / raw)
To: Boris Sagadin <boris@infosplet.com>; +Cc: pgsql-generallists.postgresql.org <pgsql-general@lists.postgresql.org>
Boris Sagadin <boris@infosplet.com> writes:
> restoring a 1.5TB database with about 800k tables on i3.4xlarge AWS
> instace, PgSQL V12.15 on Ubuntu.
> Running pg_restore with -j 16, I noticed the pg_restore is busy for an hour
> or so with IO at 80%+ and then most of processes start idling and only a
> few doing some work, disk IO at 1-2%, pg_stat_activity is mostly idle, same
> goes for CPU, and this state proceeds for further 6 hours, disk space
> increases very slowly.
Yeah, this seems like a performance bug in pg_restore, per analysis at
[1]. Until somebody gets around to doing something about that, I'd
counsel not bothering with parallelized restore for this many tables.
If the tables are mostly small, it'd likely be more useful to use
--single-transaction mode; though you'd have to make sure that
max_locks_per_transaction is set high enough to allow creating all
those tables in one transaction.
regards, tom lane
[1] https://www.postgresql.org/message-id/3612876.1689443232%40sss.pgh.pa.us
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2023-07-15 17:56 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-07-13 07:41 pg_restore mostly idle on restoring a large number of tables Boris Sagadin <boris@infosplet.com>
2023-07-15 14:51 ` Ron <ronljohnsonjr@gmail.com>
2023-07-15 17:56 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox