pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Nathan Bossart <nathan@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Convert pg_restore's ready_list to a priority queue.
Date: Tue, 19 Sep 2023 21:32:46 +0000
Message-ID: <E1qiiKr-004pij-NS@gemulon.postgresql.org> (raw)

Convert pg_restore's ready_list to a priority queue.

Presently, parallel restores spend a lot of time sorting this list
so that we pick the largest items first.  With many tables, this
sorting can become a significant bottleneck.  There are a couple of
reports from the field about this, and it is easy to reproduce.

This commit improves the performance of parallel pg_restore with
many tables by converting its ready_list to a priority queue, i.e.,
a binary heap.  We will first try to run the highest priority item,
but if it cannot be chosen due to the lock heuristic, we'll do a
sequential scan through the heap nodes until we find one that is
runnable.  This means that we might end up picking an item with a
much lower priority.  However, we expect that we will typically be
able to pick one of the first few items, which should usually have
a relatively high priority.

Suggested-by: Tom Lane
Tested-by: Pierre Ducroquet
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/3612876.1689443232%40sss.pgh.pa.us

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/9bfd44bbde4261181bf94738f3b041c629c65a7e

Modified Files
--------------
src/bin/pg_dump/pg_backup_archiver.c | 198 ++++++++++-------------------------
1 file changed, 58 insertions(+), 140 deletions(-)



Message-ID: <E1qiiKr-004pij-NS@gemulon.postgresql.org>
Permalink:  ../E1qiiKr-004pij-NS@gemulon.postgresql.org/
Also on:    postgresql.org/message-id/E1qiiKr-004pij-NS@gemulon.postgresql.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-committers@postgresql.org
  Cc: nathan@postgresql.org, pgsql-committers@lists.postgresql.org
  Subject: Re: pgsql: Convert pg_restore's ready_list to a priority queue.
  In-Reply-To: <E1qiiKr-004pij-NS@gemulon.postgresql.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox