pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Bossart, Nathan <bossartn@amazon.com>
To: Dipesh Pandit <dipesh.pandit@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Andres Freund <andres@anarazel.de>
Cc: Hannu Krosing <hannuk@google.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: .ready and .done files considered harmful
Date: Mon, 23 Aug 2021 02:31:45 +0000
Message-ID: <3ECC212F-88FD-4FB2-BAF1-C2DD1563E310@amazon.com> (raw)
In-Reply-To: <CAN1g5_EgkiSAd2Hh_T_bC9Z+GmdQTEhR=eb=edh8aKAiLDnysQ@mail.gmail.com>
References: <32FCEE06-C094-4B3F-BA6A-CCAF90FB90E2@amazon.com>
	<20210806.102606.1567426789036874091.horikyota.ntt@gmail.com>
	<2E74AA08-9BA1-495A-B782-3D7A1B6DDBA0@amazon.com>
	<20210806.133958.962850705445445360.horikyota.ntt@gmail.com>
	<CAN1g5_EZB7W=+koas6yvG4huNy-bkWau9YXHQEEFN_6Y9D0igg@mail.gmail.com>
	<05564140-9567-49FA-AFB5-252F5868BBCA@amazon.com>
	<CAN1g5_ECegnUtHM-fg66UN=f1qFXJ_rZyrC+QoGcURJMOB-+Rw@mail.gmail.com>
	<273A6E6D-DCC1-4719-96D2-609D5ADA3EA7@amazon.com>
	<CA+TgmoaOD+CcEeXmOo2+vFLL0K7jc9=HHF0t=8nJqRaiRJndSw@mail.gmail.com>
	<D68A2BF6-3BCE-4122-9CA8-40486812AFAE@amazon.com>
	<CA+TgmoY4Tu_a=4hTX88BrWCP-OdnasAxkk2Em3c_gTLqhUy9pg@mail.gmail.com>
	<27F8BC15-6242-44C8-9818-5D02A3BF8F5C@amazon.com>
	<CAN1g5_EgkiSAd2Hh_T_bC9Z+GmdQTEhR=eb=edh8aKAiLDnysQ@mail.gmail.com>

On 8/21/21, 9:29 PM, "Bossart, Nathan" <bossartn@amazon.com> wrote:
> I was curious about this, so I wrote a patch (attached) to store
> multiple files per directory scan and tested it against the latest
> patch in this thread (v9) [0].  Specifically, I set archive_command to
> 'false', created ~20K WAL segments, then restarted the server with
> archive_command set to 'true'.  Both the v9 patch and the attached
> patch completed archiving all segments in just under a minute.  (I
> tested the attached patch with NUM_FILES_PER_DIRECTORY_SCAN set to 64,
> 128, and 256 and didn't observe any significant difference.)  The
> existing logic took over 4 minutes to complete.
>
> I'm hoping to do this test again with many more (100K+) status files,
> as I believe that the v9 patch will be faster at that scale, but I'm
> not sure how much faster it will be.

I ran this again on a bigger machine with 200K WAL files pending
archive.  The v9 patch took ~5.5 minutes, the patch I sent took ~8
minutes, and the existing logic took just under 3 hours.

Nathan



view thread (118+ messages)  latest in thread

Message-ID: <3ECC212F-88FD-4FB2-BAF1-C2DD1563E310@amazon.com>
Permalink:  ../3ECC212F-88FD-4FB2-BAF1-C2DD1563E310@amazon.com/
Also on:    postgresql.org/message-id/3ECC212F-88FD-4FB2-BAF1-C2DD1563E310@amazon.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: bossartn@amazon.com, dipesh.pandit@gmail.com, robertmhaas@gmail.com, horikyota.ntt@gmail.com, jeevan.ladhe@enterprisedb.com, sfrost@snowman.net, andres@anarazel.de, hannuk@google.com
  Subject: Re: .ready and .done files considered harmful
  In-Reply-To: <3ECC212F-88FD-4FB2-BAF1-C2DD1563E310@amazon.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox