From: Bossart, Nathan <bossartn@amazon.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Dipesh Pandit <dipesh.pandit@gmail.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Andres Freund <andres@anarazel.de>
Cc: Hannu Krosing <hannuk@google.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: .ready and .done files considered harmful
Date: Mon, 23 Aug 2021 15:50:29 +0000
Message-ID: <08C406B6-1672-475B-AE03-1ABD6C9F7984@amazon.com> (raw)
In-Reply-To: <CA+TgmoYqNF1A56TPAFY0pCxoM429gjqB0OnXs+swfubmNk+1CA@mail.gmail.com>
References: <32FCEE06-C094-4B3F-BA6A-CCAF90FB90E2@amazon.com>
<20210806.102606.1567426789036874091.horikyota.ntt@gmail.com>
<2E74AA08-9BA1-495A-B782-3D7A1B6DDBA0@amazon.com>
<20210806.133958.962850705445445360.horikyota.ntt@gmail.com>
<CAN1g5_EZB7W=+koas6yvG4huNy-bkWau9YXHQEEFN_6Y9D0igg@mail.gmail.com>
<05564140-9567-49FA-AFB5-252F5868BBCA@amazon.com>
<CAN1g5_ECegnUtHM-fg66UN=f1qFXJ_rZyrC+QoGcURJMOB-+Rw@mail.gmail.com>
<273A6E6D-DCC1-4719-96D2-609D5ADA3EA7@amazon.com>
<CA+TgmoaOD+CcEeXmOo2+vFLL0K7jc9=HHF0t=8nJqRaiRJndSw@mail.gmail.com>
<D68A2BF6-3BCE-4122-9CA8-40486812AFAE@amazon.com>
<CA+TgmoY4Tu_a=4hTX88BrWCP-OdnasAxkk2Em3c_gTLqhUy9pg@mail.gmail.com>
<27F8BC15-6242-44C8-9818-5D02A3BF8F5C@amazon.com>
<CAN1g5_EgkiSAd2Hh_T_bC9Z+GmdQTEhR=eb=edh8aKAiLDnysQ@mail.gmail.com>
<3ECC212F-88FD-4FB2-BAF1-C2DD1563E310@amazon.com>
<CA+TgmoYqNF1A56TPAFY0pCxoM429gjqB0OnXs+swfubmNk+1CA@mail.gmail.com>
On 8/23/21, 6:42 AM, "Robert Haas" <robertmhaas@gmail.com> wrote:
> On Sun, Aug 22, 2021 at 10:31 PM Bossart, Nathan <bossartn@amazon.com> wrote:
>> I ran this again on a bigger machine with 200K WAL files pending
>> archive. The v9 patch took ~5.5 minutes, the patch I sent took ~8
>> minutes, and the existing logic took just under 3 hours.
>
> Hmm. On the one hand, 8 minutes > 5.5 minutes, and presumably the gap
> would only get wider if the number of files were larger or if reading
> the directory were slower. I am pretty sure that reading the directory
> must be much slower in some real deployments where this problem has
> come up. On the other hand, 8.8 minutes << 3 hours, and your patch
> would win if somehow we had a ton of gaps in the sequence of files.
> I'm not sure how likely that is to be the cause - probably not very
> likely at all if you aren't using an archive command that cheats, but
> maybe really common if you are. Hmm, but I think if the
> archive_command cheats by marking a bunch of files done when it is
> tasked with archiving just one, your patch will break, because, unless
> I'm missing something, it doesn't re-evaluate whether things have
> changed on every pass through the loop as Dipesh's patch does. So I
> guess I'm not quite sure I understand why you think this might be the
> way to go?
To handle a "cheating" archive command, I'd probably need to add a
stat() for every time pgarch_readyXLog() returned something from
arch_files. I suspect something similar might be needed in Dipesh's
patch to handle backup history files and partial WAL files.
In any case, I think Dipesh's patch is the way to go. It obviously
will perform better in the extreme cases discussed in this thread. I
think it's important to make sure the patch doesn't potentially leave
files behind to be picked up by a directory scan that might not
happen, but there are likely ways to handle that. In the worst case,
perhaps we need to force a directory scan every N files to make sure
nothing gets left behind. But maybe we can do better.
> Maintaining the binary heap in lowest-priority-first order is very
> clever, and the patch does look quite elegant. I'm just not sure I
> understand the point.
This was mostly an exploratory exercise to get some numbers for the
different approaches discussed in this thread.
Nathan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bossartn@amazon.com, robertmhaas@gmail.com, dipesh.pandit@gmail.com, horikyota.ntt@gmail.com, jeevan.ladhe@enterprisedb.com, sfrost@snowman.net, andres@anarazel.de, hannuk@google.com
Subject: Re: .ready and .done files considered harmful
In-Reply-To: <08C406B6-1672-475B-AE03-1ABD6C9F7984@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox