From: Stephen Frost <sfrost@snowman.net>
To: Dipesh Pandit <dipesh.pandit@gmail.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: Hannu Krosing <hannuk@google.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: .ready and .done files considered harmful
Date: Tue, 6 Jul 2021 09:34:58 -0400
Message-ID: <20210706133458.GE20766@tamriel.snowman.net> (raw)
In-Reply-To: <CAN1g5_GxqKZ3MnHjBpHyBYSWe9tQRCx2c+TSCnJ3Ea6v8UZhag@mail.gmail.com>
References: <CA+TgmobhAbs2yabTuTRkJTq_kkC80-+jw=pfpypdOJ7+gAbQbw@mail.gmail.com>
<20210504042755.ehuaoz5blcnjw5yk@alap3.anarazel.de>
<CA+TgmobBoMqU_fwOD48e-JY69y87SKG6=POR0FODw03soaJOUA@mail.gmail.com>
<20210506.162302.1971507615008478616.horikyota.ntt@gmail.com>
<CA+TgmoaJ+afFAisrUXKaOgEEH4nGUCgb3EzWFnx7Tx-A2YgNOw@mail.gmail.com>
<CAMT0RQR0Op5uSgunWpcTRjge5sjXL1nGY=xLG8PxFn4Smy4gcw@mail.gmail.com>
<20210506200139.ztcbtymc2mf7hncn@alap3.anarazel.de>
<CAN1g5_GxqKZ3MnHjBpHyBYSWe9tQRCx2c+TSCnJ3Ea6v8UZhag@mail.gmail.com>
Greetings,
* Dipesh Pandit (dipesh.pandit@gmail.com) wrote:
> We have addressed the O(n^2) problem which involves directory scan for
> archiving individual WAL files by maintaining a WAL counter to identify
> the next WAL file in a sequence.
This seems to have missed the concerns raised in
https://postgr.es/m/20210505170601.GF20766@tamriel.snowman.net ..?
And also the comments immediately above the ones being added here:
> @@ -596,29 +606,55 @@ pgarch_archiveXlog(char *xlog)
> * larger ID; the net result being that past timelines are given higher
> * priority for archiving. This seems okay, or at least not obviously worth
> * changing.
> + *
> + * WAL files are generated in a specific order of log segment number. The
> + * directory scan for each WAL file can be minimized by identifying the next
> + * WAL file in the sequence. This can be achieved by maintaining log segment
> + * number and timeline ID corresponding to WAL file currently being archived.
> + * The log segment number of current WAL file can be incremented by '1' upon
> + * successful archival to point to the next WAL file.
specifically about history files being given higher priority for
archiving. If we go with this change then we'd at least want to rewrite
or remove those comments, but I don't actually agree that we should
remove that preference to archive history files ahead of WAL, for the
reasons brought up previously.
As was suggested on that subthread, it seems like it should be possible
to just track the current timeline and adjust what we're doing if the
timeline changes, and we should even know what the .history file is at
that point and likely don't even need to scan the directory for it, as
it'll be the old timeline ID.
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (818B, ../20210706133458.GE20766@tamriel.snowman.net/2-signature.asc)
download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: sfrost@snowman.net, dipesh.pandit@gmail.com, andres@anarazel.de, hannuk@google.com, robertmhaas@gmail.com, horikyota.ntt@gmail.com
Subject: Re: .ready and .done files considered harmful
In-Reply-To: <20210706133458.GE20766@tamriel.snowman.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox