Date: Tue, 6 Jul 2021 09:34:58 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Dipesh Pandit <dipesh.pandit@gmail.com>
Cc: Andres Freund <andres@anarazel.de>, Hannu Krosing <hannuk@google.com>,
	Robert Haas <robertmhaas@gmail.com>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: .ready and .done files considered harmful
Message-ID: <20210706133458.GE20766@tamriel.snowman.net>
References: 
 <CA+TgmobhAbs2yabTuTRkJTq_kkC80-+jw=pfpypdOJ7+gAbQbw@mail.gmail.com>
 <20210504042755.ehuaoz5blcnjw5yk@alap3.anarazel.de>
 <CA+TgmobBoMqU_fwOD48e-JY69y87SKG6=POR0FODw03soaJOUA@mail.gmail.com>
 <20210506.162302.1971507615008478616.horikyota.ntt@gmail.com>
 <CA+TgmoaJ+afFAisrUXKaOgEEH4nGUCgb3EzWFnx7Tx-A2YgNOw@mail.gmail.com>
 <CAMT0RQR0Op5uSgunWpcTRjge5sjXL1nGY=xLG8PxFn4Smy4gcw@mail.gmail.com>
 <20210506200139.ztcbtymc2mf7hncn@alap3.anarazel.de>
 <CAN1g5_GxqKZ3MnHjBpHyBYSWe9tQRCx2c+TSCnJ3Ea6v8UZhag@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="vh89e/HiiVjD0qgf"
Content-Disposition: inline
In-Reply-To: 
 <CAN1g5_GxqKZ3MnHjBpHyBYSWe9tQRCx2c+TSCnJ3Ea6v8UZhag@mail.gmail.com>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: 
 <https://www.postgresql.org/message-id/20210706133458.GE20766%40tamriel.snowman.net>
Precedence: bulk


--vh89e/HiiVjD0qgf
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Greetings,

* Dipesh Pandit (dipesh.pandit@gmail.com) wrote:
> We have addressed the O(n^2) problem which involves directory scan for
> archiving individual WAL files by maintaining a WAL counter to identify
> the next WAL file in a sequence.

This seems to have missed the concerns raised in
https://postgr.es/m/20210505170601.GF20766@tamriel.snowman.net ..?

And also the comments immediately above the ones being added here:

> @@ -596,29 +606,55 @@ pgarch_archiveXlog(char *xlog)
>   * larger ID; the net result being that past timelines are given higher
>   * priority for archiving.  This seems okay, or at least not obviously worth
>   * changing.
> + *
> + * WAL files are generated in a specific order of log segment number. The
> + * directory scan for each WAL file can be minimized by identifying the next
> + * WAL file in the sequence. This can be achieved by maintaining log segment
> + * number and timeline ID corresponding to WAL file currently being archived.
> + * The log segment number of current WAL file can be incremented by '1' upon
> + * successful archival to point to the next WAL file.

specifically about history files being given higher priority for
archiving.  If we go with this change then we'd at least want to rewrite
or remove those comments, but I don't actually agree that we should
remove that preference to archive history files ahead of WAL, for the
reasons brought up previously.

As was suggested on that subthread, it seems like it should be possible
to just track the current timeline and adjust what we're doing if the
timeline changes, and we should even know what the .history file is at
that point and likely don't even need to scan the directory for it, as
it'll be the old timeline ID.

Thanks,

Stephen

--vh89e/HiiVjD0qgf
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJg5FwCAAoJEO1sijiDR2RVV3YP/26Sk53KxOIluOxDQDnpbd5I
dcZS8J5MXE2vKUg403XnOvKSv+Xe9T1MUFhnOpx6Jw3L0SAn1hcrDYNHDQpjaxPP
TUrinXkxP8teHRxMsrW4a+A6gHT7Anj34Acx/nsQnlbB8eFUwTmlja7Dh1qaMKWs
R6p18vY6nMuLgGPr6UWKpBs5E/MRtI9Tz/MteshAOWv/s6XEyc1H5oIJPUGRfdAZ
WHCes2hyiqvaGhjKD5XJwu3G4R+iwHU9WRp64XbJVmO4jxHrEvbFPK9VRCLVrg94
sEmCKApphdGiaFOKysgD5xy9AA0XqyOHlYzbiWOL9FgfmWGdD3bmB63kwSm7kges
hBGte2/G6EUAL1ZUyVpTNgbGULFmiDbs2pi7Fs+dkfNNMXAZxVYOdIO6gs91ClNf
npR9uD6dEBr+Gtb7S1RPNLYCGuy5W/yZ2qDc3Nf6z3pFPlGXK+TVTdnq08SXCFEg
zNJPxsM/NT33XGXqBfDqx7vSCAccbPJXIrGCx8Z73ZINukUBFzYEPX6PMqo2WgS4
bUstB5ucf/hZ29CjvymtCn5PmMHEboSDcuI5dCedp0zUZTYkWq3ss5xQFruM3ZOJ
eMezgRuoNTTfuRJyk3hUo6WvZ69rhclNwRRnyaLfnFRP5I1RPnPypAIDlUeosjE0
XfZQiE6xfj4f5j6+3Gpr
=aMQk
-----END PGP SIGNATURE-----

--vh89e/HiiVjD0qgf--