pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedFrom: Bossart, Nathan <bossartn@amazon.com>
To: Robert Haas <robertmhaas@gmail.com>
To: Dipesh Pandit <dipesh.pandit@gmail.com>
Cc: Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Andres Freund <andres@anarazel.de>
Cc: Hannu Krosing <hannuk@google.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: .ready and .done files considered harmful
Date: Tue, 3 Aug 2021 20:46:57 +0000
Message-ID: <32FCEE06-C094-4B3F-BA6A-CCAF90FB90E2@amazon.com> (raw)
In-Reply-To: <CA+TgmoYnOVT3yRnmQJc9xsNh1nPWBo8qXW3m_4ZYMz-Xb53Vhg@mail.gmail.com>
References: <CA+TgmobhAbs2yabTuTRkJTq_kkC80-+jw=pfpypdOJ7+gAbQbw@mail.gmail.com>
<20210504042755.ehuaoz5blcnjw5yk@alap3.anarazel.de>
<CA+TgmobBoMqU_fwOD48e-JY69y87SKG6=POR0FODw03soaJOUA@mail.gmail.com>
<20210506.162302.1971507615008478616.horikyota.ntt@gmail.com>
<CA+TgmoaJ+afFAisrUXKaOgEEH4nGUCgb3EzWFnx7Tx-A2YgNOw@mail.gmail.com>
<CAMT0RQR0Op5uSgunWpcTRjge5sjXL1nGY=xLG8PxFn4Smy4gcw@mail.gmail.com>
<20210506200139.ztcbtymc2mf7hncn@alap3.anarazel.de>
<CAN1g5_GxqKZ3MnHjBpHyBYSWe9tQRCx2c+TSCnJ3Ea6v8UZhag@mail.gmail.com>
<20210706133458.GE20766@tamriel.snowman.net>
<CA+Tgmoaj6PtjRSFPQfMWZyN5bD1+bF2pxzwAfgDvwCMDj06A3A@mail.gmail.com>
<CAN1g5_FcxZYg3=uC0ScpoJzxPaO087FEqgr+KXdxy7RK_+Y-Kw@mail.gmail.com>
<CAOgcT0Mggk4KZ8Lz-icYpJWY3c9WC3zgGCxXE_NyNgmJWNPYgQ@mail.gmail.com>
<4CAB59F8-1EC3-4BA4-B97A-DE927D7D694F@amazon.com>
<CA+TgmoZ8WnZjL_uY6AQfKRmbBtHerfgpzkJekEHxRExx=AhDfA@mail.gmail.com>
<FE1E60A6-49ED-4627-804D-FD5D86AD2876@amazon.com>
<CAN1g5_GwAENGJGZ56bL7CBN85MFnH+u5ZuNm=J2wmzMQA4j39Q@mail.gmail.com>
<CA+Tgmoaq5eBFoj--Xn92Mz5H+YomkE5xyv-_ETB1L-P7BjRCDA@mail.gmail.com>
<CAN1g5_GGwn-AbF2pn+=i5T4b2r5ExynXPnnNH0eRpeWDKw-C_Q@mail.gmail.com>
<CA+TgmoZXw4M-scnZoRRsS_304L0Ln6ndT59JpAHn+L2_r6-QBg@mail.gmail.com>
<CAN1g5_GbOgwxtOa=xaKDrXpwVBgxTUdzMWmFYseLFaFmNYUPEA@mail.gmail.com>
<CA+TgmoYnOVT3yRnmQJc9xsNh1nPWBo8qXW3m_4ZYMz-Xb53Vhg@mail.gmail.com>
+ /*
+ * Perform a full directory scan to identify the next log segment. There
+ * may be one of the following scenarios which may require us to perform a
+ * full directory scan.
+ *
+ * 1. This is the first cycle since archiver has started and there is no
+ * idea about the next anticipated log segment.
+ *
+ * 2. There is a timeline switch, i.e. the timeline ID tracked at archiver
+ * does not match with current timeline ID. Archive history file as part of
+ * this timeline switch.
+ *
+ * 3. The next anticipated log segment is not available.
One benefit of the current implementation of pgarch_readyXlog() is
that .ready files created out of order will be prioritized before
segments with greater LSNs. IIUC, with this patch, as long as there
is a "next anticipated" segment available, the archiver won't go back
and archive segments it missed. I don't think the archive status
files are regularly created out of order, but XLogArchiveCheckDone()
has handling for that case, and the work to avoid creating .ready
files too early [0] seems to make it more likely. Perhaps we should
also force a directory scan when we detect that we are creating a
.ready file for a segment that is older than the "next anticipated"
segment.
Nathan
[0] https://postgr.es/m/DA71434B-7340-4984-9B91-F085BC47A778%40amazon.com
view thread (118+ messages) latest in thread
Message-ID: <32FCEE06-C094-4B3F-BA6A-CCAF90FB90E2@amazon.com>
Permalink: ../32FCEE06-C094-4B3F-BA6A-CCAF90FB90E2@amazon.com/
Also on: postgresql.org/message-id/32FCEE06-C094-4B3F-BA6A-CCAF90FB90E2@amazon.com
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bossartn@amazon.com, robertmhaas@gmail.com, dipesh.pandit@gmail.com, jeevan.ladhe@enterprisedb.com, sfrost@snowman.net, andres@anarazel.de, hannuk@google.com, horikyota.ntt@gmail.com
Subject: Re: .ready and .done files considered harmful
In-Reply-To: <32FCEE06-C094-4B3F-BA6A-CCAF90FB90E2@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox