From: Bossart, Nathan <bossartn@amazon.com>
To: Dipesh Pandit <dipesh.pandit@gmail.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Andres Freund <andres@anarazel.de>
Cc: Hannu Krosing <hannuk@google.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Subject: Re: .ready and .done files considered harmful
Date: Mon, 16 Aug 2021 04:51:23 +0000
Message-ID: <CF7D0ECB-EEFF-4E76-8963-8A3648AD9052@amazon.com> (raw)
In-Reply-To: <CAN1g5_EZB7W=+koas6yvG4huNy-bkWau9YXHQEEFN_6Y9D0igg@mail.gmail.com>
References: <32FCEE06-C094-4B3F-BA6A-CCAF90FB90E2@amazon.com>
<20210806.102606.1567426789036874091.horikyota.ntt@gmail.com>
<2E74AA08-9BA1-495A-B782-3D7A1B6DDBA0@amazon.com>
<20210806.133958.962850705445445360.horikyota.ntt@gmail.com>
<CAN1g5_EZB7W=+koas6yvG4huNy-bkWau9YXHQEEFN_6Y9D0igg@mail.gmail.com>
+ * This .ready file is created out of order, notify archiver to perform
+ * a full directory scan to archive corresponding WAL file.
+ */
+ StatusFilePath(archiveStatusPath, xlog, ".ready");
+ if (stat(archiveStatusPath, &stat_buf) == 0)
+ PgArchEnableDirScan();
We may want to call PgArchWakeup() after setting the flag.
+ * Perform a full directory scan to identify the next log segment. There
+ * may be one of the following scenarios which may require us to perform a
+ * full directory scan.
...
+ * - The next anticipated log segment is not available.
I wonder if we really need to perform a directory scan in this case.
Unless there are other cases where the .ready files are created out of
order, I think this just causes an unnecessary directory scan every
time the archiver catches up.
+ * Flag to enable/disable directory scan. If this flag is set then it
+ * forces archiver to perform a full directory scan to get the next log
+ * segment.
+ */
+ pg_atomic_flag dirScan;
I personally don't think it's necessary to use an atomic here. A
spinlock or LWLock would probably work just fine, as contention seems
unlikely. If we use a lock, we also don't have to worry about memory
barriers.
Nathan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: bossartn@amazon.com, dipesh.pandit@gmail.com, horikyota.ntt@gmail.com, robertmhaas@gmail.com, jeevan.ladhe@enterprisedb.com, sfrost@snowman.net, andres@anarazel.de, hannuk@google.com
Subject: Re: .ready and .done files considered harmful
In-Reply-To: <CF7D0ECB-EEFF-4E76-8963-8A3648AD9052@amazon.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox