Subject: pgsql: Allow using syncfs() in frontend utilities.

From: Nathan Bossart <[email-masked]>
Date: Wed, 06 Sep 2023 23:28:00 +0000

Allow using syncfs() in frontend utilities.

This commit allows specifying a --sync-method in several frontend
utilities that must synchronize many files to disk (initdb,
pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade).
On Linux, users can specify "syncfs" to synchronize the relevant
file systems instead of calling fsync() for every single file.  In
many cases, using syncfs() is much faster.

As with recovery_init_sync_method, this new option comes with some
caveats.  The descriptions of these caveats have been moved to a
new appendix section in the documentation.

Co-authored-by: Justin Pryzby
Reviewed-by: Michael Paquier, Thomas Munro, Robert Haas, Justin Pryzby
Discussion: https://postgr.es/m/20210930004340.GM831%40telsasoft.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/8c16ad3b43299695f203f9157a2b27c22b9ed634

Modified Files
--------------
doc/src/sgml/config.sgml              | 12 +++---------
doc/src/sgml/filelist.sgml            |  1 +
doc/src/sgml/postgres.sgml            |  1 +
doc/src/sgml/ref/initdb.sgml          | 22 +++++++++++++++++++++
doc/src/sgml/ref/pg_basebackup.sgml   | 25 ++++++++++++++++++++++++
doc/src/sgml/ref/pg_checksums.sgml    | 22 +++++++++++++++++++++
doc/src/sgml/ref/pg_dump.sgml         | 21 ++++++++++++++++++++
doc/src/sgml/ref/pg_rewind.sgml       | 22 +++++++++++++++++++++
doc/src/sgml/ref/pgupgrade.sgml       | 23 ++++++++++++++++++++++
doc/src/sgml/syncfs.sgml              | 36 +++++++++++++++++++++++++++++++++++
src/bin/initdb/initdb.c               |  6 ++++++
src/bin/initdb/t/001_initdb.pl        | 12 ++++++++++++
src/bin/pg_basebackup/pg_basebackup.c |  7 +++++++
src/bin/pg_checksums/pg_checksums.c   |  6 ++++++
src/bin/pg_dump/pg_dump.c             |  7 +++++++
src/bin/pg_rewind/pg_rewind.c         |  8 ++++++++
src/bin/pg_upgrade/option.c           | 13 +++++++++++++
src/bin/pg_upgrade/pg_upgrade.c       |  6 ++++--
src/bin/pg_upgrade/pg_upgrade.h       |  1 +
src/fe_utils/option_utils.c           | 27 ++++++++++++++++++++++++++
src/include/fe_utils/option_utils.h   |  4 ++++
21 files changed, 271 insertions(+), 11 deletions(-)

---

From: Robert Haas <[email-masked]>
Date: Fri, 22 Mar 2024 12:52:15 -0400

> new appendix section in the documentation.

Hi,

I'd like to complain about this commit's addition of a new appendix. I
do understand the temptation to document caveats like this centrally
instead of in multiple places, but as I've been complaining about over
in the "documentation structure" thread, our top-level documentation
index is too big, and I feel strongly that we need to de-clutter it
rather than cluttering it further. This added a new chapter which is
just 5 sentences long. I understand that this was done because the
same issue applies to a bunch of different utilities and we didn't
want to duplicate this text in all of those places, but I feel like
this approach just doesn't scale. If we did this in every place where
we have this much text that we want to avoid duplicating, we'd soon
have hundreds of appendixes.

What I would suggest we do instead is pick one of the places where
this comes up and document it there, perhaps the
recovery_init_sync_method GUC. And then make the documentation for the
other say something like, you know those issues we documented for
recovery_init_sync_method? Well they also apply to this.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

---

From: Peter Eisentraut <[email-masked]>
Date: Tue, 26 Mar 2024 11:18:57 +0100

On 22.03.24 17:52, Robert Haas wrote:
> I'd like to complain about this commit's addition of a new appendix.

I already complained about that at 
<https://www.postgresql.org/message-id/[email-masked]> 
and some follow-up was announced but didn't happen.  It was on my list 
to look into cleaning up during beta.

---

From: Nathan Bossart <[email-masked]>
Date: Tue, 26 Mar 2024 09:52:10 -0500

> have hundreds of appendixes.

Sorry I missed this.  I explored a couple of options last year but the
discussion trailed off [0].

> recovery_init_sync_method? Well they also apply to this.

WFM.  I'll put together a patch.

[0] https://postgr.es/m/20231009204823.GA659480%40nathanxps13

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

---

From: Nathan Bossart <[email-masked]>
Date: Tue, 26 Mar 2024 10:11:31 -0500

On Tue, Mar 26, 2024 at 11:18:57AM +0100, Peter Eisentraut wrote:
> I already complained about that at <https://www.postgresql.org/message-id/[email-masked]>
> and some follow-up was announced but didn't happen.  It was on my list to
> look into cleaning up during beta.

Sorry about this, I lost track of it.

---

From: Nathan Bossart <[email-masked]>
Date: Tue, 26 Mar 2024 11:34:49 -0500

> Sorry about this, I lost track of it.

Here's a first attempt at a patch based on Robert's suggestion from
upthread.

---

From: Robert Haas <[email-masked]>
Date: Wed, 27 Mar 2024 10:41:42 -0400

> Here's a first attempt at a patch based on Robert's suggestion from

WFM.

---

From: Nathan Bossart <[email-masked]>
Date: Wed, 27 Mar 2024 10:25:16 -0500

>> Here's a first attempt at a patch based on Robert's suggestion from

Committed.  Again, I apologize this took so long.

---

From: Robert Haas <[email-masked]>
Date: Wed, 27 Mar 2024 11:27:46 -0400

> Committed.  Again, I apologize this took so long.

No worries from my side; I only noticed recently. I guess Peter's been
waiting a while, though. Thanks for committing.