public inbox for [email protected]help / color / mirror / Atom feed
pgsql: Allow using syncfs() in frontend utilities. 9+ messages / 4 participants [nested] [flat]
* pgsql: Allow using syncfs() in frontend utilities. @ 2023-09-06 23:28 Nathan Bossart <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Nathan Bossart @ 2023-09-06 23:28 UTC (permalink / raw) To: [email protected] Allow using syncfs() in frontend utilities. This commit allows specifying a --sync-method in several frontend utilities that must synchronize many files to disk (initdb, pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade). On Linux, users can specify "syncfs" to synchronize the relevant file systems instead of calling fsync() for every single file. In many cases, using syncfs() is much faster. As with recovery_init_sync_method, this new option comes with some caveats. The descriptions of these caveats have been moved to a new appendix section in the documentation. Co-authored-by: Justin Pryzby Reviewed-by: Michael Paquier, Thomas Munro, Robert Haas, Justin Pryzby Discussion: https://postgr.es/m/20210930004340.GM831%40telsasoft.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/8c16ad3b43299695f203f9157a2b27c22b9ed634 Modified Files -------------- doc/src/sgml/config.sgml | 12 +++--------- doc/src/sgml/filelist.sgml | 1 + doc/src/sgml/postgres.sgml | 1 + doc/src/sgml/ref/initdb.sgml | 22 +++++++++++++++++++++ doc/src/sgml/ref/pg_basebackup.sgml | 25 ++++++++++++++++++++++++ doc/src/sgml/ref/pg_checksums.sgml | 22 +++++++++++++++++++++ doc/src/sgml/ref/pg_dump.sgml | 21 ++++++++++++++++++++ doc/src/sgml/ref/pg_rewind.sgml | 22 +++++++++++++++++++++ doc/src/sgml/ref/pgupgrade.sgml | 23 ++++++++++++++++++++++ doc/src/sgml/syncfs.sgml | 36 +++++++++++++++++++++++++++++++++++ src/bin/initdb/initdb.c | 6 ++++++ src/bin/initdb/t/001_initdb.pl | 12 ++++++++++++ src/bin/pg_basebackup/pg_basebackup.c | 7 +++++++ src/bin/pg_checksums/pg_checksums.c | 6 ++++++ src/bin/pg_dump/pg_dump.c | 7 +++++++ src/bin/pg_rewind/pg_rewind.c | 8 ++++++++ src/bin/pg_upgrade/option.c | 13 +++++++++++++ src/bin/pg_upgrade/pg_upgrade.c | 6 ++++-- src/bin/pg_upgrade/pg_upgrade.h | 1 + src/fe_utils/option_utils.c | 27 ++++++++++++++++++++++++++ src/include/fe_utils/option_utils.h | 4 ++++ 21 files changed, 271 insertions(+), 11 deletions(-) ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Allow using syncfs() in frontend utilities. @ 2024-03-22 16:52 Robert Haas <[email protected]> parent: Nathan Bossart <[email protected]> 0 siblings, 2 replies; 9+ messages in thread From: Robert Haas @ 2024-03-22 16:52 UTC (permalink / raw) To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]> On Wed, Sep 6, 2023 at 7:28 PM Nathan Bossart <[email protected]> wrote: > Allow using syncfs() in frontend utilities. > > This commit allows specifying a --sync-method in several frontend > utilities that must synchronize many files to disk (initdb, > pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade). > On Linux, users can specify "syncfs" to synchronize the relevant > file systems instead of calling fsync() for every single file. In > many cases, using syncfs() is much faster. > > As with recovery_init_sync_method, this new option comes with some > caveats. The descriptions of these caveats have been moved to a > new appendix section in the documentation. Hi, I'd like to complain about this commit's addition of a new appendix. I do understand the temptation to document caveats like this centrally instead of in multiple places, but as I've been complaining about over in the "documentation structure" thread, our top-level documentation index is too big, and I feel strongly that we need to de-clutter it rather than cluttering it further. This added a new chapter which is just 5 sentences long. I understand that this was done because the same issue applies to a bunch of different utilities and we didn't want to duplicate this text in all of those places, but I feel like this approach just doesn't scale. If we did this in every place where we have this much text that we want to avoid duplicating, we'd soon have hundreds of appendixes. What I would suggest we do instead is pick one of the places where this comes up and document it there, perhaps the recovery_init_sync_method GUC. And then make the documentation for the other say something like, you know those issues we documented for recovery_init_sync_method? Well they also apply to this. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Allow using syncfs() in frontend utilities. @ 2024-03-26 10:18 Peter Eisentraut <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 1 reply; 9+ messages in thread From: Peter Eisentraut @ 2024-03-26 10:18 UTC (permalink / raw) To: Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]> On 22.03.24 17:52, Robert Haas wrote: > On Wed, Sep 6, 2023 at 7:28 PM Nathan Bossart <[email protected]> wrote: >> Allow using syncfs() in frontend utilities. >> >> This commit allows specifying a --sync-method in several frontend >> utilities that must synchronize many files to disk (initdb, >> pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade). >> On Linux, users can specify "syncfs" to synchronize the relevant >> file systems instead of calling fsync() for every single file. In >> many cases, using syncfs() is much faster. >> >> As with recovery_init_sync_method, this new option comes with some >> caveats. The descriptions of these caveats have been moved to a >> new appendix section in the documentation. > > Hi, > > I'd like to complain about this commit's addition of a new appendix. I already complained about that at <https://www.postgresql.org/message-id/[email protected]; and some follow-up was announced but didn't happen. It was on my list to look into cleaning up during beta. ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Allow using syncfs() in frontend utilities. @ 2024-03-26 14:52 Nathan Bossart <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 0 replies; 9+ messages in thread From: Nathan Bossart @ 2024-03-26 14:52 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Nathan Bossart <[email protected]>; [email protected] <[email protected]> On Fri, Mar 22, 2024 at 12:52:15PM -0400, Robert Haas wrote: > I'd like to complain about this commit's addition of a new appendix. I > do understand the temptation to document caveats like this centrally > instead of in multiple places, but as I've been complaining about over > in the "documentation structure" thread, our top-level documentation > index is too big, and I feel strongly that we need to de-clutter it > rather than cluttering it further. This added a new chapter which is > just 5 sentences long. I understand that this was done because the > same issue applies to a bunch of different utilities and we didn't > want to duplicate this text in all of those places, but I feel like > this approach just doesn't scale. If we did this in every place where > we have this much text that we want to avoid duplicating, we'd soon > have hundreds of appendixes. Sorry I missed this. I explored a couple of options last year but the discussion trailed off [0]. > What I would suggest we do instead is pick one of the places where > this comes up and document it there, perhaps the > recovery_init_sync_method GUC. And then make the documentation for the > other say something like, you know those issues we documented for > recovery_init_sync_method? Well they also apply to this. WFM. I'll put together a patch. [0] https://postgr.es/m/20231009204823.GA659480%40nathanxps13 -- Nathan Bossart Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Allow using syncfs() in frontend utilities. @ 2024-03-26 15:11 Nathan Bossart <[email protected]> parent: Peter Eisentraut <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Nathan Bossart @ 2024-03-26 15:11 UTC (permalink / raw) To: Peter Eisentraut <[email protected]>; +Cc: Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; [email protected] <[email protected]> On Tue, Mar 26, 2024 at 11:18:57AM +0100, Peter Eisentraut wrote: > On 22.03.24 17:52, Robert Haas wrote: >> I'd like to complain about this commit's addition of a new appendix. > > I already complained about that at <https://www.postgresql.org/message-id/[email protected]; > and some follow-up was announced but didn't happen. It was on my list to > look into cleaning up during beta. Sorry about this, I lost track of it. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Allow using syncfs() in frontend utilities. @ 2024-03-26 16:34 Nathan Bossart <[email protected]> parent: Nathan Bossart <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Nathan Bossart @ 2024-03-26 16:34 UTC (permalink / raw) To: Peter Eisentraut <[email protected]>; +Cc: Robert Haas <[email protected]>; Nathan Bossart <[email protected]>; [email protected] <[email protected]> On Tue, Mar 26, 2024 at 10:11:31AM -0500, Nathan Bossart wrote: > On Tue, Mar 26, 2024 at 11:18:57AM +0100, Peter Eisentraut wrote: >> On 22.03.24 17:52, Robert Haas wrote: >>> I'd like to complain about this commit's addition of a new appendix. >> >> I already complained about that at <https://www.postgresql.org/message-id/[email protected]; >> and some follow-up was announced but didn't happen. It was on my list to >> look into cleaning up during beta. > > Sorry about this, I lost track of it. Here's a first attempt at a patch based on Robert's suggestion from upthread. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Allow using syncfs() in frontend utilities. @ 2024-03-27 14:41 Robert Haas <[email protected]> parent: Nathan Bossart <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Robert Haas @ 2024-03-27 14:41 UTC (permalink / raw) To: Nathan Bossart <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Nathan Bossart <[email protected]>; [email protected] <[email protected]> On Tue, Mar 26, 2024 at 12:34 PM Nathan Bossart <[email protected]> wrote: > Here's a first attempt at a patch based on Robert's suggestion from > upthread. WFM. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Allow using syncfs() in frontend utilities. @ 2024-03-27 15:25 Nathan Bossart <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Nathan Bossart @ 2024-03-27 15:25 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Nathan Bossart <[email protected]>; [email protected] <[email protected]> On Wed, Mar 27, 2024 at 10:41:42AM -0400, Robert Haas wrote: > On Tue, Mar 26, 2024 at 12:34 PM Nathan Bossart > <[email protected]> wrote: >> Here's a first attempt at a patch based on Robert's suggestion from >> upthread. > > WFM. Committed. Again, I apologize this took so long. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: pgsql: Allow using syncfs() in frontend utilities. @ 2024-03-27 15:27 Robert Haas <[email protected]> parent: Nathan Bossart <[email protected]> 0 siblings, 0 replies; 9+ messages in thread From: Robert Haas @ 2024-03-27 15:27 UTC (permalink / raw) To: Nathan Bossart <[email protected]>; +Cc: Peter Eisentraut <[email protected]>; Nathan Bossart <[email protected]>; [email protected] <[email protected]> On Wed, Mar 27, 2024 at 11:25 AM Nathan Bossart <[email protected]> wrote: > Committed. Again, I apologize this took so long. No worries from my side; I only noticed recently. I guess Peter's been waiting a while, though. Thanks for committing. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2024-03-27 15:27 UTC | newest] Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-09-06 23:28 pgsql: Allow using syncfs() in frontend utilities. Nathan Bossart <[email protected]> 2024-03-22 16:52 ` Robert Haas <[email protected]> 2024-03-26 10:18 ` Peter Eisentraut <[email protected]> 2024-03-26 15:11 ` Nathan Bossart <[email protected]> 2024-03-26 16:34 ` Nathan Bossart <[email protected]> 2024-03-27 14:41 ` Robert Haas <[email protected]> 2024-03-27 15:25 ` Nathan Bossart <[email protected]> 2024-03-27 15:27 ` Robert Haas <[email protected]> 2024-03-26 14:52 ` Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox