pg.ddx.io  pgbackrest/pgbackrest GitHub issues and pull requests  
help / color / mirror / Atom feed
[pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
13+ messages / 2 participants
[nested] [flat]

* [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-10-27 01:03  "mmoldfield (@mmoldfield)" <noreply+mmoldfield@github.com>
  0 siblings, 0 replies; 13+ messages in thread

From: mmoldfield (@mmoldfield) @ 2026-10-27 01:03 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

I propose adding an experimental, opt-in restore mode that uses `syncfs()` instead of syncing each restored file individually. I’ve benchmarked an initial prototype and would like feedback on the design before preparing an upstream implementation.

This relates to [#2861](https://github.com/pgbackrest/pgbackrest/issues/2861), but focuses on fresh restores rather than backup repository writes.

The main motivation is faster replica creation for clusters containing many databases and small relation files. With large file counts, repeated synchronization can contribute substantially to restore time relative to the amount of data transferred.

The proposed sequence would be:
1. Open the destination directory before restore writes begin, and retain that descriptor until the final `syncfs()` so writeback errors during the restore can be reported.
2. Restore files through the existing workers, preserving checksum verification and metadata handling while deferring per-file syncs.
3. After workers and recovery-setting writes finish, perform a checked `syncfs()` before publishing the final `pg_control`.

Files would still be created, opened, written, and closed normally. Existing directory synchronization and final control-file syncing would remain in place. The aim is to preserve restore durability while reducing repeated synchronization costs that scale poorly with high file counts.

The feature would be experimental and disabled by default, requiring explicit opt-in. For example:

```bash
pgbackrest --stanza=example --restore-syncfs restore
```

Or in configuration:

```ini
[global:restore]
restore-syncfs=y
```

The option name is open for discussion, including whether `restore-sync-method=fsync|syncfs` would be preferable.

For a narrow-scope initial implementation, I’d target fresh restores into empty destinations on Linux 5.8+, including standby creation with a mapped WAL directory on the same file system as PGDATA. Would it make sense to defer delta restores, resumed restores, tablespaces, and cross-filesystem destinations for the initial prototype, or would it be preferable to include any of these from the start?

I benchmarked an initial prototype using a synthetic cluster containing 4,000 databases, each with 1,000 tables and primary-key indexes. This produced approximately 340 GB of logical data and just over 25 million restored files.

Tests ran on AWS EC2 `r8g.2xlarge` instances, with each benchmark container limited to 6 CPUs and 24 GiB of memory. Each host used a dedicated 512 GiB EBS gp3 volume, formatted with XFS and provisioned for 16,000 IOPS and 1,000 MiB/s throughput. The backup repository was Amazon S3.

The comparison used eight pairs of restores, with each stock/prototype pair running on the same host and volume. Four pairs ran stock first; four ran the prototype first. Every restore used 90 workers and the same offline full backup, restoring into a fresh destination.

Across eight matched pairs using 90 workers, the median time for restore preparation and file restoration was:

| Build | Median preparation and file-restoration time |
| --- | --- |
| Stock pgBackRest 2.60 | 65m 11s |
| pgBackRest 2.60 with the `syncfs` prototype | 44m 33s |

These results show a restore-time benefit for this synthetic workload and suggest the approach is worth testing on other clusters with high file counts relative to their total data size.

Looking forward to discussing this more.

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-10-27 11:14  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-10-27 11:14 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

Thanks for the detailed proposal and benchmark. This seems reasonable to me. PostgreSQL already supports syncfs (`recovery_init_sync_method=syncfs` in PG14 and `--sync-method=syncfs` for the client tools in PG17). Opening the directory before writes start and keeping it open until `syncfs()` is the right approach to get writeback errors on Linux 5.8+.

Please go ahead and submit a PR. Some things to address:

- **Tablespaces and links:** `syncfs()` only syncs the filesystem that contains the fd. All manifest targets need to be covered: pgdata, tablespaces, path links such as `pg_wal` and the parent paths of file links. After clean/build and before the restore jobs start, open each target, dedupe by `st_dev` and keep one fd per filesystem. After the jobs finish, call `syncfs()` on each fd. With that in place there is no need to limit the WAL mapping to the same filesystem or to defer tablespaces. Large clusters often have tablespaces, so an option that refuses them would be much less useful.
- **Delta:** I don't see a strong reason to exclude delta restores. Files that delta matches are not written or synced today, so syncfs would not weaken anything.
- **pg_control:** all `syncfs()` calls must complete before `pg_control` is renamed. The `global` path sync after the rename stays as it is.
- **Path sync:** the per-path sync loop in `cmdRestore()` can be skipped for paths on filesystems that were synced with `syncfs()`.
- **Platform:** syncfs is Linux only, so it needs a meson check and the option should error on other platforms. On kernels before 5.8 `syncfs()` does not report I/O errors. Distro backports make a kernel version check unreliable, so I would document this the way PostgreSQL does rather than try to detect it.
- **Option:** I prefer an enum such as `restore-sync-method=fsync|syncfs` over a boolean. It should be marked `beta: true` in `build/config.yaml` so it requires `--beta` while it is experimental. I 
- **Tests:** unit tests in `test/src/module/command/restoreTest.c` with full coverage. See CONTRIBUTING.md.

Also note that PostgreSQL runs `SyncDataDirectory()` at startup when `pg_control` does not show a clean shutdown, and that is always true for a restored online backup. With this many files the first start pays for another full sync unless `recovery_init_sync_method=syncfs` is set. It's not part of this change but it may affect your timings.

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-10-27 11:14  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-10-27 11:14 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele assigned @dwsteele.

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-10-27 11:14  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-10-27 11:14 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele applied label "question".

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-10-28 22:01  "mmoldfield (@mmoldfield)" <noreply+mmoldfield@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: mmoldfield (@mmoldfield) @ 2026-10-28 22:01 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele Thanks for the detailed feedback and clear direction. I’ll incorporate these points and have a PR up next week.

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-11-17 00:03  "mmoldfield (@mmoldfield)" <noreply+mmoldfield@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: mmoldfield (@mmoldfield) @ 2026-11-17 00:03 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

Hey @dwsteele, I have implemented this in #2894 

Is there anything else I need to do to get it picked up for review?

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-11-17 13:57  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-17 13:57 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@mmoldfield Just a bit of patience. Right now we have a lot of patches and few reviewers.

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-11-20 14:46  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-20 14:46 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele applied label "enhancement".

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-11-20 14:46  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-20 14:46 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele removed label "question".

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-11-20 14:46  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-20 14:46 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele applied label "module [core]".

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-11-20 14:46  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-20 14:46 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele set milestone to "2.61.0".

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-11-20 14:47  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-20 14:47 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

Closing since there is a PR to implement the feature.

^ permalink  raw  [nested|flat] 13+ messages in thread

* Re: [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores
@ 2026-11-20 14:47  "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
  11 siblings, 0 replies; 13+ messages in thread

From: dwsteele (@dwsteele) @ 2026-11-20 14:47 UTC (permalink / raw)
  To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>

@dwsteele closed this.

^ permalink  raw  [nested|flat] 13+ messages in thread


end of thread, other threads:[~2026-11-20 14:47 UTC | newest]

Thread overview: 13+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-27 01:03 [pgbackrest/pgbackrest] issue #2888: Proposal: experimental `syncfs` option for fresh restores "mmoldfield (@mmoldfield)" <noreply+mmoldfield@github.com>
2026-10-27 11:14 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-10-27 11:14 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-10-27 11:14 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-10-28 22:01 ` "mmoldfield (@mmoldfield)" <noreply+mmoldfield@github.com>
2026-11-17 00:03 ` "mmoldfield (@mmoldfield)" <noreply+mmoldfield@github.com>
2026-11-17 13:57 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-20 14:46 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-20 14:46 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-20 14:46 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-20 14:46 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-20 14:47 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-20 14:47 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox