pg.ddx.io  pgbackrest/pgbackrest GitHub issues and pull requests  
help / color / mirror / Atom feed
From: dwsteele (@dwsteele) <noreply+dwsteele@github.com>
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
Subject: Re: [pgbackrest/pgbackrest] PR #2887: DirectIO
Date: Mon, 26 Oct 2026 17:28:41 +0000
Message-ID: <gh-pgbackrest-pgbackrest-2887-c6422873492@github.com> (raw)
In-Reply-To: <gh-pgbackrest-pgbackrest-2887@github.com>
References: <gh-pgbackrest-pgbackrest-2887@github.com>

Thanks for the PR. Before getting into the code I'm a bit worried about performance.

Right now we depend on the kernel to read ahead for us. Writes go into the page cache so they return immediately and writeback happens in the background. With `O_DIRECT` each `read()`/`write()` is a synchronous round trip to the device (`buffer-size`, so 1MiB by default) and nothing is in flight while we compress/encrypt/checksum. So per process throughput goes from roughly max(io, cpu) to io + cpu. `process-max` will hide some of that but not all of it, and it gets worse as storage latency goes up (EBS, SAN, etc.).

A few other things I noticed on the performance side:

- Reads with an offset/limit always fetch a full aligned buffer from disk. Nothing is cached so repeated reads of the same region always go to disk. Block incremental restore does a lot of this, and the checksum re-read after a block delta restore expects the pages to still be in cache.
- On Linux an `O_DIRECT` read of a range with dirty pages in the page cache forces writeback of those pages first. Since PostgreSQL writes through the page cache, a backup with `pg-direct-io` will be flushing PostgreSQL's dirty pages as it reads, outside of checkpoints.
- `O_DIRECT` writes still need `fsync` for durability so there are no savings there.
- The bounce buffer `memcpy` replaces the kernel copy so there are no CPU savings either.

Do you have any benchmarks? It would also help to know what problem you are trying to solve. If it is cache eviction (backup pushing the PostgreSQL working set out of the page cache) then `posix_fadvise(POSIX_FADV_DONTNEED)` on ranges we have already read (and after `fsync` for writes) might get most of the benefit without losing readahead/writeback.

There also looks to be a correctness problem in block incremental restore. `storageWritePosixSeek()` sets `totalWritten = position` and `storageWritePosixClose()` does `ftruncate(totalWritten)`. Delta restore opens the file with `.noTruncate` and seeks to each changed block, so with `pg-direct-io` the file will be truncated at the end of the last changed block. I haven't run it but that's how the code reads.

If we do move forward with this:

- I don't think a new driver is needed. `posix-dio` is just `posix` with a flag set, so an option like `repo-io=default/direct` (and something similar for pg) would be simpler. Also, a new option doesn't need deprecations.
- There are a lot of formatting changes to existing code (comment alignment, struct initializers) and some existing comments and `{vm_covered}` markers have been removed. That makes the diff hard to review -- please revert those.
- This needs a probe in `meson.build`. We build with `_POSIX_C_SOURCE` rather than `_GNU_SOURCE` so glibc doesn't expose `O_DIRECT`, which I'm guessing is why there's a `#define O_DIRECT __O_DIRECT` (that's glibc internal). Better to probe for `O_DIRECT`, define `HAVE_O_DIRECT`, and only allow the option when it's available (like `sftp: HAVE_LIBSSH2` in `config.yaml`).
- The PR description states it falls back to posix when `O_DIRECT` isn't supported, but the code throws `FeatureNotSupportedError`. On platforms without `O_DIRECT` (e.g. macOS) it silently uses buffered I/O but still goes through the bounce buffer.
- The 4096 alignment is hard-coded. That works for most devices but ideally it would come from the device (`statx()` with `STATX_DIOALIGN` on Linux >= 6.1).

Overall I'm just struggling to see the advantage of `O_DIRECT` over `posix_fadvise(POSIX_FADV_DONTNEED)` for our specific use case, especially given the apparent performance hit. AIO solves a lot of that at the cost of complexity but that's not what we have here.

Overall I think the `fadvise` patch had more promise but it was not moving forward and I did not have time to take it over. This patch in particular looked useful to me: https://github.com/pgbackrest/pgbackrest/compare/main...fbernier:pgbackrest:fadvise?expand=1.

view thread (20+ messages)  latest in thread

Message-ID: <gh-pgbackrest-pgbackrest-2887-c6422873492@github.com>
Permalink:  ../gh-pgbackrest-pgbackrest-2887-c6422873492@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox