pg.ddx.io pgbackrest/pgbackrest GitHub issues and pull requests
help / color / mirror / Atom feed[pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
9+ messages / 2 participants
[nested] [flat]
* [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
@ 2026-11-17 16:43 "NoobInTheNet (@NoobInTheNet)" <noreply+NoobInTheNet@github.com>
0 siblings, 0 replies; 9+ messages in thread
From: NoobInTheNet (@NoobInTheNet) @ 2026-11-17 16:43 UTC (permalink / raw)
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
1. pgBackRest version: 2.60.1 (affected), 2.58.0 (for comparison)
2. PostgreSQL version: 17.11
3. Operating system/version: macOS 26.7 on arm64 for the reproduction below. I have not run it on Linux. The repositories are local directories, so the repository type should not matter.
4. Did you install pgBackRest from source or from a package? 2.58.0 from the nixpkgs package. 2.60.1 built from the `release/2.60.1` tag with the same package definition (meson).
5. Please attach the following as applicable:
- `pgbackrest.conf`:
```ini
[global]
repo1-path=/path/repo1
repo2-path=/path/repo2
archive-async=y
spool-path=/path/spool
log-path=/path/log
lock-path=/path/lock
process-max=2
log-level-file=detail
[demo]
pg1-path=/path/pgdata
```
- `postgresql.conf`: `archive_mode = on`, `archive_command = 'pgbackrest --config=/path/pgbackrest.conf --stanza=demo archive-push %p'`. The cluster was made with `initdb --wal-segsize=1`, to keep the test small.
- Errors in the PostgreSQL log, the same on both versions:
```
ERROR: [104]: raised from local-1 protocol: archive-push command encountered error(s):
repo2: [FileOpenError] unable to load info file '/path/repo2/archive/demo/archive.info' or '/path/repo2/archive/demo/archive.info.copy':
FileOpenError: unable to open file '/path/repo2/archive/demo/archive.info' for read: [13] Permission denied
```
- `demo-archive-push-async.log` of 2.60.1, one run, abbreviated. Every later run looks the same, with the same three segments:
```
INFO: push 12 WAL file(s) to archive: 000000010000000000000009...000000010000000000000014
WARN: could not push WAL file '000000010000000000000009' to the archive (will be retried): [104] raised from local-1 protocol: archive-push command encountered error(s):
repo2: [FileOpenError] unable to load info file '/path/repo2/archive/demo/archive.info' or '/path/repo2/archive/demo/archive.info.copy':
[...]
WARN: could not push WAL file '00000001000000000000000A' to the archive (will be retried): [...]
WARN: could not push WAL file '00000001000000000000000B' to the archive (will be retried): [...]
WARN: stopped archive-push after an error, remaining WAL will be processed on the next run
```
- `demo-archive-push-async.log` of 2.58.0, one run, abbreviated. It reports the same error for all 12 segments, `...09` to `...14`, and each of them is in repo1 afterwards.
6. Describe the issue:
With two repositories and `archive-async=y`, when one repository fails, 2.58.0 keeps pushing every ready WAL segment to the other one. 2.60.1 pushes a few segments to it and then stops, until the failed repository is back. We keep two repositories so that one can fail without affecting the other, so this matters to us.
Steps, the same on both versions:
1. `stanza-create`, start PostgreSQL, switch WAL three times, and wait until both repositories hold the three segments.
2. `chmod 000 /path/repo2/archive`.
3. Switch WAL 12 times, one second apart (`insert ...; select pg_switch_wal()`).
4. Wait 150 seconds, then count the segments in repo1.
| Version | New segments in repo1 | Async log |
| --- | --- | --- |
| 2.58.0 | 12 of 12 | no run stopped early |
| 2.60.1 | 4 of 12 | 23 runs logged `stopped archive-push after an error`; `...09`, `...0A` and `...0B` were retried in every run, `...0C` got through once, and nothing after it reached repo1 |
Both versions push the remaining segments to both repositories once repo2 is readable again.
As far as I can tell from the code, this follows from #2802. `archivePushAsyncCallback()` schedules no new job once `errorFound` is set, and `archivePushFile()` throws when any repository fails, even if the push to the others succeeded ("Throw any errors, even if some pushes were successful"). So with one repository down every job errors, and each run retries the same first few segments.
The reasoning in #2802 and #1848 is about one repository, where pushing past an error frees nothing. With two repositories it did have a use: repo1 stayed current while repo2 was down. That is what the multi-repository section of the user guide describes, and the 2.53 note says that `archive-push` can push ahead when `archive-async` is enabled.
Is this intended for the multi-repository case? If not, would it work to set `errorFound` only when a segment reached no repository, so that the queue check of #2802 still happens when everything is down? I am happy to test a patch.
cc @dwsteele and @pgstef (author and reviewer of #2802), @slardiere (reporter of #1848).
<details>
<summary>Reproduction script (bash; arguments: a label, the path of the pgBackRest binary, the PostgreSQL bin directory, a free port)</summary>
```bash
#!/usr/bin/env bash
# Reproduces what the healthy pgBackRest repository receives while the other
# repository fails, with archive-async and two local (posix) repositories.
#
# Usage: repro.sh <label> <pgbackrest binary> <postgres bin directory> <port>
#
# It creates a throwaway cluster under run-<label>/ beside this script, archives
# a baseline to both repositories, makes repo2 unreadable, switches WAL
# SEGMENTS times, waits, and prints how many of those segments repo1 holds.
# Safe to re-run: it removes its own run directory first.
set -euo pipefail
label="$1"
pgbackrest="$2"
pgbin="$3"
port="$4"
segments=12 # WAL switches while repo2 fails
wait_seconds=150 # more than two archiver cycles of PostgreSQL (60 s each)
here="$(cd "$(dirname "$0")" && pwd)"
work="$here/run-$label"
conf="$work/pgbackrest.conf"
if [ -d "$work" ]; then
[ -f "$work/pgdata/postmaster.pid" ] && "$pgbin/pg_ctl" -D "$work/pgdata" -m immediate stop >/dev/null 2>&1 || true
chmod -R u+rwx "$work"
rm -rf "$work"
fi
mkdir -p "$work"/{repo1,repo2,spool,log,lock}
cat >"$conf" <<EOF
[global]
repo1-path=$work/repo1
repo2-path=$work/repo2
archive-async=y
spool-path=$work/spool
log-path=$work/log
lock-path=$work/lock
process-max=2
log-level-console=warn
log-level-file=detail
[demo]
pg1-path=$work/pgdata
EOF
run_pgbackrest() { "$pgbackrest" --config="$conf" --stanza=demo "$@"; }
sql() { "$pgbin/psql" -X -q -A -t -h 127.0.0.1 -p "$port" -U postgres -d postgres "$@"; }
count_segments() { find "$1/archive/demo" -type f -name '0000000[0-9A-F]*' 2>/dev/null | wc -l | tr -d ' '; }
newest_segment() { find "$1/archive/demo" -type f -name '0000000[0-9A-F]*' 2>/dev/null | sed 's|.*/||' | cut -c1-24 | sort | tail -n 1; }
echo "== $label: $("$pgbackrest" version), $("$pgbin/postgres" --version)"
"$pgbin/initdb" -D "$work/pgdata" -U postgres --auth=trust --locale=C -E UTF8 --wal-segsize=1 >/dev/null
cat >>"$work/pgdata/postgresql.conf" <<EOF
listen_addresses = '127.0.0.1'
port = $port
unix_socket_directories = ''
archive_mode = on
archive_command = '$pgbackrest --config=$conf --stanza=demo archive-push %p'
EOF
# The cluster is stopped, so the stanza is created without a connection.
run_pgbackrest --no-online stanza-create
"$pgbin/pg_ctl" -D "$work/pgdata" -l "$work/postgres.log" -w start >/dev/null
trap '"$pgbin/pg_ctl" -D "$work/pgdata" -m fast stop >/dev/null 2>&1 || true; chmod -R u+rwx "$work/repo2" 2>/dev/null || true' EXIT
sql -c "create table filler (n int)"
switch_wal() { sql -c "insert into filler select generate_series(1, 2000)" -c "select pg_switch_wal()" >/dev/null; }
# Baseline: three segments must reach both repositories.
for _ in 1 2 3; do switch_wal; done
for _ in $(seq 1 60); do
[ "$(count_segments "$work/repo1")" -ge 3 ] && [ "$(count_segments "$work/repo2")" -ge 3 ] && break
sleep 1
done
base1="$(count_segments "$work/repo1")"
base2="$(count_segments "$work/repo2")"
echo "baseline: repo1 holds $base1 segments, repo2 holds $base2"
if [ "$base1" -lt 3 ] || [ "$base2" -lt 3 ]; then
echo "baseline failed: archiving does not work with both repositories healthy"
exit 1
fi
# repo2 fails: its archive directory becomes unreadable.
chmod 000 "$work/repo2/archive"
for _ in $(seq 1 "$segments"); do
switch_wal
sleep 1
done
echo "repo2 made unreadable, $segments WAL switches done, waiting $wait_seconds s"
sleep "$wait_seconds"
chmod 755 "$work/repo2/archive"
after1="$(count_segments "$work/repo1")"
after2="$(count_segments "$work/repo2")"
ready="$(find "$work/pgdata/pg_wal/archive_status" -name '*.ready' | wc -l | tr -d ' ')"
log="$work/log/demo-archive-push-async.log"
echo "RESULT $label: repo1 received $((after1 - base1)) of the $segments segments while repo2 failed (repo2 received $((after2 - base2)))"
echo "newest segment in repo1: $(newest_segment "$work/repo1")"
echo "segments that PostgreSQL still waits to archive (.ready): $ready"
echo "async log: $(grep -c 'pushed WAL file' "$log" || true) pushed, $(grep -c 'could not push WAL file' "$log" || true) could not push, $(grep -c 'stopped archive-push after an error' "$log" || true) runs stopped after an error"
echo "one error line from the log:"
grep -m 1 'could not push WAL file' "$log" | cut -c1-300 || true
```
</details>
^ permalink raw [nested|flat] 9+ messages in thread
* Re: [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
@ 2026-11-17 17:07 ` "NoobInTheNet (@NoobInTheNet)" <noreply+NoobInTheNet@github.com>
7 siblings, 0 replies; 9+ messages in thread
From: NoobInTheNet (@NoobInTheNet) @ 2026-11-17 17:07 UTC (permalink / raw)
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
I have now run the same script on Linux (aarch64, kernel 6.18, PostgreSQL 17.11, inside a Nix build sandbox), so the limit in point 3 above no longer applies. The result matches macOS:
| Version | New segments in repo1 | Async log |
| --- | --- | --- |
| 2.58.0 | 12 of 12 | no run stopped early |
| 2.60.1 | 3 of 12 | 20 runs logged `stopped archive-push after an error`; nothing after `...0B` reached repo1 |
Both binaries were built from the release tags with the nixpkgs package definition.
^ permalink raw [nested|flat] 9+ messages in thread
* Re: [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
@ 2026-11-17 17:08 ` "NoobInTheNet (@NoobInTheNet)" <noreply+NoobInTheNet@github.com>
7 siblings, 0 replies; 9+ messages in thread
From: NoobInTheNet (@NoobInTheNet) @ 2026-11-17 17:08 UTC (permalink / raw)
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
I have now run the same script on Linux (aarch64, kernel 6.18, PostgreSQL 17.11, inside a Nix build sandbox), so the limit in point 3 above no longer applies. The result matches macOS:
| Version | New segments in repo1 | Async log |
| --- | --- | --- |
| 2.58.0 | 12 of 12 | no run stopped early |
| 2.60.1 | 3 of 12 | 20 runs logged `stopped archive-push after an error`; nothing after `...0B` reached repo1 |
Both binaries were built from the release tags with the nixpkgs package definition.
^ permalink raw [nested|flat] 9+ messages in thread
* Re: [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
@ 2026-11-18 13:51 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
7 siblings, 0 replies; 9+ messages in thread
From: dwsteele (@dwsteele) @ 2026-11-18 13:51 UTC (permalink / raw)
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
@dwsteele assigned @dwsteele.
^ permalink raw [nested|flat] 9+ messages in thread
* Re: [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
@ 2026-11-18 13:52 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
7 siblings, 0 replies; 9+ messages in thread
From: dwsteele (@dwsteele) @ 2026-11-18 13:52 UTC (permalink / raw)
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
@dwsteele applied label "bug".
^ permalink raw [nested|flat] 9+ messages in thread
* Re: [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
@ 2026-11-18 13:52 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
7 siblings, 0 replies; 9+ messages in thread
From: dwsteele (@dwsteele) @ 2026-11-18 13:52 UTC (permalink / raw)
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
@dwsteele applied label "module [core]".
^ permalink raw [nested|flat] 9+ messages in thread
* Re: [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
@ 2026-11-18 13:53 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
7 siblings, 0 replies; 9+ messages in thread
From: dwsteele (@dwsteele) @ 2026-11-18 13:53 UTC (permalink / raw)
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
@dwsteele set milestone to "2.60.2".
^ permalink raw [nested|flat] 9+ messages in thread
* Re: [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
@ 2026-11-18 13:56 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
7 siblings, 0 replies; 9+ messages in thread
From: dwsteele (@dwsteele) @ 2026-11-18 13:56 UTC (permalink / raw)
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
Thank you for the detailed report! This was definitely introduced by #2802/a0c143593 and it was not intentional to break the multi repo case.
I've prepared a fix in #2927. Please test if you are able.
^ permalink raw [nested|flat] 9+ messages in thread
* Re: [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0)
@ 2026-11-18 13:56 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
7 siblings, 0 replies; 9+ messages in thread
From: dwsteele (@dwsteele) @ 2026-11-18 13:56 UTC (permalink / raw)
To: pgbackrest/pgbackrest <noreply+pgbackrest-pgbackrest@github.com>
@dwsteele closed this.
^ permalink raw [nested|flat] 9+ messages in thread
end of thread, other threads:[~2026-11-18 13:56 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-17 16:43 [pgbackrest/pgbackrest] issue #2926: Async archive-push no longer pushes ahead to a healthy repository when another repository fails (2.60.1 vs 2.58.0) "NoobInTheNet (@NoobInTheNet)" <noreply+NoobInTheNet@github.com>
2026-11-17 17:07 ` "NoobInTheNet (@NoobInTheNet)" <noreply+NoobInTheNet@github.com>
2026-11-17 17:08 ` "NoobInTheNet (@NoobInTheNet)" <noreply+NoobInTheNet@github.com>
2026-11-18 13:51 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-18 13:52 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-18 13:52 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-18 13:53 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-18 13:56 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
2026-11-18 13:56 ` "dwsteele (@dwsteele)" <noreply+dwsteele@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox