MIME-Version: 1.0
From: KK CHN <kkchn.in@gmail.com>
Date: Mon, 28 Apr 2025 11:53:21 +0530
Message-ID: <CAKgGyB-+xDu6S23EgCK0GRHXUhuZ-eNSgXfAZspmLRhCKWp4_g@mail.gmail.com>
Subject: pgbackrest with two remote backend servers fails
To: pgsql-general <pgsql-general@postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000c2e1010633d0b08e"
Archived-At: <https://www.postgresql.org/message-id/CAKgGyB-%2BxDu6S23EgCK0GRHXUhuZ-eNSgXfAZspmLRhCKWp4_g%40mail.gmail.com>
Precedence: bulk

--000000000000c2e1010633d0b08e
Content-Type: text/plain; charset="UTF-8"

Folks,


I am able to perfectly run pgbackrest with Single RepoServer works fine.
 (  Stanza   My_Repo created successfully,   info  Check runs successfully
, backup perfectly works)

[ PG 16 server(10.1.2.202) , RHEL9.4, Pgbackrest 2.52.1) backup to remote
repo server1 (IP 10.x.4.242, ).  works fine ]


Trying Pgbackrest with two remote RepoServers for  a Single  DB Cluster .

For redundancy I have created another repo server2 (10.x.5.242) :

While the  DB server and  Primary Repo Server are active ,   I created the
stanza    My_Repo  from the   Repo Server 2  with the stanza-create command
it succeeds. But  when I do the * info check *command  It fails with the
following error.

[root@dbbackup2   ~]# sudo -u pgbackrest pgbackrest --stanza=My_Repo
 --log-level-console=info check
2025-04-25 17:41:51.209 P00   INFO: check command begin 2.52.1:
--exec-id=626534-c49a6903 --log-level-console=info --log-level-file=debug
--pg1-host= 10.1.2.203   --pg1-host-user=postgres  --pg1-path=
/var/lib/postgresql/16/data   --pg-version-force=16
--repo1-cipher-pass=<redacted> --repo1-cipher-type=aes-256-cbc
--repo1-path=/DBBACKUP --stanza=My_Repo
2025-04-25 17:41:54.054 P00   INFO: check repo2 configuration (primary)
2025-04-25 17:41:54.386 P00   INFO: check repo2 archive for WAL (primary)
ERROR: [082]: WAL segment 0000000100000242000000B3 was not archived before
the 60000ms timeout
       HINT: check the archive_command to ensure that all options are
correct (especially --stanza).
       HINT: check the PostgreSQL server log for errors.
       HINT: run the 'start' command if the stanza was previously stopped.
2025-04-25 17:42:54.386 P00   INFO: check command end: aborted with
exception [082]
[root@  dbbackup2   ~]#


After this step,  The   primary  reposerver   backup also fails, it's
unable to perform backup due to the WAL archive  60000 ms  timeout
happening  on db server .

 Then I have to stop the  stanza on both  repo servers  and  restart  the
stanza on   repo server 1 and re-run the stanza-create  My_Repo (Is this
required ? no sure !)   on the  Repo Server 1, check command works on the
primary repo server,   it starts working and performs a backup on  repo
server1 .    But not getting succeeded for   Repo server 2  .

What may be wrong here ? Any hints much appreciated to make  both repo
servers start taking backups  ..

Thank you .
krishane


 pgbackrest confs  =>  on Repo1, Repo2 and DB server  as follows

Repo server1 (10.x.4.242)
[root@dbbackup1 ~]# cat /etc/pgbackrest/pgbackrest.conf
[My_Repo]
pg1-host=10.1.2.202
pg1-host-user=postgres
pg1-path=/var/lib/postgresql/16/data
pg-version-force=16

[global]
repo1-path=/DB_BACKUP
repo1-block=y
repo1-bundle=y
repo1-retention-full=2
repo1-retention-diff=2
repo1-cipher-type=aes-256-cbc
repo1-cipher-pass=acbd
process-max=5
log-level-console=info
log-level-file=debug
start-fast=y
delta=y

[global:archive-push]
compress-level=3


Repo Server 2 : (10.x.5.242)
[root@dbbackup2 ~]# cat /etc/pgbackrest/pgbackrest.conf
[My_Repo]
pg1-host=10.1.2.202
pg1-host-user=postgres
pg1-path= /var/lib/postgresql/16/data
pg-version-force=16

[global]
repo2-path=/DB_BACKUP
repo2-block=y
repo2-bundle=y
repo2-retention-full=2
repo2-retention-diff=2
repo2-cipher-type=aes-256-cbc
repo2-cipher-pass=acbd
process-max=5
log-level-console=info
log-level-file=debug
start-fast=y
delta=y

[global:archive-push]
compress-level=3
[root@dbbackup ~]#

ON DB  CLUSTER [root@db~]#  :  /etc/pgbackrest/pgbackrest.conf

[My_Repo]
pg1-path=  /var/lib/postgresql/16/data
pg1-port=5444
pg1-user=postgres
pg-version-force=16


[global]
repo1-host=10.x.4.242
repo1-host-user=pgbackrest

repo2-host=10.x.5.242
repo2-host-user=pgbackrest
archive-async=y
spool-path= /var/spool/pgbackrest

log-level-console=info
log-level-file=debug
log-level-stderr=info
delta=y
compress-type=zst

[global:archive-get]
process-max=2

[global:archive-push]
process-max=6

[root@db~]#

--000000000000c2e1010633d0b08e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Folks,=C2=A0</div><div><br></div><div><br></div><div>=
I am able to perfectly run pgbackrest with Single RepoServer works fine.=C2=
=A0 =C2=A0(=C2=A0 Stanza=C2=A0 =C2=A0My_Repo created successfully,=C2=A0 =
=C2=A0info=C2=A0 Check runs successfully , backup perfectly works)</div><di=
v><br></div><div>[ PG 16 server(10.1.2.202) , RHEL9.4, Pgbackrest 2.52.1) b=
ackup to remote repo server1 (IP 10.x.4.242, ).=C2=A0 works fine ]</div><di=
v><br></div><div><br></div><div><br></div><div><div>Trying Pgbackrest with =
two remote RepoServers for=C2=A0 a Single=C2=A0 DB Cluster .</div><div><br>=
</div></div><div>For redundancy I have created another repo server2 (10.x.5=
.242) :</div><div><br></div><div>While the=C2=A0 DB server and=C2=A0 Primar=
y Repo Server are active ,=C2=A0 =C2=A0I created the stanza=C2=A0 =C2=A0 My=
_Repo=C2=A0 from the=C2=A0 =C2=A0Repo Server 2=C2=A0 with the stanza-create=
 command it succeeds. But=C2=A0 when I do the=C2=A0<b> info check </b>comma=
nd=C2=A0 It fails with the following error.=C2=A0</div><div><br></div>[root=
@dbbackup2=C2=A0=C2=A0=C2=A0~]# sudo -u pgbackrest pgbackrest --stanza=3DMy=
_Repo =C2=A0--log-level-console=3Dinfo check<br>2025-04-25 17:41:51.209 P00=
 =C2=A0 INFO: check command begin 2.52.1: --exec-id=3D626534-c49a6903 --log=
-level-console=3Dinfo --log-level-file=3Ddebug --pg1-host=3D

10.1.2.203=C2=A0=C2=A0=C2=A0--pg1-host-user=3Dpostgres=C2=A0 --pg1-path=3D

/var/lib/postgresql/16/data=C2=A0=C2=A0=C2=A0--pg-version-force=3D16 --repo=
1-cipher-pass=3D&lt;redacted&gt; --repo1-cipher-type=3Daes-256-cbc --repo1-=
path=3D/DBBACKUP --stanza=3DMy_Repo<br>2025-04-25 17:41:54.054 P00 =C2=A0 I=
NFO: check repo2 configuration (primary)<br>2025-04-25 17:41:54.386 P00 =C2=
=A0 INFO: check repo2 archive for WAL (primary)<br>ERROR: [082]: WAL segmen=
t 0000000100000242000000B3 was not archived before the 60000ms timeout<br>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0HINT: check the archive_command to ensure that a=
ll options are correct (especially --stanza).<br>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0HINT: check the PostgreSQL server log for errors.<br>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0HINT: run the &#39;start&#39; command if the stanza was previousl=
y stopped.<br>2025-04-25 17:42:54.386 P00 =C2=A0 INFO: check command end: a=
borted with exception [082]<br>[root@=C2=A0

dbbackup2=C2=A0=C2=A0=C2=A0~]#<div><br></div><div><br></div><div>After this=
 step,=C2=A0 The=C2=A0 =C2=A0primary=C2=A0 reposerver=C2=A0 =C2=A0backup al=
so fails, it&#39;s unable to perform backup due to the WAL archive=C2=A0 60=
000 ms=C2=A0 timeout happening=C2=A0 on db server .=C2=A0</div><div><br></d=
iv><div>=C2=A0Then I have to stop the=C2=A0 stanza on both=C2=A0 repo serve=
rs=C2=A0 and=C2=A0 restart=C2=A0 the stanza on=C2=A0 =C2=A0repo server 1 an=
d re-run the=C2=A0stanza-create=C2=A0 My_Repo (Is this required ? no sure !=
)=C2=A0 =C2=A0on the=C2=A0 Repo Server 1, check command works on the primar=
y repo server,=C2=A0 =C2=A0it starts working and performs a backup=C2=A0on=
=C2=A0 repo server1 .=C2=A0 =C2=A0 But not getting succeeded for=C2=A0 =C2=
=A0Repo server 2=C2=A0 .=C2=A0=C2=A0</div><div><br></div><div>What may be w=
rong here ? Any hints much appreciated to make=C2=A0 both repo servers star=
t=C2=A0taking backups=C2=A0 ..</div><div><br></div><div>Thank you .=C2=A0</=
div><div>krishane</div><div><br></div><div><br></div><div><div>=C2=A0pgback=
rest confs=C2=A0 =3D&gt;=C2=A0 on Repo1, Repo2 and DB server=C2=A0 as follo=
ws</div><div><br></div><div>Repo server1 (10.x.4.242)</div><div><div>[root@=
dbbackup1 ~]# cat /etc/pgbackrest/pgbackrest.conf</div><div>[My_Repo]<br>pg=
1-host=3D10.1.2.202<br>pg1-host-user=3Dpostgres<br>pg1-path=3D/var/lib/post=
gresql/16/data<br>pg-version-force=3D16<br><br>[global]<br>repo1-path=3D/DB=
_BACKUP<br>repo1-block=3Dy<br>repo1-bundle=3Dy<br>repo1-retention-full=3D2<=
br>repo1-retention-diff=3D2<br>repo1-cipher-type=3Daes-256-cbc<br>repo1-cip=
her-pass=3Dacbd<br>process-max=3D5<br>log-level-console=3Dinfo<br>log-level=
-file=3Ddebug<br>start-fast=3Dy<br>delta=3Dy<br><br>[global:archive-push]<b=
r>compress-level=3D3</div></div><div><br></div><div><br></div><div><br></di=
v><div><br></div><div>Repo Server 2 : (10.x.5.242)</div><div>[root@dbbackup=
2 ~]# cat /etc/pgbackrest/pgbackrest.conf<br>[My_Repo]<br>pg1-host=3D10.1.2=
.202<br>pg1-host-user=3Dpostgres<br>pg1-path=3D /var/lib/postgresql/16/data=
<br>pg-version-force=3D16<br><br>[global]<br>repo2-path=3D/DB_BACKUP<br>rep=
o2-block=3Dy<br>repo2-bundle=3Dy<br>repo2-retention-full=3D2<br>repo2-reten=
tion-diff=3D2<br>repo2-cipher-type=3Daes-256-cbc<br>repo2-cipher-pass=3Dacb=
d<br>process-max=3D5<br>log-level-console=3Dinfo<br>log-level-file=3Ddebug<=
br>start-fast=3Dy<br>delta=3Dy<br><br>[global:archive-push]<br>compress-lev=
el=3D3<br>[root@dbbackup ~]#</div><div><br></div><div>ON DB=C2=A0 CLUSTER [=
root@db~]#=C2=A0 :=C2=A0 /etc/pgbackrest/pgbackrest.conf=C2=A0</div><div><b=
r></div><div>[My_Repo]<br>pg1-path=3D=C2=A0 /var/lib/postgresql/16/data<br>=
pg1-port=3D5444<br>pg1-user=3Dpostgres<br>pg-version-force=3D16<br><br><br>=
[global]<br>repo1-host=3D10.x.4.242<br>repo1-host-user=3Dpgbackrest<br><br>=
repo2-host=3D10.x.5.242</div><div>repo2-host-user=3Dpgbackrest<br>archive-a=
sync=3Dy<br>spool-path=3D /var/spool/pgbackrest<br><br>log-level-console=3D=
info<br>log-level-file=3Ddebug<br>log-level-stderr=3Dinfo<br>delta=3Dy<br>c=
ompress-type=3Dzst<br><br>[global:archive-get]<br>process-max=3D2<br><br>[g=
lobal:archive-push]<br>process-max=3D6<br><br>[root@db~]#</div><div><br></d=
iv></div></div>

--000000000000c2e1010633d0b08e--