MIME-Version: 1.0
From: KK CHN <kkchn.in@gmail.com>
Date: Mon, 7 Apr 2025 15:04:26 +0530
Message-ID: <CAKgGyB8Exn-+K1k3AzA+B8mL+PUAkzSKa_worKKJc7dwHG5tpw@mail.gmail.com>
Subject: PgBackRest fails due to filesystem full
To: pgsql-general@lists.postgresql.org
Content-Type: multipart/alternative; boundary="000000000000c903b706322ce891"
Archived-At: <https://www.postgresql.org/message-id/CAKgGyB8Exn-%2BK1k3AzA%2BB8mL%2BPUAkzSKa_worKKJc7dwHG5tpw%40mail.gmail.com>
Precedence: bulk

--000000000000c903b706322ce891
Content-Type: text/plain; charset="UTF-8"

List,

 I am running PgbackRest-2.52.1 on RHEL9.3  and  EDB16  to backup to a
remote repo server .   Everything was working fine and backups were
regularly taken  with  cron scheduler daily.

 But due to a   /  partition full 100 % utilization, the pgbackrest  backup
failed the other day.  I came to know the backup script is not working for
the backup which is scheduled daily from a cron scheduler.  I made  space
in  /  file system by removing few  log files  from
/var/pgbackrest/DBCluster1

I tried to  reschedule the backup script (after deleting some log files
from  /var and now / is having 50 % free space ) but after running for 2 or
3 minutes pgbackrest fails as follows.


[root@dbtest log]# sudo -u postgres pgbackrest --stanza=DBCluster1_Repo
--type=full backup
2025-04-07 14:29:36.171 P00   INFO: backup command begin 2.52.1: --delta
--exec-id=4175219-0893aa9e --log-level-console=info --log-level-file=debug
--pg1-host=10.x.0.y --pg1-host-user=enterprisedb
--pg1-path=/data/edb/as16/data --pg-version-force=16 --process-max=5
--repo1-block --repo1-bundle --repo1-cipher-pass=<redacted>
--repo1-cipher-type=aes-256-cbc --repo1-path=/data/DB_BKUPS
--repo1-retention-diff=6 --repo1-retention-full=3
--stanza=DBCluster1_Repo  --start-fast --type=full
2025-04-07 14:29:40.007 P00   INFO: execute non-exclusive backup start:
backup begins after the requested immediate checkpoint completes
2025-04-07 14:29:41.383 P00   INFO: backup start archive =
00000001000001EB0000004C, lsn = 1EB/4C0003D8
2025-04-07 14:29:41.383 P00   INFO: check archive for prior segment
00000001000001EB0000004B
*ERROR: [082]: WAL segment 00000001000001EB000000*4B was not archived
before the 60000ms timeout
       HINT: check the archive_command to ensure that all options are
correct (especially --stanza).
       HINT: check the PostgreSQL server log for errors.
       HINT: run the 'start' command if the stanza was previously stopped.

*Again I ran the backup script but each time it fails with error (each time
the WAL segment error with a new WAL segment number ) *

2025-04-07 14:30:41.383 P00   INFO: backup command end: aborted with
exception [082]

     2025-04-07 14:33:03.382 P00   INFO: check archive for prior segment
00000001000001EB0000004D
ERROR: [082]: WAL segment 00000001000001EB0000004D was not archived before
the 60000ms timeout
       HINT: check the archive_command to ensure that all options are
correct (especially --stanza).
       HINT: check the PostgreSQL server log for errors.
       HINT: run the 'start' command if the stanza was previously stopped.

2025-04-07 14:34:03.382 P00   INFO: backup command end: aborted with
exception [082]


  This may be due to the WAL segment from the DB server being unable to
sync that time when the file system was full at the Repo Server side which
was observed by me after 2 days !!

Any hints how can I rectify this issue and put pgbackrest working back ??

How can I  enforce  the consistency of the Backups and WAL files since
there may be missing WAL files in between when the RepoServer file system
is full ?


Thanks in advance
Krishane

--000000000000c903b706322ce891
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>List,=C2=A0</div><div><br></div><div>=C2=A0I am runni=
ng PgbackRest-2.52.1 on RHEL9.3=C2=A0 and=C2=A0 EDB16=C2=A0 to backup to a =
remote repo server .=C2=A0 =C2=A0Everything was working fine and backups we=
re regularly taken=C2=A0 with=C2=A0 cron scheduler daily.=C2=A0=C2=A0</div>=
<div><br></div><div>=C2=A0But due to a=C2=A0 =C2=A0/=C2=A0 partition=C2=A0f=
ull 100 % utilization, the pgbackrest=C2=A0 backup failed the other day.=C2=
=A0 I came to know the backup script is not working for the backup which is=
 scheduled daily from a cron scheduler.=C2=A0 I made=C2=A0 space in=C2=A0 /=
=C2=A0 file system by removing few=C2=A0=C2=A0log files=C2=A0 from /var/pgb=
ackrest/DBCluster1=C2=A0</div><div><br></div><div>I tried to=C2=A0 reschedu=
le the backup script (after deleting some log files from=C2=A0 /var and now=
 / is having 50 % free space ) but after running for 2 or 3 minutes pgbackr=
est fails as follows.=C2=A0</div><div><br></div><div><br></div><div>[root@d=
btest log]# sudo -u postgres pgbackrest --stanza=3DDBCluster1_Repo --type=
=3Dfull backup<br>2025-04-07 14:29:36.171 P00 =C2=A0 INFO: backup command b=
egin 2.52.1: --delta --exec-id=3D4175219-0893aa9e --log-level-console=3Dinf=
o --log-level-file=3Ddebug --pg1-host=3D10.x.0.y --pg1-host-user=3Denterpri=
sedb --pg1-path=3D/data/edb/as16/data --pg-version-force=3D16 --process-max=
=3D5 --repo1-block --repo1-bundle --repo1-cipher-pass=3D&lt;redacted&gt; --=
repo1-cipher-type=3Daes-256-cbc --repo1-path=3D/data/DB_BKUPS --repo1-reten=
tion-diff=3D6 --repo1-retention-full=3D3 --stanza=3DDBCluster1_Repo=C2=A0=
=C2=A0--start-fast --type=3Dfull</div><div>2025-04-07 14:29:40.007 P00 =C2=
=A0 INFO: execute non-exclusive backup start: backup begins after the reque=
sted immediate checkpoint completes<br>2025-04-07 14:29:41.383 P00 =C2=A0 I=
NFO: backup start archive =3D 00000001000001EB0000004C, lsn =3D 1EB/4C0003D=
8<br>2025-04-07 14:29:41.383 P00 =C2=A0 INFO: check archive for prior segme=
nt 00000001000001EB0000004B<br><b>ERROR: [082]: WAL segment 00000001000001E=
B000000</b>4B was not archived before the 60000ms timeout<br>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0HINT: check the archive_command to ensure that all options are=
 correct (especially --stanza).<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0HINT: check t=
he PostgreSQL server log for errors.<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0HINT: ru=
n the &#39;start&#39; command if the stanza was previously stopped.<br><br>=
</div><div><b>Again I ran the backup script=C2=A0but each time it fails wit=
h error (each time the WAL segment error with a new WAL segment=C2=A0number=
 )=C2=A0</b></div><div><br></div><div>2025-04-07 14:30:41.383 P00 =C2=A0 IN=
FO: backup command end: aborted with exception [082]</div><div><br></div><d=
iv>=C2=A0 =C2=A0 =C2=A02025-04-07 14:33:03.382 P00 =C2=A0 INFO: check archi=
ve for prior segment 00000001000001EB0000004D</div>ERROR: [082]: WAL segmen=
t 00000001000001EB0000004D was not archived before the 60000ms timeout<br>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0HINT: check the archive_command to ensure that a=
ll options are correct (especially --stanza).<br>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0HINT: check the PostgreSQL server log for errors.<br>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0HINT: run the &#39;start&#39; command if the stanza was previousl=
y stopped.<div><br>2025-04-07 14:34:03.382 P00 =C2=A0 INFO: backup command =
end: aborted with exception [082]<br><br><br><br>=C2=A0 This may be due to =
the WAL segment from the DB server being unable to sync that time when the =
file system was full at the Repo Server side which was observed by me after=
 2 days !!</div><div><br></div><div>Any hints how can I rectify this issue =
and put pgbackrest working back ??=C2=A0</div><div><br></div><div>How can I=
=C2=A0 enforce=C2=A0 the consistency of the Backups and WAL files since the=
re may be missing WAL files in between when the RepoServer file system is f=
ull ?</div><div><br></div><div><br></div><div><br></div><div>Thanks in adva=
nce</div><div>Krishane</div><div><br></div><div><br></div><div><div><br></d=
iv><div><br></div><div><br></div></div></div>

--000000000000c903b706322ce891--