Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1u2OJW-00HS1r-3Z for pgsql-general@arkaria.postgresql.org; Wed, 09 Apr 2025 05:49:30 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1u2OJT-000j1m-UH for pgsql-general@arkaria.postgresql.org; Wed, 09 Apr 2025 05:49:28 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1u2OJT-000j1b-Hx for pgsql-general@lists.postgresql.org; Wed, 09 Apr 2025 05:49:27 +0000 Received: from mail-yb1-xb33.google.com ([2607:f8b0:4864:20::b33]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1u2OJR-004H55-1B for pgsql-general@lists.postgresql.org; Wed, 09 Apr 2025 05:49:27 +0000 Received: by mail-yb1-xb33.google.com with SMTP id 3f1490d57ef6-e609cff9927so4437177276.3 for ; Tue, 08 Apr 2025 22:49:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744177763; x=1744782563; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=eE/CdSylPkPpdfbi1u4+gty/xih+pDQyjkix+kF4N4s=; b=nGw0uzgJP4decNpVXB/kM+d7U3kpcpAFNzMJmzLzSaYCcsiEKt3akTUGN4c8nWL8dx qz3bPN3q1uEv/iJWp9jnIcyIrTFi4Ao72Ei8PJLM9SQYgXq5KCxJnoZyn6nuf6vGHsOB DOs8lpK3LVyYCjCyPBrh9m1PiB8E/DuDbK3D2NIkWqVzWN7gD/qhrG8T8aKjVkpBJIP8 cmqSqmBgSDmHKQ73pBYGzBnkCHEEawz/ZKLKu8lDkh0IgZjvXm47O3M5FQyX2/WicRVq Zx7bgbUFXP7asfS6MpOs79YchIlC8WqwDs6LyKRCZPdDEnvMwUFN9DVf0woDGFDn0FkL cfOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744177763; x=1744782563; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=eE/CdSylPkPpdfbi1u4+gty/xih+pDQyjkix+kF4N4s=; b=QulPd5xhyZN1vkfGhz7HQbF/hH7L3oP/zEP6iH3J+B/z313pZEpqY1ghr3F6x48BoN B40f1i2Y2HkSQMxkUhtyLsVnC/uk8natqF1/KKLKMsWos7Hp56FIPPTzqtDmqjJW17Zm MZRiO8AjBj052PzW2WxBVclivM3NcQp/3B4xFICF8VJIYfDdaubcYMX1WrRW4x/9jNgL iLBzCxUmDISgmsehy0CUyVEGOVn3SquKfhl50sdx9fxQxZahJUB05vYrmHLD2AYosgp0 zjol8aYg7lTX2huxnYiRBJGDEGKvrkZFA1B0YalD3fBMktjZ0EPf5KghrTpzG8DfvME0 n1YQ== X-Gm-Message-State: AOJu0YwlNncYuT0/YpZbL2jKygxu4E5tldMxstGk3LYIh3gAtj02/EQx 5vIuZH7Tg+j+K4IfJtxrXTAXfCR1cptvfaDuGPx6M8/PxCF/8N2h2yaw2ukaCsH3rJoU1+PpJtY GKjDlHolnPDhvY9iMV92Mf6w3XWg= X-Gm-Gg: ASbGncviF6xxou+31WPD9oSJudSNoIvrpBHVDZsgh4MC5a4sXQ7cCFIYv+iY/hJmmgr l1CpX31oGpn+I0hSn0D87q813yUOQ/5qCE7oRhHnEXeMnovTmkvHrUYLgsnWkMU8auKJu0fMdKj dKXc/UZY/BFeJ24L9bTptHnC0= X-Google-Smtp-Source: AGHT+IEok8pwdSnRozgrsWZ5y9k74qEbb7WV5aeYRwDbzVnyCYiysuQbHG+26DUXpUw3Ufg2kw9l/4I/8h0DFfuIjs4= X-Received: by 2002:a05:6902:2511:b0:e6d:dfb6:89ec with SMTP id 3f1490d57ef6-e702f6bdf98mr2289089276.43.1744177763341; Tue, 08 Apr 2025 22:49:23 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: KK CHN Date: Wed, 9 Apr 2025 11:21:45 +0530 X-Gm-Features: ATxdqUGwsHWl0uxecE4wDPZp-1S9OU2D_oKbRx89600t7Hd45DFUcDb16ZEGbmg Message-ID: Subject: Re: PgBackRest fails due to filesystem full To: Greg Sabino Mullane Cc: pgsql-general@lists.postgresql.org Content-Type: multipart/alternative; boundary="000000000000025d4106325208c4" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000025d4106325208c4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Apr 8, 2025 at 10:28=E2=80=AFPM Greg Sabino Mullane wrote: > On Mon, Apr 7, 2025 at 5:32=E2=80=AFAM KK CHN wrote: > >> *ERROR: [082]: WAL segment 00000001000001EB000000*4B was not archived >> before the 60000ms timeout >> > > This is the part you need to focus on. Look at your Postgres logs and fin= d > out why the archiver is failing. You can also test this without trying a > whole backup by using the "check" command: > https://pgbackrest.org/command.html#command-check > I have run the check and it says successful !! [root@dbtest ~]# sudo -u postgres pgbackrest --stanza=3DDBCluster1_Repo --log-level-console=3Dinfo check [root@dbtest ~]# 2025-04-09 10:52:26.148 P00 INFO: check command begin 2.52.1: --exec-id=3D384808-715e8496 --log-level-console=3Dinfo --log-level-file=3Ddebug --pg1-host=3D10.x.x.x --pg1-host-user=3Denterpri= sedb --pg1-path=3D/data/edb/as16/data --pg-version-force=3D16 --repo1-cipher-pass=3D --repo1-cipher-type=3Daes-256-cbc --repo1-path=3D/data/DB_BKUPS --stanza=3DDBCluster1_Repo 2025-04-09 10:52:30.502 P00 INFO: check repo1 configuration (primary) 2025-04-09 10:52:31.003 P00 INFO: check repo1 archive for WAL (primary) 2025-04-09 10:52:36.305 P00 INFO: WAL segment 00000001000001ED00000017 successfully archived to '/data/DB_BKUPS/archive/DBCluster1_Repo/16-1/00000001000001ED/0000000100000= 1ED00000017-8609407e8b9a1827a9d9b3e170dcc53e7af46bac.gz' on repo1 2025-04-09 10:52:36.721 P00 INFO: check command end: completed successfully (10575ms) Then I ran [root@dbtest ~]# sudo -u postgres pgbackrest --stanza=3DDBCluster1_Repo --type=3Ddiff backup to test pgbackrest works fine !!!! It says 2025-04-09 10:53:52.521 P00 INFO*: backup '20250407-150858F' *cannot be resumed: resume only valid for full backup ^C2025-04-09 10:54:03.351 P00 INFO: backup command end: terminated on signal [SIGINT] *But the # sudo -u postgres pgbackrest --stanza=3DDBCluster1_Repo info* command *never shows such a backup 20250407-150858F exists*. The existing backups were 20250316-232631F and prior 2 full backups to this . Similarly diff backups I have the last one 20250316-232631F_20250329-172215D and prior diffs only nothing later than this date . and one INCR incr backup: 20250316-232631F_20250330-083923I noting later date than this.. So since 2025 03 30 all backups Full/diff/incr fails ( since the / partition ran out of space ) Nothing else reported by the info command.. How can I proceed to bring pgbackrest back to take backups to normal ? [ WAL files are missing then can we never take the Full backups / diff /inc ? What is the workaround / solution to deal with this situation ?] Any hints much appreciated .. Krishane > > Cheers, > Greg > > -- > Crunchy Data - https://www.crunchydata.com > Enterprise Postgres Software Products & Tech Support > > --000000000000025d4106325208c4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Apr 8, = 2025 at 10:28=E2=80=AFPM Greg Sabino Mullane <htamfids@gmail.com> wrote:
On Mon, Apr= 7, 2025 at 5:32=E2=80=AFAM KK CHN <kkchn.in@gmail.com> wrote:
ERROR: [082]: WAL segment 00000001000001EB0000004B was not a= rchived before the 60000ms timeout

<= div>This is the part you need to focus on. Look at your Postgres logs and f= ind out why the archiver is failing. You can also test this without trying = a whole backup by using the "check" command:=C2=A0https://pg= backrest.org/command.html#command-check
<= div>
I have run the check and it says successful !!

[root@dbtest ~]# sudo -u postgres pgbackrest --stanza=3DDBC= luster1_Repo =C2=A0--log-level-console=3Dinfo check=C2=A0

[root@dbte= st ~]# 2025-04-09 10:52:26.148 P00 =C2=A0 INFO: check command begin 2.52.1:= --exec-id=3D384808-715e8496 --log-level-console=3Dinfo --log-level-file=3D= debug --pg1-host=3D10.x.x.x=C2=A0 =C2=A0--pg1-host-user=3Denterprisedb --pg= 1-path=3D/data/edb/as16/data --pg-version-force=3D16 --repo1-cipher-pass=3D= <redacted> --repo1-cipher-type=3Daes-256-cbc --repo1-path=3D/data/DB_= BKUPS --stanza=3DDBCluster1_Repo
2025-04-09 10:52:30.502 P00 =C2=A0 INFO= : check repo1 configuration (primary)
2025-04-09 10:52:31.003 P00 =C2=A0= INFO: check repo1 archive for WAL (primary)
2025-04-09 10:52:36.305 P00= =C2=A0 INFO: WAL segment 00000001000001ED00000017 successfully archived to= '/data/DB_BKUPS/archive/DBCluster1_Repo/16-1/00000001000001ED/00000001= 000001ED00000017-8609407e8b9a1827a9d9b3e170dcc53e7af46bac.gz' on repo1<= br>2025-04-09 10:52:36.721 P00 =C2=A0 INFO: check command end: completed su= ccessfully (10575ms)




Then I ran=C2=A0
[root@dbtest ~]# sudo -u postgres pgbac= krest --stanza=3DDBCluster1_Repo --type=3Ddiff backup=C2=A0 =C2=A0 =C2=A0to= test pgbackrest works fine !!!!

It says=C2=A0

2025-04-09 10:= 53:52.521 P00 =C2=A0 INFO: backup '20250407-150858F' cannot = be resumed: resume only valid for full backup
^C2025-04-09 10:54:03.351 = P00 =C2=A0 INFO: backup command end: terminated on signal [SIGINT]

But the=C2=A0 # sudo -u postgres pgbackrest --stanza=3D= DBCluster1_Repo info=C2=A0 =C2=A0 =C2=A0 =C2=A0command never shows s= uch a backup=C2=A0 =C2=A020250407-150858F exists.=C2=A0 =C2=A0The exist= ing backups were=C2=A020250316-232631F and prior 2 full backups to this .= =C2=A0

Similarly=C2=A0 =C2=A0diff backups=C2=A0 I = have the last one=C2=A020250316-232631F_20250329-172215D=C2=A0 =C2=A0and pr= ior diffs only nothing later than this date .=C2=A0 and one INCR=C2=A0 =C2= =A0=C2=A0=C2=A0 incr backup: 20250316-232631F_20250330-083923I=C2=A0 =C2=A0= noting later date than this..=C2=A0 So since 2025 03 30=C2=A0 all backups= =C2=A0 =C2=A0Full/diff/incr fails=C2=A0 ( since the / partition ran out of = space )

Nothing else reported by the info=C2=A0 co= mmand..=C2=A0=C2=A0


How can I proce= ed to bring pgbackrest back to=C2=A0 take backups to normal ?=C2=A0 =C2=A0 = =C2=A0[=C2=A0 WAL files are missing then can we never take the Full backups= / diff /inc=C2=A0 ? What is the workaround / solution to deal with this si= tuation ?]

Any hints much appreciated ..=C2=A0

Krishane
=C2=A0
=C2=A0
Cheers,
Greg

--
Enterprise Postgres Software Products &= amp; Tech Support

--000000000000025d4106325208c4--