Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1u2VI3-001bBC-BQ for pgsql-general@arkaria.postgresql.org; Wed, 09 Apr 2025 13:16:27 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1u2VI1-002ego-Lm for pgsql-general@arkaria.postgresql.org; Wed, 09 Apr 2025 13:16:25 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1u2VI1-002egf-6C for pgsql-general@lists.postgresql.org; Wed, 09 Apr 2025 13:16:25 +0000 Received: from mail-oo1-xc2a.google.com ([2607:f8b0:4864:20::c2a]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1u2VHz-003sZo-1d for pgsql-general@lists.postgresql.org; Wed, 09 Apr 2025 13:16:24 +0000 Received: by mail-oo1-xc2a.google.com with SMTP id 006d021491bc7-5fcd61e9bcdso3054001eaf.0 for ; Wed, 09 Apr 2025 06:16:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1744204582; x=1744809382; darn=lists.postgresql.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=d/FN+MzUx5o4YcZNls/XEPmcs4xQvFm3Fgtxwe0QLzM=; b=NeouBiMPOWHaierNNXPdrzmeTQvnZ6qqFeqDCx4TZ/vWjarNiUuRclKw1tzw2lNulr QNJGWsMC5YG0ii4evU69oDywJHndQMbXPBufbxxjeKWxER2YwATO0bQIASk6COFm0YJo 8xt1EyRSzqzT0uh+gJ9n7ww1ae2yPJHghZEBn1bOWekqrLO6T+4HTSHo1uFOFLjrNeiV WJFyG30FeuYYObZDBbAhD9u4fXwU/kjplDfiSOwRTXKlR4X9M1m3SLsPRXpgNqzqbl+d vyKLQ1SK4LLas6iNc6Ml4K5f9WDImRnI6UvGRwsxTvaUW7vTj6HcbuV8AFlyaNDUplrs JIdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744204582; x=1744809382; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=d/FN+MzUx5o4YcZNls/XEPmcs4xQvFm3Fgtxwe0QLzM=; b=TNakJrfOU6XODHabr+tkf7ASpKJ5AUItO5h7h2z97ENgQxksg7qVZAw6OP0XC7hVbr ADAM9KFFyYHd5cj7qCdgPNwRMCdsmrDCUHmMuIhjm7fnceUUlxVlExFEJi3rG3O4eb0M y/uuBCXar+upaqwaoOQ7k3YeUOUvTXTxhKkCq+BlHqekBF253H0vcyzkG6G8chrHF4ty 8v5r4lValKJEZJIvV0u/mJXbN6TV5fDRlGXanDtuF1XWV1AeASn1gApfv3nV6MVWPRo8 rAqgPm1JdTiMEkANHo0OImSaIxErl/j+NdSaDFDNt+GzGwAZWP/lY0qxMPneesgeZpNb 4jSQ== X-Gm-Message-State: AOJu0YzThP7Cz6WeJyHi5Yu4lD9b6w6yjpxp+hRTf8gWa8xYApWy664L dq9xwXO6iLnl3P56S0ysVmxD47UDdDDCt2QMFhNjhbPCpbyHwYgjKDHAOoHEglHigf077HKKe2R kP9ZH8XuSCTy+gccyWkqu5nOijNrCnA== X-Gm-Gg: ASbGncvW9uEcuZPIai5mVoHtwE9RJB6VL7CuzRUuoEGqgnLNV99GugEucAequWOhuMD GCl4Fovfj3Tp0M5hScUsM/8/vDcjEXqYoWr9rob9ZJYrhOxRUP6S4YlDZNgkxC4ySEhrKSAybHi IQs7B00c3HlJkUu8YtAxIRqMk= X-Google-Smtp-Source: AGHT+IEEnY7WTSuQiGpfjydZagSlaqesRuM+prI/1Lx42VVwr3pwPm/zYtQ+rjuZme9iFyfUxcAng6xsqvFcZyTAoxU= X-Received: by 2002:a4a:c282:0:b0:604:5e57:80ab with SMTP id 006d021491bc7-6045e578724mr680494eaf.0.1744204582477; Wed, 09 Apr 2025 06:16:22 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ron Johnson Date: Wed, 9 Apr 2025 09:16:11 -0400 X-Gm-Features: ATxdqUE9SibMA5wnkxpYi0vP7QQfdfig4efg7aJRW-BHyGxLsvoMP59bDKcu5Uc Message-ID: Subject: Re: PgBackRest fails due to filesystem full To: "pgsql-generallists.postgresql.org" Content-Type: multipart/alternative; boundary="0000000000008df46b06325846a3" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000008df46b06325846a3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Try creating a new stanza, and doing a full backup from it. On Wed, Apr 9, 2025 at 1:49=E2=80=AFAM KK CHN wrote: > > > On Tue, Apr 8, 2025 at 10:28=E2=80=AFPM Greg Sabino Mullane > wrote: > >> On Mon, Apr 7, 2025 at 5:32=E2=80=AFAM KK CHN wrote= : >> >>> *ERROR: [082]: WAL segment 00000001000001EB000000*4B was not archived >>> before the 60000ms timeout >>> >> >> This is the part you need to focus on. Look at your Postgres logs and >> find out why the archiver is failing. You can also test this without try= ing >> a whole backup by using the "check" command: >> https://pgbackrest.org/command.html#command-check >> > > I have run the check and it says successful !! > > [root@dbtest ~]# sudo -u postgres pgbackrest --stanza=3DDBCluster1_Repo > --log-level-console=3Dinfo check > > [root@dbtest ~]# 2025-04-09 10:52:26.148 P00 INFO: check command begin > 2.52.1: --exec-id=3D384808-715e8496 --log-level-console=3Dinfo > --log-level-file=3Ddebug --pg1-host=3D10.x.x.x --pg1-host-user=3Denterp= risedb > --pg1-path=3D/data/edb/as16/data --pg-version-force=3D16 > --repo1-cipher-pass=3D --repo1-cipher-type=3Daes-256-cbc > --repo1-path=3D/data/DB_BKUPS --stanza=3DDBCluster1_Repo > 2025-04-09 10:52:30.502 P00 INFO: check repo1 configuration (primary) > 2025-04-09 10:52:31.003 P00 INFO: check repo1 archive for WAL (primary) > 2025-04-09 10:52:36.305 P00 INFO: WAL segment 00000001000001ED00000017 > successfully archived to > '/data/DB_BKUPS/archive/DBCluster1_Repo/16-1/00000001000001ED/00000001000= 001ED00000017-8609407e8b9a1827a9d9b3e170dcc53e7af46bac.gz' > on repo1 > 2025-04-09 10:52:36.721 P00 INFO: check command end: completed > successfully (10575ms) > > > > > Then I ran > [root@dbtest ~]# sudo -u postgres pgbackrest --stanza=3DDBCluster1_Repo > --type=3Ddiff backup to test pgbackrest works fine !!!! > > It says > > 2025-04-09 10:53:52.521 P00 INFO*: backup '20250407-150858F' *cannot be > resumed: resume only valid for full backup > ^C2025-04-09 10:54:03.351 P00 INFO: backup command end: terminated on > signal [SIGINT] > > *But the # sudo -u postgres pgbackrest --stanza=3DDBCluster1_Repo info* > command *never shows such a backup 20250407-150858F exists*. The > existing backups were 20250316-232631F and prior 2 full backups to this . > > Similarly diff backups I have the last > one 20250316-232631F_20250329-172215D and prior diffs only nothing late= r > than this date . and one INCR incr backup: > 20250316-232631F_20250330-083923I noting later date than this.. So sin= ce > 2025 03 30 all backups Full/diff/incr fails ( since the / partition r= an > out of space ) > > Nothing else reported by the info command.. > > > How can I proceed to bring pgbackrest back to take backups to normal ? > [ WAL files are missing then can we never take the Full backups / diff > /inc ? What is the workaround / solution to deal with this situation ?] > > Any hints much appreciated .. > > Krishane > > >> >> Cheers, >> Greg >> >> -- >> Crunchy Data - https://www.crunchydata.com >> Enterprise Postgres Software Products & Tech Support >> >> --=20 Death to , and butter sauce. Don't boil me, I'm still alive. lobster! --0000000000008df46b06325846a3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Try creating a new stanza, and doing a full backup fr= om it.

On Wed, Apr 9, 2025 at 1:49=E2= =80=AFAM KK CHN <kkchn.in@gmail.co= m> wrote:


On Tue, Apr 8, 2025 at 10:28=E2=80=AFPM Greg Sabino Mullane <htamfids@gmail.com> wrote:
On Mon, Apr 7, 2025 at 5:32=E2=80=AFAM KK CH= N <kkchn.in@gmai= l.com> wrote:
ERROR: [082]: WAL segm= ent 00000001000001EB0000004B was not archived before the 60000ms timeou= t

This is the part you need to = focus on. Look at your Postgres logs and find out why the archiver is faili= ng. You can also test this without trying a whole backup by using the "= ;check" command:=C2=A0https://pgbackrest.org/command.html#command= -check

I have run the= check and it says successful !!

[root@dbtest ~]# = sudo -u postgres pgbackrest --stanza=3DDBCluster1_Repo =C2=A0--log-level-co= nsole=3Dinfo check=C2=A0

[root@dbtest ~]# 2025-04-09 10:52:26.148 P0= 0 =C2=A0 INFO: check command begin 2.52.1: --exec-id=3D384808-715e8496 --lo= g-level-console=3Dinfo --log-level-file=3Ddebug --pg1-host=3D10.x.x.x=C2=A0= =C2=A0--pg1-host-user=3Denterprisedb --pg1-path=3D/data/edb/as16/data --pg= -version-force=3D16 --repo1-cipher-pass=3D<redacted> --repo1-cipher-t= ype=3Daes-256-cbc --repo1-path=3D/data/DB_BKUPS --stanza=3DDBCluster1_Repo<= br>2025-04-09 10:52:30.502 P00 =C2=A0 INFO: check repo1 configuration (prim= ary)
2025-04-09 10:52:31.003 P00 =C2=A0 INFO: check repo1 archive for WA= L (primary)
2025-04-09 10:52:36.305 P00 =C2=A0 INFO: WAL segment 0000000= 1000001ED00000017 successfully archived to '/data/DB_BKUPS/archive/DBCl= uster1_Repo/16-1/00000001000001ED/00000001000001ED00000017-8609407e8b9a1827= a9d9b3e170dcc53e7af46bac.gz' on repo1
2025-04-09 10:52:36.721 P00 = =C2=A0 INFO: check command end: completed successfully (10575ms)
=



Then I ran=C2= =A0
[root@dbtest ~]# sudo -u postgres pgbackrest --stanza=3DDBCluster1_R= epo --type=3Ddiff backup=C2=A0 =C2=A0 =C2=A0to test pgbackrest works fine != !!!

It says=C2=A0

2025-04-09 10:53:52.521 P00 =C2=A0 INFO:= backup '20250407-150858F' cannot be resumed: resume only valid= for full backup
^C2025-04-09 10:54:03.351 P00 =C2=A0 INFO: backup comma= nd end: terminated on signal [SIGINT]

But the= =C2=A0 # sudo -u postgres pgbackrest --stanza=3DDBCluster1_Repo info=C2= =A0 =C2=A0 =C2=A0 =C2=A0command never shows such a backup=C2=A0 =C2=A020= 250407-150858F exists.=C2=A0 =C2=A0The existing backups were=C2=A020250= 316-232631F and prior 2 full backups to this .=C2=A0

Similarly=C2=A0 =C2=A0diff backups=C2=A0 I have the last one=C2=A0202503= 16-232631F_20250329-172215D=C2=A0 =C2=A0and prior diffs only nothing later = than this date .=C2=A0 and one INCR=C2=A0 =C2=A0=C2=A0=C2=A0 incr backup: 2= 0250316-232631F_20250330-083923I=C2=A0 =C2=A0noting later date than this..= =C2=A0 So since 2025 03 30=C2=A0 all backups=C2=A0 =C2=A0Full/diff/incr fai= ls=C2=A0 ( since the / partition ran out of space )

Nothing else reported by the info=C2=A0 command..=C2=A0=C2=A0
<= br>

How can I proceed to bring pgbackrest back to= =C2=A0 take backups to normal ?=C2=A0 =C2=A0 =C2=A0[=C2=A0 WAL files are mi= ssing then can we never take the Full backups / diff /inc=C2=A0 ? What is t= he workaround / solution to deal with this situation ?]

Any hints much appreciated ..=C2=A0

Krishane=
=C2=A0
=C2=A0
Cheers,
= Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support
=


--
Death to <Redacted>, and butter sauce.Don't boil me, I'm still alive.
<Redacted> lobs= ter!
--0000000000008df46b06325846a3--