MIME-Version: 1.0
References: <CAGbX52Fg2jxqGVZzeQ_QcHSZ8fgDjnVZUJy5NUb5-PAf8fvxkw@mail.gmail.com>
 <52afa4c9-7393-4265-88bb-6393f1b0fb03@aklaver.com> <CAGbX52EyjjOQJUjA0m4+0azs_yH1by63p6hhJg7Y66=VCCqzpA@mail.gmail.com>
 <45e8c44b-2506-41b5-b999-5fdc42472644@aklaver.com> <CAGbX52ENsSHKoTyu5+XfN1o1bZ2w2CJaE1oQnxcm=fj2SyoZXg@mail.gmail.com>
 <CAKFQuwYCwp-sR=p6oCysRTLyiX4c_kj6DOtHxFqHS3Pg05u1dg@mail.gmail.com>
In-Reply-To: <CAKFQuwYCwp-sR=p6oCysRTLyiX4c_kj6DOtHxFqHS3Pg05u1dg@mail.gmail.com>
From: Koen De Groote <kdg.dev@gmail.com>
Date: Tue, 22 Oct 2024 21:50:24 +0200
Message-ID: <CAGbX52EO3zCuNEbbLOkKjXOxbVUnQf-P=vkGLFOuxDYg1nFiuw@mail.gmail.com>
Subject: Re: Basebackup fails without useful error message
To: "David G. Johnston" <david.g.johnston@gmail.com>
Cc: Adrian Klaver <adrian.klaver@aklaver.com>, 
	PostgreSQL General <pgsql-general@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000443d0606251615f7"
Archived-At: <https://www.postgresql.org/message-id/CAGbX52EO3zCuNEbbLOkKjXOxbVUnQf-P%3DvkGLFOuxDYg1nFiuw%40mail.gmail.com>
Precedence: bulk

--000000000000443d0606251615f7
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hello David,

I saw the backup fail. The backup logged that it terminated the walsender,
and correlating the moment it failed to the metrics of my storage, shows
the storage at that time was facing a huge IOWAIT. And this was a network
mounted storage.

The backup process continued, but because of a failure to stream WAL
without error(due to a local issue) the entire backup was marked as failed.
At the end, pg_basebackup will delete the backup, in this case. There's no
flag to control this final behavior.

I'll be testing restore soon without streaming WAL, since the actual
restore I perform doesn't use the pg_wal.tar.gz file. It gets the archived
WAL At least I think it doesn't need it, hence the need for testing.

Regards,
Koen De Groote

On Tue, Oct 22, 2024 at 12:34=E2=80=AFAM David G. Johnston <
david.g.johnston@gmail.com> wrote:

> On Sunday, October 20, 2024, Koen De Groote <kdg.dev@gmail.com> wrote:
>>
>>
>> I'm going to be testing this. If someone could confirm that this is how
>> writing WAL files works, that being: that it is only considered "done" w=
hen
>> the archive_command is done, that would be great.
>>
>
> The archiving of WAL files by the primary does not involve a replication
> connection of any sort and thus the =E2=80=9CWAL sender=E2=80=9D settings=
 are not relevant
> to it; or, here, whether or not you are archiving your WAL is immaterial
> since you are streaming it as it gets produced.
>
> If you are streaming WAL it seems highly unusual that you=E2=80=99d end u=
p in a
> situation where the connection goes idle long enough that it gets killed,
> especially if the backup is still happening.  I=E2=80=99d probably go wit=
h
> performing the backup under a disabled (or extremely large?) timeout thou=
gh
> and move on to other things.
>
> That isn=E2=80=99t to say I fully understand what actually is happening h=
ere=E2=80=A6
>
> David J.
>
>

--000000000000443d0606251615f7
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hello David,</div><div><br></div><div>I saw the backu=
p fail. The backup logged that it terminated the walsender, and correlating=
 the moment it failed to the metrics of my storage, shows the storage at th=
at time was facing a huge IOWAIT. And this was a network mounted storage.</=
div><div><br></div><div>The backup process continued, but because of a fail=
ure to stream WAL without error(due to a local issue) the entire backup was=
 marked as failed. At the end, pg_basebackup will delete the backup, in thi=
s case. There&#39;s no flag to control this final behavior.</div><div><br><=
/div><div>I&#39;ll be testing restore soon without streaming WAL, since the=
 actual restore I perform doesn&#39;t use the pg_wal.tar.gz file. It gets t=
he archived WAL At least I think it doesn&#39;t need it, hence the need for=
 testing.</div><div><br></div><div>Regards,</div><div>Koen De Groote<br></d=
iv></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_att=
r">On Tue, Oct 22, 2024 at 12:34=E2=80=AFAM David G. Johnston &lt;<a href=
=3D"mailto:david.g.johnston@gmail.com">david.g.johnston@gmail.com</a>&gt; w=
rote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p=
x 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sunday,=
 October 20, 2024, Koen De Groote &lt;<a href=3D"mailto:kdg.dev@gmail.com" =
target=3D"_blank">kdg.dev@gmail.com</a>&gt; wrote:<blockquote class=3D"gmai=
l_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,20=
4,204);padding-left:1ex"><div dir=3D"ltr"><div><div><br></div><div>I&#39;m =
going to be testing this. If someone could confirm that this is how writing=
 WAL files works, that being: that it is only considered &quot;done&quot; w=
hen the archive_command is done, that would be great.</div></div></div></bl=
ockquote><div><br></div><div>The archiving of WAL files by the primary does=
 not involve a replication connection of any sort and thus the =E2=80=9CWAL=
 sender=E2=80=9D settings are not relevant to it; or, here, whether or not =
you are archiving your WAL is immaterial since you are streaming it as it g=
ets produced.</div><div><br></div><div>If you are streaming WAL it seems hi=
ghly unusual that you=E2=80=99d end up in a situation where the connection =
goes idle long enough that it gets killed, especially if the backup is stil=
l happening.=C2=A0 I=E2=80=99d probably go with performing the backup under=
 a disabled (or extremely large?) timeout though and move on to other thing=
s.</div><div><br></div><div>That isn=E2=80=99t to say I fully understand wh=
at actually is happening here=E2=80=A6</div><div><br></div><div>David J.</d=
iv><div><br></div>
</blockquote></div>

--000000000000443d0606251615f7--