Date: Wed, 13 Apr 2022 10:43:19 -0500
From: Justin Pryzby <pryzby@telsasoft.com>
To: ldh@laurent-hasson.com
Cc: Ranier Vilela <ranier.vf@gmail.com>, pgsql-performance@postgresql.org
Subject: Re: An I/O error occurred while sending to the backend (PG 13.4)
Message-ID: <20220413154319.GP26620@telsasoft.com>
References: <20220228220503.GF25269@telsasoft.com>
 <MN2PR15MB25602F46F8665F53E4E0692F85029@MN2PR15MB2560.namprd15.prod.outlook.com>
 <20220301192641.GA15744@telsasoft.com>
 <MN2PR15MB2560956DE25947E5E944C96785049@MN2PR15MB2560.namprd15.prod.outlook.com>
 <CAEudQAot0tEAkPiVLU38NdQAQx8M0p-LQmRZCVm7jCQDAkBKSA@mail.gmail.com>
 <20220303164604.GH27651@telsasoft.com>
 <CAEudQApMrXHa9AUO+3-gTw_Ev78RTxF4mvMne=txCB00U2k1Vg@mail.gmail.com>
 <MN2PR15MB256090CE9954DBABD14AD78385049@MN2PR15MB2560.namprd15.prod.outlook.com>
 <CAEudQAqPndHj+ZhryJmsETRaFtyNsqCS7oJjxFjn=zdvxUpsVw@mail.gmail.com>
 <MN2PR15MB2560C2027870AA2C1DECC29485EC9@MN2PR15MB2560.namprd15.prod.outlook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
 <MN2PR15MB2560C2027870AA2C1DECC29485EC9@MN2PR15MB2560.namprd15.prod.outlook.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Archived-At: 
 <https://www.postgresql.org/message-id/20220413154319.GP26620%40telsasoft.com>
Precedence: bulk

On Wed, Apr 13, 2022 at 03:36:19PM +0000, ldh@laurent-hasson.com wrote:
> After a lot of back and forth, someone in IT informed us that the database VM is under a backup schedule using Veeam. Apparently, during the backup window, Veeam creates a snapshot and that takes the VM offline for a couple of minutes… And of course, they scheduled this right at the busiest time of the day for this machine which is during our nightly ETL. Their backup doesn’t perform very week either, which explained why the failure seemed to randomly happen at various points during our ETL (which takes about 2h30mn).
> 
> They moved the schedule out and the issue has not happened again over the past 3 weeks. This looks like it was the root cause and would explain (I think) how the database and the client simultaneously reported a connection timeout.
> 
> Thank you so much for all your help in trying to figure this out and exonerate Postgres.

Great, thanks for letting us know.
This time it wasn't postgres' fault; you're 2 for 3 ;)

One issue I've seen is if a vmware snapshot is taken and then saved for a long
time.  It can be okay if VEEM takes a transient snapshot, copies its data, and
then destroys the snapshot.  But it can be bad if multiple snapshots are taken
and then left around for a long time to use as a backup themselves.

-- 
Justin