pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Justin Pryzby <pryzby@telsasoft.com>
To: ldh@laurent-hasson.com
Cc: Ranier Vilela <ranier.vf@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: An I/O error occurred while sending to the backend (PG 13.4)
Date: Wed, 13 Apr 2022 10:43:19 -0500
Message-ID: <20220413154319.GP26620@telsasoft.com> (raw)
In-Reply-To: <MN2PR15MB2560C2027870AA2C1DECC29485EC9@MN2PR15MB2560.namprd15.prod.outlook.com>
References: <20220228220503.GF25269@telsasoft.com>
	<MN2PR15MB25602F46F8665F53E4E0692F85029@MN2PR15MB2560.namprd15.prod.outlook.com>
	<20220301192641.GA15744@telsasoft.com>
	<MN2PR15MB2560956DE25947E5E944C96785049@MN2PR15MB2560.namprd15.prod.outlook.com>
	<CAEudQAot0tEAkPiVLU38NdQAQx8M0p-LQmRZCVm7jCQDAkBKSA@mail.gmail.com>
	<20220303164604.GH27651@telsasoft.com>
	<CAEudQApMrXHa9AUO+3-gTw_Ev78RTxF4mvMne=txCB00U2k1Vg@mail.gmail.com>
	<MN2PR15MB256090CE9954DBABD14AD78385049@MN2PR15MB2560.namprd15.prod.outlook.com>
	<CAEudQAqPndHj+ZhryJmsETRaFtyNsqCS7oJjxFjn=zdvxUpsVw@mail.gmail.com>
	<MN2PR15MB2560C2027870AA2C1DECC29485EC9@MN2PR15MB2560.namprd15.prod.outlook.com>

On Wed, Apr 13, 2022 at 03:36:19PM +0000, ldh@laurent-hasson.com wrote:
> After a lot of back and forth, someone in IT informed us that the database VM is under a backup schedule using Veeam. Apparently, during the backup window, Veeam creates a snapshot and that takes the VM offline for a couple of minutes… And of course, they scheduled this right at the busiest time of the day for this machine which is during our nightly ETL. Their backup doesn’t perform very week either, which explained why the failure seemed to randomly happen at various points during our ETL (which takes about 2h30mn).
> 
> They moved the schedule out and the issue has not happened again over the past 3 weeks. This looks like it was the root cause and would explain (I think) how the database and the client simultaneously reported a connection timeout.
> 
> Thank you so much for all your help in trying to figure this out and exonerate Postgres.

Great, thanks for letting us know.
This time it wasn't postgres' fault; you're 2 for 3 ;)

One issue I've seen is if a vmware snapshot is taken and then saved for a long
time.  It can be okay if VEEM takes a transient snapshot, copies its data, and
then destroys the snapshot.  But it can be bad if multiple snapshots are taken
and then left around for a long time to use as a backup themselves.

-- 
Justin





view thread (25+ messages)

Message-ID: <20220413154319.GP26620@telsasoft.com>
Permalink:  ../20220413154319.GP26620@telsasoft.com/
Also on:    postgresql.org/message-id/20220413154319.GP26620@telsasoft.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: pryzby@telsasoft.com, ldh@laurent-hasson.com, ranier.vf@gmail.com
  Subject: Re: An I/O error occurred while sending to the backend (PG 13.4)
  In-Reply-To: <20220413154319.GP26620@telsasoft.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox