From: Stephen Frost <sfrost@snowman.net>
To: Achilleas Mantzios <achill@matrix.gatewaynet.com>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging
Date: Wed, 16 Jan 2019 15:27:58 -0500
Message-ID: <20190116202758.GG2528@tamriel.snowman.net> (raw)
In-Reply-To: <e7d0ac3f-cf49-7d04-87b7-0944a46207e0@matrix.gatewaynet.com>
References: <e7d0ac3f-cf49-7d04-87b7-0944a46207e0@matrix.gatewaynet.com>
Greetings,
* Achilleas Mantzios (achill@matrix.gatewaynet.com) wrote:
> I had this error today with postgresql 10.6 and pgbackrest 2.07. I was running test backups and during a backup
> The command given to the repo host (pg1-host=10.9.0.77) was :
> pgbackrest@smadb2cs:~$ pgbackrest --stanza=dynacomtest backup
> I noticed that the (two) ssh login sessions to the repo host just froze. No disconnections, just froze. So some network problem surely occurred.
> On the repo host pgbackrest log I got :
> 2019-01-16 12:20:38.372 P00 ERROR: [042]: unexpected EOF reading line from remote process on '10.9.0.77'
> on the pg host (10.9.0.77) I noticed that the pgbackrest backup processes
> had died, the only ones running was spawn from the archive-push , however on
> the repo host the pgbackrest processes were alive, when in fact the backup
> was stalled.
> On the repo I got :
> root@smadb2cs:/home/pgbackrest# netstat -apn | grep 10.9.0.77
> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh
> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh
> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh
> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh
> root@smadb2cs:/home/pgbackrest# netstat -anp --timers | grep 0.77
> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh on (65.50/12/0)
> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh on (69.60/12/0)
> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh on (65.50/12/0)
> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh on (61.40/12/0)
This looks like the kernel thinks the connections are still alive.
> In order to start a new backup I had to kill the 4 processes.
>
> Is this behavior normal ? Would the 4 processes be killed due to KEEPALIVE settings eventually ?
Yes, they almost certainly would have timed out eventually- but how
long were you already waiting..? The timeouts in pgbackrest are not
terribly long, in general.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: sfrost@snowman.net, achill@matrix.gatewaynet.com, pgsql-admin@lists.postgresql.org
Subject: Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging
In-Reply-To: <20190116202758.GG2528@tamriel.snowman.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox