Subject: Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from
 remote process on" and stale pgbackrest processes hanging
To: pgsql-admin@lists.postgresql.org
References: <e7d0ac3f-cf49-7d04-87b7-0944a46207e0@matrix.gatewaynet.com>
From: Achilleas Mantzios <achill@matrix.gatewaynet.com>
Message-ID: <7e5c5626-693d-9f9f-73d2-105679221fa5@matrix.gatewaynet.com>
Date: Wed, 16 Jan 2019 22:14:58 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.2.1
MIME-Version: 1.0
In-Reply-To: <e7d0ac3f-cf49-7d04-87b7-0944a46207e0@matrix.gatewaynet.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Precedence: bulk

Hello so should I post an issue to github? or just wait to see if this 
happens again?

On 16/1/19 5:24 μ.μ., Achilleas Mantzios wrote:
> Hello List,
>
> I had this error today with postgresql 10.6 and pgbackrest 2.07. I was 
> running test backups and during a backup
> The command given to the repo host (pg1-host=10.9.0.77) was :
> pgbackrest@smadb2cs:~$ pgbackrest --stanza=dynacomtest backup
> I noticed that the (two) ssh login sessions to the repo host just 
> froze. No disconnections, just froze. So some network problem surely 
> occurred.
> On the repo host pgbackrest log I got :
> 2019-01-16 12:20:38.372 P00  ERROR: [042]: unexpected EOF reading line 
> from remote process on '10.9.0.77'
> on the pg host (10.9.0.77) I noticed that the pgbackrest backup 
> processes had died, the only ones running was spawn from the 
> archive-push , however on the repo host the pgbackrest processes were 
> alive, when in fact the backup was stalled.
> On the repo I got :
> root@smadb2cs:/home/pgbackrest# netstat -apn | grep 10.9.0.77
> tcp        0     88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh
> tcp        0     88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh
> tcp        0     88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh
> tcp        0     88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh
> root@smadb2cs:/home/pgbackrest# netstat -anp --timers | grep 0.77
> tcp        0     88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 
> 11918/ssh        on (65.50/12/0)
> tcp        0     88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 
> 11920/ssh        on (69.60/12/0)
> tcp        0     88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 
> 11919/ssh        on (65.50/12/0)
> tcp        0     88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 
> 11922/ssh        on (61.40/12/0)
>
> In order to start a new backup I had to kill the 4 processes.
>
> Is this behavior normal ? Would the 4 processes be killed due to 
> KEEPALIVE settings eventually ?
>