pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedpgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging
5+ messages / 3 participants
[nested] [flat]
* pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging
@ 2019-01-16 15:24 Achilleas Mantzios <achill@matrix.gatewaynet.com>
0 siblings, 2 replies; 5+ messages in thread
From: Achilleas Mantzios @ 2019-01-16 15:24 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hello List,
I had this error today with postgresql 10.6 and pgbackrest 2.07. I was running test backups and during a backup
The command given to the repo host (pg1-host=10.9.0.77) was :
pgbackrest@smadb2cs:~$ pgbackrest --stanza=dynacomtest backup
I noticed that the (two) ssh login sessions to the repo host just froze. No disconnections, just froze. So some network problem surely occurred.
On the repo host pgbackrest log I got :
2019-01-16 12:20:38.372 P00 ERROR: [042]: unexpected EOF reading line from remote process on '10.9.0.77'
on the pg host (10.9.0.77) I noticed that the pgbackrest backup processes had died, the only ones running was spawn from the archive-push , however on the repo host the pgbackrest processes were
alive, when in fact the backup was stalled.
On the repo I got :
root@smadb2cs:/home/pgbackrest# netstat -apn | grep 10.9.0.77
tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh
tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh
tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh
tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh
root@smadb2cs:/home/pgbackrest# netstat -anp --timers | grep 0.77
tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh on (65.50/12/0)
tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh on (69.60/12/0)
tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh on (65.50/12/0)
tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh on (61.40/12/0)
In order to start a new backup I had to kill the 4 processes.
Is this behavior normal ? Would the 4 processes be killed due to KEEPALIVE settings eventually ?
--
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging
@ 2019-01-16 20:14 Achilleas Mantzios <achill@matrix.gatewaynet.com>
parent: Achilleas Mantzios <achill@matrix.gatewaynet.com>
1 sibling, 0 replies; 5+ messages in thread
From: Achilleas Mantzios @ 2019-01-16 20:14 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hello so should I post an issue to github? or just wait to see if this
happens again?
On 16/1/19 5:24 μ.μ., Achilleas Mantzios wrote:
> Hello List,
>
> I had this error today with postgresql 10.6 and pgbackrest 2.07. I was
> running test backups and during a backup
> The command given to the repo host (pg1-host=10.9.0.77) was :
> pgbackrest@smadb2cs:~$ pgbackrest --stanza=dynacomtest backup
> I noticed that the (two) ssh login sessions to the repo host just
> froze. No disconnections, just froze. So some network problem surely
> occurred.
> On the repo host pgbackrest log I got :
> 2019-01-16 12:20:38.372 P00 ERROR: [042]: unexpected EOF reading line
> from remote process on '10.9.0.77'
> on the pg host (10.9.0.77) I noticed that the pgbackrest backup
> processes had died, the only ones running was spawn from the
> archive-push , however on the repo host the pgbackrest processes were
> alive, when in fact the backup was stalled.
> On the repo I got :
> root@smadb2cs:/home/pgbackrest# netstat -apn | grep 10.9.0.77
> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh
> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh
> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh
> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh
> root@smadb2cs:/home/pgbackrest# netstat -anp --timers | grep 0.77
> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED
> 11918/ssh on (65.50/12/0)
> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED
> 11920/ssh on (69.60/12/0)
> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED
> 11919/ssh on (65.50/12/0)
> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED
> 11922/ssh on (61.40/12/0)
>
> In order to start a new backup I had to kill the 4 processes.
>
> Is this behavior normal ? Would the 4 processes be killed due to
> KEEPALIVE settings eventually ?
>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging
@ 2019-01-16 20:27 Stephen Frost <sfrost@snowman.net>
parent: Achilleas Mantzios <achill@matrix.gatewaynet.com>
1 sibling, 2 replies; 5+ messages in thread
From: Stephen Frost @ 2019-01-16 20:27 UTC (permalink / raw)
To: Achilleas Mantzios <achill@matrix.gatewaynet.com>; +Cc: pgsql-admin@lists.postgresql.org
Greetings,
* Achilleas Mantzios (achill@matrix.gatewaynet.com) wrote:
> I had this error today with postgresql 10.6 and pgbackrest 2.07. I was running test backups and during a backup
> The command given to the repo host (pg1-host=10.9.0.77) was :
> pgbackrest@smadb2cs:~$ pgbackrest --stanza=dynacomtest backup
> I noticed that the (two) ssh login sessions to the repo host just froze. No disconnections, just froze. So some network problem surely occurred.
> On the repo host pgbackrest log I got :
> 2019-01-16 12:20:38.372 P00 ERROR: [042]: unexpected EOF reading line from remote process on '10.9.0.77'
> on the pg host (10.9.0.77) I noticed that the pgbackrest backup processes
> had died, the only ones running was spawn from the archive-push , however on
> the repo host the pgbackrest processes were alive, when in fact the backup
> was stalled.
> On the repo I got :
> root@smadb2cs:/home/pgbackrest# netstat -apn | grep 10.9.0.77
> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh
> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh
> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh
> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh
> root@smadb2cs:/home/pgbackrest# netstat -anp --timers | grep 0.77
> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh on (65.50/12/0)
> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh on (69.60/12/0)
> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh on (65.50/12/0)
> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh on (61.40/12/0)
This looks like the kernel thinks the connections are still alive.
> In order to start a new backup I had to kill the 4 processes.
>
> Is this behavior normal ? Would the 4 processes be killed due to KEEPALIVE settings eventually ?
Yes, they almost certainly would have timed out eventually- but how
long were you already waiting..? The timeouts in pgbackrest are not
terribly long, in general.
Thanks!
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging
@ 2019-01-17 05:20 Achilleas Mantzios <achill@matrix.gatewaynet.com>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 0 replies; 5+ messages in thread
From: Achilleas Mantzios @ 2019-01-17 05:20 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-admin@lists.postgresql.org
Hello Stephen
On 16/1/19 10:27 μ.μ., Stephen Frost wrote:
> Greetings,
>
> * Achilleas Mantzios (achill@matrix.gatewaynet.com) wrote:
>> I had this error today with postgresql 10.6 and pgbackrest 2.07. I was running test backups and during a backup
>> The command given to the repo host (pg1-host=10.9.0.77) was :
>> pgbackrest@smadb2cs:~$ pgbackrest --stanza=dynacomtest backup
>> I noticed that the (two) ssh login sessions to the repo host just froze. No disconnections, just froze. So some network problem surely occurred.
>> On the repo host pgbackrest log I got :
>> 2019-01-16 12:20:38.372 P00 ERROR: [042]: unexpected EOF reading line from remote process on '10.9.0.77'
>> on the pg host (10.9.0.77) I noticed that the pgbackrest backup processes
>> had died, the only ones running was spawn from the archive-push , however on
>> the repo host the pgbackrest processes were alive, when in fact the backup
>> was stalled.
>> On the repo I got :
>> root@smadb2cs:/home/pgbackrest# netstat -apn | grep 10.9.0.77
>> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh
>> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh
>> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh
>> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh
>> root@smadb2cs:/home/pgbackrest# netstat -anp --timers | grep 0.77
>> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh on (65.50/12/0)
>> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh on (69.60/12/0)
>> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh on (65.50/12/0)
>> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh on (61.40/12/0)
> This looks like the kernel thinks the connections are still alive.
>
>> In order to start a new backup I had to kill the 4 processes.
>>
>> Is this behavior normal ? Would the 4 processes be killed due to KEEPALIVE settings eventually ?
> Yes, they almost certainly would have timed out eventually- but how
> long were you already waiting..? The timeouts in pgbackrest are not
> terribly long, in general.
Looking at the log I waited 9 minutes, before manually killing.
>
> Thanks!
>
> Stephen
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging
@ 2019-01-17 06:50 David Steele <david@pgmasters.net>
parent: Stephen Frost <sfrost@snowman.net>
1 sibling, 0 replies; 5+ messages in thread
From: David Steele @ 2019-01-17 06:50 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; Achilleas Mantzios <achill@matrix.gatewaynet.com>; +Cc: pgsql-admin@lists.postgresql.org
On 1/16/19 10:27 PM, Stephen Frost wrote:
> * Achilleas Mantzios (achill@matrix.gatewaynet.com) wrote:
>>
>> Is this behavior normal ? Would the 4 processes be killed due to KEEPALIVE settings eventually ?
>
> Yes, they almost certainly would have timed out eventually- but how
> long were you already waiting..? The timeouts in pgbackrest are not
> terribly long, in general.
By default the remote processes will shutdown after 30 minutes if they
have not gotten a message from the main process. This is controlled by
the `protocol-timeout` option.
It's a bit unusual for them to get stuck like this -- usually they get
an eof and exit immediately. Definitely looks like something was up
with SSH.
Regards,
--
-David
david@pgmasters.net
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2019-01-17 06:50 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-01-16 15:24 pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging Achilleas Mantzios <achill@matrix.gatewaynet.com>
2019-01-16 20:14 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2019-01-16 20:27 ` Stephen Frost <sfrost@snowman.net>
2019-01-17 05:20 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2019-01-17 06:50 ` David Steele <david@pgmasters.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox