Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gjro7-0002EK-R4 for pgsql-admin@arkaria.postgresql.org; Wed, 16 Jan 2019 20:29:03 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gjrn8-0004gZ-C7 for pgsql-admin@arkaria.postgresql.org; Wed, 16 Jan 2019 20:28:02 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gjrn8-0004gS-0y for pgsql-admin@lists.postgresql.org; Wed, 16 Jan 2019 20:28:02 +0000 Received: from tamriel.snowman.net ([96.255.250.162]) by makus.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gjrn5-0001LF-3g for pgsql-admin@lists.postgresql.org; Wed, 16 Jan 2019 20:28:00 +0000 Received: by tamriel.snowman.net (Postfix, from userid 1000) id 38DB35F79E; Wed, 16 Jan 2019 15:27:58 -0500 (EST) Date: Wed, 16 Jan 2019 15:27:58 -0500 From: Stephen Frost To: Achilleas Mantzios Cc: pgsql-admin@lists.postgresql.org Subject: Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging Message-ID: <20190116202758.GG2528@tamriel.snowman.net> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="YUIvVVgc7nskgGF6" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --YUIvVVgc7nskgGF6 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Greetings, * Achilleas Mantzios (achill@matrix.gatewaynet.com) wrote: > I had this error today with postgresql 10.6 and pgbackrest 2.07. I was ru= nning test backups and during a backup > The command given to the repo host (pg1-host=3D10.9.0.77) was : > pgbackrest@smadb2cs:~$ pgbackrest --stanza=3Ddynacomtest backup > I noticed that the (two) ssh login sessions to the repo host just froze. = No disconnections, just froze. So some network problem surely occurred. > On the repo host pgbackrest log I got : > 2019-01-16 12:20:38.372 P00=A0 ERROR: [042]: unexpected EOF reading line = =66rom remote process on '10.9.0.77' > on the pg host (10.9.0.77) I noticed that the pgbackrest backup processes > had died, the only ones running was spawn from the archive-push , however= on > the repo host the pgbackrest processes were alive, when in fact the backup > was stalled. > On the repo I got : > root@smadb2cs:/home/pgbackrest# netstat -apn | grep 10.9.0.77 > tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0 88 10.9.1.22:43837 10.9.0.77:22=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ESTABLISHED 11918/ssh > tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0 88 10.9.1.22:43838 10.9.0.77:22=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ESTABLISHED 11920/ssh > tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0 88 10.9.1.22:43839 10.9.0.77:22=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ESTABLISHED 11919/ssh > tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0 88 10.9.1.22:43840 10.9.0.77:22=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ESTABLISHED 11922/ssh > root@smadb2cs:/home/pgbackrest# netstat -anp --timers | grep 0.77 > tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0 88 10.9.1.22:43837 10.9.0.77:22=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ESTABLISHED 11918/ssh=A0=A0=A0=A0=A0=A0=A0 o= n (65.50/12/0) > tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0 88 10.9.1.22:43838 10.9.0.77:22=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ESTABLISHED 11920/ssh=A0=A0=A0=A0=A0=A0=A0 o= n (69.60/12/0) > tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0 88 10.9.1.22:43839 10.9.0.77:22=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ESTABLISHED 11919/ssh=A0=A0=A0=A0=A0=A0=A0 o= n (65.50/12/0) > tcp=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0 88 10.9.1.22:43840 10.9.0.77:22=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ESTABLISHED 11922/ssh=A0=A0=A0=A0=A0=A0=A0 o= n (61.40/12/0) This looks like the kernel thinks the connections are still alive. > In order to start a new backup I had to kill the 4 processes. >=20 > Is this behavior normal ? Would the 4 processes be killed due to KEEPALIV= E settings eventually ? Yes, they almost certainly would have timed out eventually- but how long were you already waiting..? The timeouts in pgbackrest are not terribly long, in general. Thanks! Stephen --YUIvVVgc7nskgGF6 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJcP5PNAAoJEO1sijiDR2RVDWQP/RIhryH5qTSjESZAKlNbq/jJ oYuVpx8r5FLuH/JmhMlJoIk91et7m4GmM29zfCGvDRcQPRTvQeU7v2ZdHGp2NZRT z8Wk75qnTd2q53qlLGZ0Pjo81ht5kBlufy6aoRnYsLuQzbPLoYKThKkByzsp7PaX C42tPogKXUgsytxUseRBBiGmte/U30k2XJhCvnpT00HysN+L+iWkoOt7usgFCBQ3 sAL3gInPz2FXIPskWRZu79L3FcpjfJbfGpQ9XhkUpZHHRUCkJ8wqPNFzsxroj418 WohOH/dSkOaHW4biBvluckSMoS3MX1DdW7oX0TVZTxX6z5WKqdbMv498arlHBoj8 DTqYRlFOwG/wxyipX3DU+jq1KAdX8/1SHQR5V56H+aBTkAqBnsDP9XONIXbEYoRP MKmwMJliLBbUoryk3xRkx77WLezhFNWhXI0GC6EKEB1Z7BeKe2Zmq+t9EhS03IP9 eWSMNilwxZb6FGszNOJZQCaPI358JDU1yfwrDWhaTxm9CYao2bFbX/oigI5h3W4A Au3nOC1PlxszoKXlp2dybG3vUymh9xvNUccxwThtgOLIaGzd1K0npAT+CGPwC1tf hPYk4DoUbPEIPsQLRyRWWcKlFJUTEWYW+wArTgHYNBp+71zIkwjMMIEyMsPslaVp BtaVoIKmQLdqJvPRmgeY =2qiu -----END PGP SIGNATURE----- --YUIvVVgc7nskgGF6--