From: Sreejith P <sreejith@lifetrenz.com>
To: Laurenz Albe <laurenz.albe@cybertec.at>,
	"pgsql-admin@lists.postgresql.org" <pgsql-admin@lists.postgresql.org>
Subject: Re: Replication failed
Thread-Topic: Replication failed
Thread-Index: ATZBRTRCbhZdQ22R9l20vUjQ9GnqXGw5OTFmvcX6N/I=
X-MS-Exchange-MessageSentRepresentingType: 1
Date: Fri, 18 Dec 2020 10:19:02 +0000
Message-ID: 
	<PS2PR02MB2837F275523C57499D567C97F3C30@PS2PR02MB2837.apcprd02.prod.outlook.com>
References: <E73F4CFB-E322-461E-B1EC-82FAA808FEE6@lifetrenz.com>
 <9492f3e4fff86b6fcb1d064d680920e8579d9c37.camel@cybertec.at>
In-Reply-To: <9492f3e4fff86b6fcb1d064d680920e8579d9c37.camel@cybertec.at>
Accept-Language: en-US
X-MS-Has-Attach: 
X-MS-Exchange-Organization-SCL: -1
X-MS-TNEF-Correlator: 
X-MS-Exchange-Organization-RecordReviewCfmType: 0
MIME-Version: 1.0
Content-Language: en-US
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
List-Id: <pgsql-admin.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-admin@lists.postgresql.org>
List-Owner: <mailto:pgsql-admin-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-admin>
Precedence: bulk

Yes.

There was a crash.

Master recovered automatically .. But replication failed.=20

See blow log from Master.=20

cp: error writing '/BackupVolume/hisDbBackup/WALs/0000000200000AE6000000B9'=
: No space left on device
2020-12-17 21:55:26 +04 [55822]: user=3D,db=3D,app=3D,client=3D LOG:  archi=
ve command failed with exit code 1
2020-12-17 21:55:26 +04 [55822]: user=3D,db=3D,app=3D,client=3D DETAIL:  Th=
e failed archive command was: test ! -f /BackupVolume/dbWALBack/WALs/000000=
0200000AE6000000B9 && cp pg_wal/0000000200000AE6000000B9 /BackupVolume/hisD=
bBackup/WALs/0000000200000AE6000000B9
=20
2020-12-17 21:53:56 +04 [84225]: user=3DAstDBA,db=3DAST-PROD,app=3D[unknown=
],client=3D172.18.200.100 HINT:  In a moment you should be able to reconnec=
t to the database and repeat your command.
2020-12-17 21:53:56 +04 [51102]: user=3DAstDBA,db=3DAST-PROD,app=3D[unknown=
],client=3D172.18.200.100 WARNING:  terminating connection because of crash=
 of another server process
2020-12-17 21:53:56 +04 [51102]: user=3DAstDBA,db=3DAST-PROD,app=3D[unknown=
],client=3D172.18.200.100 DETAIL:  The postmaster has commanded this server=
 process to roll back the current transaction and exit, because another ser=
ver process exited abnormally and possibly corrupted shared memory.
=20
=20
2020-12-17 21:55:20 +04 [55808]: user=3Dreplicator,db=3D[unknown],app=3D[un=
known],client=3D172.18.200.144 FATAL:  the database system is in recovery m=
ode
=20
=20
2020-12-17 21:55:25 +04 [55828]: user=3Dreplicator,db=3D[unknown],app=3Dwal=
receiver,client=3D172.18.200.148 ERROR:  requested starting point AEA/A2000=
000 is ahead of the WAL flush position of this server AEA/A1FFFA50
2020-12-17 21:55:25 +04 [55827]: user=3Dreplicator,db=3D[unknown],app=3Dwal=
receiver,client=3D172.18.200.147 ERROR:  requested starting point AEA/A2000=
000 is ahead of the WAL flush position of this server AEA/A1FFFA50
2020-12-17 21:55:25 +04 [55829]: user=3Dreplicator,db=3D[unknown],app=3Dwal=
receiver,client=3D172.18.200.145 ERROR:  requested starting point AEA/A2000=
000 is ahead of the WAL flush position of this server AEA/A1FFFA50
2020-12-17 21:55:25 +04 [55830]: user=3Dreplicator,db=3D[unknown],app=3Dwal=
receiver,client=3D172.18.200.146 ERROR:  requested starting point AEA/A2000=
000 is ahead of the WAL flush position of this server AEA/A1FFFA50
2020-12-17 21:55:25 +04 [55831]: user=3Dreplicator,db=3D[unknown],app=3Dwal=
receiver,client=3D172.18.200.144 ERROR:  requested starting point AEA/A2000=
000 is ahead of the WAL flush position of this server AEA/A1FFFA50


=EF=BB=BFOn 18/12/20, 2:13 PM, "Laurenz Albe" <laurenz.albe@cybertec.at> wr=
ote:

    On Fri, 2020-12-18 at 00:56 +0530, Sreejith P wrote:
    > We had 1 M x 4 slave server streaming replication using Postgres 10. =
Was working successfully for long time.
    >=20
    > Suddenly all replication servers got failed and getting following mes=
sage.
    >=20
    > 2020-12-17 22:24:32 +04 [1587]: [357-1] db=3D,user=3DLOG:  invalid co=
ntrecord length 2722 at AEA/A1FFF9E0.
    >=20
    > Requesting help for identifying root cause.

    Looks like this problem:
    https://postgr.es/m/77734732-44A4-4209-8C2F-3AF36C9D4D18%40amazon.com

    Was there a crash on the primary sever?

    There is a patch under development at this thread:
    https://postgr.es/m/CBDDFA01-6E40-46BB-9F98-9340F4379505%40amazon.com

    Yours,
    Laurenz Albe


--=20


=C2=A0

*Solutions for Care Anywhere*
*dWise HealthCare IT Solutions Pvt.=20
Ltd.* | www.lifetrenz.com <http://www.lifetrenz.com>
*Disclaimer*:
 The=20
information and attachments contained in this email are intended=20
for=20
exclusive use of the addressee(s) and may contain confidential or=20
privileged information. If you are not the intended recipient, please=20
notify the sender immediately and destroy all copies of this message and
=20
any attachments. The views expressed in this email are, unless=20
otherwise=20
stated, those of the author and not those of dWise HealthCare IT Solutions=
=20
or its management.