Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1g2w9g-0008Sk-HZ for pgsql-admin@arkaria.postgresql.org; Thu, 20 Sep 2018 10:25:53 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1g2w9d-0006xG-II for pgsql-admin@arkaria.postgresql.org; Thu, 20 Sep 2018 10:25:49 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1g2w9c-0006x9-Vi for pgsql-admin@lists.postgresql.org; Thu, 20 Sep 2018 10:25:49 +0000 Received: from mail-oln040092255010.outbound.protection.outlook.com ([40.92.255.10] helo=APC01-HK2-obe.outbound.protection.outlook.com) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1g2w9U-0002FJ-KP for pgsql-admin@postgresql.org; Thu, 20 Sep 2018 10:25:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lL+rTVMwADvd9MI61U295p5zPOV+OgpGZCzQJBm2Jio=; b=LNlw0MpU1LcEWjXnqEF5QA61KGJEDeTh3ESiLxtrK0YpSurkYl6aOoexjGK233S8O+dQolLna92KM34KwOV5I9oWmEwqdnEOcHIZwmztQQTxknKga6GNk/BqCrvB/y7Xl50whPz34BikL8aOmqfmPHrfp8NNbdEQAzAiG/XrMFkCENLQe6ZSvt5d+DT//LZvxw5Te9Je35V4YXHZpPkgQx5FsBdXBqE1XhUsTzL+Qadj8yJJtiWx6ZXicbdqp0r7zbfkwysPPcsbBsVSudoKiL7uFxPYepHRYvp2sM994kDaHm5V1SIc74vzu3b/QxzlU6lnZUcCcCHjKlF8keli5w== Received: from SG2APC01FT012.eop-APC01.prod.protection.outlook.com (10.152.250.58) by SG2APC01HT133.eop-APC01.prod.protection.outlook.com (10.152.251.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.20.1164.13; Thu, 20 Sep 2018 10:25:34 +0000 Received: from MA1PR0101MB1687.INDPRD01.PROD.OUTLOOK.COM (10.152.250.57) by SG2APC01FT012.mail.protection.outlook.com (10.152.250.184) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.1164.13 via Frontend Transport; Thu, 20 Sep 2018 10:25:34 +0000 Received: from MA1PR0101MB1687.INDPRD01.PROD.OUTLOOK.COM ([fe80::851c:4cdc:a11f:d54b]) by MA1PR0101MB1687.INDPRD01.PROD.OUTLOOK.COM ([fe80::851c:4cdc:a11f:d54b%3]) with mapi id 15.20.1143.017; Thu, 20 Sep 2018 10:25:34 +0000 From: soumitra bhandary To: arvind chikne , "thomas.poty@gmail.com" CC: pgsql-admin Subject: Re: Master server stoped working Thread-Topic: Master server stoped working Thread-Index: AQHUUMIBcVXtB/4uhkW7wbtAAkQD06T45YuAgAACbQCAAA5Lrw== Date: Thu, 20 Sep 2018 10:25:33 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:D570BCC2DE6741192541179296129DE3FFD0EEF0DF3355B52CC0CF60070592B0;UpperCasedChecksum:F7DE927A860D150BE07DFA98D78D6DA5076A114C7E8010D4DBC9426728524E41;SizeAsReceived:7320;Count:47 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [BzI+GuJ8I81fVdUva14x/4FoGleDx6aq] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;SG2APC01HT133;6:Vj1wTH9s19s1WlAplwA8+LBw3u+V68AwIYtm1R/pohrqcPH/qPGqFe91kC0qX1EAd5s8jh5nnTdQwWCATE+9sPmkx2AQq4a2bpEMFniKvtCQXtIaRMfbmn+e2o6ey1pux+U8K/tetev5gf7NLgJj3LwYO7qxHTU9NPt9i+LERoqgs/dTU8R/CjJmi3MMhLgux+c0TGa+cxr08ncn4Z2P99E9B0WFmDUs1T/6ywd8MScggiM3YAbqzQui1Pl80QsURTmg1doyPwjHbN3/gHktifGZjOMnAi49BnbpVyeumesZ8MJVU/tGPotXES0eLlcWY9LWUX9e24Uxa+ogynBgba5DFSTPHCMt7iBZambL28osnqMTRVFZfMXVCcm1aB2Ab3douhAVVBqTHyiQ8qKgNcrBnpiC63rVWPd61dYGOepS06YY21+fYJr0wAfK0/4vUqQTF4RRAghSJDTIPlIhjw==;5:+1EByaNShiiOeT7IuGdJpW1OdtQ+RevzlPGcsp/1yKRbAEtfZ2fV7Ajctf+blkWSINTxmyUJFnqMb6A0ytvKtBblfUyTqNb+sUAn9tm/ZbAyDFFlbQT8UlgieQ0eVKYGf9oMYoPF9u1Vyk0sHRoFMGXOmv8kB+kgHcz/2FdjNQs=;7:Up/Dkiaixm679LmhtW5wJ/BSQWKDlaa9UDF+F6hrKbZMc+TnHlwUGCvd4Z8Z9LJJ6u/7vOM+0BjlpvyeFniF9TO00ESV1/QEm70IF5Wv3FgtjgxNLJVKgyatTPuvHU0Sz87wQzQjSsKXFkNEpkiv7BGsb5E+2LILUC9QsMMJKi0qumj6M8oh5oEAi6RhWbT6ng7jv5xklkK7BzERiET3dd/5Gr5aBTEfJ4wMk97wNXNerd2W/dqiDRjDJihqKgLW x-incomingheadercount: 47 x-eopattributedmessage: 0 x-microsoft-antispam: BCL:0;PCL:0;RULEID:(7020095)(201702061078)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1601125500)(1603101475)(1701031045);SRVR:SG2APC01HT133; x-ms-traffictypediagnostic: SG2APC01HT133: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(4566010)(82015058);SRVR:SG2APC01HT133;BCL:0;PCL:0;RULEID:;SRVR:SG2APC01HT133; x-forefront-prvs: 0801F2E62B x-forefront-antispam-report: SFV:NSPM;SFS:(7070007)(189003)(199004)(53754006)(2900100001)(551934003)(105004)(102836004)(83332001)(68736007)(19627405001)(6346003)(26005)(5660300001)(87572001)(20460500001)(8936002)(6506007)(53546011)(8676002)(81156014)(3480700004)(229853002)(14454004)(9686003)(86152003)(55016002)(74316002)(99286004)(86362001)(236005)(54896002)(25786009)(256004)(14444005)(76176011)(7696005)(105586002)(4326008)(73972006)(71200400001)(71190400001)(110136005)(486006)(6436002)(476003)(446003)(11346002)(33656002)(2501003)(39060400002)(106356001)(5250100002)(97736004)(82202002)(56003)(6246003)(104016004)(15852004);DIR:OUT;SFP:1901;SCL:1;SRVR:SG2APC01HT133;H:MA1PR0101MB1687.INDPRD01.PROD.OUTLOOK.COM;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: hotmail.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=soumitra.bhandary@hotmail.com; x-microsoft-antispam-message-info: fEKZ/RCatnpeAb1QxzM7Wvs8v1Jy40AdRb22T3P2I1Qnp8BHJ5rAMcHtZU05Nah0ugB83LpXiOIPKyWvnVNH7+Gp7U2Nd/vkTVGL+zaiP6UKexyLwT6Z2OZSmr/MGiInb63mUcBGdKVyMGz1br9XHw7PT0/r+lMnMTjwF7GI89CTB0NMVMCN27m+PIDFe+vlKKdXjt+3XMrvIr3WApQgdQUHktOjA4eP6RYENYpDslQ= Content-Type: multipart/alternative; boundary="_000_MA1PR0101MB16879B4DBDDB2E085D9EDDD9E0130MA1PR0101MB1687_" MIME-Version: 1.0 X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 7181d4b0-87d6-4f4e-ba33-0d3746212cec X-MS-Exchange-CrossTenant-Network-Message-Id: be306790-f675-4705-e89d-08d61ee36634 X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 7181d4b0-87d6-4f4e-ba33-0d3746212cec X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Sep 2018 10:25:33.9545 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: SG2APC01HT133 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --_000_MA1PR0101MB16879B4DBDDB2E085D9EDDD9E0130MA1PR0101MB1687_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Arvind , Can you please check the location of postgresql.conf file ? It looks like ,= it is in different location than what you have given in the command i.e. (= -D /var/lib/postgresql/9.5/main) Thanks , Soumitra ________________________________ From: arvind chikne Sent: Thursday, September 20, 2018 3:01 PM To: thomas.poty@gmail.com Cc: pgsql-admin Subject: Re: Master server stoped working Actually its seems that there is no activity captured in the /var/log/post= gresql/postgresql-9.5-main.log pls see below the log root@Master:~# tail -f /var/log/postgresql/postgresql-9.5-main.log 2018-09-19 17:32:41 IST [325-9] dhis@up HINT: In a moment you should be ab= le to reconnect to the database and repeat your command. 2018-09-19 17:32:41 IST [31038-1] dhis@dhis WARNING: terminating connectio= n because of crash of another server process 2018-09-19 17:32:41 IST [31038-2] dhis@dhis DETAIL: The postmaster has com= manded this server process to roll back the current transaction and exit, b= ecause another server process exited abnormally and possibly corrupted shar= ed memory. 2018-09-19 17:32:41 IST [31038-3] dhis@dhis HINT: In a moment you should b= e able to reconnect to the database and repeat your command. 2018-09-19 17:32:41 IST [324-49] dhis@up WARNING: terminating connection b= ecause of crash of another server process 2018-09-19 17:32:41 IST [324-50] dhis@up DETAIL: The postmaster has comman= ded this server process to roll back the current transaction and exit, beca= use another server process exited abnormally and possibly corrupted shared = memory. 2018-09-19 17:32:41 IST [324-51] dhis@up HINT: In a moment you should be a= ble to reconnect to the database and repeat your command. 2018-09-19 17:32:41 IST [27616-3] dhis@up WARNING: terminating connection = because of crash of another server process 2018-09-19 17:32:41 IST [27616-4] dhis@up DETAIL: The postmaster has comma= nded this server process to roll back the current transaction and exit, bec= ause another server process exited abnormally and possibly corrupted shared= memory. 2018-09-19 17:32:41 IST [27616-5] dhis@up HINT: In a moment you should be = able to reconnect to the database and repeat your command. On Thu, Sep 20, 2018 at 2:53 PM Thomas Poty > wrote: Hi, Could you share your log in /var/log/postgresql/postgresql-9.5-main.log fro= m server start to the end ? Le jeu. 20 sept. 2018 =E0 11:11, arvind chikne > a =E9crit : Hi All of sudden my master server goes down pls see the error below when tried= to start it Sep 20 14:33:30 Master postgresql@9.5-main[2162]: Error: could not exec /us= r/lib/postgresql/9.5/bin/pg_ctl /usr/lib/postgresql/9.5/bin/pg_ctl start -D= /var/lib/postgresql/9.5/main -l /var/log/postgresql/postgresql-9.5-main.lo= g -s -o -c config_file=3D"/etc/postgresql/9.5/main/postgresql.conf" : Sep 20 14:33:30 Master systemd[1]: postgresql@9.5-main.service: Control pro= cess exited, code=3Dexited status=3D1 Sep 20 14:33:30 Master systemd[1]: Failed to start PostgreSQL Cluster 9.5-m= ain. Sep 20 14:33:30 Master systemd[1]: postgresql@9.5-main.service: Unit entere= d failed state. Sep 20 14:33:30 Master systemd[1]: postgresql@9.5-main.service: Failed with= result 'exit-code'. -- Thanks & Regards Arvind rao chikne 9818979986 Think Green!!! Please consider your environmental responsibility before pri= nting this e-mail -- Thanks & Regards Arvind rao chikne 9818979986 Think Green!!! Please consider your environmental responsibility before pri= nting this e-mail --_000_MA1PR0101MB16879B4DBDDB2E085D9EDDD9E0130MA1PR0101MB1687_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi Arvind , 

Can you please check the location of postgresql.conf file ? It looks like , it is in different location than what you have give= n in the command i.e. ( -D /var/l= ib/postgresql/9.5/main)

Thanks , 
Soumitra 
From: arvind chikne <arv= ind.chikne@gmail.com>
Sent: Thursday, September 20, 2018 3:01 PM
To: thomas.poty@gmail.com
Cc: pgsql-admin
Subject: Re: Master server stoped working
 
Actually its seems that there is no activity captured in t= he  /var/log/postgresql/postgresql-9.5-main.log 
pls see below the log
root@Master:~# tail -f /var/log/postgresql/postgresql-9.5-main.log
2018-09-19 17:32:41 IST [325-9] dhis@up HINT:  In a moment you sh= ould be able to reconnect to the database and repeat your command.
2018-09-19 17:32:41 IST [31038-1] dhis@dhis WARNING:  terminating= connection because of crash of another server process
2018-09-19 17:32:41 IST [31038-2] dhis@dhis DETAIL:  The postmast= er has commanded this server process to roll back the current transaction a= nd exit, because another server process exited abnormally and possibly corr= upted shared memory.
2018-09-19 17:32:41 IST [31038-3] dhis@dhis HINT:  In a moment yo= u should be able to reconnect to the database and repeat your command.
2018-09-19 17:32:41 IST [324-49] dhis@up WARNING:  terminating co= nnection because of crash of another server process
2018-09-19 17:32:41 IST [324-50] dhis@up DETAIL:  The postmaster = has commanded this server process to roll back the current transaction and = exit, because another server process exited abnormally and possibly corrupt= ed shared memory.
2018-09-19 17:32:41 IST [324-51] dhis@up HINT:  In a moment you s= hould be able to reconnect to the database and repeat your command.
2018-09-19 17:32:41 IST [27616-3] dhis@up WARNING:  terminating c= onnection because of crash of another server process
2018-09-19 17:32:41 IST [27616-4] dhis@up DETAIL:  The postmaster= has commanded this server process to roll back the current transaction and= exit, because another server process exited abnormally and possibly corrup= ted shared memory.
2018-09-19 17:32:41 IST [27616-5] dhis@up HINT:  In a moment you = should be able to reconnect to the database and repeat your command.


On Thu, Sep 20, 2018 at 2:53 PM Thomas Poty <thomas.poty@gmail.com> wrote:
Hi,
Could you share your log in /var/log/postgresql/postgresql-9.5-main.lo= g from server start
 to the end ?

Le jeu. 20 sept. 2018 =E0 11:11, arvind chikne &= lt;arvind.chik= ne@gmail.com> a =E9crit :
Hi
All of sudden my master server goes down pls see the error below when = tried to start it 

Sep 20 14:33:30 Master postgresql@9.5-main[2162]: Error: could not exe= c /usr/lib/postgresql/9.5/bin/pg_ctl /usr/lib/postgresql/9.5/bin/pg_ctl sta= rt -D /var/lib/postgresql/9.5/main -l /var/log/postgresql/postgresql-9.5-ma= in.log -s -o  -c config_file=3D"/etc/postgresql/9.5/main/postgres= ql.conf" :
Sep 20 14:33:30 Master systemd[1]: postgresql@9.5-main.service: Contro= l process exited, code=3Dexited status=3D1
Sep 20 14:33:30 Master systemd[1]: Failed to start PostgreSQL Cluster = 9.5-main.
Sep 20 14:33:30 Master systemd[1]: postgresql@9.5-main.service: Unit e= ntered failed state.
Sep 20 14:33:30 Master systemd[1]: postgresql@9.5-main.service: Failed= with result 'exit-code'.


--
Thanks & Regard= s
Arvind rao chikne
9818979986=
Think Green!!!=  Please&= nbsp;con= sider yo= ur environmental responsibility before printing this e-mail



--
Thanks & Regard= s
Arvind rao chikne
9818979986=
Think Green!!!=  Please&= nbsp;con= sider yo= ur environmental responsibility before printing this e-mail

--_000_MA1PR0101MB16879B4DBDDB2E085D9EDDD9E0130MA1PR0101MB1687_--