Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lXAPP-0007Cd-3p for pgsql-admin@arkaria.postgresql.org; Thu, 15 Apr 2021 22:24:23 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lXAPM-0001hs-M1 for pgsql-admin@arkaria.postgresql.org; Thu, 15 Apr 2021 22:24:20 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lXAPM-0001gt-DY for pgsql-admin@lists.postgresql.org; Thu, 15 Apr 2021 22:24:20 +0000 Received: from gmsmail.gms.ca ([207.195.127.194] helo=mail.gms.ca) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from ) id 1lXAPI-0008Uz-La for pgsql-admin@lists.postgresql.org; Thu, 15 Apr 2021 22:24:19 +0000 Received: from gmsEXCH1201.gms.ca (172.16.87.40) by gmsEXCH1201.gms.ca (172.16.87.40) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 15 Apr 2021 16:24:13 -0600 Received: from gmsEXCH1201.gms.ca ([172.16.87.40]) by gmsEXCH1201.gms.ca ([172.16.87.40]) with mapi id 15.00.1497.012; Thu, 15 Apr 2021 16:24:13 -0600 From: Oswald Hirschmueller To: "pgsql-admin@lists.postgresql.org" Subject: postgres cluster becomes unhealthy after draining one of the worker nodes Thread-Topic: postgres cluster becomes unhealthy after draining one of the worker nodes Thread-Index: AdcyRd72DXzuUynfSHirvz9F0L9kkw== Date: Thu, 15 Apr 2021 22:24:12 +0000 Message-ID: <71de8e84252d4becbb76b622787b5251@gmsEXCH1201.gms.ca> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [172.16.6.78] Content-Type: multipart/alternative; boundary="_000_71de8e84252d4becbb76b622787b5251gmsEXCH1201gmsca_" MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --_000_71de8e84252d4becbb76b622787b5251gmsEXCH1201gmsca_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable We are trying to upgrade the kubernetes worker nodes, but encountering an i= ssue with the postgres clusters. As soon as our system administrator drains= one of the nodes, the postgres cluster becomes unhealthy. We expected anot= her node to take over as the master, but this did not occur. Note that the = replicas are running in synchronous mode. This is the command the system adminstrator ran. The issue occurs shortly a= fter. kubectl drain $NODENAME --delete-local-data --ignore-daemonsets Here are snippets of the logs of the master and one of the replicas (the lo= g of the second replica looks the same): Master Log ---------- 021-04-15 20:44:29,986 INFO: no action. i am the leader with the lock 2021-04-15 20:44:39,907 INFO: Lock owner: dbo-dhog-5b7dc865b4-lg7c9; I am d= bo-dhog-5b7dc865b4-lg7c9 (the master was on the node that was drained) /tmp:5432 - no response 2021-04-15 20:47:13.940 UTC [186] LOG: pgaudit extension initialized 2021-04-15 20:47:13.942 UTC [186] LOG: starting PostgreSQL 12.5 on x86_64-p= c-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bi= t 2021-04-15 20:47:13.942 UTC [186] LOG: listening on IPv4 address "0.0.0.0",= port 5432 2021-04-15 20:47:14.105 UTC [186] LOG: listening on Unix socket "/tmp/.s.PG= SQL.5432" 2021-04-15 20:47:14.277 UTC [186] LOG: listening on Unix socket "/crunchyad= m/.s.PGSQL.5432" 2021-04-15 20:47:14.321 UTC [186] LOG: redirecting log output to logging co= llector process 2021-04-15 20:47:14.321 UTC [186] HINT: Future log output will appear in di= rectory "pg_log". /tmp:5432 - rejecting connections /tmp:5432 - rejecting connections /tmp:5432 - rejecting connections /tmp:5432 - accepting connections 2021-04-15 20:47:16,996 INFO: establishing a new patroni connection to the = postgres cluster 2021-04-15 20:47:17,105 INFO: following a different leader because i am not= the healthiest node Thu Apr 15 20:47:17 UTC 2021 INFO: Node dbo-dhog-5b7dc865b4-d8pv4 fully ini= tialized for cluster dbo and is ready for use 2021-04-15 20:47:27,511 INFO: following a different leader because i am not= the healthiest node 2021-04-15 20:47:37,516 INFO: following a different leader because i am not= the healthiest node 2021-04-15 20:47:47,511 INFO: following a different leader because i am not= the healthiest node Replica Log ----------- 2021-04-15 20:46:16.618 UTC [28158] LOG: listening on IPv4 address "0.0.0.0= ", port 5432 /tmp:5432 - no response 2021-04-15 20:46:16.655 UTC [28158] LOG: listening on Unix socket "/tmp/.s.= PGSQL.5432" 2021-04-15 20:46:16.697 UTC [28158] LOG: listening on Unix socket "/crunchy= adm/.s.PGSQL.5432" 2021-04-15 20:46:16.764 UTC [28158] LOG: redirecting log output to logging = collector process 2021-04-15 20:46:16.764 UTC [28158] HINT: Future log output will appear in = directory "pg_log". 2021-04-15 20:46:17,061 INFO: Lock owner: None; I am dbo-ffvg-6c5d7c576c-8t= cwt 2021-04-15 20:46:17,061 INFO: not healthy enough for leader race 2021-04-15 20:46:17,061 INFO: changing primary_conninfo and restarting in p= rogress /tmp:5432 - rejecting connections /tmp:5432 - rejecting connections /tmp:5432 - accepting connections 2021-04-15 20:46:18,729 INFO: establishing a new patroni connection to the = postgres cluster 2021-04-15 20:46:18,871 INFO: following a different leader because i am not= the healthiest node 2021-04-15 20:46:29,245 INFO: following a different leader because i am not= the healthiest node 2021-04-15 20:46:39,248 INFO: following a different leader because i am not= the healthiest node 2021-04-15 20:46:49,249 INFO: following a different leader because i am not= the healthiest node 2021-04-15 20:46:59,247 INFO: following a different leader because i am not= the healthiest node What we can do to prevent this situation from occurring? We do not want thi= s to occur when we upgrade our production nodes. Thanks, Os . --_000_71de8e84252d4becbb76b622787b5251gmsEXCH1201gmsca_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

We are trying to upgrade the kubernetes worker nodes= , but encountering an issue with the postgres clusters. As soon as our syst= em administrator drains one of the nodes, the postgres cluster becomes unhe= althy. We expected another node to take over as the master, but this did not occur. Note that the replicas ar= e running in synchronous mode.

 

This is the command the system adminstrator ran. The= issue occurs shortly after.

kubectl drain $NODENAME --delete-local-data --ignore= -daemonsets

 

Here are snippets of the logs of the master and one = of the replicas (the log of the second replica looks the same):<= /p>

 

Master Log

----------

021-04-15 20:44:29,986 INFO: no action. i am the lea= der with the lock

2021-04-15 20:44:39,907 INFO: Lock owner: dbo-dhog-5= b7dc865b4-lg7c9; I am dbo-dhog-5b7dc865b4-lg7c9

(the master was on the node that was drained)

/tmp:5432 - no response

2021-04-15 20:47:13.940 UTC [186] LOG: pgaudit exten= sion initialized

2021-04-15 20:47:13.942 UTC [186] LOG: starting Post= greSQL 12.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (R= ed Hat 4.8.5-39), 64-bit

2021-04-15 20:47:13.942 UTC [186] LOG: listening on = IPv4 address "0.0.0.0", port 5432

2021-04-15 20:47:14.105 UTC [186] LOG: listening on = Unix socket "/tmp/.s.PGSQL.5432"

2021-04-15 20:47:14.277 UTC [186] LOG: listening on = Unix socket "/crunchyadm/.s.PGSQL.5432"

2021-04-15 20:47:14.321 UTC [186] LOG: redirecting l= og output to logging collector process

2021-04-15 20:47:14.321 UTC [186] HINT: Future log o= utput will appear in directory "pg_log".

/tmp:5432 - rejecting connections

/tmp:5432 - rejecting connections

/tmp:5432 - rejecting connections

/tmp:5432 - accepting connections

2021-04-15 20:47:16,996 INFO: establishing a new pat= roni connection to the postgres cluster

2021-04-15 20:47:17,105 INFO: following a different = leader because i am not the healthiest node

Thu Apr 15 20:47:17 UTC 2021 INFO: Node dbo-dhog-5b7= dc865b4-d8pv4 fully initialized for cluster dbo and is ready for use

2021-04-15 20:47:27,511 INFO: following a different = leader because i am not the healthiest node

2021-04-15 20:47:37,516 INFO: following a different = leader because i am not the healthiest node

2021-04-15 20:47:47,511 INFO: following a different = leader because i am not the healthiest node

 

Replica Log

-----------

2021-04-15 20:46:16.618 UTC [28158] LOG: listening o= n IPv4 address "0.0.0.0", port 5432

/tmp:5432 - no response

2021-04-15 20:46:16.655 UTC [28158] LOG: listening o= n Unix socket "/tmp/.s.PGSQL.5432"

2021-04-15 20:46:16.697 UTC [28158] LOG: listening o= n Unix socket "/crunchyadm/.s.PGSQL.5432"

2021-04-15 20:46:16.764 UTC [28158] LOG: redirecting= log output to logging collector process

2021-04-15 20:46:16.764 UTC [28158] HINT: Future log= output will appear in directory "pg_log".

2021-04-15 20:46:17,061 INFO: Lock owner: None; I am= dbo-ffvg-6c5d7c576c-8tcwt

2021-04-15 20:46:17,061 INFO: not healthy enough for= leader race

2021-04-15 20:46:17,061 INFO: changing primary_conni= nfo and restarting in progress

/tmp:5432 - rejecting connections

/tmp:5432 - rejecting connections

/tmp:5432 - accepting connections

2021-04-15 20:46:18,729 INFO: establishing a new pat= roni connection to the postgres cluster

2021-04-15 20:46:18,871 INFO: following a different = leader because i am not the healthiest node

2021-04-15 20:46:29,245 INFO: following a different = leader because i am not the healthiest node

2021-04-15 20:46:39,248 INFO: following a different = leader because i am not the healthiest node

2021-04-15 20:46:49,249 INFO: following a different = leader because i am not the healthiest node

2021-04-15 20:46:59,247 INFO: following a different = leader because i am not the healthiest node

 

What we can do to prevent this situation from occurr= ing? We do not want this to occur when we upgrade our production nodes.

 

Thanks,

Os

.

 

--_000_71de8e84252d4becbb76b622787b5251gmsEXCH1201gmsca_--