pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedpostgresql lost connection to repmgr arbitrarily
4+ messages / 2 participants
[nested] [flat]
* postgresql lost connection to repmgr arbitrarily
@ 2023-10-17 06:55 Zhaoxun Yan <yan.zhaoxun@gmail.com>
0 siblings, 1 reply; 4+ messages in thread
From: Zhaoxun Yan @ 2023-10-17 06:55 UTC (permalink / raw)
To: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi!
It happens from time to time. At first I thought it was the router problem,
so I changed the host in repmgr's configuration from its intranet address
to 127.0.0.1, but it persists. Here is what happened according to repmgrd:
2023-10-17 00:21:50+0800: repmgrd_local_disconnect on node2, unable to
connect to local node - happened
2023-10-17 00:21:50.471347+08: repmgrd_local_reconnect on node2,
reconnected to local node after 0 seconds - happened
2023-10-17 10:23:05+0800: repmgrd_local_disconnect on node2, unable to
connect to local node - happened
2023-10-17 10:23:05.473997+08: repmgrd_local_reconnect on node2,
reconnected to local node after 0 seconds - happened
2023-10-17 13:22:23+0800: repmgrd_local_disconnect on node2, unable to
connect to local node - happened
2023-10-17 13:22:23.552278+08: repmgrd_local_reconnect on node2,
reconnected to local node after 0 seconds - happened
So I get to the postgresql side and its log reports:
2023-10-17 00:21:50.415 CST [2210264] LOG: could not receive data from
client: Connection reset by peer
2023-10-17 10:23:05.420 CST [2249257] LOG: could not receive data from
client: Connection reset by peer
2023-10-17 13:22:23.486 CST [2260546] LOG: could not receive data from
client: Connection reset by peer
I have set up keepalive feature in postgresql.conf to prevent router from
cutting off TCP connection:
tcp_keepalives_idle = 20
tcp_keepalives_interval = 10
tcp_keepalives_count = 3
So do you have any idea on what went wrong? BTW, postgresql version is 15.4
while repmgr version is 5.4dev.
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: postgresql lost connection to repmgr arbitrarily
@ 2023-10-17 13:39 Scott Ribe <scott_ribe@elevated-dev.com>
parent: Zhaoxun Yan <yan.zhaoxun@gmail.com>
0 siblings, 1 reply; 4+ messages in thread
From: Scott Ribe @ 2023-10-17 13:39 UTC (permalink / raw)
To: Zhaoxun Yan <yan.zhaoxun@gmail.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
I have no idea if this is related to your problem, but...
I once had a connection timeout where a big institution was using Cisco routers, which charged ongoing license fees, tiered by how many connections they would support. And they configured them to recognize keepalive packets, and drop connections which only had keepalive packets for some length of time!
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: postgresql lost connection to repmgr arbitrarily
@ 2023-10-18 02:25 Zhaoxun Yan <yan.zhaoxun@gmail.com>
parent: Scott Ribe <scott_ribe@elevated-dev.com>
0 siblings, 1 reply; 4+ messages in thread
From: Zhaoxun Yan @ 2023-10-18 02:25 UTC (permalink / raw)
To: Scott Ribe <scott_ribe@elevated-dev.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi Scott,
To avoid the problem you mentioned, I have already changed the host
address to 127.0.0.1, meaning 'localhost', and the connection is only on
that machine, without via a router.
On Tue, Oct 17, 2023 at 9:39 PM Scott Ribe <scott_ribe@elevated-dev.com>
wrote:
> I have no idea if this is related to your problem, but...
>
> I once had a connection timeout where a big institution was using Cisco
> routers, which charged ongoing license fees, tiered by how many connections
> they would support. And they configured them to recognize keepalive
> packets, and drop connections which only had keepalive packets for some
> length of time!
^ permalink raw reply [nested|flat] 4+ messages in thread
* Fwd: postgresql lost connection to repmgr arbitrarily
@ 2023-10-18 09:41 Zhaoxun Yan <yan.zhaoxun@gmail.com>
parent: Zhaoxun Yan <yan.zhaoxun@gmail.com>
0 siblings, 0 replies; 4+ messages in thread
From: Zhaoxun Yan @ 2023-10-18 09:41 UTC (permalink / raw)
To: Pgsql-admin <pgsql-admin@lists.postgresql.org>
I found the postgresql process with this line:
postgres: rep repmgr 172.17.1.2(60490) idle
It represents the TCP connection from local address 172.17.1.2:60490 and
was labeled as "idle"
I checked a local connection to 172.17.1.2, which is the address of eth0,
It is a loopback connection just like localhost:
[root@yzx2 ~]# tracepath 172.17.1.2
1: yzx2 0.057ms reached
Resume: pmtu 65535 hops 1 back 1
Thus no router is involved in the repmgr-postgresql connection otherwise
mtu<=1500
Does the "idle" label mean something?
Forwarded Conversation
Subject: postgresql lost connection to repmgr arbitrarily
------------------------
From: Zhaoxun Yan <yan.zhaoxun@gmail.com>
Date: Tue, Oct 17, 2023 at 2:55 PM
To: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi!
It happens from time to time. At first I thought it was the router problem,
so I changed the host in repmgr's configuration from its intranet address
to 127.0.0.1, but it persists. Here is what happened according to repmgrd:
2023-10-17 00:21:50+0800: repmgrd_local_disconnect on node2, unable to
connect to local node - happened
2023-10-17 00:21:50.471347+08: repmgrd_local_reconnect on node2,
reconnected to local node after 0 seconds - happened
2023-10-17 10:23:05+0800: repmgrd_local_disconnect on node2, unable to
connect to local node - happened
2023-10-17 10:23:05.473997+08: repmgrd_local_reconnect on node2,
reconnected to local node after 0 seconds - happened
2023-10-17 13:22:23+0800: repmgrd_local_disconnect on node2, unable to
connect to local node - happened
2023-10-17 13:22:23.552278+08: repmgrd_local_reconnect on node2,
reconnected to local node after 0 seconds - happened
So I get to the postgresql side and its log reports:
2023-10-17 00:21:50.415 CST [2210264] LOG: could not receive data from
client: Connection reset by peer
2023-10-17 10:23:05.420 CST [2249257] LOG: could not receive data from
client: Connection reset by peer
2023-10-17 13:22:23.486 CST [2260546] LOG: could not receive data from
client: Connection reset by peer
I have set up keepalive feature in postgresql.conf to prevent router from
cutting off TCP connection:
tcp_keepalives_idle = 20
tcp_keepalives_interval = 10
tcp_keepalives_count = 3
So do you have any idea on what went wrong? BTW, postgresql version is 15.4
while repmgr version is 5.4dev.
----------
From: Scott Ribe <scott_ribe@elevated-dev.com>
Date: Tue, Oct 17, 2023 at 9:39 PM
To: Zhaoxun Yan <yan.zhaoxun@gmail.com>
Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
I have no idea if this is related to your problem, but...
I once had a connection timeout where a big institution was using Cisco
routers, which charged ongoing license fees, tiered by how many connections
they would support. And they configured them to recognize keepalive
packets, and drop connections which only had keepalive packets for some
length of time!
----------
From: Zhaoxun Yan <yan.zhaoxun@gmail.com>
Date: Wed, Oct 18, 2023 at 10:25 AM
To: Scott Ribe <scott_ribe@elevated-dev.com>
Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi Scott,
To avoid the problem you mentioned, I have already changed the host
address to 127.0.0.1, meaning 'localhost', and the connection is only on
that machine, without via a router.
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2023-10-18 09:41 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-10-17 06:55 postgresql lost connection to repmgr arbitrarily Zhaoxun Yan <yan.zhaoxun@gmail.com>
2023-10-17 13:39 ` Scott Ribe <scott_ribe@elevated-dev.com>
2023-10-18 02:25 ` Zhaoxun Yan <yan.zhaoxun@gmail.com>
2023-10-18 09:41 ` Zhaoxun Yan <yan.zhaoxun@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox