From: Rui DeSousa <rui@crazybean.net>
To: Darron Harrison <darron@realtyserver.com>
Cc: pgsql-admin <pgsql-admin@lists.postgresql.org>
Subject: Re: Vacuum Issues
Date: Thu, 26 Mar 2020 18:26:01 -0400
Message-ID: <8D9390C5-E192-4232-B18D-BDF9F658DFEA@crazybean.net> (raw)
In-Reply-To: <17118ca7beb.bd8e3204216663.1608462906593065114@realtyserver.com>
References: <171180a0730.f301716f192269.5929166075824764633@realtyserver.com>
<8612E7A0-95FE-45DD-92BF-99FF68757BCF@crazybean.net>
<1711838a1a0.b8776dce198656.4553889266632620330@realtyserver.com>
<EB0BC3D5-EE49-4863-963F-932A5D148A2B@crazybean.net>
<171187f1f11.e7c537f6208044.4379759561125142996@realtyserver.com>
<30BC197F-1AC7-438E-BF72-AB29B5194C69@crazybean.net>
<17118ca7beb.bd8e3204216663.1608462906593065114@realtyserver.com>
> On Mar 26, 2020, at 5:40 PM, Darron Harrison <darron@realtyserver.com> wrote:
>
> There is no current lag on the replicas. Replication does traverse a firewall, but we have made no changes recently.
>
> I will say that one of the hot standbys was only recently attached, and it seems like the issues started when we began sending some longer running queries it's way. We have since placed those queries back on the master, but the vacuum issues remain.
>
> One side effect of whatever is happening, is that nightly backups are taking twice as long as normal.
It could sill be a bad/sale replication session. If the firewall drops the replication stream and does not send a reset packets (bad pratice) then the replication session might still be lingering on the Postgres server and holding on to a very old xmin. Do you know if you have TCP/IP Keepalive enabled? I don’t think in 9.2 replication sessions are listed in pg_stat_activity; thus, you’ll have to look for TCP/IP connections to the replics that should not exist.
Check all the upstream servers for stale TCP/IP replication connections; using netstat. I would also look at the system’s process list for walsender processes to see if there more more than there should be; i.e. two of them for single replica.
If you do find one; the best way to terminate it is to drop the TCP/IP connection. i.e. In FreeBSD it would be the command “tcpdrop”; for Linux there are few utilities that do same -- I just don’t recall the name of them.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: rui@crazybean.net, darron@realtyserver.com, pgsql-admin@lists.postgresql.org
Subject: Re: Vacuum Issues
In-Reply-To: <8D9390C5-E192-4232-B18D-BDF9F658DFEA@crazybean.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox