From: Jerry Sievers <gsievers19@comcast.net>
To: pgsql-admin@postgresql.org
Subject: Odd OS, SAN or related throughput issue affecting large streamer
Date: Tue, 23 Mar 2021 12:25:35 -0500
Message-ID: <87o8f9n5o0.fsf@jsievers.enova.com> (raw)
Greetings!
I do not know if $issue has anything to do w/Pg directly but would be
very grateful for any insights...
We've got ~20 servers on a beefy physical host that's using a
fibre-channel storage array backend.
Runtime and OS software was updated recently, and about 2 days ago, our
big monster system began lagging in replication. It generally is able
to stream WALs from the primary, but is no longer able to apply them and
the slowdown is orders of magnitute below what it was prior.
Full reboot of the host restores adequate throughput for several hours,
upon which time the backlogging resumes. We have whitnessed the reboot
as temp fix and then repeated falloff twice now on consecutive days.
System of interest is a churny, ~50TB warehouse w/4 tablespaces. I'
unclear on whether all or just some of them are sluggish on writes.
It is "replay" lag, not lag in streaming that is evident.
my SysEng team is so far unable to spot what's causing the issue.
Suggestions re where to look next?
Thx!
postgres=# select version();
version
-----------------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 11.11 (Ubuntu 11.11-1.pgdg16.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609, 64-bit
$ uname -a
Linux foobox.foocorp.com 4.15.0-139-generic #143~16.04.1-Ubuntu SMP Wed Mar 17 08:10:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
--
Jerry Sievers
Postgres DBA/Development Consulting
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: gsievers19@comcast.net
Subject: Re: Odd OS, SAN or related throughput issue affecting large streamer
In-Reply-To: <87o8f9n5o0.fsf@jsievers.enova.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox