Date: Sun, 10 Jul 2022 09:05:20 -0500
From: Justin Pryzby <pryzby@telsasoft.com>
To: Nathan Ward <lists+postgresql@daork.net>
Cc: pgsql-performance@lists.postgresql.org
Subject: Re: Occasional performance issue after changing table partitions
Message-ID: <20220710140519.GN13040@telsasoft.com>
References: <24A451C3-1F45-48ED-B747-4BA17C249AF1@daork.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <24A451C3-1F45-48ED-B747-4BA17C249AF1@daork.net>
User-Agent: Mutt/1.9.4 (2018-02-28)
Archived-At: 
 <https://www.postgresql.org/message-id/20220710140519.GN13040%40telsasoft.com>
Precedence: bulk

On Sun, Jul 10, 2022 at 04:55:34PM +1200, Nathan Ward wrote:
> I am running Postgres 13 on CentOS 7, installed from the yum.postgresql.org <http://yum.postgresql.org/> repo.

It doesn't sound relevant, but what kind of storage systems is postgres using ?
Filesystem, raid, device.

Is the high CPU use related to to autovacuum/autoanalyze ?

> The issue I am having, is that when the daily data usage aggregation runs, sometimes we have a big performance impact, with the following characteristics which happen *after* the aggregation job runs in it usual fast time of 12s or so:
> - The aggregation runs fast as per normal
> - Load on the server goes to 30-40 - recall we have quite high “max connections” to keep throughput high when the client is far (16ms) from the server

I suggest to install and enable autoexplain to see what's running slowly here,
and what its query plans are.  It seems possible that when the daily
aggregation script drops the old partitions, the plan changes for the worse.
I'm not sure what the fix is - maybe you just need to run vacuum or analyze on
the new partitions soon after populating them.

For good measure, also set log_autovacuum_min_duration=0 (or something other
than -1) (and while you're at it, log_checkpoints=on, and log_lock_waits=on if
you haven't already).

Note that postgres doesn't automatically analyze parent tables, so you should
maybe do that whenever the data changes enough for it to matter.

-- 
Justin