Subject: Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations
Message-ID: <3b8edd50-6062-b3b8-4001-4f41c35e665f@amazon.com>
Date: Thu, 14 Apr 2022 18:19:24 -0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.8.0
Content-Language: en-US
To: Andres Freund <andres@anarazel.de>, Peter Geoghegan <pg@bowt.ie>
CC: Justin Pryzby <pryzby@telsasoft.com>, Robert Haas <robertmhaas@gmail.com>,
	Thomas Munro <thomas.munro@gmail.com>, Masahiko Sawada
	<sawada.mshk@gmail.com>, PostgreSQL Hackers
	<pgsql-hackers@lists.postgresql.org>
References: <20220331032844.j7wu5ijjpx3qay5c@alap3.anarazel.de>
 <CAH2-WzkvYTo4rbUi_izHOvcHNw78Ur9xLPGvj1PmDgt-csc8QQ@mail.gmail.com>
 <20220331040407.ole7orep6sj4iore@alap3.anarazel.de>
 <20220331042038.udnydrj4x45i4khb@alap3.anarazel.de>
 <CAH2-WzmN4qh04TcyNph8=1awdUN-nzza_6y+TRbEo-ECiCDV0w@mail.gmail.com>
 <20220331045915.srhbzkajlmptclhz@alap3.anarazel.de>
 <CAH2-WzkPc56XrkGqWD-zto=c_YoJCv6OD=UXqxtXRbKdFejMaw@mail.gmail.com>
 <20220331175007.lgx3huuc5gyjgk3e@alap3.anarazel.de>
 <CAH2-WzmP0awh8d2+dPd7Umq+HyJ-28HxjELzcPu7gO0G7VAvcQ@mail.gmail.com>
 <CAH2-WzkwUx++AcDByZV+TvoZddbWfvSEVGhrc4roQVGVqdHJ=w@mail.gmail.com>
 <20220403190532.qp5flagk5pyyhw4y@alap3.anarazel.de>
From: Jim Nasby <nasbyj@amazon.com>
In-Reply-To: <20220403190532.qp5flagk5pyyhw4y@alap3.anarazel.de>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/3b8edd50-6062-b3b8-4001-4f41c35e665f%40amazon.com>
Precedence: bulk


On 4/3/22 12:05 PM, Andres Freund wrote:
> While I was writing the above I, again, realized that it'd be awfully nice to
> have some accumulated stats about (auto-)vacuum's effectiveness. For us to get
> feedback about improvements more easily and for users to know what aspects
> they need to tune.
>
> Knowing how many times a table was vacuumed doesn't really tell that much, and
> requiring to enable log_autovacuum_min_duration and then aggregating those
> results is pretty painful (and version dependent).
>
> If we just collected something like:
> - number of heap passes
> - time spent heap vacuuming
> - number of index scans
> - time spent index vacuuming
> - time spent delaying
The number of passes would let you know if maintenance_work_mem is too 
small (or to stop killing 187M+ tuples in one go). The timing info would 
give you an idea of the impact of throttling.
> - percentage of non-yet-removable vs removable tuples

This'd give you an idea how bad your long-running-transaction problem is.

Another metric I think would be useful is the average utilization of 
your autovac workers. No spare workers means you almost certainly have 
tables that need vacuuming but have to wait. As a single number, it'd 
also be much easier for users to understand. I'm no stats expert, but 
one way to handle that cheaply would be to maintain an 
engineering-weighted-mean of the percentage of autovac workers that are 
in use at the end of each autovac launcher cycle (though that would 
probably not work great for people that have extreme values for launcher 
delay, or constantly muck with launcher_delay).

>
> it'd start to be a heck of a lot easier to judge how well autovacuum is
> coping.
>
> If we tracked the related pieces above in the index stats (or perhaps
> additionally there), it'd also make it easier to judge the cost of different
> indexes.
>
> - Andres
>
>