Subject: Re: VACUUM and ANALYZE disagreeing on what reltuples means
To: Haribabu Kommi <kommi.haribabu@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>,
 "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
References: <16db4468-edfa-830a-f921-39a50498e77e@2ndquadrant.com>
 <19990.1500936945@sss.pgh.pa.us>
 <3d250597-384e-c5f4-7533-b513e366888f@2ndquadrant.com>
 <CAJrrPGeQcsBXhuJA1vQ25PS0wAqfo0v_mXtwmt4VcOmxKSvfCA@mail.gmail.com>
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Message-ID: <94f58897-861b-accf-a9f1-af4be816c0d3@2ndquadrant.com>
Date: Sun, 24 Sep 2017 20:39:03 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.3.0
MIME-Version: 1.0
In-Reply-To: 
 <CAJrrPGeQcsBXhuJA1vQ25PS0wAqfo0v_mXtwmt4VcOmxKSvfCA@mail.gmail.com>
Content-Type: multipart/mixed;
 boundary="------------EEE680DB6805783DFA634517"
Content-Language: en-US
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

This is a multi-part message in MIME format.
--------------EEE680DB6805783DFA634517
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit


On 09/06/2017 09:45 AM, Haribabu Kommi wrote:
> 
> 
> On Tue, Jul 25, 2017 at 9:33 PM, Tomas Vondra
> <tomas.vondra@2ndquadrant.com <mailto:tomas.vondra@2ndquadrant.com>> wrote:
> 
>     On 7/25/17 12:55 AM, Tom Lane wrote:
> 
>         Tomas Vondra <tomas.vondra@2ndquadrant.com
>         <mailto:tomas.vondra@2ndquadrant.com>> writes:
> 
>             It seems to me that VACUUM and ANALYZE somewhat disagree on what
>             exactly reltuples means. VACUUM seems to be thinking that
>             reltuples
>             = live + dead while ANALYZE apparently believes that reltuples =
>             live
> 
> 
>             The question is - which of the reltuples definitions is the
>             right
>             one? I've always assumed that "reltuples = live + dead" but
>             perhaps
>             not?
> 
> 
>         I think the planner basically assumes that reltuples is the live
>         tuple count, so maybe we'd better change VACUUM to get in step.
> 
> 
>     Attached is a patch that (I think) does just that. The disagreement
>     was caused by VACUUM treating recently dead tuples as live, while
>     ANALYZE treats both of those as dead.
> 
>     At first I was worried that this will negatively affect plans in the
>     long-running transaction, as it will get underestimates (due to
>     reltuples not including rows it can see). But that's a problem we
>     already have anyway, you just need to run ANALYZE in the other session.
> 
> 
> Thanks for the patch.
> From the mail, I understand that this patch tries to improve the
> reltuples value update in the catalog table by the vacuum command
> to consider the proper visible tuples similar like analyze command.
> 
> -num_tuples);
> +num_tuples - nkeep);
> 
> With the above correction, there is a problem in reporting the number
> of live tuples to the stats.
> 
> postgres=# select reltuples, n_live_tup, n_dead_tup
>               from pg_stat_user_tables join pg_class using (relname)
>              where relname = 't';
>  reltuples | n_live_tup | n_dead_tup 
> -----------+------------+------------
>     899818 |     799636 |     100182
> (1 row)
> 
> 
> The live tuples data value is again decremented with dead tuples
> value before sending them to stats in function lazy_vacuum_rel(),
> 
> /* report results to the stats collector, too */
> new_live_tuples = new_rel_tuples - vacrelstats->new_dead_tuples;
> 
> The fix needs a correction here also. Or change the correction in 
> lazy_vacuum_rel() function itself before updating catalog table similar
> like stats.
> 

Ah, haven't noticed that for some reason - you're right, we estimate the
reltuples based on (num_tuples - nkeep), so it doesn't make much sense
to subtract nkeep again. Attached is v2 of the fix.

I've removed the subtraction from lazy_vacuum_rel(), leaving just

    new_live_tuples = new_rel_tuples;

and now it behaves as expected (no second subtraction). That means we
can get rid of new_live_tuples altogether (and the protection against
negative values), and use new_rel_tuples directly.

Which pretty much means that in this case

    (pg_class.reltuples == pg_stat_user_tables.n_live_tup)

but I guess that's fine, based on the initial discussion in this thread.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--------------EEE680DB6805783DFA634517
Content-Type: text/x-patch;
 name="vacuum-reltuples-fix-v2.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="vacuum-reltuples-fix-v2.patch"

diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c
index 45b1859..1886f0d 100644
--- a/src/backend/commands/vacuumlazy.c
+++ b/src/backend/commands/vacuumlazy.c
@@ -198,7 +198,6 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params,
 	BlockNumber new_rel_pages;
 	double		new_rel_tuples;
 	BlockNumber new_rel_allvisible;
-	double		new_live_tuples;
 	TransactionId new_frozen_xid;
 	MultiXactId new_min_multi;
 
@@ -335,13 +334,9 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params,
 						false);
 
 	/* report results to the stats collector, too */
-	new_live_tuples = new_rel_tuples - vacrelstats->new_dead_tuples;
-	if (new_live_tuples < 0)
-		new_live_tuples = 0;	/* just in case */
-
 	pgstat_report_vacuum(RelationGetRelid(onerel),
 						 onerel->rd_rel->relisshared,
-						 new_live_tuples,
+						 new_rel_tuples,
 						 vacrelstats->new_dead_tuples);
 	pgstat_progress_end_command();
 
@@ -1267,7 +1262,7 @@ lazy_scan_heap(Relation onerel, int options, LVRelStats *vacrelstats,
 	vacrelstats->new_rel_tuples = vac_estimate_reltuples(onerel, false,
 														 nblocks,
 														 vacrelstats->tupcount_pages,
-														 num_tuples);
+														 num_tuples - nkeep);
 
 	/*
 	 * Release any remaining pin on visibility map page.

--------------EEE680DB6805783DFA634517
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

--------------EEE680DB6805783DFA634517--