From: Tom Lane <tgl@sss.pgh.pa.us>
To: Mark Dilger <mark.dilger@enterprisedb.com>
cc: Robert Haas <robertmhaas@gmail.com>, Noah Misch <noah@leadboat.com>,
        Peter Geoghegan <pg@bowt.ie>, Thomas Munro <thomas.munro@gmail.com>,
        "Andrey M. Borodin" <x4mmm@yandex-team.ru>,
        Stephen Frost <sfrost@snowman.net>,
        Michael Paquier <michael@paquier.xyz>, Amul Sul <sulamul@gmail.com>,
        Dilip Kumar <dilipbalaut@gmail.com>,
        Andres Freund <andres@anarazel.de>,
        PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: pg_amcheck contrib application
In-reply-to: <4A42A410-AA92-4C72-BCF9-CCBA6DAB70EF@enterprisedb.com>
References: <A80D68F6-E38F-482D-9522-E2FB6AAFE8A1@enterprisedb.com>
 <CA+TgmoY3_U=QuQWo4DArBSp+tg2MiCHLwDN_9U2H_iTxj3UcjA@mail.gmail.com>
 <9E76E46A-48B2-4869-BD0C-422204C1F767@enterprisedb.com>
 <F0A1FD70-A2F4-4528-8A03-8650CAEC0554@enterprisedb.com>
 <20210313184612.GA3195145@rfd.leadboat.com>
 <930621E5-5D10-456D-8CE4-CF629A6C4669@enterprisedb.com>
 <20210313191841.GB3195145@rfd.leadboat.com>
 <3012106.1615827854@sss.pgh.pa.us>
 <C2C142F3-F740-4197-B507-20B28B7395AC@enterprisedb.com>
 <3053552.1615834640@sss.pgh.pa.us>
 <20210316060913.GA3323238@rfd.leadboat.com>
 <7BCE9084-9381-471F-90F2-FAD99B7FE3DB@enterprisedb.com>
 <3254547.1615910855@sss.pgh.pa.us>
 <7CFAF7EF-4EA7-4CB6-B2C7-EC7F4512AE50@enterprisedb.com>
 <16B79A18-48F3-404D-8DC5-53528E85A7AC@enterprisedb.com>
 <CA+TgmoYFwii3cgfc-yNzEgFce=zWP08Nt56pAyffqpXjgg1UxA@mail.gmail.com>
 <3360174.1615916917@sss.pgh.pa.us>
 <4A42A410-AA92-4C72-BCF9-CCBA6DAB70EF@enterprisedb.com>
Comments: In-reply-to Mark Dilger <mark.dilger@enterprisedb.com>
	message dated "Tue, 16 Mar 2021 11:07:16 -0700"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3377688.1615918954.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Tue, 16 Mar 2021 14:22:34 -0400
Message-ID: <3377689.1615918954@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/3377689.1615918954%40sss.pgh.pa.us>
Precedence: bulk

Mark Dilger <mark.dilger@enterprisedb.com> writes:
> On Mar 16, 2021, at 10:48 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> (Too bad the message doesn't report the
>> TOAST OID it probed for, so we can see if that's sane or not.)

> I've added that and now get the toast pointer's va_valueid in the messag=
e:

> heap table "postgres"."pg_catalog"."pg_statistic", block 4, offset 2, at=
tribute 29:
>     toasted value id 13227 for attribute 29 missing from toast table
> heap table "postgres"."pg_catalog"."pg_statistic", block 4, offset 5, at=
tribute 27:
>     toasted value id 13228 for attribute 27 missing from toast table

That's awfully interesting, because OIDs less than 16384 should
only be generated during initdb.  So what we seem to be looking at
here is pg_statistic entries that were made during the ANALYZE
done by initdb (cf. vacuum_db()), and then were deleted during
a later auto-analyze (in the buildfarm) or deliberate analyze
(in your reproducer).  But if they're deleted, why is amcheck
looking for them?

I'm circling back around to the idea that amcheck is trying to
validate TOAST references that are already dead, and it's getting
burnt because something-or-other has already removed the toast
rows, though not the referencing datums.  That's legal behavior
once the rows are marked dead.  Upthread it was claimed that
amcheck isn't doing that, but this looks like a smoking gun to me.

			regards, tom lane