Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Subject: Re: new heapcheck contrib module
From: Mark Dilger <mark.dilger@enterprisedb.com>
In-Reply-To: <1C4CD961-84A7-4DD7-9F98-4E02B8E98ACF@yandex-team.ru>
Date: Sat, 29 Aug 2020 10:48:44 -0700
Cc: Robert Haas <robertmhaas@gmail.com>,
 Amul Sul <sulamul@gmail.com>,
 Dilip Kumar <dilipbalaut@gmail.com>,
 Peter Geoghegan <pg@bowt.ie>,
 Andres Freund <andres@anarazel.de>,
 PostgreSQL-development <pgsql-hackers@postgresql.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <38AF687F-8F6B-48B4-AB9E-A60CFD6CC261@enterprisedb.com>
References: <12ED3DA8-25F0-4B68-937D-D907CFBF08E7@enterprisedb.com>
 <20200420194210.sevnpi4lyghoanxi@alap3.anarazel.de>
 <23E60C3C-C20A-4508-84AC-E0CEA59E784A@enterprisedb.com>
 <CA+TgmobAm8vB1MNQ1taO1et+HxTN8ULnQvEG+grNDqNQTBjbtA@mail.gmail.com>
 <F845AC50-A479-43F3-ADFB-861A4D2AD287@enterprisedb.com>
 <5F743835-3399-419C-8324-2D424237E999@enterprisedb.com>
 <CAFiTN-syyHc3jZoou51v0SR8z0POoNfktqEO6MaGig4YS8mosA@mail.gmail.com>
 <E6F4258A-B661-4F2C-8A35-278C9436930A@enterprisedb.com>
 <CAFiTN-s-VX3DLfqv7OOo0zGCTOZ3xVU9UZ6GgzBZdJTPNnYFNg@mail.gmail.com>
 <885D2EC9-7D06-4903-A17D-72C184A1E1C5@enterprisedb.com>
 <CAFiTN-uYg_PiZ854=5R-D9skratapAHKhaDAG7fhJv-Vn5UhaQ@mail.gmail.com>
 <5CC85F28-F210-47AC-8DA8-EC4E82AF3FD5@enterprisedb.com>
 <CAFiTN-sP5iFJGERTq4a4OLTCzjZKhLC5P6_k2bxQKXnuvdmzGA@mail.gmail.com>
 <CAFiTN-tTg0R5AHvXC1ro9XphroX1wUtc5t91hU_+7DeGTpE5VQ@mail.gmail.com>
 <0C0349EB-DD85-4DA8-82C7-DF70AB714592@enterprisedb.com>
 <CAFiTN-uD3WSeggU20yU6voGou6BDb-cQcfkE8B1rAzWKZVK-Bw@mail.gmail.com>
 <BA6E870F-241C-4DDB-B92D-DB39E679C7D4@enterprisedb.com>
 <CA+TgmobQUdfM2Q7WAR606RR2YehCZ5GHRKBc-VWfRj-1ZUKqDA@mail.gmail.com>
 <C4A3E5AE-CBB2-4E3F-8BAE-29EE216762C2@enterprisedb.com>
 <CAAJ_b96_Xr2a+D0j8xKbOnCzURcVQm0V2DS+T_pfc+wt3BpL8g@mail.gmail.com>
 <9DF93B83-FE46-4EC4-A99D-77A60729FD2D@enterprisedb.com>
 <CA+TgmoZT3kCOaKq6=m5wHGKh_9XQDbCC1LZXL6w_+NMB-n1+mw@mail.gmail.com>
 <CAAJ_b94hsrF4x7UKQj03416fAEXJorUpZ9WhsZZjqgv-4+Wg9A@mail.gmail.com>
 <50F6468F-5A5E-412A-86EC-DF24B549D733@enterprisedb.com>
 <CAAJ_b94+A1Gk5r5sBYbYM2K5hH_gyp_o=+gwJuJhn5ZEm5_rqw@mail.gmail.com>
 <CAAJ_b94Wh_7kM9JuJk7wnNsYSNPdSzz35cOEaE_ZRnPcqvupCg@mail.gmail.com>
 <7D603B53-8078-4AF0-851B-803FF4416501@enterprisedb.com>
 <47D0EC13-0B9A-4D97-9435-0846B0DA2916@yandex-team.ru>
 <CA+TgmobgkU0U-XDcpRZvEWb-btOsiZXU3_Uf-15rge+JvqcJsg@mail.gmail.com>
 <773B38BD-E27F-4C49-8678-B316261C9189@yandex-team.ru>
 <CA+TgmoZo6BHLgZLW5n6ua72GtuMqDieYDhQRpUcFYm9rp-veDQ@mail.gmail.com>
 <1C4CD961-84A7-4DD7-9F98-4E02B8E98ACF@yandex-team.ru>
To: "Andrey M. Borodin" <x4mmm@yandex-team.ru>
Precedence: bulk


> On Aug 29, 2020, at 3:27 AM, Andrey M. Borodin <x4mmm@yandex-team.ru> =
wrote:
>=20
>=20
>=20
>> 29 =D0=B0=D0=B2=D0=B3. 2020 =D0=B3., =D0=B2 00:56, Robert Haas =
<robertmhaas@gmail.com> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB(=D0=B0)=
:
>>=20
>> On Fri, Aug 28, 2020 at 2:10 PM Andrey M. Borodin =
<x4mmm@yandex-team.ru> wrote:
>>> I don't think so. ISTM It's the same problem of xmax<relfrozenxid =
actually, just hidden behind detoasing.
>>> Our regular heap_check was checking xmin\xmax invariants for tables, =
but failed to recognise the problem in toast (while toast was accessible =
until CLOG truncation).
>>=20
>> The code can (and should, and I think does) refrain from looking up
>> XIDs that are out of the range thought to be valid -- but how do you
>> propose that it avoid looking up XIDs that ought to have clog data
>> associated with them despite being >=3D relfrozenxid and < nextxid?
>> TransactionIdDidCommit() does not have a suppress-errors flag, adding
>> one would be quite invasive, yet we cannot safely perform a
>> significant number of checks without knowing whether the inserting
>> transaction committed.
>=20
> What you write seems completely correct to me. I agree that CLOG =
thresholds lookup seems unnecessary.
>=20
> But I have a real corruption at hand (on testing site). If I have =
proposed here heapcheck. And I have pg_surgery from the thread nearby. =
Yet I cannot fix the problem, because cannot list affected tuples. These =
tools do not solve the problem neglected for long enough. It would be =
supercool if they could.
>=20
> This corruption like a caries had 3 stages:
> 1. incorrect VM flag that page do not need vacuum
> 2. xmin and xmax < relfrozenxid
> 3. CLOG truncated
>=20
> Stage 2 is curable with proposed toolset, stage 3 is not. But they are =
not that different.

I had an earlier version of the verify_heapam patch that included a =
non-throwing interface to clog.  Ultimately, I ripped that out.  My =
reasoning was that a simpler patch submission was more likely to be =
acceptable to the community.

If you want to submit a separate patch that creates a non-throwing =
version of the clog interface, and get the community to accept and =
commit it, I would seriously consider using that from verify_heapam.  If =
it gets committed in time, I might even do so for this release cycle.  =
But I don't want to make this patch dependent on that hypothetical patch =
getting written and accepted.

=E2=80=94
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company