pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Mark Dilger <mark.dilger@enterprisedb.com>
To: Andrey M. Borodin <x4mmm@yandex-team.ru>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Amul Sul <sulamul@gmail.com>
Cc: Dilip Kumar <dilipbalaut@gmail.com>
Cc: Peter Geoghegan <pg@bowt.ie>
Cc: Andres Freund <andres@anarazel.de>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: new heapcheck contrib module
Date: Sat, 29 Aug 2020 10:48:44 -0700
Message-ID: <38AF687F-8F6B-48B4-AB9E-A60CFD6CC261@enterprisedb.com> (raw )
In-Reply-To: <1C4CD961-84A7-4DD7-9F98-4E02B8E98ACF@yandex-team.ru >
References: <12ED3DA8-25F0-4B68-937D-D907CFBF08E7@enterprisedb.com >
<20200420194210.sevnpi4lyghoanxi@alap3.anarazel.de >
<23E60C3C-C20A-4508-84AC-E0CEA59E784A@enterprisedb.com >
<CA+TgmobAm8vB1MNQ1taO1et+HxTN8ULnQvEG+grNDqNQTBjbtA@mail.gmail.com >
<F845AC50-A479-43F3-ADFB-861A4D2AD287@enterprisedb.com >
<5F743835-3399-419C-8324-2D424237E999@enterprisedb.com >
<CAFiTN-syyHc3jZoou51v0SR8z0POoNfktqEO6MaGig4YS8mosA@mail.gmail.com >
<E6F4258A-B661-4F2C-8A35-278C9436930A@enterprisedb.com >
<CAFiTN-s-VX3DLfqv7OOo0zGCTOZ3xVU9UZ6GgzBZdJTPNnYFNg@mail.gmail.com >
<885D2EC9-7D06-4903-A17D-72C184A1E1C5@enterprisedb.com >
<CAFiTN-uYg_PiZ854=5R-D9skratapAHKhaDAG7fhJv-Vn5UhaQ@mail.gmail.com >
<5CC85F28-F210-47AC-8DA8-EC4E82AF3FD5@enterprisedb.com >
<CAFiTN-sP5iFJGERTq4a4OLTCzjZKhLC5P6_k2bxQKXnuvdmzGA@mail.gmail.com >
<CAFiTN-tTg0R5AHvXC1ro9XphroX1wUtc5t91hU_+7DeGTpE5VQ@mail.gmail.com >
<0C0349EB-DD85-4DA8-82C7-DF70AB714592@enterprisedb.com >
<CAFiTN-uD3WSeggU20yU6voGou6BDb-cQcfkE8B1rAzWKZVK-Bw@mail.gmail.com >
<BA6E870F-241C-4DDB-B92D-DB39E679C7D4@enterprisedb.com >
<CA+TgmobQUdfM2Q7WAR606RR2YehCZ5GHRKBc-VWfRj-1ZUKqDA@mail.gmail.com >
<C4A3E5AE-CBB2-4E3F-8BAE-29EE216762C2@enterprisedb.com >
<CAAJ_b96_Xr2a+D0j8xKbOnCzURcVQm0V2DS+T_pfc+wt3BpL8g@mail.gmail.com >
<9DF93B83-FE46-4EC4-A99D-77A60729FD2D@enterprisedb.com >
<CA+TgmoZT3kCOaKq6=m5wHGKh_9XQDbCC1LZXL6w_+NMB-n1+mw@mail.gmail.com >
<CAAJ_b94hsrF4x7UKQj03416fAEXJorUpZ9WhsZZjqgv-4+Wg9A@mail.gmail.com >
<50F6468F-5A5E-412A-86EC-DF24B549D733@enterprisedb.com >
<CAAJ_b94+A1Gk5r5sBYbYM2K5hH_gyp_o=+gwJuJhn5ZEm5_rqw@mail.gmail.com >
<CAAJ_b94Wh_7kM9JuJk7wnNsYSNPdSzz35cOEaE_ZRnPcqvupCg@mail.gmail.com >
<7D603B53-8078-4AF0-851B-803FF4416501@enterprisedb.com >
<47D0EC13-0B9A-4D97-9435-0846B0DA2916@yandex-team.ru >
<CA+TgmobgkU0U-XDcpRZvEWb-btOsiZXU3_Uf-15rge+JvqcJsg@mail.gmail.com >
<773B38BD-E27F-4C49-8678-B316261C9189@yandex-team.ru >
<CA+TgmoZo6BHLgZLW5n6ua72GtuMqDieYDhQRpUcFYm9rp-veDQ@mail.gmail.com >
<1C4CD961-84A7-4DD7-9F98-4E02B8E98ACF@yandex-team.ru >
> On Aug 29, 2020, at 3:27 AM, Andrey M. Borodin <x4mmm@yandex-team.ru> wrote:
>
>
>
>> 29 авг. 2020 г., в 00:56, Robert Haas <robertmhaas@gmail.com> написал(а):
>>
>> On Fri, Aug 28, 2020 at 2:10 PM Andrey M. Borodin <x4mmm@yandex-team.ru> wrote:
>>> I don't think so. ISTM It's the same problem of xmax<relfrozenxid actually, just hidden behind detoasing.
>>> Our regular heap_check was checking xmin\xmax invariants for tables, but failed to recognise the problem in toast (while toast was accessible until CLOG truncation).
>>
>> The code can (and should, and I think does) refrain from looking up
>> XIDs that are out of the range thought to be valid -- but how do you
>> propose that it avoid looking up XIDs that ought to have clog data
>> associated with them despite being >= relfrozenxid and < nextxid?
>> TransactionIdDidCommit() does not have a suppress-errors flag, adding
>> one would be quite invasive, yet we cannot safely perform a
>> significant number of checks without knowing whether the inserting
>> transaction committed.
>
> What you write seems completely correct to me. I agree that CLOG thresholds lookup seems unnecessary.
>
> But I have a real corruption at hand (on testing site). If I have proposed here heapcheck. And I have pg_surgery from the thread nearby. Yet I cannot fix the problem, because cannot list affected tuples. These tools do not solve the problem neglected for long enough. It would be supercool if they could.
>
> This corruption like a caries had 3 stages:
> 1. incorrect VM flag that page do not need vacuum
> 2. xmin and xmax < relfrozenxid
> 3. CLOG truncated
>
> Stage 2 is curable with proposed toolset, stage 3 is not. But they are not that different.
I had an earlier version of the verify_heapam patch that included a non-throwing interface to clog. Ultimately, I ripped that out. My reasoning was that a simpler patch submission was more likely to be acceptable to the community.
If you want to submit a separate patch that creates a non-throwing version of the clog interface, and get the community to accept and commit it, I would seriously consider using that from verify_heapam. If it gets committed in time, I might even do so for this release cycle. But I don't want to make this patch dependent on that hypothetical patch getting written and accepted.
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
view thread (218+ messages) latest in thread
Message-ID: <38AF687F-8F6B-48B4-AB9E-A60CFD6CC261@enterprisedb.com>
Permalink: ../38AF687F-8F6B-48B4-AB9E-A60CFD6CC261@enterprisedb.com/
Also on: postgresql.org/message-id/38AF687F-8F6B-48B4-AB9E-A60CFD6CC261@enterprisedb.com
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: mark.dilger@enterprisedb.com, x4mmm@yandex-team.ru, robertmhaas@gmail.com, sulamul@gmail.com, dilipbalaut@gmail.com, pg@bowt.ie, andres@anarazel.de
Subject: Re: new heapcheck contrib module
In-Reply-To: <38AF687F-8F6B-48B4-AB9E-A60CFD6CC261@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox