Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lTChg-00077Y-5j for pgsql-hackers@arkaria.postgresql.org; Mon, 05 Apr 2021 00:02:52 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lTChd-0008SJ-Lf for pgsql-hackers@arkaria.postgresql.org; Mon, 05 Apr 2021 00:02:49 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lTChd-0008SA-58 for pgsql-hackers@lists.postgresql.org; Mon, 05 Apr 2021 00:02:49 +0000 Received: from mail-pg1-x52c.google.com ([2607:f8b0:4864:20::52c]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1lTChZ-0004dq-DT for pgsql-hackers@postgresql.org; Mon, 05 Apr 2021 00:02:48 +0000 Received: by mail-pg1-x52c.google.com with SMTP id b17so3439586pgh.7 for ; Sun, 04 Apr 2021 17:02:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=soqV0Dv/n5hFcdayTMbpLHUcIfPozWkAexTa2FY5d18=; b=GnOSfXb4+XzN4vgk/8lz1lhcuYH92aK9oNcVgrbwSuvUXm8ojhALHQaF4Y7G7/FsX2 cjq7CuWVrD9LmV+7m55w1j+1HzU/k57Lz0QMLqZeLn9sbgPcOnJt1/Dcn4pwN93Z5sVd Pw5/yRsBC3r5XbVNm+u04n55W1uMYtHvVO0JTB0YJbYBnzMrHUo70CiCRUJeijtDjKK0 L7sIWyXWJoHeVou/2yPNOElK9re017ORMmei1hqCC1/G97E4uUH7WDGJcdqoUbFpA+xG 2Fix/YMb33gJMSTNG0LAo0TA9aSYYZyTbhatlwDFcl+BPf7kSjszWYCKeHqaCNHWoUCQ JTRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=soqV0Dv/n5hFcdayTMbpLHUcIfPozWkAexTa2FY5d18=; b=F248QDB3zSWgvxC6lcxt0bpFqvXLrpsamgXZsvY/z534D8bKHkBN6MuFyaSiZxJHw6 AgsLTxkVZgAZqxI6htwN8i7o0xkzdDTMwG5fV6C0Uj3qBDtgjsf7/Toc5QhP4FeEnnsR 3M5I2mONGvqq5AqRwEiGdZ+OEJrW9dS4ECt30ErChyuz47bj0ppV3loX+wdpiGdFzJbO y+Y7o3PUa52jTiqk6e91wMsFsV8b6jef1I4egJuuMFX402qTt+4fBHl6TUwKibcQ5QVe GbUD+BCn31S16yc2dHQ1yCu0mA+1SMbL90dlXpF/MdffhBizsrVl6qDVN/B/u0tKDdD2 nz6w== X-Gm-Message-State: AOAM532A115WF+bjrT23tpQlkDWNRu6CoLoZCsJ68h6MvwIiJFIxgDC6 FFBfUsXltjssikXJTgt5DwTlsgECWv1W1t5deJGgyXeb6Y+2d+KZWBfe7yKAXs2/uDIZVk2mONq DU1Pjg35rkrKEWqDMLRaEd4wkS9lk94A8zGdfkbXSnRNunl3L8hGfqmrUpYjiGF3Hn/fG1PXnNy GOtg/J+NV0UNR7FQDNs0psAOeY3bx+KAOB84h1 X-Google-Smtp-Source: ABdhPJxGWcv0Lhq5NIcxWWr7GBaMTHRyxyJaZA7Tpc8mrxNnT5W6735XkK+NCV+tIbX9+lr/LBXJEw== X-Received: by 2002:a63:dc49:: with SMTP id f9mr20602345pgj.361.1617580962770; Sun, 04 Apr 2021 17:02:42 -0700 (PDT) Received: from laptop280-ma-us.hitronhub.home (24-113-193-150.wavecable.com. [24.113.193.150]) by smtp.gmail.com with ESMTPSA id i7sm13272540pgq.16.2021.04.04.17.02.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 04 Apr 2021 17:02:41 -0700 (PDT) From: Mark Dilger Message-Id: Content-Type: multipart/mixed; boundary="Apple-Mail=_762805D9-89AF-4F33-9298-4A3BFD0EBE1E" Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: pg_amcheck contrib application Date: Sun, 4 Apr 2021 17:02:40 -0700 In-Reply-To: Cc: Tom Lane , Noah Misch , Peter Geoghegan , Thomas Munro , "Andrey M. Borodin" , Stephen Frost , Michael Paquier , Amul Sul , Dilip Kumar , Andres Freund , PostgreSQL-development To: Robert Haas References: <9E76E46A-48B2-4869-BD0C-422204C1F767@enterprisedb.com> <20210313184612.GA3195145@rfd.leadboat.com> <930621E5-5D10-456D-8CE4-CF629A6C4669@enterprisedb.com> <20210313191841.GB3195145@rfd.leadboat.com> <3012106.1615827854@sss.pgh.pa.us> <3053552.1615834640@sss.pgh.pa.us> <4F10540A-9022-49EE-A6FC-7FE90797793E@enterprisedb.com> <0D48534A-F511-405D-8645-84B0A1DA1E11@enterprisedb.com> <6915B9B2-B551-4DBC-89DB-C12D9AF985B8@enterprisedb.com> <23192440-3B8A-4D82-8C24-9D8161BAAB01@enterprisedb.com> <1F1A6246-854F-47F0-A179-A3CD9F05356B@enterprisedb.com> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-CLOUD-SEC-AV-Info: enterprisedb,google_mail,monitor X-CLOUD-SEC-AV-Sent: true X-Gm-Spam: 0 X-Gm-Phishy: 0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --Apple-Mail=_762805D9-89AF-4F33-9298-4A3BFD0EBE1E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Apr 1, 2021, at 1:08 PM, Robert Haas wrote: >=20 >=20 >=20 > - There are a total of two (2) calls in the current source code to > palloc0fast, and hundreds of calls to palloc0. So I think you should > forget about using the fast variant and just do what almost every > other caller does. Done. > - If you want to make this code faster, a better idea would be to > avoid doing all of this allocate and free work and just allocate an > array that's guaranteed to be big enough, and then keep track of how > many elements of that array are actually in use. Sounds like premature optimization to me. I only used palloc0fast = because the argument is compile-time known. I wasn't specifically = attempting to speed anything up. > - #ifdef DECOMPRESSION_CORRUPTION_CHECKING is not a useful way of > introducing such a feature. Either we do it for real and expose it via > SQL and pg_amcheck as an optional behavior, or we rip it out and > revisit it later. Given the nearness of feature freeze, my vote is for > the latter. >=20 > - I'd remove the USE_LZ4 bit, too. Let's not define the presence of > LZ4 data in a non-LZ4-enabled cluster as corruption. If we do, then > people will expect to be able to use this to find places where they > are dependent on LZ4 if they want to move away from it -- and if we > don't recurse into composite datums, that will not actually work. Ok, I have removed this bit. I also removed the part of the patch that = introduced a new corruption check, decompressing the data to see if it = decompresses without error. > - check_toast_tuple() has an odd and slightly unclear calling > convention for which there are no comments. I wonder if it would be > better to reverse things and make bool *error the return value and > what is now the return value into a pointer argument, but whatever we > do I think it needs a few words in the comments. We don't need to > slavishly explain every argument -- I think toasttup and ctx and tctx > are reasonably clear -- but this is not. ... > - Is there a reason we need a cross-check on both the number of chunks > and on the total size? It seems to me that we should check that each > individual chunk has the size we expect, and that the total number of > chunks is what we expect. The overall size is then necessarily > correct. Good point. I've removed the extra check on the total size, since it = cannot be wrong if the checks on the individual chunk sizes were all = correct. This eliminates the need for the odd calling convention for = check_toast_tuple(), so I've changed that to return void and not take = any return-by-reference arguments. > - To me it would be more logical to reverse the order of the > toast_pointer.va_toastrelid !=3D ctx->rel->rd_rel->reltoastrelid and > VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer) > toast_pointer.va_rawsize > - VARHDRSZ checks. Whether we're pointing at the correct relation > feels more fundamental. Done. > - If we moved the toplevel foreach loop in check_toasted_attributes() > out to the caller, say renaming the function to just > check_toasted_attribute(), we'd save a level of indentation in that > whole function and probably add a tad of clarity, too. You wouldn't > feel the need to Assert(ctx.toasted_attributes =3D=3D NIL) in the = caller > if the caller had just done list_free(ctx->toasted_attributes); > ctx->toasted_attributes =3D NIL. You're right. It looks nicer that way. Changed. > - Why are all the changes to the tests in this patch? What do they > have to do with getting the TOAST checks out from under the buffer > lock? I really need you to structure the patch series so that each > patch is about one topic and, equally, so that each topic is only > covered by one patch. Otherwise it's just way too confusing. v18-0001 - Finishes work started in commit 3b6c1259f9 that was = overlooked owing to how I had separated the changes in v17-0002 vs. = v17-0003 v18-0002 - Postpones the toast checks for a page until after the main = table page lock is released v18-0003 - Improves the corruption messages in ways already discussed = earlier in this thread. Changes the tests to expect the new messages, = but adds no new checks v18-0004 - Adding corruption checks of toast pointers. Extends the = regression tests to cover the new checks. >=20 > - I think some of these messages need a bit of word-smithing, too, but > we can leave that for when we're closer to being done with this. Ok. --Apple-Mail=_762805D9-89AF-4F33-9298-4A3BFD0EBE1E Content-Disposition: attachment; filename=v18-0001-amcheck-remove-duplicate-xid-bounds-checks.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="v18-0001-amcheck-remove-duplicate-xid-bounds-checks.patch" Content-Transfer-Encoding: quoted-printable =46rom=205ff174db4de8c35f73c3bb9ffc51d854686cb1c5=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20Mark=20Dilger=20=0A= Date:=20Fri,=202=20Apr=202021=2015:03:12=20-0700=0ASubject:=20[PATCH=20= v18=201/4]=20amcheck:=20remove=20duplicate=20xid=20bounds=20checks=0A=0A= Commit=203b6c1259f9=20resulted=20in=20the=20same=20xmin=20and=20xmax=20= bounds=20checking=0Abeing=20performed=20in=20both=20check_tuple()=20and=20= check_tuple_visibility().=0ALeaving=20the=20ones=20in=20= check_tuple_visibility()=20and=20removing=20the=0Aothers.=0A=0AWhile=20= at=20it,=20adjusting=20some=20code=20comments=20that=20should=20have=20= been=0Achanged=20in=203b6c1259f9=20but=20were=20not.=0A---=0A=20= contrib/amcheck/verify_heapam.c=20|=20130=20= ++------------------------------=0A=201=20file=20changed,=206=20= insertions(+),=20124=20deletions(-)=0A=0Adiff=20--git=20= a/contrib/amcheck/verify_heapam.c=20b/contrib/amcheck/verify_heapam.c=0A= index=2011ace483d0..1d769035f1=20100644=0A---=20= a/contrib/amcheck/verify_heapam.c=0A+++=20= b/contrib/amcheck/verify_heapam.c=0A@@=20-1390,136=20+1390,18=20@@=20= check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20static=20void=0A=20= check_tuple(HeapCheckContext=20*ctx)=0A=20{=0A-=09TransactionId=20xmin;=0A= -=09TransactionId=20xmax;=0A-=09bool=09=09fatal=20=3D=20false;=0A-=09= uint16=09=09infomask=20=3D=20ctx->tuphdr->t_infomask;=0A-=0A-=09/*=20If=20= xmin=20is=20normal,=20it=20should=20be=20within=20valid=20range=20*/=0A-=09= xmin=20=3D=20HeapTupleHeaderGetXmin(ctx->tuphdr);=0A-=09switch=20= (get_xid_status(xmin,=20ctx,=20NULL))=0A-=09{=0A-=09=09case=20= XID_INVALID:=0A-=09=09case=20XID_BOUNDS_OK:=0A-=09=09=09break;=0A-=09=09= case=20XID_IN_FUTURE:=0A-=09=09=09report_corruption(ctx,=0A-=09=09=09=09=09= =09=09=20=20psprintf("xmin=20%u=20equals=20or=20exceeds=20next=20valid=20= transaction=20ID=20%u:%u",=0A-=09=09=09=09=09=09=09=09=09=20=20=20xmin,=0A= -=09=09=09=09=09=09=09=09=09=20=20=20= EpochFromFullTransactionId(ctx->next_fxid),=0A-=09=09=09=09=09=09=09=09=09= =20=20=20XidFromFullTransactionId(ctx->next_fxid)));=0A-=09=09=09fatal=20= =3D=20true;=0A-=09=09=09break;=0A-=09=09case=20XID_PRECEDES_CLUSTERMIN:=0A= -=09=09=09report_corruption(ctx,=0A-=09=09=09=09=09=09=09=20=20= psprintf("xmin=20%u=20precedes=20oldest=20valid=20transaction=20ID=20= %u:%u",=0A-=09=09=09=09=09=09=09=09=09=20=20=20xmin,=0A-=09=09=09=09=09=09= =09=09=09=20=20=20EpochFromFullTransactionId(ctx->oldest_fxid),=0A-=09=09= =09=09=09=09=09=09=09=20=20=20= XidFromFullTransactionId(ctx->oldest_fxid)));=0A-=09=09=09fatal=20=3D=20= true;=0A-=09=09=09break;=0A-=09=09case=20XID_PRECEDES_RELMIN:=0A-=09=09=09= report_corruption(ctx,=0A-=09=09=09=09=09=09=09=20=20psprintf("xmin=20%u=20= precedes=20relation=20freeze=20threshold=20%u:%u",=0A-=09=09=09=09=09=09=09= =09=09=20=20=20xmin,=0A-=09=09=09=09=09=09=09=09=09=20=20=20= EpochFromFullTransactionId(ctx->relfrozenfxid),=0A-=09=09=09=09=09=09=09=09= =09=20=20=20XidFromFullTransactionId(ctx->relfrozenfxid)));=0A-=09=09=09= fatal=20=3D=20true;=0A-=09=09=09break;=0A-=09}=0A-=0A-=09xmax=20=3D=20= HeapTupleHeaderGetRawXmax(ctx->tuphdr);=0A-=0A-=09if=20(infomask=20&=20= HEAP_XMAX_IS_MULTI)=0A-=09{=0A-=09=09/*=20xmax=20is=20a=20multixact,=20= so=20it=20should=20be=20within=20valid=20MXID=20range=20*/=0A-=09=09= switch=20(check_mxid_valid_in_rel(xmax,=20ctx))=0A-=09=09{=0A-=09=09=09= case=20XID_INVALID:=0A-=09=09=09=09report_corruption(ctx,=0A-=09=09=09=09= =09=09=09=09=20=20pstrdup("multitransaction=20ID=20is=20invalid"));=0A-=09= =09=09=09fatal=20=3D=20true;=0A-=09=09=09=09break;=0A-=09=09=09case=20= XID_PRECEDES_RELMIN:=0A-=09=09=09=09report_corruption(ctx,=0A-=09=09=09=09= =09=09=09=09=20=20psprintf("multitransaction=20ID=20%u=20precedes=20= relation=20minimum=20multitransaction=20ID=20threshold=20%u",=0A-=09=09=09= =09=09=09=09=09=09=09=20=20=20xmax,=20ctx->relminmxid));=0A-=09=09=09=09= fatal=20=3D=20true;=0A-=09=09=09=09break;=0A-=09=09=09case=20= XID_PRECEDES_CLUSTERMIN:=0A-=09=09=09=09report_corruption(ctx,=0A-=09=09=09= =09=09=09=09=09=20=20psprintf("multitransaction=20ID=20%u=20precedes=20= oldest=20valid=20multitransaction=20ID=20threshold=20%u",=0A-=09=09=09=09= =09=09=09=09=09=09=20=20=20xmax,=20ctx->oldest_mxact));=0A-=09=09=09=09= fatal=20=3D=20true;=0A-=09=09=09=09break;=0A-=09=09=09case=20= XID_IN_FUTURE:=0A-=09=09=09=09report_corruption(ctx,=0A-=09=09=09=09=09=09= =09=09=20=20psprintf("multitransaction=20ID=20%u=20equals=20or=20exceeds=20= next=20valid=20multitransaction=20ID=20%u",=0A-=09=09=09=09=09=09=09=09=09= =09=20=20=20xmax,=0A-=09=09=09=09=09=09=09=09=09=09=20=20=20= ctx->next_mxact));=0A-=09=09=09=09fatal=20=3D=20true;=0A-=09=09=09=09= break;=0A-=09=09=09case=20XID_BOUNDS_OK:=0A-=09=09=09=09break;=0A-=09=09= }=0A-=09}=0A-=09else=0A-=09{=0A-=09=09/*=0A-=09=09=20*=20xmax=20is=20not=20= a=20multixact=20and=20is=20normal,=20so=20it=20should=20be=20within=20= the=0A-=09=09=20*=20valid=20XID=20range.=0A-=09=09=20*/=0A-=09=09switch=20= (get_xid_status(xmax,=20ctx,=20NULL))=0A-=09=09{=0A-=09=09=09case=20= XID_INVALID:=0A-=09=09=09case=20XID_BOUNDS_OK:=0A-=09=09=09=09break;=0A-=09= =09=09case=20XID_IN_FUTURE:=0A-=09=09=09=09report_corruption(ctx,=0A-=09=09= =09=09=09=09=09=09=20=20psprintf("xmax=20%u=20equals=20or=20exceeds=20= next=20valid=20transaction=20ID=20%u:%u",=0A-=09=09=09=09=09=09=09=09=09=09= =20=20=20xmax,=0A-=09=09=09=09=09=09=09=09=09=09=20=20=20= EpochFromFullTransactionId(ctx->next_fxid),=0A-=09=09=09=09=09=09=09=09=09= =09=20=20=20XidFromFullTransactionId(ctx->next_fxid)));=0A-=09=09=09=09= fatal=20=3D=20true;=0A-=09=09=09=09break;=0A-=09=09=09case=20= XID_PRECEDES_CLUSTERMIN:=0A-=09=09=09=09report_corruption(ctx,=0A-=09=09=09= =09=09=09=09=09=20=20psprintf("xmax=20%u=20precedes=20oldest=20valid=20= transaction=20ID=20%u:%u",=0A-=09=09=09=09=09=09=09=09=09=09=20=20=20= xmax,=0A-=09=09=09=09=09=09=09=09=09=09=20=20=20= EpochFromFullTransactionId(ctx->oldest_fxid),=0A-=09=09=09=09=09=09=09=09= =09=09=20=20=20XidFromFullTransactionId(ctx->oldest_fxid)));=0A-=09=09=09= =09fatal=20=3D=20true;=0A-=09=09=09=09break;=0A-=09=09=09case=20= XID_PRECEDES_RELMIN:=0A-=09=09=09=09report_corruption(ctx,=0A-=09=09=09=09= =09=09=09=09=20=20psprintf("xmax=20%u=20precedes=20relation=20freeze=20= threshold=20%u:%u",=0A-=09=09=09=09=09=09=09=09=09=09=20=20=20xmax,=0A-=09= =09=09=09=09=09=09=09=09=09=20=20=20= EpochFromFullTransactionId(ctx->relfrozenfxid),=0A-=09=09=09=09=09=09=09=09= =09=09=20=20=20XidFromFullTransactionId(ctx->relfrozenfxid)));=0A-=09=09=09= =09fatal=20=3D=20true;=0A-=09=09}=0A-=09}=0A-=0A=20=09/*=0A-=09=20*=20= Cannot=20process=20tuple=20data=20if=20tuple=20header=20was=20corrupt,=20= as=20the=20offsets=0A-=09=20*=20within=20the=20page=20cannot=20be=20= trusted,=20leaving=20too=20much=20risk=20of=20reading=0A-=09=20*=20= garbage=20if=20we=20continue.=0A-=09=20*=0A-=09=20*=20We=20also=20cannot=20= process=20the=20tuple=20if=20the=20xmin=20or=20xmax=20were=20invalid=0A-=09= =20*=20relative=20to=20relfrozenxid=20or=20relminmxid,=20as=20clog=20= entries=20for=20the=20xids=0A-=09=20*=20may=20already=20be=20gone.=0A+=09= =20*=20Check=20various=20forms=20of=20tuple=20header=20corruption,=20and=20= if=20the=20header=20is=20too=0A+=09=20*=20corrupt,=20do=20not=20continue=20= with=20other=20checks.=0A=20=09=20*/=0A-=09if=20(fatal)=0A+=09if=20= (!check_tuple_header(ctx))=0A=20=09=09return;=0A=20=0A=20=09/*=0A-=09=20= *=20Check=20various=20forms=20of=20tuple=20header=20corruption.=20=20If=20= the=20header=20is=20too=0A-=09=20*=20corrupt=20to=20continue=20checking,=20= or=20if=20the=20tuple=20is=20not=20visible=20to=20anyone,=0A-=09=20*=20= we=20cannot=20continue=20with=20other=20checks.=0A+=09=20*=20Check=20= tuple=20visibility.=20=20If=20the=20inserting=20transaction=20aborted,=20= we=0A+=09=20*=20cannot=20assume=20our=20relation=20description=20matches=20= the=20tuple=20structure,=20and=0A+=09=20*=20therefore=20cannot=20check=20= it.=0A=20=09=20*/=0A-=09if=20(!check_tuple_header(ctx))=0A-=09=09return;=0A= -=0A=20=09if=20(!check_tuple_visibility(ctx))=0A=20=09=09return;=0A=20=0A= --=20=0A2.21.1=20(Apple=20Git-122.3)=0A=0A= --Apple-Mail=_762805D9-89AF-4F33-9298-4A3BFD0EBE1E Content-Disposition: attachment; filename=v18-0002-amcheck-avoid-extra-work-while-holding-buffer-lo.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="v18-0002-amcheck-avoid-extra-work-while-holding-buffer-lo.patch" Content-Transfer-Encoding: quoted-printable =46rom=20e33e49f1e1325a4439991550789c47fdf54b7220=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20Mark=20Dilger=20=0A= Date:=20Fri,=202=20Apr=202021=2015:05:07=20-0700=0ASubject:=20[PATCH=20= v18=202/4]=20amcheck:=20avoid=20extra=20work=20while=20holding=20buffer=20= locks=0A=0ARefactoring=20verify_heapam=20so=20it=20does=20not=20look=20= up=20attributes=20in=20the=0Atoast=20table=20while=20holding=20a=20= buffer=20lock=20on=20the=20main=20table=20page.=0AInstead,=20check=20all=20= toasted=20attributes=20found=20in=20tuples=20on=20a=20main=0Atable=20= page=20after=20the=20lock=20has=20been=20released.=0A=0AThis=20changes=20= the=20user=20visible=20behavior=20slightly,=20in=20that=20when=20both=0A= toasted=20attributes=20and=20non-toasted=20attributes=20from=20the=20= same=20page=20are=0Acorrupt,=20the=20corruption=20reports=20about=20= non-toasted=20attributes=20will=0Aall=20be=20emitted=20prior=20to=20any=20= corruption=20reports=20about=20the=20toasted=0Aattributes.=20=20This=20= shouldn't=20matter,=20though,=20as=20this=20function=20was=0Afirst=20= committed=20in=20this=20development=20cycle,=20and=20no=20users=20should=20= be=0Afamiliar=20with=20the=20old=20behavior.=0A---=0A=20= contrib/amcheck/verify_heapam.c=20=20|=20236=20= +++++++++++++++++++++----------=0A=20src/tools/pgindent/typedefs.list=20= |=20=20=201=20+=0A=202=20files=20changed,=20165=20insertions(+),=2072=20= deletions(-)=0A=0Adiff=20--git=20a/contrib/amcheck/verify_heapam.c=20= b/contrib/amcheck/verify_heapam.c=0Aindex=201d769035f1..d8a3e966c4=20= 100644=0A---=20a/contrib/amcheck/verify_heapam.c=0A+++=20= b/contrib/amcheck/verify_heapam.c=0A@@=20-58,6=20+58,19=20@@=20typedef=20= enum=20SkipPages=0A=20=09SKIP_PAGES_NONE=0A=20}=20SkipPages;=0A=20=0A+/*=0A= +=20*=20Struct=20holding=20information=20about=20a=20toasted=20attribute=20= sufficient=20to=20both=0A+=20*=20check=20the=20toasted=20attribute=20= and,=20if=20found=20to=20be=20corrupt,=20to=20report=20where=20it=0A+=20= *=20was=20encountered=20in=20the=20main=20table.=0A+=20*/=0A+typedef=20= struct=20ToastedAttribute=0A+{=0A+=09struct=20varatt_external=20= toast_pointer;=0A+=09BlockNumber=20blkno;=09=09=09/*=20block=20in=20main=20= table=20*/=0A+=09OffsetNumber=20offnum;=09=09/*=20offset=20in=20main=20= table=20*/=0A+=09AttrNumber=09attnum;=09=09=09/*=20attribute=20in=20main=20= table=20*/=0A+}=20ToastedAttribute;=0A+=0A=20/*=0A=20=20*=20Struct=20= holding=20the=20running=20context=20information=20during=0A=20=20*=20a=20= lifetime=20of=20a=20verify_heapam=20execution.=0A@@=20-119,11=20+132,11=20= @@=20typedef=20struct=20HeapCheckContext=0A=20=09/*=20True=20if=20= tuple's=20xmax=20makes=20it=20eligible=20for=20pruning=20*/=0A=20=09bool=09= =09tuple_could_be_pruned;=0A=20=0A-=09/*=20Values=20for=20iterating=20= over=20toast=20for=20the=20attribute=20*/=0A-=09int32=09=09chunkno;=0A-=09= int32=09=09attrsize;=0A-=09int32=09=09endchunk;=0A-=09int32=09=09= totalchunks;=0A+=09/*=0A+=09=20*=20List=20of=20ToastedAttribute=20= structs=20for=20toasted=20attributes=20which=20are=20not=0A+=09=20*=20= eligible=20for=20pruning=20and=20should=20be=20checked=0A+=09=20*/=0A+=09= List=09=20=20=20*toasted_attributes;=0A=20=0A=20=09/*=20Whether=20= verify_heapam=20has=20yet=20encountered=20any=20corrupt=20tuples=20*/=0A=20= =09bool=09=09is_corrupt;=0A@@=20-136,13=20+149,20=20@@=20typedef=20= struct=20HeapCheckContext=0A=20/*=20Internal=20implementation=20*/=0A=20= static=20void=20sanity_check_relation(Relation=20rel);=0A=20static=20= void=20check_tuple(HeapCheckContext=20*ctx);=0A-static=20void=20= check_toast_tuple(HeapTuple=20toasttup,=20HeapCheckContext=20*ctx);=0A= +static=20void=20check_toast_tuple(HeapTuple=20toasttup,=20= HeapCheckContext=20*ctx,=0A+=09=09=09=09=09=09=09=20=20ToastedAttribute=20= *ta,=20int32=20chunkno,=0A+=09=09=09=09=09=09=09=20=20int32=20endchunk);=0A= =20=0A=20static=20bool=20check_tuple_attribute(HeapCheckContext=20*ctx);=0A= +static=20void=20check_toasted_attribute(HeapCheckContext=20*ctx,=0A+=09=09= =09=09=09=09=09=09=09ToastedAttribute=20*ta);=0A+=0A=20static=20bool=20= check_tuple_header(HeapCheckContext=20*ctx);=0A=20static=20bool=20= check_tuple_visibility(HeapCheckContext=20*ctx);=0A=20=0A=20static=20= void=20report_corruption(HeapCheckContext=20*ctx,=20char=20*msg);=0A= +static=20void=20report_toast_corruption(HeapCheckContext=20*ctx,=0A+=09=09= =09=09=09=09=09=09=09ToastedAttribute=20*ta,=20char=20*msg);=0A=20static=20= TupleDesc=20verify_heapam_tupdesc(void);=0A=20static=20FullTransactionId=20= FullTransactionIdFromXidAndCtx(TransactionId=20xid,=0A=20=09=09=09=09=09=09= =09=09=09=09=09=09=09=09const=20HeapCheckContext=20*ctx);=0A@@=20-253,6=20= +273,7=20@@=20verify_heapam(PG_FUNCTION_ARGS)=0A=20=0A=20=09memset(&ctx,=20= 0,=20sizeof(HeapCheckContext));=0A=20=09ctx.cached_xid=20=3D=20= InvalidTransactionId;=0A+=09ctx.toasted_attributes=20=3D=20NIL;=0A=20=0A=20= =09/*=0A=20=09=20*=20Any=20xmin=20newer=20than=20the=20xmin=20of=20our=20= snapshot=20can't=20become=20all-visible=0A@@=20-469,6=20+490,19=20@@=20= verify_heapam(PG_FUNCTION_ARGS)=0A=20=09=09/*=20clean=20up=20*/=0A=20=09=09= UnlockReleaseBuffer(ctx.buffer);=0A=20=0A+=09=09/*=0A+=09=09=20*=20Check=20= any=20toast=20pointers=20from=20the=20page=20whose=20lock=20we=20just=20= released=0A+=09=09=20*/=0A+=09=09if=20(ctx.toasted_attributes=20!=3D=20= NIL)=0A+=09=09{=0A+=09=09=09ListCell=20=20=20*cell;=0A+=0A+=09=09=09= foreach(cell,=20ctx.toasted_attributes)=0A+=09=09=09=09= check_toasted_attribute(&ctx,=20lfirst(cell));=0A+=09=09=09= list_free(ctx.toasted_attributes);=0A+=09=09=09ctx.toasted_attributes=20= =3D=20NIL;=0A+=09=09}=0A+=0A=20=09=09if=20(on_error_stop=20&&=20= ctx.is_corrupt)=0A=20=09=09=09break;=0A=20=09}=0A@@=20-510,14=20+544,13=20= @@=20sanity_check_relation(Relation=20rel)=0A=20}=0A=20=0A=20/*=0A-=20*=20= Record=20a=20single=20corruption=20found=20in=20the=20table.=20=20The=20= values=20in=20ctx=20should=0A-=20*=20reflect=20the=20location=20of=20the=20= corruption,=20and=20the=20msg=20argument=20should=20contain=0A-=20*=20a=20= human-readable=20description=20of=20the=20corruption.=0A-=20*=0A-=20*=20= The=20msg=20argument=20is=20pfree'd=20by=20this=20function.=0A+=20*=20= Shared=20internal=20implementation=20for=20report_corruption=20and=0A+=20= *=20report_toast_corruption.=0A=20=20*/=0A=20static=20void=0A= -report_corruption(HeapCheckContext=20*ctx,=20char=20*msg)=0A= +report_corruption_internal(Tuplestorestate=20*tupstore,=20TupleDesc=20= tupdesc,=0A+=09=09=09=09=09=09=20=20=20BlockNumber=20blkno,=20= OffsetNumber=20offnum,=0A+=09=09=09=09=09=09=20=20=20AttrNumber=20= attnum,=20char=20*msg)=0A=20{=0A=20=09Datum=09=09= values[HEAPCHECK_RELATION_COLS];=0A=20=09bool=09=09= nulls[HEAPCHECK_RELATION_COLS];=0A@@=20-525,10=20+558,10=20@@=20= report_corruption(HeapCheckContext=20*ctx,=20char=20*msg)=0A=20=0A=20=09= MemSet(values,=200,=20sizeof(values));=0A=20=09MemSet(nulls,=200,=20= sizeof(nulls));=0A-=09values[0]=20=3D=20Int64GetDatum(ctx->blkno);=0A-=09= values[1]=20=3D=20Int32GetDatum(ctx->offnum);=0A-=09values[2]=20=3D=20= Int32GetDatum(ctx->attnum);=0A-=09nulls[2]=20=3D=20(ctx->attnum=20<=20= 0);=0A+=09values[0]=20=3D=20Int64GetDatum(blkno);=0A+=09values[1]=20=3D=20= Int32GetDatum(offnum);=0A+=09values[2]=20=3D=20Int32GetDatum(attnum);=0A= +=09nulls[2]=20=3D=20(attnum=20<=200);=0A=20=09values[3]=20=3D=20= CStringGetTextDatum(msg);=0A=20=0A=20=09/*=0A@@=20-541,8=20+574,39=20@@=20= report_corruption(HeapCheckContext=20*ctx,=20char=20*msg)=0A=20=09=20*/=0A= =20=09pfree(msg);=0A=20=0A-=09tuple=20=3D=20= heap_form_tuple(ctx->tupdesc,=20values,=20nulls);=0A-=09= tuplestore_puttuple(ctx->tupstore,=20tuple);=0A+=09tuple=20=3D=20= heap_form_tuple(tupdesc,=20values,=20nulls);=0A+=09= tuplestore_puttuple(tupstore,=20tuple);=0A+}=0A+=0A+/*=0A+=20*=20Record=20= a=20single=20corruption=20found=20in=20the=20main=20table.=20=20The=20= values=20in=20ctx=20should=0A+=20*=20indicate=20the=20location=20of=20= the=20corruption,=20and=20the=20msg=20argument=20should=20contain=0A+=20= *=20a=20human-readable=20description=20of=20the=20corruption.=0A+=20*=0A= +=20*=20The=20msg=20argument=20is=20pfree'd=20by=20this=20function.=0A+=20= */=0A+static=20void=0A+report_corruption(HeapCheckContext=20*ctx,=20char=20= *msg)=0A+{=0A+=09report_corruption_internal(ctx->tupstore,=20= ctx->tupdesc,=20ctx->blkno,=0A+=09=09=09=09=09=09=09=20=20=20= ctx->offnum,=20ctx->attnum,=20msg);=0A+=09ctx->is_corrupt=20=3D=20true;=0A= +}=0A+=0A+/*=0A+=20*=20Record=20corruption=20found=20in=20the=20toast=20= table.=20=20The=20values=20in=20ta=20should=0A+=20*=20indicate=20the=20= location=20in=20the=20main=20table=20where=20the=20toast=20pointer=20was=0A= +=20*=20encountered,=20and=20the=20msg=20argument=20should=20contain=20a=20= human-readable=0A+=20*=20description=20of=20the=20toast=20table=20= corruption.=0A+=20*=0A+=20*=20As=20above,=20the=20msg=20argument=20is=20= pfree'd=20by=20this=20function.=0A+=20*/=0A+static=20void=0A= +report_toast_corruption(HeapCheckContext=20*ctx,=20ToastedAttribute=20= *ta,=0A+=09=09=09=09=09=09char=20*msg)=0A+{=0A+=09= report_corruption_internal(ctx->tupstore,=20ctx->tupdesc,=20ta->blkno,=0A= +=09=09=09=09=09=09=09=20=20=20ta->offnum,=20ta->attnum,=20msg);=0A=20=09= ctx->is_corrupt=20=3D=20true;=0A=20}=0A=20=0A@@=20-1094,9=20+1158,12=20= @@=20check_tuple_visibility(HeapCheckContext=20*ctx)=0A=20=20*=20tuples=20= that=20store=20the=20toasted=20value=20are=20retrieved=20and=20checked=20= in=20order,=20with=0A=20=20*=20each=20toast=20tuple=20being=20checked=20= against=20where=20we=20are=20in=20the=20sequence,=20as=20well=0A=20=20*=20= as=20each=20toast=20tuple=20having=20its=20varlena=20structure=20sanity=20= checked.=0A+=20*=0A+=20*=20Returns=20whether=20the=20toast=20tuple=20= passed=20the=20corruption=20checks.=0A=20=20*/=0A=20static=20void=0A= -check_toast_tuple(HeapTuple=20toasttup,=20HeapCheckContext=20*ctx)=0A= +check_toast_tuple(HeapTuple=20toasttup,=20HeapCheckContext=20*ctx,=0A+=09= =09=09=09=20=20ToastedAttribute=20*ta,=20int32=20chunkno,=20int32=20= endchunk)=0A=20{=0A=20=09int32=09=09curchunk;=0A=20=09Pointer=09=09= chunk;=0A@@=20-1111,7=20+1178,7=20@@=20check_toast_tuple(HeapTuple=20= toasttup,=20HeapCheckContext=20*ctx)=0A=20=09=09=09=09=09=09=09=09=09=09=20= ctx->toast_rel->rd_att,=20&isnull));=0A=20=09if=20(isnull)=0A=20=09{=0A-=09= =09report_corruption(ctx,=0A+=09=09report_toast_corruption(ctx,=20ta,=0A=20= =09=09=09=09=09=09=20=20pstrdup("toast=20chunk=20sequence=20number=20is=20= null"));=0A=20=09=09return;=0A=20=09}=0A@@=20-1119,7=20+1186,7=20@@=20= check_toast_tuple(HeapTuple=20toasttup,=20HeapCheckContext=20*ctx)=0A=20=09= =09=09=09=09=09=09=09=09=09ctx->toast_rel->rd_att,=20&isnull));=0A=20=09= if=20(isnull)=0A=20=09{=0A-=09=09report_corruption(ctx,=0A+=09=09= report_toast_corruption(ctx,=20ta,=0A=20=09=09=09=09=09=09=20=20= pstrdup("toast=20chunk=20data=20is=20null"));=0A=20=09=09return;=0A=20=09= }=0A@@=20-1137,7=20+1204,7=20@@=20check_toast_tuple(HeapTuple=20= toasttup,=20HeapCheckContext=20*ctx)=0A=20=09=09/*=20should=20never=20= happen=20*/=0A=20=09=09uint32=09=09header=20=3D=20((varattrib_4b=20*)=20= chunk)->va_4byte.va_header;=0A=20=0A-=09=09report_corruption(ctx,=0A+=09=09= report_toast_corruption(ctx,=20ta,=0A=20=09=09=09=09=09=09=20=20= psprintf("corrupt=20extended=20toast=20chunk=20has=20invalid=20varlena=20= header:=20%0x=20(sequence=20number=20%d)",=0A=20=09=09=09=09=09=09=09=09=20= =20=20header,=20curchunk));=0A=20=09=09return;=0A@@=20-1146,30=20= +1213,28=20@@=20check_toast_tuple(HeapTuple=20toasttup,=20= HeapCheckContext=20*ctx)=0A=20=09/*=0A=20=09=20*=20Some=20checks=20on=20= the=20data=20we've=20found=0A=20=09=20*/=0A-=09if=20(curchunk=20!=3D=20= ctx->chunkno)=0A+=09if=20(curchunk=20!=3D=20chunkno)=0A=20=09{=0A-=09=09= report_corruption(ctx,=0A+=09=09report_toast_corruption(ctx,=20ta,=0A=20=09= =09=09=09=09=09=20=20psprintf("toast=20chunk=20sequence=20number=20%u=20= does=20not=20match=20the=20expected=20sequence=20number=20%u",=0A-=09=09=09= =09=09=09=09=09=20=20=20curchunk,=20ctx->chunkno));=0A+=09=09=09=09=09=09= =09=09=20=20=20curchunk,=20chunkno));=0A=20=09=09return;=0A=20=09}=0A-=09= if=20(curchunk=20>=20ctx->endchunk)=0A+=09if=20(curchunk=20>=20endchunk)=0A= =20=09{=0A-=09=09report_corruption(ctx,=0A+=09=09= report_toast_corruption(ctx,=20ta,=0A=20=09=09=09=09=09=09=20=20= psprintf("toast=20chunk=20sequence=20number=20%u=20exceeds=20the=20end=20= chunk=20sequence=20number=20%u",=0A-=09=09=09=09=09=09=09=09=20=20=20= curchunk,=20ctx->endchunk));=0A+=09=09=09=09=09=09=09=09=20=20=20= curchunk,=20endchunk));=0A=20=09=09return;=0A=20=09}=0A=20=0A-=09= expected_size=20=3D=20curchunk=20<=20ctx->totalchunks=20-=201=20?=20= TOAST_MAX_CHUNK_SIZE=0A-=09=09:=20ctx->attrsize=20-=20((ctx->totalchunks=20= -=201)=20*=20TOAST_MAX_CHUNK_SIZE);=0A+=09expected_size=20=3D=20curchunk=20= <=20endchunk=20?=20TOAST_MAX_CHUNK_SIZE=0A+=09=09:=20= VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer)=20-=20(endchunk=20*=20= TOAST_MAX_CHUNK_SIZE);=0A+=0A=20=09if=20(chunksize=20!=3D=20= expected_size)=0A-=09{=0A-=09=09report_corruption(ctx,=0A+=09=09= report_toast_corruption(ctx,=20ta,=0A=20=09=09=09=09=09=09=20=20= psprintf("toast=20chunk=20size=20%u=20differs=20from=20the=20expected=20= size=20%u",=0A=20=09=09=09=09=09=09=09=09=20=20=20chunksize,=20= expected_size));=0A-=09=09return;=0A-=09}=0A=20}=0A=20=0A=20/*=0A@@=20= -1177,17=20+1242,17=20@@=20check_toast_tuple(HeapTuple=20toasttup,=20= HeapCheckContext=20*ctx)=0A=20=20*=20found=20in=20ctx->tupstore.=0A=20=20= *=0A=20=20*=20This=20function=20follows=20the=20logic=20performed=20by=20= heap_deform_tuple(),=20and=20in=20the=0A-=20*=20case=20of=20a=20toasted=20= value,=20optionally=20continues=20along=20the=20logic=20of=0A-=20*=20= detoast_external_attr(),=20checking=20for=20any=20conditions=20that=20= would=20result=20in=0A-=20*=20either=20of=20those=20functions=20= Asserting=20or=20crashing=20the=20backend.=20=20The=20checks=0A-=20*=20= performed=20by=20Asserts=20present=20in=20those=20two=20functions=20are=20= also=20performed=20here.=0A-=20*=20In=20cases=20where=20those=20two=20= functions=20are=20a=20bit=20cavalier=20in=20their=20assumptions=0A-=20*=20= about=20data=20being=20correct,=20we=20perform=20additional=20checks=20= not=20present=20in=20either=0A-=20*=20of=20those=20two=20functions.=20=20= Where=20some=20condition=20is=20checked=20in=20both=20of=20those=0A-=20*=20= functions,=20we=20perform=20it=20here=20twice,=20as=20we=20parallel=20= the=20logical=20flow=20of=0A-=20*=20those=20two=20functions.=20=20The=20= presence=20of=20duplicate=20checks=20seems=20a=20reasonable=0A-=20*=20= price=20to=20pay=20for=20keeping=20this=20code=20tightly=20coupled=20= with=20the=20code=20it=0A-=20*=20protects.=0A+=20*=20case=20of=20a=20= toasted=20value,=20optionally=20stores=20the=20toast=20pointer=20so=20= later=20it=20can=0A+=20*=20be=20checked=20following=20the=20logic=20of=20= detoast_external_attr(),=20checking=20for=20any=0A+=20*=20conditions=20= that=20would=20result=20in=20either=20of=20those=20functions=20Asserting=20= or=0A+=20*=20crashing=20the=20backend.=20=20The=20checks=20performed=20= by=20Asserts=20present=20in=20those=20two=0A+=20*=20functions=20are=20= also=20performed=20here=20and=20in=20check_toasted_attribute.=20=20In=20= cases=0A+=20*=20where=20those=20two=20functions=20are=20a=20bit=20= cavalier=20in=20their=20assumptions=20about=20data=0A+=20*=20being=20= correct,=20we=20perform=20additional=20checks=20not=20present=20in=20= either=20of=20those=0A+=20*=20two=20functions.=20=20Where=20some=20= condition=20is=20checked=20in=20both=20of=20those=20functions,=0A+=20*=20= we=20perform=20it=20here=20twice,=20as=20we=20parallel=20the=20logical=20= flow=20of=20those=20two=0A+=20*=20functions.=20=20The=20presence=20of=20= duplicate=20checks=20seems=20a=20reasonable=20price=20to=20pay=0A+=20*=20= for=20keeping=20this=20code=20tightly=20coupled=20with=20the=20code=20it=20= protects.=0A=20=20*=0A=20=20*=20Returns=20true=20if=20the=20tuple=20= attribute=20is=20sane=20enough=20for=20processing=20to=0A=20=20*=20= continue=20on=20to=20the=20next=20attribute,=20false=20otherwise.=0A@@=20= -1195,12=20+1260,6=20@@=20check_toast_tuple(HeapTuple=20toasttup,=20= HeapCheckContext=20*ctx)=0A=20static=20bool=0A=20= check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20{=0A-=09struct=20= varatt_external=20toast_pointer;=0A-=09ScanKeyData=20toastkey;=0A-=09= SysScanDesc=20toastscan;=0A-=09SnapshotData=20SnapshotToast;=0A-=09= HeapTuple=09toasttup;=0A-=09bool=09=09found_toasttup;=0A=20=09Datum=09=09= attdatum;=0A=20=09struct=20varlena=20*attr;=0A=20=09char=09=20=20=20*tp;=09= =09=09=09/*=20pointer=20to=20the=20tuple=20data=20*/=0A@@=20-1335,13=20= +1394,44=20@@=20check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09=09= return=20true;=0A=20=0A=20=09/*=0A-=09=20*=20Must=20copy=20attr=20into=20= toast_pointer=20for=20alignment=20considerations=0A+=09=20*=20If=20this=20= tuple=20is=20eligible=20to=20be=20pruned,=20we=20cannot=20check=20the=20= toast.=0A+=09=20*=20Otherwise,=20we=20push=20a=20copy=20of=20the=20toast=20= tuple=20so=20we=20can=20check=20it=20after=0A+=09=20*=20releasing=20the=20= main=20table=20buffer=20lock.=0A=20=09=20*/=0A-=09= VARATT_EXTERNAL_GET_POINTER(toast_pointer,=20attr);=0A+=09if=20= (!ctx->tuple_could_be_pruned)=0A+=09{=0A+=09=09ToastedAttribute=20*ta;=0A= +=0A+=09=09ta=20=3D=20(ToastedAttribute=20*)=20= palloc0(sizeof(ToastedAttribute));=0A+=0A+=09=09= VARATT_EXTERNAL_GET_POINTER(ta->toast_pointer,=20attr);=0A+=09=09= ta->blkno=20=3D=20ctx->blkno;=0A+=09=09ta->offnum=20=3D=20ctx->offnum;=0A= +=09=09ta->attnum=20=3D=20ctx->attnum;=0A+=09=09ctx->toasted_attributes=20= =3D=20lappend(ctx->toasted_attributes,=20ta);=0A+=09}=0A+=0A+=09return=20= true;=0A+}=0A+=0A+/*=0A+=20*=20For=20each=20attribute=20collected=20in=20= ctx->toasted_attributes,=20look=20up=20the=20value=0A+=20*=20in=20the=20= toast=20table=20and=20perform=20checks=20on=20it.=20=20This=20function=20= should=20only=20be=0A+=20*=20called=20on=20toast=20pointers=20which=20= cannot=20be=20vacuumed=20away=20during=20our=0A+=20*=20processing.=0A+=20= */=0A+static=20void=0A+check_toasted_attribute(HeapCheckContext=20*ctx,=20= ToastedAttribute=20*ta)=0A+{=0A+=09SnapshotData=20SnapshotToast;=0A+=09= ScanKeyData=20toastkey;=0A+=09SysScanDesc=20toastscan;=0A+=09bool=09=09= found_toasttup;=0A+=09HeapTuple=09toasttup;=0A+=09int32=09=09chunkno;=0A= +=09int32=09=09endchunk;=0A=20=0A-=09ctx->attrsize=20=3D=20= VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer);=0A-=09ctx->endchunk=20=3D=20= (ctx->attrsize=20-=201)=20/=20TOAST_MAX_CHUNK_SIZE;=0A-=09= ctx->totalchunks=20=3D=20ctx->endchunk=20+=201;=0A+=09endchunk=20=3D=20= (VARATT_EXTERNAL_GET_EXTSIZE(ta->toast_pointer)=20-=201)=20/=20= TOAST_MAX_CHUNK_SIZE;=0A=20=0A=20=09/*=0A=20=09=20*=20Setup=20a=20scan=20= key=20to=20find=20chunks=20in=20toast=20table=20with=20matching=20= va_valueid=0A@@=20-1349,7=20+1439,7=20@@=20= check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09= ScanKeyInit(&toastkey,=0A=20=09=09=09=09(AttrNumber)=201,=0A=20=09=09=09=09= BTEqualStrategyNumber,=20F_OIDEQ,=0A-=09=09=09=09= ObjectIdGetDatum(toast_pointer.va_valueid));=0A+=09=09=09=09= ObjectIdGetDatum(ta->toast_pointer.va_valueid));=0A=20=0A=20=09/*=0A=20=09= =20*=20Check=20if=20any=20chunks=20for=20this=20toasted=20object=20exist=20= in=20the=20toast=20table,=0A@@=20-1360,27=20+1450,26=20@@=20= check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09=09=09=09=09=09=09= =09=09=09=20=20=20ctx->valid_toast_index,=0A=20=09=09=09=09=09=09=09=09=09= =09=20=20=20&SnapshotToast,=201,=0A=20=09=09=09=09=09=09=09=09=09=09=20=20= =20&toastkey);=0A-=09ctx->chunkno=20=3D=200;=0A+=09chunkno=20=3D=200;=0A=20= =09found_toasttup=20=3D=20false;=0A=20=09while=20((toasttup=20=3D=0A=20=09= =09=09systable_getnext_ordered(toastscan,=0A=20=09=09=09=09=09=09=09=09=09= =20ForwardScanDirection))=20!=3D=20NULL)=0A=20=09{=0A=20=09=09= found_toasttup=20=3D=20true;=0A-=09=09check_toast_tuple(toasttup,=20= ctx);=0A-=09=09ctx->chunkno++;=0A+=09=09check_toast_tuple(toasttup,=20= ctx,=20ta,=20chunkno,=20endchunk);=0A+=09=09chunkno++;=0A=20=09}=0A-=09= if=20(!found_toasttup)=0A-=09=09report_corruption(ctx,=0A-=09=09=09=09=09= =09=20=20psprintf("toasted=20value=20for=20attribute=20%u=20missing=20= from=20toast=20table",=0A-=09=09=09=09=09=09=09=09=20=20=20= ctx->attnum));=0A-=09else=20if=20(ctx->chunkno=20!=3D=20(ctx->endchunk=20= +=201))=0A-=09=09report_corruption(ctx,=0A-=09=09=09=09=09=09=20=20= psprintf("final=20toast=20chunk=20number=20%u=20differs=20from=20= expected=20value=20%u",=0A-=09=09=09=09=09=09=09=09=20=20=20= ctx->chunkno,=20(ctx->endchunk=20+=201)));=0A=20=09= systable_endscan_ordered(toastscan);=0A=20=0A-=09return=20true;=0A+=09if=20= (!found_toasttup)=0A+=09=09report_toast_corruption(ctx,=20ta,=0A+=09=09=09= =09=09=09=09=09psprintf("toasted=20value=20for=20attribute=20%u=20= missing=20from=20toast=20table",=0A+=09=09=09=09=09=09=09=09=09=09=20= ta->attnum));=0A+=09else=20if=20(chunkno=20!=3D=20(endchunk=20+=201))=0A= +=09=09report_toast_corruption(ctx,=20ta,=0A+=09=09=09=09=09=09=09=09= psprintf("final=20toast=20chunk=20number=20%u=20differs=20from=20= expected=20value=20%u",=0A+=09=09=09=09=09=09=09=09=09=09=20chunkno,=20= (endchunk=20+=201)));=0A=20}=0A=20=0A=20/*=0A@@=20-1391,8=20+1480,8=20@@=20= static=20void=0A=20check_tuple(HeapCheckContext=20*ctx)=0A=20{=0A=20=09= /*=0A-=09=20*=20Check=20various=20forms=20of=20tuple=20header=20= corruption,=20and=20if=20the=20header=20is=20too=0A-=09=20*=20corrupt,=20= do=20not=20continue=20with=20other=20checks.=0A+=09=20*=20Check=20= various=20forms=20of=20tuple=20header=20corruption,=20and=20if=20the=20= header=20is=0A+=09=20*=20too=20corrupt,=20do=20not=20continue=20with=20= other=20checks.=0A=20=09=20*/=0A=20=09if=20(!check_tuple_header(ctx))=0A=20= =09=09return;=0A@@=20-1423,7=20+1512,10=20@@=20= check_tuple(HeapCheckContext=20*ctx)=0A=20=09=20*=20Check=20each=20= attribute=20unless=20we=20hit=20corruption=20that=20confuses=20what=20to=20= do=0A=20=09=20*=20next,=20at=20which=20point=20we=20abort=20further=20= attribute=20checks=20for=20this=20tuple.=0A=20=09=20*=20Note=20that=20we=20= don't=20abort=20for=20all=20types=20of=20corruption,=20only=20for=20= those=0A-=09=20*=20types=20where=20we=20don't=20know=20how=20to=20= continue.=0A+=09=20*=20types=20where=20we=20don't=20know=20how=20to=20= continue.=20=20We=20also=20don't=20abort=20the=0A+=09=20*=20checking=20= of=20toasted=20attributes=20collected=20from=20the=20tuple=20prior=20to=0A= +=09=20*=20aborting.=20=20Those=20will=20still=20be=20checked=20later=20= along=20with=20other=20toasted=0A+=09=20*=20attributes=20collected=20= from=20the=20page.=0A=20=09=20*/=0A=20=09ctx->offset=20=3D=200;=0A=20=09= for=20(ctx->attnum=20=3D=200;=20ctx->attnum=20<=20ctx->natts;=20= ctx->attnum++)=0Adiff=20--git=20a/src/tools/pgindent/typedefs.list=20= b/src/tools/pgindent/typedefs.list=0Aindex=209e6777e9d0..862a6df949=20= 100644=0A---=20a/src/tools/pgindent/typedefs.list=0A+++=20= b/src/tools/pgindent/typedefs.list=0A@@=20-2558,6=20+2558,7=20@@=20= TmFromChar=0A=20TmToChar=0A=20ToastAttrInfo=0A=20ToastTupleContext=0A= +ToastedAttribute=0A=20TocEntry=0A=20TokenAuxData=0A=20TokenizedLine=0A= --=20=0A2.21.1=20(Apple=20Git-122.3)=0A=0A= --Apple-Mail=_762805D9-89AF-4F33-9298-4A3BFD0EBE1E Content-Disposition: attachment; filename=v18-0003-amcheck-improving-corruption-messages.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="v18-0003-amcheck-improving-corruption-messages.patch" Content-Transfer-Encoding: quoted-printable =46rom=204195bb5d6e39ec0cc2b323b0ad5b24c4c843df88=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20Mark=20Dilger=20=0A= Date:=20Fri,=202=20Apr=202021=2015:06:50=20-0700=0ASubject:=20[PATCH=20= v18=203/4]=20amcheck:=20improving=20corruption=20messages.=0A=0ARemoving=20= redundant=20mention=20of=20attnum=20in=20the=20corruption=20message=20= text,=0Aas=20the=20attnum=20is=20already=20its=20own=20separate=20= column.=0A=0AWhen=20reporting=20toast=20corruption,=20mentioning=20the=20= toast=20value=20in=20the=0Amessage=20since=20that=20information=20is=20= not=20otherwise=20reported.=0A---=0A=20contrib/amcheck/verify_heapam.c=20= =20=20=20=20=20=20=20=20=20=20|=2061=20+++++++++++++----------=0A=20= src/bin/pg_amcheck/t/004_verify_heapam.pl=20|=20=204=20+-=0A=202=20files=20= changed,=2037=20insertions(+),=2028=20deletions(-)=0A=0Adiff=20--git=20= a/contrib/amcheck/verify_heapam.c=20b/contrib/amcheck/verify_heapam.c=0A= index=20d8a3e966c4..cd1f2c4113=20100644=0A---=20= a/contrib/amcheck/verify_heapam.c=0A+++=20= b/contrib/amcheck/verify_heapam.c=0A@@=20-1179,7=20+1179,8=20@@=20= check_toast_tuple(HeapTuple=20toasttup,=20HeapCheckContext=20*ctx,=0A=20=09= if=20(isnull)=0A=20=09{=0A=20=09=09report_toast_corruption(ctx,=20ta,=0A= -=09=09=09=09=09=09=20=20pstrdup("toast=20chunk=20sequence=20number=20is=20= null"));=0A+=09=09=09=09=09=09=09=09psprintf("toast=20value=20%u=20has=20= toast=20chunk=20with=20null=20sequence=20number",=0A+=09=09=09=09=09=09=09= =09=09=09=20ta->toast_pointer.va_valueid));=0A=20=09=09return;=0A=20=09}=0A= =20=09chunk=20=3D=20DatumGetPointer(fastgetattr(toasttup,=203,=0A@@=20= -1187,7=20+1188,8=20@@=20check_toast_tuple(HeapTuple=20toasttup,=20= HeapCheckContext=20*ctx,=0A=20=09if=20(isnull)=0A=20=09{=0A=20=09=09= report_toast_corruption(ctx,=20ta,=0A-=09=09=09=09=09=09=20=20= pstrdup("toast=20chunk=20data=20is=20null"));=0A+=09=09=09=09=09=09=09=09= psprintf("toast=20value=20%u=20chunk=20data=20is=20null",=0A+=09=09=09=09= =09=09=09=09=09=09=20ta->toast_pointer.va_valueid));=0A=20=09=09return;=0A= =20=09}=0A=20=09if=20(!VARATT_IS_EXTENDED(chunk))=0A@@=20-1205,8=20= +1207,9=20@@=20check_toast_tuple(HeapTuple=20toasttup,=20= HeapCheckContext=20*ctx,=0A=20=09=09uint32=09=09header=20=3D=20= ((varattrib_4b=20*)=20chunk)->va_4byte.va_header;=0A=20=0A=20=09=09= report_toast_corruption(ctx,=20ta,=0A-=09=09=09=09=09=09=20=20= psprintf("corrupt=20extended=20toast=20chunk=20has=20invalid=20varlena=20= header:=20%0x=20(sequence=20number=20%d)",=0A-=09=09=09=09=09=09=09=09=20= =20=20header,=20curchunk));=0A+=09=09=09=09=09=09=09=09psprintf("toast=20= value=20%u=20corrupt=20extended=20chunk=20has=20invalid=20varlena=20= header:=20%0x=20(sequence=20number=20%d)",=0A+=09=09=09=09=09=09=09=09=09= =09=20ta->toast_pointer.va_valueid,=0A+=09=09=09=09=09=09=09=09=09=09=20= header,=20curchunk));=0A=20=09=09return;=0A=20=09}=0A=20=0A@@=20-1216,15=20= +1219,17=20@@=20check_toast_tuple(HeapTuple=20toasttup,=20= HeapCheckContext=20*ctx,=0A=20=09if=20(curchunk=20!=3D=20chunkno)=0A=20=09= {=0A=20=09=09report_toast_corruption(ctx,=20ta,=0A-=09=09=09=09=09=09=20=20= psprintf("toast=20chunk=20sequence=20number=20%u=20does=20not=20match=20= the=20expected=20sequence=20number=20%u",=0A-=09=09=09=09=09=09=09=09=20=20= =20curchunk,=20chunkno));=0A+=09=09=09=09=09=09=09=09psprintf("toast=20= value=20%u=20chunk=20sequence=20number=20%u=20does=20not=20match=20the=20= expected=20sequence=20number=20%u",=0A+=09=09=09=09=09=09=09=09=09=09=20= ta->toast_pointer.va_valueid,=0A+=09=09=09=09=09=09=09=09=09=09=20= curchunk,=20chunkno));=0A=20=09=09return;=0A=20=09}=0A=20=09if=20= (curchunk=20>=20endchunk)=0A=20=09{=0A=20=09=09= report_toast_corruption(ctx,=20ta,=0A-=09=09=09=09=09=09=20=20= psprintf("toast=20chunk=20sequence=20number=20%u=20exceeds=20the=20end=20= chunk=20sequence=20number=20%u",=0A-=09=09=09=09=09=09=09=09=20=20=20= curchunk,=20endchunk));=0A+=09=09=09=09=09=09=09=09psprintf("toast=20= value=20%u=20chunk=20sequence=20number=20%u=20exceeds=20the=20end=20= chunk=20sequence=20number=20%u",=0A+=09=09=09=09=09=09=09=09=09=09=20= ta->toast_pointer.va_valueid,=0A+=09=09=09=09=09=09=09=09=09=09=20= curchunk,=20endchunk));=0A=20=09=09return;=0A=20=09}=0A=20=0A@@=20= -1233,8=20+1238,9=20@@=20check_toast_tuple(HeapTuple=20toasttup,=20= HeapCheckContext=20*ctx,=0A=20=0A=20=09if=20(chunksize=20!=3D=20= expected_size)=0A=20=09=09report_toast_corruption(ctx,=20ta,=0A-=09=09=09= =09=09=09=20=20psprintf("toast=20chunk=20size=20%u=20differs=20from=20= the=20expected=20size=20%u",=0A-=09=09=09=09=09=09=09=09=20=20=20= chunksize,=20expected_size));=0A+=09=09=09=09=09=09=09=09psprintf("toast=20= value=20%u=20chunk=20size=20%u=20differs=20from=20the=20expected=20size=20= %u",=0A+=09=09=09=09=09=09=09=09=09=09=20ta->toast_pointer.va_valueid,=0A= +=09=09=09=09=09=09=09=09=09=09=20chunksize,=20expected_size));=0A=20}=0A= =20=0A=20/*=0A@@=20-1265,6=20+1271,7=20@@=20= check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09char=09=20=20=20= *tp;=09=09=09=09/*=20pointer=20to=20the=20tuple=20data=20*/=0A=20=09= uint16=09=09infomask;=0A=20=09Form_pg_attribute=20thisatt;=0A+=09struct=20= varatt_external=20toast_pointer;=0A=20=0A=20=09infomask=20=3D=20= ctx->tuphdr->t_infomask;=0A=20=09thisatt=20=3D=20= TupleDescAttr(RelationGetDescr(ctx->rel),=20ctx->attnum);=0A@@=20-1274,8=20= +1281,7=20@@=20check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09if=20= (ctx->tuphdr->t_hoff=20+=20ctx->offset=20>=20ctx->lp_len)=0A=20=09{=0A=20= =09=09report_corruption(ctx,=0A-=09=09=09=09=09=09=20=20= psprintf("attribute=20%u=20with=20length=20%u=20starts=20at=20offset=20= %u=20beyond=20total=20tuple=20length=20%u",=0A-=09=09=09=09=09=09=09=09=20= =20=20ctx->attnum,=0A+=09=09=09=09=09=09=20=20psprintf("attribute=20with=20= length=20%u=20starts=20at=20offset=20%u=20beyond=20total=20tuple=20= length=20%u",=0A=20=09=09=09=09=09=09=09=09=20=20=20thisatt->attlen,=0A=20= =09=09=09=09=09=09=09=09=20=20=20ctx->tuphdr->t_hoff=20+=20ctx->offset,=0A= =20=09=09=09=09=09=09=09=09=20=20=20ctx->lp_len));=0A@@=20-1295,8=20= +1301,7=20@@=20check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09=09= if=20(ctx->tuphdr->t_hoff=20+=20ctx->offset=20>=20ctx->lp_len)=0A=20=09=09= {=0A=20=09=09=09report_corruption(ctx,=0A-=09=09=09=09=09=09=09=20=20= psprintf("attribute=20%u=20with=20length=20%u=20ends=20at=20offset=20%u=20= beyond=20total=20tuple=20length=20%u",=0A-=09=09=09=09=09=09=09=09=09=20=20= =20ctx->attnum,=0A+=09=09=09=09=09=09=09=20=20psprintf("attribute=20with=20= length=20%u=20ends=20at=20offset=20%u=20beyond=20total=20tuple=20length=20= %u",=0A=20=09=09=09=09=09=09=09=09=09=20=20=20thisatt->attlen,=0A=20=09=09= =09=09=09=09=09=09=09=20=20=20ctx->tuphdr->t_hoff=20+=20ctx->offset,=0A=20= =09=09=09=09=09=09=09=09=09=20=20=20ctx->lp_len));=0A@@=20-1328,8=20= +1333,7=20@@=20check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09=09= if=20(va_tag=20!=3D=20VARTAG_ONDISK)=0A=20=09=09{=0A=20=09=09=09= report_corruption(ctx,=0A-=09=09=09=09=09=09=09=20=20psprintf("toasted=20= attribute=20%u=20has=20unexpected=20TOAST=20tag=20%u",=0A-=09=09=09=09=09= =09=09=09=09=20=20=20ctx->attnum,=0A+=09=09=09=09=09=09=09=20=20= psprintf("toasted=20attribute=20has=20unexpected=20TOAST=20tag=20%u",=0A=20= =09=09=09=09=09=09=09=09=09=20=20=20va_tag));=0A=20=09=09=09/*=20We=20= can't=20know=20where=20the=20next=20attribute=20begins=20*/=0A=20=09=09=09= return=20false;=0A@@=20-1343,8=20+1347,7=20@@=20= check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09if=20= (ctx->tuphdr->t_hoff=20+=20ctx->offset=20>=20ctx->lp_len)=0A=20=09{=0A=20= =09=09report_corruption(ctx,=0A-=09=09=09=09=09=09=20=20= psprintf("attribute=20%u=20with=20length=20%u=20ends=20at=20offset=20%u=20= beyond=20total=20tuple=20length=20%u",=0A-=09=09=09=09=09=09=09=09=20=20=20= ctx->attnum,=0A+=09=09=09=09=09=09=20=20psprintf("attribute=20with=20= length=20%u=20ends=20at=20offset=20%u=20beyond=20total=20tuple=20length=20= %u",=0A=20=09=09=09=09=09=09=09=09=20=20=20thisatt->attlen,=0A=20=09=09=09= =09=09=09=09=09=20=20=20ctx->tuphdr->t_hoff=20+=20ctx->offset,=0A=20=09=09= =09=09=09=09=09=09=20=20=20ctx->lp_len));=0A@@=20-1371,12=20+1374,17=20= @@=20check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=0A=20=09/*=20= It=20is=20external,=20and=20we're=20looking=20at=20a=20page=20on=20disk=20= */=0A=20=0A+=09/*=0A+=09=20*=20Must=20copy=20attr=20into=20toast_pointer=20= for=20alignment=20considerations=0A+=09=20*/=0A+=09= VARATT_EXTERNAL_GET_POINTER(toast_pointer,=20attr);=0A+=0A=20=09/*=20The=20= tuple=20header=20better=20claim=20to=20contain=20toasted=20values=20*/=0A= =20=09if=20(!(infomask=20&=20HEAP_HASEXTERNAL))=0A=20=09{=0A=20=09=09= report_corruption(ctx,=0A-=09=09=09=09=09=09=20=20psprintf("attribute=20= %u=20is=20external=20but=20tuple=20header=20flag=20HEAP_HASEXTERNAL=20= not=20set",=0A-=09=09=09=09=09=09=09=09=20=20=20ctx->attnum));=0A+=09=09=09= =09=09=09=20=20psprintf("toast=20value=20%u=20is=20external=20but=20= tuple=20header=20flag=20HEAP_HASEXTERNAL=20not=20set",=0A+=09=09=09=09=09= =09=09=09=20=20=20toast_pointer.va_valueid));=0A=20=09=09return=20true;=0A= =20=09}=0A=20=0A@@=20-1384,8=20+1392,8=20@@=20= check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09if=20= (!ctx->rel->rd_rel->reltoastrelid)=0A=20=09{=0A=20=09=09= report_corruption(ctx,=0A-=09=09=09=09=09=09=20=20psprintf("attribute=20= %u=20is=20external=20but=20relation=20has=20no=20toast=20relation",=0A-=09= =09=09=09=09=09=09=09=20=20=20ctx->attnum));=0A+=09=09=09=09=09=09=20=20= psprintf("toast=20value=20%u=20is=20external=20but=20relation=20has=20no=20= toast=20relation",=0A+=09=09=09=09=09=09=09=09=20=20=20= toast_pointer.va_valueid));=0A=20=09=09return=20true;=0A=20=09}=0A=20=0A= @@=20-1464,12=20+1472,13=20@@=20check_toasted_attribute(HeapCheckContext=20= *ctx,=20ToastedAttribute=20*ta)=0A=20=0A=20=09if=20(!found_toasttup)=0A=20= =09=09report_toast_corruption(ctx,=20ta,=0A-=09=09=09=09=09=09=09=09= psprintf("toasted=20value=20for=20attribute=20%u=20missing=20from=20= toast=20table",=0A-=09=09=09=09=09=09=09=09=09=09=20ta->attnum));=0A+=09=09= =09=09=09=09=09=09psprintf("toast=20value=20%u=20not=20found=20in=20= toast=20table",=0A+=09=09=09=09=09=09=09=09=09=09=20= ta->toast_pointer.va_valueid));=0A=20=09else=20if=20(chunkno=20!=3D=20= (endchunk=20+=201))=0A=20=09=09report_toast_corruption(ctx,=20ta,=0A-=09=09= =09=09=09=09=09=09psprintf("final=20toast=20chunk=20number=20%u=20= differs=20from=20expected=20value=20%u",=0A-=09=09=09=09=09=09=09=09=09=09= =20chunkno,=20(endchunk=20+=201)));=0A+=09=09=09=09=09=09=09=09= psprintf("toast=20value=20%u=20was=20expected=20to=20end=20at=20chunk=20= %u,=20but=20ended=20at=20chunk=20%u",=0A+=09=09=09=09=09=09=09=09=09=09=20= ta->toast_pointer.va_valueid,=0A+=09=09=09=09=09=09=09=09=09=09=20= (endchunk=20+=201),=20chunkno));=0A=20}=0A=20=0A=20/*=0Adiff=20--git=20= a/src/bin/pg_amcheck/t/004_verify_heapam.pl=20= b/src/bin/pg_amcheck/t/004_verify_heapam.pl=0Aindex=20= 36607596b1..307f14611c=20100644=0A---=20= a/src/bin/pg_amcheck/t/004_verify_heapam.pl=0A+++=20= b/src/bin/pg_amcheck/t/004_verify_heapam.pl=0A@@=20-480,7=20+480,7=20@@=20= for=20(my=20$tupidx=20=3D=200;=20$tupidx=20<=20ROWCOUNT;=20$tupidx++)=0A=20= =0A=20=09=09$header=20=3D=20header(0,=20$offnum,=201);=0A=20=09=09push=20= @expected,=0A-=09=09=09qr/${header}attribute=20\d+=20with=20length=20\d+=20= ends=20at=20offset=20\d+=20beyond=20total=20tuple=20length=20\d+/;=0A+=09= =09=09qr/${header}attribute=20with=20length=20\d+=20ends=20at=20offset=20= \d+=20beyond=20total=20tuple=20length=20\d+/;=0A=20=09}=0A=20=09elsif=20= ($offnum=20=3D=3D=2013)=0A=20=09{=0A@@=20-489,7=20+489,7=20@@=20for=20= (my=20$tupidx=20=3D=200;=20$tupidx=20<=20ROWCOUNT;=20$tupidx++)=0A=20=0A=20= =09=09$header=20=3D=20header(0,=20$offnum,=202);=0A=20=09=09push=20= @expected,=0A-=09=09=09qr/${header}toasted=20value=20for=20attribute=202=20= missing=20from=20toast=20table/;=0A+=09=09=09qr/${header}toast=20value=20= \d+=20not=20found=20in=20toast=20table/;=0A=20=09}=0A=20=09elsif=20= ($offnum=20=3D=3D=2014)=0A=20=09{=0A--=20=0A2.21.1=20(Apple=20Git-122.3)=0A= =0A= --Apple-Mail=_762805D9-89AF-4F33-9298-4A3BFD0EBE1E Content-Disposition: attachment; filename=v18-0004-amcheck-adding-toast-pointer-corruption-checks.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="v18-0004-amcheck-adding-toast-pointer-corruption-checks.patch" Content-Transfer-Encoding: quoted-printable =46rom=20c3645b0af895767c40b1ead2f2e076c7c33e482b=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20Mark=20Dilger=20=0A= Date:=20Fri,=202=20Apr=202021=2015:07:52=20-0700=0ASubject:=20[PATCH=20= v18=204/4]=20amcheck:=20adding=20toast=20pointer=20corruption=20checks=0A= =0AVerifying=20that=20toast=20pointer=20va_toastrelid=20fields=20match=20= their=20heap=0Atable's=20reltoastrelid.=0A=0AChecking=20the=20extsize=20= for=20a=20toast=20pointer=20against=20the=20raw=20size.=20=20This=0A= check=20could=20fail=20if=20buggy=20compression=20logic=20fails=20to=20= notice=20that=0Acompressing=20the=20attribute=20makes=20it=20bigger.=20=20= But=20assuming=20the=20logic=0Afor=20that=20is=20correct,=20overlarge=20= extsize=20indicates=20a=20corrupted=20toast=0Apointer.=0A=0AChecking=20= the=20toast=20is=20not=20too=20large=20to=20be=20allocated.=20=20No=20= such=0Atoasted=20value=20should=20ever=20be=20stored,=20but=20a=20= corrupted=20toast=20pointer=0Acould=20record=20an=20unreasonbly=20large=20= size,=20so=20check=20that.=0A---=0A=20contrib/amcheck/verify_heapam.c=20=20= =20=20=20=20=20=20=20=20=20|=2034=20+++++++++++++++++++-=0A=20= src/bin/pg_amcheck/t/004_verify_heapam.pl=20|=2039=20= +++++++++++++++++++++--=0A=202=20files=20changed,=2069=20insertions(+),=20= 4=20deletions(-)=0A=0Adiff=20--git=20a/contrib/amcheck/verify_heapam.c=20= b/contrib/amcheck/verify_heapam.c=0Aindex=20cd1f2c4113..b2e121ed38=20= 100644=0A---=20a/contrib/amcheck/verify_heapam.c=0A+++=20= b/contrib/amcheck/verify_heapam.c=0A@@=20-1397,6=20+1397,28=20@@=20= check_tuple_attribute(HeapCheckContext=20*ctx)=0A=20=09=09return=20true;=0A= =20=09}=0A=20=0A+=09/*=20The=20toast=20pointer=20had=20better=20point=20= at=20the=20relation's=20toast=20table=20*/=0A+=09if=20= (toast_pointer.va_toastrelid=20!=3D=20ctx->rel->rd_rel->reltoastrelid)=0A= +=09{=0A+=09=09report_corruption(ctx,=0A+=09=09=09=09=09=09=20=20= psprintf("toast=20value=20%u=20toast=20relation=20oid=20%u=20differs=20= from=20expected=20oid=20%u",=0A+=09=09=09=09=09=09=09=09=20=20=20= toast_pointer.va_valueid,=0A+=09=09=09=09=09=09=09=09=20=20=20= toast_pointer.va_toastrelid,=0A+=09=09=09=09=09=09=09=09=20=20=20= ctx->rel->rd_rel->reltoastrelid));=0A+=09=09return=20true;=0A+=09}=0A+=0A= +=09/*=20Compression=20should=20never=20expand=20the=20attribute=20*/=0A= +=09if=20(VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer)=20>=20= toast_pointer.va_rawsize=20-=20VARHDRSZ)=0A+=09{=0A+=09=09= report_corruption(ctx,=0A+=09=09=09=09=09=09=20=20psprintf("toast=20= value=20%u=20external=20size=20%u=20exceeds=20maximum=20expected=20for=20= rawsize=20%u",=0A+=09=09=09=09=09=09=09=09=20=20=20= toast_pointer.va_valueid,=0A+=09=09=09=09=09=09=09=09=20=20=20= VARATT_EXTERNAL_GET_EXTSIZE(toast_pointer),=0A+=09=09=09=09=09=09=09=09=20= =20=20toast_pointer.va_rawsize));=0A+=09=09return=20true;=0A+=09}=0A+=0A=20= =09/*=20If=20we=20were=20told=20to=20skip=20toast=20checking,=20then=20= we're=20done.=20*/=0A=20=09if=20(ctx->toast_rel=20=3D=3D=20NULL)=0A=20=09= =09return=20true;=0A@@=20-1471,14=20+1493,24=20@@=20= check_toasted_attribute(HeapCheckContext=20*ctx,=20ToastedAttribute=20= *ta)=0A=20=09systable_endscan_ordered(toastscan);=0A=20=0A=20=09if=20= (!found_toasttup)=0A+=09{=0A=20=09=09report_toast_corruption(ctx,=20ta,=0A= =20=09=09=09=09=09=09=09=09psprintf("toast=20value=20%u=20not=20found=20= in=20toast=20table",=0A=20=09=09=09=09=09=09=09=09=09=09=20= ta->toast_pointer.va_valueid));=0A-=09else=20if=20(chunkno=20!=3D=20= (endchunk=20+=201))=0A+=09=09return;=0A+=09}=0A+=0A+=09if=20(chunkno=20= !=3D=20(endchunk=20+=201))=0A=20=09=09report_toast_corruption(ctx,=20ta,=0A= =20=09=09=09=09=09=09=09=09psprintf("toast=20value=20%u=20was=20expected=20= to=20end=20at=20chunk=20%u,=20but=20ended=20at=20chunk=20%u",=0A=20=09=09= =09=09=09=09=09=09=09=09=20ta->toast_pointer.va_valueid,=0A=20=09=09=09=09= =09=09=09=09=09=09=20(endchunk=20+=201),=20chunkno));=0A+=0A+=09if=20= (!AllocSizeIsValid(ta->toast_pointer.va_rawsize))=0A+=09=09= report_toast_corruption(ctx,=20ta,=0A+=09=09=09=09=09=09=09=09= psprintf("toast=20value=20%u=20rawsize=20%u=20too=20large=20to=20be=20= allocated",=0A+=09=09=09=09=09=09=09=09=09=09=20= ta->toast_pointer.va_valueid,=0A+=09=09=09=09=09=09=09=09=09=09=20= ta->toast_pointer.va_rawsize));=0A=20}=0A=20=0A=20/*=0Adiff=20--git=20= a/src/bin/pg_amcheck/t/004_verify_heapam.pl=20= b/src/bin/pg_amcheck/t/004_verify_heapam.pl=0Aindex=20= 307f14611c..de525fbdd8=20100644=0A---=20= a/src/bin/pg_amcheck/t/004_verify_heapam.pl=0A+++=20= b/src/bin/pg_amcheck/t/004_verify_heapam.pl=0A@@=20-224,7=20+224,7=20@@=20= my=20$rel=20=3D=20$node->safe_psql('postgres',=20qq(SELECT=20= pg_relation_filepath('public.te=0A=20my=20$relpath=20=3D=20= "$pgdata/$rel";=0A=20=0A=20#=20Insert=20data=20and=20freeze=20= public.test=0A-use=20constant=20ROWCOUNT=20=3D>=2016;=0A+use=20constant=20= ROWCOUNT=20=3D>=2019;=0A=20$node->safe_psql('postgres',=20qq(=0A=20=09= INSERT=20INTO=20public.test=20(a,=20b,=20c)=0A=20=09=09VALUES=20(=0A@@=20= -240,6=20+240,13=20@@=20my=20$relfrozenxid=20=3D=20= $node->safe_psql('postgres',=0A=20my=20$datfrozenxid=20=3D=20= $node->safe_psql('postgres',=0A=20=09q(select=20datfrozenxid=20from=20= pg_database=20where=20datname=20=3D=20'postgres'));=0A=20=0A+#=20Find=20= our=20toast=20relation=20id=0A+my=20$toastrelid=20=3D=20= $node->safe_psql('postgres',=20qq(=0A+=09SELECT=20c.reltoastrelid=0A+=09=09= FROM=20pg_catalog.pg_class=20c=0A+=09=09WHERE=20c.oid=20=3D=20= 'public.test'::regclass=0A+=09=09));=0A+=0A=20#=20Sanity=20check=20that=20= our=20'test'=20table=20has=20a=20relfrozenxid=20newer=20than=20the=0A=20= #=20datfrozenxid=20for=20the=20database,=20and=20that=20the=20= datfrozenxid=20is=20greater=20than=20the=0A=20#=20first=20normal=20xid.=20= =20We=20rely=20on=20these=20invariants=20in=20some=20of=20our=20tests.=0A= @@=20-296,7=20+303,7=20@@=20close($file)=0A=20$node->start;=0A=20=0A=20#=20= Ok,=20Xids=20and=20page=20layout=20look=20ok.=20=20We=20can=20run=20= corruption=20tests.=0A-plan=20tests=20=3D>=2019;=0A+plan=20tests=20=3D>=20= 22;=0A=20=0A=20#=20Check=20that=20pg_amcheck=20runs=20against=20the=20= uncorrupted=20table=20without=20error.=0A=20= $node->command_ok(['pg_amcheck',=20'-p',=20$port,=20'postgres'],=0A@@=20= -501,7=20+508,7=20@@=20for=20(my=20$tupidx=20=3D=200;=20$tupidx=20<=20= ROWCOUNT;=20$tupidx++)=0A=20=09=09push=20@expected,=0A=20=09=09=09= qr/${header}multitransaction=20ID=204=20equals=20or=20exceeds=20next=20= valid=20multitransaction=20ID=201/;=0A=20=09}=0A-=09elsif=20($offnum=20= =3D=3D=2015)=09#=20Last=20offnum=20must=20equal=20ROWCOUNT=0A+=09elsif=20= ($offnum=20=3D=3D=2015)=0A=20=09{=0A=20=09=09#=20Set=20both=20= HEAP_XMAX_COMMITTED=20and=20HEAP_XMAX_IS_MULTI=0A=20=09=09= $tup->{t_infomask}=20|=3D=20HEAP_XMAX_COMMITTED;=0A@@=20-511,6=20+518,32=20= @@=20for=20(my=20$tupidx=20=3D=200;=20$tupidx=20<=20ROWCOUNT;=20= $tupidx++)=0A=20=09=09push=20@expected,=0A=20=09=09=09= qr/${header}multitransaction=20ID=204000000000=20precedes=20relation=20= minimum=20multitransaction=20ID=20threshold=201/;=0A=20=09}=0A+=09elsif=20= ($offnum=20=3D=3D=2016)=0A+=09{=0A+=09=09#=20Corrupt=20column=20c's=20= toast=20pointer=20va_toastrelid=20field=0A+=09=09my=20$otherid=20=3D=20= $toastrelid=20+=201;=0A+=09=09$tup->{c_va_toastrelid}=20=3D=20$otherid;=0A= +=09=09$header=20=3D=20header(0,=20$offnum,=202);=0A+=09=09push=20= @expected,=0A+=09=09=09qr/${header}toast=20value=20\d+=20toast=20= relation=20oid=20$otherid=20differs=20from=20expected=20oid=20= $toastrelid/;=0A+=09}=0A+=09elsif=20($offnum=20=3D=3D=2017)=0A+=09{=0A+=09= =09#=20Corrupt=20column=20c's=20toast=20pointer=20va_extinfo=20field=0A+=09= =09$tup->{c_va_extinfo}=20=3D=207654321;=0A+=09=09$header=20=3D=20= header(0,=20$offnum,=202);=0A+=09=09push=20@expected,=0A+=09=09=09= qr/${header}toast=20value=20\d+=20external=20size=207654321=20exceeds=20= maximum=20expected=20for=20rawsize=2010004/;=0A+=09}=0A+=09elsif=20= ($offnum=20=3D=3D=2018)=09#=20Last=20offnum=20should=20equal=20= ROWCOUNT-1=0A+=09{=0A+=09=09#=20Corrupt=20column=20c's=20toast=20pointer=20= va_rawsize=20field=20with=20a=20value=0A+=09=09#=20exceeding=20maximum=20= allowable=20allocation=20size=0A+=09=09$tup->{c_va_rawsize}=20=3D=20= 0x40000000;=0A+=09=09$header=20=3D=20header(0,=20$offnum,=202);=0A+=09=09= push=20@expected,=0A+=09=09=09qr/${header}toast=20value=20\d+=20rawsize=20= 1073741824=20too=20large=20to=20be=20allocated/;=0A+=09}=0A=20=09= write_tuple($file,=20$offset,=20$tup);=0A=20}=0A=20close($file)=0A--=20=0A= 2.21.1=20(Apple=20Git-122.3)=0A=0A= --Apple-Mail=_762805D9-89AF-4F33-9298-4A3BFD0EBE1E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 =E2=80=94 Mark Dilger EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company --Apple-Mail=_762805D9-89AF-4F33-9298-4A3BFD0EBE1E--