Subject: Re: IRe: BUG #16792: silent corruption of GIN index resulting in
 SELECTs returning non-matching rows
To: Pawel Kudzia <kudzia@gmail.com>
Cc: Peter Geoghegan <pg@bowt.ie>, Alexander Korotkov <aekorotkov@gmail.com>,
 PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>,
 Tom Lane <tgl@sss.pgh.pa.us>
References: <16792-b1913b6b4e098331@postgresql.org>
 <CAJYBUS84aSgCNhcusOJQfeWFy1V3U55AOmRUWcpd+SsF9_qp=A@mail.gmail.com>
 <920041ab-e975-afe3-77d0-934a7c50965d@iki.fi>
 <CAJYBUS8S-k7r-VQ3Vv7Asz9KiW1u50VT5k7fq5BSgVXJ_W4KYA@mail.gmail.com>
 <06cedb17-6cbe-6586-caa6-2ff618a35517@iki.fi>
 <CAH2-Wz=ocXi1NeaWVAO5JndKkppF4WNLBXU=MmCd__B5en3pWw@mail.gmail.com>
 <CAJYBUS8aBQQL22oHsAwjHdwYfdB_NMzt7-sZxhxiOdEdn7cOkw@mail.gmail.com>
 <e31fc916-1aee-18bc-ba9a-c612b3cf4f82@iki.fi>
 <CAJYBUS_nx9oehEkTi+phrAvEcdQAcTGkofLOhGdGoLLHK=kGAg@mail.gmail.com>
 <d4f4d5e2-35ef-58e4-4f31-5fdfb2c146a0@iki.fi>
 <CAJYBUS9_VUUYfAen+_pWGa4xxRR-2uvX+_Ycg6W+NqBXVwLY2Q@mail.gmail.com>
 <470ba78f-4784-f90d-91ba-dcc05d6d4738@iki.fi>
 <CAJYBUS_O8ZYvKgF8A+gwR3=ja0SZGmBBee9gVWBpHYrULMOHVQ@mail.gmail.com>
 <14dfcfab-36df-a92d-a42c-01065acf0036@iki.fi>
 <CAJYBUS-ZUZJffsacq8bX_2CQJOu7kr2=UYBKGgjDTtO9Arcm8A@mail.gmail.com>
 <8b21e29c-f236-ef40-452f-3fb829b37990@iki.fi>
 <CAJYBUS-cPiVDonc7M-OESZNt+aKOuLbtuMY5=4VKxndRLzqeQA@mail.gmail.com>
From: Heikki Linnakangas <hlinnaka@iki.fi>
Message-ID: <9fdbb584-1e10-6a55-ecc2-9ba8b5dca1cf@iki.fi>
Date: Thu, 22 Jul 2021 10:25:43 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: 
 <CAJYBUS-cPiVDonc7M-OESZNt+aKOuLbtuMY5=4VKxndRLzqeQA@mail.gmail.com>
Content-Type: multipart/mixed;
 boundary="------------2EA3246A77B82151CCF0F806"
Content-Language: en-US
Archived-At: 
 <https://www.postgresql.org/message-id/9fdbb584-1e10-6a55-ecc2-9ba8b5dca1cf%40iki.fi>
Precedence: bulk

This is a multi-part message in MIME format.
--------------2EA3246A77B82151CCF0F806
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

On 20/07/2021 18:58, Pawel Kudzia wrote:
> On Tue, Jul 20, 2021 at 5:34 PM Heikki Linnakangas <hlinnaka@iki.fi> wrote:
>> Pawel: Can you test once again with the attached amcheck version?
> 
> patch v4-0001-Amcheck-for-GIN-13stable.patch applied to
> postgresql-13_13.3.orig.tar.bz2
> attached - log and result of the check.
> 
> executing "select
> gin_index_parent_check('entity_attribute_name_ids_gin');" took
> considerably longer this time and it ended with :
> 
> [524767.920498] postgres[29716]: segfault at fffffffffffffffe ip
> 00007f214c20a2d8 sp 00007ffe2a5fd040 error 5 in
> amcheck.so[7f214c209000+6000]
> [524767.920521] Code: 00 00 41 0f b7 4e 04 41 8b 56 0c 41 8b 76 10 49
> 89 c1 48 8d 04 40 c1 e1 10 45 0f b7 46 08 48 8d 7c 43 fa 41 0f b7 46
> 06 09 c1 <0f> b7 47 04 50 0f b7 07 0f b7 7f 02 c1 e0 10 09 f8 50 0f b7
> 43 04

Hmph, another set of bugs in the amcheck patch. It didn't handle empty 
entry tree items, nor empty posting tree pages. And there was another 
bug, which caused it to scan entry tree pages multiple times; that's why 
the same WARNING was printed multiple times.

Fixed those bugs, new patch version attached. Pawel, can you test this 
again, please? At this point, I'm pretty sure this isn't going to reveal 
any more information about the original problem, but at least we're 
ironing out bugs from the 'amcheck' patch..

I'm grasping at straws here, but here's one more thing we could try: the 
query returned these incorrect tuples:

      ctid     | entity_id
--------------+-----------
  (4002784,1)  |  38048120
  (4002869,14) |  95333744
(2 rows)

We know those entries are in the GIN index with key '1373', when they 
shouldn't be. But I wonder if the correct keys for those tuples are 
present? Pawel, can you try this, please:

-- persuade planner to use a bitmap index scan
set enable_seqscan=off;
set enable_tidscan=off;
-- avoid lossy bitmaps
set work_mem='1GB';

-- Explain analyze first to check it's using a bitmap index scan and
-- no lossy pages.
explain analyze
SELECT ctid, entity_id FROM entity WHERE attribute_name_ids && 
'{38048120}' and ctid='(4002784,1)';
SELECT ctid, entity_id FROM entity WHERE attribute_name_ids && 
'{38048120}' and ctid='(4002784,1)';

explain analyze
SELECT ctid, entity_id FROM entity WHERE attribute_name_ids && 
'{95333744}' and ctid='(4002869,14)';
SELECT ctid, entity_id FROM entity WHERE attribute_name_ids && 
'{95333744}' and ctid='(4002869,14)';


- Heikki

--------------2EA3246A77B82151CCF0F806
Content-Type: text/x-patch; charset=UTF-8;
 name="v5-0001-Amcheck-for-GIN-13stable.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="v5-0001-Amcheck-for-GIN-13stable.patch"