Subject: Re: [PATCHES] GIN improvements
From: Jeff Davis <pgsql@j-davis.com>
To: Teodor Sigaev <teodor@sigaev.ru>
Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>,
	Tom Lane <tgl@sss.pgh.pa.us>, Pgsql Hackers <pgsql-hackers@postgresql.org>
In-Reply-To: <49708008.8070109@sigaev.ru>
References: <483FD205.4090904@sigaev.ru> <486BC501.6060605@sigaev.ru>
	<27880.1215810535@sss.pgh.pa.us> <487A6E0C.3090600@sigaev.ru>
	<8993.1216843701@sss.pgh.pa.us> <4888A594.6010102@sigaev.ru>
	<490B07BA.6030109@sigaev.ru> <492F04D9.5070404@enterprisedb.com>
	<4942A137.3000502@sigaev.ru> <1229910966.2285.76.camel@jdavis>
	<49708008.8070109@sigaev.ru>
Content-Type: text/plain
Date: Sun, 18 Jan 2009 23:08:25 -0800
Message-Id: <1232348905.7281.114.camel@jdavis>
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit

On Fri, 2009-01-16 at 15:39 +0300, Teodor Sigaev wrote:
> > START_CRIT_SECTION();
> > ...
> > l = PageAddItem(...);
> > if (l == InvalidOffsetNumber)
> >     elog(ERROR, "failed to add item to index page in \"%s\"",
> >          RelationGetRelationName(index));
> > 
> > It's no use using ERROR, because it will turn into PANIC, which is
> I did that similar to other GIN/GiST places. BTW, BTree  directly emits PANIC if
> PageAddItem fails
> 

I'd still prefer PANIC over an ERROR that will always turn into a PANIC.
I'll leave it as you did, though.

> > 
> > 4. Heikki mentioned:
> > http://archives.postgresql.org/pgsql-hackers/2008-11/msg01832.php
> > 
> > "To make things worse, a query will fail if all the matching 
> > fast-inserted tuples don't fit in the non-lossy tid bitmap."
> > 
> > That issue still remains, correct? Is there a resolution to that?
> 
> Now gincostestimate can forbid index scan by disable_cost (see Changes). Of 
> course, it doesn't prevent failure in case of large update (for example), but it 
> prevents in most cases. BTW, because of sequential scan of pending list cost of 
> scan grows up fast and index scan becomes non-optimal.

Is this a 100% bulletproof solution, or is it still possible for a query
to fail due to the pending list? It relies on the stats collector, so
perhaps in rare cases it could still fail?

It might be surprising though, that after an UPDATE and before a VACUUM,
the gin index just stops working (if work_mem is too low). For many
use-cases, if GIN is not used, it's just as bad as the query failing,
because it would be so slow.

Can you explain why the tbm must not be lossy?

Also, can you clarify why a large update can cause a problem? In the
previous discussion, you suggested that it force normal index inserts
after a threshold based on work_mem:

http://archives.postgresql.org/pgsql-hackers/2008-12/msg00065.php

Regards,
	Jeff Davis