From: Teodor Sigaev <teodor@sigaev.ru>
To: Jeff Davis <pgsql@j-davis.com>
Cc: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Pgsql Hackers <pgsql-hackers@postgresql.org>
Subject: Re: [PATCHES] GIN improvements
Date: Mon, 19 Jan 2009 19:53:22 +0300
Message-ID: <4974B002.3040202@sigaev.ru> (raw)
In-Reply-To: <1232348905.7281.114.camel@jdavis>
References: <483FD205.4090904@sigaev.ru>
<486BC501.6060605@sigaev.ru>
<27880.1215810535@sss.pgh.pa.us>
<487A6E0C.3090600@sigaev.ru>
<8993.1216843701@sss.pgh.pa.us>
<4888A594.6010102@sigaev.ru>
<490B07BA.6030109@sigaev.ru>
<492F04D9.5070404@enterprisedb.com>
<4942A137.3000502@sigaev.ru>
<1229910966.2285.76.camel@jdavis>
<49708008.8070109@sigaev.ru>
<1232348905.7281.114.camel@jdavis>
Changes:
Results of pernding list's scan now are placed directly in resulting
tidbitmap. This saves cycles for filtering results and reduce memory usage.
Also, it allows to not check losiness of tbm.
> Is this a 100% bulletproof solution, or is it still possible for a query
> to fail due to the pending list? It relies on the stats collector, so
> perhaps in rare cases it could still fail?
Yes :(
> Can you explain why the tbm must not be lossy?
The problem with lossy tbm has two aspects:
- amgettuple interface hasn't possibility to work with page-wide result instead
of exact ItemPointer. amgettuple can not return just a block number as
amgetbitmap can.
- Because of concurrent vacuum process: while we scan pending list, it's
content could be transferred into regular structure of index and then we will
find the same tuple twice. Again, amgettuple hasn't protection from that,
only amgetbitmap has it. So, we need to filter results from regular GIN
by results from pending list. ANd for filtering we can't use lossy tbm.
v0.21 prevents from that fail on call of gingetbitmap, because now all results
are collected in single resulting tidbitmap.
> Also, can you clarify why a large update can cause a problem? In the
If query looks like
UPDATE tbl SET col=... WHERE col ... and planner choose GIN indexscan over col
then there is a probability of increasing of pending list over non-lossy limit.
> previous discussion, you suggested that it force normal index inserts
> after a threshold based on work_mem:
>
> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00065.php
I see only two guaranteed solution of the problem:
- after limit is reached, force normal index inserts. One of the motivation of
patch was frequent question from users: why update of whole table with GIN index
is so slow? So this way will not resolve this question.
- after limit is reached, force cleanup of pending list by calling
gininsertcleanup. Not very good, because users sometimes will see a huge
execution time of simple insert. Although users who runs a huge update should be
satisfied.
I have difficulties in a choice of way. Seems to me, the better will be second
way: if user gets very long time of insertion then (auto)vacuum of his
installation should tweaked.
--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/Attachments:
fast_insert_gin-0.21.gz (application/x-tar, 23.3K) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-patches@postgresql.org
Cc: teodor@sigaev.ru, pgsql@j-davis.com, heikki.linnakangas@enterprisedb.com, tgl@sss.pgh.pa.us, pgsql-hackers@postgresql.org
Subject: Re: [PATCHES] GIN improvements
In-Reply-To: <4974B002.3040202@sigaev.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox