Subject: Re: [WIP] [B-Tree] Retail IndexTuple deletion
To: Peter Geoghegan <pg@bowt.ie>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>,
 Masahiko Sawada <sawada.mshk@gmail.com>
References: <425db134-8bba-005c-b59d-56e50de3b41e@postgrespro.ru>
 <CAH2-WzkOmpQ30nRy6=fdQ8Um8MKBRZ9yU6MwEFDQnWcVb3dY8Q@mail.gmail.com>
 <CAH2-Wz=bhuwATGhiQVACbEcVrdGJ-TKyTCg7rZtm1ncRzSuGfw@mail.gmail.com>
 <CAD21AoC-31Fr2JGmCpQPjJNZb-bvH38gURAC7256aJg3jO2C7w@mail.gmail.com>
 <CAH2-Wz=U-m6rDUdj2JcGycX3OT+0588ZCsauCfC7Fdv3_UmKGQ@mail.gmail.com>
 <5affa50e-72a9-aef5-d71f-f5f551e01aaf@postgrespro.ru>
 <CAH2-WzmpjdeU77aJf1gtte3QhWrnwMffx3pPJQpEDHA1zbx3gw@mail.gmail.com>
 <e4cec53a-ab12-4858-d9d8-5fac9948a4c7@postgrespro.ru>
 <CAH2-Wz=mkrkQJJxHjDgiTv25yk0yqrx0EOn5exAHvPyQRR6iNQ@mail.gmail.com>
From: "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>
Message-ID: <9a008ab3-f591-e5e9-8e03-c0dd1dc13579@postgrespro.ru>
Date: Thu, 28 Jun 2018 10:36:44 +0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.8.0
MIME-Version: 1.0
In-Reply-To: 
 <CAH2-Wz=mkrkQJJxHjDgiTv25yk0yqrx0EOn5exAHvPyQRR6iNQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Precedence: bulk


On 28.06.2018 05:00, Peter Geoghegan wrote:
> On Tue, Jun 26, 2018 at 11:40 PM, Andrey V. Lepikhov
> <a.lepikhov@postgrespro.ru> wrote:
>> I still believe that the patch for physical TID ordering in btree:
>> 1) has its own value, not only for target deletion,
>> 2) will require only a few local changes in my code,
>> and this patches can be developed independently.
> 
> I want to be clear on something now: I just don't think that this
> patch has any chance of getting committed without something like my
> own patch to go with it. The worst case for your patch without that
> component is completely terrible. It's not really important for you to
> actually formally make it part of your patch, so I'm not going to
> insist on that or anything, but the reality is that my patch does not
> have independent value -- and neither does yours.
> 
As I wrote before in the last email, I will integrate TID sorting to my 
patches right now. Please, give me access to the last version of your 
code, if it possible.
You can track the progress at https://github.com/danolivo/postgres git 
repository

> I'm sorry if that sounds harsh, but this is a difficult, complicated
> project. It's better to be clear about this stuff earlier on.

Ok. It is clear now.
> 
>> I prepare third version of the patches. Summary:
>> 1. Mask DEAD tuples at a page during consistency checking (See comments for
>> the mask_dead_tuples() function).
>> 2. Still not using physical TID ordering.
>> 3. Index cleanup() after each quick_vacuum_index() call was excluded.
> 
> How does this patch affect opportunistic pruning in particular? Not
> being able to immediately reclaim tuple space in the event of a dead
> hot chain that is marked LP_DEAD could hurt quite a lot, including
> with very common workloads, such as pgbench (pgbench accounts tuples
> are quite a lot wider than a raw item pointer, and opportunistic
> pruning is much more important than vacuuming). Is that going to be
> acceptable, do you think? Have you measured the effects? Can we do
> something about it, like make pruning behave differently when it's
> opportunistic?

This is the most "tasty" part of the work. I plan some experimental 
research on it at the end of patches developing (including TID sort)
and parametrized opportunistic pruning for flexibility of switching 
between strategies on the fly.
My current opinion on this question: we can develop flexible strategy 
based on parameters: free space at a block, frequency of UPDATE/DELETE 
queries, percent of DEAD tuples in a block/relation.
Background cleaner, raised by heap_page_prune(), give an opportunity for 
using different ways for each block or relation.
This technique should be able to configure from fully non-storage DEAD 
tuples+vacuum to all-storage DEAD tuples+target deletion by DB admin.

> 
> Are you aware of the difference between _bt_delitems_delete() and
> _bt_delitems_vacuum(), and the considerations for hot standby? I think
> that that's another TODO list item for this patch.
> 

Ok

-- 
Andrey Lepikhov
Postgres Professional:
https://postgrespro.com
The Russian Postgres Company