Message-ID: <4E43AEEA.2040105@enterprisedb.com>
Date: Thu, 11 Aug 2011 13:28:58 +0300
From: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fi-FI;
	rv:1.9.2.18) Gecko/20110626 Icedove/3.1.11
MIME-Version: 1.0
To: Alexander Korotkov <aekorotkov@gmail.com>
CC: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: WIP: Fast GiST index build
References: <BANLkTikgOQ1+r=A9b08szNTeEWsk-cNMMg@mail.gmail.com>
	<CAPpHfdvoOh78ycX3eRePTS29635pHCSTfLdFDzHZhxTKsggCuQ@mail.gmail.com>
	<4E2DBB60.80604@enterprisedb.com>
	<4E2EF83A.8000304@enterprisedb.com>
	<CAPpHfdso-oOo4+3r9yBn=PATz3BC4z4cxK9Ew8tFRT1MkVVoiA@mail.gmail.com>
	<4E31D03D.70206@enterprisedb.com>
	<CAPpHfdtmgwOjNr7J15eCe6sXrtmq_c_52oNh-=D934SKekoGxg@mail.gmail.com>
	<CAPpHfdv=HQDmUGTARKjH+AiHMDpXa+8+WwuqjUSfZnVdC-LJbQ@mail.gmail.com>
	<CAPpHfduAhSNo8LHMPj_pqQ_2W6ZDPzdQBPzHbnpqaKz6kf4nrw@mail.gmail.com>
	<4E3977D8.1020506@enterprisedb.com>
	<CAPpHfducs8Ussw+t8iP3WS3ZTuX2J-77FOY=jAZUg-K6AuOC0g@mail.gmail.com>
	<CAPpHfdusd_b_dtvNY2uuJict6=VEUTGAS8ELXGck=o=kOdMnOw@mail.gmail.com>
	<4E3FAB1D.1060105@enterprisedb.com>
	<CAPpHfdsovwbgs30TojJc_zDsc+0ghapm+-AALfP58UaHiYjr2A@mail.gmail.com>
	<CAPpHfdvi5WZKdRD+s60R+pdm8dLczRQ5_RYt4L8OA_bhorPjpg@mail.gmail.com>
	<CAPpHfdunxCQ-FU=eNHRKqgmw5wbXHOdj+28X_WqsZxU9r2X21w@mail.gmail.com>
In-Reply-To: 
 <CAPpHfdunxCQ-FU=eNHRKqgmw5wbXHOdj+28X_WqsZxU9r2X21w@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 10.08.2011 22:44, Alexander Korotkov wrote:
> Manual and readme updates.

Thanks, I'm reviewing these now.

Do we want to expose the level-step and buffersize parameters to users? 
They've been useful during testing, but I'm thinking we should be able 
to guess good enough values for them automatically, and just remove the 
options. It's pretty much impossible for a user to tune them correctly, 
it would require deep knowledge of the buffering algorithm.

I'm thinking that even when you explicitly turn buffering on, we should 
still process the first 10000 or so tuples with simple inserts. That way 
we always have a sample of tuples to calculate the average tuple size 
from. It's plausible that if the input data is ordered, looking at the 
first N tuples will give skewed sample, but I don't think there's much 
danger of that in practice. Even if the data is ordered, the length of 
GiST tuples shouldn't vary much.

What happens if we get the levelstep and pagesPerBuffer estimates wrong? 
How sensitive is the algorithm to that? Or will we run out of memory? 
Would it be feasible to adjust those in the middle of the index build, 
if we e.g exceed the estimated memory usage greatly?

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com