pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>
To: Alexander Korotkov <aekorotkov@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: WIP: Fast GiST index build
Date: Wed, 13 Jul 2011 16:59:39 +0300
Message-ID: <4E1DA4CB.9020209@enterprisedb.com> (raw)
In-Reply-To: <CAPpHfdvcAo2L2RVtnr+BtPE18Q4mJLaLf-R7SrKMTUousv8ChA@mail.gmail.com>
References: <BANLkTikgOQ1+r=A9b08szNTeEWsk-cNMMg@mail.gmail.com>
	<BANLkTinMB-GSjFDThc0o9hRWa+_dJJXPZg@mail.gmail.com>
	<4DF8676F.9040801@enterprisedb.com>
	<BANLkTimr___BFy+98t36tkbUvvavUzjV9A@mail.gmail.com>
	<BANLkTi=x0Ck1z_NbkqeqvuKXs9kHOR+5WQ@mail.gmail.com>
	<BANLkTikP6RAYmUriquQEdKjEBviTH2vWRQ@mail.gmail.com>
	<4DFA4AE5.1010806@enterprisedb.com>
	<BANLkTinN6C8q61Fb8ASZVOPbYq7VoqoHOw@mail.gmail.com>
	<BANLkTin=gyX5PJW-JaN59uqRBSgdDHVNNA@mail.gmail.com>
	<BANLkTikSbaun6Sf8hJSkO0D4nu7VC2O35A@mail.gmail.com>
	<4E044D68.5070605@enterprisedb.com>
	<BANLkTi=umLx8LBCSmALKpoGTFguqQxQ61w@mail.gmail.com>
	<BANLkTinK-ZL-yQCNtKZLfGS=8VGmO13b8g@mail.gmail.com>
	<4E0894FF.8080202@enterprisedb.com>
	<BANLkTi=1TL+pH2sxiR7nf3oMFdWAHHgSyA@mail.gmail.com>
	<CAPpHfdt6ER8LKEEmnVnbYuaODKpDb4S5UuGRa5prio--Kp=LFw@mail.gmail.com>
	<26338.1310134685@sss.pgh.pa.us>
	<CAPpHfduE1LM7aCk7MZ_u_n2z58Jxps=TBD0eX4WX4gwBrkk=Yw@mail.gmail.com>
	<CAPpHfdvcAo2L2RVtnr+BtPE18Q4mJLaLf-R7SrKMTUousv8ChA@mail.gmail.com>

On 12.07.2011 11:34, Alexander Korotkov wrote:
> New version of patch with a little more refactoring and comments.

Great! The README helps tremendously to understand this, thanks for that.

One thing that caught my eye is that when you empty a buffer, you load 
the entire subtree below that buffer, down to the next buffered or leaf 
level, into memory. Every page in that subtree is kept pinned. That is a 
problem; in the general case, the buffer manager can only hold a modest 
number of pages pinned at a time. Consider that the minimum value for 
shared_buffers is just 16. That's unrealistically low for any real 
system, but the default is only 32MB, which equals to just 4096 buffers. 
A subtree could easily be larger than that.

I don't think you're benefiting at all from the buffering that BufFile 
does for you, since you're reading/writing a full block at a time 
anyway. You might as well use the file API in fd.c directly, ie. 
OpenTemporaryFile/FileRead/FileWrite.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com



view thread (114+ messages)  latest in thread

Message-ID: <4E1DA4CB.9020209@enterprisedb.com>
Permalink:  ../4E1DA4CB.9020209@enterprisedb.com/
Also on:    postgresql.org/message-id/4E1DA4CB.9020209@enterprisedb.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: heikki.linnakangas@enterprisedb.com, aekorotkov@gmail.com
  Subject: Re: WIP: Fast GiST index build
  In-Reply-To: <4E1DA4CB.9020209@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox