X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by postgresql.org (Postfix) with ESMTP id AA1F39DC801
	for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
	Sat, 21 Jan 2006 12:25:44 -0400 (AST)
Received: from postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 92499-10
	for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
	Sat, 21 Jan 2006 12:25:40 -0400 (AST)
X-Greylist: from auto-whitelisted by SQLgrey-
Received: from svana.org (unknown [125.62.94.225])
	by postgresql.org (Postfix) with ESMTP id 48BE29DC819
	for <pgsql-performance@postgresql.org>;
	Sat, 21 Jan 2006 12:25:39 -0400 (AST)
Received: from kleptog by svana.org with local (Exim 4.50)
	id 1F0KHw-0002st-PV; Sun, 22 Jan 2006 02:04:24 +1100
Date: Sat, 21 Jan 2006 16:04:24 +0100
From: Martijn van Oosterhout <kleptog@svana.org>
To: Oleg Bartunov <oleg@sai.msu.su>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Ron <rjpeace@earthlink.net>,
	pgsql-performance@postgresql.org
Subject: Re: [GENERAL] Creation of tsearch2 index is very slow
Message-ID: <20060121150424.GB9955@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <26153.1137784469@sss.pgh.pa.us>
 <20060120201036.GE31908@svana.org>
	<26583.1137788505@sss.pgh.pa.us> <20060120205132.GF31908@svana.org>
	<26958.1137791955@sss.pgh.pa.us> <20060120213754.GG31908@svana.org>
	<7.0.1.0.2.20060120174141.039b03e8@earthlink.net>
	<6111.1137797436@sss.pgh.pa.us> <20060121130830.GA9955@svana.org>
	<Pine.GSO.4.63.0601211619541.14417@ra.sai.msu.su>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="9zSXsLTf0vkW971A"
Content-Disposition: inline
In-Reply-To: <Pine.GSO.4.63.0601211619541.14417@ra.sai.msu.su>
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
User-Agent: Mutt/1.5.9i
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: kleptog@svana.org
X-SA-Exim-Scanned: No (on svana.org); SAEximRunCond expanded to false
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, score=0.129 required=5 tests=[AWL=0.129]
X-Spam-Score: 0.129
X-Spam-Level: 
X-Archive-Number: 200601/391
X-Sequence-Number: 16869


--9zSXsLTf0vkW971A
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sat, Jan 21, 2006 at 04:29:13PM +0300, Oleg Bartunov wrote:
> Martijn, you're right! We want not only to split page to very
> different parts, but not to increase the number of sets bits in
> resulted signatures, which are union (OR'ed) of all signatures=20
> in part. We need not only fast index creation (thanks, Tom !),
> but a better index. Some information is available here
> http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_internals
> There are should be more detailed document, but I don't remember where:)

I see how it works, what I don't quite get is whether the "inverted
index" you refer to is what we're working with here, or just what's in
tsearchd?

> >That's harder though (this algorithm does approximate it sort of)
> >and I havn't come up with an algorithm yet
>=20
> Don't ask how hard we thought :)

Well, looking at how other people are struggling with it, it's
definitly a Hard Problem. One thing though, I don't think the picksplit
algorithm as is really requires you to strictly have the longest
distance, just something reasonably long. So I think the alternate
algorithm I posted should produce equivalent results. No idea how to
test it though...

Have a nice day,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--9zSXsLTf0vkW971A
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFD0k14IB7bNG8LQkwRAqi9AJ4sYMjyepJIRWRypeD02wQJjpWCmwCfaFz6
+3szZsgPCCayrEUyYjAumic=
=RjgF
-----END PGP SIGNATURE-----

--9zSXsLTf0vkW971A--