From: Andrew Lazarus <andrew@pillette.com>
To: Mark Kirkwood <markir@paradise.net.nz>
Cc: pgsql-performance@postgresql.org
Subject: Re: index structure for 114-dimension vector
Date: Fri, 20 Apr 2007 17:28:20 -0700
Message-ID: <3810566934.20070420172820@pillette.com> (raw)
In-Reply-To: <46294FEB.9010909@paradise.net.nz>
References: <b42b73150704170917n33ebcdc8wc852a3fa02fd8bb5@mail.gmail.com>
<01db01c781e4$c700bf30$2e00a8c0@tridecap.com>
<b42b73150704181716wb1600aey8761b6793d3fd702@mail.gmail.com>
<1217328326.20070420120729@pillette.com>
<1177105284.28383.31.camel@dogma.v10.wvs>
<46294FEB.9010909@paradise.net.nz>
Because I know the 25 closest are going to be fairly close in each
coordinate, I did try a multicolumn index on the last 6 columns and
used a +/- 0.1 or 0.2 tolerance on each. (The 25 best are very probably inside
that hypercube on the distribution of data in question.)
This hypercube tended to have 10-20K records, and took at least 4
seconds to retrieve. I was a little surprised by how long that took.
So I'm wondering if my data representation is off the wall.
I should mention I also tried a cube index using gist on all 114
elements, but CREATE INDEX hadn't finished in 36 hours, when I killed
it, and I wasn't in retrospect sure an index that took something like
6GB by itself would be helpful on a 2GB of RAM box.
MK> I don't think that will work for the vector norm i.e:
MK> |x - y| = sqrt(sum over j ((x[j] - y[j])^2))
MK> Cheers
MK> Mark
--
Sincerely,
Andrew Lazarus mailto:andrew@pillette.com
Attachments:vCard.VCF (text/x-vcard, 377B) download
BEGIN:VCARD
VERSION:2.1
N:Lazarus;Andrew;;;Ph.D.
FN:Andrew Lazarus, Ph.D.
EMAIL;PREF;INTERNET:andrew@pillette.com
TITLE:Director of R&D
ADR;WORK:;800-366-0688;3028 Fillmore Street;San Francisco;CA;94123;USA
LABEL;WORK;ENCODING=QUOTED-PRINTABLE:800-366-0688=0D=0A3028 Fillmore S=
treet=0D=0ASan Francisco=0D=0ACA=0D=0A94123=0D=0AUSA
X-GENDER:Male
REV:18991230T080000Z
END:VCARD
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: andrew@pillette.com, markir@paradise.net.nz
Subject: Re: index structure for 114-dimension vector
In-Reply-To: <3810566934.20070420172820@pillette.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox