pg.ddx.io  pgvector/pgvector GitHub issues and pull requests  
help / color / mirror / Atom feed
From: VladlenPopolitov (@VladlenPopolitov) <noreply+VladlenPopolitov@github.com>
To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>
Subject: Re: [pgvector/pgvector] issue #761: Enhancement: multicolumn vector index design
Date: Wed, 29 Jan 2025 07:04:08 +0000
Message-ID: <gh-pgvector-pgvector-761-c2620858209@github.com> (raw)
In-Reply-To: <gh-pgvector-pgvector-761@github.com>
References: <gh-pgvector-pgvector-761@github.com>

@VoVAllen @ankane  @jkatz  Hi!
First of all - these benchmarks compare different algorithms with WHERE clause in SELECT, and these algorithms have big differences in algorithms themselves, not in software realisation or machine performance.
 Search with filter depends on dataset and filters properties:
- selectivity - the share of vectors that have filter condition 'true'. pgvector without relaxed option decreases recall, pgvector with relaxed decreases rate, multicolumn index increases rate (and slightly recall)
- correlation - how filtered vectors are combined in clusters in own vector space - it can increase or decrease search rate depending on algorithm.
Multicolumn index has big advantage for filtered search, but without filters it has additional  overhead.
1) Example for selectivity 10 values (10%) with condition 'WHERE field=value'.

![Image](https://github.com/user-attachments/assets/6a7dbebe-0096-4a06-a41e-04754a0061bd)
you can see, that mc_hnsw has higher rate (due to less vectors to analyze), but on low selectivity it comparable with pg_vector with relaxed algorithm.

2) Example for selectivity 1000 values (0.1%) with condition 'WHERE field=value'.

![Image](https://github.com/user-attachments/assets/01603814-80d2-4267-b023-e8f2a6ada698)
pgvector without relaxed - decreased recall, pgvector relaxed - keeps recall with decrease of rate, mc_hnsw - increased rate and recall and becomes the dot in top-right corner of the graph.

It is no question of software quality, it is algorithm property. 
It is why I propose to include multicolumn version of the index into pgvector. We could fast run the path, that everyone alone walked for year.
 

view thread (18+ messages)  latest in thread

Message-ID: <gh-pgvector-pgvector-761-c2620858209@github.com>
Permalink:  ../gh-pgvector-pgvector-761-c2620858209@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox