pgvector.git / summary / log / commit / refs
commit 67eeade63cb8f9da1e653b429ff4f27d913a8c68
Author: Andrew Kane <andrew@ankane.org>
Date: Tue Jan 09 21:04:18 2024 +0000
Moved HNSW first in readme [skip ci]
README.md | 94 +++++++++++++++++++++++++++++++--------------------------------
1 file changed, 47 insertions(+), 47 deletions(-)
diff --git a/README.md b/README.md
index a813ff5e..e6b857d7 100644
--- a/README.md
+++ b/README.md
@@ -161,56 +161,63 @@ You can add an index to use approximate nearest neighbor search, which trades so
Supported index types are:
-- [IVFFlat](#ivfflat)
- [HNSW](#hnsw) - added in 0.5.0
+- [IVFFlat](#ivfflat)
-## IVFFlat
-
-An IVFFlat index divides vectors into lists, and then searches a subset of those lists that are closest to the query vector. It has faster build times and uses less memory than HNSW, but has lower query performance (in terms of speed-recall tradeoff).
-
-Three keys to achieving good recall are:
+## HNSW
-1. Create the index *after* the table has some data
-2. Choose an appropriate number of lists - a good place to start is `rows / 1000` for up to 1M rows and `sqrt(rows)` for over 1M rows
-3. When querying, specify an appropriate number of [probes](#query-options) (higher is better for recall, lower is better for speed) - a good place to start is `sqrt(lists)`
+An HNSW index creates a multilayer graph. It has slower build times and uses more memory than IVFFlat, but has better query performance (in terms of speed-recall tradeoff). There’s no training step like IVFFlat, so the index can be created without any data in the table.
Add an index for each distance function you want to use.
L2 distance
```sql
-CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops) WITH (lists = 100);
+CREATE INDEX ON items USING hnsw (embedding vector_l2_ops);
```
Inner product
```sql
-CREATE INDEX ON items USING ivfflat (embedding vector_ip_ops) WITH (lists = 100);
+CREATE INDEX ON items USING hnsw (embedding vector_ip_ops);
```
Cosine distance
```sql
-CREATE INDEX ON items USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
+CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops);
```
Vectors with up to 2,000 dimensions can be indexed.
+### Index Options
+
+Specify HNSW parameters
+
+- `m` - the max number of connections per layer (16 by default)
+- `ef_construction` - the size of the dynamic candidate list for constructing the graph (64 by default)
+
+```sql
+CREATE INDEX ON items USING hnsw (embedding vector_l2_ops) WITH (m = 16, ef_construction = 64);
+```
+
+A higher value of `ef_construction` provides better recall at the cost of index build time / insert speed.
+
### Query Options
-Specify the number of probes (1 by default)
+Specify the size of the dynamic candidate list for search (40 by default)
```sql
-SET ivfflat.probes = 10;
+SET hnsw.ef_search = 100;
```
-A higher value provides better recall at the cost of speed, and it can be set to the number of lists for exact nearest neighbor search (at which point the planner won’t use the index)
+A higher value provides better recall at the cost of speed.
Use `SET LOCAL` inside a transaction to set it for a single query
```sql
BEGIN;
-SET LOCAL ivfflat.probes = 10;
+SET LOCAL hnsw.ef_search = 100;
SELECT ...
COMMIT;
```
@@ -220,72 +227,61 @@ COMMIT;
Check [indexing progress](https://www.postgresql.org/docs/current/progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING) with Postgres 12+
```sql
-SELECT phase, round(100.0 * tuples_done / nullif(tuples_total, 0), 1) AS "%" FROM pg_stat_progress_create_index;
+SELECT phase, round(100.0 * blocks_done / nullif(blocks_total, 0), 1) AS "%" FROM pg_stat_progress_create_index;
```
-The phases for IVFFlat are:
+The phases for HNSW are:
1. `initializing`
-2. `performing k-means`
-3. `assigning tuples`
-4. `loading tuples`
+2. `loading tuples`
-Note: `%` is only populated during the `loading tuples` phase
+## IVFFlat
-## HNSW
+An IVFFlat index divides vectors into lists, and then searches a subset of those lists that are closest to the query vector. It has faster build times and uses less memory than HNSW, but has lower query performance (in terms of speed-recall tradeoff).
-An HNSW index creates a multilayer graph. It has slower build times and uses more memory than IVFFlat, but has better query performance (in terms of speed-recall tradeoff). There’s no training step like IVFFlat, so the index can be created without any data in the table.
+Three keys to achieving good recall are:
+
+1. Create the index *after* the table has some data
+2. Choose an appropriate number of lists - a good place to start is `rows / 1000` for up to 1M rows and `sqrt(rows)` for over 1M rows
+3. When querying, specify an appropriate number of [probes](#query-options) (higher is better for recall, lower is better for speed) - a good place to start is `sqrt(lists)`
Add an index for each distance function you want to use.
L2 distance
```sql
-CREATE INDEX ON items USING hnsw (embedding vector_l2_ops);
+CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops) WITH (lists = 100);
```
Inner product
```sql
-CREATE INDEX ON items USING hnsw (embedding vector_ip_ops);
+CREATE INDEX ON items USING ivfflat (embedding vector_ip_ops) WITH (lists = 100);
```
Cosine distance
```sql
-CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops);
+CREATE INDEX ON items USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);
```
Vectors with up to 2,000 dimensions can be indexed.
-### Index Options
-
-Specify HNSW parameters
-
-- `m` - the max number of connections per layer (16 by default)
-- `ef_construction` - the size of the dynamic candidate list for constructing the graph (64 by default)
-
-```sql
-CREATE INDEX ON items USING hnsw (embedding vector_l2_ops) WITH (m = 16, ef_construction = 64);
-```
-
-A higher value of `ef_construction` provides better recall at the cost of index build time / insert speed.
-
### Query Options
-Specify the size of the dynamic candidate list for search (40 by default)
+Specify the number of probes (1 by default)
```sql
-SET hnsw.ef_search = 100;
+SET ivfflat.probes = 10;
```
-A higher value provides better recall at the cost of speed.
+A higher value provides better recall at the cost of speed, and it can be set to the number of lists for exact nearest neighbor search (at which point the planner won’t use the index)
Use `SET LOCAL` inside a transaction to set it for a single query
```sql
BEGIN;
-SET LOCAL hnsw.ef_search = 100;
+SET LOCAL ivfflat.probes = 10;
SELECT ...
COMMIT;
```
@@ -295,13 +291,17 @@ COMMIT;
Check [indexing progress](https://www.postgresql.org/docs/current/progress-reporting.html#CREATE-INDEX-PROGRESS-REPORTING) with Postgres 12+
```sql
-SELECT phase, round(100.0 * blocks_done / nullif(blocks_total, 0), 1) AS "%" FROM pg_stat_progress_create_index;
+SELECT phase, round(100.0 * tuples_done / nullif(tuples_total, 0), 1) AS "%" FROM pg_stat_progress_create_index;
```
-The phases for HNSW are:
+The phases for IVFFlat are:
1. `initializing`
-2. `loading tuples`
+2. `performing k-means`
+3. `assigning tuples`
+4. `loading tuples`
+
+Note: `%` is only populated during the `loading tuples` phase
## Filtering
[parent: 108fb09d7b32]