pg.ddx.io  pgvector/pgvector GitHub issues and pull requests  
help / color / mirror / Atom feed
[pgvector/pgvector] PR #1053: Serve loaded indexes to many PostgreSQL hosts over the network
2+ messages / 1 participants
[nested] [flat]

* [pgvector/pgvector] PR #1053: Serve loaded indexes to many PostgreSQL hosts over the network
@ 2026-10-31 14:05  "chenqin (@chenqin)" <noreply+chenqin@github.com>
  0 siblings, 0 replies; 2+ messages in thread

From: chenqin (@chenqin) @ 2026-10-31 14:05 UTC (permalink / raw)
  To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>

## Summary

PostgreSQL hosts evaluate SQL filters on their own CPUs and send only vector search to shared GPU nodes, so hybrid search capacity is no longer limited by the GPU host's CPUs.                                                                    - **Networked GPU nodes.** The worker listens on TCP (`--listen-tcpHOST:PORT --auth-token-file FILE`). Clients must authenticate with a shared token before any other frame, and shared-memory operations are refused over TCP.
- **PostgreSQL client in C** (`src/annremote.c`). `ann.worker_address` lists the GPU nodes. Each backend picks a node, fails over to the next one, and loads go to every node. `ann_index_attach(name)` lets other hosts use an index that one host loaded; the `ann_index_catalog` table records each index's source table and columns.                                                   - **On-demand routing.** `ann_hybrid_search(name, query, k,       filter_sql)` counts the filter's rows with a capped scan. Filters    under `ann.gpu_min_eligible` (default 1000) run exactly on the hoslarger ones stream their keys to a GPU node as delta varints (1–2 bytes per key). `ann_index_search` routes small key filters the same way.                                                              - **Heartbeats and failover.** Idle connections are probed before    reuse (`ann.worker_heartbeat_ms`), TCP keepalives detect dead peeron both sides, failed nodes are skipped for `ann.worker_retry_ms`,   and requests retry on the next node. Unauthenticated peers have 10to authenticate.                                                     - **Also included:** an optional exact GPU brute-force route for  selective filters (`--filtered-bruteforce`, off by default), and     `key`/`vector` columns in index manifests.                                                                                             
Docs: `docs/ann-networked.md`, with links from `docs/README.md` and `ann/README.md`.

## Measured scaling

One DGX B200 partitioned into a 96-core GPU node and two 64-core CPU
hosts. 10M × 128 vectors, recall@10 = 1.0 on every workload:

| Workload | 1 CPU host | 2 CPU hosts | Bottleneck |
| --- | --- | --- | --- |
| ANN | 295,787 QPS | 391,781 | GPU node |
| 1% filter | 1,203 | 1,298 | GPU node |
| 0.1% filter, GPU route | 13,895 | 14,320 | GPU node CPUs |
| 0.1% filter, local route | 3,345 | 5,605 | CPU hosts (scales) |

Local scoring scales with CPU hosts but costs about 4× more per query
than the GPU node's exact scan, hence the default threshold of 1000
rows. The filtered brute-force route was slower under load (about 800 vs 1,200 QPS), so it is off by default.

## Not included

AWS EFA / RDMA transport. The test host is a DGX B200 with ConnectX-7 InfiniBand, no EFA, and no libfabric installed. Building libfabric from source needs approval, and its IPoIB interfaces are down. The intended design is in `docs/ann-networked.md`.

## Testing

- `test/ann/test_remote.py` (new, 13 tests): token auth and disconnects, refusal of shared memory over TCP, packed-filter decoding including malformed gaps, ANN over TCP, both routes of `ann_hybrid_search` and of key filters equal to exact SQL, read-only filter enforcement, attach and catalog, a wrong token, failover with heartbeats when a node dies while idle, and every node down.
- On the GPU host: `make installcheck` (16 tests), plus the worker, batching, server, index, remote, pipeline, filtered, native, postgres, cuvs, cuvs_gathered and cuvs_index suites, all passing.
- C code formatted with PostgreSQL's `pgindent` (clean re-check); `git diff --check` clean.

The PR will contain all four rewritten commits, not just the networked one. That's because rewriting the three master commits changed their IDs, so until master is force-pushed too, the branch no longer sits on top of the remote master. Once both force pushes are done, the PR shows only the networked-mode commit.

inline patch 1: .gitignore, Makefile, Makefile.win and 67 more (+13785/-8 lines) download inline-1.patchdiff --git a/.gitignore b/.gitignore index 9d51a94f6..685a5a52d 100644 --- a/.gitignore +++ b/.gitignore @@ -12,3 +12,14 @@ regression.* *.obj *.lib *.exp +/ann/ann-worker +/ann/ann-worker.dSYM/ +/ann/ann-worker-cuvs +/ann/ann-worker-cuvs.dSYM/ +__pycache__/ +/ann/libannbatch.a +/ann/ann-batch-tests +/ann/ann-batch-tests.dSYM/ +/ann/ann-batch-bench +/ann/ann-batch-bench.dSYM/ +cufile.log diff --git a/Makefile b/Makefile index 207a19b2d..a709e5a12 100644 --- a/Makefile +++ b/Makefile @@ -1,10 +1,10 @@ EXTENSION = vector -EXTVERSION = 0.8.7 +EXTVERSION = 0.9.0 MODULE_big = vector DATA = $(wildcard sql/*--*--*.sql) DATA_built = sql/$(EXTENSION)--$(EXTVERSION).sql -OBJS = src/bitutils.o src/bitvec.o src/halfutils.o src/halfvec.o src/hnsw.o src/hnswbuild.o src/hnswinsert.o src/hnswscan.o src/hnswutils.o src/hnswvacuum.o src/ivfbuild.o src/ivfflat.o src/ivfinsert.o src/ivfkmeans.o src/ivfscan.o src/ivfutils.o src/ivfvacuum.o src/sparsevec.o src/vector.o +OBJS = src/ann.o src/anncustom.o src/annindex.o src/annremote.o src/annplan.o src/annbuild.o src/anninsert.o src/annscan.o src/annutils.o src/annvacuum.o src/annworker.o src/bitutils.o src/bitvec.o src/halfutils.o src/halfvec.o src/hnsw.o src/hnswbuild.o src/hnswinsert.o src/hnswscan.o src/hnswutils.o src/hnswvacuum.o src/ivfbuild.o src/ivfflat.o src/ivfinsert.o src/ivfkmeans.o src/ivfscan.o src/ivfutils.o src/ivfvacuum.o src/sparsevec.o src/vector.o HEADERS = src/halfvec.h src/sparsevec.h src/vector.h TESTS = $(wildcard test/sql/*.sql) diff --git a/Makefile.win b/Makefile.win index 597108877..9aaf4b5c5 100644 --- a/Makefile.win +++ b/Makefile.win @@ -1,11 +1,11 @@ EXTENSION = vector -EXTVERSION = 0.8.7 +EXTVERSION = 0.9.0 DATA_built = sql\$(EXTENSION)--$(EXTVERSION).sql -OBJS = src\bitutils.obj src\bitvec.obj src\halfutils.obj src\halfvec.obj src\hnsw.obj src\hnswbuild.obj src\hnswinsert.obj src\hnswscan.obj src\hnswutils.obj src\hnswvacuum.obj src\ivfbuild.obj src\ivfflat.obj src\ivfinsert.obj src\ivfkmeans.obj src\ivfscan.obj src\ivfutils.obj src\ivfvacuum.obj src\sparsevec.obj src\vector.obj +OBJS = src\ann.obj src\anncustom.obj src\annindex.obj src\annremote.obj src\annplan.obj src\annbuild.obj src\anninsert.obj src\annscan.obj src\annutils.obj src\annvacuum.obj src\annworker.obj src\bitutils.obj src\bitvec.obj src\halfutils.obj src\halfvec.obj src\hnsw.obj src\hnswbuild.obj src\hnswinsert.obj src\hnswscan.obj src\hnswutils.obj src\hnswvacuum.obj src\ivfbuild.obj src\ivfflat.obj src\ivfinsert.obj src\ivfkmeans.obj src\ivfscan.obj src\ivfutils.obj src\ivfvacuum.obj src\sparsevec.obj src\vector.obj HEADERS = src\halfvec.h src\sparsevec.h src\vector.h -REGRESS = bit btree cast copy halfvec hnsw_bit hnsw_halfvec hnsw_sparsevec hnsw_vector ivfflat_bit ivfflat_halfvec ivfflat_vector sparsevec vector_type +REGRESS = ann_filtered ann_vector bit btree cast copy halfvec hnsw_bit hnsw_halfvec hnsw_sparsevec hnsw_vector ivfflat_bit ivfflat_halfvec ivfflat_vector sparsevec vector_type REGRESS_OPTS = --inputdir=test --load-extension=$(EXTENSION) # For /arch flags diff --git a/README.md b/README.md index f5fde78f3..2fe637ab0 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,8 @@ Plus [ACID](https://en.wikipedia.org/wiki/ACID) compliance, point-in-time recove Have a lot of vectors? Use [quantization](#scaling) to scale +This fork also includes [experimental accelerator search](#experimental-accelerator-search), with a CPU reference worker, frozen-table Postgres reference tests, and optional cuVS adapter source. + [![Build Status](https://github.com/pgvector/pgvector/actions/workflows/build.yml/badge.svg)](https://github.com/pgvector/pgvector/actions) ## Installation @@ -547,6 +549,60 @@ SET ivfflat.max_probes = 100; Note: If this is lower than `ivfflat.probes`, `ivfflat.probes` will be used +## Experimental Accelerator Search + +This fork is developing filtered vector search across accelerator backends. The standalone C runtime implements exact CPU search and bounded batching/admission. Test fixtures combine Postgres full-text/metadata predicates with that worker. The native C `ann` access method now provides an exact L2 storage/lifecycle baseline. An opt-in exact CustomScan scores vectors after PostgreSQL predicates; cuVS CAGRA remains a deferred draft. + +| Component | Status | +| --- | --- | +| CPU reference worker | Implemented and tested: L2, negative inner product, cosine, shared/per-query filters and canonical import/export. Wire timeouts currently belong to the test client; C external-client deadlines remain pending. | +| CPU ready-query batching | Implemented and tested: native C batching, request/byte/work limits, deadlines, cancellation, and overload reporting. See the [scheduler guide](ann/README.md#c-batching-and-admission-api). | +| Native PostgreSQL index | Experimental C exact L2 `ann` AM: bulk build, insert, ordered scans, WAL, MVCC heap checks, vacuum/page reuse, and reindex. The ordinary AM filters in the executor; an opt-in CustomScan scores only eligible vectors. Worker integration is pending. | +| Frozen-table PostgreSQL test reference | Python test fixtures validate SQL eligibility and freeze/failure behavior; test tooling only. | +| NVIDIA cuVS | Deferred TODO: CAGRA integration, cuVS SDK compilation, and GPU validation. Unvalidated adapter draft retained. | +| AMD hipVS | Planned; vendor capabilities require independent validation. | +| Fused NVFP4 | Planned; a concrete kernel API and precision/filtering contract are still needed. | + +### Try the CPU worker + +From the root of a checkout of this fork, with a C compiler, Make, and Python 3.9+: + +```sh +make -C ann +make -C ann check +python3 benchmarks/ann/search.py --mode shared --batch 16 +python3 benchmarks/ann/search.py --mode unique --batch 16 +``` + +This build is independent of PGXS and does not require Postgres or a GPU SDK. The benchmark uses precomputed synthetic eligibility and reports worker batch requests/sec and vector searches/sec separately. Its results are not SQL or GPU throughput. Tests requiring a database or GPU skip unless explicitly enabled. + +### Native C runtime and PostgreSQL reference tests + +Worker, backend, and batching/admission runtime code is C, following pgvector's implementation language. See [the native C API](ann/batching.h) and [worker guide](ann/README.md) for build, ownership, and cancellation details. Python is restricted to test helpers and benchmark scripts. + +The frozen-table Python adapter is a test reference only. Run its SQL eligibility and freeze/failure tests against a test database with pgvector installed: + +```sh +python3 -m pip install -r test/ann/requirements-postgres.txt +ANN_PG_DSN='dbname=your_test_database' make -C ann check-postgres +``` + +See the [native index guide](docs/ann-native-index.md) for SQL, architecture diagrams, source layout, and tests. The [native filtered CustomScan](docs/ann-filtered-search.md) is opt-in (`LOAD 'vector'; SET ann.enable_filtered_search = on;`). The production C protocol client remains a todo. There is no supported Python runtime search API in this fork. + +### Validation and next steps + +The current implementation has passed 12 worker tests, 7 native scheduler scenarios, 26 live PostgreSQL tests, worker/scheduler sanitizer checks, all 16 SQL regression tests, and 10 native WAL/recovery assertions on a temporary Postgres 17.6 installation. GPU tests have not run on this development host. + +To run live integration tests against a test database with pgvector installed: + +```sh +ANN_PG_DSN='dbname=your_test_database' make -C ann check-postgres +``` + +These tests create/remove uniquely named schemas and need the corresponding database permissions. GPU validation uses the separate `check-cuvs` target described in the worker guide; explicit GPU tests fail when required dependencies or hardware are missing. + +CPU ready-query batching/admission is implemented. The exact native AM supplies initial WAL/MVCC/vacuum coverage. Selective exact CustomScan execution is implemented. The pipelined worker route, with shared-memory transport and process-wide admission across backends, is implemented with a mocked GPU backend ([guide](docs/ann-worker-pipeline.md)). Immutable sharded CAGRA indexes, built offline and loaded across all GPUs, serve ANN and hybrid queries through `ann_index_search` ([guide](docs/ann-cuvs-index.md)). Full throughput validation on accelerator hardware remains pending. CAGRA integration, cuVS SDK compilation/GPU validation, and multi-GPU tuning remain deferred todos. The [implementation plan](docs/cuvs-filtered-ann-plan.md) tracks the phased todos and vendor boundary; the [performance analysis](docs/cuvs-search-performance.md) covers eligibility reuse, exact-versus-ANN routing, and throughput experiments. + ## Half-Precision Vectors Use the `halfvec` type to store half-precision vectors diff --git a/ann/Makefile b/ann/Makefile new file mode 100644 index 000000000..3a282592e --- /dev/null +++ b/ann/Makefile @@ -0,0 +1,81 @@ +# Standalone reference worker; independent of PGXS and GPU SDKs. +CC ?= cc +CFLAGS ?= -O2 -g +CFLAGS += -std=c11 -Wall -Wextra -Werror -pedantic +CUDA_HOME ?= /usr/local/cuda + +# USE_CUVS=1 also builds the cuVS worker (requires CUVS_PREFIX); hosts +# without a GPU SDK keep the default CPU-only build. +USE_CUVS ?= 0 + +ifeq ($(USE_CUVS),1) +all: ann-worker libannbatch.a ann-worker-cuvs +else +all: ann-worker libannbatch.a +endif + +batching.o: batching.c batching.h backend.h + $(CC) $(CFLAGS) -pthread -c batching.c -o $@ + +cpu.o: cpu.c backend.h indexfile.h + $(CC) $(CFLAGS) -c cpu.c -o $@ + +indexfile.o: indexfile.c indexfile.h backend.h + $(CC) $(CFLAGS) -pthread -c indexfile.c -o $@ + +libannbatch.a: batching.o cpu.o indexfile.o + $(AR) rcs $@ batching.o cpu.o indexfile.o + +ann-batch-tests: ../test/ann/batch_test.c batching.c cpu.c indexfile.c batching.h backend.h + $(CC) $(CFLAGS) -pthread -I. ../test/ann/batch_test.c batching.c cpu.c indexfile.c -lm -o $@ + +ann-batch-bench: ../benchmarks/ann/batch.c batching.c cpu.c indexfile.c batching.h backend.h + $(CC) $(CFLAGS) -pthread -I. ../benchmarks/ann/batch.c batching.c cpu.c indexfile.c -lm -o $@ + +WORKER_SRCS = worker.c cpu.c mockgpu.c batching.c server.c indexfile.c +WORKER_DEPS = $(WORKER_SRCS) batching.h backend.h server.h indexfile.h + +ann-worker: $(WORKER_DEPS) + $(CC) $(CFLAGS) -pthread $(WORKER_SRCS) -lm -o $@ + +# Requires the cuVS 26.10 C SDK (libcuvs_c), DLPack, and CUDA runtime headers. +# A different executable keeps the dependency-free CPU build reproducible. +ann-worker-cuvs: $(WORKER_DEPS) cuvs.c + @test -n "$(CUVS_PREFIX)" || { echo 'Set CUVS_PREFIX to the cuVS 26.10 installation' >&2; exit 1; } + $(CC) $(CFLAGS) -pthread -DANN_USE_CUVS -I"$(CUVS_PREFIX)/include" -I"$(CUDA_HOME)/include" $(CPPFLAGS) $(WORKER_SRCS) cuvs.c -L"$(CUVS_PREFIX)/lib" -L"$(CUVS_PREFIX)/lib64" -L"$(CUDA_HOME)/lib64" $(LDFLAGS) -lcuvs_c -lcudart -lm -o $@ + +check: ann-worker ann-batch-tests + python3 ../test/ann/test_worker.py + python3 ../test/ann/test_batching.py + python3 ../test/ann/test_server.py + python3 ../test/ann/test_index.py + python3 ../test/ann/test_cuvs.py + python3 ../test/ann/test_postgres.py + python3 ../test/ann/test_native.py + python3 ../test/ann/test_filtered.py + python3 ../test/ann/test_pipeline.py + +check-cuvs: ann-worker ann-worker-cuvs + ANN_CUVS_TEST=1 python3 ../test/ann/test_cuvs.py + ANN_CUVS_TEST=1 python3 ../test/ann/test_cuvs_gathered.py + ANN_CUVS_TEST=1 python3 ../test/ann/test_cuvs_index.py + +check-postgres: ann-worker + @test -n "$$ANN_PG_DSN" || { echo 'Set ANN_PG_DSN to a test database with pgvector installed' >&2; exit 1; } + python3 ../test/ann/test_postgres.py + python3 ../test/ann/test_native.py + python3 ../test/ann/test_filtered.py + python3 ../test/ann/test_pipeline.py + python3 ../test/ann/test_index.py + +clean: + rm -f ann-worker + rm -f batching.o cpu.o indexfile.o libannbatch.a ann-batch-tests + rm -f ann-batch-bench + rm -rf ann-batch-bench.dSYM + rm -rf ann-batch-tests.dSYM + rm -f ann-worker-cuvs + rm -rf ann-worker.dSYM + rm -rf ann-worker-cuvs.dSYM + +.PHONY: all check check-cuvs check-postgres clean diff --git a/ann/README.md b/ann/README.md new file mode 100644 index 000000000..86405cd92 --- /dev/null +++ b/ann/README.md @@ -0,0 +1,123 @@ +# Native C search worker and scheduler + +Runtime code in this directory is C. Python is used only by test helpers and benchmark launchers under `test/ann/` and `benchmarks/ann/`. The [native C PostgreSQL AM](../docs/ann-native-index.md) provides an exact L2 baseline; an [opt-in exact CustomScan](../docs/ann-filtered-search.md) scores eligible vectors, optionally [pipelined to this worker](../docs/ann-worker-pipeline.md), and [loaded cuVS indexes](../docs/ann-cuvs-index.md) serve sharded multi-GPU ANN and hybrid search from SQL; the Python frozen-table implementation is retained solely as a test reference, not an application API. + +## Build and test + +A C11 compiler, POSIX threads, and Make are required for the standalone worker on Linux/macOS. PostgreSQL's main build is independent of this experimental worker. Windows is not a supported worker platform. + +```sh +make -C ann +make -C ann check +# GPU hosts only: also build ann-worker-cuvs (pip SDK paths: see docs/ann-cuvs-index.md) +make -C ann USE_CUVS=1 CUVS_PREFIX=/path/to/libcuvs CUDA_HOME=/path/to/cuda +make -C ann check-cuvs CUVS_PREFIX=/path/to/libcuvs +``` + +`USE_CUVS` defaults to 0, so hosts without a GPU SDK build and test every CPU path. Loaded indexes are served by both builds. The CPU worker scans them exactly, which makes it the test reference; the cuVS worker searches CAGRA shards on all GPUs ([guide](../docs/ann-cuvs-index.md)). + +`make` builds the C executable and `libannbatch.a`, without Python or a GPU SDK. `make check` uses Python to drive C protocol and scheduler tests. Live PostgreSQL and GPU tests are explicitly gated. + +## C batching and admission API + +[batching.h](batching.h) exposes the native API; [batching.c](batching.c) implements scheduling, resource accounting, deadlines, cancellation, and request/result ownership. There is no Python scheduler or runtime client dependency. + +```c +#include "batching.h" + +int main(void) +{ + float vectors[] = {0, 1, 1, 1, 2, 1}; + float query[] = {1, 1}; + AnnBatchOptions options; + AnnBatcher *batcher; + AnnRequest *request; + uint32_t count; + + AnnBatchDefaultOptions(&options); + if (AnnBatchCreate(AnnCpuBackend(), 1, 3, 2, vectors, + &options, &batcher) != ANN_OK) + return 1; + if (AnnBatchSubmit(batcher, query, 1, 0, NULL, 0, + 30000000, &request) != ANN_OK) + { + AnnBatchClose(batcher); + return 1; + } + if (AnnRequestWait(request, 30000000) == ANN_OK) + { + const AnnCandidate *results = AnnRequestResults(request, &count); + /* Consume results[0..count); storage belongs to request. */ + (void) results; + } + AnnRequestRelease(request); + AnnBatchClose(batcher); + return 0; +} +``` + +Link with `-Iann -Lann -lannbatch -pthread -lm`. The scheduler owns a canonical index copy by default. The worker uses `AnnBatchCreateBorrowed()` to share its immutable index; it closes the scheduler before unloading or replacing that index. Failed replacement builds preserve the old generation. + +Admission returns `ANN_OVERLOADED` without queueing when request count, bytes, or estimated work exceed capacity. The C byte charge includes the request structure, query, filter, and reserved result array. It excludes the resident index, allocator overhead, bounded scheduler metadata, input wire frames, and results retained by callers after completion. Work is estimated as eligible count times dimension. Limits apply to queued and running work; timed-out running requests retain reservations until execution finishes. + +Compatible jobs share metric, effective k, filter presence, and a coarse selectivity class. The oldest request anchors selection. The dispatcher groups jobs up to row/byte/work limits or the collection delay. CPU distance evaluation remains serial; there is no claimed GPU kernel batching or automatic cross-process aggregation. CPU worker search frames use this C scheduler directly. External protocol frames are still read serially. + +A monotonic-clock watcher expires queued and running requests. Queued cancellation removes the job; running cancellation is not allowed. `AnnBatchClose()` cancels queued work, marks unfinished results canceled, waits for backend work to finish, and releases the index. It does not forcibly terminate a C thread or preempt a GPU kernel. Request handles remain valid until `AnnRequestRelease()`, even after service close. Caller-held results are released with their handle. + +Lifecycle rules: coordinate producer threads before closing the service; do not use a batcher handle after `AnnBatchClose()`. Do not release a request handle while another caller is using that same handle. Backend descriptors must outlive the service. Worker threads must never call PostgreSQL backend APIs; future backend integration uses PostgreSQL's memory, lock, interrupt, and resource mechanisms in the backend process. + +## Protocol + +The C worker uses version 1, 32-byte little-endian headers: magic `PVAN`, u16 version, u16 operation, u64 request ID, u64 generation, u32 payload size, u32 status. Frames are capped at 64 MiB. Generation zero is reserved, and successful builds increase generation monotonically per process/session. + +| Operation | Request | Success result | +| --- | --- | --- | +| 0 capabilities | Empty | JSON describing compiled backend capabilities | +| 1 build | u32 rows, dimensions; row-major FP32 vectors | Empty | +| 2 search | u32 queries, dimensions, k, metric, filter; vectors; optional sparse allowlists | Per query: u32 count and `(u32 ordinal, FP64 distance)` records | +| 3 unload | Empty | Empty | +| 4 export | Empty | Canonical build payload, not a graph artifact | +| 5 diagnostic probe | Bounded bytes | Empty; never a search result | +| 6 index info | Empty | u32 rows and dimensions | +| 7–10 | Gathered and shared-memory search for the [pipelined CustomScan](../docs/ann-worker-pipeline.md) (`--listen` mode) | | +| 11 load index | u32 name length, name, directory path | Text `rows N dimensions D shards S dtype T source NAME` | +| 12 index search | u32 name length, name, u32 dimensions, k, filter count (UINT32_MAX = none); FP32 query; i64 keys | u32 count, u32 route (0 graph, 1 filtered graph, 2 exact), then `(i64 key, f32 distance)` per hit | +| 13–15 | Network auth, packed-filter index search, index info ([networked guide](../docs/ann-networked.md)) | | + +Metric values: 0 L2, 1 negative inner product, 2 cosine. Filter values: 0 none, 1 shared IDs, 2 IDs per query. Allowlists are u32 count followed by strictly increasing unique u32 ordinals; an empty list differs from no filter. Capability masks use `1 << enum_value`. The CPU backend uses FP32 input and double accumulation. + +Statuses 0–5 are success, invalid, unsupported, stale, allocation failure, and backend failure. The C scheduling API adds 6 pending, 7 overloaded, 8 timed out, 9 canceled, and 10 closed. Search errors have empty protocol payloads. The PostgreSQL client enforces `ann.worker_timeout_ms`, and server mode sets a 10 s reply write timeout; timeouts in the Python test client are test infrastructure, not a shipped C client feature. + +## Test references and benchmarks + +[test/ann/support/](../test/ann/support/) contains Python-only protocol clients and a frozen-table SQL oracle. Snapshot checksum/save/load helpers there are test conveniences; production canonical export/import is implemented in C, while a durable C artifact-file format remains pending. + +```sh +python3 -m pip install -r test/ann/requirements-postgres.txt +ANN_PG_DSN='dbname=your_test_database' make -C ann check-postgres +python3 benchmarks/ann/postgres.py --rows 10000 --dimensions 32 > /tmp/ann-baseline.sql +psql -X -v ON_ERROR_STOP=1 -f /tmp/ann-baseline.sql +python3 benchmarks/ann/search.py --mode shared --batch 16 +python3 benchmarks/ann/batching.py --requests 1000 --sequential +python3 benchmarks/ann/batching.py --requests 1000 --pending 1024 --batch-rows 64 +``` + +The batching launcher runs [a C benchmark](../benchmarks/ann/batch.c); search and scheduling execute entirely in C. It reports successful/offered/rejected counts, elapsed time, and batch occupancy. The test-client search benchmark measures IPC and uses an independent distance oracle. Neither benchmark includes SQL eligibility or establishes production QPS. The PostgreSQL fixtures create temporary or uniquely named test objects and require the corresponding database permissions. + +C tests cover batching/filter isolation, input copying, budgets, queued/running deadlines, concurrent producers, early handle release, shutdown during execution, backend errors, and result lifetime after close. Python protocol tests independently validate metrics and filters. Sanitizer checks: + +```sh +make -C ann clean +make -C ann CFLAGS='-O1 -g -std=c11 -Wall -Wextra -Werror -pedantic -fsanitize=address,undefined -fno-omit-frame-pointer' check +``` + +## cuVS integration + +`cuvs.c` is compiled and validated against cuVS 26.10 on NVIDIA B200 GPUs (`make -C ann check-cuvs`): + +- **Gathered search** serves the [pipelined CustomScan](../docs/ann-worker-pipeline.md) with exact GPU brute force. +- **Loaded indexes** serve sharded CAGRA search across all GPUs, with bitset filters and an exact route for selective filters ([guide](../docs/ann-cuvs-index.md)). + +The stdin-mode resident CAGRA build (operations 1–4 with `--backend cuvs`) is still the older unvalidated draft. Use loaded indexes instead. hipVS and NVFP4 adapters are not implemented: cuVS 26.10 CAGRA supports FP32, FP16, int8 and uint8 datasets, and no FP4 or FP8 path. + +C PostgreSQL worker integration is implemented: `--listen PATH` serves pipelined gathered searches (operations 7–10) to `AnnFilteredSearch`, with a `mock-gpu` backend standing in for accelerators ([guide](../docs/ann-worker-pipeline.md)). Sustained-load validation remains. Python remains test/benchmark tooling only. diff --git a/ann/backend.h b/ann/backend.h new file mode 100644 index 000000000..58d98a27f --- /dev/null +++ b/ann/backend.h @@ -0,0 +1,125 @@ +#ifndef ANN_BACKEND_H +#define ANN_BACKEND_H + +#include <stddef.h> +#include <stdint.h> + +#define ANN_MAX_BYTES (64U * 1024U * 1024U) +#define ANN_MAX_DIM 16000U + +typedef enum AnnStatus +{ + ANN_OK, + ANN_INVALID, + ANN_UNSUPPORTED, + ANN_STALE, + ANN_NOMEM, + ANN_BACKEND_ERROR, + ANN_PENDING, + ANN_OVERLOADED, + ANN_TIMED_OUT, + ANN_CANCELLED, + ANN_CLOSED +} AnnStatus; + +typedef struct AnnIndex +{ + uint64_t generation; + uint32_t rows; + uint32_t dimensions; + float *vectors; + void *backend_state; +} AnnIndex; + +typedef struct AnnCandidate +{ + uint32_t id; + double distance; +} AnnCandidate; + +/* + * A gathered job scores caller-supplied eligible vectors against one query. + * Results hold every row within a small tolerance of the k-th distance, so a + * caller recomputing distances with different rounding cannot lose a row of + * its own top k. Results capacity must be at least rows. + */ +typedef struct AnnGatheredJob +{ + uint32_t rows; + uint32_t dimensions; + uint32_t k; + uint32_t metric; + const float *query; + const float *vectors; + AnnCandidate *results; + uint32_t nresults; + AnnStatus status; +} AnnGatheredJob; + +/* A loaded on-disk index result: table key and L2 distance. */ +typedef struct AnnIndexHit +{ + int64_t key; + float distance; +} AnnIndexHit; + +#define ANN_ROUTE_ANN 0 /* unfiltered graph search */ +#define ANN_ROUTE_FILTERED 1 /* graph search with an eligibility bitset */ +#define ANN_ROUTE_EXACT 2 /* exact scan of eligible rows */ + +/* + * One query against a named loaded index. The filter, when present, is a + * sorted unique key list; keys missing from the index are ignored. Results + * hold up to k hits sorted by distance. + */ +typedef struct AnnIndexJob +{ + char name[64]; + uint32_t dimensions; + uint32_t k; + const float *query; + const int64_t *filter; + uint32_t nfilter; + int filtered; + AnnIndexHit *results; + uint32_t nresults; + uint32_t route; + AnnStatus status; +} AnnIndexJob; + +/* Vendor-specific objects and precision/layout conversion stay behind this API. */ +typedef struct AnnBackend +{ + const char *name; + uint32_t metrics; + uint32_t filters; + const char *algorithm; + const char *storage; + int exact; + uint32_t min_rows; + uint32_t max_k; + AnnStatus (*build) (AnnIndex *, uint64_t, uint32_t, uint32_t, const float *); + AnnStatus (*search) (const AnnIndex *, const float *, uint32_t, + const uint32_t *, uint32_t, uint32_t, + AnnCandidate *, uint32_t *); + void (*destroy) (AnnIndex *); + /* One device submission for a batch of gathered jobs; sets each status. */ + void (*search_gathered) (AnnGatheredJob *, uint32_t); + /* Gathered jobs settled on a device, or NULL for host-only backends. */ + uint64_t (*device_jobs) (void); + /* Load an immutable index directory under a name; search loaded ones. */ + AnnStatus (*load_index) (const char *, const char *); + void (*search_index) (AnnIndexJob *, uint32_t); +} AnnBackend; + +#define ANN_GATHERED_TOLERANCE 1e-3 + +void AnnGatheredExact(AnnGatheredJob *job); + +const AnnBackend *AnnCpuBackend(void); +const AnnBackend *AnnMockGpuBackend(void); +#ifdef ANN_USE_CUVS +const AnnBackend *AnnCuvsBackend(void); +#endif + +#endif diff --git a/ann/batching.c b/ann/batching.c new file mode 100644 index 000000000..52737604e --- /dev/null +++ b/ann/batching.c @@ -0,0 +1,645 @@ +/* POSIX standalone-worker scheduler. PostgreSQL backends use their own APIs. */ +#ifndef __APPLE__ +#define _POSIX_C_SOURCE 200809L +#endif +#include "batching.h" + +#include <errno.h> +#include <math.h> +#include <pthread.h> +#include <stdatomic.h> +#include <stdlib.h> +#include <string.h> +#include <time.h> + +#define ANN_DAY_US UINT64_C(86400000000) + +struct AnnRequest +{ + AnnBatcher *owner; + AnnRequest *next; + atomic_uint references; + pthread_mutex_t mutex; + pthread_cond_t condition; + float *query; + uint32_t *ids; + AnnCandidate *results; + uint32_t nids; + uint32_t k; + uint32_t metric; + uint32_t count; + uint32_t bucket; + int filtered; + int state; /* 0 queued, 1 running, 2 physically complete */ + uint64_t bytes; + uint64_t work; + uint64_t arrived; + uint64_t deadline; + AnnStatus status; /* Public result status; may expire during + * execution */ +}; + +struct AnnBatcher +{ + const AnnBackend *backend; + AnnIndex index; + AnnBatchOptions options; + AnnBatchStats stats; + atomic_uint references; + pthread_mutex_t mutex; + pthread_cond_t condition; + pthread_t dispatcher; + pthread_t watcher; + AnnRequest *head; + AnnRequest *tail; + AnnRequest **active; + uint32_t nactive; + int stopping; + int owns_index; +}; + +static uint64_t +Now(void) +{ + struct timespec ts; + + (void) clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t) ts.tv_sec * UINT64_C(1000000000) + (uint64_t) ts.tv_nsec; +} + +static int +InitCondition(pthread_cond_t *condition) +{ +#ifdef __APPLE__ + return pthread_cond_init(condition, NULL); +#else + pthread_condattr_t attr; + int result; + + if (pthread_condattr_init(&attr)) + return -1; + result = pthread_condattr_setclock(&attr, CLOCK_MONOTONIC); + if (!result) + result = pthread_cond_init(condition, &attr); + (void) pthread_condattr_destroy(&attr); + return result; +#endif +} + +static void +WaitUntil(pthread_cond_t *condition, pthread_mutex_t *mutex, uint64_t deadline) +{ + struct timespec ts; +#ifdef __APPLE__ + uint64_t now = Now(); + + if (now >= deadline) + return; + deadline -= now; +#endif + ts.tv_sec = (time_t) (deadline / UINT64_C(1000000000)); + ts.tv_nsec = (long) (deadline % UINT64_C(1000000000)); +#ifdef __APPLE__ + (void) pthread_cond_timedwait_relative_np(condition, mutex, &ts); +#else + (void) pthread_cond_timedwait(condition, mutex, &ts); +#endif +} + +static void +ReleaseBatcher(AnnBatcher *batcher) +{ + if (atomic_fetch_sub(&batcher->references, 1) == 1) + { + (void) pthread_cond_destroy(&batcher->condition); + (void) pthread_mutex_destroy(&batcher->mutex); + free(batcher->active); + free(batcher); + } +} + +void +AnnRequestRelease(AnnRequest *request) +{ + if (request && atomic_fetch_sub(&request->references, 1) == 1) + { + AnnBatcher *owner = request->owner; + + (void) pthread_cond_destroy(&request->condition); + (void) pthread_mutex_destroy(&request->mutex); + free(request->query); + free(request->ids); + free(request->results); + free(request); + ReleaseBatcher(owner); + } +} + +/* Called with the scheduler mutex held; lock order is scheduler then request. */ +static void +SetStatus(AnnBatcher *batcher, AnnRequest *request, AnnStatus status) +{ + (void) pthread_mutex_lock(&request->mutex); + if (request->status == ANN_PENDING) + { + request->status = status; + if (status == ANN_OK) + batcher->stats.completed++; + else if (status == ANN_TIMED_OUT) + batcher->stats.timed_out++; + else if (status == ANN_CANCELLED) + batcher->stats.cancelled++; + else + batcher->stats.failed++; + (void) pthread_cond_broadcast(&request->condition); + } + (void) pthread_mutex_unlock(&request->mutex); +} + +static void +Finish(AnnBatcher *batcher, AnnRequest *request, AnnStatus status) +{ + if (status == ANN_OK && Now() >= request->deadline) + status = ANN_TIMED_OUT; + SetStatus(batcher, request, status); + request->state = 2; + free(request->query); + free(request->ids); + request->query = NULL; + request->ids = NULL; + batcher->stats.pending_requests--; + batcher->stats.pending_bytes -= request->bytes; + batcher->stats.pending_work -= request->work; + AnnRequestRelease(request); /* Release scheduler reference after execution */ +} + +static void +Remove(AnnBatcher *batcher, AnnRequest *request, AnnRequest *previous) +{ + if (previous) + previous->next = request->next; + else + batcher->head = request->next; + if (batcher->tail == request) + batcher->tail = previous; + request->next = NULL; +} + +static int +Compatible(const AnnRequest *a, const AnnRequest *b) +{ + return a->metric == b->metric && a->k == b->k && + a->filtered == b->filtered && a->bucket == b->bucket; +} + +static void +Expire(AnnBatcher *batcher) +{ + AnnRequest *request = batcher->head; + AnnRequest *previous = NULL; + uint64_t now = Now(); + + while (request) + { + AnnRequest *next = request->next; + + if (now >= request->deadline) + { + Remove(batcher, request, previous); + Finish(batcher, request, ANN_TIMED_OUT); + } + else + previous = request; + request = next; + } + for (uint32_t i = 0; i < batcher->nactive; i++) + if (now >= batcher->active[i]->deadline) + SetStatus(batcher, batcher->active[i], ANN_TIMED_OUT); +} + +static void * +Watch(void *arg) +{ + AnnBatcher *batcher = arg; + + (void) pthread_mutex_lock(&batcher->mutex); + while (!batcher->stopping) + { + uint64_t deadline = UINT64_MAX; + + Expire(batcher); + for (AnnRequest *r = batcher->head; r; r = r->next) + if (r->deadline < deadline) + deadline = r->deadline; + for (uint32_t i = 0; i < batcher->nactive; i++) + { + AnnRequest *r = batcher->active[i]; + + if (r->deadline > Now() && r->deadline < deadline) + deadline = r->deadline; + } + (void) pthread_cond_broadcast(&batcher->condition); + if (deadline == UINT64_MAX) + (void) pthread_cond_wait(&batcher->condition, &batcher->mutex); + else + WaitUntil(&batcher->condition, &batcher->mutex, deadline); + } + (void) pthread_mutex_unlock(&batcher->mutex); + return NULL; +} + +static void * +Dispatch(void *arg) +{ + AnnBatcher *batcher = arg; + + (void) pthread_mutex_lock(&batcher->mutex); + while (!batcher->stopping) + { + AnnRequest *first; + uint64_t bytes = 0; + uint64_t work = 0; + uint32_t count = 0; + int limited = 0; + + Expire(batcher); + if (!batcher->head) + { + (void) pthread_cond_wait(&batcher->condition, &batcher->mutex); + continue; + } + first = batcher->head; + for (AnnRequest *r = first; r; r = r->next) + { + if (!Compatible(first, r)) + continue; + if (count == batcher->options.max_batch_rows || + bytes + r->bytes > batcher->options.max_batch_bytes || + work + r->work > batcher->options.max_batch_work) + { + limited = 1; + continue; + } + batcher->active[count++] = r; + bytes += r->bytes; + work += r->work; + } + if (!limited && count < batcher->options.max_batch_rows && + Now() < first->arrived + batcher->options.delay_us * 1000) + { + WaitUntil(&batcher->condition, &batcher->mutex, + first->arrived + batcher->options.delay_us * 1000); + continue; + } + for (uint32_t i = 0; i < count; i++) + { + AnnRequest *r = batcher->active[i]; + AnnRequest *previous = NULL; + + for (AnnRequest *p = batcher->head; p != r; p = p->next) + previous = p; + Remove(batcher, r, previous); + r->state = 1; + } + batcher->nactive = count; + batcher->stats.batches++; + batcher->stats.batch_rows += count; + if (count > batcher->stats.max_observed_batch) + batcher->stats.max_observed_batch = count; + (void) pthread_cond_broadcast(&batcher->condition); + (void) pthread_mutex_unlock(&batcher->mutex); + for (uint32_t i = 0; i < count; i++) + { + AnnRequest *r = batcher->active[i]; + AnnStatus status; + uint32_t nresults = 0; + + status = batcher->backend->search(&batcher->index, r->query, r->metric, + r->ids, r->nids, r->k, r->results, &nresults); + (void) pthread_mutex_lock(&batcher->mutex); + (void) pthread_mutex_lock(&r->mutex); + r->count = nresults; + (void) pthread_mutex_unlock(&r->mutex); + + /* + * Keep the request in the active set until the entire batch + * finishes. + */ + SetStatus(batcher, r, Now() >= r->deadline ? ANN_TIMED_OUT : status); + (void) pthread_mutex_unlock(&batcher->mutex); + } + (void) pthread_mutex_lock(&batcher->mutex); + for (uint32_t i = 0; i < count; i++) + Finish(batcher, batcher->active[i], ANN_BACKEND_ERROR); + batcher->nactive = 0; + (void) pthread_cond_broadcast(&batcher->condition); + } + (void) pthread_mutex_unlock(&batcher->mutex); + return NULL; +} + +void +AnnBatchDefaultOptions(AnnBatchOptions *options) +{ + *options = (AnnBatchOptions) + { + 64, 256, 8 * 1024 * 1024, 16000000, + 64 * 1024 * 1024, 128000000, 100 + }; +} + +static AnnStatus +Create(const AnnBackend *backend, uint64_t generation, uint32_t rows, + uint32_t dimensions, const float *vectors, + const AnnIndex *borrowed, const AnnBatchOptions *options, AnnBatcher **result) +{ + AnnBatcher *batcher; + AnnStatus status; + + *result = NULL; + if (!generation || !backend || !options || !backend->build || !backend->search || !backend->destroy || + !dimensions || dimensions > ANN_MAX_DIM || (!vectors && rows) || + (uint64_t) rows * dimensions > ANN_MAX_BYTES / sizeof(float) || + !options->max_batch_rows || options->max_batch_rows > 1024 || + !options->max_pending_requests || !options->max_batch_bytes || + !options->max_batch_work || !options->max_pending_bytes || !options->max_pending_work || + options->delay_us > 60000000) + return ANN_INVALID; + batcher = calloc(1, sizeof(*batcher)); + if (!batcher) + return ANN_NOMEM; + batcher->backend = backend; + batcher->options = *options; + atomic_init(&batcher->references, 1); + batcher->active = calloc(options->max_batch_rows, sizeof(*batcher->active)); + if (!batcher->active) + { + free(batcher); + return ANN_NOMEM; + } + batcher->owns_index = borrowed == NULL; + if (borrowed) + { + batcher->index = *borrowed; + status = ANN_OK; + } + else + status = backend->build(&batcher->index, generation, rows, dimensions, vectors); + if (status != ANN_OK) + goto fail; + if (pthread_mutex_init(&batcher->mutex, NULL)) + goto init_fail; + if (InitCondition(&batcher->condition)) + { + (void) pthread_mutex_destroy(&batcher->mutex); + goto init_fail; + } + if (pthread_create(&batcher->dispatcher, NULL, Dispatch, batcher)) + goto thread_fail; + if (pthread_create(&batcher->watcher, NULL, Watch, batcher)) + { + (void) pthread_mutex_lock(&batcher->mutex); + batcher->stopping = 1; + (void) pthread_cond_broadcast(&batcher->condition); + (void) pthread_mutex_unlock(&batcher->mutex); + (void) pthread_join(batcher->dispatcher, NULL); + goto thread_fail; + } + *result = batcher; + return ANN_OK; +thread_fail: + (void) pthread_cond_destroy(&batcher->condition); + (void) pthread_mutex_destroy(&batcher->mutex); +init_fail: + status = ANN_BACKEND_ERROR; +fail: + if (batcher->owns_index) + backend->destroy(&batcher->index); + free(batcher->active); + free(batcher); + return status; +} + +AnnStatus +AnnBatchCreate(const AnnBackend *backend, uint64_t generation, uint32_t rows, + uint32_t dimensions, const float *vectors, + const AnnBatchOptions *options, AnnBatcher **result) +{ + return Create(backend, generation, rows, dimensions, vectors, NULL, options, result); +} + +AnnStatus +AnnBatchCreateBorrowed(const AnnBackend *backend, const AnnIndex *index, + const AnnBatchOptions *options, AnnBatcher **result) +{ + if (!index) + return ANN_INVALID; + return Create(backend, index->generation, index->rows, index->dimensions, + index->vectors, index, options, result); +} + +AnnStatus +AnnBatchSubmit(AnnBatcher *batcher, const float *query, uint32_t k, uint32_t metric, + const uint32_t *ids, uint32_t nids, uint64_t timeout_us, AnnRequest **result) +{ + AnnRequest *request; + uint32_t count = ids ? nids : batcher->index.rows; + uint64_t work = (uint64_t) count * batcher->index.dimensions; + uint64_t bytes; + uint32_t bucket = 0; + uint32_t bits = count; + AnnStatus status = ANN_NOMEM; + + *result = NULL; + if (!query || metric > 2 || !(batcher->backend->metrics & (1U << metric)) || + !timeout_us || timeout_us > ANN_DAY_US || (ids && nids > batcher->index.rows)) + return ANN_INVALID; + { + double norm = 0; + + for (uint32_t i = 0; i < batcher->index.dimensions; i++) + { + if (!isfinite(query[i])) + return ANN_INVALID; + norm += (double) query[i] * query[i]; + } + if (metric == 2 && norm == 0) + return ANN_INVALID; + } + for (uint32_t i = 0; ids && i < nids; i++) + if (ids[i] >= batcher->index.rows || (i && ids[i] <= ids[i - 1])) + return ANN_INVALID; + if (k > count) + k = count; + bytes = sizeof(AnnRequest) + (uint64_t) batcher->index.dimensions * sizeof(float) + + (ids ? (uint64_t) (nids ? nids : 1) * sizeof(uint32_t) : 0) + + (uint64_t) (k ? k : 1) * sizeof(AnnCandidate); + if (!work) + work = 1; + while (bits) + { + bucket++; + bits >>= 1; + } + (void) pthread_mutex_lock(&batcher->mutex); + if (batcher->stopping) + { + status = ANN_CLOSED; + goto done; + } + if (bytes > batcher->options.max_batch_bytes || work > batcher->options.max_batch_work || + batcher->stats.pending_requests >= batcher->options.max_pending_requests || + bytes > batcher->options.max_pending_bytes - batcher->stats.pending_bytes || + work > batcher->options.max_pending_work - batcher->stats.pending_work) + { + batcher->stats.rejected++; + status = ANN_OVERLOADED; + goto done; + } + request = calloc(1, sizeof(*request)); + if (!request) + goto done; + if (pthread_mutex_init(&request->mutex, NULL)) + { + free(request); + goto done; + } + if (InitCondition(&request->condition)) + { + (void) pthread_mutex_destroy(&request->mutex); + free(request); + goto done; + } + atomic_init(&request->references, 2); + atomic_fetch_add(&batcher->references, 1); + request->owner = batcher; + request->query = malloc((size_t) batcher->index.dimensions * sizeof(float)); + request->ids = ids ? malloc((size_t) (nids ? nids : 1) * sizeof(uint32_t)) : NULL; + request->results = malloc((size_t) (k ? k : 1) * sizeof(AnnCandidate)); + if (!request->query || (ids && !request->ids) || !request->results) + { + AnnRequestRelease(request); + AnnRequestRelease(request); + goto done; + } + memcpy(request->query, query, (size_t) batcher->index.dimensions * sizeof(float)); + if (ids) + memcpy(request->ids, ids, (size_t) nids * sizeof(uint32_t)); + request->filtered = ids != NULL; + request->nids = nids; + request->k = k; + request->metric = metric; + request->bucket = bucket / 4; + request->bytes = bytes; + request->work = work; + request->status = ANN_PENDING; + request->arrived = Now(); + request->deadline = request->arrived + timeout_us * 1000; + if (batcher->tail) + batcher->tail->next = request; + else + batcher->head = request; + batcher->tail = request; + batcher->stats.admitted++; + batcher->stats.pending_requests++; + batcher->stats.pending_bytes += bytes; + batcher->stats.pending_work += work; + *result = request; + status = ANN_OK; + (void) pthread_cond_broadcast(&batcher->condition); +done: + (void) pthread_mutex_unlock(&batcher->mutex); + return status; +} + +AnnStatus +AnnRequestWait(AnnRequest *request, uint64_t wait_us) +{ + AnnStatus status; + uint64_t deadline; + + if (wait_us > ANN_DAY_US) + return ANN_INVALID; + deadline = Now() + wait_us * 1000; + (void) pthread_mutex_lock(&request->mutex); + while (request->status == ANN_PENDING && Now() < deadline) + WaitUntil(&request->condition, &request->mutex, deadline); + status = request->status; + (void) pthread_mutex_unlock(&request->mutex); + return status; +} + +AnnStatus +AnnRequestPoll(AnnRequest *request) +{ + return AnnRequestWait(request, 0); +} + +const AnnCandidate * +AnnRequestResults(AnnRequest *request, uint32_t *count) +{ + const AnnCandidate *results = NULL; + + (void) pthread_mutex_lock(&request->mutex); + *count = 0; + if (request->status == ANN_OK) + { + *count = request->count; + results = request->results; + } + (void) pthread_mutex_unlock(&request->mutex); + return results; +} + +int +AnnRequestCancel(AnnRequest *request) +{ + AnnBatcher *batcher = request->owner; + int cancelled = 0; + + (void) pthread_mutex_lock(&batcher->mutex); + if (request->state == 0) + { + AnnRequest *previous = NULL; + + for (AnnRequest *r = batcher->head; r != request; r = r->next) + previous = r; + Remove(batcher, request, previous); + Finish(batcher, request, ANN_CANCELLED); + cancelled = 1; + (void) pthread_cond_broadcast(&batcher->condition); + } + (void) pthread_mutex_unlock(&batcher->mutex); + return cancelled; +} + +void +AnnBatchGetStats(AnnBatcher *batcher, AnnBatchStats *stats) +{ + (void) pthread_mutex_lock(&batcher->mutex); + *stats = batcher->stats; + (void) pthread_mutex_unlock(&batcher->mutex); +} + +void +AnnBatchClose(AnnBatcher *batcher) +{ + (void) pthread_mutex_lock(&batcher->mutex); + batcher->stopping = 1; + while (batcher->head) + { + AnnRequest *r = batcher->head; + + Remove(batcher, r, NULL); + Finish(batcher, r, ANN_CANCELLED); + } + for (uint32_t i = 0; i < batcher->nactive; i++) + SetStatus(batcher, batcher->active[i], ANN_CANCELLED); + (void) pthread_cond_broadcast(&batcher->condition); + (void) pthread_mutex_unlock(&batcher->mutex); + (void) pthread_join(batcher->dispatcher, NULL); + (void) pthread_join(batcher->watcher, NULL); + if (batcher->owns_index) + batcher->backend->destroy(&batcher->index); + ReleaseBatcher(batcher); +} diff --git a/ann/batching.h b/ann/batching.h new file mode 100644 index 000000000..8780bdca9 --- /dev/null +++ b/ann/batching.h @@ -0,0 +1,58 @@ +#ifndef ANN_BATCHING_H +#define ANN_BATCHING_H + +#include "backend.h" + +/* Standalone worker API. Never call PostgreSQL backend APIs on these threads. */ +typedef struct AnnBatcher AnnBatcher; +typedef struct AnnRequest AnnRequest; + +typedef struct AnnBatchOptions +{ + uint32_t max_batch_rows; + uint32_t max_pending_requests; + uint64_t max_batch_bytes; + uint64_t max_batch_work; + uint64_t max_pending_bytes; + uint64_t max_pending_work; + uint64_t delay_us; +} AnnBatchOptions; + +typedef struct AnnBatchStats +{ + uint64_t admitted; + uint64_t rejected; + uint64_t completed; + uint64_t failed; + uint64_t cancelled; + uint64_t timed_out; + uint64_t batches; + uint64_t batch_rows; + uint64_t max_observed_batch; + uint64_t pending_requests; + uint64_t pending_bytes; + uint64_t pending_work; +} AnnBatchStats; + +void AnnBatchDefaultOptions(AnnBatchOptions *options); +AnnStatus AnnBatchCreate(const AnnBackend *backend, uint64_t generation, + uint32_t rows, uint32_t dimensions, const float *vectors, + const AnnBatchOptions *options, AnnBatcher **result); + +/* Caller keeps the immutable index alive until AnnBatchClose has returned. */ +AnnStatus AnnBatchCreateBorrowed(const AnnBackend *backend, const AnnIndex *index, + const AnnBatchOptions *options, AnnBatcher **result); +AnnStatus AnnBatchSubmit(AnnBatcher *batcher, const float *query, uint32_t k, + uint32_t metric, const uint32_t *ids, uint32_t nids, + uint64_t timeout_us, AnnRequest **result); +AnnStatus AnnRequestWait(AnnRequest *request, uint64_t wait_us); +AnnStatus AnnRequestPoll(AnnRequest *request); +int AnnRequestCancel(AnnRequest *request); +const AnnCandidate *AnnRequestResults(AnnRequest *request, uint32_t *count); +void AnnRequestRelease(AnnRequest *request); +void AnnBatchGetStats(AnnBatcher *batcher, AnnBatchStats *stats); + +/* Cancels queued requests and waits for in-flight backend work to finish. */ +void AnnBatchClose(AnnBatcher *batcher); + +#endif diff --git a/ann/cpu.c b/ann/cpu.c new file mode 100644 index 000000000..473b5b7b6 --- /dev/null +++ b/ann/cpu.c @@ -0,0 +1,405 @@ +#include "backend.h" +#include "indexfile.h" + +#include <math.h> +#include <stdlib.h> +#include <string.h> + +static void +Destroy(AnnIndex *index) +{ + free(index->vectors); + memset(index, 0, sizeof(*index)); +} + +static AnnStatus +Build(AnnIndex *index, uint64_t generation, uint32_t rows, + uint32_t dimensions, const float *vectors) +{ + uint64_t wideCount = (uint64_t) rows * dimensions; + size_t count; + float *copy; + + if (generation == 0 || dimensions == 0 || dimensions > ANN_MAX_DIM || + wideCount > ANN_MAX_BYTES / sizeof(float) || (wideCount && vectors == NULL)) + return ANN_INVALID; + count = (size_t) wideCount; + for (size_t i = 0; i < count; i++) + if (!isfinite(vectors[i])) + return ANN_INVALID; + copy = malloc(count ? count * sizeof(float) : 1); + if (copy == NULL) + return ANN_NOMEM; + if (count) + memcpy(copy, vectors, count * sizeof(float)); + Destroy(index); + index->vectors = copy; + index->rows = rows; + index->dimensions = dimensions; + index->generation = generation; + return ANN_OK; +} + +static int +Compare(const AnnCandidate *a, const AnnCandidate *b) +{ + if (a->distance != b->distance) + return a->distance < b->distance ? -1 : 1; + return a->id < b->id ? -1 : a->id > b->id; +} + +/* Keep the worst result at the root of a bounded top-k heap. */ +static void +SiftDown(AnnCandidate *heap, uint32_t count, uint32_t parent) +{ + for (;;) + { + uint32_t child = parent * 2 + 1; + AnnCandidate tmp; + + if (child >= count) + break; + if (child + 1 < count && Compare(&heap[child], &heap[child + 1]) < 0) + child++; + if (Compare(&heap[parent], &heap[child]) >= 0) + break; + tmp = heap[parent]; + heap[parent] = heap[child]; + heap[child] = tmp; + parent = child; + } +} + +static int +SortCompare(const void *a, const void *b) +{ + return Compare(a, b); +} + +static AnnStatus +Search(const AnnIndex *index, const float *query, uint32_t metric, + const uint32_t *ids, uint32_t nids, uint32_t k, + AnnCandidate *results, uint32_t *nresults) +{ + uint32_t count = ids ? nids : index->rows; + double queryNorm = 0; + + *nresults = 0; + if (metric > 2) + return ANN_UNSUPPORTED; + for (uint32_t d = 0; d < index->dimensions; d++) + { + if (!isfinite(query[d])) + return ANN_INVALID; + queryNorm += (double) query[d] * query[d]; + } + if (metric == 2 && queryNorm == 0) + return ANN_INVALID; + + /* + * Sorted unique local IDs prevent duplicate output and permit linear + * validation. + */ + for (uint32_t i = 0; ids && i < nids; i++) + if (ids[i] >= index->rows || (i && ids[i] <= ids[i - 1])) + return ANN_INVALID; + for (uint32_t i = 0; i < count; i++) + { + uint32_t id = ids ? ids[i] : i; + const float *vector = index->vectors + (size_t) id * index->dimensions; + double distance = 0; + double norm = 0; + AnnCandidate candidate; + + for (uint32_t d = 0; d < index->dimensions; d++) + { + double a = vector[d]; + double b = query[d]; + + distance += metric == 0 ? (a - b) * (a - b) : a * b; + norm += a * a; + } + if (metric == 2 && norm == 0) + continue; + candidate.id = id; + candidate.distance = metric == 0 ? sqrt(distance) : + metric == 1 ? -distance : + 1 - fmax(-1, fmin(1, distance / sqrt(norm * queryNorm))); + if (k == 0) + continue; + if (*nresults < k) + { + uint32_t child = (*nresults)++; + + results[child] = candidate; + while (child && Compare(&results[(child - 1) / 2], &results[child]) < 0) + { + AnnCandidate tmp = results[child]; + + results[child] = results[(child - 1) / 2]; + results[(child - 1) / 2] = tmp; + child = (child - 1) / 2; + } + } + else if (Compare(&candidate, &results[0]) < 0) + { + results[0] = candidate; + SiftDown(results, *nresults, 0); + } + } + qsort(results, *nresults, sizeof(*results), SortCompare); + return ANN_OK; +} + +/* Eight partial sums let compilers vectorize without fast-math flags. */ +static float +L2Squared(const float *a, const float *b, uint32_t dimensions) +{ + float sums[8] = {0}; + float sum = 0; + uint32_t d = 0; + + for (; d + 8 <= dimensions; d += 8) + for (int j = 0; j < 8; j++) + { + float diff = a[d + j] - b[d + j]; + + sums[j] += diff * diff; + } + for (; d < dimensions; d++) + sum += (a[d] - b[d]) * (a[d] - b[d]); + return sum + ((sums[0] + sums[1]) + (sums[2] + sums[3])) + + ((sums[4] + sums[5]) + (sums[6] + sums[7])); +} + +/* Partition so position n holds the n-th smallest distance. */ +static void +Select(AnnCandidate *items, uint32_t count, uint32_t n) +{ + uint32_t left = 0; + uint32_t right = count - 1; + + while (left < right) + { + double pivot = items[left + (right - left) / 2].distance; + uint32_t i = left; + uint32_t j = right; + + while (i <= j) + { + while (items[i].distance < pivot) + i++; + while (items[j].distance > pivot) + j--; + if (i <= j) + { + AnnCandidate tmp = items[i]; + + items[i] = items[j]; + items[j] = tmp; + i++; + if (j == 0) + break; + j--; + } + } + if (n <= j) + right = j; + else if (n >= i) + left = i; + else + break; + } +} + +void +AnnGatheredExact(AnnGatheredJob *job) +{ + double threshold; + uint32_t count = 0; + + job->nresults = 0; + if (job->dimensions == 0 || job->dimensions > ANN_MAX_DIM || job->metric > 2) + { + job->status = ANN_INVALID; + return; + } + if (job->metric != 0) + { + AnnIndex index = {0}; + + index.rows = job->rows; + index.dimensions = job->dimensions; + index.vectors = (float *) job->vectors; + job->status = Search(&index, job->query, job->metric, NULL, 0, + job->rows, job->results, &job->nresults); + } + else + { + for (uint32_t i = 0; i < job->rows; i++) + { + double distance = sqrt((double) L2Squared(job->vectors + (size_t) i * job->dimensions, + job->query, job->dimensions)); + + if (!isfinite(distance)) + { + job->status = ANN_INVALID; + return; + } + job->results[i].id = i; + job->results[i].distance = distance; + } + job->nresults = job->rows; + job->status = ANN_OK; + } + if (job->status != ANN_OK) + return; + if (job->k == 0) + { + job->nresults = 0; + return; + } + if (job->k < job->nresults) + { + /* Keep boundary ties and near-ties from differently rounded callers. */ + Select(job->results, job->nresults, job->k - 1); + threshold = job->results[job->k - 1].distance; + threshold += ANN_GATHERED_TOLERANCE * fabs(threshold) + 1e-6; + for (uint32_t i = 0; i < job->nresults; i++) + if (job->results[i].distance <= threshold) + job->results[count++] = job->results[i]; + job->nresults = count; + } + qsort(job->results, job->nresults, sizeof(*job->results), SortCompare); +} + +static void +SearchGathered(AnnGatheredJob *jobs, uint32_t count) +{ + for (uint32_t i = 0; i < count; i++) + AnnGatheredExact(&jobs[i]); +} + +static AnnStatus +LoadIndex(const char *name, const char *directory) +{ + AnnIndexFile *file; + AnnStatus status = AnnIndexFileOpen(name, directory, &file); + + if (status == ANN_OK && (status = AnnIndexRegister(file)) != ANN_OK) + AnnIndexFileClose(file); + return status; +} + +/* + * Exact reference for loaded indexes: each shard's key range is scanned + * separately and the per-shard top-k lists are tree-merged, mirroring the + * device route without approximation. + */ +static void +SearchIndexJob(AnnIndexJob *job) +{ + AnnIndexFile *file = AnnIndexGet(job->name); + uint64_t *ordinals = NULL; + uint64_t nordinals = 0; + uint32_t nshards; + AnnIndexHit *lists = NULL; + AnnIndexHit **heads = NULL; + uint32_t *counts = NULL; + + job->nresults = 0; + if (file == NULL) + { + job->status = ANN_STALE; + return; + } + if (job->dimensions != file->dimensions) + { + job->status = ANN_INVALID; + goto done; + } + if (job->filtered) + { + ordinals = malloc(sizeof(*ordinals) * (job->nfilter ? job->nfilter : 1)); + if (ordinals == NULL) + { + job->status = ANN_NOMEM; + goto done; + } + nordinals = AnnIndexOrdinals(file, job->filter, job->nfilter, ordinals); + } + nshards = file->nshards ? file->nshards : 1; + lists = malloc(sizeof(*lists) * ((size_t) job->k + 1) * nshards); + heads = malloc(sizeof(*heads) * nshards); + counts = malloc(sizeof(*counts) * nshards); + if (lists == NULL || heads == NULL || counts == NULL) + { + job->status = ANN_NOMEM; + goto done; + } + for (uint32_t s = 0; s < nshards; s++) + { + AnnIndexJob part = *job; + uint64_t first = file->nshards ? file->shards[s].first : 0; + uint64_t end = file->nshards ? first + file->shards[s].rows : file->rows; + uint64_t *range = NULL; + uint64_t nrange = 0; + uint64_t *all = NULL; + + part.results = heads[s] = lists + (size_t) s * (job->k + 1); + if (ordinals) + { + /* Ordinals are sorted; take this shard's slice. */ + uint64_t lo = 0; + + while (lo < nordinals && ordinals[lo] < first) + lo++; + range = ordinals + lo; + while (lo + nrange < nordinals && ordinals[lo + nrange] < end) + nrange++; + } + else + { + all = malloc(sizeof(*all) * (end - first ? end - first : 1)); + if (all == NULL) + { + job->status = ANN_NOMEM; + goto done; + } + for (uint64_t i = first; i < end; i++) + all[i - first] = i; + range = all; + nrange = end - first; + } + AnnIndexExact(file, &part, range, nrange); + counts[s] = part.nresults; + free(all); + } + job->nresults = AnnIndexTreeMerge(heads, counts, nshards, job->k, job->results); + job->route = job->filtered ? ANN_ROUTE_EXACT : ANN_ROUTE_ANN; + job->status = ANN_OK; +done: + free(lists); + free(heads); + free(counts); + free(ordinals); + AnnIndexPut(file); +} + +static void +SearchIndex(AnnIndexJob *jobs, uint32_t count) +{ + for (uint32_t i = 0; i < count; i++) + SearchIndexJob(&jobs[i]); +} + +const AnnBackend * +AnnCpuBackend(void) +{ + static const AnnBackend backend = { + "cpu-exact", 7, 7, "flat", "fp32", 1, 0, UINT32_MAX, Build, Search, Destroy, + SearchGathered, NULL, LoadIndex, SearchIndex + }; + + return &backend; +} diff --git a/ann/cuvs.c b/ann/cuvs.c new file mode 100644 index 000000000..0cf91e8af --- /dev/null +++ b/ann/cuvs.c @@ -0,0 +1,1463 @@ +/* + * Resident CAGRA search is an unvalidated draft. Gathered search (exact + * brute force over each request's eligible vectors) is GPU-validated. + */ +#include "backend.h" +#include "indexfile.h" + +#include <cuda_runtime_api.h> +#include <cuvs/neighbors/brute_force.h> +#include <cuvs/neighbors/cagra.h> +#include <math.h> +#include <pthread.h> +#include <stdatomic.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +typedef struct AnnCuvsState +{ + cuvsResources_t resources; + cuvsCagraIndex_t graph; + cuvsDataset_t dataset; + cuvsCagraIndexParams_t build; + cuvsCagraSearchParams_t search; + int device; +} AnnCuvsState; + +/* Error paths synchronize before releasing objects referenced by queued work. */ +#define CuvsCall(call) \ + do { \ + if ((call) != CUVS_SUCCESS) { \ + const char *message = cuvsGetLastErrorText(); \ + fprintf(stderr, "cuVS: %s\n", message ? message : "operation failed"); \ + goto done; \ + } \ + } while (0) + +#define CudaCall(call) \ + do { \ + cudaError_t error = (call); \ + if (error != cudaSuccess) { \ + fprintf(stderr, "CUDA: %s\n", cudaGetErrorString(error)); \ + goto done; \ + } \ + } while (0) + +static DLManagedTensor + Tensor(void *data, DLDeviceType deviceType, int device, + uint8_t code, int dimensions, int64_t *shape); + +/* + * Gathered search: every request carries its own eligible vectors, so each + * job is an exact brute-force search on a GPU. A lane is one worker thread; + * lanes are spread round-robin across visible devices and keep their own + * resources, stream, and device buffers. + */ +#define ANN_CUVS_MIN_ROWS 2048 /* smaller jobs are cheaper on the CPU */ +#define ANN_CUVS_MAX_K 1024 +#define ANN_CUVS_ERROR 2e-3 /* bound on fp32 relative distance error */ + +typedef struct AnnCuvsLane +{ + int ready; /* 0 unset, 1 usable, -1 no device */ + int device; + cuvsResources_t resources; + float *vectors; /* device: every job's rows, then queries */ + int64_t *neighbors; /* device */ + float *distances; /* device */ + int64_t *hostNeighbors; /* pinned */ + float *hostDistances; /* pinned */ + size_t vectorCapacity; /* floats */ + size_t resultCapacity; /* candidates */ +} AnnCuvsLane; + +static _Thread_local AnnCuvsLane lane; +static atomic_uint nextLane; +static atomic_ullong deviceJobs; + +static int +LaneReady(void) +{ + int devices = 0; + + if (lane.ready) + return lane.ready > 0; + lane.ready = -1; + if (cudaGetDeviceCount(&devices) != cudaSuccess || devices <= 0) + return 0; + lane.device = (int) (atomic_fetch_add(&nextLane, 1) % (unsigned) devices); + if (cudaSetDevice(lane.device) != cudaSuccess || + (cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync) != cudaSuccess && + cudaGetLastError() != cudaErrorSetOnActiveProcess) || + cuvsResourcesCreate(&lane.resources) != CUVS_SUCCESS) + return 0; + lane.ready = 1; + return 1; +} + +/* Grow lane buffers geometrically; buffers are reused across submissions. */ +static int +Reserve(size_t floats, size_t candidates) +{ + if (floats > lane.vectorCapacity) + { + if (lane.vectors) + (void) cudaFree(lane.vectors); + lane.vectorCapacity = 0; + floats += floats / 2; + if (cudaMalloc((void **) &lane.vectors, floats * sizeof(float)) != cudaSuccess) + { + lane.vectors = NULL; + return 0; + } + lane.vectorCapacity = floats; + } + if (candidates > lane.resultCapacity) + { + (void) cudaFree(lane.neighbors); + (void) cudaFree(lane.distances); + (void) cudaFreeHost(lane.hostNeighbors); + (void) cudaFreeHost(lane.hostDistances); + lane.neighbors = NULL; + lane.distances = NULL; + lane.hostNeighbors = NULL; + lane.hostDistances = NULL; + lane.resultCapacity = 0; + candidates += candidates / 2; + if (cudaMalloc((void **) &lane.neighbors, candidates * sizeof(int64_t)) != cudaSuccess || + cudaMalloc((void **) &lane.distances, candidates * sizeof(float)) != cudaSuccess || + cudaMallocHost((void **) &lane.hostNeighbors, candidates * sizeof(int64_t)) != cudaSuccess || + cudaMallocHost((void **) &lane.hostDistances, candidates * sizeof(float)) != cudaSuccess) + return 0; + lane.resultCapacity = candidates; + } + return 1; +} + +static int CandidateCompare(const void *left, const void *right); + +/* + * Re-score GPU candidates in double precision and keep rows within the + * gathered tolerance of the k-th exact distance. A row the GPU did not return + * has an fp32 distance of at least the last returned one; if that bound + * cannot exclude it from the threshold, the caller scores the job on the CPU. + */ +static int +FinishJob(AnnGatheredJob *job, const int64_t *neighbors, const float *distances, uint32_t count) +{ + double threshold; + uint32_t kept = 0; + + for (uint32_t i = 0; i < count; i++) + { + const float *vector; + double squared = 0; + + if (neighbors[i] < 0 || neighbors[i] >= job->rows) + return 0; + vector = job->vectors + (size_t) neighbors[i] * job->dimensions; + for (uint32_t d = 0; d < job->dimensions; d++) + { + double delta = (double) vector[d] - job->query[d]; + + squared += delta * delta; + } + job->results[i] = (AnnCandidate) + { + (uint32_t) neighbors[i], sqrt(squared) + }; + } + qsort(job->results, count, sizeof(*job->results), CandidateCompare); + threshold = job->results[job->k - 1].distance; + threshold += ANN_GATHERED_TOLERANCE * fabs(threshold) + 1e-6; + if (count < job->rows && + (double) distances[count - 1] * (1 - ANN_CUVS_ERROR) - 1e-6 <= threshold) + return 0; + while (kept < count && job->results[kept].distance <= threshold) + kept++; + job->nresults = kept; + job->status = ANN_OK; + return 1; +} + +static uint32_t +Candidates(const AnnGatheredJob *job) +{ + uint32_t count = job->k * 2 + 64; + + if (lane.ready <= 0 || job->metric != 0 || job->k == 0 || + job->rows < ANN_CUVS_MIN_ROWS || job->k > ANN_CUVS_MAX_K / 2) + return 0; + if (count > ANN_CUVS_MAX_K) + count = ANN_CUVS_MAX_K; + return count < job->rows ? count : job->rows; +} + +/* + * Queue copies, build, search, and result copies for every job on the lane's + * stream, then synchronize once. Jobs the GPU skips or cannot settle are + * scored by the exact CPU kernel. + */ +static void +SearchGathered(AnnGatheredJob *jobs, uint32_t count) +{ + size_t floats = 0; + size_t candidates = 0; + size_t vectorOffset = 0; + size_t queryOffset = 0; + size_t resultOffset = 0; + cudaStream_t stream; + cuvsBruteForceIndex_t *indexes = calloc(count, sizeof(*indexes)); + int synced; + + (void) LaneReady(); + for (uint32_t i = 0; i < count; i++) + { + uint32_t n = Candidates(&jobs[i]); + + jobs[i].status = ANN_PENDING; + if (n == 0) + continue; + floats += ((size_t) jobs[i].rows + 1) * jobs[i].dimensions; + queryOffset += (size_t) jobs[i].rows * jobs[i].dimensions; + candidates += n; + } + if (candidates == 0 || indexes == NULL || !Reserve(floats, candidates) || + cuvsStreamGet(lane.resources, &stream) != CUVS_SUCCESS) + goto cpu; + for (uint32_t i = 0; i < count; i++) + { + AnnGatheredJob *job = &jobs[i]; + uint32_t n = Candidates(job); + int64_t dataShape[2] = {job->rows, job->dimensions}; + int64_t queryShape[2] = {1, job->dimensions}; + int64_t resultShape[2] = {1, n}; + float *data = lane.vectors + vectorOffset; + float *query = lane.vectors + queryOffset; + DLManagedTensor dataTensor; + DLManagedTensor queryTensor; + DLManagedTensor neighborTensor; + DLManagedTensor distanceTensor; + cuvsBruteForceIndex_t index = NULL; + cuvsFilter filter = {0, NO_FILTER}; + + if (n == 0) + continue; + vectorOffset += (size_t) job->rows * job->dimensions; + queryOffset += job->dimensions; + dataTensor = Tensor(data, kDLCUDA, lane.device, kDLFloat, 2, dataShape); + queryTensor = Tensor(query, kDLCUDA, lane.device, kDLFloat, 2, queryShape); + neighborTensor = Tensor(lane.neighbors + resultOffset, kDLCUDA, lane.device, kDLInt, 2, resultShape); + neighborTensor.dl_tensor.dtype.bits = 64; + distanceTensor = Tensor(lane.distances + resultOffset, kDLCUDA, lane.device, kDLFloat, 2, resultShape); + if (cudaMemcpyAsync(data, job->vectors, (size_t) job->rows * job->dimensions * sizeof(float), + cudaMemcpyHostToDevice, stream) == cudaSuccess && + cudaMemcpyAsync(query, job->query, job->dimensions * sizeof(float), + cudaMemcpyHostToDevice, stream) == cudaSuccess && + cuvsBruteForceIndexCreate(&index) == CUVS_SUCCESS && + cuvsBruteForceBuild(lane.resources, &dataTensor, L2SqrtUnexpanded, 0, index) == CUVS_SUCCESS && + cuvsBruteForceSearch(lane.resources, index, &queryTensor, &neighborTensor, + &distanceTensor, filter) == CUVS_SUCCESS && + cudaMemcpyAsync(lane.hostNeighbors + resultOffset, lane.neighbors + resultOffset, + n * sizeof(int64_t), cudaMemcpyDeviceToHost, stream) == cudaSuccess && + cudaMemcpyAsync(lane.hostDistances + resultOffset, lane.distances + resultOffset, + n * sizeof(float), cudaMemcpyDeviceToHost, stream) == cudaSuccess) + job->status = ANN_BACKEND_ERROR; /* queued; settled after sync */ + else + { + const char *message = cuvsGetLastErrorText(); + + fprintf(stderr, "cuVS gathered search: %s\n", message ? message : "failed"); + } + + /* An index references its dataset; destroy it after the sync. */ + indexes[i] = index; + resultOffset += n; + } + synced = cuvsStreamSync(lane.resources) == CUVS_SUCCESS; + resultOffset = 0; + for (uint32_t i = 0; i < count; i++) + { + uint32_t n = Candidates(&jobs[i]); + + if (indexes[i]) + (void) cuvsBruteForceIndexDestroy(indexes[i]); + if (n == 0) + continue; + if (!(synced && jobs[i].status == ANN_BACKEND_ERROR && + FinishJob(&jobs[i], lane.hostNeighbors + resultOffset, + lane.hostDistances + resultOffset, n))) + jobs[i].status = ANN_PENDING; + else + atomic_fetch_add(&deviceJobs, 1); + resultOffset += n; + } +cpu: + free(indexes); + for (uint32_t i = 0; i < count; i++) + if (jobs[i].status != ANN_OK) + AnnGatheredExact(&jobs[i]); +} + +static DLManagedTensor +Tensor(void *data, DLDeviceType deviceType, int device, + uint8_t code, int dimensions, int64_t *shape) +{ + DLManagedTensor tensor = {0}; + + tensor.dl_tensor.data = data; + tensor.dl_tensor.device.device_type = deviceType; + tensor.dl_tensor.device.device_id = device; + tensor.dl_tensor.ndim = dimensions; + tensor.dl_tensor.dtype = (DLDataType) + { + code, 32, 1 + }; + tensor.dl_tensor.shape = shape; + return tensor; +} + +static void +Destroy(AnnIndex *index) +{ + AnnCuvsState *state = index->backend_state; + + if (state) + { + if (state->resources) + (void) cuvsStreamSync(state->resources); + if (state->graph) + (void) cuvsCagraIndexDestroy(state->graph); + if (state->dataset) + (void) cuvsDatasetDestroy(state->dataset); + if (state->search) + (void) cuvsCagraSearchParamsDestroy(state->search); + if (state->build) + (void) cuvsCagraIndexParamsDestroy(state->build); + if (state->resources) + (void) cuvsResourcesDestroy(state->resources); + free(state); + } + AnnCpuBackend()->destroy(index); +} + +static AnnStatus +Build(AnnIndex *index, uint64_t generation, uint32_t rows, + uint32_t dimensions, const float *vectors) +{ + AnnIndex next = {0}; + AnnCuvsState *state; + AnnStatus status; + int64_t shape[2] = {rows, dimensions}; + DLManagedTensor source; + uint16_t major; + uint16_t minor; + uint16_t patch; + + /* Conservative initial envelope for the chosen graph degrees. */ + if (rows < 128) + return ANN_UNSUPPORTED; + status = AnnCpuBackend()->build(&next, generation, rows, dimensions, vectors); + if (status != ANN_OK) + return status; + state = calloc(1, sizeof(*state)); + if (state == NULL) + { + AnnCpuBackend()->destroy(&next); + return ANN_NOMEM; + } + next.backend_state = state; + status = ANN_BACKEND_ERROR; + cuvsSetLogLevel(CUVS_LOG_LEVEL_OFF); + CuvsCall(cuvsVersionGet(&major, &minor, &patch)); + if (major != 26 || minor != 10) + { + fprintf(stderr, "cuVS: expected 26.10, found %u.%u.%u\n", major, minor, patch); + status = ANN_UNSUPPORTED; + goto done; + } + CuvsCall(cuvsResourcesCreate(&state->resources)); + CuvsCall(cuvsDeviceIdGet(state->resources, &state->device)); + CuvsCall(cuvsCagraIndexParamsCreate(&state->build)); + CuvsCall(cuvsCagraSearchParamsCreate(&state->search)); + CuvsCall(cuvsCagraIndexCreate(&state->graph)); + state->build->metric = L2Expanded; + state->build->graph_degree = 32; + state->build->intermediate_graph_degree = 64; + state->search->max_queries = 1; + state->search->persistent = false; + source = Tensor(next.vectors, kDLCPU, 0, kDLFloat, 2, shape); + CuvsCall(cuvsDatasetMakePadded(state->resources, &source, + CUVS_DATASET_MEM_TYPE_DEVICE, &state->dataset)); + CuvsCall(cuvsCagraBuild(state->resources, state->build, state->dataset, state->graph)); + CuvsCall(cuvsStreamSync(state->resources)); + /* Publish only after all construction work has completed successfully. */ + Destroy(index); + *index = next; + memset(&next, 0, sizeof(next)); + status = ANN_OK; +done: + Destroy(&next); + return status; +} + +static int +IdCompare(const void *left, const void *right) +{ + uint32_t a = *(const uint32_t *) left; + uint32_t b = *(const uint32_t *) right; + + return a < b ? -1 : a > b; +} + +static int +CandidateCompare(const void *left, const void *right) +{ + const AnnCandidate *a = left; + const AnnCandidate *b = right; + + if (a->distance != b->distance) + return a->distance < b->distance ? -1 : 1; + return a->id < b->id ? -1 : a->id > b->id; +} + +static AnnStatus +Search(const AnnIndex *index, const float *query, uint32_t metric, + const uint32_t *ids, uint32_t nids, uint32_t k, + AnnCandidate *results, uint32_t *nresults) +{ + AnnCuvsState *state = index->backend_state; + float *deviceQuery = NULL; + float *deviceDistances = NULL; + uint32_t *deviceNeighbors = NULL; + uint32_t *deviceMask = NULL; + uint32_t *mask = NULL; + uint32_t *neighbors = NULL; + size_t words = ((size_t) index->rows + 31) / 32; + int64_t queryShape[2] = {1, index->dimensions}; + int64_t resultShape[2]; + int64_t maskShape[1] = {(int64_t) words}; + DLManagedTensor queryTensor; + DLManagedTensor neighborTensor; + DLManagedTensor distanceTensor; + DLManagedTensor maskTensor; + cuvsFilter filter = {0}; + AnnStatus status = ANN_BACKEND_ERROR; + + *nresults = 0; + if (metric != 0) + return ANN_UNSUPPORTED; + if (k > 1024) + return ANN_UNSUPPORTED; + for (uint32_t d = 0; d < index->dimensions; d++) + if (!isfinite(query[d])) + return ANN_INVALID; + for (uint32_t i = 0; ids && i < nids; i++) + if (ids[i] >= index->rows || (i && ids[i] <= ids[i - 1])) + return ANN_INVALID; + if (ids && k > nids) + k = nids; + if (k == 0) + return ANN_OK; + if (state == NULL) + return ANN_STALE; + resultShape[0] = 1; + resultShape[1] = k; + neighbors = malloc((size_t) k * sizeof(*neighbors)); + if (neighbors == NULL) + return ANN_NOMEM; + CudaCall(cudaMalloc((void **) &deviceQuery, index->dimensions * sizeof(float))); + CudaCall(cudaMalloc((void **) &deviceNeighbors, (size_t) k * sizeof(uint32_t))); + CudaCall(cudaMalloc((void **) &deviceDistances, (size_t) k * sizeof(float))); + CudaCall(cudaMemset(deviceNeighbors, 0xff, (size_t) k * sizeof(uint32_t))); + CudaCall(cudaMemcpy(deviceQuery, query, index->dimensions * sizeof(float), cudaMemcpyHostToDevice)); + queryTensor = Tensor(deviceQuery, kDLCUDA, state->device, kDLFloat, 2, queryShape); + neighborTensor = Tensor(deviceNeighbors, kDLCUDA, state->device, kDLUInt, 2, resultShape); + distanceTensor = Tensor(deviceDistances, kDLCUDA, state->device, kDLFloat, 2, resultShape); + filter.type = NO_FILTER; + if (ids) + { + mask = calloc(words, sizeof(*mask)); + if (mask == NULL) + { + status = ANN_NOMEM; + goto done; + } + for (uint32_t i = 0; i < nids; i++) + mask[ids[i] / 32] |= UINT32_C(1) << (ids[i] % 32); + CudaCall(cudaMalloc((void **) &deviceMask, words * sizeof(uint32_t))); + CudaCall(cudaMemcpy(deviceMask, mask, words * sizeof(uint32_t), cudaMemcpyHostToDevice)); + maskTensor = Tensor(deviceMask, kDLCUDA, state->device, kDLUInt, 1, maskShape); + filter.type = BITSET; + filter.addr = (uintptr_t) &maskTensor; + } + state->search->itopk_size = k < 128 ? 128 : ((size_t) k + 31) / 32 * 32; + CuvsCall(cuvsCagraSearch(state->resources, state->search, state->graph, + &queryTensor, &neighborTensor, &distanceTensor, filter)); + CuvsCall(cuvsStreamSync(state->resources)); + CudaCall(cudaMemcpy(neighbors, deviceNeighbors, (size_t) k * sizeof(uint32_t), cudaMemcpyDeviceToHost)); + for (uint32_t i = 0; i < k; i++) + { + uint32_t id = neighbors[i]; + double squared = 0; + + if (id == UINT32_MAX) + continue; + if (id >= index->rows || (ids && !bsearch(&id, ids, nids, sizeof(*ids), IdCompare))) + goto done; + for (uint32_t j = 0; j < *nresults; j++) + if (results[j].id == id) + goto done; + + /* + * Canonical reranking fixes score precision, not approximate + * membership. + */ + for (uint32_t d = 0; d < index->dimensions; d++) + { + double delta = (double) index->vectors[(size_t) id * index->dimensions + d] - query[d]; + + squared += delta * delta; + } + results[*nresults] = (AnnCandidate) + { + id, sqrt(squared) + }; + (*nresults)++; + } + qsort(results, *nresults, sizeof(*results), CandidateCompare); + status = ANN_OK; +done: + (void) cuvsStreamSync(state->resources); + if (deviceMask) + (void) cudaFree(deviceMask); + if (deviceDistances) + (void) cudaFree(deviceDistances); + if (deviceNeighbors) + (void) cudaFree(deviceNeighbors); + if (deviceQuery) + (void) cudaFree(deviceQuery); + free(neighbors); + free(mask); + return status; +} + +/* + * Loaded indexes: one CAGRA graph per shard file, each on GPU (shard modulo + * devices). Lanes search with their own per-device resources and buffers. + */ +#define ANN_CUVS_MAX_DEVICES 16 +#define ANN_CUVS_WORKSPACE_BYTES (64 * 1024 * 1024) + +typedef struct AnnCuvsShard +{ + int device; + cuvsResources_t resources; /* load-time owner of the index memory */ + cuvsCagraIndex_t index; + cuvsDataset_t dataset; + float *vectors; /* FP32 device copy for filtered brute force */ + cuvsBruteForceIndex_t bruteForce; +} AnnCuvsShard; + +/* + * Shard s of copy c is shards[c * nshards + s]. Without replication there is + * one copy and shard s lives on GPU s mod devices; with it, every GPU holds a + * full copy and each lane searches its home GPU's copy only. + */ +typedef struct AnnCuvsIndex +{ + uint32_t nshards; + uint32_t copies; + int fp16; + AnnCuvsShard *shards; +} AnnCuvsIndex; + +static _Thread_local int laneHome = -1; +static atomic_uint nextHome; + +typedef struct AnnCuvsDevice +{ + cuvsResources_t resources; + cuvsCagraSearchParams_t params; + void *queries; + uint32_t *neighbors; + float *distances; + uint32_t *bitset; + uint32_t *hostNeighbors; /* pinned */ + float *hostDistances; /* pinned */ + int64_t *neighbors64; /* brute-force results, device then pinned */ + int64_t *hostNeighbors64; + size_t queryBytes; + size_t resultCount; + size_t result64Count; + size_t bitsetWords; +} AnnCuvsDevice; + +static _Thread_local AnnCuvsDevice laneDevices[ANN_CUVS_MAX_DEVICES]; + +static uint16_t +FloatToHalf(float value) +{ + uint32_t bits; + uint32_t sign; + int32_t exponent; + uint32_t mantissa; + + memcpy(&bits, &value, sizeof(bits)); + sign = (bits >> 16) & 0x8000; + exponent = (int32_t) ((bits >> 23) & 0xff) - 127 + 15; + mantissa = bits & 0x7fffff; + if (exponent >= 31) + return (uint16_t) (sign | 0x7c00); + if (exponent <= 0) + { + if (exponent < -10) + return (uint16_t) sign; + mantissa |= 0x800000; + { + uint32_t shift = (uint32_t) (14 - exponent); + uint32_t half = mantissa >> shift; + uint32_t rest = mantissa & ((1U << shift) - 1); + uint32_t middle = 1U << (shift - 1); + + if (rest > middle || (rest == middle && (half & 1))) + half++; + + return (uint16_t) (sign | half); + } + } + { + uint32_t half = sign | ((uint32_t) exponent << 10) | (mantissa >> 13); + uint32_t rest = mantissa & 0x1fff; + + if (rest > 0x1000 || (rest == 0x1000 && (half & 1))) + half++; + return (uint16_t) half; + } +} + +static int +GrowDevice(void **pointer, size_t *capacity, size_t needed, size_t unit) +{ + if (needed <= *capacity) + return 1; + if (*pointer) + (void) cudaFree(*pointer); + *pointer = NULL; + *capacity = 0; + needed += needed / 2; + if (cudaMalloc(pointer, needed * unit) != cudaSuccess) + { + *pointer = NULL; + return 0; + } + *capacity = needed; + return 1; +} + +/* Lazily create this lane's state on a device and size its buffers. */ +static AnnCuvsDevice * +LaneDevice(int device, size_t queryBytes, size_t results, size_t bitsetWords) +{ + AnnCuvsDevice *lane = &laneDevices[device]; + + if (cudaSetDevice(device) != cudaSuccess) + return NULL; + + /* + * A per-lane workspace pool keeps concurrent searches from serializing on + * device-wide allocation. + */ + if (lane->resources == 0 && + (cuvsResourcesCreate(&lane->resources) != CUVS_SUCCESS || + cuvsResourcesSetWorkspacePool(lane->resources, ANN_CUVS_WORKSPACE_BYTES) != CUVS_SUCCESS || + cuvsCagraSearchParamsCreate(&lane->params) != CUVS_SUCCESS)) + return NULL; + if (!GrowDevice(&lane->queries, &lane->queryBytes, queryBytes, 1) || + !GrowDevice((void **) &lane->bitset, &lane->bitsetWords, bitsetWords, sizeof(uint32_t))) + return NULL; + if (results > lane->resultCount) + { + size_t count = results + results / 2; + + (void) cudaFree(lane->neighbors); + (void) cudaFree(lane->distances); + (void) cudaFreeHost(lane->hostNeighbors); + (void) cudaFreeHost(lane->hostDistances); + lane->neighbors = NULL; + lane->distances = NULL; + lane->hostNeighbors = NULL; + lane->hostDistances = NULL; + lane->resultCount = 0; + if (cudaMalloc((void **) &lane->neighbors, count * sizeof(uint32_t)) != cudaSuccess || + cudaMalloc((void **) &lane->distances, count * sizeof(float)) != cudaSuccess || + cudaMallocHost((void **) &lane->hostNeighbors, count * sizeof(uint32_t)) != cudaSuccess || + cudaMallocHost((void **) &lane->hostDistances, count * sizeof(float)) != cudaSuccess) + return NULL; + lane->resultCount = count; + } + return lane; +} + +typedef struct LoadArgs +{ + AnnIndexFile *file; + AnnCuvsIndex *state; + uint32_t shard; /* shard in the manifest */ + uint32_t slot; /* entry in state->shards */ + int device; + int threaded; + AnnStatus status; +} LoadArgs; + +static void * +LoadShard(void *argument) +{ + LoadArgs *args = argument; + AnnCuvsShard *shard = &args->state->shards[args->slot]; + char path[1400]; + int64_t dims = 0, + size = 0; + + args->status = ANN_BACKEND_ERROR; + shard->device = args->device; + snprintf(path, sizeof(path), "%s/%s", args->file->directory, args->file->shards[args->shard].file); + if (cudaSetDevice(shard->device) != cudaSuccess || + cuvsResourcesCreate(&shard->resources) != CUVS_SUCCESS || + cuvsCagraIndexCreate(&shard->index) != CUVS_SUCCESS || + cuvsCagraDeserializeGraphAndDataset(shard->resources, path, shard->index, + &shard->dataset) != CUVS_SUCCESS || + cuvsStreamSync(shard->resources) != CUVS_SUCCESS) + { + const char *message = cuvsGetLastErrorText(); + + fprintf(stderr, "cuVS: cannot load %s: %s\n", path, message ? message : "failed"); + return NULL; + } + if (cuvsCagraIndexGetDims(shard->index, &dims) != CUVS_SUCCESS || + cuvsCagraIndexGetSize(shard->index, &size) != CUVS_SUCCESS || + dims != args->file->dimensions || size != args->file->shards[args->shard].rows) + { + fprintf(stderr, "cuVS: %s has %lld rows x %lld dims, manifest says %u x %u\n", path, + (long long) size, (long long) dims, args->file->shards[args->shard].rows, + args->file->dimensions); + args->status = ANN_INVALID; + return NULL; + } + + /* + * Selective filters run exact brute force over an FP32 device copy of the + * shard, masked by the eligibility bitset; build its norms once here. + */ + if (AnnIndexConfig.filtered_bruteforce) + { + size_t count = (size_t) args->file->shards[args->shard].rows * args->file->dimensions; + int64_t shape[2] = {args->file->shards[args->shard].rows, args->file->dimensions}; + DLManagedTensor data; + + if (cudaMalloc((void **) &shard->vectors, count * sizeof(float)) != cudaSuccess || + cudaMemcpy(shard->vectors, args->file->vectors + args->file->shards[args->shard].first * args->file->dimensions, + count * sizeof(float), cudaMemcpyHostToDevice) != cudaSuccess || + cuvsBruteForceIndexCreate(&shard->bruteForce) != CUVS_SUCCESS || + (data = Tensor(shard->vectors, kDLCUDA, shard->device, kDLFloat, 2, shape), + cuvsBruteForceBuild(shard->resources, &data, L2Expanded, 0, shard->bruteForce)) != CUVS_SUCCESS || + cuvsStreamSync(shard->resources) != CUVS_SUCCESS) + { + fprintf(stderr, "cuVS: cannot build brute-force copy of %s\n", path); + return NULL; + } + } + args->status = ANN_OK; + return NULL; +} + +static void +DestroyIndexState(AnnCuvsIndex *state) +{ + if (state == NULL) + return; + for (uint32_t s = 0; state->shards && s < state->nshards * state->copies; s++) + { + AnnCuvsShard *shard = &state->shards[s]; + + (void) cudaSetDevice(shard->device); + if (shard->bruteForce) + (void) cuvsBruteForceIndexDestroy(shard->bruteForce); + if (shard->vectors) + (void) cudaFree(shard->vectors); + if (shard->index) + (void) cuvsCagraIndexDestroy(shard->index); + if (shard->dataset) + (void) cuvsDatasetDestroy(shard->dataset); + if (shard->resources) + (void) cuvsResourcesDestroy(shard->resources); + } + free(state->shards); + free(state); +} + +/* Shards load in parallel, one thread each; any failure rejects the load. */ +static AnnStatus +LoadIndex(const char *name, const char *directory) +{ + AnnIndexFile *file; + AnnCuvsIndex *state; + LoadArgs *args; + pthread_t *threads; + uint32_t copies; + uint32_t total; + int devices = 0; + AnnStatus status = AnnIndexFileOpen(name, directory, &file); + + if (status != ANN_OK) + return status; + if (file->nshards == 0 || cudaGetDeviceCount(&devices) != cudaSuccess || devices <= 0) + { + /* No CAGRA shards or devices: serve exact results from the files. */ + if ((status = AnnIndexRegister(file)) != ANN_OK) + AnnIndexFileClose(file); + return status; + } + if (devices > ANN_CUVS_MAX_DEVICES) + devices = ANN_CUVS_MAX_DEVICES; + state = calloc(1, sizeof(*state)); + copies = AnnIndexConfig.replicate ? (uint32_t) devices : 1; + total = file->nshards * copies; + args = calloc(total, sizeof(*args)); + threads = calloc(total, sizeof(*threads)); + if (state) + state->shards = calloc(total, sizeof(*state->shards)); + if (state == NULL || state->shards == NULL || args == NULL || threads == NULL) + { + if (state) + free(state->shards); + free(state); + free(args); + free(threads); + AnnIndexFileClose(file); + return ANN_NOMEM; + } + state->nshards = file->nshards; + state->copies = copies; + state->fp16 = strcmp(file->dtype, "float16") == 0; + cuvsSetLogLevel(CUVS_LOG_LEVEL_OFF); + + /* + * Blocking synchronization lets waiting lanes sleep instead of spinning a + * core each, leaving CPUs to PostgreSQL. It must precede context + * creation. Stream-ordered allocation outside lane pools avoids + * device-wide syncs. + */ + for (int d = 0; d < devices; d++) + if (cudaSetDevice(d) != cudaSuccess || + (cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync) != cudaSuccess && + cudaGetLastError() != cudaErrorSetOnActiveProcess) || + cuvsRMMAsyncMemoryResourceEnable() != CUVS_SUCCESS) + fprintf(stderr, "cuVS: asynchronous memory resource unavailable on device %d\n", d); + for (uint32_t s = 0; s < total; s++) + { + uint32_t copy = s / file->nshards; + uint32_t shard = s % file->nshards; + + args[s] = (LoadArgs) + { + file, state, shard, s, copies > 1 ? (int) copy : (int) (shard % (uint32_t) devices), + 1, ANN_BACKEND_ERROR + }; + if (pthread_create(&threads[s], NULL, LoadShard, &args[s]) != 0) + { + args[s].threaded = 0; + LoadShard(&args[s]); + } + } + for (uint32_t s = 0; s < total; s++) + { + if (args[s].threaded) + (void) pthread_join(threads[s], NULL); + if (args[s].status != ANN_OK && status == ANN_OK) + status = args[s].status; + } + free(args); + free(threads); + file->device = state; + if (status == ANN_OK) + status = AnnIndexRegister(file); + if (status != ANN_OK) + { + DestroyIndexState(state); + AnnIndexFileClose(file); + } + return status; +} + +static int +HitCompare(const void *left, const void *right) +{ + const AnnIndexHit *a = left; + const AnnIndexHit *b = right; + + if (a->distance != b->distance) + return a->distance < b->distance ? -1 : 1; + return a->key < b->key ? -1 : a->key > b->key; +} + +static float +ExactDistance(const AnnIndexFile *file, uint64_t row, const float *query) +{ + const float *vector = file->vectors + row * file->dimensions; + double sum = 0; + + for (uint32_t d = 0; d < file->dimensions; d++) + { + double delta = (double) vector[d] - query[d]; + + sum += delta * delta; + } + return (float) sqrt(sum); +} + +/* + * Turn one shard's CAGRA neighbours into a sorted hit list, re-scoring each + * with FP32 vectors so FP16 datasets still return exact distances. Rows + * outside the eligibility bitset (when given) and padding are dropped. + */ +static uint32_t +ShardHits(const AnnIndexFile *file, uint32_t s, const uint32_t *neighbors, uint32_t n, + const float *query, const uint32_t *bitset, AnnIndexHit *hits) +{ + uint32_t count = 0; + + for (uint32_t i = 0; i < n; i++) + { + uint32_t local = neighbors[i]; + + if (local >= file->shards[s].rows || + (bitset && !(bitset[local / 32] & (UINT32_C(1) << (local % 32))))) + continue; + hits[count].key = file->keys[file->shards[s].first + local]; + hits[count].distance = ExactDistance(file, file->shards[s].first + local, query); + count++; + } + qsort(hits, count, sizeof(*hits), HitCompare); + return count; +} + +static DLManagedTensor +QueryTensor(void *data, int device, int half, int64_t *shape) +{ + DLManagedTensor tensor = Tensor(data, kDLCUDA, device, kDLFloat, 2, shape); + + if (half) + tensor.dl_tensor.dtype.bits = 16; + return tensor; +} + +/* Copy queries to the device, converting to FP16 for half datasets. */ +static int +UploadQueries(AnnCuvsDevice *lane, const AnnIndexJob *const *jobs, uint32_t m, uint32_t dims, + int half, cudaStream_t stream, void *host) +{ + size_t unit = half ? 2 : 4; + + for (uint32_t j = 0; j < m; j++) + for (uint32_t d = 0; d < dims; d++) + { + if (half) + ((uint16_t *) host)[(size_t) j * dims + d] = FloatToHalf(jobs[j]->query[d]); + else + ((float *) host)[(size_t) j * dims + d] = jobs[j]->query[d]; + } + return cudaMemcpyAsync(lane->queries, host, (size_t) m * dims * unit, cudaMemcpyHostToDevice, + stream) == cudaSuccess; +} + +static uint32_t +RoundTopK(uint32_t value) +{ + value = (value + 31) / 32 * 32; + return value < 32 ? 32 : value; +} + +/* + * Search every shard for a set of queries, filtered by one bitset per shard + * when given (single query). Returns 0 on any device error. + */ +static int +SearchShards(const AnnIndexFile *file, AnnCuvsIndex *state, const AnnIndexJob *const *jobs, + uint32_t m, uint32_t k, uint32_t itopk, uint32_t **bitsets, const int *useShard, + AnnIndexHit *lists, uint32_t *counts) +{ + uint32_t dims = file->dimensions; + size_t unit = state->fp16 ? 2 : 4; + void *host = malloc((size_t) m * dims * unit); + int uploaded[ANN_CUVS_MAX_DEVICES] = {0}; + size_t slot[ANN_INDEX_MAX_SHARDS]; + size_t bitsetSlot[ANN_INDEX_MAX_SHARDS]; + size_t perDevice[ANN_CUVS_MAX_DEVICES] = {0}; + size_t bitsetWords[ANN_CUVS_MAX_DEVICES] = {0}; + AnnCuvsShard *shards; + int ok = host != NULL; + + if (laneHome < 0) + laneHome = (int) (atomic_fetch_add(&nextHome, 1) % ANN_CUVS_MAX_DEVICES); + shards = state->shards + (size_t) (laneHome % state->copies) * state->nshards; + + /* + * Each shard gets m * k result slots, and its own bitset slot, in its + * device's buffers, so no shard waits for another before launching. + */ + for (uint32_t s = 0; s < state->nshards; s++) + { + int device = shards[s].device; + + slot[s] = perDevice[device]; + bitsetSlot[s] = bitsetWords[device]; + if (!useShard[s]) + continue; + perDevice[device] += (size_t) m * k; + if (bitsets) + bitsetWords[device] += ((size_t) file->shards[s].rows + 31) / 32; + } + for (uint32_t s = 0; ok && s < state->nshards; s++) + { + AnnCuvsShard *shard = &shards[s]; + size_t words = ((size_t) file->shards[s].rows + 31) / 32; + AnnCuvsDevice *lane; + cudaStream_t stream; + int64_t queryShape[2] = {m, dims}; + int64_t resultShape[2] = {m, k}; + int64_t bitsetShape[1] = {(int64_t) words}; + DLManagedTensor queries, + neighbors, + distances, + mask; + cuvsFilter filter = {0, NO_FILTER}; + + if (!useShard[s]) + continue; + lane = LaneDevice(shard->device, (size_t) m * dims * unit, perDevice[shard->device], + bitsets ? bitsetWords[shard->device] : 1); + if (lane == NULL || cuvsStreamGet(lane->resources, &stream) != CUVS_SUCCESS) + { + ok = 0; + break; + } + if (!uploaded[shard->device] && + !(uploaded[shard->device] = UploadQueries(lane, jobs, m, dims, state->fp16, stream, host))) + { + ok = 0; + break; + } + lane->params->itopk_size = itopk; + lane->params->persistent = AnnIndexConfig.persistent != 0; + queries = QueryTensor(lane->queries, shard->device, state->fp16, queryShape); + neighbors = Tensor(lane->neighbors + slot[s], kDLCUDA, shard->device, kDLUInt, 2, resultShape); + distances = Tensor(lane->distances + slot[s], kDLCUDA, shard->device, kDLFloat, 2, resultShape); + if (bitsets) + { + if (cudaMemcpyAsync(lane->bitset + bitsetSlot[s], bitsets[s], words * sizeof(uint32_t), + cudaMemcpyHostToDevice, stream) != cudaSuccess) + { + ok = 0; + break; + } + mask = Tensor(lane->bitset + bitsetSlot[s], kDLCUDA, shard->device, kDLUInt, 1, bitsetShape); + filter.type = BITSET; + filter.addr = (uintptr_t) &mask; + } + if (cuvsCagraSearch(lane->resources, lane->params, shard->index, &queries, &neighbors, + &distances, filter) != CUVS_SUCCESS || + cudaMemcpyAsync(lane->hostNeighbors + slot[s], lane->neighbors + slot[s], + (size_t) m * k * sizeof(uint32_t), cudaMemcpyDeviceToHost, stream) != cudaSuccess) + { + const char *message = cuvsGetLastErrorText(); + + fprintf(stderr, "cuVS index search: %s\n", message ? message : "failed"); + ok = 0; + break; + } + } + for (int d = 0; d < ANN_CUVS_MAX_DEVICES; d++) + if (laneDevices[d].resources && perDevice[d] && + (cudaSetDevice(d) != cudaSuccess || cuvsStreamSync(laneDevices[d].resources) != CUVS_SUCCESS)) + ok = 0; + for (uint32_t s = 0; ok && s < state->nshards; s++) + for (uint32_t j = 0; j < m; j++) + { + AnnIndexHit *hits = lists + ((size_t) j * state->nshards + s) * k; + + counts[j * state->nshards + s] = useShard[s] ? + ShardHits(file, s, laneDevices[shards[s].device].hostNeighbors + slot[s] + (size_t) j * k, + k, jobs[j]->query, bitsets ? bitsets[s] : NULL, hits) : 0; + } + free(host); + return ok; +} + +/* Merge per-shard lists for query j into its job's results. */ +static void +MergeJob(AnnIndexJob *job, AnnIndexHit *lists, uint32_t *counts, uint32_t nshards, uint32_t k) +{ + AnnIndexHit *heads[ANN_INDEX_MAX_SHARDS]; + + for (uint32_t s = 0; s < nshards; s++) + heads[s] = lists + (size_t) s * k; + job->nresults = AnnIndexTreeMerge(heads, counts, nshards, job->k, job->results); + job->status = ANN_OK; +} + +/* + * Exact filtered search of one query on the GPU: brute force over each + * shard's FP32 copy, masked by its eligibility bitset. Each shard returns up + * to n candidates, re-scored and sorted like graph results. + */ +static int +SearchShardsBruteForce(const AnnIndexFile *file, AnnCuvsIndex *state, const AnnIndexJob *job, + uint32_t k, uint32_t **bitsets, const int *useShard, const uint64_t *eligible, + AnnIndexHit *lists, uint32_t *counts) +{ + uint32_t dims = file->dimensions; + size_t slot[ANN_INDEX_MAX_SHARDS]; + size_t bitsetSlot[ANN_INDEX_MAX_SHARDS]; + uint32_t wanted[ANN_INDEX_MAX_SHARDS]; + size_t perDevice[ANN_CUVS_MAX_DEVICES] = {0}; + size_t bitsetWords[ANN_CUVS_MAX_DEVICES] = {0}; + int uploaded[ANN_CUVS_MAX_DEVICES] = {0}; + AnnCuvsShard *shards; + int ok = 1; + + if (laneHome < 0) + laneHome = (int) (atomic_fetch_add(&nextHome, 1) % ANN_CUVS_MAX_DEVICES); + shards = state->shards + (size_t) (laneHome % state->copies) * state->nshards; + for (uint32_t s = 0; s < state->nshards; s++) + { + int device = shards[s].device; + + wanted[s] = (uint32_t) ((uint64_t) k + 32 < eligible[s] ? (uint64_t) k + 32 : eligible[s]); + slot[s] = perDevice[device]; + bitsetSlot[s] = bitsetWords[device]; + if (!useShard[s]) + continue; + perDevice[device] += wanted[s]; + bitsetWords[device] += ((size_t) file->shards[s].rows + 31) / 32; + } + for (uint32_t s = 0; ok && s < state->nshards; s++) + { + AnnCuvsShard *shard = &shards[s]; + size_t words = ((size_t) file->shards[s].rows + 31) / 32; + int64_t queryShape[2] = {1, dims}; + int64_t resultShape[2] = {1, wanted[s]}; + int64_t bitmapShape[1] = {(int64_t) words}; + DLManagedTensor query, + neighbors, + distances, + bitmap; + cuvsFilter filter; + AnnCuvsDevice *lane; + cudaStream_t stream; + + if (!useShard[s]) + continue; + lane = LaneDevice(shard->device, dims * sizeof(float), perDevice[shard->device], + bitsetWords[shard->device]); + if (lane == NULL || cuvsStreamGet(lane->resources, &stream) != CUVS_SUCCESS || + !GrowDevice((void **) &lane->neighbors64, &lane->result64Count, perDevice[shard->device], sizeof(int64_t))) + { + ok = 0; + break; + } + if (lane->hostNeighbors64 == NULL || lane->result64Count > lane->resultCount) + { + (void) cudaFreeHost(lane->hostNeighbors64); + lane->hostNeighbors64 = NULL; + if (cudaMallocHost((void **) &lane->hostNeighbors64, lane->result64Count * sizeof(int64_t)) != cudaSuccess) + { + ok = 0; + break; + } + } + if (!uploaded[shard->device] && + !(uploaded[shard->device] = cudaMemcpyAsync(lane->queries, job->query, dims * sizeof(float), + cudaMemcpyHostToDevice, stream) == cudaSuccess)) + { + ok = 0; + break; + } + query = Tensor(lane->queries, kDLCUDA, shard->device, kDLFloat, 2, queryShape); + neighbors = Tensor(lane->neighbors64 + slot[s], kDLCUDA, shard->device, kDLInt, 2, resultShape); + neighbors.dl_tensor.dtype.bits = 64; + distances = Tensor(lane->distances + slot[s], kDLCUDA, shard->device, kDLFloat, 2, resultShape); + bitmap = Tensor(lane->bitset + bitsetSlot[s], kDLCUDA, shard->device, kDLUInt, 1, bitmapShape); + filter.addr = (uintptr_t) &bitmap; + filter.type = BITMAP; + if (cudaMemcpyAsync(lane->bitset + bitsetSlot[s], bitsets[s], words * sizeof(uint32_t), + cudaMemcpyHostToDevice, stream) != cudaSuccess || + cuvsBruteForceSearch(lane->resources, shard->bruteForce, &query, &neighbors, &distances, + filter) != CUVS_SUCCESS || + cudaMemcpyAsync(lane->hostNeighbors64 + slot[s], lane->neighbors64 + slot[s], + wanted[s] * sizeof(int64_t), cudaMemcpyDeviceToHost, stream) != cudaSuccess) + { + const char *message = cuvsGetLastErrorText(); + + fprintf(stderr, "cuVS filtered brute force: %s\n", message ? message : "failed"); + ok = 0; + } + } + for (int d = 0; d < ANN_CUVS_MAX_DEVICES; d++) + if (laneDevices[d].resources && perDevice[d] && + (cudaSetDevice(d) != cudaSuccess || cuvsStreamSync(laneDevices[d].resources) != CUVS_SUCCESS)) + ok = 0; + for (uint32_t s = 0; ok && s < state->nshards; s++) + { + const int64_t *found; + uint32_t *local; + + if (!useShard[s]) + continue; + found = laneDevices[shards[s].device].hostNeighbors64 + slot[s]; + local = malloc(sizeof(uint32_t) * (wanted[s] ? wanted[s] : 1)); + if (local == NULL) + { + ok = 0; + break; + } + for (uint32_t i = 0; i < wanted[s]; i++) + local[i] = found[i] < 0 || found[i] >= file->shards[s].rows ? UINT32_MAX : (uint32_t) found[i]; + /* Lists are k + 32 apart: a shard may return that many candidates. */ + counts[s] = ShardHits(file, s, local, wanted[s], job->query, bitsets[s], lists + (size_t) s * (k + 32)); + if (counts[s] > k) + counts[s] = k; + free(local); + } + return ok; +} + +/* + * Filtered query: shards whose eligible slice is small are scanned exactly; + * the rest run CAGRA with a bitset, with a larger internal top-k when the + * filter is selective. All per-shard lists are tree-merged. + */ +static void +SearchFiltered(const AnnIndexFile *file, AnnCuvsIndex *state, AnnIndexJob *job) +{ + uint64_t *ordinals = malloc(sizeof(uint64_t) * (job->nfilter ? job->nfilter : 1)); + uint64_t n; + uint32_t k = job->k; + uint32_t *bitsets[ANN_INDEX_MAX_SHARDS] = {0}; + int useShard[ANN_INDEX_MAX_SHARDS] = {0}; + uint32_t counts[ANN_INDEX_MAX_SHARDS] = {0}; + AnnIndexHit *lists = malloc(sizeof(AnnIndexHit) * (size_t) k * state->nshards); + const AnnIndexJob *jobs[1] = {job}; + uint64_t lo = 0; + uint64_t eligible[ANN_INDEX_MAX_SHARDS] = {0}; + uint32_t graphShards = 0; + double selectivity; + uint32_t itopk; + + job->status = ANN_NOMEM; + if (ordinals == NULL || lists == NULL) + goto done; + n = AnnIndexOrdinals(file, job->filter, job->nfilter, ordinals); + if (n <= AnnIndexConfig.exact_limit) + { + AnnIndexExact(file, job, ordinals, n); + goto done; + } + for (uint32_t s = 0; s < state->nshards; s++) + { + uint64_t first = file->shards[s].first; + uint64_t end = first + file->shards[s].rows; + uint64_t start = lo; + + while (lo < n && ordinals[lo] < end) + lo++; + eligible[s] = lo - start; + if (lo - start <= AnnIndexConfig.exact_limit / state->nshards) + { + AnnIndexJob part = *job; + + part.results = lists + (size_t) s * k; + AnnIndexExact(file, &part, ordinals + start, lo - start); + counts[s] = part.nresults; + continue; + } + bitsets[s] = calloc(((size_t) file->shards[s].rows + 31) / 32, sizeof(uint32_t)); + if (bitsets[s] == NULL) + goto done; + for (uint64_t i = start; i < lo; i++) + bitsets[s][(ordinals[i] - first) / 32] |= UINT32_C(1) << ((ordinals[i] - first) % 32); + useShard[s] = 1; + graphShards++; + } + if (graphShards) + { + uint32_t graphCounts[ANN_INDEX_MAX_SHARDS] = {0}; + AnnIndexHit *graphLists = malloc(sizeof(AnnIndexHit) * (size_t) k * state->nshards); + + selectivity = (double) n / (double) file->rows; + + /* + * Selective filters: exact GPU brute force beats a mostly-masked + * graph walk. + */ + if (AnnIndexConfig.filtered_bruteforce && state->shards[0].bruteForce && selectivity < 0.25) + { + AnnIndexHit *bfLists = malloc(sizeof(AnnIndexHit) * (size_t) (k + 32) * state->nshards); + uint32_t bfCounts[ANN_INDEX_MAX_SHARDS] = {0}; + + if (bfLists == NULL || + !SearchShardsBruteForce(file, state, job, k, bitsets, useShard, eligible, bfLists, bfCounts)) + { + free(bfLists); + job->status = ANN_BACKEND_ERROR; + goto done; + } + for (uint32_t s = 0; s < state->nshards; s++) + if (useShard[s]) + { + memcpy(lists + (size_t) s * k, bfLists + (size_t) s * (k + 32), sizeof(AnnIndexHit) * bfCounts[s]); + counts[s] = bfCounts[s]; + } + free(bfLists); + MergeJob(job, lists, counts, state->nshards, k); + job->route = ANN_ROUTE_EXACT; + goto done; + } + itopk = RoundTopK(k); + if (itopk < AnnIndexConfig.itopk) + itopk = AnnIndexConfig.itopk; + if (selectivity < 0.25) + itopk = AnnIndexConfig.filtered_itopk > itopk ? AnnIndexConfig.filtered_itopk : itopk; + if (graphLists == NULL || + !SearchShards(file, state, jobs, 1, k, itopk, bitsets, useShard, graphLists, graphCounts)) + { + free(graphLists); + job->status = ANN_BACKEND_ERROR; + goto done; + } + for (uint32_t s = 0; s < state->nshards; s++) + if (useShard[s]) + { + memcpy(lists + (size_t) s * k, graphLists + (size_t) s * k, sizeof(AnnIndexHit) * graphCounts[s]); + counts[s] = graphCounts[s]; + } + free(graphLists); + } + MergeJob(job, lists, counts, state->nshards, k); + job->route = graphShards ? ANN_ROUTE_FILTERED : ANN_ROUTE_EXACT; +done: + for (uint32_t s = 0; s < state->nshards; s++) + free(bitsets[s]); + free(ordinals); + free(lists); +} + +/* + * Unfiltered jobs of one index share each CAGRA submission: all queries go + * to every shard at once with the batch's largest k. + */ +static void +SearchUnfiltered(const AnnIndexFile *file, AnnCuvsIndex *state, AnnIndexJob **jobs, uint32_t m) +{ + uint32_t k = 0; + int useShard[ANN_INDEX_MAX_SHARDS]; + AnnIndexHit *lists; + uint32_t *counts; + uint32_t itopk; + + for (uint32_t j = 0; j < m; j++) + if (jobs[j]->k > k) + k = jobs[j]->k; + itopk = RoundTopK(k) > AnnIndexConfig.itopk ? RoundTopK(k) : AnnIndexConfig.itopk; + for (uint32_t s = 0; s < state->nshards; s++) + useShard[s] = 1; + lists = malloc(sizeof(AnnIndexHit) * (size_t) m * state->nshards * k); + counts = calloc((size_t) m * state->nshards, sizeof(uint32_t)); + if (lists == NULL || counts == NULL || + !SearchShards(file, state, (const AnnIndexJob *const *) jobs, m, k, itopk, NULL, useShard, lists, counts)) + { + for (uint32_t j = 0; j < m; j++) + jobs[j]->status = ANN_BACKEND_ERROR; + } + else + for (uint32_t j = 0; j < m; j++) + { + MergeJob(jobs[j], lists + (size_t) j * state->nshards * k, counts + (size_t) j * state->nshards, + state->nshards, k); + jobs[j]->route = ANN_ROUTE_ANN; + } + free(lists); + free(counts); +} + +static void +SearchIndex(AnnIndexJob *jobs, uint32_t count) +{ + AnnIndexJob **group = malloc(sizeof(*group) * (count ? count : 1)); + char *done = calloc(count ? count : 1, 1); + + for (uint32_t i = 0; i < count; i++) + { + AnnIndexFile *file; + AnnCuvsIndex *state; + uint32_t m = 0; + + if (done[i]) + continue; + file = AnnIndexGet(jobs[i].name); + state = file ? file->device : NULL; + for (uint32_t j = i; j < count; j++) + { + if (done[j] || strcmp(jobs[j].name, jobs[i].name) != 0) + continue; + done[j] = 1; + jobs[j].nresults = 0; + if (file == NULL) + jobs[j].status = ANN_STALE; + else if (jobs[j].dimensions != file->dimensions) + jobs[j].status = ANN_INVALID; + else if (state == NULL) + { + /* Files without CAGRA shards are served exactly. */ + uint64_t *ordinals = NULL; + uint64_t n = 0; + + if (jobs[j].filtered && (ordinals = malloc(sizeof(uint64_t) * (jobs[j].nfilter + 1)))) + n = AnnIndexOrdinals(file, jobs[j].filter, jobs[j].nfilter, ordinals); + if (jobs[j].filtered && ordinals == NULL) + jobs[j].status = ANN_NOMEM; + else + AnnIndexExact(file, &jobs[j], ordinals, n); + free(ordinals); + } + else if (jobs[j].filtered) + SearchFiltered(file, state, &jobs[j]); + else + group[m++] = &jobs[j]; + } + if (m && group && file && state) + SearchUnfiltered(file, state, group, m); + if (file) + AnnIndexPut(file); + } + free(group); + free(done); +} + +static uint64_t +DeviceJobs(void) +{ + return atomic_load(&deviceJobs); +} + +const AnnBackend * +AnnCuvsBackend(void) +{ + static const AnnBackend backend = { + "cuvs", 1, 7, "cagra", "fp32", 0, 128, 1024, Build, Search, Destroy, + SearchGathered, DeviceJobs, LoadIndex, SearchIndex + }; + + return &backend; +} diff --git a/ann/indexfile.c b/ann/indexfile.c new file mode 100644 index 000000000..d561f0bd3 --- /dev/null +++ b/ann/indexfile.c @@ -0,0 +1,416 @@ +#include "indexfile.h" + +#include <errno.h> +#include <fcntl.h> +#include <math.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <sys/mman.h> +#include <sys/stat.h> +#include <unistd.h> + +AnnIndexOptions AnnIndexConfig = {128, 512, 20000, 0, 0, 0}; + +typedef struct Entry +{ + AnnIndexFile *file; + int references; + struct Entry *next; +} Entry; + +static pthread_mutex_t registryMutex = PTHREAD_MUTEX_INITIALIZER; +static Entry *registry; + +static int +MapFile(const char *directory, const char *name, size_t expected, const void **data, size_t *size) +{ + char path[1400]; + struct stat info; + void *mapping; + int fd; + + snprintf(path, sizeof(path), "%s/%s", directory, name); + fd = open(path, O_RDONLY); + if (fd < 0) + { + fprintf(stderr, "ann index: cannot open %s: %s\n", path, strerror(errno)); + return 0; + } + if (fstat(fd, &info) != 0 || (size_t) info.st_size != expected) + { + fprintf(stderr, "ann index: %s has %lld bytes, expected %zu\n", path, + (long long) info.st_size, expected); + (void) close(fd); + return 0; + } + mapping = expected ? mmap(NULL, expected, PROT_READ, MAP_SHARED, fd, 0) : NULL; + (void) close(fd); + if (mapping == MAP_FAILED) + return 0; + *data = mapping; + *size = expected; + return 1; +} + +/* Parse "key value" lines; unknown keys are rejected so typos fail loudly. */ +static AnnStatus +ParseManifest(AnnIndexFile *file) +{ + char path[1400]; + char line[1400]; + FILE *stream; + int version = 0; + uint64_t covered = 0; + + snprintf(path, sizeof(path), "%s/manifest", file->directory); + stream = fopen(path, "r"); + if (stream == NULL) + { + fprintf(stderr, "ann index: cannot open %s: %s\n", path, strerror(errno)); + return ANN_INVALID; + } + strcpy(file->dtype, "float32"); + while (fgets(line, sizeof(line), stream)) + { + char key[64]; + unsigned long long a, + b, + c; + char text[ 256]; + + if (line[0] == '#' || line[0] == '\n' || sscanf(line, "%63s", key) != 1) + continue; + if (strcmp(key, "pgvector-ann-index") == 0 && sscanf(line, "%*s %d", &version) == 1) + continue; + if (strcmp(key, "dimensions") == 0 && sscanf(line, "%*s %llu", &a) == 1 && a > 0 && a <= ANN_MAX_DIM) + file->dimensions = (uint32_t) a; + else if (strcmp(key, "rows") == 0 && sscanf(line, "%*s %llu", &a) == 1) + file->rows = a; + else if (strcmp(key, "metric") == 0 && sscanf(line, "%*s %255s", text) == 1 && strcmp(text, "l2") == 0) + continue; + else if (strcmp(key, "dtype") == 0 && sscanf(line, "%*s %15s", file->dtype) == 1 && + (strcmp(file->dtype, "float32") == 0 || strcmp(file->dtype, "float16") == 0)) + continue; + else if (strcmp(key, "source") == 0 && sscanf(line, "%*s %255s", file->source) == 1) + continue; + else if (strcmp(key, "key") == 0 && sscanf(line, "%*s %63s", file->key) == 1) + continue; + else if (strcmp(key, "vector") == 0 && sscanf(line, "%*s %63s", file->vector) == 1) + continue; + else if (strcmp(key, "shards") == 0 && sscanf(line, "%*s %llu", &a) == 1 && a <= ANN_INDEX_MAX_SHARDS) + continue; + else if (strcmp(key, "shard") == 0 && + sscanf(line, "%*s %llu %llu %llu %255s", &a, &b, &c, text) == 4 && + a == file->nshards && file->nshards < ANN_INDEX_MAX_SHARDS && + b == covered && c > 0 && c <= UINT32_MAX && !strchr(text, '/')) + { + AnnIndexShard *shard = &file->shards[file->nshards++]; + + shard->first = b; + shard->rows = (uint32_t) c; + strcpy(shard->file, text); + covered += c; + } + else + { + fprintf(stderr, "ann index: invalid manifest line: %s", line); + (void) fclose(stream); + return ANN_INVALID; + } + } + (void) fclose(stream); + if (version != 1 || file->dimensions == 0 || file->rows == 0 || + (file->nshards && covered != file->rows)) + { + fprintf(stderr, "ann index: incomplete manifest in %s\n", file->directory); + return ANN_INVALID; + } + return ANN_OK; +} + +AnnStatus +AnnIndexFileOpen(const char *name, const char *directory, AnnIndexFile **result) +{ + AnnIndexFile *file; + AnnStatus status; + const void *data; + + if (strlen(name) == 0 || strlen(name) > ANN_INDEX_NAME_MAX || + strlen(directory) >= sizeof(file->directory)) + return ANN_INVALID; + file = calloc(1, sizeof(*file)); + if (file == NULL) + return ANN_NOMEM; + strcpy(file->name, name); + strcpy(file->directory, directory); + status = ParseManifest(file); + if (status == ANN_OK && + (!MapFile(directory, "keys.i64", file->rows * sizeof(int64_t), &data, &file->keysBytes) || + !(file->keys = data, 1) || + !MapFile(directory, "vectors.f32", file->rows * file->dimensions * sizeof(float), + &data, &file->vectorsBytes) || + !(file->vectors = data, 1))) + status = ANN_INVALID; + + /* Ordinals rely on strictly increasing keys. */ + for (uint64_t i = 1; status == ANN_OK && i < file->rows; i++) + if (file->keys[i] <= file->keys[i - 1]) + { + fprintf(stderr, "ann index: keys.i64 is not strictly increasing\n"); + status = ANN_INVALID; + } + if (status != ANN_OK) + { + AnnIndexFileClose(file); + return status; + } + *result = file; + return ANN_OK; +} + +void +AnnIndexFileClose(AnnIndexFile *file) +{ + if (file == NULL) + return; + if (file->keys) + (void) munmap((void *) file->keys, file->keysBytes); + if (file->vectors) + (void) munmap((void *) file->vectors, file->vectorsBytes); + free(file); +} + +/* Names are unique; replacing a loaded index requires a worker restart. */ +AnnStatus +AnnIndexRegister(AnnIndexFile *file) +{ + Entry *entry; + + (void) pthread_mutex_lock(&registryMutex); + for (entry = registry; entry; entry = entry->next) + if (strcmp(entry->file->name, file->name) == 0) + { + (void) pthread_mutex_unlock(&registryMutex); + return ANN_STALE; + } + entry = calloc(1, sizeof(*entry)); + if (entry == NULL) + { + (void) pthread_mutex_unlock(&registryMutex); + return ANN_NOMEM; + } + entry->file = file; + entry->references = 1; + entry->next = registry; + registry = entry; + (void) pthread_mutex_unlock(&registryMutex); + return ANN_OK; +} + +AnnIndexFile * +AnnIndexGet(const char *name) +{ + AnnIndexFile *file = NULL; + + (void) pthread_mutex_lock(&registryMutex); + for (Entry *entry = registry; entry; entry = entry->next) + if (strcmp(entry->file->name, name) == 0) + { + entry->references++; + file = entry->file; + break; + } + (void) pthread_mutex_unlock(&registryMutex); + return file; +} + +void +AnnIndexPut(AnnIndexFile *file) +{ + (void) pthread_mutex_lock(&registryMutex); + for (Entry *entry = registry; entry; entry = entry->next) + if (entry->file == file) + { + entry->references--; + break; + } + (void) pthread_mutex_unlock(&registryMutex); +} + +uint64_t +AnnIndexOrdinals(const AnnIndexFile *file, const int64_t *keys, uint32_t nkeys, uint64_t *ordinals) +{ + uint64_t count = 0; + uint64_t low = 0; + + /* + * Both lists are sorted: gallop forward from the previous position, then + * binary search the bracketed range. Dense filters touch nearby keys + * only. + */ + for (uint32_t i = 0; i < nkeys && low < file->rows; i++) + { + uint64_t high; + uint64_t step = 1; + + while (low + step < file->rows && file->keys[low + step] < keys[i]) + { + low += step; + step *= 2; + } + high = low + step < file->rows ? low + step + 1 : file->rows; + while (low < high) + { + uint64_t middle = low + (high - low) / 2; + + if (file->keys[middle] < keys[i]) + low = middle + 1; + else + high = middle; + } + if (low < file->rows && file->keys[low] == keys[i]) + ordinals[count++] = low++; + } + return count; +} + +static int +HitCompare(const AnnIndexHit *a, const AnnIndexHit *b) +{ + if (a->distance != b->distance) + return a->distance < b->distance ? -1 : 1; + return a->key < b->key ? -1 : a->key > b->key; +} + +static int +HitSortCompare(const void *a, const void *b) +{ + return HitCompare(a, b); +} + +static float +Distance(const float *a, const float *b, uint32_t dimensions) +{ + float sums[8] = {0}; + float sum = 0; + uint32_t d = 0; + + for (; d + 8 <= dimensions; d += 8) + for (int j = 0; j < 8; j++) + { + float diff = a[d + j] - b[d + j]; + + sums[j] += diff * diff; + } + for (; d < dimensions; d++) + sum += (a[d] - b[d]) * (a[d] - b[d]); + sum += ((sums[0] + sums[1]) + (sums[2] + sums[3])) + ((sums[4] + sums[5]) + (sums[6] + sums[7])); + return sqrtf(sum); +} + +/* Bounded max-heap on distance keeps the best k hits. */ +static void +Offer(AnnIndexHit *heap, uint32_t *count, uint32_t k, AnnIndexHit hit) +{ + uint32_t i; + + if (*count < k) + { + i = (*count)++; + while (i && HitCompare(&heap[(i - 1) / 2], &hit) < 0) + { + heap[i] = heap[(i - 1) / 2]; + i = (i - 1) / 2; + } + heap[i] = hit; + return; + } + if (HitCompare(&hit, &heap[0]) >= 0) + return; + i = 0; + for (;;) + { + uint32_t child = i * 2 + 1; + + if (child >= *count) + break; + if (child + 1 < *count && HitCompare(&heap[child], &heap[child + 1]) < 0) + child++; + if (HitCompare(&heap[child], &hit) <= 0) + break; + heap[i] = heap[child]; + i = child; + } + heap[i] = hit; +} + +void +AnnIndexExact(const AnnIndexFile *file, AnnIndexJob *job, const uint64_t *ordinals, uint64_t nordinals) +{ + uint64_t count = ordinals ? nordinals : file->rows; + + job->nresults = 0; + for (uint64_t i = 0; i < count && job->k; i++) + { + uint64_t row = ordinals ? ordinals[i] : i; + AnnIndexHit hit; + + hit.key = file->keys[row]; + hit.distance = Distance(file->vectors + row * file->dimensions, job->query, file->dimensions); + Offer(job->results, &job->nresults, job->k, hit); + } + qsort(job->results, job->nresults, sizeof(*job->results), HitSortCompare); + job->route = ANN_ROUTE_EXACT; + job->status = ANN_OK; +} + +static uint32_t +MergeTwo(const AnnIndexHit *a, uint32_t na, const AnnIndexHit *b, uint32_t nb, + uint32_t k, AnnIndexHit *output) +{ + uint32_t i = 0, + j = 0, + n = 0; + + while (n < k && (i < na || j < nb)) + { + if (j >= nb || (i < na && HitCompare(&a[i], &b[j]) <= 0)) + output[n++] = a[i++]; + else + output[n++] = b[j++]; + } + return n; +} + +/* + * Each round merges neighbouring lists, halving their number, as a GPU + * reduction would. Lists must each be sorted and hold distinct keys; shards + * cover disjoint key ranges, so the merged list holds distinct keys too. + * Every list buffer must hold at least k hits; merged lists are stored back. + */ +uint32_t +AnnIndexTreeMerge(AnnIndexHit **lists, uint32_t *counts, uint32_t nlists, + uint32_t k, AnnIndexHit *output) +{ + AnnIndexHit *scratch; + uint32_t n; + + if (nlists == 0) + return 0; + scratch = malloc(sizeof(*scratch) * (size_t) k * nlists); + if (scratch == NULL) + return 0; + for (uint32_t width = 1; width < nlists; width *= 2) + for (uint32_t i = 0; i + width < nlists; i += width * 2) + { + AnnIndexHit *target = scratch + (size_t) i * k; + + counts[i] = MergeTwo(lists[i], counts[i], lists[i + width], counts[i + width], k, target); + memcpy(lists[i], target, sizeof(*target) * counts[i]); + } + n = counts[0] < k ? counts[0] : k; + memcpy(output, lists[0], sizeof(*output) * n); + free(scratch); + return n; +} diff --git a/ann/indexfile.h b/ann/indexfile.h new file mode 100644 index 000000000..79cd79726 --- /dev/null +++ b/ann/indexfile.h @@ -0,0 +1,82 @@ +#ifndef ANN_INDEXFILE_H +#define ANN_INDEXFILE_H + +#include "backend.h" + +#include <pthread.h> + +/* + * Immutable on-disk index produced by scripts/ann/build_cuvs_index.py: + * + * manifest text, one "key value..." pair per line (see ann/README.md) + * keys.i64 rows sorted unique little-endian int64 table keys + * vectors.f32 rows x dimensions little-endian FP32 vectors, key order + * shard-N.cagra optional cuVS CAGRA graph and dataset for one key range + * + * Global ordinal i is the row with the i-th smallest key. Shards cover + * contiguous ordinal ranges, so key to shard mapping is a binary search. + */ +#define ANN_INDEX_MAX_SHARDS 64 +#define ANN_INDEX_NAME_MAX 63 + +typedef struct AnnIndexShard +{ + uint64_t first; /* first global ordinal */ + uint32_t rows; + char file[256]; /* CAGRA file name inside the directory */ +} AnnIndexShard; + +typedef struct AnnIndexFile +{ + char name[ANN_INDEX_NAME_MAX + 1]; + char directory[1024]; + char source[256]; /* table the index was built from, or empty */ + char key[64]; /* key column name in source, or empty */ + char vector[64]; /* vector column name in source, or empty */ + char dtype[16]; /* CAGRA dataset type: float32 or float16 */ + uint32_t dimensions; + uint64_t rows; + uint32_t nshards; + AnnIndexShard shards[ANN_INDEX_MAX_SHARDS]; + const int64_t *keys; /* mapped */ + const float *vectors; /* mapped */ + size_t keysBytes; + size_t vectorsBytes; + void *device; /* backend-owned device state */ +} AnnIndexFile; + +/* Worker-wide tuning shared by index backends; set once before serving. */ +typedef struct AnnIndexOptions +{ + uint32_t itopk; /* CAGRA internal top-k for unfiltered queries */ + uint32_t filtered_itopk; /* ceiling for filtered queries */ + uint32_t exact_limit; /* filtered queries this selective run exact */ + int persistent; /* CAGRA persistent kernels */ + int replicate; /* full shard set on every GPU */ + int filtered_bruteforce; /* exact GPU brute force for selective + * filters */ +} AnnIndexOptions; + +extern AnnIndexOptions AnnIndexConfig; + +AnnStatus AnnIndexFileOpen(const char *name, const char *directory, AnnIndexFile **result); +void AnnIndexFileClose(AnnIndexFile *file); + +/* Registry of loaded indexes; lookups take a reference released by Put. */ +AnnStatus AnnIndexRegister(AnnIndexFile *file); +AnnIndexFile *AnnIndexGet(const char *name); +void AnnIndexPut(AnnIndexFile *file); + +/* Map sorted unique keys to sorted global ordinals; returns the count found. */ +uint64_t AnnIndexOrdinals(const AnnIndexFile *file, const int64_t *keys, uint32_t nkeys, + uint64_t *ordinals); + +/* Exact search over all rows, or over sorted global ordinals when given. */ +void AnnIndexExact(const AnnIndexFile *file, AnnIndexJob *job, + const uint64_t *ordinals, uint64_t nordinals); + +/* Merge sorted hit lists pairwise until one top-k list remains; buffers hold k. */ +uint32_t AnnIndexTreeMerge(AnnIndexHit **lists, uint32_t *counts, uint32_t nlists, + uint32_t k, AnnIndexHit *output); + +#endif diff --git a/ann/mockgpu.c b/ann/mockgpu.c new file mode 100644 index 000000000..d2874b170 --- /dev/null +++ b/ann/mockgpu.c @@ -0,0 +1,58 @@ +#include "backend.h" + +#include <stdlib.h> +#include <time.h> + +/* + * Stand-in for an accelerator adapter. Each search_gathered call models one + * device submission: a fixed launch latency plus host-to-device transfer time, + * then exact CPU scoring for the results. The waits use no CPU, so batching + * amortizes launch cost the way it would on a device. It never claims real + * accelerator performance; it only exercises scheduling and pipelining. + */ +static uint64_t launchNs = 100000; +static double bytesPerNs = 12.0; + +static void +Wait(uint64_t ns) +{ + struct timespec ts = {(time_t) (ns / 1000000000), (long) (ns % 1000000000)}; + + while (ns && nanosleep(&ts, &ts) != 0) + ; +} + +static void +SearchGathered(AnnGatheredJob *jobs, uint32_t count) +{ + double bytes = 0; + + for (uint32_t i = 0; i < count; i++) + bytes += ((double) jobs[i].rows + 1) * jobs[i].dimensions * sizeof(float); + Wait(launchNs + (uint64_t) (bytes / bytesPerNs)); + for (uint32_t i = 0; i < count; i++) + AnnGatheredExact(&jobs[i]); +} + +const AnnBackend * +AnnMockGpuBackend(void) +{ + static AnnBackend backend; + + /* Called once from the worker's main thread before any service starts. */ + if (backend.name == NULL) + { + const char *launch = getenv("ANN_MOCK_GPU_LAUNCH_US"); + const char *bandwidth = getenv("ANN_MOCK_GPU_GBPS"); + + if (launch) + launchNs = strtoull(launch, NULL, 10) * 1000; + if (bandwidth && strtod(bandwidth, NULL) > 0) + bytesPerNs = strtod(bandwidth, NULL); + backend = *AnnCpuBackend(); + backend.name = "mock-gpu"; + backend.storage = "fp32-mock-device"; + backend.search_gathered = SearchGathered; + } + return &backend; +} diff --git a/ann/server.c b/ann/server.c new file mode 100644 index 000000000..8bf21d369 --- /dev/null +++ b/ann/server.c @@ -0,0 +1,1081 @@ +/* getaddrinfo and TCP keepalive options are POSIX/GNU, not strict C11. */ +#ifdef __linux__ +#define _GNU_SOURCE +#endif + +#include "server.h" +#include "indexfile.h" + +#include <errno.h> +#include <poll.h> +#include <pthread.h> +#include <signal.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <fcntl.h> +#include <sys/mman.h> +#include <netdb.h> +#include <netinet/in.h> +#include <netinet/tcp.h> +#include <sys/socket.h> +#include <sys/stat.h> +#include <sys/time.h> +#include <sys/un.h> +#include <time.h> +#include <unistd.h> + +typedef struct Connection +{ + int fd; + int references; + int broken; + int remote; /* accepted on the TCP listener */ + int authenticated; + const unsigned char *shared; /* client-written read-only vectors */ + size_t shared_size; + pthread_mutex_t mutex; /* serializes replies and guards references */ +} Connection; + +#define JOB_GATHERED 0 +#define JOB_INDEX 1 + +typedef struct Job +{ + int kind; + AnnGatheredJob job; + AnnIndexJob index; + int64_t *keys; /* sorted filter keys owned by the job */ + Connection *connection; + uint16_t operation; + uint64_t request; + unsigned char *payload; + uint64_t charge; + struct Job *next; +} Job; + +typedef struct Service +{ + const AnnBackend *backend; + AnnServeOptions options; + pthread_mutex_t mutex; + pthread_cond_t condition; + Job *head; + Job *tail; + uint64_t pending_bytes; + uint64_t queued; + uint32_t idle; + int stopping; + uint64_t jobs; + uint64_t rejected; + uint64_t failed; + uint64_t batches; + uint64_t max_batch; + uint64_t rows; +} Service; + +static volatile sig_atomic_t stopRequested; + +void +AnnServeDefaultOptions(AnnServeOptions *options) +{ + long cpus = sysconf(_SC_NPROCESSORS_ONLN); + + options->lanes = cpus > 0 ? (uint32_t) cpus : 2; + options->max_batch_jobs = 64; + options->max_batch_bytes = 32U * 1024U * 1024U; + options->max_pending_bytes = 512U * 1024U * 1024U; + options->delay_us = 0; + options->tcp = NULL; + options->token = NULL; +} + +static void +OnSignal(int signal) +{ + (void) signal; + stopRequested = 1; +} + +static uint64_t +Now(void) +{ + struct timespec ts; + + (void) clock_gettime(CLOCK_REALTIME, &ts); + return (uint64_t) ts.tv_sec * UINT64_C(1000000000) + (uint64_t) ts.tv_nsec; +} + +static int +LittleEndian(void) +{ + const uint16_t value = 1; + + return *(const unsigned char *) &value == 1; +} + +static uint64_t +Read(const unsigned char *data, unsigned int width) +{ + uint64_t value = 0; + + for (unsigned int i = 0; i < width; i++) + value |= (uint64_t) data[i] << (8 * i); + return value; +} + +static void +Write(unsigned char *data, uint64_t value, unsigned int width) +{ + for (unsigned int i = 0; i < width; i++) + data[i] = (unsigned char) (value >> (8 * i)); +} + +static int +ReadFull(int fd, unsigned char *data, size_t size) +{ + while (size) + { + ssize_t n = read(fd, data, size); + + if (n < 0 && errno == EINTR) + continue; + if (n <= 0) + return 0; + data += n; + size -= (size_t) n; + } + return 1; +} + +static int +WriteFull(int fd, const unsigned char *data, size_t size) +{ + while (size) + { + ssize_t n = write(fd, data, size); + + if (n < 0 && errno == EINTR) + continue; + if (n <= 0) + return 0; + data += n; + size -= (size_t) n; + } + return 1; +} + +static void +Release(Connection *connection) +{ + int last; + + (void) pthread_mutex_lock(&connection->mutex); + last = --connection->references == 0; + (void) pthread_mutex_unlock(&connection->mutex); + if (last) + { + (void) close(connection->fd); + if (connection->shared) + (void) munmap((void *) connection->shared, connection->shared_size); + (void) pthread_mutex_destroy(&connection->mutex); + free(connection); + } +} + +/* Replies are written whole under the connection mutex, in completion order. */ +static void +Reply(Connection *connection, uint16_t operation, uint64_t request, + AnnStatus status, const unsigned char *data, size_t size) +{ + unsigned char header[32] = {'P', 'V', 'A', 'N'}; + + Write(header + 4, 1, 2); + Write(header + 6, operation, 2); + Write(header + 8, request, 8); + Write(header + 16, 0, 8); + Write(header + 24, size, 4); + Write(header + 28, status, 4); + (void) pthread_mutex_lock(&connection->mutex); + if (!connection->broken && + (!WriteFull(connection->fd, header, sizeof(header)) || + (size && !WriteFull(connection->fd, data, size)))) + { + connection->broken = 1; + (void) shutdown(connection->fd, SHUT_RDWR); + } + (void) pthread_mutex_unlock(&connection->mutex); +} + +static void +Finish(Service *service, Job *job) +{ + unsigned char *data = NULL; + size_t size = 0; + + if (job->kind == JOB_INDEX) + { + /* u32 count, u32 route, then (i64 key, f32 distance) per hit */ + job->job.status = job->index.status; + if (job->index.status == ANN_OK) + { + size = 8 + (size_t) job->index.nresults * 12; + data = malloc(size); + if (data == NULL) + job->job.status = ANN_NOMEM; + else + { + Write(data, job->index.nresults, 4); + Write(data + 4, job->index.route, 4); + for (uint32_t i = 0; i < job->index.nresults; i++) + { + uint32_t bits; + + memcpy(&bits, &job->index.results[i].distance, sizeof(bits)); + Write(data + 8 + (size_t) i * 12, (uint64_t) job->index.results[i].key, 8); + Write(data + 16 + (size_t) i * 12, bits, 4); + } + } + } + } + else if (job->job.status == ANN_OK) + { + size = 4 + (size_t) job->job.nresults * 12; + data = malloc(size); + if (data == NULL) + job->job.status = ANN_NOMEM; + else + { + Write(data, job->job.nresults, 4); + for (uint32_t i = 0; i < job->job.nresults; i++) + { + uint64_t bits; + + memcpy(&bits, &job->job.results[i].distance, sizeof(bits)); + Write(data + 4 + (size_t) i * 12, job->job.results[i].id, 4); + Write(data + 8 + (size_t) i * 12, bits, 8); + } + } + } + Reply(job->connection, job->operation, job->request, job->job.status, + data, job->job.status == ANN_OK ? size : 0); + free(data); + (void) pthread_mutex_lock(&service->mutex); + service->pending_bytes -= job->charge; + if (job->job.status != ANN_OK) + service->failed++; + (void) pthread_mutex_unlock(&service->mutex); + Release(job->connection); + free(job->job.results); + free(job->index.results); + free(job->keys); + free(job->payload); + free(job); +} + +/* + * A lane takes the oldest job and every queued job that fits the batch + * limits, optionally waiting delay_us for more, then makes one submission. + */ +static void * +Lane(void *argument) +{ + Service *service = argument; + Job **batch = calloc(service->options.max_batch_jobs, sizeof(*batch)); + AnnGatheredJob *jobs = calloc(service->options.max_batch_jobs, sizeof(*jobs)); + AnnIndexJob *indexJobs = calloc(service->options.max_batch_jobs, sizeof(*indexJobs)); + + if (batch == NULL || jobs == NULL || indexJobs == NULL) + abort(); + for (;;) + { + uint32_t count = 0; + uint32_t limit; + uint64_t bytes = 0; + uint64_t deadline; + + (void) pthread_mutex_lock(&service->mutex); + service->idle++; + while (service->head == NULL && !service->stopping) + (void) pthread_cond_wait(&service->condition, &service->mutex); + service->idle--; + if (service->head == NULL) + { + (void) pthread_mutex_unlock(&service->mutex); + break; + } + deadline = Now() + service->options.delay_us * 1000; + + /* + * Leave a share of the queue for idle lanes instead of serializing + * it. + */ + limit = (uint32_t) ((service->queued + service->idle) / (service->idle + 1)); + if (limit > service->options.max_batch_jobs) + limit = service->options.max_batch_jobs; + for (;;) + { + Job *job = service->head; + + if (job && count < limit) + { + uint64_t size = (uint64_t) job->job.rows * job->job.dimensions * 4; + + /* One submission serves one kind of job. */ + if (count && (job->kind != batch[0]->kind || + bytes + size > service->options.max_batch_bytes)) + break; + service->head = job->next; + service->queued--; + if (service->head == NULL) + service->tail = NULL; + batch[count++] = job; + bytes += size; + continue; + } + + /* Optionally hold a partial batch only while no lane is idle. */ + if (job || count >= service->options.max_batch_jobs || service->idle || + service->stopping || Now() >= deadline) + break; + { + struct timespec ts = {(time_t) (deadline / 1000000000), + (long) (deadline % 1000000000)}; + + (void) pthread_cond_timedwait(&service->condition, &service->mutex, &ts); + } + limit = service->options.max_batch_jobs; + } + service->batches++; + service->jobs += count; + if (count > service->max_batch) + service->max_batch = count; + for (uint32_t i = 0; i < count; i++) + service->rows += batch[i]->job.rows; + (void) pthread_mutex_unlock(&service->mutex); + if (batch[0]->kind == JOB_INDEX) + { + for (uint32_t i = 0; i < count; i++) + indexJobs[i] = batch[i]->index; + service->backend->search_index(indexJobs, count); + for (uint32_t i = 0; i < count; i++) + batch[i]->index = indexJobs[i]; + } + else + { + for (uint32_t i = 0; i < count; i++) + jobs[i] = batch[i]->job; + service->backend->search_gathered(jobs, count); + for (uint32_t i = 0; i < count; i++) + batch[i]->job = jobs[i]; + } + for (uint32_t i = 0; i < count; i++) + Finish(service, batch[i]); + } + free(batch); + free(jobs); + free(indexJobs); + return NULL; +} + +/* + * Inline payload: u32 rows, dimensions, k, metric; FP32 query; row-major FP32 + * vectors. Shared payload: the same four fields, u64 byte offset into the + * attached segment, then the FP32 query; vectors stay in the segment. + */ +static AnnStatus +Submit(Service *service, Connection *connection, uint16_t operation, + uint64_t request, unsigned char *payload, uint32_t size) +{ + Job *job; + uint32_t rows, + dimensions; + uint64_t floats; + uint64_t charge; + size_t header = operation == ANN_OP_SHARED_SEARCH ? 24 : 16; + float *values = (float *) (void *) (payload + header); + const float *vectors; + + if (size < header) + return ANN_INVALID; + rows = (uint32_t) Read(payload, 4); + dimensions = (uint32_t) Read(payload + 4, 4); + floats = operation == ANN_OP_SHARED_SEARCH ? dimensions : ((uint64_t) rows + 1) * dimensions; + if (rows == 0 || dimensions == 0 || dimensions > ANN_MAX_DIM || + Read(payload + 12, 4) > 2 || floats * 4 != (uint64_t) size - header) + return ANN_INVALID; + if (service->backend->search_gathered == NULL) + return ANN_UNSUPPORTED; + if (operation == ANN_OP_SHARED_SEARCH) + { + uint64_t offset = Read(payload + 16, 8); + uint64_t bytes = (uint64_t) rows * dimensions * 4; + + /* The client must not rewrite this range until the reply arrives. */ + if (connection->shared == NULL || offset % 4 != 0 || + offset > connection->shared_size || bytes > connection->shared_size - offset) + return ANN_INVALID; + vectors = (const float *) (const void *) (connection->shared + offset); + } + else + vectors = values + dimensions; + + /* Decode in place; the payload buffer is malloc-aligned. */ + for (uint64_t i = 0; !LittleEndian() && i < floats; i++) + { + uint32_t bits = (uint32_t) Read(payload + header + i * 4, 4); + + memcpy(&values[i], &bits, sizeof(bits)); + } + if (operation == ANN_OP_SHARED_SEARCH && !LittleEndian()) + return ANN_UNSUPPORTED; + charge = (uint64_t) size + (uint64_t) rows * sizeof(AnnCandidate); + job = calloc(1, sizeof(*job)); + if (job == NULL || (job->job.results = malloc((size_t) rows * sizeof(AnnCandidate))) == NULL) + { + free(job); + return ANN_NOMEM; + } + job->job.rows = rows; + job->job.dimensions = dimensions; + job->job.k = (uint32_t) Read(payload + 8, 4); + job->job.metric = (uint32_t) Read(payload + 12, 4); + job->job.query = values; + job->job.vectors = vectors; + job->connection = connection; + job->operation = operation; + job->request = request; + job->payload = payload; + job->charge = charge; + (void) pthread_mutex_lock(&service->mutex); + if (service->pending_bytes + charge > service->options.max_pending_bytes) + { + service->rejected++; + (void) pthread_mutex_unlock(&service->mutex); + free(job->job.results); + free(job); + return ANN_OVERLOADED; + } + service->pending_bytes += charge; + (void) pthread_mutex_lock(&connection->mutex); + connection->references++; + (void) pthread_mutex_unlock(&connection->mutex); + if (service->tail) + service->tail->next = job; + else + service->head = job; + service->tail = job; + service->queued++; + (void) pthread_cond_signal(&service->condition); + (void) pthread_mutex_unlock(&service->mutex); + return ANN_PENDING; +} + +/* Map a client's POSIX shared memory segment read-only, once per connection. */ +static AnnStatus +Attach(Connection *connection, const unsigned char *name, uint32_t size) +{ + char path[64]; + struct stat info; + void *mapping; + int fd; + + if (connection->shared || size < 8 || size >= sizeof(path) || + memcmp(name, "/pgann.", 7) != 0 || memchr(name, '\0', size)) + return ANN_INVALID; + memcpy(path, name, size); + path[size] = '\0'; + if (strchr(path + 1, '/')) + return ANN_INVALID; + fd = shm_open(path, O_RDONLY, 0); + if (fd < 0) + return ANN_INVALID; + if (fstat(fd, &info) != 0 || info.st_size <= 0) + { + (void) close(fd); + return ANN_INVALID; + } + mapping = mmap(NULL, (size_t) info.st_size, PROT_READ, MAP_SHARED, fd, 0); + (void) close(fd); + if (mapping == MAP_FAILED) + return ANN_NOMEM; + connection->shared = mapping; + connection->shared_size = (size_t) info.st_size; + return ANN_OK; +} + +/* Admit a prepared job and queue it; false when over the pending budget. */ +static int +Enqueue(Service *service, Connection *connection, Job *job) +{ + (void) pthread_mutex_lock(&service->mutex); + if (service->pending_bytes + job->charge > service->options.max_pending_bytes) + { + service->rejected++; + (void) pthread_mutex_unlock(&service->mutex); + return 0; + } + service->pending_bytes += job->charge; + (void) pthread_mutex_lock(&connection->mutex); + connection->references++; + (void) pthread_mutex_unlock(&connection->mutex); + if (service->tail) + service->tail->next = job; + else + service->head = job; + service->tail = job; + service->queued++; + (void) pthread_cond_signal(&service->condition); + (void) pthread_mutex_unlock(&service->mutex); + return 1; +} + +static int +KeyCompare(const void *a, const void *b) +{ + int64_t x = *(const int64_t *) a; + int64_t y = *(const int64_t *) b; + + return x < y ? -1 : x > y; +} + +/* + * Index search payload: u32 name length, name, u32 dimensions, k, filter + * count (UINT32_MAX for none); FP32 query; i64 filter keys in any order. + */ +static AnnStatus +SubmitIndex(Service *service, Connection *connection, uint16_t operation, uint64_t request, + unsigned char *payload, uint32_t size) +{ + Job *job; + uint32_t nameLength, + dimensions, + k, + nfilter, + packedBytes = 0; + size_t offset; + uint64_t keys; + int packed = operation == ANN_OP_INDEX_SEARCH_PACKED; + + if (service->backend->search_index == NULL) + return ANN_UNSUPPORTED; + if (size < 4 || (nameLength = (uint32_t) Read(payload, 4)) == 0 || + nameLength > ANN_INDEX_NAME_MAX || (uint64_t) size < 16 + (uint64_t) nameLength) + return ANN_INVALID; + offset = 4 + nameLength; + dimensions = (uint32_t) Read(payload + offset, 4); + k = (uint32_t) Read(payload + offset + 4, 4); + nfilter = (uint32_t) Read(payload + offset + 8, 4); + offset += 12; + if (packed) + { + if ((uint64_t) size < offset + 4) + return ANN_INVALID; + packedBytes = (uint32_t) Read(payload + offset, 4); + offset += 4; + } + keys = nfilter == UINT32_MAX ? 0 : nfilter; + if (dimensions == 0 || dimensions > ANN_MAX_DIM || k == 0 || k > 10000 || + memchr(payload + 4, '\0', nameLength) || + (uint64_t) size != offset + (uint64_t) dimensions * 4 + (packed ? packedBytes : keys * 8)) + return ANN_INVALID; + job = calloc(1, sizeof(*job)); + if (job == NULL) + return ANN_NOMEM; + job->kind = JOB_INDEX; + memcpy(job->index.name, payload + 4, nameLength); + job->index.dimensions = dimensions; + job->index.k = k; + job->index.filtered = nfilter != UINT32_MAX; + job->index.query = (const float *) (const void *) payload; /* replaced below */ + job->index.results = malloc(sizeof(AnnIndexHit) * k); + job->keys = malloc(sizeof(int64_t) * (keys ? keys : 1)); + if (job->index.results == NULL || job->keys == NULL) + { + free(job->index.results); + free(job->keys); + free(job); + return ANN_NOMEM; + } + + /* Decode the query in place at an aligned offset and copy the keys. */ + { + float *query = (float *) (void *) payload; + + for (uint32_t d = 0; d < dimensions; d++) + { + uint32_t bits = (uint32_t) Read(payload + offset + (size_t) d * 4, 4); + + memcpy(&query[d], &bits, sizeof(bits)); + } + job->index.query = query; + } + offset += (size_t) dimensions * 4; + if (packed) + { + /* + * Strictly increasing keys: a zigzag varint for the first, then + * varint gaps. Dense filters cost one or two bytes per key. + */ + const unsigned char *data = payload + offset; + const unsigned char *end = data + packedBytes; + uint64_t previous = 0; + + for (uint64_t i = 0; i < keys; i++) + { + uint64_t value = 0; + int shift = 0; + + do + { + if (data == end || shift > 63) + { + free(job->index.results); + free(job->keys); + free(job); + return ANN_INVALID; + } + value |= (uint64_t) (*data & 0x7f) << shift; + shift += 7; + } while (*data++ & 0x80); + if (i == 0) + previous = (value >> 1) ^ (~(value & 1) + 1); + else if (value == 0) + { + free(job->index.results); + free(job->keys); + free(job); + return ANN_INVALID; + } + else + previous += value; + job->keys[i] = (int64_t) previous; + } + if (data != end) + { + free(job->index.results); + free(job->keys); + free(job); + return ANN_INVALID; + } + } + else if (LittleEndian()) + memcpy(job->keys, payload + offset, keys * 8); + else + for (uint64_t i = 0; i < keys; i++) + job->keys[i] = (int64_t) Read(payload + offset + i * 8, 8); + + /* Clients normally send sorted keys; sort only when they did not. */ + for (uint64_t i = 1; i < keys; i++) + if (job->keys[i] < job->keys[i - 1]) + { + qsort(job->keys, keys, sizeof(int64_t), KeyCompare); + break; + } + job->index.nfilter = 0; + for (uint64_t i = 0; i < keys; i++) + if (i == 0 || job->keys[i] != job->keys[i - 1]) + job->keys[job->index.nfilter++] = job->keys[i]; + job->index.filter = job->keys; + job->connection = connection; + job->operation = operation; + job->request = request; + job->payload = payload; + job->charge = (uint64_t) size + keys * 8 + sizeof(AnnIndexHit) * k; + if (!Enqueue(service, connection, job)) + { + free(job->index.results); + free(job->keys); + free(job); + return ANN_OVERLOADED; + } + return ANN_PENDING; +} + +/* + * Load payload: u32 name length, name, then the directory path. The reply + * is text: "rows N dimensions D shards S dtype T source NAME". + */ +static AnnStatus +LoadIndex(Service *service, const unsigned char *payload, uint32_t size, + char *reply, size_t replySize, size_t *replyLength) +{ + char name[ANN_INDEX_NAME_MAX + 1]; + char directory[1024]; + uint32_t nameLength; + AnnIndexFile *file; + AnnStatus status; + int written; + + if (service->backend->load_index == NULL) + return ANN_UNSUPPORTED; + if (size < 4 || (nameLength = (uint32_t) Read(payload, 4)) == 0 || + nameLength > ANN_INDEX_NAME_MAX || size - 4 <= nameLength || + size - 4 - nameLength >= sizeof(directory)) + return ANN_INVALID; + memcpy(name, payload + 4, nameLength); + name[nameLength] = '\0'; + memcpy(directory, payload + 4 + nameLength, size - 4 - nameLength); + directory[size - 4 - nameLength] = '\0'; + if (strlen(name) != nameLength || strlen(directory) != size - 4 - nameLength) + return ANN_INVALID; + status = service->backend->load_index(name, directory); + if (status != ANN_OK) + return status; + file = AnnIndexGet(name); + if (file == NULL) + return ANN_BACKEND_ERROR; + written = snprintf(reply, replySize, "rows %llu dimensions %u shards %u dtype %s source %s key %s vector %s", + (unsigned long long) file->rows, file->dimensions, file->nshards, + file->dtype, file->source[0] ? file->source : "-", + file->key[0] ? file->key : "-", file->vector[0] ? file->vector : "-"); + AnnIndexPut(file); + *replyLength = written > 0 && (size_t) written < replySize ? (size_t) written : 0; + return ANN_OK; +} + +/* Index info payload: the index name. Replies like LoadIndex. */ +static AnnStatus +IndexInfo (const unsigned char *payload, uint32_t size, char *reply, size_t replySize, size_t *replyLength) +{ + char name[ANN_INDEX_NAME_MAX + 1]; + AnnIndexFile *file; + int written; + + if (size == 0 || size > ANN_INDEX_NAME_MAX || memchr(payload, '\0', size)) + return ANN_INVALID; + memcpy(name, payload, size); + name[size] = '\0'; + if ((file = AnnIndexGet(name)) == NULL) + return ANN_STALE; + written = snprintf(reply, replySize, "rows %llu dimensions %u shards %u dtype %s source %s key %s vector %s", + (unsigned long long) file->rows, file->dimensions, file->nshards, + file->dtype, file->source[0] ? file->source : "-", + file->key[0] ? file->key : "-", file->vector[0] ? file->vector : "-"); + AnnIndexPut(file); + *replyLength = written > 0 && (size_t) written < replySize ? (size_t) written : 0; + return ANN_OK; +} + +/* Constant-time token comparison; lengths are not secret. */ +static int +TokenMatches(const char *token, const unsigned char *given, uint32_t size) +{ + size_t length = strlen(token); + unsigned char difference = 0; + + if (size != length) + return 0; + for (size_t i = 0; i < length; i++) + difference |= (unsigned char) token[i] ^ given[i]; + return difference == 0; +} + +typedef struct ReaderArgs +{ + Service *service; + Connection *connection; +} ReaderArgs; + +static void * +Reader(void *argument) +{ + ReaderArgs *args = argument; + Service *service = args->service; + Connection *connection = args->connection; + + free(args); + for (;;) + { + unsigned char header[32]; + unsigned char *payload; + uint16_t operation; + uint64_t request; + uint32_t length; + AnnStatus status; + + if (!ReadFull(connection->fd, header, sizeof(header)) || + memcmp(header, "PVAN", 4) != 0 || Read(header + 4, 2) != 1 || + Read(header + 28, 4) != 0 || (length = (uint32_t) Read(header + 24, 4)) > ANN_MAX_BYTES) + break; + operation = (uint16_t) Read(header + 6, 2); + request = Read(header + 8, 8); + payload = malloc(length ? length : 1); + if (payload == NULL || !ReadFull(connection->fd, payload, length)) + { + free(payload); + break; + } + if (connection->remote && !connection->authenticated) + { + /* + * The first network frame must carry the token; anything else + * closes. + */ + int ok = operation == ANN_OP_AUTH && service->options.token && + TokenMatches(service->options.token, payload, length); + + free(payload); + if (!ok) + break; + connection->authenticated = 1; + { + struct timeval none = {0, 0}; + + /* Authenticated clients may idle between queries. */ + (void) setsockopt(connection->fd, SOL_SOCKET, SO_RCVTIMEO, &none, sizeof(none)); + } + Reply(connection, operation, request, ANN_OK, NULL, 0); + continue; + } + if (connection->remote && + (operation == ANN_OP_ATTACH_SHARED || operation == ANN_OP_SHARED_SEARCH)) + { + /* Shared memory only works on the same host. */ + free(payload); + Reply(connection, operation, request, ANN_UNSUPPORTED, NULL, 0); + continue; + } + if (operation == ANN_OP_GATHERED_SEARCH || operation == ANN_OP_SHARED_SEARCH) + { + status = Submit(service, connection, operation, request, payload, length); + if (status == ANN_PENDING) + continue; /* the lane owns the payload */ + Reply(connection, operation, request, status, NULL, 0); + } + else if (operation == 0 || operation == ANN_OP_STATS) + { + char text[ 512]; + int size; + + (void) pthread_mutex_lock(&service->mutex); + if (operation == 0) + size = snprintf(text, sizeof(text), + "{\"backend\":\"%s\",\"mode\":\"server\",\"operations\":[0,5,7,8,9,10,11,12,13,14,15]," + "\"metrics_mask\":%u,\"exact\":%s,\"lanes\":%u,\"max_batch_jobs\":%u," + "\"max_frame_bytes\":%u}", + service->backend->name, service->backend->metrics, + service->backend->exact ? "true" : "false", + service->options.lanes, service->options.max_batch_jobs, ANN_MAX_BYTES); + else + size = snprintf(text, sizeof(text), + "{\"jobs\":%llu,\"batches\":%llu,\"max_batch\":%llu,\"rows\":%llu," + "\"rejected\":%llu,\"failed\":%llu,\"pending_bytes\":%llu,\"device_jobs\":%llu}", + (unsigned long long) service->jobs, (unsigned long long) service->batches, + (unsigned long long) service->max_batch, (unsigned long long) service->rows, + (unsigned long long) service->rejected, (unsigned long long) service->failed, + (unsigned long long) service->pending_bytes, + (unsigned long long) (service->backend->device_jobs ? + service->backend->device_jobs() : 0)); + (void) pthread_mutex_unlock(&service->mutex); + Reply(connection, operation, request, ANN_OK, (unsigned char *) text, + size > 0 && (size_t) size < sizeof(text) ? (size_t) size : 0); + } + else if (operation == ANN_OP_INDEX_SEARCH || operation == ANN_OP_INDEX_SEARCH_PACKED) + { + status = SubmitIndex(service, connection, operation, request, payload, length); + if (status == ANN_PENDING) + continue; + Reply(connection, operation, request, status, NULL, 0); + } + else if (operation == ANN_OP_LOAD_INDEX) + { + char text[ 512]; + size_t textLength = 0; + + /* Loading blocks only this connection's reader. */ + status = LoadIndex(service, payload, length, text, sizeof(text), &textLength); + Reply(connection, operation, request, status, (unsigned char *) text, + status == ANN_OK ? textLength : 0); + } + else if (operation == ANN_OP_INDEX_INFO) + { + char text[ 512]; + size_t textLength = 0; + + status = IndexInfo (payload, length, text, sizeof(text), &textLength); + + Reply(connection, operation, request, status, (unsigned char *) text, + status == ANN_OK ? textLength : 0); + } + else if (operation == ANN_OP_AUTH) + Reply(connection, operation, request, ANN_OK, NULL, 0); + else if (operation == ANN_OP_ATTACH_SHARED) + Reply(connection, operation, request, Attach(connection, payload, length), NULL, 0); + else if (operation == 5) + Reply(connection, operation, request, ANN_OK, NULL, 0); + else + Reply(connection, operation, request, ANN_UNSUPPORTED, NULL, 0); + free(payload); + } + (void) pthread_mutex_lock(&connection->mutex); + connection->broken = 1; + (void) shutdown(connection->fd, SHUT_RDWR); + (void) pthread_mutex_unlock(&connection->mutex); + Release(connection); + return NULL; +} + +int +AnnServe(const char *path, const AnnBackend *backend, const AnnServeOptions *options) +{ + Service service = {0}; + struct sockaddr_un address = {0}; + struct sigaction action = {0}; + pthread_t *lanes; + int listener; + int network = -1; + + if (options->lanes == 0 || options->max_batch_jobs == 0 || + strlen(path) >= sizeof(address.sun_path)) + { + fprintf(stderr, "invalid server options or socket path\n"); + return 2; + } + service.backend = backend; + service.options = *options; + (void) pthread_mutex_init(&service.mutex, NULL); + (void) pthread_cond_init(&service.condition, NULL); + action.sa_handler = OnSignal; + (void) sigaction(SIGINT, &action, NULL); + (void) sigaction(SIGTERM, &action, NULL); + (void) signal(SIGPIPE, SIG_IGN); + + address.sun_family = AF_UNIX; + strcpy(address.sun_path, path); + (void) unlink(path); + listener = socket(AF_UNIX, SOCK_STREAM, 0); + if (listener < 0 || bind(listener, (struct sockaddr *) &address, sizeof(address)) != 0 || + listen(listener, 128) != 0) + { + perror("ann-worker listen"); + return 1; + } + if (options->tcp) + { + char host[256]; + const char *colon = strrchr(options->tcp, ':'); + struct addrinfo hints = {0}; + struct addrinfo *result; + int on = 1; + + if (colon == NULL || colon == options->tcp || (size_t) (colon - options->tcp) >= sizeof(host) || + options->token == NULL || strlen(options->token) < 16) + { + fprintf(stderr, "--listen-tcp needs HOST:PORT and an --auth-token-file of 16+ bytes\n"); + return 2; + } + memcpy(host, options->tcp, (size_t) (colon - options->tcp)); + host[colon - options->tcp] = '\0'; + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + hints.ai_flags = AI_PASSIVE; + if (getaddrinfo(host, colon + 1, &hints, &result) != 0 || + (network = socket(result->ai_family, SOCK_STREAM, 0)) < 0 || + setsockopt(network, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) != 0 || + bind(network, result->ai_addr, result->ai_addrlen) != 0 || listen(network, 1024) != 0) + { + perror("ann-worker tcp listen"); + return 1; + } + freeaddrinfo(result); + } + lanes = calloc(options->lanes, sizeof(*lanes)); + if (lanes == NULL) + return 1; + for (uint32_t i = 0; i < options->lanes; i++) + if (pthread_create(&lanes[i], NULL, Lane, &service) != 0) + return 1; + fprintf(stderr, "ann-worker: backend %s listening on %s%s%s\n", backend->name, path, + options->tcp ? " and " : "", options->tcp ? options->tcp : ""); + while (!stopRequested) + { + struct pollfd pollers[2] = {{listener, POLLIN, 0}, {network, POLLIN, 0}}; + struct timeval timeout = {10, 0}; + int buffer = 4 * 1024 * 1024; + Connection *connection; + ReaderArgs *args; + pthread_t thread; + int fd; + + int remote; + + if (poll(pollers, network >= 0 ? 2 : 1, 200) <= 0) + continue; + remote = !(pollers[0].revents & POLLIN); + fd = accept(remote ? network : listener, NULL, NULL); + if (fd < 0) + continue; + if (remote) + { + int on = 1; + int idle = 10, + interval = 5, + count = 3; + struct timeval authDeadline = {10, 0}; + + /* Small request/reply frames: disable Nagle batching. */ + (void) setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &on, sizeof(on)); + + /* + * Keepalives drop connections from PostgreSQL hosts that crashed + * or vanished, releasing their reader and queued replies. + */ + (void) setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &on, sizeof(on)); +#ifdef TCP_KEEPIDLE + (void) setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &idle, sizeof(idle)); +#elif defined(TCP_KEEPALIVE) + (void) setsockopt(fd, IPPROTO_TCP, TCP_KEEPALIVE, &idle, sizeof(idle)); +#endif +#ifdef TCP_KEEPINTVL + (void) setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &interval, sizeof(interval)); +#endif +#ifdef TCP_KEEPCNT + (void) setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT, &count, sizeof(count)); +#endif + /* Unauthenticated peers get a short deadline to send the token. */ + (void) setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &authDeadline, sizeof(authDeadline)); + (void) idle; + (void) interval; + (void) count; + } + /* A stalled client cannot block a lane indefinitely. */ + (void) setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &timeout, sizeof(timeout)); + (void) setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &buffer, sizeof(buffer)); + (void) setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &buffer, sizeof(buffer)); + connection = calloc(1, sizeof(*connection)); + args = malloc(sizeof(*args)); + if (connection == NULL || args == NULL) + { + free(connection); + free(args); + (void) close(fd); + continue; + } + connection->fd = fd; + connection->remote = remote; + connection->references = 1; + (void) pthread_mutex_init(&connection->mutex, NULL); + args->service = &service; + args->connection = connection; + if (pthread_create(&thread, NULL, Reader, args) != 0) + { + free(args); + Release(connection); + continue; + } + (void) pthread_detach(thread); + } + (void) close(listener); + if (network >= 0) + (void) close(network); + (void) unlink(path); + (void) pthread_mutex_lock(&service.mutex); + service.stopping = 1; + (void) pthread_cond_broadcast(&service.condition); + (void) pthread_mutex_unlock(&service.mutex); + for (uint32_t i = 0; i < options->lanes; i++) + (void) pthread_join(lanes[i], NULL); + free(lanes); + return 0; +} diff --git a/ann/server.h b/ann/server.h new file mode 100644 index 000000000..a706a180f --- /dev/null +++ b/ann/server.h @@ -0,0 +1,39 @@ +#ifndef ANN_SERVER_H +#define ANN_SERVER_H + +#include "backend.h" + +#define ANN_OP_GATHERED_SEARCH 7 +#define ANN_OP_STATS 8 +#define ANN_OP_ATTACH_SHARED 9 +#define ANN_OP_SHARED_SEARCH 10 +#define ANN_OP_LOAD_INDEX 11 +#define ANN_OP_INDEX_SEARCH 12 +#define ANN_OP_AUTH 13 +#define ANN_OP_INDEX_SEARCH_PACKED 14 +#define ANN_OP_INDEX_INFO 15 + +typedef struct AnnServeOptions +{ + uint32_t lanes; /* concurrent backend submissions */ + uint32_t max_batch_jobs; /* gathered jobs per submission */ + uint64_t max_batch_bytes; /* vector bytes per submission */ + uint64_t max_pending_bytes; /* queued and running payload bytes */ + uint64_t delay_us; /* wait for more jobs before a partial batch */ + const char *tcp; /* optional "host:port" network listener */ + const char *token; /* shared secret network clients must send */ +} AnnServeOptions; + +void AnnServeDefaultOptions(AnnServeOptions *options); + +/* + * Accept connections on a Unix socket, and optionally on TCP for remote + * PostgreSQL hosts, until SIGINT/SIGTERM. Network clients must authenticate + * with the shared token first and cannot use shared-memory operations. Each connection + * may pipeline requests; replies carry the request ID and can arrive out of + * order. Jobs from all connections share one admission budget and batcher. + */ +int AnnServe(const char *path, const AnnBackend *backend, + const AnnServeOptions *options); + +#endif diff --git a/ann/worker.c b/ann/worker.c new file mode 100644 index 000000000..111200d25 --- /dev/null +++ b/ann/worker.c @@ -0,0 +1,503 @@ +#include "backend.h" +#include "batching.h" +#include "server.h" +#include "indexfile.h" + +#include <float.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +_Static_assert(sizeof(float) == 4 && FLT_RADIX == 2 && FLT_MANT_DIG == 24, + "protocol requires IEEE binary32"); +_Static_assert(sizeof(double) == 8 && DBL_MANT_DIG == 53, + "protocol requires IEEE binary64"); + +typedef struct Cursor +{ + unsigned char *data; + size_t size; + size_t offset; + int valid; +} Cursor; + +static uint64_t +ReadInteger(Cursor *cursor, unsigned int width) +{ + uint64_t value = 0; + + if (!cursor->valid || width > cursor->size - cursor->offset) + { + cursor->valid = 0; + return 0; + } + for (unsigned int i = 0; i < width; i++) + value |= (uint64_t) cursor->data[cursor->offset++] << (8 * i); + return value; +} + +static void +WriteInteger(Cursor *cursor, uint64_t value, unsigned int width) +{ + if (!cursor->valid || width > cursor->size - cursor->offset) + { + cursor->valid = 0; + return; + } + for (unsigned int i = 0; i < width; i++) + cursor->data[cursor->offset++] = (unsigned char) (value >> (8 * i)); +} + +static float +ReadFloat(Cursor *cursor) +{ + uint32_t bits = (uint32_t) ReadInteger(cursor, 4); + float value; + + memcpy(&value, &bits, sizeof(value)); + return value; +} + +static void +WriteDouble(Cursor *cursor, double value) +{ + uint64_t bits; + + memcpy(&bits, &value, sizeof(bits)); + WriteInteger(cursor, bits, 8); +} + +static uint32_t * +ReadIds(Cursor *cursor, uint32_t rows, uint32_t *count) +{ + uint32_t *ids; + + *count = (uint32_t) ReadInteger(cursor, 4); + if (!cursor->valid || *count > rows || *count > (cursor->size - cursor->offset) / 4) + { + cursor->valid = 0; + return NULL; + } + /* Non-NULL distinguishes an empty allowlist from no filter. */ + ids = malloc(*count ? (size_t) *count * sizeof(*ids) : sizeof(*ids)); + if (ids == NULL) + { + cursor->valid = 0; + return NULL; + } + for (uint32_t i = 0; i < *count; i++) + ids[i] = (uint32_t) ReadInteger(cursor, 4); + return ids; +} + +static AnnStatus +Build(const AnnBackend *backend, AnnIndex *index, uint64_t generation, Cursor *input) +{ + uint32_t rows = (uint32_t) ReadInteger(input, 4); + uint32_t dimensions = (uint32_t) ReadInteger(input, 4); + uint64_t count = (uint64_t) rows * dimensions; + float *vectors; + AnnStatus status; + + if (!input->valid || dimensions == 0 || dimensions > ANN_MAX_DIM || + count > ANN_MAX_BYTES / 4 || count * 4 != input->size - input->offset) + return ANN_INVALID; + vectors = malloc(count ? (size_t) count * sizeof(*vectors) : 1); + if (vectors == NULL) + return ANN_NOMEM; + for (size_t i = 0; i < (size_t) count; i++) + vectors[i] = ReadFloat(input); + status = backend->build(index, generation, rows, dimensions, vectors); + free(vectors); + return status; +} + +static AnnStatus +Search(const AnnBackend *backend, const AnnIndex *index, AnnBatcher *batcher, + Cursor *input, Cursor *output) +{ + uint32_t queries = (uint32_t) ReadInteger(input, 4); + uint32_t dimensions = (uint32_t) ReadInteger(input, 4); + uint32_t k = (uint32_t) ReadInteger(input, 4); + uint32_t metric = (uint32_t) ReadInteger(input, 4); + uint32_t filter = (uint32_t) ReadInteger(input, 4); + uint32_t *shared = NULL; + uint32_t sharedCount = 0; + float *vectors = NULL; + AnnCandidate *results = NULL; + AnnRequest **requests = NULL; + AnnStatus status = ANN_INVALID; + uint64_t count = (uint64_t) queries * dimensions; + uint64_t capacity; + + if (!input->valid || queries == 0 || queries > 1024 || + dimensions != index->dimensions || filter > 2 || + count > (input->size - input->offset) / 4) + return ANN_INVALID; + if (metric > 2 || !(backend->metrics & (1U << metric)) || + !(backend->filters & (1U << filter))) + return ANN_UNSUPPORTED; + if (k > index->rows) + k = index->rows; + capacity = (uint64_t) queries * (4 + (uint64_t) k * 12); + if (capacity > ANN_MAX_BYTES) + return ANN_INVALID; + vectors = malloc((size_t) count * sizeof(*vectors)); + results = malloc(k ? (size_t) k * sizeof(*results) : sizeof(*results)); + output->data = malloc((size_t) capacity); + output->size = (size_t) capacity; + requests = calloc(queries, sizeof(*requests)); + if (vectors == NULL || results == NULL || output->data == NULL || requests == NULL) + { + status = ANN_NOMEM; + goto done; + } + for (size_t i = 0; i < (size_t) count; i++) + vectors[i] = ReadFloat(input); + if (filter == 1) + shared = ReadIds(input, index->rows, &sharedCount); + for (uint32_t i = 0; i < queries; i++) + { + uint32_t nids = sharedCount; + uint32_t *ids = shared; + uint32_t nresults; + + if (filter == 2) + ids = ReadIds(input, index->rows, &nids); + if (!input->valid) + { + if (filter == 2) + free(ids); + status = ANN_INVALID; + goto done; + } + if (batcher) + status = AnnBatchSubmit(batcher, vectors + (size_t) i * dimensions, + k, metric, ids, nids, 30000000, &requests[i]); + else + status = backend->search(index, vectors + (size_t) i * dimensions, + metric, ids, nids, k, results, &nresults); + if (filter == 2) + free(ids); + if (status != ANN_OK) + goto done; + if (batcher) + continue; + WriteInteger(output, nresults, 4); + for (uint32_t j = 0; j < nresults; j++) + { + WriteInteger(output, results[j].id, 4); + WriteDouble(output, results[j].distance); + } + } + if (!input->valid || input->offset != input->size || !output->valid) + { + status = ANN_INVALID; + goto done; + } + for (uint32_t i = 0; batcher && i < queries; i++) + { + uint32_t nresults; + const AnnCandidate *found; + + status = AnnRequestWait(requests[i], 30000000); + if (status == ANN_PENDING) + status = ANN_TIMED_OUT; + if (status != ANN_OK) + goto done; + found = AnnRequestResults(requests[i], &nresults); + WriteInteger(output, nresults, 4); + for (uint32_t j = 0; j < nresults; j++) + { + WriteInteger(output, found[j].id, 4); + WriteDouble(output, found[j].distance); + } + } +done: + for (uint32_t i = 0; requests && i < queries; i++) + if (requests[i]) + { + (void) AnnRequestCancel(requests[i]); + AnnRequestRelease(requests[i]); + } + free(requests); + free(shared); + free(vectors); + free(results); + return status; +} + +static int +Reply(uint16_t operation, uint64_t request, uint64_t generation, + AnnStatus status, const void *data, size_t size) +{ + unsigned char bytes[32] = {'P', 'V', 'A', 'N'}; + Cursor header = {bytes, sizeof(bytes), 4, 1}; + + WriteInteger(&header, 1, 2); + WriteInteger(&header, operation, 2); + WriteInteger(&header, request, 8); + WriteInteger(&header, generation, 8); + WriteInteger(&header, size, 4); + WriteInteger(&header, status, 4); + return fwrite(bytes, 1, sizeof(bytes), stdout) == sizeof(bytes) && + (size == 0 || fwrite(data, 1, size, stdout) == size) && fflush(stdout) == 0; +} + +int +main(int argc, char **argv) +{ + const AnnBackend *backend = AnnCpuBackend(); + AnnIndex index = {0}; + AnnBatcher *batcher = NULL; + AnnServeOptions serveOptions; + const char *listen = NULL; + uint64_t lastGeneration = 0; + int exitCode = 0; + + AnnServeDefaultOptions(&serveOptions); + for (int i = 1; i < argc; i += 2) + { + const char *value = i + 1 < argc ? argv[i + 1] : NULL; + + if (value == NULL) + goto usage; + if (strcmp(argv[i], "--backend") == 0) + { + if (strcmp(value, "cpu-exact") == 0) + backend = AnnCpuBackend(); + else if (strcmp(value, "mock-gpu") == 0) + backend = AnnMockGpuBackend(); +#ifdef ANN_USE_CUVS + else if (strcmp(value, "cuvs") == 0) + backend = AnnCuvsBackend(); +#endif + else + goto usage; + } + else if (strcmp(argv[i], "--listen") == 0) + listen = value; + else if (strcmp(argv[i], "--listen-tcp") == 0) + serveOptions.tcp = value; + else if (strcmp(argv[i], "--auth-token-file") == 0) + { + static char token[256]; + FILE *stream = fopen(value, "r"); + size_t length; + + if (stream == NULL || fgets(token, sizeof(token), stream) == NULL) + { + fprintf(stderr, "cannot read %s\n", value); + return 2; + } + (void) fclose(stream); + length = strcspn(token, "\r\n"); + token[length] = '\0'; + serveOptions.token = token; + } + else if (strcmp(argv[i], "--lanes") == 0) + serveOptions.lanes = (uint32_t) strtoul(value, NULL, 10); + else if (strcmp(argv[i], "--max-batch") == 0) + serveOptions.max_batch_jobs = (uint32_t) strtoul(value, NULL, 10); + else if (strcmp(argv[i], "--delay-us") == 0) + serveOptions.delay_us = strtoull(value, NULL, 10); + else if (strcmp(argv[i], "--max-pending-mb") == 0) + serveOptions.max_pending_bytes = strtoull(value, NULL, 10) * 1024 * 1024; + else if (strcmp(argv[i], "--itopk") == 0) + { + AnnIndexConfig.itopk = (uint32_t) strtoul(value, NULL, 10); + if (AnnIndexConfig.itopk > 512) + AnnIndexConfig.itopk = 512; + } + else if (strcmp(argv[i], "--filtered-itopk") == 0) + { + /* Single-CTA CAGRA search, the cuVS default, supports up to 512. */ + AnnIndexConfig.filtered_itopk = (uint32_t) strtoul(value, NULL, 10); + if (AnnIndexConfig.filtered_itopk > 512) + AnnIndexConfig.filtered_itopk = 512; + } + else if (strcmp(argv[i], "--exact-limit") == 0) + AnnIndexConfig.exact_limit = (uint32_t) strtoul(value, NULL, 10); + else if (strcmp(argv[i], "--cagra-persistent") == 0) + AnnIndexConfig.persistent = strcmp(value, "on") == 0; + else if (strcmp(argv[i], "--replicate") == 0) + AnnIndexConfig.replicate = strcmp(value, "on") == 0; + else if (strcmp(argv[i], "--filtered-bruteforce") == 0) + AnnIndexConfig.filtered_bruteforce = strcmp(value, "on") == 0; + else + goto usage; + } + if (listen) + return AnnServe(listen, backend, &serveOptions); + for (;;) + { + unsigned char bytes[32]; + Cursor header = {bytes, sizeof(bytes), 4, 1}; + Cursor input = {NULL, 0, 0, 1}; + Cursor output = {NULL, 0, 0, 1}; + size_t received = fread(bytes, 1, sizeof(bytes), stdin); + uint16_t version; + uint16_t operation; + uint64_t request; + uint64_t generation; + uint32_t length; + uint32_t requestStatus; + AnnStatus status = ANN_INVALID; + + if (received == 0 && feof(stdin)) + break; + if (received != sizeof(bytes) || memcmp(bytes, "PVAN", 4) != 0) + { + exitCode = 1; + break; + } + version = (uint16_t) ReadInteger(&header, 2); + operation = (uint16_t) ReadInteger(&header, 2); + request = ReadInteger(&header, 8); + generation = ReadInteger(&header, 8); + length = (uint32_t) ReadInteger(&header, 4); + requestStatus = (uint32_t) ReadInteger(&header, 4); + if (version != 1 || length > ANN_MAX_BYTES || requestStatus != 0) + { + exitCode = 1; + break; + } + input.data = malloc(length ? length : 1); + input.size = length; + if (input.data == NULL || fread(input.data, 1, length, stdin) != length) + { + free(input.data); + exitCode = 1; + break; + } + if (operation == 0 && length == 0) + { + char caps[512]; + int size = snprintf(caps, sizeof(caps), + "{\"backend\":\"%s\",\"algorithm\":\"%s\",\"storage\":\"%s\"," + "\"metrics_mask\":%u,\"filters_mask\":%u,\"input\":\"fp32\"," + "\"min_rows\":%u,\"max_k\":%u," + "\"exact\":%s,\"max_frame_bytes\":%u,\"max_queries\":1024}", + backend->name, backend->algorithm, backend->storage, + backend->metrics, backend->filters, backend->min_rows, backend->max_k, + backend->exact ? "true" : "false", + ANN_MAX_BYTES); + + status = ANN_OK; + if (size < 0 || (size_t) size >= sizeof(caps)) + { + exitCode = 1; + free(input.data); + break; + } + output.offset = (size_t) size; + output.data = malloc(output.offset); + if (output.data == NULL) + status = ANN_NOMEM; + else + memcpy(output.data, caps, output.offset); + } + else if (operation == 1) + { + AnnIndex next = {0}; + AnnBatcher *nextBatcher = NULL; + + status = generation <= lastGeneration ? ANN_STALE : + Build(backend, &next, generation, &input); + if (status == ANN_OK && backend == AnnCpuBackend()) + { + AnnBatchOptions options; + + AnnBatchDefaultOptions(&options); + options.max_pending_requests = 1024; + status = AnnBatchCreateBorrowed(backend, &next, &options, &nextBatcher); + } + if (status == ANN_OK) + { + if (batcher) + AnnBatchClose(batcher); + backend->destroy(&index); + index = next; + memset(&next, 0, sizeof(next)); + batcher = nextBatcher; + lastGeneration = generation; + } + backend->destroy(&next); + } + else if (generation == 0 || generation != index.generation) + status = ANN_STALE; + else if (operation == 2) + status = Search(backend, &index, batcher, &input, &output); + else if (operation == 3 && length == 0) + { + if (batcher) + AnnBatchClose(batcher); + batcher = NULL; + backend->destroy(&index); + status = ANN_OK; + } + else if (operation == 4 && length == 0) + { + size_t count = (size_t) index.rows * index.dimensions; + + output.size = 8 + count * 4; + output.data = malloc(output.size); + status = output.data ? ANN_OK : ANN_NOMEM; + if (output.data) + { + WriteInteger(&output, index.rows, 4); + WriteInteger(&output, index.dimensions, 4); + for (size_t i = 0; i < count; i++) + { + uint32_t bits; + + memcpy(&bits, &index.vectors[i], sizeof(bits)); + WriteInteger(&output, bits, 4); + } + } + } + else if (operation == 5) + { + /* Diagnostic transport probe, never a search result. */ + status = ANN_OK; + } + else if (operation == 6 && length == 0) + { + output.size = 8; + output.data = malloc(output.size); + status = output.data ? ANN_OK : ANN_NOMEM; + if (output.data) + { + WriteInteger(&output, index.rows, 4); + WriteInteger(&output, index.dimensions, 4); + } + } + else + status = ANN_UNSUPPORTED; + if (!Reply(operation, request, generation, status, + output.data, status == ANN_OK ? output.offset : 0)) + exitCode = 1; + free(input.data); + free(output.data); + if (exitCode) + break; + } + if (batcher) + AnnBatchClose(batcher); + backend->destroy(&index); + return exitCode; + +usage: +#ifdef ANN_USE_CUVS + fprintf(stderr, "usage: ann-worker [--backend cpu-exact|mock-gpu|cuvs] [--listen PATH [--listen-tcp HOST:PORT --auth-token-file FILE] " + "[--lanes N] [--max-batch N] [--delay-us N] [--max-pending-mb N]\n" + " [--itopk N] [--filtered-itopk N] [--exact-limit ROWS] [--cagra-persistent on|off] [--replicate on|off] [--filtered-bruteforce on|off]]\n" + "hipvs and fused-nvfp4 require vendor adapters\n"); +#else + fprintf(stderr, "usage: ann-worker [--backend cpu-exact|mock-gpu] [--listen PATH [--listen-tcp HOST:PORT --auth-token-file FILE] " + "[--lanes N] [--max-batch N] [--delay-us N] [--max-pending-mb N]\n" + " [--itopk N] [--filtered-itopk N] [--exact-limit ROWS] [--cagra-persistent on|off] [--replicate on|off] [--filtered-bruteforce on|off]]\n" + "cuvs, hipvs, and fused-nvfp4 require vendor adapters\n"); +#endif + return 2; +} diff --git a/benchmarks/ann/batch.c b/benchmarks/ann/batch.c new file mode 100644 index 000000000..0c979b020 --- /dev/null +++ b/benchmarks/ann/batch.c @@ -0,0 +1,165 @@ +#ifndef __APPLE__ +#define _POSIX_C_SOURCE 200809L +#endif +#include "batching.h" + +#include <inttypes.h> +#include <stdio.h> +#include <stdlib.h> +#include <time.h> + +static double +Seconds(void) +{ + struct timespec ts; + + (void) clock_gettime(CLOCK_MONOTONIC, &ts); + return ts.tv_sec + ts.tv_nsec / 1000000000.0; +} + +static uint32_t +Number(const char *text, uint32_t maximum) +{ + char *end; + unsigned long value = strtoul(text, &end, 10); + + if (!*text ||*text == '-' || *end || value > maximum) + { + fprintf(stderr, "invalid benchmark argument: %s\n", text); + exit(2); + } + return (uint32_t) value; +} + +int +main(int argc, char **argv) +{ + uint32_t rows; + uint32_t dims; + uint32_t offered; + uint32_t passing; + int sequential; + int unique; + AnnBatchOptions options; + AnnBatcher *batcher = NULL; + AnnIndex index = {0}; + AnnBatchStats stats = {0}; + AnnRequest **requests; + uint32_t *expected; + uint32_t *ids; + float *vectors; + uint64_t success = 0; + uint64_t rejected = 0; + uint64_t failed = 0; + double start; + double elapsed; + + if (argc != 10) + return 2; + rows = Number(argv[1], 1000000); + dims = Number(argv[2], ANN_MAX_DIM); + offered = Number(argv[3], 100000); + passing = Number(argv[4], rows); + AnnBatchDefaultOptions(&options); + options.max_batch_rows = Number(argv[5], 1024); + options.max_pending_requests = Number(argv[6], 100000); + options.delay_us = Number(argv[7], 60000000); + unique = (int) Number(argv[8], 1); + sequential = (int) Number(argv[9], 1); + if (!rows || !dims || !offered || (uint64_t) rows * dims > ANN_MAX_BYTES / 4) + return 2; + vectors = malloc((size_t) rows * dims * sizeof(float)); + ids = malloc((size_t) (passing ? passing : 1) * sizeof(uint32_t)); + requests = calloc(offered, sizeof(*requests)); + expected = calloc(offered, sizeof(*expected)); + if (!vectors || !ids || !requests || !expected) + return 2; + for (uint32_t i = 0; i < rows; i++) + for (uint32_t d = 0; d < dims; d++) + vectors[(size_t) i * dims + d] = (float) i + (float) d / (dims + 1); + if (sequential) + { + if (AnnCpuBackend()->build(&index, 1, rows, dims, vectors) != ANN_OK) + return 2; + } + else if (AnnBatchCreate(AnnCpuBackend(), 1, rows, dims, vectors, &options, &batcher) != ANN_OK) + return 2; + start = Seconds(); + for (uint32_t i = 0; i < offered; i++) + { + uint32_t offset = unique ? i % (rows - passing + 1) : 0; + AnnStatus status; + + /* + * An empty subset has no query row requirement; avoid the end + * ordinal. + */ + if (offset >= rows) + offset = 0; + expected[i] = offset; + for (uint32_t j = 0; j < passing; j++) + ids[j] = offset + j; + if (sequential) + { + AnnCandidate results[10]; + uint32_t count; + + status = AnnCpuBackend()->search(&index, vectors + (size_t) offset * dims, + 0, ids, passing, 10, results, &count); + if (status != ANN_OK || (passing && (!count || results[0].id != offset || results[0].distance != 0))) + failed++; + else + success++; + } + else + { + status = AnnBatchSubmit(batcher, vectors + (size_t) offset * dims, + 10, 0, ids, passing, 30000000, &requests[i]); + if (status == ANN_OVERLOADED) + rejected++; + else if (status != ANN_OK) + failed++; + } + } + for (uint32_t i = 0; !sequential && i < offered; i++) + { + if (requests[i]) + { + uint32_t count; + AnnStatus status = AnnRequestWait(requests[i], 30000000); + const AnnCandidate *results = AnnRequestResults(requests[i], &count); + + if (status != ANN_OK || (passing && (!count || results[0].id != expected[i] || results[0].distance != 0))) + failed++; + else + success++; + AnnRequestRelease(requests[i]); + } + } + if (batcher) + { + do + { + struct timespec pause = {0, 100000}; + + AnnBatchGetStats(batcher, &stats); + if (stats.pending_requests) + (void) nanosleep(&pause, NULL); + } while (stats.pending_requests); + } + elapsed = Seconds() - start; + printf("{\"implementation\":\"native-c\",\"offered\":%u,\"successful\":%" PRIu64 + ",\"rejected\":%" PRIu64 ",\"failed\":%" PRIu64 ",\"elapsed_seconds\":%.9f," + "\"successful_vector_searches_per_second\":%.3f,\"batches\":%" PRIu64 + ",\"max_observed_batch\":%" PRIu64 ",\"pending_bytes\":%" PRIu64 "}\n", + offered, success, rejected, failed, elapsed, success / elapsed, + stats.batches, stats.max_observed_batch, stats.pending_bytes); + if (batcher) + AnnBatchClose(batcher); + AnnCpuBackend()->destroy(&index); + free(vectors); + free(ids); + free(requests); + free(expected); + return failed ? 1 : 0; +} diff --git a/benchmarks/ann/batching.py b/benchmarks/ann/batching.py new file mode 100644 index 000000000..0253c3133 --- /dev/null +++ b/benchmarks/ann/batching.py @@ -0,0 +1,33 @@ +"""Test/benchmark launcher only: all scheduling and search execute in C.""" + +import argparse +from pathlib import Path +import subprocess + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--rows', type=int, default=10000) + parser.add_argument('--dimensions', type=int, default=32) + parser.add_argument('--requests', type=int, default=1000) + parser.add_argument('--batch-rows', type=int, default=64) + parser.add_argument('--batch-delay-ms', type=float, default=.1) + parser.add_argument('--pending', type=int, default=256) + parser.add_argument('--survivor-fraction', type=float, default=.01) + parser.add_argument('--mode', choices=['shared', 'unique'], default='shared') + parser.add_argument('--sequential', action='store_true') + args = parser.parse_args() + if not 0 <= args.survivor_fraction <= 1 or args.batch_delay_ms < 0: + parser.error('invalid survivor fraction or delay') + root = Path(__file__).resolve().parents[2] + subprocess.run(['make', '-C', str(root / 'ann'), 'ann-batch-bench'], + check=True, stdout=subprocess.DEVNULL) + values = [args.rows, args.dimensions, args.requests, + int(args.rows * args.survivor_fraction), args.batch_rows, + args.pending, int(args.batch_delay_ms * 1000), + int(args.mode == 'unique'), int(args.sequential)] + subprocess.run([str(root / 'ann/ann-batch-bench'), *map(str, values)], check=True) + + +if __name__ == '__main__': + main() diff --git a/benchmarks/ann/cuvs_load.py b/benchmarks/ann/cuvs_load.py new file mode 100644 index 000000000..7638c5578 --- /dev/null +++ b/benchmarks/ann/cuvs_load.py @@ -0,0 +1,216 @@ +#!/usr/bin/env python3 +"""Load test for loaded cuVS indexes served through PostgreSQL. + + # 1. synthetic table: id, tenant (0.1% each), category (1% each), emb vector(D) + python3 benchmarks/ann/cuvs_load.py generate --dsn ... --table bench10m --rows 10000000 --dimensions 128 + # 2. offline index build (one shard per GPU) + python3 scripts/ann/build_cuvs_index.py --dsn ... --table public.bench10m --vector emb --out /data/bench10m + # 3. sweep client concurrency with pgbench for ANN and hybrid queries + python3 benchmarks/ann/cuvs_load.py bench --dsn ... --table bench10m --index bench10m \\ + --path /data/bench10m --socket /tmp/ann.sock --clients 64,128,256,512 + +The bench step loads the index once, measures recall@10 against exact SQL on +sample queries, then runs pgbench per workload and client count, printing +throughput and latency percentiles. Test tooling only: the query path is C. +""" + +import argparse +import concurrent.futures +import os +import re +import subprocess +import sys +import tempfile + +import numpy as np + +WORKLOADS = { + 'ann': "SELECT id FROM ann_index_search(:idx, :v::vector, 10);", + # 1% category filter: about 100k eligible keys per query on 10M rows (filtered CAGRA). + 'hybrid-1pct': ("SELECT id FROM ann_index_search(:idx, :v::vector, 10, " + "ARRAY(SELECT id FROM {table} WHERE category = :c));"), + # 0.1% tenant filter: about 10k eligible keys (exact route). + 'hybrid-0.1pct': ("SELECT id FROM ann_index_search(:idx, :v::vector, 10, " + "ARRAY(SELECT id FROM {table} WHERE tenant = :t));"), + # ann_hybrid_search evaluates the SQL filter itself and routes by its size: + # the 1% filter goes to a GPU node, the 0.1% filter stays exact on this host. + 'sql-1pct': "SELECT id FROM ann_hybrid_search(:idx, :v::vector, 10, 'category = ' || :c);", + 'sql-0.1pct': "SELECT id FROM ann_hybrid_search(:idx, :v::vector, 10, 'tenant = ' || :t);", +} + + +CLUSTERS = 1024 +LATENT = 16 + + +def sample(rng, n, dimensions): + """Embedding-like vectors: clusters in a 16-d latent space, randomly projected, plus noise. + + Real embeddings have low intrinsic dimension; isotropic random vectors in + 128-d have no neighborhood structure and defeat every graph index. + """ + shared = np.random.default_rng(12345) + centroids = (shared.normal(size=(CLUSTERS, LATENT)) * 2).astype(np.float32) + projection = (shared.normal(size=(LATENT, dimensions)) / np.sqrt(LATENT)).astype(np.float32) + latent = centroids[rng.integers(0, CLUSTERS, n)] + rng.normal(size=(n, LATENT)).astype(np.float32) + return (latent @ projection + rng.normal(scale=0.05, size=(n, dimensions))).astype(np.float32) + + +def copy_part(dsn, table, first, rows, dimensions, seed): + import psycopg + + rng = np.random.default_rng(seed) + layout = np.dtype([('fields', '>i2'), ('l1', '>i4'), ('id', '>i8'), ('l2', '>i4'), ('tenant', '>i4'), + ('l3', '>i4'), ('category', '>i4'), ('l4', '>i4'), ('dim', '>u2'), ('unused', '>u2'), + ('vector', '>f4', (dimensions,))]) + with psycopg.connect(dsn) as db: + with db.cursor().copy(f'COPY {table} (id, tenant, category, emb) FROM STDIN (FORMAT binary)') as copy: + copy.write(b'PGCOPY\n\xff\r\n\x00' + b'\x00' * 8) + for start in range(first, first + rows, 100000): + n = min(100000, first + rows - start) + batch = np.zeros(n, dtype=layout) + ids = np.arange(start + 1, start + n + 1) + batch['fields'] = 4 + batch['l1'] = 8 + batch['id'] = ids + batch['l2'] = batch['l3'] = 4 + batch['tenant'] = ids % 1000 + batch['category'] = (ids // 7) % 100 + batch['l4'] = 4 + 4 * dimensions + batch['dim'] = dimensions + batch['vector'] = sample(rng, n, dimensions) + copy.write(batch.tobytes()) + copy.write(b'\xff\xff') + + +def generate(args): + import psycopg + + with psycopg.connect(args.dsn, autocommit=True) as db: + db.execute(f'DROP TABLE IF EXISTS {args.table}') + db.execute(f'CREATE UNLOGGED TABLE {args.table} (id bigint, tenant int, category int, ' + f'emb vector({args.dimensions}))') + parts = args.jobs + bounds = [args.rows * i // parts for i in range(parts + 1)] + with concurrent.futures.ProcessPoolExecutor(parts) as pool: + list(pool.map(copy_part, [args.dsn] * parts, [args.table] * parts, bounds[:-1], + [b - a for a, b in zip(bounds, bounds[1:])], [args.dimensions] * parts, range(parts))) + with psycopg.connect(args.dsn, autocommit=True) as db: + db.execute('SET maintenance_work_mem = \'8GB\'') + db.execute(f'ALTER TABLE {args.table} ADD PRIMARY KEY (id)') + # Covering (filter, key) indexes let eligibility arrays come from + # index-only scans; heap-scattered keys otherwise dominate hybrid cost. + db.execute(f'CREATE INDEX ON {args.table} (tenant, id)') + db.execute(f'CREATE INDEX ON {args.table} (category, id)') + db.execute(f'DROP TABLE IF EXISTS {args.table}_queries') + # Queries come from the same distribution but are not rows of the table. + db.execute(f'CREATE TABLE {args.table}_queries (q int PRIMARY KEY, v text)') + queries = sample(np.random.default_rng(999), 1000, args.dimensions) + with db.cursor().copy(f'COPY {args.table}_queries FROM STDIN') as copy: + for i, row in enumerate(queries, 1): + copy.write_row((i, '[' + ','.join(f'{x:.6g}' for x in row) + ']')) + db.execute(f'VACUUM ANALYZE {args.table}') + db.execute(f'ANALYZE {args.table}_queries') + print(f'generated {args.rows} rows in {args.table}', file=sys.stderr) + + +def psql(args, sql): + result = subprocess.run(['psql', args.dsn, '-X', '-qAt', '-v', 'ON_ERROR_STOP=1'], + input=(f"SET ann.worker_socket = '{args.socket}';\n" if args.socket else '') + sql, + capture_output=True, text=True) + if result.returncode: + raise SystemExit(result.stderr) + return result.stdout.strip() + + +def recall(args, workload): + """Mean recall@10 over sample queries against exact SQL.""" + where = {'ann': 'true', 'hybrid-1pct': 't.category = q.c', 'hybrid-0.1pct': 't.tenant = q.t', + 'sql-1pct': 't.category = q.c', 'sql-0.1pct': 't.tenant = q.t'}[workload] + fltr = {'ann': '', 'hybrid-1pct': f', ARRAY(SELECT id FROM {args.table} t WHERE {where})', + 'hybrid-0.1pct': f', ARRAY(SELECT id FROM {args.table} t WHERE {where})'}.get(workload) + if workload.startswith('sql-'): + column = 'category' if workload == 'sql-1pct' else 'tenant' + value = 'c' if column == 'category' else 't' + return float(psql(args, f""" + SELECT round(avg(cardinality(ARRAY(SELECT unnest(a) INTERSECT SELECT unnest(b)))) / 10, 3) FROM ( + SELECT ARRAY(SELECT s.id FROM ann_hybrid_search('{args.index}', q.v, 10, '{column} = ' || q.{value}) s) a, + ARRAY(SELECT t.id FROM {args.table} t WHERE {where} ORDER BY t.emb <-> q.v LIMIT 10) b + FROM (SELECT v::vector AS v, q % 100 AS c, q % 1000 AS t FROM {args.table}_queries + WHERE q <= {args.recall_queries} OFFSET 0) q) x""")) + return float(psql(args, f""" + SET max_parallel_workers_per_gather = 16; + SELECT round(avg(cardinality(ARRAY(SELECT unnest(a) INTERSECT SELECT unnest(b)))) / 10, 3) FROM ( + SELECT ARRAY(SELECT s.id FROM ann_index_search('{args.index}', q.v, 10{fltr}) s) a, + ARRAY(SELECT t.id FROM {args.table} t WHERE {where} ORDER BY t.emb <-> q.v LIMIT 10) b + FROM (SELECT v::vector AS v, q % 100 AS c, q % 1000 AS t FROM {args.table}_queries + WHERE q <= {args.recall_queries} OFFSET 0) q) x""")) + + +def bench(args): + if args.path: + print('loaded', psql(args, f"SELECT ann_index_load('{args.index}', '{args.path}')"), 'rows', flush=True) + workloads = args.workloads.split(',') + for workload in workloads: + if not args.no_recall: + print(f'{workload}: recall@10 {recall(args, workload)}', flush=True) + with tempfile.TemporaryDirectory() as directory: + for workload in workloads: + script = os.path.join(directory, f'{workload}.sql') + with open(script, 'w') as out: + out.write('\\set q random(1, 1000)\n\\set c random(0, 99)\n\\set t random(0, 999)\n') + out.write(f'SELECT v FROM {args.table}_queries WHERE q = :q \\gset\n') + out.write(WORKLOADS[workload].format(table=args.table) + '\n') + for clients in [int(c) for c in args.clients.split(',')]: + # Networked GPU nodes are configured on the server (ALTER SYSTEM). + env = dict(os.environ, PGOPTIONS=(f'-c ann.worker_socket={args.socket} ' if args.socket else '') + + args.options) + command = ['pgbench', args.dsn, '-n', '-M', 'prepared', '-c', str(clients), + '-j', str(min(clients, args.threads)), '-T', str(args.seconds), '-D', f'idx={args.index}', + '-f', script, '--log', '--log-prefix', os.path.join(directory, f'{workload}-{clients}')] + output = subprocess.run(command, capture_output=True, text=True, env=env).stdout + tps = re.search(r'tps = ([\d.]+)', output) + failed = re.search(r'number of failed transactions: (\d+)', output) + latencies = [] + for name in os.listdir(directory): + if name.startswith(f'{workload}-{clients}.'): + with open(os.path.join(directory, name)) as log: + latencies += [int(line.split()[2]) for line in log] + os.unlink(os.path.join(directory, name)) + latencies.sort() + + def pct(p): + return latencies[min(len(latencies) - 1, int(len(latencies) * p))] / 1000 if latencies else 0 + print(f'{workload:14} clients={clients:<5} qps={float(tps.group(1)) if tps else 0:9.0f} ' + f'p50={pct(0.5):7.2f}ms p95={pct(0.95):7.2f}ms p99={pct(0.99):7.2f}ms ' + f'failed={failed.group(1) if failed else "?"}', flush=True) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + sub = parser.add_subparsers(dest='command', required=True) + g = sub.add_parser('generate') + g.add_argument('--dsn', required=True) + g.add_argument('--table', required=True) + g.add_argument('--rows', type=int, default=10_000_000) + g.add_argument('--dimensions', type=int, default=128) + g.add_argument('--jobs', type=int, default=32) + b = sub.add_parser('bench') + b.add_argument('--dsn', required=True) + b.add_argument('--table', required=True) + b.add_argument('--index', required=True) + b.add_argument('--path', help='load this index directory first') + b.add_argument('--socket', default='', help='local worker socket; omit for ann.worker_address') + b.add_argument('--clients', default='64,128,256,512') + b.add_argument('--threads', type=int, default=64) + b.add_argument('--seconds', type=int, default=20) + b.add_argument('--workloads', default='ann,hybrid-1pct,hybrid-0.1pct') + b.add_argument('--no-recall', action='store_true') + b.add_argument('--recall-queries', type=int, default=20) + b.add_argument('--options', default='', help='extra PGOPTIONS, e.g. "-c ann.gpu_min_eligible=0"') + args = parser.parse_args() + generate(args) if args.command == 'generate' else bench(args) + + +if __name__ == '__main__': + main() diff --git a/benchmarks/ann/pipeline.sh b/benchmarks/ann/pipeline.sh new file mode 100755 index 000000000..8e3caafd2 --- /dev/null +++ b/benchmarks/ann/pipeline.sh @@ -0,0 +1,57 @@ +#!/bin/sh +# Compare filtered exact search scored in the backend with the pipelined +# worker route. Usage: +# ANN_PG_DSN='host=... dbname=...' benchmarks/ann/pipeline.sh [rows] [dimensions] +# The worker runs locally, so PostgreSQL must be on this host. Requires a +# superuser (ann.worker_socket is superuser-only), psql, and pgbench on PATH. +set -eu + +ROOT=$(cd "$(dirname "$0")/../.." && pwd) +ROWS=${1:-100000} +DIM=${2:-256} +CLIENTS=${ANN_BENCH_CLIENTS:-"1 4 8 16"} +SECONDS_PER_RUN=${ANN_BENCH_SECONDS:-10} +BACKEND=${ANN_BENCH_BACKEND:-mock-gpu} +: "${ANN_PG_DSN:?set ANN_PG_DSN}" + +WORK=$(mktemp -d) +SOCKET=$WORK/ann.sock +trap 'kill $WORKER 2>/dev/null || true; rm -rf "$WORK"' EXIT INT TERM + +make -s -C "$ROOT/ann" ann-worker +"$ROOT/ann/ann-worker" --backend "$BACKEND" --listen "$SOCKET" 2>"$WORK/worker.log" & +WORKER=$! +while [ ! -S "$SOCKET" ]; do sleep 0.1; done + +psql "$ANN_PG_DSN" -X -q -v ON_ERROR_STOP=1 <<EOF +CREATE EXTENSION IF NOT EXISTS vector; +DROP TABLE IF EXISTS ann_pipeline_bench, ann_pipeline_queries; +CREATE TABLE ann_pipeline_bench (id int, tenant int, emb vector($DIM)); +INSERT INTO ann_pipeline_bench SELECT i, i % 4, + (SELECT array_agg(random())::real[] FROM generate_series(1, $DIM) WHERE i > 0)::vector + FROM generate_series(1, $ROWS) i; +CREATE INDEX ON ann_pipeline_bench USING ann (emb vector_l2_ops); +CREATE INDEX ON ann_pipeline_bench (tenant); +CREATE TABLE ann_pipeline_queries AS SELECT id, emb FROM ann_pipeline_bench WHERE id <= 200; +ANALYZE ann_pipeline_bench, ann_pipeline_queries; +EOF + +cat > "$WORK/query.sql" <<'EOF' +\set q random(1, 200) +SELECT emb::text AS v FROM ann_pipeline_queries WHERE id = :q \gset +SELECT id FROM ann_pipeline_bench WHERE tenant = 1 ORDER BY emb <-> :v::vector, id LIMIT 10; +EOF + +BASE="-c ann.enable_filtered_search=on -c max_parallel_workers_per_gather=0 -c session_preload_libraries=vector" +for route in local worker; do + OPTIONS=$BASE + [ "$route" = worker ] && OPTIONS="$BASE -c ann.worker_socket=$SOCKET" + for clients in $CLIENTS; do + result=$(PGOPTIONS="$OPTIONS" pgbench "$ANN_PG_DSN" -n -M prepared -c "$clients" \ + -j "$clients" -T "$SECONDS_PER_RUN" -f "$WORK/query.sql" 2>&1) + printf '%-6s clients=%-3s %s %s\n' "$route" "$clients" \ + "$(echo "$result" | grep '^tps' | sed 's/ (without.*//')" \ + "$(echo "$result" | grep 'latency average')" + done +done +psql "$ANN_PG_DSN" -X -q -c 'DROP TABLE ann_pipeline_bench, ann_pipeline_queries' diff --git a/benchmarks/ann/postgres.py b/benchmarks/ann/postgres.py new file mode 100644 index 000000000..3e80788dc --- /dev/null +++ b/benchmarks/ann/postgres.py @@ -0,0 +1,74 @@ +"""Emit a PostgreSQL baseline fixture; execute with psql -X -v ON_ERROR_STOP=1.""" + +import argparse +import random + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--rows', type=int, default=10000) + parser.add_argument('--dimensions', type=int, default=32) + parser.add_argument('--seed', type=int, default=42) + parser.add_argument('--k', type=int, default=20) + parser.add_argument('--filtered-native', action='store_true', + help='also measure the opt-in native exact filtered path') + args = parser.parse_args() + if not 1 <= args.dimensions <= 2000 or args.rows <= 0 or args.k <= 0: + parser.error('rows/k must be positive and dimensions must be in 1..2000') + rng = random.Random(args.seed) + def vector(): + return '[' + ','.join(format(rng.random(), '.9g') for _ in range(args.dimensions)) + ']' + print('BEGIN;') + print('CREATE EXTENSION IF NOT EXISTS vector;') + if args.filtered_native: + print("LOAD 'vector';") + print('SET LOCAL gin_fuzzy_search_limit = 0;') + print(f'CREATE TEMP TABLE ann_bench (id integer PRIMARY KEY, tenant text, ' + f'tags text[], search_text tsvector, embedding vector({args.dimensions}));') + print('COPY ann_bench (id, tenant, tags, search_text, embedding) FROM STDIN;') + for i in range(args.rows): + term = rng.choice(['database', 'query', 'storage', 'network']) + tenant = rng.randrange(16) + print(f"{i}\ttenant{tenant}\t{{{term}}}\t'{term}':1\t{vector()}") + print('\\.') + print('CREATE INDEX ON ann_bench (tenant);') + print('CREATE INDEX ON ann_bench USING gin (tags);') + print('CREATE INDEX ON ann_bench USING gin (search_text);') + print('CREATE INDEX ON ann_bench USING hnsw (embedding vector_l2_ops);') + if args.filtered_native: + print('CREATE INDEX ON ann_bench USING ann (embedding vector_l2_ops);') + print('ANALYZE ann_bench;') + print('SET LOCAL hnsw.iterative_scan = strict_order;') + query = vector() + predicates = { + 'metadata': "tenant = 'tenant0' AND tags @> ARRAY['database']", + 'text': "search_text @@ to_tsquery('simple', 'database')", + 'combined': "tenant = 'tenant0' AND search_text @@ to_tsquery('simple', 'database')", + } + for label, predicate in predicates.items(): + print(f"SELECT '{label}: eligibility';") + print(f'EXPLAIN (ANALYZE, BUFFERS) SELECT id FROM ann_bench WHERE {predicate};') + print(f"SELECT '{label}: exact filtered reference';") + exact = (f'WITH eligible AS MATERIALIZED (SELECT id, embedding FROM ann_bench ' + f'WHERE {predicate}) SELECT id, embedding <-> \'{query}\' AS distance ' + f'FROM eligible ORDER BY distance, id LIMIT {args.k};') + print('EXPLAIN (ANALYZE, BUFFERS) ' + exact) + print(exact) + print(f"SELECT '{label}: native planner choice with iterative HNSW available';") + ann = (f'SELECT id, embedding <-> \'{query}\' AS distance FROM ann_bench ' + f'WHERE {predicate} ORDER BY distance LIMIT {args.k};') + print('EXPLAIN (ANALYZE, BUFFERS) ' + ann) + print(ann) + if args.filtered_native: + print(f"SELECT '{label}: planner choice with native filtered exact path enabled';") + print('SET LOCAL ann.enable_filtered_search = on;') + filtered = (f'SELECT id, embedding <-> \'{query}\' AS distance FROM ann_bench ' + f'WHERE {predicate} ORDER BY distance, id LIMIT {args.k};') + print('EXPLAIN (ANALYZE, BUFFERS) ' + filtered) + print(filtered) + print('SET LOCAL ann.enable_filtered_search = off;') + print('ROLLBACK;') + + +if __name__ == '__main__': + main() diff --git a/benchmarks/ann/search.py b/benchmarks/ann/search.py new file mode 100644 index 000000000..9399b1eff --- /dev/null +++ b/benchmarks/ann/search.py @@ -0,0 +1,93 @@ +"""Reproducible single-client baseline; this does not measure PostgreSQL or GPU QPS.""" + +import argparse +import json +import math +from pathlib import Path +import random +import struct +import sys +import time + +sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'test/ann')) +from support.client import Worker + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--rows', type=int, default=10000) + parser.add_argument('--dimensions', type=int, default=32) + parser.add_argument('--batch', type=int, default=16) + parser.add_argument('--iterations', type=int, default=20) + parser.add_argument('--survivor-fraction', type=float, default=.01) + parser.add_argument('--mode', choices=['shared', 'unique'], default='shared') + parser.add_argument('--seed', type=int, default=42) + parser.add_argument('--backend', choices=['cpu-exact', 'cuvs'], default='cpu-exact') + parser.add_argument('--worker', type=Path) + parser.add_argument('--min-recall', type=float, default=.9) + args = parser.parse_args() + if min(args.rows, args.dimensions, args.batch, args.iterations) <= 0: + parser.error('sizes must be positive') + if not 0 <= args.survivor_fraction <= 1: + parser.error('survivor fraction must be between zero and one') + if not 0 <= args.min_recall <= 1: + parser.error('minimum recall must be between zero and one') + rng = random.Random(args.seed) + def vector(): + return [struct.unpack('<f', struct.pack('<f', rng.random()))[0] + for _ in range(args.dimensions)] + vectors = [vector() for _ in range(args.rows)] + queries = [vector() for _ in range(args.batch)] + passing = int(args.rows * args.survivor_fraction) + filters = [sorted(rng.sample(range(args.rows), passing)) + for _ in range(1 if args.mode == 'shared' else args.batch)] + allowed = filters[0] if args.mode == 'shared' else filters + latencies = [] + executable = args.worker + if executable is None and args.backend == 'cuvs': + executable = Path(__file__).resolve().parents[2] / 'ann/ann-worker-cuvs' + recalls = [] + with Worker(executable, args.backend) as worker: + caps = worker.capabilities() + start = time.perf_counter() + worker.build(1, vectors) + build_seconds = time.perf_counter() - start + result = worker.search(1, queries, 10, allowed=allowed, per_query=args.mode == 'unique') + # Independent exact oracle outside the timed interval. + for i, query in enumerate(queries): + ids = filters[0] if args.mode == 'shared' else filters[i] + ranked = sorted(ids, key=lambda j: ( + sum((a - b) ** 2 for a, b in zip(vectors[j], query)), j))[:10] + if caps['exact'] and [r[0] for r in result[i]] != ranked: + raise RuntimeError('reference validation failed') + if any(r[0] not in ids for r in result[i]): + raise RuntimeError('ineligible result') + recall = len(set(ranked) & {r[0] for r in result[i]}) / len(ranked) if ranked else 1 + recalls.append(recall) + if recall < args.min_recall: + raise RuntimeError(f'initial batch recall {recall} below required {args.min_recall}') + start = time.perf_counter() + for _ in range(args.iterations): + began = time.perf_counter() + worker.search(1, queries, 10, allowed=allowed, per_query=args.mode == 'unique') + latencies.append(time.perf_counter() - began) + elapsed = time.perf_counter() - start + latencies.sort() + options = vars(args).copy() + options['worker'] = str(executable) if executable else None + report = options | { + 'scope': 'single-client worker; precomputed synthetic eligibility', + 'capabilities': caps, 'oracle_validated': True, + 'initial_batch_min_recall_at_k': min(recalls), + 'quality_scope': 'initial batch only', + 'build_seconds': build_seconds, 'elapsed_seconds': elapsed, + 'batch_requests_per_second': args.iterations / elapsed, + 'vector_searches_per_second': args.iterations * args.batch / elapsed, + 'batch_p50_ms': latencies[math.ceil(.5 * len(latencies)) - 1] * 1000, + 'batch_p99_ms': latencies[math.ceil(.99 * len(latencies)) - 1] * 1000, + } + print(json.dumps(report, indent=2, sort_keys=True)) + + +if __name__ == '__main__': + main() diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..db77cb6b9 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,13 @@ +# Design documents + +- [cuVS bulk build and filtered ANN implementation plan](cuvs-filtered-ann-plan.md): proposed GPU-backed indexing and search for full-text and text metadata predicates, including architecture, execution diagrams, repository code conventions, build/upgrade integration, delivery steps, and validation gates. +- [Native CPU staging index](ann-native-index.md): implemented C access method, SQL usage, storage/locking diagrams, source conventions, and lifecycle tests. +- [Native filtered exact search](ann-filtered-search.md): opt-in C CustomScan, supported query shapes, predicate-first execution, EXPLAIN, and tests. +- [Loaded cuVS indexes](ann-cuvs-index.md): offline sharded CAGRA build, multi-GPU loading, `ann_index_search` ANN and hybrid search from SQL, the `USE_CUVS` build flag, B200 throughput options, and load-test results. +- [Networked GPU nodes](ann-networked.md): many PostgreSQL hosts evaluate filters and send vector search to GPU nodes over authenticated TCP, with size-based routing, heartbeats and failover, and scaling measurements. +- [Worker pipeline for filtered search](ann-worker-pipeline.md): pipelined, shared-memory worker route for the filtered CustomScan, mocked GPU backend, fallbacks, protocol operations 7–10, and measured throughput. +- [High-throughput filtered-search analysis](cuvs-search-performance.md): CPU eligibility and GPU execution optimization tracks, capacity estimates, batching/admission design, and end-to-end benchmark gates. + +The plans distinguish proposed work from the implemented features described in the native index and filtered-search guides. + +The [standalone C worker and C scheduler](../ann/README.md) are implemented. Python is restricted to test helpers and benchmark scripts, including the frozen-table SQL reference; the native C exact L2 access method is implemented, with an opt-in selective exact CustomScan that can pipeline scoring to the worker. Loaded cuVS indexes serve sharded CAGRA search on GPUs, validated with cuVS 26.10 on B200s; the pipeline's mock GPU backend remains for hosts without one. Accelerator generation/delta integration remains pending. diff --git a/docs/ann-cuvs-index.md b/docs/ann-cuvs-index.md new file mode 100644 index 000000000..d4c49c6a4 --- /dev/null +++ b/docs/ann-cuvs-index.md @@ -0,0 +1,152 @@ +# Loaded cuVS indexes: sharded multi-GPU search from PostgreSQL + +An offline Python script builds an immutable, sharded CAGRA index directory from a table. The C worker loads it across all GPUs, and two C SQL functions serve ANN and hybrid (filtered) queries. Everything on the query path is C: the PostgreSQL functions, the worker, and the cuVS adapter. Python is used only by the offline builder, tests and benchmarks. + +## Quick start + +```sh +# GPU host: build the worker with cuVS (default builds are CPU-only) +make -C ann USE_CUVS=1 CUVS_PREFIX=/path/to/libcuvs CUDA_HOME=/path/to/cuda +ann/ann-worker-cuvs --backend cuvs --listen /tmp/ann.sock --lanes 64 + +# offline build: one shard per visible GPU, FP16 dataset for bandwidth +python3 scripts/ann/build_cuvs_index.py --dsn 'dbname=app' --table public.docs \ + --key id --vector embedding --out /data/ann/docs --dtype float16 +``` + +```sql +-- postgresql.conf: ann.worker_socket = '/tmp/ann.sock' +SELECT ann_index_load('docs', '/data/ann/docs'); -- superuser; returns rows loaded + +-- ANN +SELECT d.* FROM ann_index_search('docs', $1, 10) s JOIN docs d USING (id) ORDER BY s.distance; + +-- hybrid: any SQL predicate selects the eligible keys +SELECT d.* FROM ann_index_search('docs', $1, 10, + ARRAY(SELECT id FROM docs WHERE tenant = $2 AND body @@ to_tsquery('english', $3))) s +JOIN docs d USING (id) ORDER BY s.distance; +``` + +`ann_index_search(name, query, k, filter bigint[] DEFAULT NULL)` returns up to `k` rows of `(id bigint, distance real)`, sorted by L2 distance. Distances are exact (FP32 vectors re-score every candidate), so they agree with `<->`. A NULL filter means no filter; an empty array, or one whose keys are all absent from the index, returns no rows. NULL elements are ignored. + +## On-disk format + +| File | Content | +| --- | --- | +| `manifest` | Text `key value` lines: `pgvector-ann-index 1`, `dimensions`, `rows`, `metric l2`, `dtype float32\|float16`, `source schema.table`, `shards N`, then `shard i first rows file` per shard. Written last, so partial builds are never loadable. | +| `keys.i64` | `rows` strictly increasing little-endian int64 table keys. Global ordinal *i* is the *i*-th smallest key. | +| `vectors.f32` | `rows × dimensions` little-endian FP32 vectors in key order. They're memory-mapped for the exact route and for re-scoring. | +| `shard-i.cagra` | CAGRA graph and dataset (`cagra.save(..., include_dataset=True)`) for ordinals `[first, first + rows)`. | + +Shards are contiguous key ranges, so mapping a key to its shard is a binary search, and shard results never overlap. The worker validates the manifest, file sizes, key order, and each shard's dimensions and row count before registering the index. + +**Immutability.** A directory is never modified, and the builder refuses a non-empty output directory. `ann_index_load` compares the source table's current row count with the manifest and refuses a stale index. After a table changes, build a new directory and load it under a new name. Joining results back to the table drops deleted keys, but rows added after the build are invisible. A loaded name stays until the worker restarts. + +## Search execution + +```mermaid +flowchart LR + SQL[ann_index_search] -->|query, k, sorted keys| W[ann-worker lanes] + W --> U{filter?} + U -->|no| B[Batch unfiltered queries of the index] + B --> G0[CAGRA shard 0 on GPU 0] + B --> G1[CAGRA shard 1 on GPU 1] + B --> Gn[... shard N on GPU N mod devices] + U -->|"eligible ≤ exact limit"| X[Exact scan of eligible rows] + U -->|larger| F[Per-shard bitsets] + F --> G0 & G1 & Gn + F -->|"small shard slice"| X + G0 & G1 & Gn --> R[Re-score candidates with FP32 vectors] + R --> M[Pairwise tree merge of per-shard top-k] + X --> M + M --> SQL +``` + +- **Lanes and devices.** A lane is a worker thread with its own cuVS resources, stream, workspace pool and buffers on every GPU. Shard *i* lives on GPU *i* mod the device count, or on every GPU with `--replicate on`. Requests from all PostgreSQL connections share one queue and one admission budget (`--max-pending-mb`). +- **Unfiltered batching.** A lane takes up to `--max-batch` queued queries. Unfiltered queries against the same index go to each shard as one query matrix. Each shard is searched asynchronously on its own GPU, then the lane synchronizes every device once. +- **Filtered queries.** The worker maps the eligible keys to ordinals. If the total is at most `--exact-limit` (default 20,000 rows), it scans those rows exactly on the CPU, which is cheaper than a graph search and gives full recall. Otherwise each shard whose slice is at most `exact-limit / shards` is scanned exactly, and the remaining shards run CAGRA with a bitset of their eligible rows. Selective filters (under 25% of rows) raise the CAGRA internal top-k to `--filtered-itopk` (default 512) to keep recall up. +- **Merge.** The worker re-scores each shard's candidates with FP32 vectors, sorts them, and merges the shard lists pairwise, halving their number each round, down to the global top-k. + +## B200 throughput options + +| Option | Effect | +| --- | --- | +| `--dtype float16` (builder) | Stores the CAGRA dataset in FP16: half the HBM traffic and footprint (8.5 GB vs 11 GB on disk for 10M×128). The worker converts queries to FP16, and FP32 re-scoring keeps returned distances exact. | +| `--itopk N` (worker) | CAGRA internal top-k for unfiltered queries, at most 512. This is the main recall/throughput dial. The default is 128; 64 kept recall@10 at 1.0 on the 10M benchmark below. | +| `--filtered-itopk N` | Internal top-k for selective filters, at most 512 (single-CTA CAGRA search). | +| `--max-batch N`, `--delay-us N` | Queries per CAGRA submission, and an optional wait to fill a batch. | +| `--lanes N` | Concurrent submissions across all GPUs. | +| `--cagra-persistent on` | cuVS persistent search kernels, for many small batches. | +| `--replicate on` (worker) | Loads every shard on every GPU, and each lane searches its home GPU's copy. Combined with `--shards 1` in the builder, a query is a single CAGRA search on one GPU: the highest ANN throughput measured. Use it when one copy of the index fits in one GPU's memory. | +| `--exact-limit ROWS` | Where filtered queries switch from exact scans to filtered CAGRA. | + +cuVS 26.10's CAGRA supports FP32, FP16, int8 and uint8 datasets. It has no NVFP4 or FP8 path, so those aren't offered. int8/uint8 need a quantization scale chosen when building the index; that's left for a later builder option. + +## Build flag + +The extension never links CUDA: it talks to the worker over a Unix socket, so the extension and the CPU worker build anywhere, including hosts without a GPU. `make -C ann USE_CUVS=1` additionally builds `ann-worker-cuvs` against cuVS (`CUVS_PREFIX`, `CUDA_HOME`, plus `CPPFLAGS`/`LDFLAGS` for pip-installed SDKs). Without it, the CPU worker's `cpu-exact` backend serves loaded indexes exactly from `keys.i64` and `vectors.f32`, using the same per-shard tree merge. That makes it the reference for tests. + +## Tests + +| Test | Needs | Covers | +| --- | --- | --- | +| `test/ann/test_index.py` | CPU worker; `ANN_PG_DSN` for the SQL tests | File format, corrupt files, sharded tree merge against brute force, filters (unknown keys, duplicates), protocol errors. Through SQL: ANN and hybrid results equal to `ORDER BY <->`, joins, empty/NULL filters, unavailable worker, stale-index refusal, superuser-only load. | +| `test/ann/test_cuvs_index.py` | `ANN_CUVS_TEST=1`, GPU, cuVS Python | Builder output for FP32 and FP16; recall ≥ 0.9 for ANN and filtered CAGRA; filtered results stay within the filter; exact route equal to SQL; exact distances. | +| `test/ann/test_cuvs_gathered.py` | `ANN_CUVS_TEST=1`, GPU | GPU brute force for the pipelined CustomScan route. | + +```sh +make -C ann check +ANN_PG_DSN='host=/tmp dbname=postgres' python3 test/ann/test_index.py +ANN_CUVS_TEST=1 ANN_PG_DSN='host=/tmp dbname=postgres' python3 test/ann/test_cuvs_index.py +``` + +## Load test + +`benchmarks/ann/cuvs_load.py` generates a synthetic table (`tenant` filters 0.1% of rows, `category` 1%), then sweeps `pgbench` client counts for each workload, reporting QPS, latency percentiles and recall@10 against exact SQL. + +### Results on 8× NVIDIA B200 + +Setup: +- **Host:** 224 CPUs, 2 TB RAM, 8× B200 (driver 580.126), cuVS 26.10, PostgreSQL 17.6. PostgreSQL, `pgbench`, and the worker all ran on the GPU host. +- **Data:** 10,000,000 rows × 128 dimensions of clustered, low-intrinsic-dimension synthetic vectors. Query vectors are drawn from the same distribution but aren't rows of the table. +- **Queries:** k = 10, `pgbench -M prepared`, worker `--lanes 96 --itopk 64`. Recall@10 is measured against exact SQL on 20 sample queries. +- **Filters:** covering `(filter, id)` B-tree indexes produce the eligibility arrays. + +Peak throughput by layout: + +| Layout | Workload | Recall@10 | Peak QPS (clients) | p50 / p99 at peak | +| --- | --- | --- | --- | --- | +| 8 shards, 1 per GPU, FP32 | ANN | 1.0 | 362,889 (1,536) | 3.4 / 12.3 ms | +| 8 shards, FP16 | ANN | 0.985 | 354,250 (1,024) | 2.4 / 8.4 ms | +| 1 shard replicated to every GPU (`--replicate on`) | ANN | 1.0 | **388,572** (1,536) | 3.0 / 12.9 ms | +| 8 shards | hybrid, 1% filter (100k keys, filtered CAGRA) | 1.0 | 3,592 (256) | 38 / 412 ms | +| 1 shard replicated | hybrid, 1% filter | 0.68 | 7,507 (256) | 29 / 165 ms | +| 8 shards | hybrid, 0.1% filter (10k keys, exact route) | 1.0 | 14,535 (256) | 9.3 / 98 ms | +| 1 shard replicated | hybrid, 0.1% filter | 1.0 | 14,465 (2,048) | 110 / 492 ms | + +ANN scaling, 8 shards FP32: + +| Clients | 64 | 128 | 256 | 512 | 1,024 | 1,536 | 2,048 | +| --- | --- | --- | --- | --- | --- | --- | --- | +| QPS | 73,152 | 96,107 | 185,933 | 305,658 | 355,550 | 362,889 | 343,608 | +| p99 ms | 1.2 | 2.3 | 2.2 | 3.0 | 8.3 | 12.3 | 16.8 | + +What the measurements showed: + +- **ANN saturates the host at about 350–390k QPS from 1,024 clients on.** All eight GPUs then report about 99% utilization, and the host CPUs are about 95% busy; the worker accounts for about 150 cores, mostly in kernel time. Per query, the host side (PostgreSQL, the socket round trip, lane wake-ups and device synchronization) costs about as much as the GPU work, so FP16 adds little at this scale. +- **Per-lane cuVS workspace pools were the largest single fix.** Without them, concurrent searches serialize on device allocation: ANN reached 26,891 QPS with a flat 28 ms median. +- **Hybrid throughput is bounded by producing the eligibility array in PostgreSQL.** With a covering index, a 100k-key array costs about 7 ms, roughly 11k arrays per second at best; a 10k-key array costs about 1 ms (about 34k per second). Without the covering index the heap visits dominate: about 3k and 9k arrays per second. +- **Filtered CAGRA recall depends on shard size.** Eight 1.25M-row graphs keep recall@10 at 1.0 with a 1% filter. One 10M-row graph drops to 0.68, because CAGRA's internal top-k is capped at 512. With a single replicated shard, raise `--exact-limit` (exact scans of up to about 100k rows cost a few milliseconds) or keep shards smaller. +- **Synthetic data choice matters.** Isotropic random vectors (uniform, or Gaussian clusters with full-dimensional noise) defeat every graph index: CAGRA reached recall@10 of 0.13 on them even when driven directly from Python. The generator therefore projects 16-dimensional clustered latents to 128 dimensions. Measure your own embeddings before relying on these numbers. + +Reproduce: + +```sh +python3 benchmarks/ann/cuvs_load.py generate --dsn "$DSN" --table bench10m --rows 10000000 --dimensions 128 +python3 scripts/ann/build_cuvs_index.py --dsn "$DSN" --table public.bench10m --vector emb --out /data/bench10m +ann/ann-worker-cuvs --backend cuvs --listen /tmp/ann.sock --lanes 96 --itopk 64 +python3 benchmarks/ann/cuvs_load.py bench --dsn "$DSN" --table bench10m --index b32 --path /data/bench10m \ + --socket /tmp/ann.sock --clients 64,256,1024,1536 --threads 96 +``` + +`max_connections` must exceed the largest client count, and the shell running `pgbench` needs enough file descriptors. diff --git a/docs/ann-filtered-search.md b/docs/ann-filtered-search.md new file mode 100644 index 000000000..5410abfd2 --- /dev/null +++ b/docs/ann-filtered-search.md @@ -0,0 +1,80 @@ +# Native filtered exact search + +`AnnFilteredSearch` is an opt-in C CustomScan. It computes exact distances over rows produced by a normal PostgreSQL predicate plan, so a restrictive text/metadata filter can avoid reading and scoring every vector in the `ann` index. With `ann.worker_socket` set, scoring can be pipelined to the standalone worker; see the [worker pipeline guide](ann-worker-pipeline.md). + +## Enable and inspect + +```sql +LOAD 'vector'; +SET ann.enable_filtered_search = on; +SET gin_fuzzy_search_limit = 0; + +EXPLAIN (ANALYZE, BUFFERS) +SELECT id, embedding <-> '[1,2,3]' AS distance +FROM documents +WHERE tenant = 'example' AND terms @@ to_tsquery('simple', 'database') +ORDER BY distance, id LIMIT 10; +``` + +The relation must have a valid, non-partial, plain-column `ann` index matching the distance operator's support function. Predicate indexes are optional: PostgreSQL chooses between sequential, B-tree, and bitmap heap scans, including GIN. The custom route reads vectors from the predicate child's visible heap/index tuples, rather than from the `ann` pages. The index currently supplies the supported-column/metric contract. + +`LOAD` reliably installs hooks even when no earlier vector input/function call has loaded the extension in that backend. The GUC defaults to off. Changing it affects subsequent planning; already cached prepared plans retain their route until replanned. Deallocate/reprepare or use `DISCARD PLANS` to change those plans. + +## Implemented execution + +```mermaid +flowchart TD + SQL[Filtered ordered SELECT] --> Hook[AnnCreateUpperPaths: opt-in whitelist] + Hook --> Normal[Keep ordinary paths for unsupported shapes] + Hook --> Child[Normal PostgreSQL predicate child plan] + Child --> Access[Sequential / B-tree / bitmap heap scan] + Access --> Visibility[Heap visibility, index rechecks, RLS, WHERE] + Visibility --> Eligible[Qualified rows and source vectors] + Eligible --> Distance[Exact support-function distance in C] + Distance --> Sort[Tuplesort: full SQL ordering and NULL handling] + Sort --> Limit[LIMIT / OFFSET / WITH TIES handled by PostgreSQL] + Limit --> Result[Final projection and rows] +``` + +[src/annplan.c](../src/annplan.c) chains the previous upper-path hook and replaces compatible full SortPaths with equivalent CustomPaths. It retains PostgreSQL's predicate path choice, pathkeys, and ordinary predicate/projection/sort cost estimates; there is no fabricated GPU cost discount. Ordered access-method scans, incremental sorts, and other alternatives remain available to the planner. + +[src/anncustom.c](../src/anncustom.c) initializes the child through the normal executor. Its plan changes the child's distance-output column into a vector-input column. The custom executor computes that distance after the child has applied all its quals, and sorts output tuples using the original ordering operators, collations, and NULL rules. Query vectors are copied and detoasted into query-owned memory before per-tuple contexts are reset. Moved operator calls retain function EXECUTE checks and PostgreSQL function-access hooks. The output target list remains typed as the original SQL distance expression. + +A positive, finite `LIMIT + OFFSET` uses a bounded sort. Bounds come from execution-time Const/external Param values, including generic prepared statements, rather than estimated row counts. NULL/unbounded limits, overflow, and WITH TIES use an unbounded sort; PostgreSQL retains final limit/offset/tie semantics. Sorting respects `work_mem` and spills as needed. The executor checks interrupts while consuming rows and relies on PostgreSQL resource owners and query contexts for error cleanup. Scroll cursors receive core materialization when required. + +## Supported initial contract + +- Top-level, single ordinary table SELECT, with predicates and a matching unconditional plain-column `ann` index. +- Ascending distance as the first ordering expression, NULLS LAST, with the indexed column on the left and a vector Const/external Param on the right. Secondary ordering keys retain their SQL operators and collations. +- A strict, immutable support function matching the SQL operator. The shipped opclass is fixed-dimension `vector` L2. +- Complete sequential/B-tree/bitmap predicate paths. Accelerator-ordered, index-only, and parameterized/parallel paths are not converted. +- Const/external Param LIMIT/OFFSET, including NULL and generic prepared parameters. WITH TIES retains every peer. + +Joins, subquery roots, CTEs, aggregates/grouping, DISTINCT, windows, SRFs, row locks, volatile target/predicate expressions, expression/partial ANN indexes, alternate distance orderings, and complex limit expressions keep ordinary PostgreSQL execution. Enabling the GUC does not force a custom plan when another path wins or this whitelist does not match. + +Complete eligibility requires `gin_fuzzy_search_limit = 0`. Planning skips the custom route otherwise. Execution also checks this setting so a cached plan cannot silently change its eligibility contract after a GUC change; such a cached custom plan reports an error until the setting is restored or the query is replanned. + +## EXPLAIN and validation + +EXPLAIN reports `Search Route: exact CPU after predicates`, or the worker route described in the [pipeline guide](ann-worker-pipeline.md). ANALYZE also reports total `Eligible Rows`, custom-stage `Distance Evaluations`, and sort method/space. NULL operands skip the distance call. Counts accumulate over executions/rescans of that node; they are not whole-query GPU or IPC metrics. + +A local 4,096-row fixture with 40 qualifying rows used a B-tree child scan, evaluated 40 distances, and used a top-N heapsort for LIMIT 10. This checks selective execution; it is not a production throughput measurement or a claimed speedup over PostgreSQL's ordinary filtered exact sort. + +[test/ann/test_filtered.py](../test/ann/test_filtered.py) compares this route to ordinary exact SQL and checks that the CustomScan was used. It covers B-tree/GIN predicates, combined filters, projection, spill, ties, NULLs, LIMIT/OFFSET and generic prepared parameters, snapshots/writes, RLS, function EXECUTE permissions, toasted 2,000-dimension vectors, cancellation cleanup, fuzzy-GIN cached-plan checks, index-only fallback, and unsupported-shape fallback. [The SQL regression](../test/sql/ann_filtered.sql) adds prepared queries and scroll-cursor behavior to the normal pgvector harness. + +```sh +make && make install +make installcheck +ANN_PG_DSN='dbname=test_database' make -C ann check-postgres +``` + +Runtime code remains C; Python drives tests and benchmarks only. Local testing uses PostgreSQL 17.6/macOS ARM64; the full PostgreSQL CI matrix is still required. Native worker IPC and process-wide batching are implemented by the [worker pipeline](ann-worker-pipeline.md); accelerator mappings/generations and sustained-load validation are next milestones. CAGRA integration and compilation against cuVS remain deferred. + +The SQL benchmark generator can include this route alongside the existing exact and HNSW baselines: + +```sh +python3 benchmarks/ann/postgres.py --rows 10000 --dimensions 32 --filtered-native > /tmp/ann-benchmark.sql +psql -X -v ON_ERROR_STOP=1 -f /tmp/ann-benchmark.sql test_database +``` + +It uses a temporary relation and rolls back. Inspect each EXPLAIN: enabling the route does not guarantee that the planner selects it when ordered/index-only alternatives are available. diff --git a/docs/ann-native-index.md b/docs/ann-native-index.md new file mode 100644 index 000000000..40e389480 --- /dev/null +++ b/docs/ann-native-index.md @@ -0,0 +1,81 @@ +# Native CPU staging index + +The experimental `ann` access method is implemented in C inside `vector`, using pgvector's callback and file layout. It provides an exact L2 storage/lifecycle baseline for later accelerator integration. It does not call the standalone worker or GPU libraries. Its scan reads and sorts every index entry before returning results, including for a small LIMIT. It should not be used as a throughput optimization over sequential exact search. + +## SQL and installation + +Build/install the extension using the repository's normal PGXS commands. Fresh installs use version 0.9.0; existing 0.8.7 installs use `ALTER EXTENSION vector UPDATE TO '0.9.0'` after installing the updated library and SQL files. No standalone worker dependency is introduced. + +```sql +CREATE EXTENSION vector; +CREATE TABLE documents (id bigint, tenant text, terms tsvector, embedding vector(3)); +CREATE INDEX documents_ann ON documents USING ann (embedding vector_l2_ops); +SELECT id FROM documents +WHERE tenant = 'example' AND terms @@ to_tsquery('simple', 'database') +ORDER BY embedding <-> '[1,2,3]' LIMIT 10; +``` + +Only fixed-dimension `vector` with L2 is registered, up to 2,000 dimensions. NULL vectors remain in the index and sort last. Zero vectors are valid. PostgreSQL checks heap visibility and evaluates predicates. The planner can choose another path: the cost model charges for a complete index scan and sort. `SET enable_seqscan = off` is useful in tests to exercise this AM; it is not a production tuning recommendation. + +## Code structure and naming + +| File | Responsibility | +| --- | --- | +| [ann.h](../src/ann.h) | Shared page format, constants, callback declarations | +| [ann.c](../src/ann.c) | Handler, cost estimate, opclass validation, properties | +| [annplan.c](../src/annplan.c) | Opt-in filtered upper-path hook, whitelist, normal predicate/sort costing | +| [anncustom.c](../src/anncustom.c) | Predicate child plan, exact eligible-vector scoring, bounded sort, EXPLAIN | +| [annbuild.c](../src/annbuild.c) | Table-AM build callback, main/init forks, build WAL | +| [anninsert.c](../src/anninsert.c) | Tuple formation, insertion locks, FSM hints, generic WAL | +| [annscan.c](../src/annscan.c) | Ordered heap-TID scan, rescan, spillable tuplesort | +| [annutils.c](../src/annutils.c) | Type/schema resolution, dimension/page validation, metapage creation | +| [annvacuum.c](../src/annvacuum.c) | Dead-TID removal, page reuse, vacuum statistics | + +Callbacks use lower-case AM names (`annbuild`, `anngettuple`); shared helpers/types use `Ann` prefixes, constants use `ANN_`, and private helpers follow existing build/scan naming. Sources include `postgres.h` first, use PostgreSQL allocation contexts, buffer locks, interrupts, table-AM scans, and generic WAL. Files have been formatted with PostgreSQL's `pgindent`. The POSIX scheduler's threads remain outside PostgreSQL and never call backend APIs. Python appears only in testing and benchmarking. + +## Execution and storage + +```mermaid +flowchart LR + SQL[Ordered SQL query] --> Planner[PostgreSQL planner] + Planner --> Scan[anngettuple] + Scan --> Pages[Scan index pages under shared locks] + Pages --> Sort[Distance and heap TID tuplesort] + Sort --> Heap[PostgreSQL heap fetch and MVCC check] + Heap --> Qual[Executor text and metadata predicates] + Qual --> Limit[LIMIT and results] +``` + +This diagram describes ordinary AM execution. The opt-in [filtered CustomScan](ann-filtered-search.md) instead uses a PostgreSQL predicate child plan and scores its eligible vectors. Its worker pipeline sends eligible vectors, not `ann` pages, to the worker. GIN and B-tree indexes are not automatically intersected with `ann` pages. + +```mermaid +flowchart TB + Meta[Block 0: magic, version, dimensions, insertion hint] + Meta --> Data[Blocks 1..N: standard index tuples with vector and heap TID] + Insert[Insert] --> ML[Exclusive metapage lock] + ML --> DL[Exclusive data-page lock] + DL --> WAL[Generic WAL: metapage and data page] + Vacuum[VACUUM] --> CL[Cleanup lock on one data page] + CL --> Delete[Remove dead TIDs with generic WAL] + Delete --> FSM[Record free space for reuse] +``` + +The metapage serializes writers and establishes the metapage-before-data lock order. Vacuum only locks data pages and never acquires the metapage. Free-space-map entries are hints: insertion checks real free space under the data-page lock. Build uses `table_index_build_scan`, per-row temporary contexts, and logs the constructed range. Unlogged indexes receive an init fork. Scan distance evaluation uses the opclass support function; sorting uses `work_mem` and can spill. Physical pages left blank by aborted extension are skipped and can be initialized for reuse. + +This format is a correctness baseline. It stores full vectors rather than accelerator row mappings or graphs, has no generation/delta protocol, and does not provide shared admission across PostgreSQL processes. Index concurrency is currently limited by the insertion metapage lock. + +## Tests and remaining work + +- [SQL regression](../test/sql/ann_vector.sql): ordering, filtered LIMIT, NULLs, lateral rescans, updates/deletes, vacuum, reindex, truncate, and invalid dimensions. +- [Native SQL oracle tests](../test/ann/test_native.py): randomized query vectors, restrictive text/metadata predicates, spill-size sorts, NULL ordering, read-own-writes and abort, repeatable-read snapshots, concurrent writers, pre-build HOT chains, concurrent builds, table rewrites, maximum dimensions, free-space reuse, unlogged/temporary indexes. Plans are checked to ensure the AM was exercised. +- [WAL TAP test](../test/t/050_ann_wal.pl): exact primary/replica results through build, mutation/vacuum, crash recovery, concurrent reindex, and truncate. + +```sh +make && make install +make installcheck +make prove_installcheck PROVE_TESTS=test/t/050_ann_wal.pl +python3 -m pip install -r test/ann/requirements-postgres.txt +ANN_PG_DSN='dbname=test_database' make -C ann check-postgres +``` + +Fresh install and 0.8.7-to-0.9.0 upgrade were also checked, including extension schema relocation. Local validation uses PostgreSQL 17.6 on macOS ARM64. The repository's PostgreSQL CI matrix still needs to run; compatibility beyond the tested host is not claimed. Production rollout also needs broader fault, concurrent-vacuum, and sustained-load tests. Selective exact eligibility execution is now implemented in the opt-in CustomScan. A bounded C worker client with admission is implemented ([worker pipeline](ann-worker-pipeline.md)). CAGRA integration and compilation against cuVS remain deferred, as do hipVS and fused NVFP4 adapters. diff --git a/docs/ann-networked.md b/docs/ann-networked.md new file mode 100644 index 000000000..1c5741ef4 --- /dev/null +++ b/docs/ann-networked.md @@ -0,0 +1,121 @@ +# Networked GPU nodes: hybrid search across PostgreSQL hosts + +[Loaded cuVS indexes](ann-cuvs-index.md) can be served by GPU nodes on the network, so many PostgreSQL hosts can evaluate SQL filters on their own CPUs and send only vector search to the GPUs. The query path is C on both sides: `src/annremote.c` and `src/annindex.c` in the extension, `ann/server.c` in the worker. + +```mermaid +flowchart LR + subgraph CPU host A + PA[PostgreSQL] --> FA[SQL filter → eligible keys] + end + subgraph CPU host B + PB[PostgreSQL] --> FB[SQL filter → eligible keys] + end + FA -->|"small filter"| XA[exact on host A] + FB -->|"small filter"| XB[exact on host B] + FA -->|"TCP: query + delta-varint keys"| G1[GPU node 1] + FB -->|TCP| G2[GPU node 2] + G1 & G2 -->|"top-k (key, distance)"| PA & PB +``` + +## Setup + +```sh +# GPU node: Unix socket for local tooling, TCP for PostgreSQL hosts +head -c 32 /dev/urandom | base64 > /etc/ann/token && chmod 600 /etc/ann/token +ann/ann-worker-cuvs --backend cuvs --listen /run/ann.sock \ + --listen-tcp 0.0.0.0:7700 --auth-token-file /etc/ann/token --lanes 96 --itopk 64 +``` + +```sql +-- every PostgreSQL host (superuser settings) +ALTER SYSTEM SET ann.worker_address = 'gpu1:7700,gpu2:7700'; +ALTER SYSTEM SET ann.worker_auth_token = '...'; -- same token as the nodes +SELECT pg_reload_conf(); + +-- one host loads the index on every node; the other hosts attach +SELECT ann_index_load('docs', '/data/ann/docs'); -- path on the GPU nodes +SELECT ann_index_attach('docs'); -- verifies this host's table, records it +``` + +`ann_index_load` sends the load to every listed node and records the index in `ann_index_catalog` (the source table plus key and vector columns, taken from the manifest). `ann_index_attach` asks a node for the same information, checks this host's copy of the source table against the index's row count, and records it. On physical replicas the catalog arrives through replication. + +## Queries and routing + +```sql +-- the host evaluates the filter, then routes by its size +SELECT * FROM ann_hybrid_search('docs', $1, 10, format('tenant = %s AND lang = %L', $2, $3)); + +-- or pass eligible keys yourself; small key lists are scored on this host too +SELECT * FROM ann_index_search('docs', $1, 10, ARRAY(SELECT id FROM docs WHERE tenant = $2)); +``` + +`ann_hybrid_search` counts the filter's rows with a scan capped at `ann.gpu_min_eligible` (default 1000): + +- **Below the threshold:** the host runs `ORDER BY vector <-> query LIMIT k` itself, so the result is exact and no network is used. +- **At or above it:** the host streams the matching keys and sends them to a GPU node as a zigzag varint first key followed by varint gaps. For dense filters that's one or two bytes per key instead of eight. + +The filter is a SQL boolean expression over the source table. It is prepared as one read-only statement and runs with the caller's privileges, like any dynamic SQL; build it with `format()` so values are quoted. + +**Choosing the threshold.** The measurements below show the trade-off. The GPU node's exact scan over memory-mapped vectors is far cheaper per query than PostgreSQL fetching scattered heap rows. Local scoring wins only when CPU hosts are numerous enough, or the eligible rows cheap enough to read (for example, a covering index that includes the vector column), to outrun the GPU node's own CPUs. + +## Failures and heartbeats + +| Mechanism | Behavior | +| --- | --- | +| Node selection | Each backend picks a node by process ID, which spreads backends across nodes, and keeps that connection. | +| Failover | A failed request or connection marks the node down and retries once on the next node. Searches are idempotent. | +| `ann.worker_retry_ms` (10 s) | How long a down node is skipped; it is tried again only after this, or when every node is down. | +| `ann.worker_heartbeat_ms` (5 s) | A connection idle this long is probed (operation 5) before reuse, so a node that died while idle costs one short probe instead of a request timeout. The same value sets TCP keepalive timing on the client. | +| `ann.worker_timeout_ms` (30 s) | Upper bound on any request. | +| Worker keepalive | The worker enables TCP keepalive (10 s idle, 3 probes) on network connections, so readers belonging to PostgreSQL hosts that crashed or were partitioned are released. | +| Auth deadline | An unauthenticated connection must send the token within 10 s, or it is closed. | +| Cancellation | Waits service interrupts; a cancelled request leaves the connection to be reopened rather than reused mid-reply. | + +## Security + +Network clients must send the shared token (operation 13) before any other frame; any other first frame closes the connection, and the comparison is constant-time. Shared-memory operations (9 and 10) are refused over TCP. `ann.worker_address` and `ann.worker_auth_token` are superuser settings, and the token is hidden from non-superusers. The transport is not encrypted: run it on a private network or VPN, and treat the token as a credential. + +## Protocol additions + +| Operation | Payload | Reply | +| --- | --- | --- | +| 13 auth | Token bytes | Empty | +| 14 packed index search | u32 name length, name, u32 dimensions, k, key count (UINT32_MAX = no filter), u32 packed bytes; FP32 query; varint keys | As operation 12 | +| 15 index info | Index name | Same text as operation 11, including `key` and `vector` columns | + +## Measured scaling + +One DGX B200 host (224 CPUs, 8× B200) was partitioned to stand in for separate machines: + +- **GPU node:** cores 0–95, running the networked cuVS worker. +- **CPU host A:** cores 96–159. +- **CPU host B:** cores 160–223. + +Each CPU host ran its own PostgreSQL 17.6 cluster and `pgbench`, and they connected to the worker over TCP on loopback. The setup otherwise matches the [single-host benchmark](ann-cuvs-index.md#results-on-8-nvidia-b200): 10M × 128 dimensions, 8 FP32 shards, `--itopk 64`, recall@10 = 1.0 on every workload. QPS figures are the best of 256 or 512 clients per host. + +| Workload | Host A alone | Hosts A + B (total) | Bottleneck | +| --- | --- | --- | --- | +| ANN | 295,787 | 391,781 | GPU node (as on one host) | +| `ann_hybrid_search`, 1% filter (GPU, filtered CAGRA) | 1,203 | 1,298 | GPU node | +| `ann_hybrid_search`, 0.1% filter, GPU route (`gpu_min_eligible = 0`) | 13,895 | 14,320 | GPU node's CPUs (exact scan) | +| `ann_hybrid_search`, 0.1% filter, local route | 3,345 | 5,605 | CPU hosts: scales with hosts | + +- A single 64-core CPU host already drives the GPU node to about 300k ANN QPS. Adding hosts helps until the node saturates, at the same ~390k ceiling measured when everything shared one machine. +- Scoring on the CPU hosts is the part that scales with more hosts, but it costs about 4× more per query than the GPU node's exact route on this schema. That's why the default `ann.gpu_min_eligible` is 1000. +- **Filtered brute force didn't help.** `--filtered-bruteforce on` adds exact GPU brute force over a device copy of each shard for selective filters. Under load it reached about 800 QPS on the 1% filter, against 1,200 for filtered CAGRA, so it is off by default. +- **Larger-scale caveats.** These hosts shared one machine's memory system and talked over loopback. A real network adds round-trip latency (about 10–50 µs in a data center) and per-query bandwidth (about 150 KB for a 1% filter on 10M keys). + +## RDMA (EFA, InfiniBand) + +Not implemented yet. The intended design is a libfabric RDM transport for search frames: libfabric is the native API for AWS EFA, and it also drives InfiniBand through its verbs provider. Authentication and fabric address exchange would stay on the existing TCP connection, and completion queues would be polled from user space. The test host has ConnectX-7 InfiniBand adapters but no AWS EFA, and no libfabric installed. + +## Tests + +`test/ann/test_remote.py` needs `ANN_PG_DSN`; it starts local networked workers. It covers: + +- Token authentication and unauthenticated disconnects, and refusal of shared memory over TCP. +- Packed-filter decoding, including malformed gaps. +- ANN over TCP. +- Both routes of `ann_hybrid_search` and of the key-array filter, each equal to exact SQL. +- Read-only filter enforcement, attach and catalog, a wrong token. +- Failover with heartbeats after a node dies while idle, and every node down. diff --git a/docs/ann-worker-pipeline.md b/docs/ann-worker-pipeline.md new file mode 100644 index 000000000..8af86c379 --- /dev/null +++ b/docs/ann-worker-pipeline.md @@ -0,0 +1,116 @@ +# Worker pipeline for filtered search + +The opt-in `AnnFilteredSearch` CustomScan can hand distance scoring to the standalone [C worker](../ann/README.md) while PostgreSQL keeps producing eligible rows. GPU execution is mocked: the `mock-gpu` backend models one device submission per batch (launch latency plus transfer time) and computes exact results on the CPU. There is no Windows support for the worker route; the extension still builds there and always scores locally. + +## Enable + +```sh +make -C ann +ann/ann-worker --backend mock-gpu --listen /var/run/postgresql/ann.sock +``` + +```sql +LOAD 'vector'; +SET ann.enable_filtered_search = on; +SET ann.worker_socket = '/var/run/postgresql/ann.sock'; -- superuser only +EXPLAIN (ANALYZE) SELECT id FROM documents WHERE tenant = 7 +ORDER BY embedding <-> '[...]', id LIMIT 10; +``` + +The worker must run on the database host as a user that can create sockets where the server can connect. Relative socket paths resolve against the data directory. + +| Setting | Default | Meaning | +| --- | --- | --- | +| `ann.worker_socket` | empty (off) | Worker Unix socket. Superuser-only because backends send row vectors to it. | +| `ann.worker_chunk_rows` | 4096 | Eligible rows per worker request. | +| `ann.worker_pipeline_depth` | 4 | Requests a scan keeps in flight. | +| `ann.worker_shared_memory_mb` | 64 | Per-backend segment for chunk vectors; 0 sends vectors over the socket. Superuser-only. | +| `ann.worker_timeout_ms` | 30000 | Time without worker progress before the scan scores locally. | + +| Worker option | Default | Meaning | +| --- | --- | --- | +| `--backend` | `cpu-exact` | `cpu-exact` or `mock-gpu` (`cuvs` in the separate cuVS build). | +| `--lanes` | online CPUs | Concurrent backend submissions. | +| `--max-batch` | 64 | Jobs per submission. Lanes leave queued work for idle lanes. | +| `--delay-us` | 0 | Optional wait to fill a partial batch while no lane is idle. | +| `--max-pending-mb` | 512 | Admission budget for queued and running work across all connections. | + +`ANN_MOCK_GPU_LAUNCH_US` (default 100) and `ANN_MOCK_GPU_GBPS` (default 12) tune the mock device. + +## Execution + +```mermaid +flowchart LR + Child[Predicate child plan] --> Rows[Visible qualified rows] + Rows -->|vector into shared slot,<br>row without vector into chunk| Chunk[Chunk of N rows] + Chunk -->|offset + query, async| Worker[ann-worker lanes] + Worker -->|"rows within tolerance of chunk top bound"| Rerank[Re-score with SQL operator] + Rerank --> Sort[Bounded tuplesort] + Sort --> Limit[LIMIT / OFFSET] + Worker -. error, overload, timeout, exit .-> Local[Score that chunk locally] + Local --> Sort +``` + +The route applies only when `LIMIT + OFFSET` is a positive bound no larger than half a chunk, and the query vector is not NULL. Otherwise the scan uses the existing exact CPU path. + +1. The child plan runs normally, so visibility, row-level security, rechecks and quals are unchanged. +2. Each eligible row is split: its vector goes into this chunk's slot of a POSIX shared-memory segment that the worker maps read-only, and the rest of the row is kept as a minimal tuple without the vector. NULL vectors sort last, so only the first `bound` of them are kept. Vectors whose dimensions differ from the query go through the SQL operator, which raises its usual error. +3. A full chunk is sent as a request carrying an offset and the query. The scan keeps reading child rows while up to `ann.worker_pipeline_depth` requests are outstanding. Replies may arrive in any order. Ready replies are collected without blocking. If the pipeline is still full, the backend scores the new chunk itself instead of waiting. +4. For each chunk, the worker returns every row whose distance is within a 1e-3 relative tolerance of the chunk's `bound`-th distance, including ties. Each row in the global top `bound` is also in its own chunk's top `bound`, so only returned rows can matter. The backend rebuilds those rows, recomputes distances with the SQL operator, and feeds them to the bounded tuplesort. Secondary sort keys, ties and NULL ordering therefore follow normal SQL semantics. The tolerance covers the gap between the worker's float sums and PostgreSQL's own rounding for up to 2,000 dimensions. +5. Overload, request errors, malformed replies, timeouts, a lost connection or a worker exit make the affected chunks score locally. After a transport failure, the rest of that scan stays local. Results stay exact in every case. + +Each backend keeps one worker connection and segment across scans. The segment is unlinked as soon as the worker attaches, so it disappears when either process exits. A scan that errors out leaves replies outstanding, so the next scan reconnects. A scan nested inside another scan (for example, through a function in the predicate) scores locally, so it never reuses the outer scan's slots. Worker threads never call PostgreSQL APIs. + +EXPLAIN reports `Search Route: worker pipeline after predicates` with `Worker Chunks`, `Local Chunks` (fallbacks and chunks scored locally while the pipeline was full) and `Worker Candidates` when the route ran. `Distance Evaluations` then counts only the backend re-scores. + +## Protocol additions + +Operations 7–10 are served only in `--listen` mode, together with 0 (capabilities), 5 (probe) and 8 (statistics); index operations remain in stdin mode. Headers keep the version 1 layout, and the generation field is zero. + +| Operation | Request | Success result | +| --- | --- | --- | +| 7 gathered search | u32 rows, dimensions, k, metric; FP32 query; FP32 row-major vectors | u32 count, `(u32 row, FP64 distance)` sorted by distance | +| 8 statistics | Empty | JSON: jobs, batches, max batch, rows, rejected, failed, pending bytes | +| 9 attach shared | Segment name `/pgann.*` | Empty; maps it read-only for this connection | +| 10 shared search | u32 rows, dimensions, k, metric; u64 byte offset; FP32 query | As operation 7 | + +Status 7 (overloaded) is returned without queueing when the admission budget would be exceeded. The client must not modify a shared range until its reply arrives. + +## Measured throughput + +Run on a 10-core Apple M-series laptop with PostgreSQL 17.6 and one `mock-gpu` worker (default options). Data: 100,000 rows, 256 dimensions, a 25% tenant filter (25,000 eligible rows), `LIMIT 10`, random query vectors, `pgbench -M prepared` with sleep disabled. A macOS background daemon used up to about 3 cores intermittently during these runs, so local and worker runs were interleaved: 7 rounds of 4 s each, reporting the median. + +| Clients | Local exact (tps) | Worker pipeline (tps) | Change | +| --- | --- | --- | --- | +| 1 | 86 | 113 | +33% | +| 4 | 246 | 292 | +19% | +| 8 | 359 | 373 | +4% | +| 16 | 361 | 390 | +8% | + +At low concurrency, pipelining overlaps the predicate scan with scoring on other cores. Measured alone, a query costs 11.7 ms of CPU locally against 9.2 ms on the worker route (8.2 ms in the backend plus 1.0 ms in the worker). The backend sorts about 1% of the rows instead of all of them. Under full load both routes cost more CPU per query, and the margin narrows to the single digits because the mock still scores on the host. + +Single uninterleaved runs under that background load varied by up to 25% between consecutive runs and sometimes showed the worker route behind. Use interleaved medians when comparing routes on a shared machine. An adaptive router that chose routes by measured elapsed time per row was tried and removed: it did worse than always using the worker, because elapsed time hides CPU used by other processes and exploration scans cost throughput. + +Changes that each made a measurable difference during development: + +- **Shared-memory vectors.** Sending about 25 MB per query through a Unix socket capped throughput near 210 tps; shared memory removed that ceiling. +- **Vectorizable float kernel with linear selection.** This replaced a full heap and sort over every row. +- **Fair lane batching.** Previously one lane could take the whole queue and run it serially while other lanes sat idle. +- **Chunk tuples without the vector.** The vector is copied once, into the shared slot, instead of twice. +- **Score locally instead of waiting.** When the pipeline is full, the backend scores the chunk itself instead of blocking, so a busy worker sheds load to its clients. + +At 64 dimensions, where scoring is cheap, the worker route is slightly slower than local scoring (`benchmarks/ann/pipeline.sh 20000 64`). + +## Tests and benchmark + +```sh +make -C ann check # includes test/ann/test_server.py +make installcheck # unavailable-worker fallback in ann_filtered +ANN_PG_DSN='host=... dbname=...' python3 test/ann/test_pipeline.py +ANN_PG_DSN='host=... dbname=...' benchmarks/ann/pipeline.sh 100000 256 +``` + +- `test_server.py` covers pipelined out-of-order replies across connections, batching under a slow mock launch, boundary ties, invalid frames, admission rejection and disconnects with pending work. +- `test_pipeline.py` runs through `psql`, and its server must be local. It compares the worker route with local scoring over 75 query/limit/offset combinations, and covers the socket transport without shared memory, overload fallback, timeout fallback, and a worker killed mid-scan. + +Remaining work: validate on a real cuVS host, fit the route decision to measured costs instead of the fixed half-chunk rule, supervise the worker process from PostgreSQL, and run sustained-load and fault tests on Linux. diff --git a/docs/cuvs-filtered-ann-plan.md b/docs/cuvs-filtered-ann-plan.md new file mode 100644 index 000000000..337f0f339 --- /dev/null +++ b/docs/cuvs-filtered-ann-plan.md @@ -0,0 +1,731 @@ +# cuVS bulk build and filtered ANN implementation plan + +Status: CPU worker and batching/admission runtime are implemented in C in [ann/](../ann/README.md). Frozen-table PostgreSQL behavior is validated by test-only Python fixtures; the native C exact L2 AM now provides a storage/lifecycle baseline ([guide](ann-native-index.md)). The opt-in selective exact CustomScan is implemented ([guide](ann-filtered-search.md)); the pipelined native worker route is implemented with a mocked GPU backend ([guide](ann-worker-pipeline.md)). CAGRA integration and compilation against cuVS are deferred todos; existing adapter source is an unvalidated draft. Accelerator generation/delta storage remains proposed. + +Prepared: 2026-10-30. cuVS capability checks below refer to released **v26.10.00**. Compile and runtime smoke tests remain implementation work. + +Companion analysis: [high-throughput filtered-search performance](cuvs-search-performance.md). Section 12 integrates its two optimization tracks into this plan. + +## 1. Objective and scope + +Build a PostgreSQL integration that uses cuVS for bulk vector index construction and query execution while supporting both: + +- Full-text predicates expressed with `tsvector`, `tsquery`, and PostgreSQL text-search configurations. +- Text metadata predicates such as tenant equality, categories, tags, and supported substring searches. + +“Highly selective” means a filter permits only a small fraction of the indexed rows. Text predicates determine eligibility; vector distance determines result order. Combining lexical relevance scores with vector scores is a separate feature and is outside the initial scope. + +The core design is **PostgreSQL eligibility evaluation followed by adaptive vector search over eligible IDs**. For a small matching set, exact search is a first-class execution strategy. For a larger set, use filtered CAGRA search. + +The target SQL shape is ordinary single-table filtered nearest-neighbor search: + +```sql +SELECT id, embedding <=> $1 AS distance +FROM documents +WHERE tenant = 'acme' + AND tags @> ARRAY['database'] + AND search_text @@ websearch_to_tsquery('english', 'query optimization') +ORDER BY distance +LIMIT 20; +``` + +This query illustrates the intended integration, not an existing cuVS execution path. Initial prototypes use an explicit experimental entry point; planner integration follows after correctness and performance are established. + +### Initial assumptions + +- A Linux host with a supported NVIDIA GPU runs the cuVS worker. The PostgreSQL adapter and documentation can be developed separately from GPU execution. +- The first prototype operates on an explicitly immutable dataset with a stable primary key. +- Start with one GPU, dense FP32 vectors, L2 distance, and a supported PostgreSQL major version selected during setup. Add cosine and inner product behind semantic tests. +- Initial prototype inputs are non-null and satisfy the selected metric's validation rules. Unsupported inputs or query shapes must fail explicitly or retain the ordinary PostgreSQL plan. +- Dataset size, embedding dimension, GPU memory, write rate, and latency/throughput targets are not yet specified. Benchmark-derived defaults and hardware sizing remain open decisions. + +### Result contract + +- Every returned row must satisfy the SQL predicates and be visible under the query snapshot. +- Exact routes compute top-k within the eligible set, subject to documented numeric tolerance and tie handling. +- ANN routes approximate membership in top-k; returned candidates are reranked and emitted in distance order. Returning k rows does not establish recall. +- Aim to return `min(k, M)` rows, where `M` is the number of eligible searchable vectors. If ANN underfills, expand the search or use the exact route. A deadline or resource failure must not be reported as proof that no more matches exist. +- PostgreSQL's distance conventions remain authoritative. Normalize or convert cuVS distances as necessary before exposing results. + +## 2. Why the existing scan path needs more than a GPU replacement + +The current HNSW and IVFFlat access methods return candidate heap TIDs. PostgreSQL fetches the heap rows, checks visibility, and applies predicates afterward. Iterative scanning can request additional candidates, but the ANN scan itself does not receive the qualifying text ID set. + +Relevant local code: + +| File | Relevant behavior | +| --- | --- | +| [src/hnsw.c](../src/hnsw.c) | Registers the HNSW access method; currently has no predicate strategies, multicolumn support, or bitmap callback. | +| [src/hnswscan.c](../src/hnswscan.c) | `GetScanItems`, `ResumeScanItems`, and `hnswgettuple` produce and resume ANN candidates. | +| [src/hnswbuild.c](../src/hnswbuild.c) | `table_index_build_scan`, build callbacks, and durable index construction provide patterns for native integration. | +| [src/ivfflat.c](../src/ivfflat.c) | Registers the IVFFlat access method with similar filtering integration limits. | +| [src/ivfscan.c](../src/ivfscan.c) | Selects lists, sorts candidate batches, and emits heap TIDs. | +| [README.md: Filtering](../README.md#filtering) | Documents post-scan filtering and iterative scans. | + +Use the vendor-neutral `ann` access method following the existing HNSW/IVFFlat lifecycle, then an opt-in `CustomPath`/`CustomScan` for supported combined queries. Keep ordinary AM scanning and combined-filter execution separate: the AM takes order-by keys and emits heap TIDs; the custom executor connects PostgreSQL predicate evaluation to the GPU allowlist. Simply creating a GIN index and a cuVS index will not make PostgreSQL perform this connection automatically. + +PostgreSQL describes custom scans as an extension mechanism for alternative execution, including hardware acceleration. Its interfaces require version-specific integration and validation. See [Writing a Custom Scan Provider](https://www.postgresql.org/docs/current/custom-scan.html). + +## 3. Component architecture + +```mermaid +flowchart TB + SQL[Filtered nearest-neighbor SQL] --> Planner[Planner hook and CustomPath] + Planner --> Exec[CustomScan executor] + Exec --> Text[PostgreSQL predicate child plan] + Text --> Indexes[GIN, B-tree, or trigram indexes] + Text --> Eligible[Snapshot-valid eligible row identities] + Eligible --> Mapping[Generation and ordinal mapping] + Mapping --> Router[Per-shard search router] + Router --> CPU[CPU exact search] + Router --> IPC[Bounded worker protocol] + IPC --> Worker[Persistent C worker with vendor C API] + Worker --> Exact[GPU exact search] + Worker --> CAGRA[Filtered CAGRA search] + CPU --> Merge[Merge, recheck, and rerank] + Exact --> Merge + CAGRA --> Merge + Merge --> Results[Ordered results] + Storage[(PostgreSQL durable vectors, mappings, and delta)] --> CPU + Storage --> Worker + Storage --> Merge +``` + +### PostgreSQL backend + +- Own the query snapshot, SQL expression evaluation, permissions, row visibility, cancellation, and final tuple production. +- Evaluate predicates through an ordinary PostgreSQL child plan, retaining required security qualifications. +- Translate row identities into vector ordinals and choose a search route using actual eligible counts. +- Merge base and delta results, recheck as required, compute authoritative distances, and sort before returning rows. + +### Persistent GPU worker + +- Own CUDA contexts, streams, resident indexes, reusable device buffers, and cuVS objects. +- Execute build, load, exact-search, and ANN-search requests through a versioned protocol. +- Use bounded admission queues, memory reservations, request deadlines, and cancellation checks. +- Cancellation stops admission and additional work and discards unwanted results. Already submitted CUDA work may continue; retain buffers, allowlist owners, and index references until stream/event completion before reclaiming them. +- Report structured failures and per-stage timings. A worker crash must not corrupt PostgreSQL state. +- Start CUDA in a dedicated process with a defined lifecycle; do not initialize CUDA in the postmaster or create one independent resident index copy per backend. + +Prefer a local worker for the first deployment. Remote GPU transport adds network latency, authentication, and failure modes and should be a separate milestone. + +### Durable state and GPU cache + +PostgreSQL index relation pages are authoritative for vectors, tuple mappings, generation metadata, and the mutable delta. Use the existing buffer-manager and generic-WAL patterns for recoverable changes. GPU-resident indexes are reconstructible caches. + +Persist graph artifacts as versioned, checksummed caches that can be rebuilt from durable vectors and mappings. An incompatible or missing graph artifact triggers reload/rebuild or an exact fallback; it must not silently remove rows from the searchable dataset. Design the index page/metapage layouts and generic-WAL update boundaries before implementing mutable storage. Introduce custom WAL records only if a demonstrated limitation requires them. + +## 4. cuVS capabilities and binding constraints + +Pin the initial dependency to [cuVS v26.10.00](https://github.com/NVIDIA/cuvs/releases/tag/v26.10.00), along with a compatible CUDA/toolchain configuration. Do not infer released behavior solely from rolling documentation or another language binding. + +| Capability | Implementation choice or constraint | +| --- | --- | +| Bulk ANN construction | Build CAGRA shards; compare NN-Descent and IVF-PQ initial-graph builders. | +| Shared filter | A dense bitset permits or rejects each dataset row; bit 1 means allowed. | +| Different filters per query | Preserve each filter in the C adapter. Use supported C API submissions or the exact CPU route; advanced Roaring batching remains deferred until a suitable vendor C binding exists. | +| C binding | Released CAGRA C search accepts `NO_FILTER` and `BITSET`, and rejects `BITMAP`. | +| Roaring batching | Direct C++ `cagra::search` supports Roaring; cuVS dynamic batching and tiered search reject this filter type. | +| Multi-GPU wrapper | The released C multi-GPU CAGRA search signature has no filter argument. Use explicit filtered shard searches and merge. | +| Exact search | Compare gathered eligible vectors plus brute force with cuVS filtered brute force. Include CPU exact search for tiny sets. | + +Sources: [released CAGRA C implementation](https://github.com/NVIDIA/cuvs/blob/v26.10.00/c/src/neighbors/cagra.cpp), [C++ filter contracts](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/include/cuvs/neighbors/common.hpp), and [multi-GPU C implementation](https://github.com/NVIDIA/cuvs/blob/v26.10.00/c/src/neighbors/mg_cagra.cpp). + +At configuration time, compile smoke programs covering every required capability. At runtime, validate build/search/serialize/reload, filter polarity, ID domains, dtype, dimensions, metrics, invalid-neighbor sentinels, and fewer-than-k behavior. + +Keep graph navigation intact when applying output eligibility. Removing rejected nodes from navigation can disconnect useful search paths. Use cuVS's supported filtering interface rather than adding an ad hoc graph-pruning rule. Filtered CAGRA still has approximate recall. + +## 5. Text eligibility and filter transport + +### Predicate support + +| Predicate | Initial handling | +| --- | --- | +| `tsvector @@ tsquery` | Native PostgreSQL text execution, normally with GIN acceleration. Preserve dictionaries, language configuration, phrases, and weights. | +| Text equality and `IN` | Native comparisons and available B-tree/bitmap paths, preserving collation and null semantics. | +| Tags and array membership | Native array semantics and suitable GIN paths. | +| `LIKE` / `ILIKE` | Ordinary PostgreSQL evaluation; use a supported trigram index when available. | +| Boolean combinations | Let the child plan evaluate the complete supported expression; do not intersect IDs for an `OR` expression. | +| Joins, volatile predicates, complex subqueries | Retain the ordinary plan until a dedicated supported execution contract exists. | + +GIN and bitmap results can require heap and expression rechecks. A raw index bitmap must not be treated as an exact list of visible matches. PostgreSQL's [text-index documentation](https://www.postgresql.org/docs/current/textsearch-indexes.html) explains relevant recheck requirements. + +Require `gin_fuzzy_search_limit = 0` when GIN supplies complete eligibility. Nonzero values can omit matches; reject that configuration for this path or select a complete alternative. Track pending-list behavior under writes as part of throughput testing. [PostgreSQL GIN documentation](https://www.postgresql.org/docs/current/gin.html) + +For the first implementation, materialize the exact qualifying set before vector search. This makes correctness and routing clear but can be expensive for broad predicates. Measure that cost. A later optimization may use a conservative superset with residual rechecks and refill; it requires a separate correctness gate and may never exclude valid candidates because of an approximation in filter construction. + +### ID conversion and representation + +1. Execute the predicate child plan under the query snapshot. +2. Resolve each eligible row to its indexed vector-version identity. +3. Separate IDs by pinned generation, shard, and delta segment. +4. Produce exact local eligible counts. +5. Use sorted sparse IDs for exact gathers, Roaring for selective ANN allowlists, or a dense bitset when appropriate. +6. Validate that every encoded ID belongs to the target shard's ordinal domain. + +A dense mask costs approximately `N / 8` bytes per shard/filter. Distinct dense masks for `Q` queries cost `Q * N / 8`; at 100 million vectors, one mask is 12.5 MB. Reuse buffers and avoid constructing a full dense mask for every sparse request. + +Roaring allowlist owners must outlive all searches referencing their views. Batch compatible queries by generation, shard, metric, dtype, and selectivity. A sparse query can inflate CAGRA's batch-wide filtering estimate, so group disparate selectivities separately. These constraints are specified in the [released C++ filter API](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/include/cuvs/neighbors/common.hpp). + +Cache eligible sets only with an explicit validity model covering generation, predicate parameters, role/security context, relevant data changes, and snapshot visibility. A normalized SQL string or generation ID alone is not sufficient for caching a snapshot-valid allowlist. + +## 6. Bulk build and publication + +```mermaid +flowchart TD + Start[Begin build under supported locking rules] --> Scan[PostgreSQL table index-build scan] + Scan --> Pack[Validate vectors and assign dense ordinals] + Pack --> Durable[Persist canonical vectors and identity mappings] + Durable --> Batch[Pack bounded pinned host batches] + Batch --> GPU[Transfer a memory-budgeted shard] + GPU --> Build[Build and prune CAGRA graph] + Build --> Cache[Write unpublished graph cache and checksums] + Cache --> Check[Reload and validate counts, IDs, and recall] + Check --> Publish[Publish durable generation metadata atomically] + Publish --> Serve[Make generation available to queries] + Scan --> Fail[Abort and clean up unpublished resources] + Build --> Fail + Check --> Fail +``` + +### Build steps + +1. Validate the table, metric, dimension, input constraints, PostgreSQL version, GPU capabilities, and memory budget. +2. For the prototype, freeze writes to the source dataset. For the native AM, use PostgreSQL's index-build APIs and initially support a build that blocks writes. +3. Scan through `table_index_build_scan` for native integration. Do not replace PostgreSQL's index-build visibility rules with an ordinary `SELECT` snapshot: index contents must cover tuple versions that supported snapshots may need. +4. Assign dense ordinals within a generation and shard. Persist the vector and its durable row-version mapping. +5. Pack vectors into contiguous batches and overlap host preparation with transfer where measured to help. Bounded transfers do not make the CAGRA graph builder itself out-of-core; each build shard must fit its peak budget. +6. Build each shard with the selected CAGRA parameters. Record input size, builder, graph degree, metric, dtype, seed where applicable, versions, and timing. +7. Serialize to an unpublished temporary graph artifact. Reload it and validate ID ranges, count consistency, checksums, and sample recall against exact neighbors. +8. Publish the complete generation through PostgreSQL-managed metadata. Partial generations remain unavailable to queries. +9. Release temporary buffers. Retain an older serving generation until its readers and required snapshot coverage permit retirement. + +CAGRA constructs an initial graph and prunes it. Construction workspace can materially exceed the final index size; profile both initial graph builders on representative data. See the [CAGRA build and memory guide](https://docs.nvidia.com/cuvs/user-guide/api-guides/indexing-guide/cagra). + +### Memory planning + +For `N` vectors, dimension `D`, component width `B`, graph degree `G`, and neighbor-ID width `I`: + +```text +Vector bytes = N * D * B +Final graph bytes ≈ N * G * I +One dense filter bytes ≈ N / 8 +Required memory = resident vectors + graph + mappings + filters + + search workspace + build workspace + + overlapping generations + allocator overhead +``` + +For 10 million 768-dimensional FP32 vectors, the vectors occupy 30.72 GB and a degree-64 graph with 32-bit neighbors adds about 2.56 GB, using decimal units. These are component sizes, not a GPU sizing recommendation. Concurrent builds, serving traffic, and old/new generations need additional headroom. + +Start with explicit independently searchable shards. Partition pruning is allowed only when metadata proves that a shard cannot contain a match. Hash sharding generally requires searching every nonempty eligible shard; tenant partitioning helps only when the predicate permits sound pruning. + +## 7. Adaptive query execution + +```mermaid +flowchart TD + Eligible[Exact eligible IDs for one shard] --> Count{Eligible count M} + Count -->|Zero| Empty[Skip shard] + Count -->|Nonzero| Cost[Estimate CPU, GPU exact, and ANN costs] + Cost --> CPU[CPU exact top-k] + Cost --> Exact[Gather eligible vectors and GPU exact top-k] + Cost --> ANN[CAGRA with exact allowlist] + ANN --> Enough{"Valid count reaches min(k, M)?"} + Enough -->|No| Retry{More ANN work within budget?} + Retry -->|Yes| ANN + Retry -->|No| Exact + Enough -->|Yes| Merge[Merge candidates across shards and delta] + CPU --> Merge + Exact --> Merge + Merge --> Sort[Recheck, rerank, and sort before output] +``` + +### Routing model + +Estimate the following using measured coefficients, not fixed universal percentages: + +```text +CPU exact cost = vector access + CPU distance evaluation(M, D) + top-k +GPU exact cost = queue + eligible-vector gather + transfers + + GPU distance evaluation(M, D) + top-k +GPU ANN cost = queue + allowlist construction/upload + + filtered traversal + candidate transfers + reranking +``` + +Include shared predicate evaluation and ID-conversion costs when comparing the complete custom path to an ordinary PostgreSQL plan. Routing should consider `M`, `D`, `k`, residency, eligible-set layout, concurrency, and batch size. + +For example, 0.1% of 10 million vectors leaves 10,000 candidates. Exact search over those candidates may be faster than filtered graph traversal. Gather only eligible vectors, or benchmark cuVS's filtered brute-force implementation; do not assume that filtering after a full distance matrix gives the same cost. NVIDIA recommends brute force for heavily filtered workloads in its [brute-force guide](https://docs.nvidia.com/cuvs/user-guide/api-guides/indexing-guide/brute-force). + +### End-to-end request sequence + +```mermaid +sequenceDiagram + participant Client + participant PG as PostgreSQL backend + participant Filter as Predicate child plan + participant State as Durable index state + participant GPU as GPU worker + Client->>PG: Filtered ORDER BY distance LIMIT k + PG->>State: Pin generation and establish delta coverage + PG->>Filter: Execute predicates under query snapshot + Filter-->>PG: Eligible row identities + PG->>State: Resolve identities to shard ordinals and delta entries + State-->>PG: Eligible IDs, counts, and coverage metadata + PG->>PG: Choose route for each shard + alt CPU exact route + PG->>State: Read eligible vectors + State-->>PG: Vectors + PG->>PG: Compute exact top-k + else GPU route + PG->>GPU: Generation, queries, allowlists, budgets, deadline + GPU-->>PG: Candidate ordinals, distances, status, timings + end + PG->>State: Search required delta and resolve returned identities + PG->>PG: Recheck, refill if needed, merge, rerank, sort + PG-->>Client: Ordered matching rows + PG->>State: Release generation pin +``` + +### Ordering, refill, and planner restrictions + +- Materialize the candidate set before advertising ordered output. Retry batches can discover closer candidates than earlier batches. +- Treat CAGRA search as a top-k request, not a resumable pgvector `ResumeScanItems` cursor. Expanding the candidate/search budget launches another search; deduplicate the accumulated candidates by generation, shard, and ordinal, and materialize their union before final ordering. +- Match distance semantics: cuVS squared L2 may require conversion for pgvector L2 output; cosine normalization and inner-product sign conventions require tests. Final reranking should use the same defined metric as the SQL operator. +- Discard invalid-neighbor sentinels before ID lookup. Handle `M < k`, duplicates, equal distances, and empty shards explicitly. +- If every shard search is exact and filtering is complete, each shard's local top-k is sufficient for exact global top-k. ANN searches need measured oversampling and recall validation. +- Search all shards that might contain eligible rows. Query base and delta without gaps, and deduplicate overlapping index-entry identities during transitions. +- Push `LIMIT` into the custom execution only for proven query shapes. OFFSET, joins, residual qualifications, security qualifications, and parameterized plans require explicit handling; otherwise preserve the standard plan. +- Keep a sort above the custom scan until its path-ordering claim is verified. Do not label an approximate ANN result as exact because its final distances are sorted. +- For broad predicates, allow the planner to choose another path when exact eligibility materialization dominates total cost. + +## 8. Identity, MVCC, and mutable operation + +### Prototype identity + +On an immutable dataset, a stable SQL primary key can map to a dense ordinal. This shortcut is valid only while the dataset is frozen. Reusing a primary key after deletion or storing multiple vector versions requires production identity handling. + +Enforce immutability at the table level. For the experimental build/query test window, a dedicated coordinator can hold a PostgreSQL `SHARE` table lock on each source table; release or loss of that lock invalidates the prototype's registration. A copied source table still needs protection against writes and rewrites. Checking only an index insert callback is insufficient: DELETE, TRUNCATE, and HOT updates can bypass that callback. This long-held lock is a test-only mechanism, not the mutable production design. + +### Production identity + +Use `(index generation, shard, ordinal)` for GPU references. Ordinals are never reused within a generation. The durable mapping points to the appropriate indexed tuple/version lineage. + +PostgreSQL heap identity requires special care: + +- A filter scan may return a visible HOT-chain member, while the index entry refers to the HOT root. +- Resolve the root/member relationship under appropriate table/buffer synchronization, or preserve root identity while rechecking the child-plan results. +- Coordinate ordinal retirement with VACUUM and heap TID reuse. A stale GPU hit must never resolve to an unrelated row that later occupied the same heap location. +- Initially support heap tables only. Other table access methods need their own identity and fetch contract. + +### Immutable base plus durable delta + +Capture inserts and indexed updates synchronously in PostgreSQL-managed WAL-protected state. GPU refresh is asynchronous; search visibility cannot depend on refresh completing. + +The delta records vector entries and their durable identities. Non-HOT indexed updates create new entries. Deletes need not immediately erase graph nodes: heap visibility rejects deleted versions when appropriate, while old snapshots may still require them. Physical cleanup follows PostgreSQL's safe reclamation horizon. + +Queries must include committed changes visible to their snapshot and the transaction's own visible writes. WAL order, cache freshness, and a global live/dead bitset are not substitutes for MVCC. + +### Generation lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Building + Building --> Validating: Durable vectors and mapping complete + Building --> Aborted: Build fails or transaction aborts + Validating --> Ready: Checks and cache validation pass + Validating --> Aborted: Validation fails + Ready --> Serving: Atomic durable publication + Serving --> Retiring: Replacement generation published + Retiring --> Reclaimed: No readers and safe reclamation horizon + Aborted --> Reclaimed: Unpublished resources cleaned up + Reclaimed --> [*] +``` + +Publication must define an explicit coverage boundary between the base and delta. An LSN or entry-sequence watermark can identify copied data, but visibility still comes from PostgreSQL. Concurrent and long-running transactions must not fall through a gap at that boundary. + +Compaction must preserve versions required by supported snapshots. Retain old base/delta coverage until it is safe to retire it; reader pins alone do not prove that old tuple versions can be discarded. Specify and test these invariants before enabling mutable tables or concurrent rebuilds. + +### Recovery and availability + +- Persist canonical vector and mapping changes through PostgreSQL's recovery machinery. Graph cache files alone do not provide transactional durability. +- On restart, validate metadata and artifact checksums before loading a graph. Rebuild an incompatible or missing graph from canonical state. +- Recover or clean up unpublished generations without exposing partial data. +- Define backup/restore and replica behavior. GPU-less replicas should retain a correct exact fallback or report a documented unavailable path, rather than silently using stale data. +- Tag every worker request and response with generation identity and validate it before consuming results. +- If the worker fails, release request resources and run an authorized fallback within the query deadline, or return an explicit error. + +## 9. Repository architecture and code-style alignment + +The PostgreSQL adapter should look like another pgvector access method. Keep the genuinely new pieces, filtered CustomScan execution and the accelerator worker, behind narrow interfaces. Vendor selection belongs in the worker's small build/search/destroy interface, not a general PostgreSQL backend registry, new storage engine, or alternate text-query language. + +### Multiple vendors and precision paths + +The worker must support independently versioned implementations such as NVIDIA cuVS, AMD hipVS, and a fused NVFP4 scoring path. The initial [AnnBackend interface](../ann/backend.h) provides a statically selected worker-local boundary, with a tested CPU reference implementation. Separate worker processes own each vendor/device context. PostgreSQL continues to own predicate semantics, tuple identity, durability, and final results. + +Each adapter must advertise and validate its algorithm, metrics, input/storage precision, filter formats, batch/dimension limits, exactness, and serialization compatibility. cuVS capability findings in this document do not establish hipVS capability parity. AMD describes hipVS as aligned with cuVS 25.10, rather than the 26.10 release used here. [AMD hipVS documentation](https://rocm.docs.amd.com/projects/hipVS/en/latest/) + +NVFP4 denotes a precision/kernel path, not a complete ANN access method. Establish its kernel API, scaling/packing format, filtering support, top-k contract, and device requirements before implementing an adapter. Preserve canonical vectors for reranking; compressed scoring is not automatically exact. An adapter offering scoring only must not advertise graph build or native filtered ANN. + +Bind native artifacts to backend, library/format version, algorithm, precision/scales, metric, and generation. Include these fields in batching/cache compatibility. Never reinterpret one vendor's graph or compressed vectors as another's. Unsupported backend selections fail explicitly; CPU fallback must be a declared route, not silently presented as GPU execution. + +The `cuvs` AM/GUC/module names below remain provisional design names. Choose one common PostgreSQL-facing name before adding shipped SQL; do not duplicate the AM lifecycle, text executor, or durable identity model for each vendor. The implemented portable worker lives in `ann/`, superseding the earlier proposed `cuvs/` directory. + +### Existing conventions to follow + +| Repository reference | Apply to the cuVS implementation | +| --- | --- | +| [.editorconfig](../.editorconfig) | Tabs with width 4 for C, headers, Perl, and SQL. Match neighboring PostgreSQL-style code. | +| [src/vector.c](../src/vector.c) | One extension entry point, `_PG_init()`, with a new `AnnInit()` alongside `HnswInit()` and `IvfflatInit()`. | +| [src/hnsw.c](../src/hnsw.c) | Handler registration, bounded reloptions, GUC definitions, costing, opclass validation, and version guards. | +| [src/hnsw.h](../src/hnsw.h) | AM-specific constants, support procedures, page layouts, state structs, and shared helper declarations. | +| [src/hnswscan.c](../src/hnswscan.c) | Begin/rescan/gettuple/end callbacks, opaque scan state, and scan-local memory ownership. | +| [src/hnswbuild.c](../src/hnswbuild.c) | Build state, table-AM scan callback, progress, index forks, and build-time page WAL. | +| [src/hnswinsert.c](../src/hnswinsert.c), [src/hnswvacuum.c](../src/hnswvacuum.c) | Durable insertion, vacuum callbacks, buffer locks, and resource cleanup. | +| [src/hnswutils.c](../src/hnswutils.c) | Opclass support lookup, metapage/page helpers, and generic WAL updates. | +| [Makefile](../Makefile), [Makefile.win](../Makefile.win) | PGXS/Windows C builds, SQL generation, installation, and existing test entry points. | +| [sql/vector.sql](../sql/vector.sql) | Canonical install SQL, handlers, access methods, operators, and opclasses. | + +### AM scanning and CustomScan are separate paths + +```mermaid +flowchart TD + Normal[PostgreSQL IndexScan] --> AM[annscan.c: AM callbacks] + Combined[Opt-in combined-filter CustomScan] --> Custom[anncustom.c: executor state] + Custom --> Predicate[PostgreSQL predicate child plan] + Predicate --> Filter[annfilter.c: eligible ordinal sets] + Filter --> Search[Typed Cuvs search and generation helpers] + AM --> Search + Search --> Pages[annutils.c: index pages and support procedures] + Search --> CPU[Exact CPU route] + Search --> Client[annworker.c: bounded IPC] + Client --> GPU[Separate C worker and vendor SDK] + Build[annbuild.c] --> Pages + Insert[anninsert.c: durable delta append] --> Pages + Vacuum[annvacuum.c: PostgreSQL vacuum callbacks] --> Pages +``` + +- The ordinary AM implements `annbeginscan`, `annrescan`, `anngettuple`, and `annendscan` with `AnnScanOpaqueData`. Read the scan descriptor's order-by keys and return `xs_heaptid`; set recheck flags according to the actual distance/qual contract. This path does not receive arbitrary WHERE expressions or the query's LIMIT. +- The custom executor has a distinct `AnnCustomScanState`. It owns its predicate child plan, eligible sets, safe top-k pushdown, and final row production. Do not pass executor state through AM-private fields or assume that a normal scan has the custom path's information. +- Share narrowly typed helpers such as `AnnSearch` and generation lookup. The inputs describe the metric, query vector, optional eligible ordinals, and search budget; results contain generation-scoped candidates and status. An opaque PostgreSQL `PlanState`, CUDA pointer, or C++ object is not a cross-process request format. +- Keep HNSW/IVFFlat scans and their on-disk formats independent. Reuse public vector representations and distance semantics; do not make cuVS depend on HNSW graph internals. + +### Proposed modules + +The basic AM files now exist as the exact CPU staging implementation. CustomScan, IPC, and accelerator storage responsibilities below remain proposed. Add files as their responsibilities are implemented, rather than creating empty layers in advance. + +| Proposed location | Responsibility | +| --- | --- | +| `src/ann.c`, `src/ann.h` | `AnnInit`, `annhandler`, options, support constants, state structs, and shared declarations. | +| `src/annbuild.c` | `annbuild`, `annbuildempty`, table build callback, packing, ordinal assignment, and publication. | +| `src/annscan.c` | Ordinary AM begin/rescan/gettuple/end callbacks and `AnnScanOpaqueData`. | +| `src/anninsert.c` | `anninsert`, validation, synchronous delta append, and refresh notification. | +| `src/annvacuum.c` | `annbulkdelete`, `annvacuumcleanup`, safe reclamation, and vacuum statistics. | +| `src/annutils.c` | Support-procedure lookup, index page/metapage helpers, generic WAL, mapping and generation access. | +| `src/annplan.c` | Opt-in planner hooks, supported-shape detection, costing, and CustomPath creation. | +| `src/anncustom.c` | CustomScan callbacks, predicate child plan, routing, merge, reranking, and row production. | +| `src/annfilter.c` | Eligible-ID conversion, shard grouping, and portable filter encoding. | +| `src/annworker.c` | Portable C client protocol, request ownership, availability checks, and error translation. | +| `ann/` | Standalone worker protocol, CPU reference, client, and future isolated cuVS/hipVS/precision adapters. | +| `test/sql/`, `test/expected/`, `test/t/` | Regression pairs and TAP tests following the existing harness. | +| `benchmarks/cuvs/` | Reproducible bulk-build and filtered-query benchmarks, separate from regression expectations. | + +The delta is an index data structure maintained by insertion, vacuum, and generation helpers. Start within these AM modules. Split out a dedicated storage or delta file only when real implementation size and ownership justify it; do not create a second transactional subsystem. + +### C style and state ownership + +- Put `#include "postgres.h"` first in backend C files. Follow neighboring include grouping, brace placement, declarations, and short `/* ... */` comments explaining intent or invariants. +- Use lowercase AM callbacks (`annbuild`), `Cuvs`-prefixed shared helpers and types (`AnnBuildState`), uppercase constants (`ANN_*`), and `static` file-local helpers (`BuildCallback`, `InitBuildState`). Keep headers private unless an external consumer needs them. +- Use the existing handler export convention, including `FUNCTION_PREFIX PG_FUNCTION_INFO_V1(annhandler)`. Match local `PG_VERSION_NUM` guards where PostgreSQL callback signatures or AM fields differ. +- Allocate backend state with PostgreSQL allocation APIs and build/scan/vacuum memory contexts. Reset `tmpCtx` between appropriate batches or tuples. Use checked size arithmetic such as `add_size` and `mul_size` for vector, ordinal, and filter buffers, including 32-bit builds. +- Use `ereport` with suitable SQLSTATE/detail/hint for user-facing failures and the existing `elog` style for internal invariants. Place `CHECK_FOR_INTERRUPTS()` in long CPU loops and use interruptible PostgreSQL wait facilities for worker requests. +- Release generation pins, external handles, and requests on rescan, normal end, ERROR, cancellation, and transaction abort through appropriate PostgreSQL resource/error cleanup. Memory-context deletion alone does not finish external GPU work. +- Keep project runtime code in C and use vendor C APIs. SDK implementation details, CUDA headers, and cuVS resource ownership stay behind the worker backend boundary. C++-only SDK features remain deferred unless a vendor C binding becomes available. Return structured errors over the protocol. Define fixed-width wire fields and explicit encoding; do not serialize raw PostgreSQL structs or process pointers. +- `.editorconfig` currently does not define `.cpp` or `.cu` formatting. Add a scoped worker formatting policy when introducing those files; do not reformat existing C sources as part of the integration. + +### Initialization, SQL semantics, and configuration + +`AnnInit()` should register bounded index options and `cuvs.*` GUCs and reserve the GUC prefix. It must not connect to a worker, create a CUDA context, allocate large queues, or require a worker merely to load `vector`. + +Use reloptions for persistent build properties such as graph degree, and GUCs for per-query search budgets and experimental planner enablement. Derive backend CPU budgets from appropriate PostgreSQL settings such as `work_mem` and `maintenance_work_mem`, with separate explicit GPU/queue limits. Do not imitate HNSW's tiny shared-memory tranche-ID allocation for substantial worker queues; shared memory would require proper sizing and lifecycle registration. + +Planner hooks are new to this repository. Make installation deliberate, chain any previous hooks, and keep combined-filter planning opt-in initially. Establish the enabled state before a statement is planned; an AM handler loaded only during execution is too late. Test fresh sessions, prepared plans, and disabled/unsupported cases. Do not require `shared_preload_libraries` for ordinary pgvector use; any future worker-supervision preload requirement is an explicit deployment option. + +Read distance and normalization support through the index opclass, following `HnswInitSupport`. Map only verified supported procedures to GPU metrics; unfamiliar opclasses must use a semantically valid CPU path or be rejected. Do not hard-code operator OIDs or infer a metric from an unqualified SQL name. Reuse the existing `vector` type and operators, with cuVS-specific opclass entries such as `vector_l2_ops ... USING cuvs`. + +Advertise only implemented AM capabilities. Initially keep multicolumn, INCLUDE, bitmap, parallel scan, and parallel build support disabled unless implemented and tested. GPU parallelism is not PostgreSQL parallel-index-scan/build support. GPU failures and exact fallbacks must preserve the declared SQL/operator contract. + +### Storage, WAL, and locks + +- Give the cuVS index its own versioned metapage and page/tuple layouts. Keep canonical vectors, ordinal mappings, delta entries, and generation metadata in index-owned storage. +- Follow `ReadBuffer`/`ReadBufferExtended`, buffer locks, `GenericXLogStart`/`GenericXLogRegisterBuffer`/`GenericXLogFinish`, and `UnlockReleaseBuffer` patterns for updates. Follow build-time `log_newpage_range` and `RelationNeedsWAL` handling where appropriate. Prefer PostgreSQL generic WAL to a new custom resource manager. +- Use `table_index_build_scan`, return `IndexBuildResult` counts, and report progress through the existing build-phase conventions. Handle `MAIN_FORKNUM` and `INIT_FORKNUM` deliberately; implement unlogged/temporary behavior correctly or reject it explicitly during the experimental phase. +- Have `anninsert` append durable entries before an asynchronous refresh can observe them. Have `annbulkdelete` use PostgreSQL's deletion callback to determine reclaimable heap identities, with `annvacuumcleanup` reporting statistics. A worker's view of live rows does not replace these callbacks. +- Copy bounded batches under the necessary buffer locks, pin the logical generation, release buffer content locks, then submit GPU work. Never wait for IPC, queue admission, or GPU completion while holding buffer content locks or inside a critical section. +- Specify lock order for publication, insertion, and vacuum. Keep these logical locks and generation pins distinct from short-lived buffer locks and from the worker's device-reference lifetime. +- Invalidate cached identities on relevant DROP, TRUNCATE, REINDEX, and relation-rewrite transitions. Cache keys must include database, physical index identity, and generation; an index OID alone is insufficient. The worker must not modify PostgreSQL relation files or act as an independent WAL writer. + +### Build, installation, and upgrade contract + +The chosen packaging model is a **portable C adapter in `MODULE_big = vector`, with a separately built optional GPU worker**. Every symbol referenced by installed SQL exists in CPU-only and GPU installations. Unavailable worker operations return a documented error or a supported exact fallback; they do not cause `CREATE EXTENSION` to fail because a handler was omitted from the library. + +1. Add portable adapter objects to both `Makefile` and `Makefile.win`. Keep CUDA headers, C++ linking, and cuVS package discovery out of the top-level PGXS build. Use portable unavailable-worker behavior where the initial Linux transport is unsupported. +2. Build the worker through an explicit subdirectory build or separate target. Ordinary `make`, installation, tests, and uninstall must not download or require CUDA. Worker lifecycle/supervision is explicit and separate from loading the extension. +3. Add shipped SQL definitions to canonical `sql/vector.sql` and the appropriate future upgrade script. Do not edit only generated `sql/vector--<version>.sql`, which the Makefile copies from the canonical source. +4. Follow the existing handler, access-method, opclass, and COMMENT conventions. Preserve extension membership and relocatability; do not assume objects live in `public`. +5. Test fresh installation and upgrade for identical catalogs, with and without a worker, and in a non-default extension schema. Keep experimental entry points out of release SQL until their supported behavior is defined. +6. Coordinate release versions in both Makefiles, `vector.control`, and `PG_MODULE_MAGIC_EXT` when the feature ships, and update the control-file description then. The checkout currently declares 0.8.7 and includes a draft 0.8.7-to-0.9.0 upgrade; this plan does not select or bump the release. + +Conditionally removing handler objects while installing their SQL is not this design. A separately packaged extension would require its own control/install/upgrade lifecycle and a deliberate revision of this packaging decision. + +### Tests and CI fit + +- Add deterministic regression SQL and matching expected output, such as `test/sql/cuvs.sql` and `test/expected/cuvs.out`, for catalogs, options, supported CPU behavior, and unavailable-worker errors. Configure a controlled unavailable endpoint for those error cases rather than relying on the host having no worker. The Unix Makefile auto-discovers SQL tests; Windows has an explicit `REGRESS` list that must be updated. +- Use numbered TAP tests with `PostgreSQL::Test::Cluster`, `safe_psql`, `Test::More`, and `done_testing()` for planner, lifecycle, recovery, recall, and worker failures. Follow [the existing WAL test](../test/t/010_hnsw_wal.pl) and [iterative recall test](../test/t/044_hnsw_iterative_scan_recall.pl). +- Ordinary regression tests must run without GPU hardware. Gate GPU-specific TAP tests with an explicit opt-in and `plan skip_all` on ordinary hosts; the dedicated GPU CI mode must fail, rather than skip, when its required worker/device is missing. +- Preserve the [current CPU workflow](../.github/workflows/build.yml): Linux/ARM, macOS, Windows, i386, assertions, warnings-as-errors, static analysis, and Valgrind. Add a separate supported-Linux GPU job with pinned cuVS/CUDA. Limiting the first custom/GPU path to one PostgreSQL major must not break the ordinary extension's existing build matrix. +- Use `make installcheck` and `make prove_installcheck`, including existing `REGRESS` and `PROVE_TESTS` selection when developing a focused change. Keep performance benchmarks and GPU latency assertions out of deterministic expected files. + +## 10. Implementation steps and delivery gates + +```mermaid +flowchart LR + P0[0: Capability and workload baseline] --> P1[1: Standalone GPU worker] + P1 --> P2[2: Immutable PostgreSQL prototype] + P2 --> P3[3: Native AM and CustomScan] + P3 --> P4[4: Transactional mutable indexing] + P4 --> P5[5: Multi-GPU and concurrent builds] +``` + +### Phase 0: Pin dependencies and establish the baseline + +- [ ] Record target PostgreSQL major version, hardware, row counts, dimension, metrics, write rate, `k`, and latency/throughput/recall objectives. +- [ ] Pin cuVS, CUDA, compiler, and worker build configuration; confirm that the portable C adapter fits PGXS builds without a CUDA dependency. Windows is not a supported worker or accelerator platform. +- [ ] Review the AM callback boundaries, naming, state ownership, page layout, and proposed SQL objects against the repository conventions in section 9. +- [ ] Compile and run capability smoke tests for bulk build, reload, bitsets, per-query Roaring, exact search, and result edge cases. +- [x] Generate representative predicates for both full-text and metadata, including their conjunctions (`benchmarks/ann/postgres.py`; executed on the temporary PostgreSQL 17.6 test cluster). +- [ ] Compute exact filtered ground truth and benchmark current pgvector iterative HNSW and PostgreSQL filter-then-exact plans. +- [ ] Measure the CPU eligibility/mapping ceiling with a diagnostic no-compute worker and record raw candidates, visited heap pages, and final eligible counts. + +Deliverable: reproducible harness, capability report, and workload manifest. Gate: the selected API combination runs on the target machine and ground truth is independently validated. + +### Phase 1: Standalone build/search worker + +- [ ] Complete CAGRA integration (deferred; existing adapter source is a draft). +- [x] Compile/link against cuVS 26.10 and validate on a supported GPU host (8× B200, `make -C ann check-cuvs`). +- [x] Define fixed-width request/response encoding and generation-scoped ID types without leaking PostgreSQL structs or C++ objects across the process boundary (`ann/worker.c`, `test/ann/support/client.py`; serialized reference transport). +- [x] Implement bounded vector ingestion, CAGRA shard build, serialization, load, and unload: offline `scripts/ann/build_cuvs_index.py`, immutable directories, multi-GPU loading ([guide](ann-cuvs-index.md)). Unloading requires a worker restart. +- [ ] Implement CPU exact reference, GPU gathered exact, GPU filtered brute force, and filtered CAGRA search. +- [ ] Implement same-filter shared-bitset batches and per-query Roaring batches with explicit ownership and range validation. +- [x] Add per-shard routing, result merge, memory accounting, timings, deadlines, and cancellation behavior: shard-per-GPU search, pairwise tree merge, shared admission budget, client timeouts. +- [ ] Define request cleanup after PostgreSQL errors and GPU stream completion; verify that submission never requires retaining a backend buffer content lock. +- [ ] Fit preliminary route crossovers from complete search costs, including filter setup and transfers. +- [ ] Add compatible direct-search microbatches, reusable buffers, and admission by estimated work and bytes; sweep batch delay and submission lanes at fixed recall and latency. CPU ready-query microbatching, request/byte/work admission, deadline/cancel handling, and a burst benchmark are implemented in `ann/batching.c`; Server mode adds cross-connection lanes, fair batch splitting and shared-memory request buffers with a mocked device; real GPU buffers and full sweeps remain pending. + +Deliverable: worker executable and benchmark reports. Gate: all returned IDs satisfy the supplied allowlist, exact results match the reference, and ANN recall is measured after reload as well as immediately after build. + +### Phase 2: Immutable PostgreSQL prototype + +- [ ] Implement the native C table freeze; the test-only Python reference enforces the table freeze for the full build/query window and build stable integer-key-to-ordinal mappings; live tests cover external mutations, context expiry, and failed-build cleanup (`test/ann/support/postgres.py`). +- [x] Implement a C PostgreSQL search entry point. `AnnFilteredSearch` pipelines eligible chunks to the worker ([guide](ann-worker-pipeline.md)); `FrozenIndex` remains a test-only Python SQL reference. +- [ ] Implement combined text/metadata eligibility in the native C PostgreSQL path. Test-only Python fixtures already match SQL exact results. +- [ ] Translate eligible identities to shard IDs; route and merge results; rerank and return rows. Single-generation CPU prototype is implemented; shard/adaptive routing and GPU validation remain pending. +- [ ] Instrument filter evaluation, identity mapping, gathering, queue time, upload, search, download, and reranking. +- [ ] Compare end-to-end performance with existing PostgreSQL plans and current pgvector. +- [ ] Evaluate an explicit batch entry point reusing one predicate result across multiple query vectors; compare CPU savings with cross-backend GPU batching. The frozen-table client now implements shared eligibility batches; throughput comparison remains pending. +- [x] Bound host eligibility materialization before worker admission: scans hold at most `ann.worker_pipeline_depth + 1` chunks plus candidates, and concurrent prepared queries are benchmarked. Compact ID arrays and page-local mapping still apply to the future resident-index route. + +Deliverable: a working combined-query prototype on immutable data. Gate: qualifying-row correctness, exact-route correctness, adequate filtered recall, and a measured benefit on the target workload. If exact search wins for the target's selective queries, retain it as the chosen route. + +### Phase 3: Native access method and planner integration + +Implement and review the ordinary AM lifecycle before layering in the custom executor. Each part should be independently reviewable and testable. + +- [ ] Add `AnnInit` and handler/options/support validation using existing initialization, export, and PostgreSQL-version conventions. +- [ ] Implement build/buildempty callbacks, progress, fork handling, and `IndexBuildResult` using PostgreSQL index-build visibility rules. +- [x] Implement ordinary begin/rescan/gettuple/end callbacks in `annscan.c`, with opaque scan state and PostgreSQL resource cleanup (exact L2 CPU baseline). +- [ ] Implement canonical index pages, mappings, metadata, generic WAL, and atomic generation publication using existing buffer-manager patterns. +- [ ] Resolve existing HOT root/member identities and test pre-existing HOT chains even while writes are frozen. +- [x] Add native AM objects to both Makefiles, canonical install SQL, and matching 0.8.7-to-0.9.0 upgrade SQL; validate fresh install/upgrade without CUDA. The CPU AM has no worker dependency; external-worker availability behavior remains pending. +- [x] Introduce opt-in CustomPath/CustomScan support in separate planner/custom-executor modules, with hook chaining and a documented whitelist of query shapes (exact CPU route; worker integration remains pending). +- [ ] Cost predicate materialization, mapping, transfer, and vector search rather than only the GPU kernel. +- [ ] Calibrate exact-versus-ANN routes and planner startup/spill costs from the performance experiments; expose actual route, batch, queue, and recheck costs. +- [x] Add ordered-output validation, `EXPLAIN` instrumentation, prepared-parameter handling, and fallback selection (worker route: re-scored candidates, per-chunk local fallback). +- [ ] Test cancellation, worker restart, invalid artifacts, transaction abort during build, backup/restore, and GPU-less behavior. +- [ ] Keep table-level freeze enforcement in this stage until Phase 4 passes; an access-method insert callback alone cannot reject every mutation. Test DELETE, TRUNCATE, HOT updates, rewrites, and freeze loss. + +Deliverable: native integration with an explicit supported contract. Gate: correct plan selection, restart/recovery behavior, and no exposure of incomplete generations. + +### Phase 4: Transactional writes and compaction + +- [ ] Extend the baseline root-TID/version mapping to concurrent HOT updates, ordinal retirement, VACUUM, and TID-reuse invariants. +- [ ] Implement synchronous generic-WAL-protected delta append through `anninsert` and snapshot-aware result validation. +- [ ] Search base plus every required delta entry, including the querying transaction's own writes. +- [ ] Implement compaction with explicit coverage boundaries and atomic replacement publication. +- [ ] Implement `annbulkdelete`/`annvacuumcleanup` using PostgreSQL's deletion callback and statistics conventions; preserve old snapshots and active generation readers. +- [ ] Test cache identity invalidation through DROP, TRUNCATE, REINDEX, and table rewrite alongside insertion/vacuum lock ordering. +- [ ] Test aborts, deletes, HOT and non-HOT updates, repeated primary keys, long-running transactions, and crash/replay at publication boundaries. +- [ ] Measure sustained offered-load behavior while GIN pending entries, delta segments, compaction, and retained snapshots compete with search. + +Deliverable: mutable production-capable indexing. Gate: the MVCC and recovery test matrix passes before enabling mutable workloads by default. + +### Phase 5: Capacity and concurrency + +- [ ] Extend explicit shard placement across GPUs, with per-shard local allowlists and global merging. +- [ ] Tune batching by compatible generation, metric, representation, and selectivity without relying on unsupported cuVS wrappers. +- [ ] Add concurrent initial builds/rebuilds only after delta capture, validation, and publication rules are integrated with PostgreSQL's required index lifecycle. +- [ ] Evaluate compression or IVF alternatives only when memory/throughput measurements justify them; rerank compressed candidates with full vectors where required. +- [ ] Stress test queue fairness, shard skew, simultaneous builds and searches, memory pressure, and partial device failures. +- [ ] Compare GPU replicas with filtered shard fanout where capacity permits, and test persistent search separately on stable versus heterogeneous filter traffic. + +Deliverable: a documented capacity envelope and operational tuning defaults. Gate: target latency, throughput, and recall under sustained concurrency and failure injection. + +## 11. Validation and benchmark plan + +### Correctness cases + +- Empty eligible sets; fewer than k matches; duplicate vectors; equal distances; invalid-neighbor sentinels; empty shards. +- Full-text phrases, weights, language configurations, boolean expressions, tags, equality, and supported substring predicates. +- Lossy bitmap rechecks and full-text/metadata conjunctions with very different selectivities. +- L2 squared/unsquared conversion, cosine normalization, inner-product ordering, dimension validation, and supported dtype conversions. +- Correct final order across ANN retries, multiple shards, and base/delta merging. +- Own writes, committed and aborted transactions, old snapshots, deletes, HOT/non-HOT updates, repeated primary keys, and TID reuse. +- Build abort, interrupted publication, worker crash, database restart, replay, artifact corruption, and incompatible serialized versions. +- Unsupported query shapes retaining the ordinary plan; no unsafe LIMIT pushdown or skipped security qualifications. +- Fresh installation versus upgrade, non-default extension schemas, CPU-only symbol availability, and supported PostgreSQL/platform builds. +- AM rescan/end and CustomScan rescan/end cleanup, repeated cancellation, and waits that do not retain buffer content locks. + +Compare exact results with PostgreSQL's CPU distance operators. Define numeric tolerances and tie-aware comparison explicitly. For ANN, calculate recall against exact top-k among qualifying visible rows, never against unfiltered ground truth. + +### Workload matrix + +| Axis | Suggested coverage | +| --- | --- | +| Survivor fraction | 100%, 10%, 1%, 0.1%, 0.01%, 0.001%, and empty. | +| Predicate distribution | Independent, correlated, and anticorrelated with vector neighborhoods. | +| Predicate kind | Full-text only, metadata only, and combined predicates. | +| Dimensions | Target dimensions; initial examples 384, 768, and 1536. | +| Dataset size | Small validation set, then 1M, 10M, and target scale where hardware permits. | +| k | 10, 100, and larger workload-specific values. | +| Residency | Warm resident index, cold load, partial residency, and build/search overlap. | +| Concurrency | Single query, batches, concurrent clients, and mixed filter selectivity. | +| Mutability | Frozen data initially; inserts, updates, deletes, and long snapshots after Phase 4. | +| Sharding | One shard, multiple shards, skewed partitions, and multiple GPUs after Phase 5. | + +Compare CPU exact, GPU gathered exact, GPU filtered brute force, filtered CAGRA, current pgvector iterative HNSW, and ordinary PostgreSQL filter-then-exact execution. Include IVF variants only as measured alternatives. + +### Metrics and proposed EXPLAIN fields + +- End-to-end p50/p95/p99 latency and throughput at fixed concurrency and recall. +- Filtered recall@k, valid returned count, exact-route agreement, and candidate refill frequency. +- Build rows/second, total build time, serialization/load time, peak host memory, and peak VRAM. +- Predicate time, mapping time, gather time, queue time, filter encoding/upload, GPU search, result download, heap recheck, and rerank time. +- Eligible count, selected route, generation, shard count, base/delta coverage, candidate budget, and fallback reason. +- Transfer bytes, filter bytes, cache hit rate, and memory reserved for simultaneous generations. + +Choose route thresholds and default search budgets only after the target workload meets its agreed objectives. Do not publish speedup or latency claims based solely on a warm GPU kernel benchmark. + +## 12. Performance optimization options + +Both options below are part of the implementation plan. They address different bottlenecks and should be evaluated independently before combining them. The detailed capacity model, diagrams, and experiment sequence are in [the throughput analysis](cuvs-search-performance.md). + +### Option A: reduce and reuse text eligibility work + +Optimize PostgreSQL predicate execution, heap rechecks, HOT resolution, and ordinal conversion before increasing GPU capacity. Materialize narrow contiguous ID arrays, group mapping accesses by page where safe, and give the producer an explicit host-memory budget. + +Evaluate a supported batch-search entry point that accepts multiple query vectors sharing one predicate and statement snapshot. It can evaluate eligibility once and reuse the ordinal mapping, device allowlist, and potentially a compacted exact-search dataset. Independent SQL statements still require their own correct snapshot handling; cross-backend GPU batching alone does not amortize this CPU work. + +Decision criteria: prefer this work when predicate/mapping CPU or heap I/O limits QPS, or when repeated same-predicate query vectors are common. Validate it first with the diagnostic no-compute worker and then with full SQL requests. A generation-only filter cache is insufficient because text metadata can change without a vector change. + +### Option B: batch and route vector search efficiently + +Make exact subset search and filtered CAGRA explicit alternatives: + +| Search option | Expected opportunity | Costs to measure | +| --- | --- | --- | +| CPU/GPU exact over eligible vectors | Tiny or selective matching sets; reusable same-filter subsets. | CPU work, gather/read bytes, transfer/queue overhead, and exact top-k. | +| Filtered CAGRA | Larger eligible sets where graph search reduces distance work at the recall target. | Allowlist setup, filter-adjusted search capacity, workspace, retries, and reranking. | + +The router chooses per shard using actual eligible count, dimension, residency, candidate budget, batch type, and load. Do not use one selectivity percentage as a universal threshold. + +Use a C backend interface with compatible microbatches, separate selectivity/work classes, resident vectors/graphs, and reusable buffers. Bound batch delay, workspace, and queued bytes. Reuse resources per submission lane and keep each query's identity/filter mapping intact. Test persistent kernels and cuVS's own batching wrapper as targeted later experiments, not default solutions for arbitrary text filters. + +Decision criteria: prioritize this work when the producer can supply sufficient work and measurements show GPU idle gaps, allocation/transfer overhead, inefficient batches, or costly ANN traversal. Compare whole-query throughput at equal filtered recall; a faster search with lower recall is a different operating point. + +### Shared performance gates + +- Define target successful QPS, p95/p99 latency, filtered recall, and host/device memory limits before choosing defaults. +- Report SQL statements/second and vector searches/second separately, with vectors per statement, full statement latency, and per-vector recall; shared-filter batches change the relationship between those rates. +- Include eligibility, mapping, queueing, transfer, search, delta, heap checks, and result production in measurements. +- Use both concurrency sweeps and open-loop offered-load tests. Report rejected, failed, timed-out, and late requests alongside completed QPS. +- Keep per-tenant fairness and snapshot/generation retention bounded under overload. +- Recheck correctness and performance during writes, compaction, cold loads, and worker failures. + +The priority order is producer-ceiling measurement, compact mapping and safe shared-filter reuse, exact/ANN crossover measurement, then microbatch and concurrency tuning. GPU scaling follows only after these experiments identify a remaining GPU capacity limit. + +## 13. Open decisions before implementation + +1. Target row count, dimension, metrics, filter distributions, and write rate. +2. GPU model/count, VRAM, host memory, local storage, and deployment topology. +3. Latency, throughput, and filtered recall requirements, including exact-fallback behavior under deadlines. +4. Initial custom/GPU execution PostgreSQL major version and supported native table/query shapes, while preserving the repository's broader CPU build matrix. +5. Native index page/metapage layouts, generic-WAL update boundaries, and generation/delta coverage protocol. +6. Shard sizing and whether tenant partitioning can soundly reduce query fanout. +7. Experimental SQL entry point and the opt-in mechanism for planner-selected ANN behavior. +8. Common native AM name, vendor adapter SDK versions/test hosts, and the fused NVFP4 kernel repository/API. + +## 14. Current implementation and next executable gates + +Implemented: standalone C worker, CPU exact top-k with shared/per-query sparse filters, explicit unsupported-vendor selection, bounded versioned frames, generation checks, canonical checksummed snapshot reload, test-only Python client, independent-oracle tests, and a synthetic throughput harness. See [worker documentation](../ann/README.md) for commands and protocol details. + +Test reference only: `FrozenIndex` now resides under `test/ann/support/`; it validates SQL predicates, table freezes, and failure behavior. It is not a runtime implementation. Native PostgreSQL integration will be C. `ann/cuvs.c` is an unvalidated FP32 L2 CAGRA draft retained for future integration. CAGRA integration and cuVS SDK compilation are deferred at the user's request and do not count as completed implementation milestones. The draft invokes GPU search serially per query; GPU microbatching and persistent GPU graph artifacts are not implemented. + +Validated locally: compiler warnings as errors, CPU correctness/protocol tests, AddressSanitizer/UndefinedBehaviorSanitizer, and live frozen-table integration tests. A checksum-verified PostgreSQL 17.6 source build was installed in a temporary development directory. This checkout's pgvector compiled, all 15 SQL regression tests passed, and generated text/metadata/exact/HNSW baselines executed successfully. The test cluster is temporary, not a system installation. + +This macOS ARM64 host still lacks CUDA and ROCm SDKs/devices. The optional cuVS adapter has not been compiled against its SDK or executed on GPU here; its explicit hardware test target must pass before claiming GPU support. hipVS and fused NVFP4 adapters, GPU performance, worker integration, and accelerator generation/delta semantics remain incomplete. The selective exact CustomScan is implemented. The exact CPU native AM has initial WAL/MVCC/vacuum coverage. Canonical snapshots are not native graph serialization or PostgreSQL WAL. + +CPU ready-query batching/admission is implemented with correctness and concurrency tests. Next active work is native worker integration, pre-eligibility host admission, and sustained-load performance validation. Preserve the tested frozen-table path as a correctness reference. Deferred GPU gates are completing CAGRA integration, compiling against cuVS, and validating capabilities, filtered recall, and performance on supported hardware. The C worker runs CPU frames through `AnnBatcher`; Python protocol clients are test infrastructure only. Multi-process scheduling, GPU workspace admission, and production kernel cancellation remain pending. + +Test-only Python helpers validate finite wire timeouts, corruption handling, SQL statement timeouts, freeze cleanup, and stage timings. A native C external protocol client and PostgreSQL worker integration of those controls remain todos. C scheduler deadlines and queued cancellation are implemented. Dedicated tests cover stalled workers, malformed responses, SQL timeout cleanup, and session loss. Whole-request admission/deadlines and GPU kernel cancellation remain separate work. + +The first implementation slice is Phase 0 through Phase 2: bulk build, native PostgreSQL eligibility, and adaptive exact/CAGRA search on an immutable dataset. The transaction and storage work becomes a production release gate rather than an assumption hidden inside the prototype. + +## References + +- [cuVS v26.10.00 release](https://github.com/NVIDIA/cuvs/releases/tag/v26.10.00) +- [Released CAGRA C implementation](https://github.com/NVIDIA/cuvs/blob/v26.10.00/c/src/neighbors/cagra.cpp) +- [Released C++ filtering contracts](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/include/cuvs/neighbors/common.hpp) +- [Released Roaring allowlist example](https://github.com/NVIDIA/cuvs/blob/v26.10.00/examples/cpp/src/cagra_roaring_bitmap_filter_example.c...) +- [Released multi-GPU CAGRA C implementation](https://github.com/NVIDIA/cuvs/blob/v26.10.00/c/src/neighbors/mg_cagra.cpp) +- [CAGRA build/search/memory guide](https://docs.nvidia.com/cuvs/user-guide/api-guides/indexing-guide/cagra) +- [Brute-force and filtered exact search guide](https://docs.nvidia.com/cuvs/user-guide/api-guides/indexing-guide/brute-force) +- [PostgreSQL custom scans](https://www.postgresql.org/docs/current/custom-scan.html) +- [PostgreSQL text-search indexes](https://www.postgresql.org/docs/current/textsearch-indexes.html) + +## 15. CPU batching implementation status + +Implemented in [ann/batching.c](../ann/batching.c): C backend/index lifetime ownership, compatible batches, private filters, finite collection delay, outstanding request/allocation-byte/work budgets, queued cancellation, monotonic queue/running deadlines, per-request backend errors, and physical-completion accounting. CPU protocol search frames use the C scheduler. Request handles remain valid after close until explicitly released. The [burst benchmark](../benchmarks/ann/batch.c) compares sequential and scheduled requests and includes overload outcomes. This is a tested single-process ready-query scheduler; GPU integration is deferred. + +Admission charges the C request object, copied query/filter arrays, and reserved result storage, plus estimated distance work. Resident index memory, allocator overhead, retained caller results, and upstream SQL materialization have separate lifetimes. Full arrival-rate sweeps, deployment SLOs, cross-process fairness, and measured production throughput are not complete. Native PostgreSQL integration must reserve eligibility resources separately before producing large ID sets. + +## 16. Runtime language and test boundary + +All runtime implementation is C. [ann/batching.c](../ann/batching.c) implements scheduling and admission; [ann/batching.h](../ann/batching.h) defines ownership and request APIs. The C worker uses this scheduler for CPU search frames. No Python package or runtime scheduler is shipped under `ann/`. Python helpers are confined to `test/ann/support/` and benchmark/test launchers; the batching benchmark executes C code. + +The standalone worker currently uses POSIX threads on Linux/macOS. These threads never call PostgreSQL backend APIs. Native backend code uses PostgreSQL memory contexts, buffer locks, interrupts, and generic WAL in C. Native wire-client deadlines and table-freeze/runtime eligibility remain explicit todos; Python reference tests are not completion of those runtime milestones. CAGRA integration and cuVS compilation remain deferred. + +## 17. Native exact CPU AM implementation + +The vendor-neutral `ann` AM now follows the existing source split: `ann.c`, `annbuild.c`, `anninsert.c`, `annscan.c`, `annutils.c`, and `annvacuum.c`, with shared declarations in `ann.h`. Callbacks, helper names, memory contexts, buffer locks, generic WAL, and formatting follow the existing pgvector architecture. [The native index guide](ann-native-index.md) contains implemented execution/storage diagrams and validation commands. + +The registered opclass supports fixed-dimension `vector` L2 through 2,000 dimensions, including NULLs. Scans evaluate all stored vectors, spillably sort distances/TIDs, and rely on PostgreSQL for heap MVCC checks and executor predicates. This is a correctness and lifecycle baseline; no filtered-search throughput improvement or GPU execution is claimed. + +Completed tests cover SQL ordering/rescans, selective executor filters, updates/deletes/abort, repeatable-read visibility, free-space reuse, concurrent writers, unlogged/temporary indexes, generic-WAL replay to a streaming replica, immediate-shutdown recovery, concurrent reindex, and truncate. Local PostgreSQL 17.6 results: 15 SQL regressions, 8 native oracle tests, and 10 WAL/recovery assertions pass. The existing 6 frozen-table reference tests remain separate. + +Remaining steps: C IPC and whole-request admission/deadlines, then sustained-load and concurrency/fault validation. Accelerator row mapping, generations/deltas, and durable graph publication are still separate milestones. CAGRA integration/cuVS compilation, hipVS, and fused NVFP4 remain deferred. + +## 18. Selective exact CustomScan implementation + +The opt-in native C `AnnFilteredSearch` path is implemented in `src/annplan.c` and `src/anncustom.c`. It retains a normal PostgreSQL predicate child plan, moves distance scoring after child qualification, and uses execution-parameter bounds for exact top-k sorting. It preserves ordinary path costs and alternative plans. It does not yet contact the standalone worker. [The guide](ann-filtered-search.md) contains the execution diagram, supported-shape whitelist, GUC/cache behavior, and test commands. + +The first route supports top-level single-table queries with a matching unconditional ANN column index, complete sequential/B-tree/bitmap predicates, ascending distance with secondary tie keys, and Const/external Param query vectors and limits. RLS and snapshot handling remain in the PostgreSQL child. Unsupported shapes keep ordinary plans. Fuzzy GIN is rejected for this exact route, including an execution guard for cached plans. + +Validation adds twelve live oracle scenarios and an SQL regression for filtered ordering, prepared parameters, and scroll cursors. EXPLAIN reports route, eligible rows, custom-stage distance evaluations, and sort space/method. A 4,096-row fixture evaluated only its 40 eligible vectors using a B-tree child and bounded top-N sort. No SQL throughput gain over core exact search or GPU execution is claimed. Native C worker IPC, cross-backend admission, and per-request deadlines are now implemented by the [worker pipeline](ann-worker-pipeline.md). diff --git a/docs/cuvs-search-performance.md b/docs/cuvs-search-performance.md new file mode 100644 index 000000000..43d471a72 --- /dev/null +++ b/docs/cuvs-search-performance.md @@ -0,0 +1,271 @@ +# cuVS filtered-search throughput analysis + +Status: proposed optimization design and experiment plan; no sustained throughput measurements have been collected for this implementation. The opt-in [native exact CustomScan](ann-filtered-search.md) now provides predicate-first execution and EXPLAIN counters; `benchmarks/ann/postgres.py --filtered-native` adds it to the SQL baseline fixture. Accelerator execution and cross-process batching remain pending. + +Prepared: 2026-10-30. Release-specific observations refer to cuVS v26.10.00. This document extends the [overall implementation plan](cuvs-filtered-ann-plan.md), including its PostgreSQL architecture, snapshot, and result contracts. + +## 1. Two optimization tracks + +Optimize the complete SQL request at a fixed filtered-recall and tail-latency target. GPU kernel QPS alone does not establish application throughput. + +| Track | Primary question | Options to evaluate | +| --- | --- | --- | +| A: text eligibility and ID preparation | Can PostgreSQL supply correct eligible IDs fast enough? | Better predicate child plans, compact/page-local mapping, bounded materialization, and same-statement shared-filter batches. | +| B: vector execution and GPU utilization | Can the worker process those IDs efficiently under concurrent load? | CPU/GPU exact subset search, filtered CAGRA, compatible microbatches, resident data, and a small number of asynchronous submission lanes. | + +These tracks complement each other. Cross-backend GPU batching does not remove repeated GIN scans, heap rechecks, or ordinal conversion. Conversely, faster eligibility does not help once the worker is saturated. + +The two vector execution options are **exact search over matching rows** and **filtered ANN over a larger index**. Route per shard using measured costs. Do not force highly selective queries through CAGRA merely because the table is large. + +## 2. Capacity and latency model + +Distinguish **completed SQL statements/second** from **completed vector searches/second**. For a fixed B query vectors per successful statement, vector-search rate equals B times statement rate. Report B, full statement latency, and per-vector recall. Otherwise the shared-filter batch experiment can appear to improve QPS merely by changing what is counted. + +For one SQL statement, measure the following elapsed-time components. Overlapping work must be identified rather than counted twice: + +```text +Tsql = parse/plan + predicate evaluation + identity mapping + filter encoding + + queue/batch wait + transfers + vector search + + delta search + visibility recheck + reranking/projection + response +``` + +Normalize CPU and transfer demand per logical vector search, amortizing shared statement work over its B vectors. For a fixed workload mix: + +```text +CPU vector-rate ceiling = available CPU cores / CPU-seconds per vector search +Link vector-rate ceiling = effective bytes per second / bytes per vector search +Serial GPU batch rate = completed query rows per batch / batch service seconds +Measured GPU vector rate = total completed query rows / elapsed measurement window + across all active submission lanes +SQL statement rate = completed vector-search rate / B (fixed B) +``` + +The CPU/link ceilings are diagnostic bounds, not predictions. The serial batch rate is a baseline, not an upper bound when batches overlap. Estimate sustainable GPU capacity from load curves across all lanes; the slowest resource constrains the overall vector rate. Shared cores, cache contention, variable filters, and retries change service demand. Adding isolated benchmark rates is invalid. + +Report both successful statement and vector-search rates at an agreed recall and statement-p99 latency target. Also report offered rates, timeouts, rejected requests, errors, and late completions. Never improve the headline by dropping slow requests from the denominator or lowering recall without reporting it. + +Use Little's law only as an average consistency check: mean in-flight statements equal statement arrival rate times mean statement residence time for a stable system. For example, 10,000 statements/second at 20 ms mean residence implies 200 mean in-flight statements. This does not define a safe queue cap or a p99 bound. + +### Worked byte-budget example + +Assume 10 million FP32 vectors, dimension 768, and independent query-specific filters. The following is arithmetic, not measured hardware performance: + +| Eligible count M | Survivor fraction | Matching vector content | Content rate at 1,000 vector searches/s | +| --- | --- | --- | --- | +| 1,000 | 0.01% | 3.072 MB | 3.072 GB/s | +| 10,000 | 0.1% | 30.72 MB | 30.72 GB/s | +| 100,000 | 1% | 307.2 MB | 307.2 GB/s | + +If the vectors are already resident, these bytes are device-side reads rather than host uploads. A gather may also write a compacted matrix and search may read it again. If each query uploads its eligible vectors from PostgreSQL, the host/device link must carry this traffic too. Residency can therefore change the winning route. + +At M = 10,000, the 32-bit eligible ID array is only 40 KB, the query vector is 3,072 bytes, and a full 10-million-row bitset is 1.25 MB. Keep vectors resident and compare sparse-ID transport with dense masks before optimizing small result messages. Include ID conversion and encoding time in that comparison. + +## 3. Track A: reduce the PostgreSQL producer cost + +### Establish the CPU ceiling first + +Add a diagnostic worker mode that accepts requests and returns without vector computation. It is a benchmark fixture, never a query path that claims valid search results. This isolates predicate execution, visibility checks, HOT resolution, ordinal lookup, encoding, and IPC. + +Record at least three different cardinalities: + +- `F`: raw predicate/index candidates requiring evaluation or recheck. +- `H`: heap pages visited during eligibility evaluation. +- `M`: eligible searchable vectors after all required checks. + +A tiny M can follow a large F and H. For example, a rare tenant combined with common text terms may still read substantial postings and heap data. Benchmark B-tree-first, GIN-first, bitmap-intersection, and ordinary sequential child plans. Do not force GIN simply because a full-text predicate is present. + +Use `EXPLAIN (ANALYZE, BUFFERS)` on representative child queries and collect lossy/exact heap blocks, recheck removals, buffer reads, and I/O timing when enabled. Test generic and custom prepared plans with skewed tenant/query parameters; runtime vector routing cannot repair a poor predicate child plan. + +### Preserve complete eligibility + +Require `gin_fuzzy_search_limit = 0` when a GIN path supplies an exact allowlist. A nonzero setting can omit matches; such a result is neither exact nor a conservative superset. Reject that configuration for this path or choose a complete eligibility plan. Under writes, measure GIN pending-list size and cleanup activity because queries also search pending entries. [PostgreSQL GIN behavior](https://www.postgresql.org/docs/current/gin.html) + +Measure phrase/weight rechecks and lossy bitmaps separately from simple terms. Sweep `work_mem`, but do not treat it as a cap on extension-owned allocations. Concurrent executor operations and sessions multiply memory demand; custom ID arrays and caches need explicit accounting. [PostgreSQL resource settings](https://www.postgresql.org/docs/current/runtime-config-resource.html) + +### Make mapping compact and local + +1. Return only identity and required mapping fields from the child plan; defer vector detoasting and full result projection. +2. Append IDs to bounded contiguous arrays, not one PostgreSQL `List` node or allocation per match. +3. Resolve HOT roots and ordinal mappings in page-local groups where possible, releasing buffer content locks before any GPU/IPC wait. +4. Benchmark bounded per-backend lookup caches against a shared immutable generation map. Do not duplicate an N-entry reverse map in every backend. +5. Account for mapping-cache bytes, cache misses, spilled materialization, and cold versus warm lookup behavior. +6. Deduplicate and translate to shard-local ordinals before claiming a sorted local-ID representation. Heap-TID order need not preserve ordinal order. + +### Two kinds of batching have different benefits + +```mermaid +flowchart LR + A[Independent SQL statements] --> A1[Predicate and mapping per statement] + A1 --> A2[Worker microbatch] + A2 --> A3[Amortized GPU launch overhead] + B[Many query vectors in one statement] --> B1[One shared predicate and snapshot] + B1 --> B2[One eligible set and ordinal mapping] + B2 --> B3[Shared-filter GPU batch] + B3 --> B4[Amortized CPU preparation and GPU work] +``` + +Prototype an explicit batch-search entry point for multiple query vectors sharing one predicate, security context, and statement snapshot. Restrict the reusable predicate to the supported nonvolatile shape and preserve each query's own top-k result. This can reuse eligibility, mapping, a device allowlist, and an exact gathered subset. + +Do not silently combine independent SQL snapshots. Read Committed statements can see different committed rows, and transactions can see their own writes. Even repeatable snapshots need command/own-write awareness. Statement-scoped eligible-set reuse is the initial optimization. [PostgreSQL transaction isolation](https://www.postgresql.org/docs/current/transaction-iso.html) + +The vector generation being immutable does not make text metadata immutable: a HOT update may change text while leaving the vector unchanged. A cache keyed only by vector generation and predicate text is unsafe. Cross-statement predicate caches require their own proven invalidation and snapshot model. Identical already-validated allowlist contents may share a device allocation while each backend retains responsibility for its query's visibility. + +## 4. Track B: choose exact or ANN execution by measured work + +### Exact subset search + +For tiny M, CPU exact search can avoid queueing and GPU launch costs. Compare it with the ordinary PostgreSQL filter-then-exact plan, which may already have the relevant heap tuple and avoid a second canonical-vector gather. + +For larger selective sets, evaluate two GPU implementations: + +- Explicit sparse-ID gather followed by exact search over the compacted vectors. +- The library's filtered brute-force path over resident full vectors. + +Released cuVS can internally select sparse masked multiplication, gather-and-search, or dense distance evaluation followed by masking. Shared bitsets can use one common gathered subset; heterogeneous per-query bitmaps cannot use that same common-gather branch. Therefore a filtered API call does not guarantee distance work proportional to M. Record which path executes and measure setup costs. [Released exact-search implementation](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/src/neighbors/detail/knn_brute_force.cuh) + +For B queries sharing an identical eligible set, gather once and reuse the subset. A dot-product formulation has approximately `2 * B * M * D` multiply/add work, excluding norms and top-k selection. Vector reuse can improve arithmetic intensity, but top-k and scratch space still matter. Tile computation so the service does not require an unbounded B-by-M distance matrix. + +For distinct filters, benchmark independent compacted subsets against masked execution. A union of eligible sets requires per-query masking and can become much larger than each original set; it is not automatically a better batch. + +### Filtered CAGRA + +Measure candidate capacity and visited-state workspace after cuVS's filter adjustment, not just requested k and input bytes. In the released MULTI_CTA implementation, for survivor fraction s and initial capacity I, the adjustment is approximately: + +```text +adjusted capacity = round_up_to_32(k / s + (I - k) / sqrt(s)) +``` + +With k = 20, I = 64, and s = 0.001, this is approximately 21,408 intermediate candidates. It is an internal capacity, not a visited-row count or a recall guarantee. It explains why highly selective queries can become expensive ANN jobs despite small output. [Released search-plan adjustment](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/src/neighbors/detail/cagra/search_plan.cuh) + +Start with automatic kernel selection, then compare supported SINGLE_CTA and MULTI_CTA settings at equal recall. Sweep `itopk_size`, `search_width`, candidate count, and batch size together; reserve explicit workspace headroom. Keep `max_queries` bounded rather than treating its automatic value as a service admission policy. Parameter validity and algorithm constraints come from the [released search API](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/include/cuvs/neighbors/cagra.hpp). + +Use measured route tables indexed by M, dimension, residency, k/candidate budget, batch type, and current queue load. Add hysteresis so small estimate changes do not continually switch routes. Recalibrate when the index generation, hardware, or workload distribution changes. Sending work to a saturated CPU fallback is not a free overload remedy. + +Filling k results is separate from matching the recall target. Tune and compare complete pipelines, including reranking and retries. A CAGRA retry is another search request; charge its work and deduplicate accumulated candidates before final sorting. + +## 5. Microbatching and admission control + +```mermaid +flowchart TD + Request[Incoming SQL work] --> Host[Bounded host preparation admission] + Host --> Filter[Predicate evaluation and ID mapping] + Filter --> Route{Per-shard route} + Route --> CPU[CPU exact lane] + Route --> Ready[Ready GPU requests with byte and work estimates] + Ready --> Group[Compatible microbatch queues] + Group --> Admit[Reserve workspace and staging capacity] + Admit --> Submit[Direct cuVS search on submission lane] + Submit --> Event[CUDA completion event] + Event --> Return[Return compact candidates to originating backend] + Return --> Release[Release buffers and generation references] +``` + +### Compatibility rules + +Group by generation, shard/local-ID domain, metric, dtype/layout, route, search-parameter profile, candidate-budget class, and selectivity class. Shared-bitset batches additionally require the same allowlist. Direct C++ Roaring batches retain each query's own allowlist and result mapping. + +Remove empty requests before dispatch. Separate extremely selective ANN requests from broad ones: Roaring's automatic filtering estimate uses the most selective query in the batch. Reuse device allowlist owners only while their validity and lifetime are established. The sorted-ID construction shortcut requires strictly increasing, unique, in-range local IDs. [Filter contract](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/include/cuvs/neighbors/common.hpp), [Roaring construction contract](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/include/cuvs/core/roaring_allowlist.hpp) + +Avoid overfragmenting queues into one bucket per tenant or exact k unless required for correctness. Compatible requests can use a common larger candidate budget and truncate per query, but only when the extra work and recall behavior are acceptable. Age-based dispatch must prevent rare buckets from waiting indefinitely. + +### Dispatch and memory bounds + +Dispatch at the earliest of a row cap, a workspace/work cap, or the oldest request's permitted batch delay. Use each request's remaining deadline and reserve time for result transfer and PostgreSQL completion. Batch-formation delay is only one part of end-to-end latency. + +Start experiments with batch rows `1, 4, 16, 64, 128, 256`, batch-delay budgets `0, 25, 50, 100, 250, 500` microseconds, and `1, 2, 4` submission lanes. These are a search grid, not recommended defaults. Prune unsupported combinations and reserve workspace before launch. + +Bound both producer memory and worker memory. A bounded GPU queue does not stop many PostgreSQL backends from allocating large ID sets simultaneously. Reserve host preparation capacity before large materialization, then GPU workspace only after a request is ready. Specify spill, alternate-plan, or explicit error behavior without violating the query contract. + +Track queued bytes and estimated service work as well as request count. A 1,000-ID query and a million-ID query must not consume indistinguishable admission units. Preserve per-tenant fairness and cap retry work; long queues retain snapshots and generations and can delay vacuum/compaction. + +### Why not use the cuVS batch wrapper by default? + +The released dynamic-batching wrapper fixes k, upstream parameters, and its optional shared filter at construction. It rejects Roaring, and its dispatch timeout does not bound complete request latency. It can suit stable same-filter or unfiltered traffic; use application-controlled direct calls for heterogeneous text filters. [Released dynamic-batching API](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/include/cuvs/neighbors/dynamic_batching.hpp) + +## 6. Residency, asynchronous execution, and persistent search + +Keep active graphs and vector data resident within the declared capacity envelope. Pool pinned host staging, query/result buffers, allowlists, and workspace rather than allocating them on every request. Pinned-memory budgets must be bounded; using more pinned memory is not an unlimited optimization. + +Use one reusable resources object per submission thread/lane, or serialize its use. cuVS resources are not concurrently thread-safe. Chain preparation, search, and result transfer on streams with explicit dependencies; wait when results or resource reclamation require completion. Different streams permit overlap but do not guarantee it. [cuVS resource ownership](https://docs.nvidia.com/cuvs/user-guide/api-guides/core-types/resources) + +Try preparing batch n+1 while batch n executes and batch n-1 completes. Measure whether memory bandwidth or workspace contention cancels the benefit. Avoid device-wide synchronization in the request loop and PostgreSQL buffer locks across any worker wait. CUDA's guidance emphasizes reducing transfers and using appropriate pinned/asynchronous transfers for overlap. [CUDA best practices](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html) + +Canceled requests may already have GPU work in flight. Discard their results while retaining referenced buffers and generations until completion. Cancellation tests must cover allocation reuse and index eviction, not only client-visible timeout latency. + +Treat persistent CAGRA as a separate experiment for stable repeated workloads. The API restricts it to SINGLE_CTA and documents progress hazards with concurrent work, allocation, and pageable transfers. Keep it disabled for the initial mixed exact/ANN service. [Persistent search contract](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/include/cuvs/neighbors/cagra.hpp) + +Persistent filtering is not universally unsupported. However, the released runner's reuse depends on graph, parameters, and filter payload identity. Fresh allowlists or alternating shards can defeat reuse. Measure runner churn before treating persistence as a throughput improvement. [Released persistent launcher](https://github.com/NVIDIA/cuvs/blob/v26.10.00/cpp/src/neighbors/detail/cagra/search_single_cta_kerne...) + +## 7. Shards, delta, and result production + +### Shards and replicas + +Prune only shards proven irrelevant by partition metadata or an empty eligible intersection. Every remaining shard adds search work and candidates. With S shards returning r candidates each, transport and reranking scale with approximately S times r, even when every local kernel is fast. + +Compare fewer large shards with many small shards at the same memory limit and recall. Batch compatible work per shard in the worker; avoid a backend RPC per row. Use exact local top-k when its eligible subset is small and ANN for other shards, then merge comparable distances. + +When a full index fits each GPU, replicas may improve throughput by assigning complete requests to different devices. Sharding primarily solves capacity and can add fanout and merge costs. Benchmark both layouts when feasible; neither fixes a CPU eligibility bottleneck. + +### Delta and compaction + +Map eligible new/updated entries directly into the relevant delta segments. Avoid a full mutable-delta scan for every query. Measure cost versus delta cardinality, segment count, write rate, and time since compaction. + +Trigger compaction decisions using observed query cost and memory pressure as well as entry count. Bound build resources so compaction does not consume serving workspace or cause repeated cache eviction. Preserve the overall plan's base/delta coverage and old-snapshot rules; performance shortcuts cannot hide fresh writes or discard still-visible versions. + +### Heap fetch and reranking + +Keep worker results to compact identities and distances. Delay full row projection until after candidate selection. Deduplicate version identities, perform required visibility/qualification checks, and use bounded top-k selection rather than sorting an unnecessarily large candidate union. + +Count original-vector reads and detoasting during authoritative reranking. Reranking every shard's excessive oversampling can shift the bottleneck back to the backend. Measure candidate count, duplicate rate, recheck failures, vector bytes, heap fetches, and refill rate together. + +## 8. Experiment sequence and acceptance gates + +Run these in order so optimization effort follows measured bottlenecks: + +| Experiment | Controlled comparison | Decision it enables | +| --- | --- | --- | +| E1: producer ceiling | Complete eligibility/mapping with diagnostic no-compute worker, at increasing client concurrency. | Whether Track A limits target QPS before GPU tuning. | +| E2: mapping layout | Narrow contiguous IDs, page grouping, bounded caches; cold/warm HOT and ordinal lookup. | Mapping design and host memory budget. | +| E3: safe reuse | Independent statements versus many query vectors under one statement/predicate/snapshot; report both statement and vector-search rates. | Value of an explicit shared-filter batch API. | +| E4: exact crossover | Native PostgreSQL exact, adapter CPU exact, GPU gathered exact, and library filtered brute force. | Route thresholds by M, dimension, residency, and batch type. | +| E5: filtered CAGRA | Candidate/iteration/search-width profiles and kernel modes, at equal filtered recall. | ANN cost and workspace profiles. | +| E6: filter transport | Dense masks, sparse IDs, Roaring; first-use and reused buffers; shared and unique filters. | Representation and cache policy. | +| E7: batching | Batch rows, formation delay, lanes, mixed selectivity, and skewed tenants. | Dispatch limits and fairness settings. | +| E8: offered load | Arrival-rate sweep through saturation with all waits/rejections recorded. | Sustainable statement/vector rates at p99/recall objectives. | +| E9: write interference | GIN pending entries, delta growth, long snapshots, and compaction during queries. | Maintenance limits and retention budget. | +| E10: fanout | Shard count, skew, pruning, replicas, and rerank budgets. | Capacity layout and multi-GPU strategy. | +| E11: persistent mode | Stable shared filters versus fresh per-query filters and alternating shards. | Whether persistence earns its complexity. | +| E12: cold/failure behavior | First load/JIT, filter creation, cache eviction, worker restart, and cancellation. | Warmup policy and bounded degradation. | + +Use survivor fractions from 100% to 0.001%, plus empty and fewer-than-k cases. Include text/vector correlation and anti-correlation, common-word/rare-tenant combinations, phrase queries, and varied k. Test target dimensions and some non-aligned dimensions to expose layout-dependent batching behavior. Record full dataset, filter, hardware, version, and parameter manifests. + +Use closed-loop clients for concurrency sweeps and open-loop scheduled arrivals to expose overload. Measure from intended arrival to completion as well as server receipt to completion; otherwise a blocked generator can hide queueing. Report offered/accepted/completed rates and preserve timeout/error samples. Run sufficiently long to observe maintenance effects and report variation across repeated runs. + +Compute exact ground truth among qualifying visible rows for frozen snapshots or controlled fixtures. For mutable tests, do not compare against a later snapshot and label the difference a recall error. Use tie-aware recall and compare configurations only at the same quality target. The cuVS benchmark methodology also emphasizes recall/throughput comparisons. [cuVS methodology](https://docs.nvidia.com/cuvs/user-guide/benchmarking-guide/methodologies) + +### Instrumentation + +Record per-request timestamps and a request ID across the backend/worker boundary. Within a host, use consistent monotonic elapsed timing; measure GPU intervals with CUDA events and complete them before reading elapsed values. + +| Layer | Required observations | +| --- | --- | +| Predicate | F, H, M, plan shape, rechecks, buffers, I/O, pending-list effects. | +| Mapping | HOT resolution, lookup misses, allocated/spilled bytes, sorted/deduplicated IDs. | +| Admission | Host and device reserved bytes, queue age, rejects, deadline remaining. | +| Batch | Actual/maximum rows, route, selectivity distribution, shared-filter ratio. | +| GPU | Chosen algorithm, adjusted budgets, workspace, transfers, allocation counts, device time. | +| Completion | Base/delta/shard candidate counts, rerank bytes/time, heap fetches, refills. | +| Service | Successful/late/error statement and vector rates, vectors/statement, statement p50/p95/p99, CPU/GPU utilization, retained generations. | + +Use system profiling to locate idle gaps, copy/compute serialization, allocator stalls, and host submission delays. Use kernel profiling for memory throughput, cache behavior, occupancy, and top-k costs. Rerun final throughput outside profiling because profiling can serialize work and alter cache behavior. [Nsight Compute profiling guide](https://docs.nvidia.com/nsight-compute/ProfilingGuide/) + +## 9. Changes to the implementation sequence + +- Phase 0: define quality/latency targets and add producer-only plus native PostgreSQL exact baselines. +- Phase 1: implement reusable buffers, explicit batch/workspace caps, route telemetry, and direct-search microbatching before persistent mode. +- Phase 2: benchmark statement-scoped shared-filter batches, host admission, compact mapping, and complete SQL throughput. +- Phase 3: cost eligibility startup and spill; preserve independent AM and CustomScan state; expose timings and route decisions through diagnostics/EXPLAIN. +- Phase 4: test pending-list, delta, snapshot-retention, and compaction interference at sustained offered load. +- Phase 5: choose shard/replica layout and persistent search only after their end-to-end gains survive equal-recall and p99 comparisons. + +Do not accept an optimization on kernel QPS alone. Require unchanged row/predicate correctness, the agreed filtered recall, bounded host/device memory, and improved successful throughput at the latency target. Until hardware and workload targets are supplied, these experiments define the work; they do not imply a promised QPS. diff --git a/scripts/ann/build_cuvs_index.py b/scripts/ann/build_cuvs_index.py new file mode 100644 index 000000000..65eddb8dc --- /dev/null +++ b/scripts/ann/build_cuvs_index.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +"""Build an immutable, sharded cuVS CAGRA index directory from a PostgreSQL table. + + python3 scripts/ann/build_cuvs_index.py --dsn 'dbname=app' --table public.docs \\ + --key id --vector embedding --out /data/ann/docs --shards 8 --dtype float16 + +The directory holds a text manifest, keys.i64 (sorted int64 keys), vectors.f32 +(FP32 vectors in key order, used for exact routes and reranking) and one +shard-N.cagra file per contiguous key range. Shard N is built on GPU N modulo +the number of visible devices, in parallel. Load it from SQL with +ann_index_load(name, path); see docs/ann-cuvs-index.md. + +This is offline tooling: Python, NumPy, psycopg, CuPy and the cuVS Python +package are needed here only, never on the query path. --no-cagra writes the +manifest, keys and vectors only, for CPU-only testing. +""" + +import argparse +import concurrent.futures +import os +import struct +import sys +import time + +import numpy as np + + +def log(message): + print(f'[{time.strftime("%H:%M:%S")}] {message}', file=sys.stderr, flush=True) + + +def export(dsn, table, key, vector, limit=None): + """Stream (key, vector) in key order with binary COPY; returns keys and FP32 matrix.""" + import psycopg + from psycopg import sql + + query = sql.SQL('SELECT {key}::bigint, {vector} FROM {table} WHERE {vector} IS NOT NULL ORDER BY 1').format( + key=sql.Identifier(key), vector=sql.Identifier(vector), + table=sql.Identifier(*table.split('.'))) + if limit: + query = sql.SQL('{} LIMIT {}').format(query, sql.Literal(limit)) + keys = [] + chunks = [] + dimensions = None + with psycopg.connect(dsn) as db: + db.execute('SET TRANSACTION ISOLATION LEVEL REPEATABLE READ') + with db.cursor().copy(sql.SQL('COPY ({}) TO STDOUT (FORMAT binary)').format(query)) as copy: + buffer = bytearray() + for data in copy: + buffer += data + raw = bytes(buffer) + # Binary COPY: 19-byte header, then per row int16 fields, int32 length + bytes per field. + offset = 19 + struct.unpack_from('>I', raw, 15)[0] + rows = [] + while True: + fields = struct.unpack_from('>h', raw, offset)[0] + offset += 2 + if fields == -1: + break + length = struct.unpack_from('>i', raw, offset)[0] + keys.append(struct.unpack_from('>q', raw, offset + 4)[0]) + offset += 4 + length + length = struct.unpack_from('>i', raw, offset)[0] + # pgvector binary: int16 dim, int16 unused, dim big-endian float4 + dim = struct.unpack_from('>H', raw, offset + 4)[0] + if dimensions is None: + dimensions = dim + elif dim != dimensions: + raise SystemExit(f'vector dimensions differ: {dim} and {dimensions}') + rows.append(raw[offset + 8:offset + 8 + 4 * dim]) + offset += 4 + length + if len(rows) >= 1 << 16: + chunks.append(np.frombuffer(b''.join(rows), dtype='>f4').reshape(-1, dimensions)) + rows = [] + if rows: + chunks.append(np.frombuffer(b''.join(rows), dtype='>f4').reshape(-1, dimensions)) + if not keys: + raise SystemExit('table has no non-null vectors') + keys = np.asarray(keys, dtype=np.int64) + if np.any(keys[1:] <= keys[:-1]): + raise SystemExit(f'key column {key} must be unique and non-null') + return keys, np.ascontiguousarray(np.vstack(chunks).astype('<f4')) + + +def build_shard(index, first, vectors, out, dtype, graph_degree, intermediate_degree, devices): + import cupy + from cuvs.neighbors import cagra + + device = index % devices + with cupy.cuda.Device(device): + started = time.time() + data = cupy.asarray(vectors, dtype=cupy.float16 if dtype == 'float16' else cupy.float32) + params = cagra.IndexParams(metric='sqeuclidean', graph_degree=graph_degree, + intermediate_graph_degree=intermediate_degree) + built = cagra.build(params, data) + cagra.save(os.path.join(out, f'shard-{index}.cagra'), built, include_dataset=True) + cupy.cuda.Device(device).synchronize() + log(f'shard {index}: {len(vectors)} rows on GPU {device} in {time.time() - started:.1f}s') + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument('--dsn', required=True) + parser.add_argument('--table', required=True, help='schema-qualified table, e.g. public.docs') + parser.add_argument('--key', default='id', help='unique integer key column') + parser.add_argument('--vector', default='embedding', help='vector column') + parser.add_argument('--out', required=True, help='new or empty output directory') + parser.add_argument('--shards', type=int, default=0, help='default: number of visible GPUs') + parser.add_argument('--dtype', choices=['float32', 'float16'], default='float32', + help='CAGRA dataset precision; float16 halves GPU memory traffic') + parser.add_argument('--graph-degree', type=int, default=32) + parser.add_argument('--intermediate-graph-degree', type=int, default=64) + parser.add_argument('--limit', type=int, help='export at most this many rows (testing)') + parser.add_argument('--no-cagra', action='store_true', help='write keys/vectors only (CPU tests)') + args = parser.parse_args() + + if os.path.exists(args.out) and os.listdir(args.out): + raise SystemExit(f'{args.out} is not empty; index directories are immutable') + os.makedirs(args.out, exist_ok=True) + started = time.time() + keys, vectors = export(args.dsn, args.table, args.key, args.vector, args.limit) + rows, dimensions = vectors.shape + log(f'exported {rows} rows x {dimensions} dims in {time.time() - started:.1f}s') + keys.astype('<i8').tofile(os.path.join(args.out, 'keys.i64')) + vectors.tofile(os.path.join(args.out, 'vectors.f32')) + + shards = [] + if not args.no_cagra: + import cupy + devices = cupy.cuda.runtime.getDeviceCount() + count = args.shards or devices + bounds = [rows * i // count for i in range(count + 1)] + shards = [(i, bounds[i], bounds[i + 1] - bounds[i]) for i in range(count) if bounds[i + 1] > bounds[i]] + with concurrent.futures.ThreadPoolExecutor(max_workers=min(len(shards), devices)) as pool: + futures = [pool.submit(build_shard, i, first, vectors[first:first + n], args.out, args.dtype, + args.graph_degree, args.intermediate_graph_degree, devices) + for i, first, n in shards] + for future in futures: + future.result() + + # The manifest is written last, so a partial build is never loadable. + lines = ['pgvector-ann-index 1', f'dimensions {dimensions}', f'rows {rows}', 'metric l2', + f'dtype {args.dtype}', f'source {args.table}', f'key {args.key}', f'vector {args.vector}', + f'shards {len(shards)}'] + lines += [f'shard {i} {first} {n} shard-{i}.cagra' for i, first, n in shards] + with open(os.path.join(args.out, 'manifest.tmp'), 'w') as manifest: + manifest.write('\n'.join(lines) + '\n') + os.replace(os.path.join(args.out, 'manifest.tmp'), os.path.join(args.out, 'manifest')) + log(f'wrote {args.out} ({len(shards)} shards) in {time.time() - started:.1f}s') + + +if __name__ == '__main__': + main() diff --git a/sql/vector--0.8.7--0.9.0.sql b/sql/vector--0.8.7--0.9.0.sql index f29765079..1cbe9a39b 100644 --- a/sql/vector--0.8.7--0.9.0.sql +++ b/sql/vector--0.8.7--0.9.0.sql @@ -1,7 +1,67 @@ -- complain if script is sourced in psql, rather than via CREATE EXTENSION \echo Use "ALTER EXTENSION vector UPDATE TO '0.9.0'" to load this file. \quit -CREATE FUNCTION vector_dims(sparsevec) RETURNS integer +CREATE OR REPLACE FUNCTION vector_dims(sparsevec) RETURNS integer AS 'MODULE_PATHNAME', 'sparsevec_vector_dims' LANGUAGE C IMMUTABLE STRICT PARALLEL SAFE; COMMENT ON FUNCTION vector_dims(sparsevec) IS 'number of dimensions'; + +-- Experimental exact CPU staging access method + +CREATE FUNCTION annhandler(internal) RETURNS index_am_handler + AS 'MODULE_PATHNAME' LANGUAGE C; + +COMMENT ON FUNCTION annhandler(internal) IS 'ann index access method handler'; + +CREATE ACCESS METHOD ann TYPE INDEX HANDLER annhandler; + +COMMENT ON ACCESS METHOD ann IS 'experimental exact vector staging index'; + +CREATE OPERATOR CLASS vector_l2_ops + FOR TYPE vector USING ann AS + OPERATOR 1 <-> (vector, vector) FOR ORDER BY float_ops, + FUNCTION 1 l2_distance(vector, vector); + +-- loaded cuVS index functions + +CREATE FUNCTION ann_index_load(name text, path text) RETURNS bigint + AS 'MODULE_PATHNAME' LANGUAGE C STRICT VOLATILE; + +COMMENT ON FUNCTION ann_index_load(text, text) IS 'load an immutable ann index directory into the worker'; + +REVOKE ALL ON FUNCTION ann_index_load(text, text) FROM PUBLIC; + +CREATE FUNCTION ann_index_search(name text, query vector, k integer, filter bigint[] DEFAULT NULL, + OUT id bigint, OUT distance real) RETURNS SETOF record + AS 'MODULE_PATHNAME' LANGUAGE C VOLATILE ROWS 10; + +COMMENT ON FUNCTION ann_index_search(text, vector, integer, bigint[]) IS 'top-k search of a loaded ann index, optionally restricted to filter keys'; + +-- where loaded indexes come from on this host (for local exact routing) + +CREATE TABLE ann_index_catalog ( + name text PRIMARY KEY, + source regclass NOT NULL, + key_column name NOT NULL, + vector_column name NOT NULL, + rows bigint NOT NULL, + loaded_at timestamptz NOT NULL DEFAULT now() +); + +COMMENT ON TABLE ann_index_catalog IS 'source tables of loaded ann indexes, recorded by ann_index_load and ann_index_attach'; + +REVOKE ALL ON ann_index_catalog FROM PUBLIC; +GRANT SELECT ON ann_index_catalog TO PUBLIC; + +CREATE FUNCTION ann_index_attach(name text) RETURNS bigint + AS 'MODULE_PATHNAME' LANGUAGE C STRICT VOLATILE; + +COMMENT ON FUNCTION ann_index_attach(text) IS 'verify and record an ann index already loaded on the GPU nodes'; + +REVOKE ALL ON FUNCTION ann_index_attach(text) FROM PUBLIC; + +CREATE FUNCTION ann_hybrid_search(name text, query vector, k integer, filter text, + OUT id bigint, OUT distance real) RETURNS SETOF record + AS 'MODULE_PATHNAME' LANGUAGE C VOLATILE ROWS 10; + +COMMENT ON FUNCTION ann_hybrid_search(text, vector, integer, text) IS 'hybrid search: selective SQL filters run exactly here, broad ones on GPU nodes'; diff --git a/sql/vector.sql b/sql/vector.sql index fece8beb6..04b81d88f 100644 --- a/sql/vector.sql +++ b/sql/vector.sql @@ -1215,3 +1215,63 @@ CREATE OPERATOR CLASS sparsevec_l1_ops OPERATOR 1 <+> (sparsevec, sparsevec) FOR ORDER BY float_ops, FUNCTION 1 l1_distance(sparsevec, sparsevec), FUNCTION 3 hnsw_sparsevec_support(internal); + +-- Experimental exact CPU staging access method + +CREATE FUNCTION annhandler(internal) RETURNS index_am_handler + AS 'MODULE_PATHNAME' LANGUAGE C; + +COMMENT ON FUNCTION annhandler(internal) IS 'ann index access method handler'; + +CREATE ACCESS METHOD ann TYPE INDEX HANDLER annhandler; + +COMMENT ON ACCESS METHOD ann IS 'experimental exact vector staging index'; + +CREATE OPERATOR CLASS vector_l2_ops + FOR TYPE vector USING ann AS + OPERATOR 1 <-> (vector, vector) FOR ORDER BY float_ops, + FUNCTION 1 l2_distance(vector, vector); + +-- loaded cuVS index functions + +CREATE FUNCTION ann_index_load(name text, path text) RETURNS bigint + AS 'MODULE_PATHNAME' LANGUAGE C STRICT VOLATILE; + +COMMENT ON FUNCTION ann_index_load(text, text) IS 'load an immutable ann index directory into the worker'; + +REVOKE ALL ON FUNCTION ann_index_load(text, text) FROM PUBLIC; + +CREATE FUNCTION ann_index_search(name text, query vector, k integer, filter bigint[] DEFAULT NULL, + OUT id bigint, OUT distance real) RETURNS SETOF record + AS 'MODULE_PATHNAME' LANGUAGE C VOLATILE ROWS 10; + +COMMENT ON FUNCTION ann_index_search(text, vector, integer, bigint[]) IS 'top-k search of a loaded ann index, optionally restricted to filter keys'; + +-- where loaded indexes come from on this host (for local exact routing) + +CREATE TABLE ann_index_catalog ( + name text PRIMARY KEY, + source regclass NOT NULL, + key_column name NOT NULL, + vector_column name NOT NULL, + rows bigint NOT NULL, + loaded_at timestamptz NOT NULL DEFAULT now() +); + +COMMENT ON TABLE ann_index_catalog IS 'source tables of loaded ann indexes, recorded by ann_index_load and ann_index_attach'; + +REVOKE ALL ON ann_index_catalog FROM PUBLIC; +GRANT SELECT ON ann_index_catalog TO PUBLIC; + +CREATE FUNCTION ann_index_attach(name text) RETURNS bigint + AS 'MODULE_PATHNAME' LANGUAGE C STRICT VOLATILE; + +COMMENT ON FUNCTION ann_index_attach(text) IS 'verify and record an ann index already loaded on the GPU nodes'; + +REVOKE ALL ON FUNCTION ann_index_attach(text) FROM PUBLIC; + +CREATE FUNCTION ann_hybrid_search(name text, query vector, k integer, filter text, + OUT id bigint, OUT distance real) RETURNS SETOF record + AS 'MODULE_PATHNAME' LANGUAGE C VOLATILE ROWS 10; + +COMMENT ON FUNCTION ann_hybrid_search(text, vector, integer, text) IS 'hybrid search: selective SQL filters run exactly here, broad ones on GPU nodes'; diff --git a/src/ann.c b/src/ann.c new file mode 100644 index 000000000..e07c76387 --- /dev/null +++ b/src/ann.c @@ -0,0 +1,92 @@ +#include "postgres.h" + +#include <math.h> + +#include "access/amvalidate.h" +#include "ann.h" +#include "catalog/pg_type_d.h" +#include "commands/vacuum.h" +#include "optimizer/optimizer.h" +#include "utils/float.h" +#include "utils/lsyscache.h" +#include "utils/selfuncs.h" + +/* + * The CPU staging implementation reads and sorts the complete index. + */ +static void +anncostestimate(PlannerInfo *root, IndexPath *path, double loop_count, + Cost *indexStartupCost, Cost *indexTotalCost, + Selectivity *indexSelectivity, double *indexCorrelation, + double *indexPages) +{ + GenericCosts costs = {0}; + double tuples = path->indexinfo->tuples; + + if (path->indexorderbys == NIL) + { + *indexStartupCost = *indexTotalCost = get_float8_infinity(); + *indexSelectivity = *indexCorrelation = *indexPages = 0; + return; + } + costs.numIndexTuples = tuples; + genericcostestimate(root, path, loop_count, &costs); + *indexStartupCost = costs.indexTotalCost + + tuples * cpu_operator_cost * (1 + (tuples > 1 ? log(tuples) / log(2.0) : 0)); + *indexTotalCost = *indexStartupCost; + *indexSelectivity = costs.indexSelectivity; + *indexCorrelation = 0; + *indexPages = costs.numIndexPages; +} + +static bool +annvalidate(Oid opclassoid) +{ + Oid type = get_opclass_input_type(opclassoid); + Oid proc = get_opfamily_proc(get_opclass_family(opclassoid), type, type, ANN_DISTANCE_PROC); + + return type == AnnVectorTypeOid() && OidIsValid(proc) && + check_amproc_signature(proc, FLOAT8OID, true, 2, 2, type, type); +} + +static bool +annproperty(Oid index_oid, int attno, IndexAMProperty prop, const char *propname, + bool *res, bool *isnull) +{ + if (attno == 0 || prop != AMPROP_DISTANCE_ORDERABLE) + return false; + *res = true; + *isnull = false; + return true; +} + +/* + * Register only implemented access-method capabilities. + */ +FUNCTION_PREFIX PG_FUNCTION_INFO_V1(annhandler); +Datum +annhandler(PG_FUNCTION_ARGS) +{ + IndexAmRoutine *amroutine = makeNode(IndexAmRoutine); + + amroutine->amstrategies = 0; + amroutine->amsupport = ANN_NPROCS; + amroutine->amcanorderbyop = true; + amroutine->amoptionalkey = true; + amroutine->amparallelvacuumoptions = VACUUM_OPTION_PARALLEL_BULKDEL; + amroutine->amkeytype = InvalidOid; + amroutine->ambuild = annbuild; + amroutine->ambuildempty = annbuildempty; + amroutine->aminsert = anninsert; + amroutine->ambulkdelete = annbulkdelete; + amroutine->amvacuumcleanup = annvacuumcleanup; + amroutine->amcostestimate = anncostestimate; + amroutine->amproperty = annproperty; + amroutine->amvalidate = annvalidate; + amroutine->ambeginscan = annbeginscan; + amroutine->amrescan = annrescan; + amroutine->amgettuple = anngettuple; + amroutine->amendscan = annendscan; + + PG_RETURN_POINTER(amroutine); +} diff --git a/src/ann.h b/src/ann.h new file mode 100644 index 000000000..c31a0dc44 --- /dev/null +++ b/src/ann.h @@ -0,0 +1,93 @@ +#ifndef ANN_H +#define ANN_H + +#include "postgres.h" + +#include "access/amapi.h" +#include "access/genam.h" +#include "access/itup.h" +#include "access/relscan.h" +#include "nodes/execnodes.h" +#include "nodes/extensible.h" +#include "storage/bufmgr.h" +#include "utils/rel.h" +#include "vector.h" + +#define ANN_MAGIC_NUMBER 0xA991A991 +#define ANN_VERSION 1 +#define ANN_PAGE_ID 0xFF92 +#define ANN_METAPAGE_BLKNO 0 +#define ANN_MAX_DIM 2000 +#define ANN_DISTANCE_PROC 1 +#define ANN_NPROCS 1 + +typedef struct AnnPageOpaqueData +{ + uint16 page_id; +} AnnPageOpaqueData; + +typedef struct AnnMetaPageData +{ + uint32 magicNumber; + uint32 version; + uint32 dimensions; + BlockNumber insertPage; +} AnnMetaPageData; + +#define AnnPageGetMeta(page) ((AnnMetaPageData *) PageGetContents(page)) +#define AnnPageGetOpaque(page) ((AnnPageOpaqueData *) PageGetSpecialPointer(page)) + +extern const CustomPathMethods AnnCustomPathMethods; +extern const CustomScanMethods AnnCustomScanMethods; +void AnnInit(void); + +extern char *ann_worker_socket; +extern int ann_worker_chunk_rows; +extern int ann_worker_pipeline_depth; +extern int ann_worker_timeout_ms; +extern int ann_worker_shared_memory_mb; +void AnnWorkerInit(void); +bool AnnWorkerBegin(void); +void AnnWorkerEnd(void); +float *AnnWorkerSharedBuffer(size_t *size); +uint64 AnnWorkerSubmit(uint32 rows, uint32 dimensions, uint32 k, + const float *query, const float *vectors); +bool AnnWorkerReady(void); +extern char *ann_worker_address; +extern char *ann_worker_auth_token; +extern int ann_gpu_min_eligible; +extern int ann_worker_heartbeat_ms; +extern int ann_worker_retry_ms; +void AnnRemoteInit(void); +bool AnnRemoteConfigured(void); +bool AnnRemoteCall(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength); +int AnnRemoteBroadcast(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength); +bool AnnWorkerCall(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength); +bool AnnWorkerReceive(uint64 *request, uint32 *status, uint32 **ids, uint32 *count); + +Oid AnnVectorTypeOid(void); +void AnnCheckPage(Page page); +void AnnCheckIndex(Relation index); +void AnnInitMetaPage(Relation index, ForkNumber forkNum); +void AnnCheckMetaPage(Page page); +void AnnInsertTuple(Relation index, Datum *values, bool *isnull, + ItemPointer tid, bool building); + +IndexBuildResult *annbuild(Relation heap, Relation index, IndexInfo *indexInfo); +void annbuildempty(Relation index); +bool anninsert(Relation index, Datum *values, bool *isnull, ItemPointer tid, + Relation heap, IndexUniqueCheck checkUnique, bool indexUnchanged, + IndexInfo *indexInfo); +IndexScanDesc annbeginscan(Relation index, int nkeys, int norderbys); +void annrescan(IndexScanDesc scan, ScanKey keys, int nkeys, + ScanKey orderbys, int norderbys); +bool anngettuple(IndexScanDesc scan, ScanDirection dir); +void annendscan(IndexScanDesc scan); +IndexBulkDeleteResult *annbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, + IndexBulkDeleteCallback callback, void *callback_state); +IndexBulkDeleteResult *annvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats); + +#endif diff --git a/src/annbuild.c b/src/annbuild.c new file mode 100644 index 000000000..690d9b469 --- /dev/null +++ b/src/annbuild.c @@ -0,0 +1,49 @@ +#include "postgres.h" + +#include "access/tableam.h" +#include "access/xloginsert.h" +#include "ann.h" +#include "utils/memutils.h" + +typedef struct AnnBuildState +{ + double indtuples; + MemoryContext tmpCtx; +} AnnBuildState; + +static void +BuildCallback(Relation index, ItemPointer tid, Datum *values, bool *isnull, + bool tupleIsAlive, void *state) +{ + AnnBuildState *buildstate = state; + MemoryContext oldCtx = MemoryContextSwitchTo(buildstate->tmpCtx); + + AnnInsertTuple(index, values, isnull, tid, true); + buildstate->indtuples++; + MemoryContextSwitchTo(oldCtx); + MemoryContextReset(buildstate->tmpCtx); +} + +IndexBuildResult * +annbuild(Relation heap, Relation index, IndexInfo *indexInfo) +{ + AnnBuildState buildstate = {0}; + IndexBuildResult *result = palloc0_object(IndexBuildResult); + + AnnInitMetaPage(index, MAIN_FORKNUM); + buildstate.tmpCtx = AllocSetContextCreate(CurrentMemoryContext, "Ann build temporary context", ALLOCSET_DEFAULT_SIZES); + result->heap_tuples = table_index_build_scan(heap, index, indexInfo, true, true, + BuildCallback, &buildstate, NULL); + result->index_tuples = buildstate.indtuples; + if (RelationNeedsWAL(index)) + log_newpage_range(index, MAIN_FORKNUM, 0, RelationGetNumberOfBlocks(index), true); + MemoryContextDelete(buildstate.tmpCtx); + return result; +} + +void +annbuildempty(Relation index) +{ + AnnInitMetaPage(index, INIT_FORKNUM); + log_newpage_range(index, INIT_FORKNUM, 0, 1, true); +} diff --git a/src/anncustom.c b/src/anncustom.c new file mode 100644 index 000000000..d7da4147a --- /dev/null +++ b/src/anncustom.c @@ -0,0 +1,686 @@ +#include "postgres.h" + +#include "access/gin.h" +#include "access/htup_details.h" +#include "ann.h" +#include "catalog/objectaccess.h" +#include "catalog/pg_proc.h" +#include "executor/executor.h" +#include "miscadmin.h" +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/optimizer.h" +#include "parser/parsetree.h" +#include "utils/acl.h" +#include "utils/lsyscache.h" +#include "utils/memutils.h" +#include "utils/tuplesort.h" + +/* Eligible rows awaiting worker candidates; tuples keep the full child row. */ +typedef struct AnnChunk +{ + uint64 request; + MemoryContext context; + MinimalTuple *tuples; + float *vectors; + float *shared; /* this chunk's slot in worker shared memory */ + int count; +} AnnChunk; + +/* custom_private: sort columns, operators, collations, null order, distance. */ +typedef struct AnnCustomScanState +{ + CustomScanState css; + PlanState *child; + TupleTableSlot *input; + TupleTableSlot *output; + Tuplesortstate *sortstate; + ExprState *query; + ExprState *limit; + ExprState *offset; + FmgrInfo distance; + AttrNumber distanceAttno; + int nkeys; + AttrNumber *sortColIdx; + Oid *sortOperators; + Oid *collations; + bool *nullsFirst; + bool withTies; + uint64 eligibleRows; + uint64 distanceEvaluations; + + /* Worker route state, valid while rows are collected. */ + bool useWorker; + bool workerUsed; + Datum queryDatum; + bool queryIsNull; + int dimensions; + int chunkRows; + int64 bound; + TupleTableSlot *chunkSlot; + bool *nulls; + AnnChunk *chunks; + int nchunks; /* chunks array length: depth + 1 */ + int filling; /* chunk being filled, or -1 */ + int inflight; + uint64 workerChunks; + uint64 localChunks; + uint64 workerCandidates; +} AnnCustomScanState; + +static Plan *PlanCustomPath(PlannerInfo *root, RelOptInfo *rel, CustomPath *path, + List *tlist, List *clauses, List *custom_plans); +static Node *CreateCustomScanState(CustomScan *scan); +static void BeginCustomScan(CustomScanState *node, EState *estate, int eflags); +static TupleTableSlot *ExecCustomScan(CustomScanState *node); +static void EndCustomScan(CustomScanState *node); +static void ReScanCustomScan(CustomScanState *node); +static void ExplainCustomScan(CustomScanState *node, List *ancestors, ExplainState *es); + +const CustomPathMethods AnnCustomPathMethods = { + .CustomName = "AnnFilteredSearch", + .PlanCustomPath = PlanCustomPath +}; + +const CustomScanMethods AnnCustomScanMethods = { + .CustomName = "AnnFilteredSearch", + .CreateCustomScanState = CreateCustomScanState +}; + +static const CustomExecMethods AnnCustomExecMethods = { + .CustomName = "AnnFilteredSearch", + .BeginCustomScan = BeginCustomScan, + .ExecCustomScan = ExecCustomScan, + .EndCustomScan = EndCustomScan, + .ReScanCustomScan = ReScanCustomScan, + .ExplainCustomScan = ExplainCustomScan +}; + +static Plan * +PlanCustomPath(PlannerInfo *root, RelOptInfo *rel, CustomPath *path, + List *tlist, List *clauses, List *custom_plans) +{ + CustomScan *scan = makeNode(CustomScan); + Plan *child = linitial(custom_plans); + OpExpr *distance = lsecond(path->custom_private); + List *sorts = linitial(path->custom_private); + List *columns = NIL; + List *operators = NIL; + List *collations = NIL; + List *nulls = NIL; + TargetEntry *entry; + ListCell *lc; + int distanceAttno = 0; + + scan->scan.plan.targetlist = tlist; + scan->flags = path->flags; + scan->methods = &AnnCustomScanMethods; + scan->custom_scan_tlist = copyObject(child->targetlist); + foreach(lc, sorts) + { + SortGroupClause *sort = lfirst(lc); + + entry = get_sortgroupclause_tle(sort, child->targetlist); + columns = lappend_int(columns, entry->resno); + operators = lappend_oid(operators, sort->sortop); + collations = lappend_oid(collations, exprCollation((Node *) entry->expr)); + nulls = lappend_int(nulls, sort->nulls_first); + } + + /* + * Replace the distance projection with its vector input in the predicate + * child. The custom executor fills that output column only after quals. + */ + foreach(lc, child->targetlist) + { + entry = lfirst(lc); + if (equal(entry->expr, distance)) + { + distanceAttno = entry->resno; + entry->expr = copyObject(linitial(distance->args)); + break; + } + } + if (!distanceAttno) + elog(ERROR, "missing distance column in ann predicate plan"); + scan->custom_plans = custom_plans; + scan->custom_exprs = list_make3(copyObject(lsecond(distance->args)), + copyObject(list_nth(path->custom_private, 2)), + copyObject(list_nth(path->custom_private, 3))); + scan->custom_private = list_make5(columns, operators, collations, nulls, + list_make3(makeInteger(distanceAttno), + makeInteger((int) get_opcode(distance->opno)), + copyObject(list_nth(path->custom_private, 4)))); + return &scan->scan.plan; +} + +static Node * +CreateCustomScanState(CustomScan *scan) +{ + AnnCustomScanState *state = palloc0_object(AnnCustomScanState); + + NodeSetTag(&state->css, T_CustomScanState); + state->css.methods = &AnnCustomExecMethods; + return (Node *) state; +} + +static void +BeginCustomScan(CustomScanState *node, EState *estate, int eflags) +{ + AnnCustomScanState *state = (AnnCustomScanState *) node; + CustomScan *scan = (CustomScan *) node->ss.ps.plan; + List *data = scan->custom_private; + List *distance = list_nth(data, 4); + Oid proc = (Oid) intVal(lsecond(distance)); + AclResult aclresult; + + /* Cached plans must preserve complete eligibility after GUC changes. */ + if (GinFuzzySearchLimit != 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ann filtered search requires gin_fuzzy_search_limit = 0"))); + state->child = ExecInitNode(linitial(scan->custom_plans), estate, eflags); + node->custom_ps = list_make1(state->child); + state->input = MakeSingleTupleTableSlot(node->ss.ss_ScanTupleSlot->tts_tupleDescriptor, &TTSOpsVirtual); + state->output = MakeSingleTupleTableSlot(node->ss.ss_ScanTupleSlot->tts_tupleDescriptor, &TTSOpsMinimalTuple); + state->query = ExecInitExpr(linitial(scan->custom_exprs), &node->ss.ps); + state->limit = ExecInitExpr(lsecond(scan->custom_exprs), &node->ss.ps); + state->offset = ExecInitExpr(lthird(scan->custom_exprs), &node->ss.ps); + state->distanceAttno = intVal(linitial(distance)); + state->withTies = intVal(lthird(distance)); + /* The moved operator must retain ordinary executor function checks. */ +#if PG_VERSION_NUM >= 160000 + aclresult = object_aclcheck(ProcedureRelationId, proc, GetUserId(), ACL_EXECUTE); +#else + aclresult = pg_proc_aclcheck(proc, GetUserId(), ACL_EXECUTE); +#endif + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_FUNCTION, get_func_name(proc)); + InvokeFunctionExecuteHook(proc); + fmgr_info(proc, &state->distance); + fmgr_info_set_expr((Node *) get_tle_by_resno(scan->custom_scan_tlist, state->distanceAttno)->expr, + &state->distance); + state->nkeys = list_length(linitial(data)); + state->sortColIdx = palloc(mul_size(sizeof(AttrNumber), state->nkeys)); + state->sortOperators = palloc(mul_size(sizeof(Oid), state->nkeys)); + state->collations = palloc(mul_size(sizeof(Oid), state->nkeys)); + state->nullsFirst = palloc(mul_size(sizeof(bool), state->nkeys)); + for (int i = 0; i < state->nkeys; i++) + { + state->sortColIdx[i] = list_nth_int(linitial(data), i); + state->sortOperators[i] = list_nth_oid(lsecond(data), i); + state->collations[i] = list_nth_oid(lthird(data), i); + state->nullsFirst[i] = list_nth_int(lfourth(data), i); + } +} + +/* Bounds come from current execution parameters, never planner estimates. */ +static int64 +GetSortBound(AnnCustomScanState *state) +{ + ExprContext *context = state->css.ss.ps.ps_ExprContext; + bool isnull; + int64 limit; + int64 offset = 0; + Datum value; + + if (state->withTies || !state->limit) + return -1; + value = ExecEvalExprSwitchContext(state->limit, context, &isnull); + if (isnull) + return -1; + limit = DatumGetInt64(value); + if (state->offset) + { + value = ExecEvalExprSwitchContext(state->offset, context, &isnull); + if (!isnull) + offset = DatumGetInt64(value); + } + if (limit <= 0 || offset < 0 || limit > PG_INT64_MAX - offset) + return -1; + return limit + offset; +} + +/* Score one child row with the PostgreSQL operator and add it to the sort. */ +static void +PutRow(AnnCustomScanState *state, TupleTableSlot *slot) +{ + ExprContext *context = state->css.ss.ps.ps_ExprContext; + TupleTableSlot *input = state->input; + int attno = state->distanceAttno - 1; + MemoryContext oldCtx; + + slot_getallattrs(slot); + ExecClearTuple(input); + memcpy(input->tts_values, slot->tts_values, sizeof(Datum) * slot->tts_tupleDescriptor->natts); + memcpy(input->tts_isnull, slot->tts_isnull, sizeof(bool) * slot->tts_tupleDescriptor->natts); + oldCtx = MemoryContextSwitchTo(context->ecxt_per_tuple_memory); + if (state->queryIsNull || input->tts_isnull[attno]) + { + input->tts_isnull[attno] = true; + input->tts_values[attno] = (Datum) 0; + } + else + { + input->tts_values[attno] = FunctionCall2Coll(&state->distance, InvalidOid, + PointerGetDatum(DatumGetVector(input->tts_values[attno])), + state->queryDatum); + state->distanceEvaluations++; + } + ExecStoreVirtualTuple(input); + MemoryContextSwitchTo(oldCtx); + tuplesort_puttupleslot(state->sortstate, input); + ExecClearTuple(input); + ResetExprContext(context); +} + +/* Chunk tuples omit the vector; rebuild it from the chunk's floats. */ +static void +PutChunkRow(AnnCustomScanState *state, AnnChunk *chunk, int i) +{ + TupleTableSlot *slot = state->chunkSlot; + int attno = state->distanceAttno - 1; + MemoryContext oldCtx = MemoryContextSwitchTo(chunk->context); + Vector *vector = InitVector(state->dimensions); + + MemoryContextSwitchTo(oldCtx); + memcpy(vector->x, chunk->vectors + (size_t) i * state->dimensions, + sizeof(float) * state->dimensions); + ExecStoreMinimalTuple(chunk->tuples[i], slot, false); + slot_getallattrs(slot); + slot->tts_values[attno] = PointerGetDatum(vector); + slot->tts_isnull[attno] = false; + PutRow(state, slot); + ExecClearTuple(slot); +} + +static void +ResetChunk(AnnChunk *chunk) +{ + MemoryContextReset(chunk->context); + chunk->request = 0; + chunk->count = 0; + chunk->tuples = NULL; + chunk->vectors = NULL; +} + +static void +ScoreChunkLocally(AnnCustomScanState *state, AnnChunk *chunk) +{ + for (int i = 0; i < chunk->count; i++) + PutChunkRow(state, chunk, i); + state->localChunks++; + ResetChunk(chunk); +} + +/* Stop using the worker; rows already sent are scored here instead. */ +static void +AbandonWorker(AnnCustomScanState *state) +{ + state->useWorker = false; + if (state->filling >= 0) + { + ScoreChunkLocally(state, &state->chunks[state->filling]); + state->filling = -1; + } + for (int i = 0; i < state->nchunks; i++) + if (state->chunks[i].request != 0) + ScoreChunkLocally(state, &state->chunks[i]); + state->inflight = 0; +} + +/* + * The worker returns every row of a chunk near or within that chunk's top + * bound. Each global top-bound row is in its chunk's top bound, so the sort + * only needs these candidates, re-scored with the SQL operator. + */ +static void +ReceiveChunk(AnnCustomScanState *state) +{ + uint64 request; + uint32 status; + uint32 *ids; + uint32 count; + AnnChunk *chunk = NULL; + + if (!AnnWorkerReceive(&request, &status, &ids, &count)) + { + AbandonWorker(state); + return; + } + for (int i = 0; i < state->nchunks; i++) + if (state->chunks[i].request == request) + chunk = &state->chunks[i]; + if (chunk == NULL) + { + AbandonWorker(state); + return; + } + state->inflight--; + if (status != 0) + { + /* Overload and other request errors fall back for this chunk only. */ + ScoreChunkLocally(state, chunk); + return; + } + for (uint32 i = 0; i < count; i++) + if (ids[i] >= (uint32) chunk->count || (i && ids[i] == ids[i - 1])) + { + pfree(ids); + ScoreChunkLocally(state, chunk); + return; + } + for (uint32 i = 0; i < count; i++) + PutChunkRow(state, chunk, ids[i]); + pfree(ids); + state->workerChunks++; + state->workerCandidates += count; + ResetChunk(chunk); +} + +static void +SendChunk(AnnCustomScanState *state) +{ + AnnChunk *chunk = &state->chunks[state->filling]; + + state->filling = -1; + while (state->useWorker && state->inflight > 0 && AnnWorkerReady()) + ReceiveChunk(state); + + /* + * With the pipeline full, score here instead of waiting. A busy worker + * then sheds work to its clients rather than adding queueing delay. + */ + if (!state->useWorker || state->inflight >= state->nchunks - 1) + { + ScoreChunkLocally(state, chunk); + return; + } + chunk->request = AnnWorkerSubmit(chunk->count, state->dimensions, (uint32) state->bound, + DatumGetVector(state->queryDatum)->x, chunk->vectors); + if (chunk->request == 0) + { + ScoreChunkLocally(state, chunk); + AbandonWorker(state); + return; + } + state->inflight++; +} + +static void +AddChunkRow(AnnCustomScanState *state, TupleTableSlot *slot, Vector *vector) +{ + AnnChunk *chunk; + MemoryContext oldCtx; + + if (state->filling < 0) + { + for (int i = 0; i < state->nchunks; i++) + if (state->chunks[i].request == 0) + { + state->filling = i; + break; + } + Assert(state->filling >= 0); + chunk = &state->chunks[state->filling]; + oldCtx = MemoryContextSwitchTo(chunk->context); + chunk->tuples = palloc(sizeof(MinimalTuple) * state->chunkRows); + chunk->vectors = chunk->shared ? chunk->shared : + palloc(mul_size(sizeof(float) * state->dimensions, state->chunkRows)); + MemoryContextSwitchTo(oldCtx); + } + chunk = &state->chunks[state->filling]; + memcpy(state->nulls, slot->tts_isnull, sizeof(bool) * slot->tts_tupleDescriptor->natts); + state->nulls[state->distanceAttno - 1] = true; + oldCtx = MemoryContextSwitchTo(chunk->context); +#if PG_VERSION_NUM >= 180000 + chunk->tuples[chunk->count] = heap_form_minimal_tuple(slot->tts_tupleDescriptor, + slot->tts_values, state->nulls, 0); +#else + chunk->tuples[chunk->count] = heap_form_minimal_tuple(slot->tts_tupleDescriptor, + slot->tts_values, state->nulls); +#endif + MemoryContextSwitchTo(oldCtx); + memcpy(chunk->vectors + (size_t) chunk->count * state->dimensions, vector->x, + sizeof(float) * state->dimensions); + if (++chunk->count == state->chunkRows) + SendChunk(state); +} + +/* + * Use the worker only when a bound makes per-chunk candidates much smaller + * than the chunk. The chunk size and depth are fixed for the whole scan. + */ +static void +BeginWorkerRoute(AnnCustomScanState *state) +{ + MemoryContext queryCtx = state->css.ss.ps.state->es_query_cxt; + int depth = ann_worker_pipeline_depth; + size_t sharedSize; + float *shared; + size_t slotBytes; + + state->useWorker = false; + if (state->queryIsNull || state->bound <= 0 || + state->bound > ann_worker_chunk_rows / 2 || !AnnWorkerBegin()) + return; + state->dimensions = DatumGetVector(state->queryDatum)->dim; + state->chunkRows = ann_worker_chunk_rows; + if (state->chunks == NULL) + { + state->nchunks = depth + 1; + state->chunks = MemoryContextAllocZero(queryCtx, sizeof(AnnChunk) * state->nchunks); + for (int i = 0; i < state->nchunks; i++) + state->chunks[i].context = AllocSetContextCreate(queryCtx, "ann worker chunk", + ALLOCSET_DEFAULT_SIZES); + state->chunkSlot = MakeSingleTupleTableSlot(ExecGetResultType(state->child), + &TTSOpsMinimalTuple); + state->nulls = MemoryContextAlloc(queryCtx, sizeof(bool) * + ExecGetResultType(state->child)->natts); + } + + /* Split shared memory into one slot per chunk, shrinking chunks to fit. */ + shared = AnnWorkerSharedBuffer(&sharedSize); + slotBytes = sharedSize / state->nchunks / sizeof(float) * sizeof(float); + if (shared && slotBytes / (sizeof(float) * state->dimensions) >= (size_t) state->bound * 2) + { + state->chunkRows = Min(state->chunkRows, slotBytes / (sizeof(float) * state->dimensions)); + for (int i = 0; i < state->nchunks; i++) + state->chunks[i].shared = shared + i * (slotBytes / sizeof(float)); + } + else + for (int i = 0; i < state->nchunks; i++) + state->chunks[i].shared = NULL; + state->filling = -1; + state->inflight = 0; + state->useWorker = true; + state->workerUsed = true; +} + +static void +CollectChildRows(AnnCustomScanState *state) +{ + ExprContext *context = state->css.ss.ps.ps_ExprContext; + int attno = state->distanceAttno - 1; + int64 bound = state->bound; + uint64 nullRows = 0; + + for (;;) + { + TupleTableSlot *slot; + + CHECK_FOR_INTERRUPTS(); + slot = ExecProcNode(state->child); + if (TupIsNull(slot)) + break; + state->eligibleRows++; + slot_getallattrs(slot); + if (state->useWorker) + { + Vector *vector; + MemoryContext oldCtx; + + /* NULL vectors sort last, so only the first bound can matter. */ + if (slot->tts_isnull[attno]) + { + if (nullRows++ < (uint64) bound) + PutRow(state, slot); + continue; + } + oldCtx = MemoryContextSwitchTo(context->ecxt_per_tuple_memory); + vector = DatumGetVector(slot->tts_values[attno]); + MemoryContextSwitchTo(oldCtx); + + /* Mismatched dimensions raise the operator's own error. */ + if (vector->dim == state->dimensions) + { + AddChunkRow(state, slot, vector); + ResetExprContext(context); + continue; + } + } + PutRow(state, slot); + } + if (state->useWorker && state->filling >= 0) + SendChunk(state); + while (state->useWorker && state->inflight > 0) + ReceiveChunk(state); +} + +static void +CollectRows(AnnCustomScanState *state) +{ + CustomScanState *node = &state->css; + ExprContext *context = node->ss.ps.ps_ExprContext; + int64 bound = GetSortBound(state); + + state->sortstate = tuplesort_begin_heap(state->input->tts_tupleDescriptor, state->nkeys, + state->sortColIdx, state->sortOperators, + state->collations, state->nullsFirst, + work_mem, NULL, bound > 0 ? TUPLESORT_ALLOWBOUNDED : TUPLESORT_NONE); + if (bound > 0) + tuplesort_set_bound(state->sortstate, bound); + state->queryDatum = ExecEvalExprSwitchContext(state->query, context, &state->queryIsNull); + if (!state->queryIsNull) + { + MemoryContext oldCtx = MemoryContextSwitchTo(node->ss.ps.state->es_query_cxt); + + state->queryDatum = PointerGetDatum(PG_DETOAST_DATUM_COPY(state->queryDatum)); + MemoryContextSwitchTo(oldCtx); + } + state->bound = bound; + BeginWorkerRoute(state); + if (!state->useWorker) + CollectChildRows(state); + else + { + PG_TRY(); + { + CollectChildRows(state); + } + PG_FINALLY(); + { + AnnWorkerEnd(); + } + PG_END_TRY(); + } + tuplesort_performsort(state->sortstate); + if (!state->queryIsNull) + pfree(DatumGetPointer(state->queryDatum)); +} + +static TupleTableSlot * +GetNextRow(ScanState *scan) +{ + AnnCustomScanState *state = (AnnCustomScanState *) scan; + TupleTableSlot *slot = state->output; + + Assert(ScanDirectionIsForward(scan->ps.state->es_direction)); + if (!state->sortstate) + CollectRows(state); + if (!tuplesort_gettupleslot(state->sortstate, true, false, slot, NULL)) + return NULL; + return slot; +} + +static bool +RecheckRow(ScanState *scan, TupleTableSlot *slot) +{ + return true; +} + +static TupleTableSlot * +ExecCustomScan(CustomScanState *node) +{ + return ExecScan(&node->ss, GetNextRow, RecheckRow); +} + +static void +EndCustomScan(CustomScanState *node) +{ + AnnCustomScanState *state = (AnnCustomScanState *) node; + + ExecClearTuple(node->ss.ss_ScanTupleSlot); + ExecDropSingleTupleTableSlot(state->input); + ExecDropSingleTupleTableSlot(state->output); + if (state->chunkSlot) + ExecDropSingleTupleTableSlot(state->chunkSlot); + if (state->sortstate) + tuplesort_end(state->sortstate); + ExecEndNode(state->child); +#if PG_VERSION_NUM >= 170000 + /* Earlier versions release these in ExecEndCustomScan itself. */ + FreeExprContext(node->ss.ps.ps_ExprContext, true); + ExecClearTuple(node->ss.ps.ps_ResultTupleSlot); +#endif +} + +static void +ReScanCustomScan(CustomScanState *node) +{ + AnnCustomScanState *state = (AnnCustomScanState *) node; + + ExecClearTuple(node->ss.ss_ScanTupleSlot); + ExecClearTuple(state->output); + if (state->sortstate) + { + tuplesort_end(state->sortstate); + state->sortstate = NULL; + } + if (node->ss.ps.chgParam) + UpdateChangedParamSet(state->child, node->ss.ps.chgParam); + ExecReScan(state->child); +} + +static void +ExplainCustomScan(CustomScanState *node, List *ancestors, ExplainState *es) +{ + AnnCustomScanState *state = (AnnCustomScanState *) node; + + if (es->analyze && state->workerUsed) + ExplainPropertyText("Search Route", "worker pipeline after predicates", es); + else if (ann_worker_socket && ann_worker_socket[0] != '\0') + ExplainPropertyText("Search Route", "worker pipeline or exact CPU after predicates", es); + else + ExplainPropertyText("Search Route", "exact CPU after predicates", es); + if (es->analyze) + { + if (state->workerUsed) + { + ExplainPropertyInteger("Worker Chunks", NULL, state->workerChunks, es); + ExplainPropertyInteger("Local Chunks", NULL, state->localChunks, es); + ExplainPropertyInteger("Worker Candidates", NULL, state->workerCandidates, es); + } + ExplainPropertyInteger("Eligible Rows", NULL, state->eligibleRows, es); + ExplainPropertyInteger("Distance Evaluations", NULL, state->distanceEvaluations, es); + if (state->sortstate) + { + TuplesortInstrumentation stats; + + tuplesort_get_stats(state->sortstate, &stats); + ExplainPropertyText("Sort Method", tuplesort_method_name(stats.sortMethod), es); + ExplainPropertyInteger("Sort Space Used", "kB", stats.spaceUsed, es); + ExplainPropertyText("Sort Space Type", tuplesort_space_type_name(stats.spaceType), es); + } + } +} diff --git a/src/annindex.c b/src/annindex.c new file mode 100644 index 000000000..f699f8f21 --- /dev/null +++ b/src/annindex.c @@ -0,0 +1,607 @@ +#include "postgres.h" + +#include "ann.h" +#include "catalog/namespace.h" +#include "catalog/pg_type_d.h" +#include "executor/spi.h" +#include "fmgr.h" +#include "funcapi.h" +#include "miscadmin.h" +#include "utils/array.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/regproc.h" +#include "utils/tuplestore.h" +#include "vector.h" + +#define ANN_OP_LOAD_INDEX 11 +#define ANN_OP_INDEX_SEARCH_PACKED 14 +#define ANN_OP_INDEX_INFO 15 +#define ANN_INDEX_NAME_MAX 63 +#define ANN_INDEX_MAX_K 10000 +#define ANN_LOAD_TIMEOUT_MS (30 * 60 * 1000) +#define ANN_KEY_FETCH 65536 + +/* Statuses shared with ann/backend.h. */ +#define ANN_STATUS_OK 0 +#define ANN_STATUS_INVALID 1 +#define ANN_STATUS_UNSUPPORTED 2 +#define ANN_STATUS_STALE 3 +#define ANN_STATUS_OVERLOADED 7 + +/* What the worker reports for a loaded index. */ +typedef struct AnnIndexInfo +{ + uint64 rows; + char source[256]; + char key[64]; + char vector[64]; +} AnnIndexInfo; + +/* Catalog row: where an index's vectors live on this host. */ +typedef struct AnnCatalogEntry +{ + char *source; /* quoted relation name */ + char *key; /* quoted column names */ + char *vector; +} AnnCatalogEntry; + +static void +Put32(StringInfo buffer, uint32 value) +{ + appendBinaryStringInfo(buffer, (char *) &value, sizeof(value)); +} + +static const char * +CheckName(text *name) +{ + char *value = text_to_cstring(name); + + if (strlen(value) == 0 || strlen(value) > ANN_INDEX_NAME_MAX) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("ann index name must have 1 to %d bytes", ANN_INDEX_NAME_MAX))); + return value; +} + +/* Networked GPU nodes when configured, otherwise the local worker socket. */ +static bool +IndexCall(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength) +{ + if (AnnRemoteConfigured()) + return AnnRemoteCall(operation, payload, length, timeoutMs, status, reply, replyLength); + return AnnWorkerCall(operation, payload, length, timeoutMs, status, reply, replyLength); +} + +static void +CheckWorker(bool ok, uint32 status, const char *name) +{ + if (!ok) + ereport(ERROR, + (errcode(ERRCODE_CONNECTION_FAILURE), + errmsg("ann worker is unavailable"), + errhint("Set ann.worker_address or ann.worker_socket to a running ann-worker."))); + if (status == ANN_STATUS_OK) + return; + if (status == ANN_STATUS_OVERLOADED) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_RESOURCES), + errmsg("ann worker is overloaded"))); + if (status == ANN_STATUS_STALE) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("ann index \"%s\" is not loaded, or is already loaded", name))); + if (status == ANN_STATUS_UNSUPPORTED) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ann worker backend does not serve loaded indexes"))); + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("ann worker rejected the request for index \"%s\" (status %u)", name, status))); +} + +static void +ParseInfo(const char *reply, AnnIndexInfo *info) +{ + unsigned long long rows; + + strcpy(info->key, "-"); + strcpy(info->vector, "-"); + if (sscanf(reply, "rows %llu dimensions %*u shards %*u dtype %*s source %255s key %63s vector %63s", + &rows, info->source, info->key, info->vector) < 2) + elog(ERROR, "unexpected ann worker reply: %s", reply); + info->rows = rows; +} + +/* Schema-qualified catalog name, in the extension's schema. */ +static char * +CatalogName(FunctionCallInfo fcinfo) +{ + char *schema = get_namespace_name(get_func_namespace(fcinfo->flinfo->fn_oid)); + + return psprintf("%s.ann_index_catalog", quote_identifier(schema)); +} + +static char * +QualifiedSource(const char *source) +{ + List *names = stringToQualifiedNameList(source +#if PG_VERSION_NUM >= 160000 + ,NULL +#endif + ); + + return NameListToQuotedString(names); +} + +/* + * The index is immutable: refuse one whose source table on this host no + * longer has the row count it was built from. Rebuild after changing the + * table. + */ +static void +CheckSource(const AnnIndexInfo *info) +{ + char *query; + bool isnull; + int64 count; + + if (strcmp(info->source, "-") == 0) + return; + query = psprintf("SELECT count(*) FROM %s", QualifiedSource(info->source)); + if (SPI_connect() != SPI_OK_CONNECT) + elog(ERROR, "SPI_connect failed"); + if (SPI_execute(query, true, 1) != SPI_OK_SELECT || SPI_processed != 1) + elog(ERROR, "could not count rows of %s", info->source); + count = DatumGetInt64(SPI_getbinval(SPI_tuptable->vals[0], SPI_tuptable->tupdesc, 1, &isnull)); + SPI_finish(); + if (count != (int64) info->rows) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("ann index is stale: %s has " INT64_FORMAT " rows, index has " UINT64_FORMAT, + info->source, count, info->rows), + errhint("Rebuild the index with scripts/ann/build_cuvs_index.py."))); +} + +/* Record where the index's rows live, so this host can route exact scans. */ +static void +RecordCatalog(FunctionCallInfo fcinfo, const char *name, const AnnIndexInfo *info) +{ + Oid types[5] = {TEXTOID, TEXTOID, TEXTOID, TEXTOID, INT8OID}; + Datum values[5]; + + if (strcmp(info->source, "-") == 0 || strcmp(info->key, "-") == 0 || strcmp(info->vector, "-") == 0) + return; + values[0] = CStringGetTextDatum(name); + values[1] = CStringGetTextDatum(QualifiedSource(info->source)); + values[2] = CStringGetTextDatum(info->key); + values[3] = CStringGetTextDatum(info->vector); + values[4] = Int64GetDatum((int64) info->rows); + if (SPI_connect() != SPI_OK_CONNECT) + elog(ERROR, "SPI_connect failed"); + if (SPI_execute_with_args(psprintf("INSERT INTO %s (name, source, key_column, vector_column, rows) " + "VALUES ($1, $2::regclass, $3, $4, $5) ON CONFLICT (name) DO UPDATE " + "SET source = EXCLUDED.source, key_column = EXCLUDED.key_column, " + "vector_column = EXCLUDED.vector_column, rows = EXCLUDED.rows, " + "loaded_at = now()", CatalogName(fcinfo)), + 5, types, values, NULL, false, 0) != SPI_OK_INSERT) + elog(ERROR, "could not record ann index \"%s\"", name); + SPI_finish(); +} + +static bool +LookupCatalog(FunctionCallInfo fcinfo, const char *name, AnnCatalogEntry *entry) +{ + Oid types[1] = {TEXTOID}; + Datum values[1]; + bool found; + + values[0] = CStringGetTextDatum(name); + if (SPI_connect() != SPI_OK_CONNECT) + elog(ERROR, "SPI_connect failed"); + if (SPI_execute_with_args(psprintf("SELECT source::text, key_column::text, vector_column::text " + "FROM %s WHERE name = $1", CatalogName(fcinfo)), + 1, types, values, NULL, true, 1) != SPI_OK_SELECT) + elog(ERROR, "could not read the ann index catalog"); + found = SPI_processed == 1; + if (found) + { + MemoryContext old = MemoryContextSwitchTo(fcinfo->flinfo->fn_mcxt); + + entry->source = pstrdup(SPI_getvalue(SPI_tuptable->vals[0], SPI_tuptable->tupdesc, 1)); + entry->key = pstrdup(quote_identifier(SPI_getvalue(SPI_tuptable->vals[0], SPI_tuptable->tupdesc, 2))); + entry->vector = pstrdup(quote_identifier(SPI_getvalue(SPI_tuptable->vals[0], SPI_tuptable->tupdesc, 3))); + MemoryContextSwitchTo(old); + } + SPI_finish(); + return found; +} + +static void +RequireSuperuser(const char *action) +{ + if (!superuser()) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("must be superuser to %s an ann index", action))); +} + +/* + * ann_index_load(name, path): load an index directory into the worker, or + * into every node of ann.worker_address. Workers read the files, so the path + * is resolved on each GPU node. + */ +FUNCTION_PREFIX PG_FUNCTION_INFO_V1(ann_index_load); +Datum +ann_index_load(PG_FUNCTION_ARGS) +{ + const char *name = CheckName(PG_GETARG_TEXT_PP(0)); + char *path = text_to_cstring(PG_GETARG_TEXT_PP(1)); + StringInfoData payload; + uint32 status = 0; + char *reply = NULL; + uint32 replyLength; + AnnIndexInfo info; + bool ok; + + RequireSuperuser("load"); + initStringInfo(&payload); + Put32(&payload, strlen(name)); + appendBinaryStringInfo(&payload, name, strlen(name)); + appendBinaryStringInfo(&payload, path, strlen(path)); + if (AnnRemoteConfigured()) + ok = AnnRemoteBroadcast(ANN_OP_LOAD_INDEX, payload.data, payload.len, ANN_LOAD_TIMEOUT_MS, + &status, &reply, &replyLength) > 0 || status != PG_UINT32_MAX; + else + ok = AnnWorkerCall(ANN_OP_LOAD_INDEX, payload.data, payload.len, ANN_LOAD_TIMEOUT_MS, + &status, &reply, &replyLength); + CheckWorker(ok, status, name); + ParseInfo(reply, &info); + CheckSource(&info); + RecordCatalog(fcinfo, name, &info); + PG_RETURN_INT64((int64) info.rows); +} + +/* + * ann_index_attach(name): use an index another host already loaded on the + * GPU nodes. Verifies this host's copy of the source table, then records it. + */ +FUNCTION_PREFIX PG_FUNCTION_INFO_V1(ann_index_attach); +Datum +ann_index_attach(PG_FUNCTION_ARGS) +{ + const char *name = CheckName(PG_GETARG_TEXT_PP(0)); + uint32 status = 0; + char *reply = NULL; + uint32 replyLength; + AnnIndexInfo info; + bool ok; + + RequireSuperuser("attach"); + ok = IndexCall(ANN_OP_INDEX_INFO, name, strlen(name), 0, &status, &reply, &replyLength); + CheckWorker(ok, status, name); + ParseInfo(reply, &info); + CheckSource(&info); + RecordCatalog(fcinfo, name, &info); + PG_RETURN_INT64((int64) info.rows); +} + +static int +KeyCompare(const void *a, const void *b) +{ + int64 x = *(const int64 *) a; + int64 y = *(const int64 *) b; + + return x < y ? -1 : x > y; +} + +/* Sort when needed and drop duplicates; returns the new count. */ +static int +SortKeys(int64 *keys, int nkeys) +{ + int n = 0; + + for (int i = 1; i < nkeys; i++) + if (keys[i] < keys[i - 1]) + { + qsort(keys, nkeys, sizeof(int64), KeyCompare); + break; + } + for (int i = 0; i < nkeys; i++) + if (i == 0 || keys[i] != keys[i - 1]) + keys[n++] = keys[i]; + return n; +} + +static void +PutVarint(StringInfo buffer, uint64 value) +{ + char bytes[10]; + int n = 0; + + do + { + bytes[n] = (char) (value & 0x7f); + value >>= 7; + if (value) + bytes[n] |= (char) 0x80; + n++; + } while (value); + appendBinaryStringInfo(buffer, bytes, n); +} + +static void +InitResult(FunctionCallInfo fcinfo, TupleDesc *tupdesc, Tuplestorestate **store) +{ + ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; + MemoryContext oldcontext; + + if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo) || + !(rsinfo->allowedModes & SFRM_Materialize)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("set-valued function called in context that cannot accept a set"))); + if (get_call_result_type(fcinfo, NULL, tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR, "return type must be a row type"); + oldcontext = MemoryContextSwitchTo(rsinfo->econtext->ecxt_per_query_memory); + *tupdesc = CreateTupleDescCopy(*tupdesc); + *store = tuplestore_begin_heap(true, false, work_mem); + MemoryContextSwitchTo(oldcontext); + rsinfo->returnMode = SFRM_Materialize; + rsinfo->setResult = *store; + rsinfo->setDesc = *tupdesc; +} + +/* + * Ask a GPU node for the top k, restricted to sorted unique keys when + * filtered. Keys travel as a zigzag varint first key and varint gaps. + */ +static void +SearchGpu(const char *name, Vector *query, int32 k, bool filtered, const int64 *keys, int nkeys, + TupleDesc tupdesc, Tuplestorestate *store) +{ + StringInfoData payload; + StringInfoData packed; + uint32 status = 0; + char *reply = NULL; + uint32 replyLength; + uint32 count; + bool ok; + + initStringInfo(&packed); + for (int i = 0; i < nkeys; i++) + { + if (i == 0) + PutVarint(&packed, ((uint64) keys[0] << 1) ^ (uint64) (keys[0] >> 63)); + else + PutVarint(&packed, (uint64) keys[i] - (uint64) keys[i - 1]); + } + initStringInfo(&payload); + Put32(&payload, strlen(name)); + appendBinaryStringInfo(&payload, name, strlen(name)); + Put32(&payload, query->dim); + Put32(&payload, k); + Put32(&payload, filtered ? (uint32) nkeys : PG_UINT32_MAX); + Put32(&payload, packed.len); + appendBinaryStringInfo(&payload, (char *) query->x, sizeof(float) * query->dim); + appendBinaryStringInfo(&payload, packed.data, packed.len); + ok = IndexCall(ANN_OP_INDEX_SEARCH_PACKED, payload.data, payload.len, 0, &status, &reply, &replyLength); + CheckWorker(ok, status, name); + if (replyLength < 8 || (count = *(uint32 *) reply) != (replyLength - 8) / 12 || + (Size) count * 12 + 8 != replyLength) + elog(ERROR, "malformed ann worker search reply"); + for (uint32 i = 0; i < count; i++) + { + Datum values[2]; + bool nulls[2] = {false, false}; + int64 key; + float distance; + + memcpy(&key, reply + 8 + (Size) i * 12, sizeof(key)); + memcpy(&distance, reply + 16 + (Size) i * 12, sizeof(distance)); + values[0] = Int64GetDatum(key); + values[1] = Float4GetDatum(distance); + tuplestore_putvalues(store, tupdesc, values, nulls); + } +} + +/* + * Exact top k on this host with the SQL operator, over either eligible keys + * or a filter predicate. The query plan is PostgreSQL's own. + */ +static void +SearchLocal(FunctionCallInfo fcinfo, const AnnCatalogEntry *entry, Datum query, int32 k, + const char *filter, const int64 *keys, int nkeys, TupleDesc tupdesc, Tuplestorestate *store) +{ + Oid types[3]; + Datum values[3]; + char *sql; + SPIPlanPtr plan; + + types[0] = get_fn_expr_argtype(fcinfo->flinfo, 1); + values[0] = query; + types[1] = INT4OID; + values[1] = Int32GetDatum(k); + if (filter) + sql = psprintf("SELECT %s::bigint, (%s <-> $1)::real FROM %s WHERE (%s) ORDER BY 2, 1 LIMIT $2", + entry->key, entry->vector, entry->source, filter); + else + { + Datum *elements = palloc(sizeof(Datum) * Max(nkeys, 1)); + + for (int i = 0; i < nkeys; i++) + elements[i] = Int64GetDatum(keys[i]); + types[2] = INT8ARRAYOID; + values[2] = PointerGetDatum(construct_array(elements, nkeys, INT8OID, 8, FLOAT8PASSBYVAL, TYPALIGN_DOUBLE)); + sql = psprintf("SELECT %s::bigint, (%s <-> $1)::real FROM %s WHERE %s = ANY ($3) ORDER BY 2, 1 LIMIT $2", + entry->key, entry->vector, entry->source, entry->key); + } + if (SPI_connect() != SPI_OK_CONNECT) + elog(ERROR, "SPI_connect failed"); + + /* One prepared read-only statement: a filter cannot smuggle in writes. */ + plan = SPI_prepare(sql, filter ? 2 : 3, types); + if (plan == NULL) + elog(ERROR, "could not prepare ann local search: %s", SPI_result_code_string(SPI_result)); + if (SPI_execute_plan(plan, values, NULL, true, 0) != SPI_OK_SELECT) + elog(ERROR, "ann local search failed"); + for (uint64 i = 0; i < SPI_processed; i++) + { + Datum row[2]; + bool nulls[2]; + + row[0] = SPI_getbinval(SPI_tuptable->vals[i], SPI_tuptable->tupdesc, 1, &nulls[0]); + row[1] = SPI_getbinval(SPI_tuptable->vals[i], SPI_tuptable->tupdesc, 2, &nulls[1]); + tuplestore_putvalues(store, tupdesc, row, nulls); + } + SPI_finish(); +} + +static int32 +CheckK(int32 k) +{ + if (k < 1 || k > ANN_INDEX_MAX_K) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("k must be between 1 and %d", ANN_INDEX_MAX_K))); + return k; +} + +/* + * ann_index_search(name, query, k, filter): top-k (id, distance) from a + * loaded index. A non-NULL filter restricts results to those keys, so any + * SQL predicate can drive hybrid search; NULL elements are ignored. Filters + * smaller than ann.gpu_min_eligible are scored exactly on this host when the + * index is in the catalog. + */ +FUNCTION_PREFIX PG_FUNCTION_INFO_V1(ann_index_search); +Datum +ann_index_search(PG_FUNCTION_ARGS) +{ + TupleDesc tupdesc; + Tuplestorestate *store; + const char *name; + int32 k; + int64 *keys = NULL; + int nkeys = 0; + bool filtered = !PG_ARGISNULL(3); + AnnCatalogEntry entry; + + InitResult(fcinfo, &tupdesc, &store); + if (PG_ARGISNULL(0) || PG_ARGISNULL(1) || PG_ARGISNULL(2)) + return (Datum) 0; + name = CheckName(PG_GETARG_TEXT_PP(0)); + k = CheckK(PG_GETARG_INT32(2)); + if (filtered) + { + ArrayType *array = PG_GETARG_ARRAYTYPE_P(3); + Datum *values; + bool *nulls; + int n; + + deconstruct_array(array, INT8OID, 8, FLOAT8PASSBYVAL, TYPALIGN_DOUBLE, &values, &nulls, &n); + keys = palloc(sizeof(int64) * Max(n, 1)); + for (int i = 0; i < n; i++) + if (!nulls[i]) + keys[nkeys++] = DatumGetInt64(values[i]); + + /* No eligible keys means no rows; skip the round trip. */ + if (nkeys == 0) + return (Datum) 0; + nkeys = SortKeys(keys, nkeys); + if (nkeys < ann_gpu_min_eligible && LookupCatalog(fcinfo, name, &entry)) + { + SearchLocal(fcinfo, &entry, PG_GETARG_DATUM(1), k, NULL, keys, nkeys, tupdesc, store); + return (Datum) 0; + } + } + SearchGpu(name, PG_GETARG_VECTOR_P(1), k, filtered, keys, nkeys, tupdesc, store); + return (Datum) 0; +} + +/* + * ann_hybrid_search(name, query, k, filter): hybrid search where this host + * evaluates the SQL filter on the index's source table. A capped count + * decides the route: filters under ann.gpu_min_eligible rows run exactly + * here; larger ones stream their keys to a GPU node. The filter is a SQL + * boolean expression over the source table, run read-only with the + * caller's privileges, like any dynamic SQL; quote values with format(). + */ +FUNCTION_PREFIX PG_FUNCTION_INFO_V1(ann_hybrid_search); +Datum +ann_hybrid_search(PG_FUNCTION_ARGS) +{ + TupleDesc tupdesc; + Tuplestorestate *store; + const char *name; + int32 k; + char *filter; + AnnCatalogEntry entry; + SPIPlanPtr plan; + Portal portal; + int64 *keys; + int nkeys = 0; + int capacity = ANN_KEY_FETCH; + int64 limit = (int64) ann_gpu_min_eligible; + bool isnull; + int64 count; + MemoryContext caller = CurrentMemoryContext; + + InitResult(fcinfo, &tupdesc, &store); + if (PG_ARGISNULL(0) || PG_ARGISNULL(1) || PG_ARGISNULL(2) || PG_ARGISNULL(3)) + return (Datum) 0; + name = CheckName(PG_GETARG_TEXT_PP(0)); + k = CheckK(PG_GETARG_INT32(2)); + filter = text_to_cstring(PG_GETARG_TEXT_PP(3)); + if (!LookupCatalog(fcinfo, name, &entry)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("ann index \"%s\" is not in this database's catalog", name), + errhint("Run ann_index_load or ann_index_attach on this host."))); + if (SPI_connect() != SPI_OK_CONNECT) + elog(ERROR, "SPI_connect failed"); + plan = SPI_prepare(psprintf("SELECT count(*) FROM (SELECT FROM %s WHERE (%s) LIMIT " INT64_FORMAT ") s", + entry.source, filter, limit), 0, NULL); + if (plan == NULL || SPI_execute_plan(plan, NULL, NULL, true, 1) != SPI_OK_SELECT) + elog(ERROR, "could not evaluate the ann hybrid filter"); + count = DatumGetInt64(SPI_getbinval(SPI_tuptable->vals[0], SPI_tuptable->tupdesc, 1, &isnull)); + if (count < limit) + { + SPI_finish(); + SearchLocal(fcinfo, &entry, PG_GETARG_DATUM(1), k, filter, NULL, 0, tupdesc, store); + return (Datum) 0; + } + + /* Broad filter: stream the eligible keys, then search on a GPU node. */ + keys = MemoryContextAlloc(caller, sizeof(int64) * capacity); + plan = SPI_prepare(psprintf("SELECT %s::bigint FROM %s WHERE (%s)", entry.key, entry.source, filter), 0, NULL); + if (plan == NULL || (portal = SPI_cursor_open(NULL, plan, NULL, NULL, true)) == NULL) + elog(ERROR, "could not evaluate the ann hybrid filter"); + for (;;) + { + SPI_cursor_fetch(portal, true, ANN_KEY_FETCH); + if (SPI_processed == 0) + break; + if (nkeys + SPI_processed > (uint64) capacity) + { + while (nkeys + SPI_processed > (uint64) capacity) + capacity *= 2; + keys = repalloc_huge(keys, sizeof(int64) * capacity); + } + for (uint64 i = 0; i < SPI_processed; i++) + { + Datum value = SPI_getbinval(SPI_tuptable->vals[i], SPI_tuptable->tupdesc, 1, &isnull); + + if (!isnull) + keys[nkeys++] = DatumGetInt64(value); + } + SPI_freetuptable(SPI_tuptable); + CHECK_FOR_INTERRUPTS(); + } + SPI_cursor_close(portal); + SPI_finish(); + nkeys = SortKeys(keys, nkeys); + SearchGpu(name, PG_GETARG_VECTOR_P(1), k, true, keys, nkeys, tupdesc, store); + return (Datum) 0; +} diff --git a/src/anninsert.c b/src/anninsert.c new file mode 100644 index 000000000..407ca784e --- /dev/null +++ b/src/anninsert.c @@ -0,0 +1,113 @@ +#include "postgres.h" + +#include "access/generic_xlog.h" +#include "ann.h" +#include "miscadmin.h" +#include "storage/freespace.h" +#include "utils/memutils.h" + +/* + * The metapage serializes insertion. VACUUM only locks data pages, so it never + * reverses this lock order. Free-space-map entries are hints, not authority. + */ +void +AnnInsertTuple(Relation index, Datum *values, bool *isnull, ItemPointer tid, bool building) +{ + Buffer metaBuf = ReadBuffer(index, ANN_METAPAGE_BLKNO); + Buffer buf; + Page metaPage; + Page page; + BlockNumber target; + GenericXLogState *state = NULL; + IndexTuple tuple; + Size size; + Size freeSpace; + Datum tupleValues[1]; + bool newPage = false; + + LockBuffer(metaBuf, BUFFER_LOCK_EXCLUSIVE); + metaPage = BufferGetPage(metaBuf); + AnnCheckMetaPage(metaPage); + tupleValues[0] = isnull[0] ? (Datum) 0 : values[0]; + if (!isnull[0]) + { + Vector *vector = DatumGetVector(values[0]); + + if (vector->dim != (int) AnnPageGetMeta(metaPage)->dimensions) + ereport(ERROR, (errcode(ERRCODE_DATA_EXCEPTION), errmsg("different vector dimensions"))); + tupleValues[0] = PointerGetDatum(vector); + } + tuple = index_form_tuple(RelationGetDescr(index), tupleValues, isnull); + tuple->t_tid = *tid; + size = IndexTupleSize(tuple); + if (size > BLCKSZ - SizeOfPageHeaderData - MAXALIGN(sizeof(AnnPageOpaqueData)) - sizeof(ItemIdData)) + ereport(ERROR, (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), errmsg("ann index tuple is too large"))); + target = AnnPageGetMeta(metaPage)->insertPage; + for (;;) + { + CHECK_FOR_INTERRUPTS(); + if (!BlockNumberIsValid(target)) + { + buf = ReadBufferExtended(index, MAIN_FORKNUM, P_NEW, RBM_NORMAL, NULL); + LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); + newPage = true; + break; + } + buf = ReadBuffer(index, target); + LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); + page = BufferGetPage(buf); + if (PageIsNew(page)) + { + newPage = true; + break; + } + AnnCheckPage(page); + freeSpace = PageGetFreeSpace(page); + if (freeSpace >= size) + break; + UnlockReleaseBuffer(buf); + target = RecordAndGetPageWithFreeSpace(index, target, freeSpace, size); + } + if (!building) + { + state = GenericXLogStart(index); + metaPage = GenericXLogRegisterBuffer(state, metaBuf, 0); + page = GenericXLogRegisterBuffer(state, buf, newPage ? GENERIC_XLOG_FULL_IMAGE : 0); + } + else + page = BufferGetPage(buf); + if (newPage) + { + PageInit(page, BufferGetPageSize(buf), sizeof(AnnPageOpaqueData)); + AnnPageGetOpaque(page)->page_id = ANN_PAGE_ID; + } + if (PageAddItem(page, (char *) tuple, size, InvalidOffsetNumber, false, false) == InvalidOffsetNumber) + elog(ERROR, "failed to add ann index tuple"); + AnnPageGetMeta(metaPage)->insertPage = BufferGetBlockNumber(buf); + freeSpace = PageGetFreeSpace(page); + target = BufferGetBlockNumber(buf); + if (building) + { + MarkBufferDirty(buf); + MarkBufferDirty(metaBuf); + } + else + GenericXLogFinish(state); + UnlockReleaseBuffer(buf); + UnlockReleaseBuffer(metaBuf); + RecordPageWithFreeSpace(index, target, freeSpace); + pfree(tuple); +} + +bool +anninsert(Relation index, Datum *values, bool *isnull, ItemPointer tid, + Relation heap, IndexUniqueCheck checkUnique, bool indexUnchanged, IndexInfo *indexInfo) +{ + MemoryContext ctx = AllocSetContextCreate(CurrentMemoryContext, "Ann insert temporary context", ALLOCSET_DEFAULT_SIZES); + MemoryContext oldCtx = MemoryContextSwitchTo(ctx); + + AnnInsertTuple(index, values, isnull, tid, false); + MemoryContextSwitchTo(oldCtx); + MemoryContextDelete(ctx); + return false; +} diff --git a/src/annplan.c b/src/annplan.c new file mode 100644 index 000000000..b0ed7cc34 --- /dev/null +++ b/src/annplan.c @@ -0,0 +1,168 @@ +#include "postgres.h" + +#include "access/gin.h" +#include "ann.h" +#include "catalog/pg_am_d.h" +#include "catalog/pg_class.h" +#include "catalog/pg_operator_d.h" +#include "catalog/pg_proc.h" +#include "commands/defrem.h" +#include "nodes/makefuncs.h" +#include "optimizer/optimizer.h" +#include "optimizer/planner.h" +#include "optimizer/tlist.h" +#include "parser/parsetree.h" +#include "utils/guc.h" +#include "utils/lsyscache.h" + +static bool ann_enable_filtered_search; +static create_upper_paths_hook_type previous_upper_paths_hook; + +/* + * A complete PostgreSQL predicate scan supplies visible, qualified rows. Keep + * parameterized, parallel, and accelerator index scans out of this first path. + */ +static bool +IsPredicatePath(Path *path) +{ + if (path->param_info || path->parallel_aware) + return false; + if (IsA(path, ProjectionPath)) + return IsPredicatePath(((ProjectionPath *) path)->subpath); + if (IsA(path, IndexPath)) + return ((IndexPath *) path)->indexinfo->relam == BTREE_AM_OID && + path->pathtype != T_IndexOnlyScan && + ((IndexPath *) path)->indexorderbys == NIL; + return path->pathtype == T_SeqScan || IsA(path, BitmapHeapPath); +} + +static bool +IsLimitExpression(Node *node) +{ + return node == NULL || IsA(node, Const) || + (IsA(node, Param) && ((Param *) node)->paramkind == PARAM_EXTERN); +} + +/* + * Match the actual support function and indexed column, not search-path names. + * Unsupported shapes retain their original PostgreSQL plans. + */ +static OpExpr * +GetDistanceExpression(PlannerInfo *root, Oid amoid) +{ + Query *query = root->parse; + RangeTblRef *ref; + RangeTblEntry *rte; + RelOptInfo *rel; + SortGroupClause *sort; + Node *expression; + OpExpr *distance; + Var *column; + Node *argument; + ListCell *lc; + + if (root->parent_root || query->commandType != CMD_SELECT || + query->sortClause == NIL || query->rowMarks || query->hasAggs || + query->hasWindowFuncs || query->hasTargetSRFs || query->hasSubLinks || + query->groupClause || query->groupingSets || query->havingQual || + query->distinctClause || query->setOperations || query->cteList || + list_length(query->jointree->fromlist) != 1 || + !IsLimitExpression(query->limitCount) || !IsLimitExpression(query->limitOffset) || + contain_volatile_functions((Node *) query->targetList) || + contain_volatile_functions(query->jointree->quals)) + return NULL; + ref = linitial(query->jointree->fromlist); + if (!IsA(ref, RangeTblRef)) + return NULL; + rte = planner_rt_fetch(ref->rtindex, root); + rel = root->simple_rel_array[ref->rtindex]; + if (!rel || rte->rtekind != RTE_RELATION || rte->relkind != RELKIND_RELATION || + rel->baserestrictinfo == NIL) + return NULL; + sort = linitial_node(SortGroupClause, query->sortClause); + expression = (Node *) get_sortgroupclause_expr(sort, query->targetList); + if (!IsA(expression, OpExpr) || sort->nulls_first || sort->sortop != Float8LessOperator) + return NULL; + distance = (OpExpr *) expression; + if (list_length(distance->args) != 2) + return NULL; + column = linitial(distance->args); + argument = lsecond(distance->args); + if (!IsA(column, Var) || column->varno != ref->rtindex || column->varlevelsup || + column->varattno <= 0 || !(IsA(argument, Const) || + (IsA(argument, Param) && ((Param *) argument)->paramkind == PARAM_EXTERN))) + return NULL; + foreach(lc, rel->indexlist) + { + IndexOptInfo *index = lfirst(lc); + Oid proc; + + if (index->relam != amoid || index->nkeycolumns != 1 || + index->indexkeys[0] != column->varattno || index->indpred || index->indexprs) + continue; + proc = get_opfamily_proc(index->opfamily[0], index->opcintype[0], + index->opcintype[0], ANN_DISTANCE_PROC); + if (OidIsValid(proc) && proc == get_opcode(distance->opno) && + func_strict(proc) && func_volatile(proc) == PROVOLATILE_IMMUTABLE) + return distance; + } + return NULL; +} + +static void +AnnCreateUpperPaths(PlannerInfo *root, UpperRelationKind stage, + RelOptInfo *input_rel, RelOptInfo *output_rel, void *extra) +{ + Oid amoid; + OpExpr *distance; + ListCell *lc; + + if (previous_upper_paths_hook) + previous_upper_paths_hook(root, stage, input_rel, output_rel, extra); + if (!ann_enable_filtered_search || stage != UPPERREL_ORDERED || GinFuzzySearchLimit != 0) + return; + amoid = get_am_oid("ann", true); + if (!OidIsValid(amoid) || !(distance = GetDistanceExpression(root, amoid))) + return; + + /* Replace equivalent full sorts, preserving PostgreSQL's path costs. */ + foreach(lc, output_rel->pathlist) + { + Path *path = lfirst(lc); + SortPath *sort; + CustomPath *custom; + + if (!IsA(path, SortPath)) + continue; + sort = (SortPath *) path; + if (!IsPredicatePath(sort->subpath)) + continue; + custom = makeNode(CustomPath); + custom->path = *path; + NodeSetTag(custom, T_CustomPath); + custom->path.pathtype = T_CustomScan; + custom->path.parallel_safe = false; + custom->flags = CUSTOMPATH_SUPPORT_PROJECTION; + custom->custom_paths = list_make1(sort->subpath); + custom->custom_private = list_make5(copyObject(root->parse->sortClause), + copyObject(distance), copyObject(root->parse->limitCount), + copyObject(root->parse->limitOffset), + makeInteger(root->parse->limitOption == LIMIT_OPTION_WITH_TIES)); + custom->methods = &AnnCustomPathMethods; + lfirst(lc) = custom; + } +} + +void +AnnInit(void) +{ + DefineCustomBoolVariable("ann.enable_filtered_search", + "Enable the experimental exact filtered-search path.", + NULL, &ann_enable_filtered_search, false, PGC_USERSET, 0, + NULL, NULL, NULL); + AnnWorkerInit(); + AnnRemoteInit(); + RegisterCustomScanMethods(&AnnCustomScanMethods); + previous_upper_paths_hook = create_upper_paths_hook; + create_upper_paths_hook = AnnCreateUpperPaths; +} diff --git a/src/annremote.c b/src/annremote.c new file mode 100644 index 000000000..ec3521938 --- /dev/null +++ b/src/annremote.c @@ -0,0 +1,482 @@ +#include "postgres.h" + +#include "ann.h" +#include "miscadmin.h" +#include "portability/instr_time.h" +#include "utils/guc.h" +#include "utils/memutils.h" + +char *ann_worker_address = NULL; +char *ann_worker_auth_token = NULL; +int ann_gpu_min_eligible = 1000; +int ann_worker_heartbeat_ms = 5000; +int ann_worker_retry_ms = 10000; + +#if defined(WIN32) || defined(WORDS_BIGENDIAN) + +bool +AnnRemoteConfigured(void) +{ + return false; +} + +bool +AnnRemoteCall(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength) +{ + return false; +} + +int +AnnRemoteBroadcast(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength) +{ + return -1; +} + +#else + +#include <errno.h> +#include <fcntl.h> +#include <netdb.h> +#include <netinet/in.h> +#include <netinet/tcp.h> +#include <poll.h> +#include <sys/socket.h> +#include <unistd.h> + +#define ANN_HEADER_SIZE 32 +#define ANN_OP_PROBE 5 +#define ANN_OP_AUTH 13 +#define ANN_MAX_NODES 64 + +/* + * One connection per backend to one GPU node, chosen from ann.worker_address + * by process ID so backends spread across nodes. A failed node is skipped + * and the next one tried. Requests are synchronous: one outstanding at most. + */ +static int remoteFd = -1; +static int remoteNode = -1; +static uint64 remoteRequest = 1; +static bool remoteBusy = false; /* an error interrupted an exchange */ +static instr_time remoteUsed; /* last successful exchange */ +static instr_time downSince[ANN_MAX_NODES]; /* zero when believed healthy */ + +static void +CloseRemote(void) +{ + if (remoteFd >= 0) + close(remoteFd); + remoteFd = -1; +} + +static double +ElapsedMs(instr_time since) +{ + instr_time now; + + INSTR_TIME_SET_CURRENT(now); + INSTR_TIME_SUBTRACT(now, since); + return INSTR_TIME_GET_MILLISEC(now); +} + +/* A node that failed is skipped until ann.worker_retry_ms has passed. */ +static void +MarkDown(int node) +{ + INSTR_TIME_SET_CURRENT(downSince[node]); +} + +static bool +NodeAvailable(int node) +{ + if (INSTR_TIME_IS_ZERO(downSince[node])) + return true; + if (ElapsedMs(downSince[node]) < ann_worker_retry_ms) + return false; + INSTR_TIME_SET_ZERO(downSince[node]); + return true; +} + +/* + * Kernel keepalives detect a node that vanished mid-request (host down or + * partitioned), which a blocked receive would otherwise only notice at the + * request timeout. + */ +static void +SetKeepalive(int fd) +{ + int on = 1; + int idle = Max(1, ann_worker_heartbeat_ms / 1000); + int interval = Max(1, idle / 2); + int count = 3; + + (void) setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &on, sizeof(on)); +#ifdef TCP_KEEPIDLE + (void) setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &idle, sizeof(idle)); +#elif defined(TCP_KEEPALIVE) + (void) setsockopt(fd, IPPROTO_TCP, TCP_KEEPALIVE, &idle, sizeof(idle)); +#endif +#ifdef TCP_KEEPINTVL + (void) setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &interval, sizeof(interval)); +#endif +#ifdef TCP_KEEPCNT + (void) setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT, &count, sizeof(count)); +#endif +} + +bool +AnnRemoteConfigured(void) +{ + return ann_worker_address != NULL && ann_worker_address[0] != '\0'; +} + +/* Split ann.worker_address into host and port strings; returns the count. */ +static int +ParseNodes(char **hosts, char **ports) +{ + char *list = pstrdup(ann_worker_address); + char *token; + char *save = NULL; + int count = 0; + + for (token = strtok_r(list, ", ", &save); token && count < ANN_MAX_NODES; + token = strtok_r(NULL, ", ", &save)) + { + char *colon = strrchr(token, ':'); + + if (colon == NULL || colon == token) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("invalid ann.worker_address entry \"%s\"", token), + errhint("Use a comma-separated list of host:port."))); + *colon = '\0'; + hosts[count] = token; + ports[count] = colon + 1; + count++; + } + return count; +} + +static bool +WaitFor(int fd, short events, instr_time start, int timeoutMs) +{ + for (;;) + { + struct pollfd poller = {fd, events, 0}; + instr_time now; + int result; + + CHECK_FOR_INTERRUPTS(); + INSTR_TIME_SET_CURRENT(now); + INSTR_TIME_SUBTRACT(now, start); + if (INSTR_TIME_GET_MILLISEC(now) > timeoutMs) + return false; + result = poll(&poller, 1, 100); + if (result > 0) + return (poller.revents & (POLLERR | POLLNVAL)) == 0 || (poller.revents & events); + if (result < 0 && errno != EINTR) + return false; + } +} + +static bool +SendAll(int fd, const char *data, size_t size, instr_time start, int timeoutMs) +{ + while (size > 0) + { +#ifdef MSG_NOSIGNAL + ssize_t n = send(fd, data, size, MSG_NOSIGNAL); +#else + ssize_t n = send(fd, data, size, 0); +#endif + + if (n > 0) + { + data += n; + size -= n; + } + else if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)) + { + if (!WaitFor(fd, POLLOUT, start, timeoutMs)) + return false; + } + else + return false; + } + return true; +} + +static bool +RecvAll(int fd, char *data, size_t size, instr_time start, int timeoutMs) +{ + while (size > 0) + { + ssize_t n = recv(fd, data, size, 0); + + if (n > 0) + { + data += n; + size -= n; + } + else if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)) + { + if (!WaitFor(fd, POLLIN, start, timeoutMs)) + return false; + } + else + return false; + } + return true; +} + +static void +PutHeader(char *header, uint16 operation, uint64 request, uint32 length) +{ + uint16 version = 1; + uint64 generation = 0; + uint32 status = 0; + + memcpy(header, "PVAN", 4); + memcpy(header + 4, &version, 2); + memcpy(header + 6, &operation, 2); + memcpy(header + 8, &request, 8); + memcpy(header + 16, &generation, 8); + memcpy(header + 24, &length, 4); + memcpy(header + 28, &status, 4); +} + +/* + * Send one frame and read its reply on fd. The reply payload is palloc'd in + * the current context and NUL-terminated. + */ +static bool +Exchange(int fd, uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength) +{ + char header[ANN_HEADER_SIZE]; + uint64 request = remoteRequest++; + uint64 replyRequest; + uint32 size; + instr_time start; + + INSTR_TIME_SET_CURRENT(start); + PutHeader(header, operation, request, (uint32) length); + if (!SendAll(fd, header, sizeof(header), start, timeoutMs) || + (length && !SendAll(fd, payload, length, start, timeoutMs)) || + !RecvAll(fd, header, sizeof(header), start, timeoutMs) || + memcmp(header, "PVAN", 4) != 0) + return false; + memcpy(&replyRequest, header + 8, 8); + memcpy(&size, header + 24, 4); + memcpy(status, header + 28, 4); + if (replyRequest != request || size > 64 * 1024 * 1024) + return false; + *reply = palloc(size + 1); + if (!RecvAll(fd, *reply, size, start, timeoutMs)) + return false; + (*reply)[size] = '\0'; + *replyLength = size; + return true; +} + +/* Connect and authenticate to one node; returns the socket or -1. */ +static int +ConnectNode(const char *host, const char *port, int timeoutMs) +{ + struct addrinfo hints = {0}; + struct addrinfo *addresses; + int fd = -1; + instr_time start; + + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + if (getaddrinfo(host, port, &hints, &addresses) != 0) + return -1; + INSTR_TIME_SET_CURRENT(start); + for (struct addrinfo *a = addresses; a && fd < 0; a = a->ai_next) + { + int on = 1; + int error = 0; + socklen_t errorLength = sizeof(error); + + fd = socket(a->ai_family, SOCK_STREAM, 0); + if (fd < 0) + continue; + if (fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK) != 0 || + (connect(fd, a->ai_addr, a->ai_addrlen) != 0 && + (errno != EINPROGRESS || !WaitFor(fd, POLLOUT, start, timeoutMs) || + getsockopt(fd, SOL_SOCKET, SO_ERROR, &error, &errorLength) != 0 || error != 0))) + { + close(fd); + fd = -1; + continue; + } + (void) setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &on, sizeof(on)); + if (ann_worker_heartbeat_ms > 0) + SetKeepalive(fd); +#ifdef SO_NOSIGPIPE + (void) setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &on, sizeof(on)); +#endif + } + freeaddrinfo(addresses); + if (fd >= 0) + { + const char *token = ann_worker_auth_token ? ann_worker_auth_token : ""; + uint32 status; + char *reply; + uint32 replyLength; + + if (!Exchange(fd, ANN_OP_AUTH, token, strlen(token), timeoutMs, &status, &reply, &replyLength) || + status != 0) + { + elog(DEBUG1, "ann worker %s:%s rejected authentication", host, port); + close(fd); + fd = -1; + } + } + return fd; +} + +/* + * Synchronous request to this backend's node. A transport failure moves to + * the next node and retries once there; requests sent this way are + * idempotent searches and lookups. + */ +bool +AnnRemoteCall(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength) +{ + char *hosts[ANN_MAX_NODES]; + char *ports[ANN_MAX_NODES]; + int count = ParseNodes(hosts, ports); + + if (timeoutMs <= 0) + timeoutMs = ann_worker_timeout_ms; + if (count == 0) + return false; + + /* A cancelled exchange may have left part of a reply unread. */ + if (remoteBusy) + CloseRemote(); + + /* + * Heartbeat: a connection idle longer than ann.worker_heartbeat_ms is + * probed first, so a node that died while idle costs one short probe + * rather than a full request timeout. + */ + if (remoteFd >= 0 && ann_worker_heartbeat_ms > 0 && + ElapsedMs(remoteUsed) > ann_worker_heartbeat_ms) + { + uint32 probeStatus; + char *probeReply; + uint32 probeLength; + + remoteBusy = true; + if (!Exchange(remoteFd, ANN_OP_PROBE, NULL, 0, ann_worker_heartbeat_ms, + &probeStatus, &probeReply, &probeLength)) + { + elog(DEBUG1, "ann worker %s:%s missed a heartbeat", hosts[remoteNode], ports[remoteNode]); + MarkDown(remoteNode); + CloseRemote(); + } + remoteBusy = false; + } + for (int attempt = 0; attempt < count + 1; attempt++) + { + if (remoteFd < 0) + { + /* + * Prefer healthy nodes; try cooling-down ones only as a last + * resort. + */ + for (int pass = 0; pass < 2 && remoteFd < 0; pass++) + for (int i = 0; i < count && remoteFd < 0; i++) + { + int node = (remoteNode < 0 ? MyProcPid + i : remoteNode + 1 + i) % count; + + if (pass == 0 && !NodeAvailable(node)) + continue; + remoteFd = ConnectNode(hosts[node], ports[node], Min(timeoutMs, Max(ann_worker_heartbeat_ms, 1000))); + if (remoteFd >= 0) + remoteNode = node; + else + MarkDown(node); + } + if (remoteFd < 0) + return false; + } + remoteBusy = true; + if (Exchange(remoteFd, operation, payload, length, timeoutMs, status, reply, replyLength)) + { + remoteBusy = false; + INSTR_TIME_SET_CURRENT(remoteUsed); + return true; + } + remoteBusy = false; + elog(DEBUG1, "ann worker %s:%s failed; trying the next node", hosts[remoteNode], ports[remoteNode]); + MarkDown(remoteNode); + CloseRemote(); + } + return false; +} + +/* + * Send one request to every node, for loads. Returns the number of nodes on + * success, or -1 after the first failure; *status holds that node's status. + */ +int +AnnRemoteBroadcast(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength) +{ + char *hosts[ANN_MAX_NODES]; + char *ports[ANN_MAX_NODES]; + int count = ParseNodes(hosts, ports); + + for (int i = 0; i < count; i++) + { + int fd = ConnectNode(hosts[i], ports[i], timeoutMs); + bool ok; + + if (fd < 0) + { + *status = UINT32_MAX; + return -1; + } + ok = Exchange(fd, operation, payload, length, timeoutMs, status, reply, replyLength); + close(fd); + if (!ok) + *status = UINT32_MAX; + if (!ok || *status != 0) + return -1; + } + return count; +} + +#endif + +void +AnnRemoteInit(void) +{ + DefineCustomStringVariable("ann.worker_address", + "Comma-separated host:port list of networked ann GPU workers.", + "When set, loaded-index functions use these nodes instead of ann.worker_socket.", + &ann_worker_address, "", PGC_SUSET, 0, NULL, NULL, NULL); + DefineCustomStringVariable("ann.worker_auth_token", + "Shared secret sent to networked ann workers.", + NULL, &ann_worker_auth_token, "", PGC_SUSET, + GUC_SUPERUSER_ONLY | GUC_NO_SHOW_ALL, NULL, NULL, NULL); + DefineCustomIntVariable("ann.worker_heartbeat_ms", + "Idle time after which a GPU node connection is probed before reuse.", + "Also sets TCP keepalive timing; 0 disables heartbeats.", + &ann_worker_heartbeat_ms, 5000, 0, INT_MAX, PGC_USERSET, GUC_UNIT_MS, + NULL, NULL, NULL); + DefineCustomIntVariable("ann.worker_retry_ms", + "Time a failed GPU node is skipped before being tried again.", + NULL, &ann_worker_retry_ms, 10000, 0, INT_MAX, PGC_USERSET, GUC_UNIT_MS, + NULL, NULL, NULL); + DefineCustomIntVariable("ann.gpu_min_eligible", + "Filters with fewer eligible rows are scored exactly on this host.", + "Filtered loaded-index searches below this count never leave the host.", + &ann_gpu_min_eligible, 1000, 0, INT_MAX, PGC_USERSET, 0, NULL, NULL, NULL); +} diff --git a/src/annscan.c b/src/annscan.c new file mode 100644 index 000000000..c014bb574 --- /dev/null +++ b/src/annscan.c @@ -0,0 +1,171 @@ +#include "postgres.h" + +#include "ann.h" +#include "catalog/pg_operator_d.h" +#include "catalog/pg_type_d.h" +#include "executor/tuptable.h" +#include "miscadmin.h" +#include "pgstat.h" +#include "utils/memutils.h" +#include "utils/snapmgr.h" +#include "utils/tuplesort.h" + +typedef struct AnnScanOpaqueData +{ + bool first; + FmgrInfo *procinfo; + TupleDesc tupdesc; + TupleTableSlot *input; + TupleTableSlot *output; + Tuplesortstate *sortstate; + MemoryContext tmpCtx; +} AnnScanOpaqueData; + +typedef AnnScanOpaqueData *AnnScanOpaque; + +static void +GetScanItems(IndexScanDesc scan) +{ + AnnScanOpaque so = scan->opaque; + Relation index = scan->indexRelation; + BlockNumber blocks; + Buffer buf; + Datum query = scan->orderByData[0].sk_argument; + bool queryIsNull = (scan->orderByData[0].sk_flags & SK_ISNULL) != 0; + + buf = ReadBuffer(index, ANN_METAPAGE_BLKNO); + LockBuffer(buf, BUFFER_LOCK_SHARE); + AnnCheckMetaPage(BufferGetPage(buf)); + UnlockReleaseBuffer(buf); + blocks = RelationGetNumberOfBlocks(index); + for (BlockNumber blkno = 1; blkno < blocks; blkno++) + { + Page page; + OffsetNumber maxoff; + + CHECK_FOR_INTERRUPTS(); + buf = ReadBuffer(index, blkno); + LockBuffer(buf, BUFFER_LOCK_SHARE); + page = BufferGetPage(buf); + /* An aborted extension can leave an unpublished, empty physical page. */ + if (PageIsNew(page)) + { + UnlockReleaseBuffer(buf); + continue; + } + AnnCheckPage(page); + maxoff = PageGetMaxOffsetNumber(page); + for (OffsetNumber offno = FirstOffsetNumber; offno <= maxoff; offno++) + { + ItemId itemid = PageGetItemId(page, offno); + IndexTuple tuple; + bool isnull; + Datum value; + MemoryContext oldCtx; + + if (!ItemIdIsNormal(itemid)) + continue; + tuple = (IndexTuple) PageGetItem(page, itemid); + oldCtx = MemoryContextSwitchTo(so->tmpCtx); + value = index_getattr(tuple, 1, RelationGetDescr(index), &isnull); + ExecClearTuple(so->input); + so->input->tts_isnull[0] = isnull || queryIsNull; + so->input->tts_values[0] = (isnull || queryIsNull) ? (Datum) 0 : + FunctionCall2Coll(so->procinfo, index->rd_indcollation[0], + PointerGetDatum(DatumGetVector(value)), query); + so->input->tts_isnull[1] = false; + so->input->tts_values[1] = PointerGetDatum(&tuple->t_tid); + ExecStoreVirtualTuple(so->input); + MemoryContextSwitchTo(oldCtx); + tuplesort_puttupleslot(so->sortstate, so->input); + ExecClearTuple(so->input); + MemoryContextReset(so->tmpCtx); + } + UnlockReleaseBuffer(buf); + } + tuplesort_performsort(so->sortstate); +} + +IndexScanDesc +annbeginscan(Relation index, int nkeys, int norderbys) +{ + IndexScanDesc scan = RelationGetIndexScan(index, nkeys, norderbys); + AnnScanOpaque so = palloc0_object(AnnScanOpaqueData); + AttrNumber attNums[] = {1}; + Oid operators[] = {Float8LessOperator}; + Oid collations[] = {InvalidOid}; + bool nullsFirst[] = {false}; + + so->first = true; + so->procinfo = index_getprocinfo(index, 1, ANN_DISTANCE_PROC); + so->tmpCtx = AllocSetContextCreate(CurrentMemoryContext, "Ann scan temporary context", ALLOCSET_DEFAULT_SIZES); + so->tupdesc = CreateTemplateTupleDesc(2); + TupleDescInitEntry(so->tupdesc, 1, "distance", FLOAT8OID, -1, 0); + TupleDescInitEntry(so->tupdesc, 2, "heaptid", TIDOID, -1, 0); +#if PG_VERSION_NUM >= 190000 + TupleDescFinalize(so->tupdesc); +#endif + so->input = MakeSingleTupleTableSlot(so->tupdesc, &TTSOpsVirtual); + so->output = MakeSingleTupleTableSlot(so->tupdesc, &TTSOpsMinimalTuple); + so->sortstate = tuplesort_begin_heap(so->tupdesc, 1, attNums, operators, collations, + nullsFirst, work_mem, NULL, false); + scan->opaque = so; + return scan; +} + +void +annrescan(IndexScanDesc scan, ScanKey keys, int nkeys, ScanKey orderbys, int norderbys) +{ + AnnScanOpaque so = scan->opaque; + + so->first = true; + ExecClearTuple(so->output); + tuplesort_reset(so->sortstate); + MemoryContextReset(so->tmpCtx); + if (keys && scan->numberOfKeys > 0) + memmove(scan->keyData, keys, scan->numberOfKeys * sizeof(ScanKeyData)); + if (orderbys && scan->numberOfOrderBys > 0) + memmove(scan->orderByData, orderbys, scan->numberOfOrderBys * sizeof(ScanKeyData)); +} + +bool +anngettuple(IndexScanDesc scan, ScanDirection dir) +{ + AnnScanOpaque so = scan->opaque; + Datum value; + bool isnull; + + Assert(ScanDirectionIsForward(dir)); + if (so->first) + { + if (scan->numberOfOrderBys != 1 || !scan->orderByData) + elog(ERROR, "cannot scan ann index without one distance order"); + if (!IsMVCCSnapshot(scan->xs_snapshot)) + elog(ERROR, "non-MVCC snapshots are not supported with ann"); + pgstat_count_index_scan(scan->indexRelation); + GetScanItems(scan); + so->first = false; + } + if (!tuplesort_gettupleslot(so->sortstate, true, false, so->output, NULL)) + return false; + value = slot_getattr(so->output, 2, &isnull); + Assert(!isnull); + scan->xs_heaptid = *DatumGetItemPointer(value); + scan->xs_recheck = false; + scan->xs_recheckorderby = false; + return true; +} + +void +annendscan(IndexScanDesc scan) +{ + AnnScanOpaque so = scan->opaque; + + ExecDropSingleTupleTableSlot(so->input); + ExecDropSingleTupleTableSlot(so->output); + tuplesort_end(so->sortstate); + FreeTupleDesc(so->tupdesc); + MemoryContextDelete(so->tmpCtx); + pfree(so); + scan->opaque = NULL; +} diff --git a/src/annutils.c b/src/annutils.c new file mode 100644 index 000000000..5cf274c09 --- /dev/null +++ b/src/annutils.c @@ -0,0 +1,79 @@ +#include "postgres.h" + +#include "access/generic_xlog.h" +#include "ann.h" +#include "catalog/pg_type.h" +#include "commands/extension.h" +#include "utils/lsyscache.h" +#include "utils/syscache.h" + +/* + * Resolve the extension type in its actual schema, not the session search path. + */ +Oid +AnnVectorTypeOid(void) +{ + Oid namespace = get_extension_schema(get_extension_oid("vector", false)); + + return GetSysCacheOid2(TYPENAMENSP, Anum_pg_type_oid, + CStringGetDatum("vector"), ObjectIdGetDatum(namespace)); +} + +void +AnnCheckIndex(Relation index) +{ + Form_pg_attribute attr = TupleDescAttr(RelationGetDescr(index), 0); + + if (attr->atttypid != AnnVectorTypeOid()) + ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("ann indexes require vector values"))); + if (attr->atttypmod < 1) + ereport(ERROR, (errcode(ERRCODE_INVALID_PARAMETER_VALUE), errmsg("column does not have dimensions"))); + if (attr->atttypmod > ANN_MAX_DIM) + ereport(ERROR, (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("column cannot have more than %d dimensions for ann index", ANN_MAX_DIM))); +} + +void +AnnCheckPage(Page page) +{ + if (PageIsNew(page) || PageGetSpecialSize(page) != MAXALIGN(sizeof(AnnPageOpaqueData)) || + AnnPageGetOpaque(page)->page_id != ANN_PAGE_ID) + ereport(ERROR, (errcode(ERRCODE_INDEX_CORRUPTED), errmsg("invalid ann index page"))); +} + +void +AnnCheckMetaPage(Page page) +{ + AnnMetaPageData *meta; + + AnnCheckPage(page); + meta = AnnPageGetMeta(page); + if (meta->magicNumber != ANN_MAGIC_NUMBER || meta->version != ANN_VERSION || + meta->dimensions < 1 || meta->dimensions > ANN_MAX_DIM) + ereport(ERROR, (errcode(ERRCODE_INDEX_CORRUPTED), errmsg("invalid ann index metapage"))); +} + +void +AnnInitMetaPage(Relation index, ForkNumber forkNum) +{ + Buffer buf; + Page page; + AnnMetaPageData *meta; + + AnnCheckIndex(index); + buf = ReadBufferExtended(index, forkNum, P_NEW, RBM_NORMAL, NULL); + LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); + if (BufferGetBlockNumber(buf) != ANN_METAPAGE_BLKNO) + elog(ERROR, "ann index is not empty"); + page = BufferGetPage(buf); + PageInit(page, BufferGetPageSize(buf), sizeof(AnnPageOpaqueData)); + AnnPageGetOpaque(page)->page_id = ANN_PAGE_ID; + meta = AnnPageGetMeta(page); + meta->magicNumber = ANN_MAGIC_NUMBER; + meta->version = ANN_VERSION; + meta->dimensions = TupleDescAttr(RelationGetDescr(index), 0)->atttypmod; + meta->insertPage = InvalidBlockNumber; + ((PageHeader) page)->pd_lower = SizeOfPageHeaderData + sizeof(*meta); + MarkBufferDirty(buf); + UnlockReleaseBuffer(buf); +} diff --git a/src/annvacuum.c b/src/annvacuum.c new file mode 100644 index 000000000..399932745 --- /dev/null +++ b/src/annvacuum.c @@ -0,0 +1,85 @@ +#include "postgres.h" + +#include "access/generic_xlog.h" +#include "ann.h" +#include "commands/vacuum.h" +#include "storage/freespace.h" + +#if PG_VERSION_NUM >= 180000 +#define vacuum_delay_point() vacuum_delay_point(false) +#endif + +IndexBulkDeleteResult * +annbulkdelete(IndexVacuumInfo *info, IndexBulkDeleteResult *stats, + IndexBulkDeleteCallback callback, void *callback_state) +{ + Relation index = info->index; + BlockNumber blocks = RelationGetNumberOfBlocks(index); + + if (!stats) + stats = palloc0_object(IndexBulkDeleteResult); + /* A later bulk-delete cycle rescans survivors; do not count them twice. */ + stats->num_index_tuples = 0; + for (BlockNumber blkno = 1; blkno < blocks; blkno++) + { + Buffer buf; + Page page; + GenericXLogState *state; + OffsetNumber deletable[MaxOffsetNumber]; + OffsetNumber maxoff; + int count = 0; + Size freeSpace; + + vacuum_delay_point(); + buf = ReadBuffer(index, blkno); + LockBufferForCleanup(buf); + page = BufferGetPage(buf); + if (PageIsNew(page)) + { + UnlockReleaseBuffer(buf); + RecordPageWithFreeSpace(index, blkno, BLCKSZ - SizeOfPageHeaderData - + MAXALIGN(sizeof(AnnPageOpaqueData)) - sizeof(ItemIdData)); + continue; + } + AnnCheckPage(page); + state = GenericXLogStart(index); + page = GenericXLogRegisterBuffer(state, buf, 0); + maxoff = PageGetMaxOffsetNumber(page); + for (OffsetNumber offno = FirstOffsetNumber; offno <= maxoff; offno++) + { + IndexTuple tuple = (IndexTuple) PageGetItem(page, PageGetItemId(page, offno)); + + if (callback(&tuple->t_tid, callback_state)) + { + deletable[count++] = offno; + stats->tuples_removed++; + } + else + stats->num_index_tuples++; + } + if (count) + { + PageIndexMultiDelete(page, deletable, count); + freeSpace = PageGetFreeSpace(page); + GenericXLogFinish(state); + } + else + { + freeSpace = PageGetFreeSpace(page); + GenericXLogAbort(state); + } + UnlockReleaseBuffer(buf); + RecordPageWithFreeSpace(index, blkno, freeSpace); + } + FreeSpaceMapVacuum(index); + return stats; +} + +IndexBulkDeleteResult * +annvacuumcleanup(IndexVacuumInfo *info, IndexBulkDeleteResult *stats) +{ + if (info->analyze_only || !stats) + return stats; + stats->num_pages = RelationGetNumberOfBlocks(info->index); + return stats; +} diff --git a/src/annworker.c b/src/annworker.c new file mode 100644 index 000000000..f4c6feb49 --- /dev/null +++ b/src/annworker.c @@ -0,0 +1,569 @@ +#include "postgres.h" + +#include "ann.h" +#include "miscadmin.h" +#include "portability/instr_time.h" +#include "utils/guc.h" +#include "utils/memutils.h" + +char *ann_worker_socket = NULL; +int ann_worker_chunk_rows = 4096; +int ann_worker_pipeline_depth = 4; +int ann_worker_timeout_ms = 30000; +int ann_worker_shared_memory_mb = 64; + +#if defined(WIN32) || defined(WORDS_BIGENDIAN) + +/* The worker route is POSIX and little-endian only; scans stay local. */ +bool +AnnWorkerBegin(void) +{ + return false; +} + +void +AnnWorkerEnd(void) +{ +} + +float * +AnnWorkerSharedBuffer(size_t *size) +{ + *size = 0; + return NULL; +} + +uint64 +AnnWorkerSubmit(uint32 rows, uint32 dimensions, uint32 k, const float *query, + const float *vectors) +{ + return 0; +} + +bool +AnnWorkerReceive(uint64 *request, uint32 *status, uint32 **ids, uint32 *count) +{ + return false; +} + +bool +AnnWorkerReady(void) +{ + return false; +} + +bool +AnnWorkerCall(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength) +{ + return false; +} + +#else + +#include <errno.h> +#include <fcntl.h> +#include <poll.h> +#include <sys/mman.h> +#include <sys/socket.h> +#include <sys/stat.h> +#include <sys/un.h> +#include <unistd.h> + +#define ANN_HEADER_SIZE 32 +#define ANN_OP_GATHERED_SEARCH 7 +#define ANN_OP_ATTACH_SHARED 9 +#define ANN_OP_SHARED_SEARCH 10 + +/* + * One connection per backend, reused across scans. Replies may arrive in any + * order. A scan that errors out leaves replies outstanding; the next scan + * reconnects rather than reading them. + */ +static int workerFd = -1; +static uint32 outstanding = 0; +static uint64 nextRequest = 1; +static char *recvBuffer = NULL; +static size_t recvLength = 0; +static size_t recvCapacity = 0; + +/* + * Chunk vectors are written directly into a segment the worker maps + * read-only, so requests carry offsets instead of vector bytes. A failed + * connection keeps the mapping until no scan can still be reading it. + */ +static char *shared = NULL; +static size_t sharedSize = 0; +static bool sharedDetached = false; +static bool scanActive = false; +static int waitTimeoutMs = 30000; + +static void +Disconnect(void) +{ + if (workerFd >= 0) + close(workerFd); + workerFd = -1; + outstanding = 0; + recvLength = 0; + if (shared) + sharedDetached = true; +} + +static void +Unmap(void) +{ + if (shared) + munmap(shared, sharedSize); + shared = NULL; + sharedSize = 0; + sharedDetached = false; +} + +static bool Send(const char *data, size_t size); +static void PutHeader(char *header, uint16 operation, uint64 request, uint32 length); +static bool ReceiveFrame(uint64 *request, uint32 *status, const char **payload, uint32 *length); +static void ConsumeFrame(uint32 length); + +/* Create, attach, and immediately unlink a segment; failure keeps inline mode. */ +static void +AttachShared(void) +{ + static uint32 counter = 0; + char name[32]; + char header[ANN_HEADER_SIZE]; + size_t size = (size_t) ann_worker_shared_memory_mb * 1024 * 1024; + uint64 request = nextRequest++; + uint64 replyRequest; + uint32 status; + uint32 length; + const char *payload; + int fd; + + if (size == 0) + return; + snprintf(name, sizeof(name), "/pgann.%d.%u", MyProcPid, counter++); + fd = shm_open(name, O_CREAT | O_EXCL | O_RDWR, 0600); + if (fd < 0) + return; + if (ftruncate(fd, size) != 0 || + (shared = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0)) == MAP_FAILED) + { + shared = NULL; + close(fd); + shm_unlink(name); + return; + } + close(fd); + sharedSize = size; + PutHeader(header, ANN_OP_ATTACH_SHARED, request, strlen(name)); + if (Send(header, sizeof(header)) && Send(name, strlen(name))) + { + outstanding++; + if (ReceiveFrame(&replyRequest, &status, &payload, &length)) + { + ConsumeFrame(length); + if (replyRequest == request && status == 0) + { + shm_unlink(name); + return; + } + } + } + shm_unlink(name); + Unmap(); +} + +static bool +Connect(void) +{ + struct sockaddr_un address = {0}; + + waitTimeoutMs = ann_worker_timeout_ms; + if (ann_worker_socket == NULL || ann_worker_socket[0] == '\0') + return false; + if (outstanding > 0) + Disconnect(); + if (sharedDetached) + Unmap(); + if (workerFd >= 0) + return true; + if (strlen(ann_worker_socket) >= sizeof(address.sun_path)) + return false; + address.sun_family = AF_UNIX; + strcpy(address.sun_path, ann_worker_socket); + workerFd = socket(AF_UNIX, SOCK_STREAM, 0); + if (workerFd < 0) + return false; + if (connect(workerFd, (struct sockaddr *) &address, sizeof(address)) != 0 || + fcntl(workerFd, F_SETFL, fcntl(workerFd, F_GETFL) | O_NONBLOCK) != 0) + { + elog(DEBUG1, "ann worker unavailable at \"%s\": %m", ann_worker_socket); + Disconnect(); + return false; + } + { + /* + * Chunks are megabytes; small default Unix socket buffers cost + * syscalls. + */ + int buffer = 4 * 1024 * 1024; + + (void) setsockopt(workerFd, SOL_SOCKET, SO_SNDBUF, &buffer, sizeof(buffer)); + (void) setsockopt(workerFd, SOL_SOCKET, SO_RCVBUF, &buffer, sizeof(buffer)); + } +#ifdef SO_NOSIGPIPE + { + int on = 1; + + (void) setsockopt(workerFd, SOL_SOCKET, SO_NOSIGPIPE, &on, sizeof(on)); + } +#endif + if (recvBuffer == NULL) + { + recvCapacity = 64 * 1024; + recvBuffer = MemoryContextAlloc(TopMemoryContext, recvCapacity); + } + AttachShared(); + return workerFd >= 0; +} + +bool +AnnWorkerBegin(void) +{ + if (scanActive || !Connect()) + return false; + scanActive = true; + return true; +} + +/* Callers pair this with a successful AnnWorkerBegin, including on error. */ +void +AnnWorkerEnd(void) +{ + scanActive = false; +} + +float * +AnnWorkerSharedBuffer(size_t *size) +{ + *size = workerFd >= 0 && !sharedDetached ? sharedSize : 0; + return *size ? (float *) shared : NULL; +} + +/* Read whatever is available without blocking; false on EOF or error. */ +static bool +ReadAvailable(void) +{ + for (;;) + { + ssize_t n; + + if (recvLength == recvCapacity) + { + recvCapacity *= 2; + recvBuffer = repalloc(recvBuffer, recvCapacity); + } + n = recv(workerFd, recvBuffer + recvLength, recvCapacity - recvLength, 0); + if (n > 0) + { + recvLength += n; + continue; + } + if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)) + return true; + return false; + } +} + +/* + * Wait for the socket while servicing interrupts. Reading while blocked on + * writes keeps a pipelined worker from stalling on a full reply buffer. + */ +static bool +Wait(short events, instr_time start) +{ + struct pollfd poller = {workerFd, events | POLLIN, 0}; + instr_time now; + int result; + + CHECK_FOR_INTERRUPTS(); + INSTR_TIME_SET_CURRENT(now); + INSTR_TIME_SUBTRACT(now, start); + if (INSTR_TIME_GET_MILLISEC(now) > waitTimeoutMs) + { + elog(DEBUG1, "ann worker timed out"); + return false; + } + result = poll(&poller, 1, 100); + if (result < 0 && errno != EINTR) + return false; + if (result > 0 && (poller.revents & (POLLIN | POLLHUP | POLLERR))) + return ReadAvailable(); + return true; +} + +static bool +Send(const char *data, size_t size) +{ + instr_time start; + + INSTR_TIME_SET_CURRENT(start); + while (size > 0) + { +#ifdef MSG_NOSIGNAL + ssize_t n = send(workerFd, data, size, MSG_NOSIGNAL); +#else + ssize_t n = send(workerFd, data, size, 0); +#endif + + if (n > 0) + { + data += n; + size -= n; + continue; + } + if (n < 0 && errno != EAGAIN && errno != EWOULDBLOCK && errno != EINTR) + return false; + if (!Wait(POLLOUT, start)) + return false; + } + return true; +} + +static void +Put32(char *data, uint32 value) +{ + memcpy(data, &value, sizeof(value)); +} + +static uint32 +Get32(const char *data) +{ + uint32 value; + + memcpy(&value, data, sizeof(value)); + return value; +} + +static uint64 +Get64(const char *data) +{ + uint64 value; + + memcpy(&value, data, sizeof(value)); + return value; +} + +static void +PutHeader(char *header, uint16 operation, uint64 request, uint32 length) +{ + uint16 version = 1; + uint64 generation = 0; + + memcpy(header, "PVAN", 4); + memcpy(header + 4, &version, 2); + memcpy(header + 6, &operation, 2); + memcpy(header + 8, &request, 8); + memcpy(header + 16, &generation, 8); + Put32(header + 24, length); + Put32(header + 28, 0); +} + +/* + * Vectors inside the shared buffer are sent by offset and must stay unchanged + * until the reply. Returns the request ID, or 0 after closing the connection. + */ +uint64 +AnnWorkerSubmit(uint32 rows, uint32 dimensions, uint32 k, const float *query, + const float *vectors) +{ + char header[ANN_HEADER_SIZE + 24]; + uint64 request = nextRequest++; + size_t vectorBytes = (size_t) rows * dimensions * sizeof(float); + bool inShared = shared && !sharedDetached && (const char *) vectors >= shared && + (const char *) vectors + vectorBytes <= shared + sharedSize; + size_t fields = inShared ? 24 : 16; + size_t payload = fields + dimensions * sizeof(float) + (inShared ? 0 : vectorBytes); + + if (workerFd < 0 || payload > 64 * 1024 * 1024) + return 0; + PutHeader(header, inShared ? ANN_OP_SHARED_SEARCH : ANN_OP_GATHERED_SEARCH, + request, (uint32) payload); + Put32(header + 32, rows); + Put32(header + 36, dimensions); + Put32(header + 40, k); + Put32(header + 44, 0); /* L2 */ + if (inShared) + { + uint64 offset = (const char *) vectors - shared; + + memcpy(header + 48, &offset, 8); + } + if (!Send(header, ANN_HEADER_SIZE + fields) || + !Send((const char *) query, dimensions * sizeof(float)) || + (!inShared && !Send((const char *) vectors, vectorBytes))) + { + Disconnect(); + return 0; + } + outstanding++; + return request; +} + +/* True when a complete reply is buffered, reading without blocking. */ +bool +AnnWorkerReady(void) +{ + if (workerFd < 0 || outstanding == 0) + return false; + if (!ReadAvailable()) + { + /* A real failure surfaces on the next blocking receive. */ + return true; + } + return recvLength >= ANN_HEADER_SIZE && + recvLength >= ANN_HEADER_SIZE + (size_t) Get32(recvBuffer + 24); +} + +/* Wait for a complete frame at the front of the receive buffer. */ +static bool +ReceiveFrame(uint64 *request, uint32 *status, const char **payload, uint32 *length) +{ + instr_time start; + + if (workerFd < 0 || outstanding == 0) + return false; + INSTR_TIME_SET_CURRENT(start); + for (;;) + { + if (recvLength >= ANN_HEADER_SIZE) + { + *length = Get32(recvBuffer + 24); + if (memcmp(recvBuffer, "PVAN", 4) != 0 || *length > 64 * 1024 * 1024) + { + Disconnect(); + return false; + } + if (recvLength >= ANN_HEADER_SIZE + (size_t) *length) + break; + } + if (!Wait(0, start)) + { + Disconnect(); + return false; + } + } + *request = Get64(recvBuffer + 8); + *status = Get32(recvBuffer + 28); + *payload = recvBuffer + ANN_HEADER_SIZE; + return true; +} + +static void +ConsumeFrame(uint32 length) +{ + size_t total = ANN_HEADER_SIZE + (size_t) length; + + memmove(recvBuffer, recvBuffer + total, recvLength - total); + recvLength -= total; + outstanding--; +} + +/* + * Wait for the next search reply. On success, *ids is palloc'd in the current + * context. Returns false after closing a failed or malformed connection. + */ +bool +AnnWorkerReceive(uint64 *request, uint32 *status, uint32 **ids, uint32 *count) +{ + const char *payload; + uint32 length; + + if (!ReceiveFrame(request, status, &payload, &length)) + return false; + *ids = NULL; + *count = 0; + if (*status == 0) + { + if (length < 4 || (*count = Get32(payload)) > (length - 4) / 12 || + (size_t) *count * 12 + 4 != length) + { + Disconnect(); + return false; + } + *ids = palloc(sizeof(uint32) * Max(*count, 1)); + for (uint32 i = 0; i < *count; i++) + (*ids)[i] = Get32(payload + 4 + (size_t) i * 12); + } + ConsumeFrame(length); + return true; +} + +/* + * One synchronous request outside a scan. On success the reply payload is + * palloc'd and NUL-terminated. Returns false when the worker is unavailable + * or the connection failed; *status carries the worker's status otherwise. + */ +bool +AnnWorkerCall(uint16 operation, const char *payload, size_t length, int timeoutMs, + uint32 *status, char **reply, uint32 *replyLength) +{ + char header[ANN_HEADER_SIZE]; + uint64 request; + uint64 replyRequest; + const char *data; + uint32 dataLength; + + if (scanActive || length > 64 * 1024 * 1024 || !Connect()) + return false; + waitTimeoutMs = timeoutMs > 0 ? timeoutMs : ann_worker_timeout_ms; + request = nextRequest++; + PutHeader(header, operation, request, (uint32) length); + if (!Send(header, sizeof(header)) || (length && !Send(payload, length))) + { + Disconnect(); + return false; + } + outstanding++; + if (!ReceiveFrame(&replyRequest, status, &data, &dataLength) || replyRequest != request) + { + Disconnect(); + return false; + } + *reply = palloc(dataLength + 1); + memcpy(*reply, data, dataLength); + (*reply)[dataLength] = '\0'; + *replyLength = dataLength; + ConsumeFrame(dataLength); + waitTimeoutMs = ann_worker_timeout_ms; + return true; +} + +#endif + +void +AnnWorkerInit(void) +{ + DefineCustomStringVariable("ann.worker_socket", + "Unix socket of the ann worker for filtered search.", + "Empty disables the worker route.", + &ann_worker_socket, "", PGC_SUSET, 0, NULL, NULL, NULL); + DefineCustomIntVariable("ann.worker_chunk_rows", + "Eligible rows sent to the ann worker per request.", + NULL, &ann_worker_chunk_rows, 4096, 64, 1 << 20, + PGC_USERSET, 0, NULL, NULL, NULL); + DefineCustomIntVariable("ann.worker_pipeline_depth", + "Requests a scan keeps in flight to the ann worker.", + NULL, &ann_worker_pipeline_depth, 4, 1, 64, + PGC_USERSET, 0, NULL, NULL, NULL); + DefineCustomIntVariable("ann.worker_shared_memory_mb", + "Shared memory per backend for vectors sent to the ann worker.", + "Zero sends vectors through the socket instead.", + &ann_worker_shared_memory_mb, 64, 0, 4096, + PGC_SUSET, GUC_UNIT_MB, NULL, NULL, NULL); + DefineCustomIntVariable("ann.worker_timeout_ms", + "Time to wait for ann worker progress before scoring locally.", + NULL, &ann_worker_timeout_ms, 30000, 1, INT_MAX, + PGC_USERSET, GUC_UNIT_MS, NULL, NULL, NULL); +} diff --git a/src/vector.c b/src/vector.c index b4b71d0d8..ad441f1f0 100644 --- a/src/vector.c +++ b/src/vector.c @@ -2,6 +2,7 @@ #include <math.h> +#include "ann.h" #include "bitutils.h" #include "bitvec.h" #include "catalog/pg_type.h" @@ -46,7 +47,7 @@ #endif #if PG_VERSION_NUM >= 180000 -PG_MODULE_MAGIC_EXT(.name = "vector", .version = "0.8.7"); +PG_MODULE_MAGIC_EXT(.name = "vector", .version = "0.9.0"); #else PG_MODULE_MAGIC; #endif @@ -58,6 +59,7 @@ PGDLLEXPORT void _PG_init(void); void _PG_init(void) { + AnnInit(); BitvecInit(); HalfvecInit(); HnswInit(); diff --git a/test/ann/batch_test.c b/test/ann/batch_test.c new file mode 100644 index 000000000..9a03c3b06 --- /dev/null +++ b/test/ann/batch_test.c @@ -0,0 +1,364 @@ +#ifndef __APPLE__ +#define _POSIX_C_SOURCE 200809L +#endif +#include "batching.h" + +#include <assert.h> +#include <pthread.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <time.h> + +static void +Sleep(uint64_t milliseconds) +{ + struct timespec ts = {(time_t) (milliseconds / 1000), (long) ((milliseconds % 1000) * 1000000)}; + + (void) nanosleep(&ts, NULL); +} + +static AnnStatus +FailSearch(const AnnIndex *index, const float *query, uint32_t metric, + const uint32_t *ids, uint32_t nids, uint32_t k, + AnnCandidate *results, uint32_t *count) +{ + (void) index; + (void) query; + (void) metric; + (void) ids; + (void) nids; + (void) k; + (void) results; + *count = 0; + return ANN_BACKEND_ERROR; +} + +static AnnStatus +SlowSearch(const AnnIndex *index, const float *query, uint32_t metric, + const uint32_t *ids, uint32_t nids, uint32_t k, + AnnCandidate *results, uint32_t *count) +{ + Sleep(50); + return AnnCpuBackend()->search(index, query, metric, ids, nids, k, results, count); +} + +static AnnBatcher * +Create(AnnBatchOptions *options, int slow) +{ + float vectors[128]; + AnnBatcher *batcher; + static AnnBackend backend; + + for (uint32_t i = 0; i < 64; i++) + { + vectors[i * 2] = (float) i; + vectors[i * 2 + 1] = 1; + } + backend = *AnnCpuBackend(); + if (slow) + backend.search = SlowSearch; + if (slow == 2) + backend.search = FailSearch; + assert(AnnBatchCreate(&backend, 1, 64, 2, vectors, options, &batcher) == ANN_OK); + return batcher; +} + +static AnnRequest * +Submit(AnnBatcher *batcher, uint32_t ordinal, uint64_t timeout) +{ + float query[2] = {(float) ordinal, 1}; + uint32_t ids[2] = {ordinal, ordinal + 1}; + AnnRequest *request; + + assert(AnnBatchSubmit(batcher, query, UINT32_MAX, 0, ids, 2, timeout, &request) == ANN_OK); + /* Inputs can change immediately after submission. */ + query[0] = -100; + ids[0] = 63; + return request; +} + +static void +Result (AnnRequest *request, uint32_t ordinal) +{ + uint32_t count; + const AnnCandidate *results; + + assert(AnnRequestWait(request, 3000000) == ANN_OK); + results = AnnRequestResults(request, &count); + assert(count == 2 && results[0].id == ordinal && results[0].distance == 0); + assert(results[1].id == ordinal + 1 && results[1].distance == 1); +} + +static void +Idle(AnnBatcher *batcher) +{ + AnnBatchStats stats; + + for (int i = 0; i < 3000; i++) + { + AnnBatchGetStats(batcher, &stats); + if (!stats.pending_requests) + break; + Sleep(1); + } + assert(stats.pending_requests == 0 && stats.pending_bytes == 0 && stats.pending_work == 0); +} + +static void +Batch(void) +{ + AnnBatchOptions options; + AnnBatcher *batcher; + AnnRequest *requests[4]; + AnnBatchStats stats; + + AnnBatchDefaultOptions(&options); + options.max_batch_rows = 4; + options.delay_us = 1000000; + batcher = Create(&options, 0); + for (uint32_t i = 0; i < 4; i++) + requests[i] = Submit(batcher, i * 5, 3000000); + for (uint32_t i = 0; i < 4; i++) + Result (requests[i], i * 5); + + Idle(batcher); + AnnBatchGetStats(batcher, &stats); + assert(stats.batches == 1 && stats.max_observed_batch == 4 && stats.completed == 4); + AnnBatchClose(batcher); + /* Result handles outlive the service and its index safely. */ + for (uint32_t i = 0; i < 4; i++) + { + Result (requests[i], i * 5); + + AnnRequestRelease(requests[i]); + } +} + +static void +Admission(void) +{ + AnnBatchOptions options; + AnnBatcher *batcher; + AnnRequest *a; + AnnRequest *b; + float query[2] = {1, 1}; + AnnBatchStats stats; + uint64_t one_request_bytes; + + AnnBatchDefaultOptions(&options); + options.delay_us = 1000000; + batcher = Create(&options, 0); + a = Submit(batcher, 1, 3000000); + AnnBatchGetStats(batcher, &stats); + one_request_bytes = stats.pending_bytes; + assert(AnnRequestCancel(a)); + AnnRequestRelease(a); + AnnBatchClose(batcher); + for (int mode = 0; mode < 3; mode++) + { + AnnBatchDefaultOptions(&options); + options.delay_us = 1000000; + if (mode == 0) + options.max_pending_requests = 1; + if (mode == 1) + options.max_pending_bytes = one_request_bytes + one_request_bytes / 2; + if (mode == 2) + options.max_pending_work = 5; + batcher = Create(&options, 0); + a = Submit(batcher, 1, 3000000); + assert(AnnBatchSubmit(batcher, query, 2, 0, NULL, 0, 3000000, &b) == ANN_OVERLOADED); + assert(b == NULL && AnnRequestCancel(a)); + assert(AnnRequestPoll(a) == ANN_CANCELLED); + Idle(batcher); + AnnBatchGetStats(batcher, &stats); + assert(stats.rejected == 1 && stats.cancelled == 1); + AnnRequestRelease(a); + AnnBatchClose(batcher); + } +} + +static void +Deadlines(void) +{ + AnnBatchOptions options; + AnnBatcher *batcher; + AnnRequest *request; + AnnBatchStats stats; + + AnnBatchDefaultOptions(&options); + options.delay_us = 1000000; + batcher = Create(&options, 0); + request = Submit(batcher, 1, 10000); + assert(AnnRequestWait(request, 1000000) == ANN_TIMED_OUT); + Idle(batcher); + AnnBatchGetStats(batcher, &stats); + assert(stats.batches == 0 && stats.timed_out == 1); + AnnRequestRelease(request); + AnnBatchClose(batcher); + + options.delay_us = 0; + batcher = Create(&options, 1); + request = Submit(batcher, 1, 20000); + assert(AnnRequestWait(request, 1000000) == ANN_TIMED_OUT); + AnnBatchGetStats(batcher, &stats); + assert(stats.pending_requests == 1 && stats.pending_bytes > 0); + assert(!AnnRequestCancel(request)); + Idle(batcher); + AnnRequestRelease(request); + AnnBatchClose(batcher); +} + +typedef struct Producer +{ + AnnBatcher *batcher; + uint32_t offset; +} Producer; + +static void * +Produce(void *arg) +{ + Producer *producer = arg; + AnnRequest *requests[16]; + + for (uint32_t i = 0; i < 16; i++) + requests[i] = Submit(producer->batcher, (producer->offset + i) % 60, 3000000); + for (uint32_t i = 0; i < 16; i++) + { + Result (requests[i], (producer->offset + i) % 60); + + AnnRequestRelease(requests[i]); + } + return NULL; +} + +static void +Concurrent(void) +{ + AnnBatchOptions options; + AnnBatcher *batcher; + pthread_t threads[8]; + Producer producers[8]; + AnnBatchStats stats; + + AnnBatchDefaultOptions(&options); + options.max_batch_rows = 16; + options.delay_us = 5000; + batcher = Create(&options, 0); + for (uint32_t i = 0; i < 8; i++) + { + producers[i] = (Producer) + { + batcher, i * 8 + }; + assert(!pthread_create(&threads[i], NULL, Produce, &producers[i])); + } + for (uint32_t i = 0; i < 8; i++) + assert(!pthread_join(threads[i], NULL)); + Idle(batcher); + AnnBatchGetStats(batcher, &stats); + assert(stats.completed == 128 && stats.max_observed_batch <= 16 && stats.batches < 128); + AnnBatchClose(batcher); +} + +static void +Lifetime(void) +{ + AnnBatchOptions options; + AnnBatcher *batcher; + AnnBatchStats stats; + AnnRequest *request; + uint32_t count; + + AnnBatchDefaultOptions(&options); + batcher = Create(&options, 0); + for (uint32_t i = 0; i < 100; i++) + AnnRequestRelease(Submit(batcher, i % 60, 3000000)); + Idle(batcher); + AnnBatchGetStats(batcher, &stats); + assert(stats.completed == 100); + AnnBatchClose(batcher); + + options.delay_us = 0; + batcher = Create(&options, 1); + request = Submit(batcher, 1, 3000000); + for (int i = 0; i < 1000; i++) + { + AnnBatchGetStats(batcher, &stats); + if (stats.batches) + break; + Sleep(1); + } + assert(stats.batches == 1 && !AnnRequestCancel(request)); + assert(AnnRequestWait(request, 1) == ANN_PENDING); + AnnBatchClose(batcher); + assert(AnnRequestPoll(request) == ANN_CANCELLED); + assert(AnnRequestResults(request, &count) == NULL && count == 0); + AnnRequestRelease(request); +} + +static void +Errors(void) +{ + AnnBatchOptions options; + AnnBatcher *batcher; + AnnRequest *request; + AnnBatchStats stats; + + AnnBatchDefaultOptions(&options); + options.delay_us = 0; + batcher = Create(&options, 2); + request = Submit(batcher, 1, 3000000); + assert(AnnRequestWait(request, 1000000) == ANN_BACKEND_ERROR); + Idle(batcher); + AnnBatchGetStats(batcher, &stats); + assert(stats.failed == 1); + AnnRequestRelease(request); + AnnBatchClose(batcher); +} + +static void +Validate(void) +{ + AnnBatchOptions options; + AnnBatcher *batcher; + AnnRequest *request; + float query[2] = {1, 1}; + uint32_t ids[2] = {2, 1}; + + AnnBatchDefaultOptions(&options); + options.delay_us = 1000000; + batcher = Create(&options, 0); + assert(AnnBatchSubmit(batcher, query, 1, 99, NULL, 0, 1000, &request) == ANN_INVALID); + assert(AnnBatchSubmit(batcher, query, 1, 0, ids, 2, 1000, &request) == ANN_INVALID); + assert(AnnBatchSubmit(batcher, query, 1, 0, ids, 2, 0, &request) == ANN_INVALID); + request = Submit(batcher, 1, 3000000); + AnnBatchClose(batcher); + assert(AnnRequestPoll(request) == ANN_CANCELLED); + AnnRequestRelease(request); +} + +int +main(int argc, char **argv) +{ + if (argc != 2) + return 2; + if (!strcmp(argv[1], "batch")) + Batch(); + else if (!strcmp(argv[1], "admission")) + Admission(); + else if (!strcmp(argv[1], "deadlines")) + Deadlines(); + else if (!strcmp(argv[1], "concurrent")) + Concurrent(); + else if (!strcmp(argv[1], "validation")) + Validate(); + else if (!strcmp(argv[1], "lifetime")) + Lifetime(); + else if (!strcmp(argv[1], "errors")) + Errors(); + else + return 2; + puts("ok"); + return 0; +} diff --git a/test/ann/requirements-postgres.txt b/test/ann/requirements-postgres.txt new file mode 100644 index 000000000..627fec51b --- /dev/null +++ b/test/ann/requirements-postgres.txt @@ -0,0 +1 @@ +psycopg[binary]==3.2.13 diff --git a/test/ann/support/__init__.py b/test/ann/support/__init__.py new file mode 100644 index 000000000..f5df96ebf --- /dev/null +++ b/test/ann/support/__init__.py @@ -0,0 +1 @@ +"""Test-only protocol clients, SQL fixtures, and native scheduler bindings.""" diff --git a/test/ann/support/client.py b/test/ann/support/client.py new file mode 100644 index 000000000..6bbce36df --- /dev/null +++ b/test/ann/support/client.py @@ -0,0 +1,192 @@ +"""Test-only binary protocol driver; not a runtime client.""" + +import json +import hashlib +import os +import struct +import subprocess +import tempfile +import math +import threading +from concurrent.futures import ThreadPoolExecutor, TimeoutError as FutureTimeout +from pathlib import Path + +HEADER = struct.Struct('<4sHHQQII') +MAX_BYTES = 64 * 1024 * 1024 + + +class WorkerError(RuntimeError): + def __init__(self, status): + self.status = status + super().__init__(f'worker returned status {status}') + + +class Worker: + def __init__(self, executable=None, backend='cpu-exact', timeout=30): + if not isinstance(timeout, (int, float)) or not math.isfinite(timeout) or timeout <= 0: + raise ValueError('timeout must be a positive finite number of seconds') + executable = executable or Path(__file__).resolve().parents[3] / 'ann/ann-worker' + self.process = subprocess.Popen( + [str(executable), '--backend', backend], + stdin=subprocess.PIPE, stdout=subprocess.PIPE) + self.request_id = 0 + self.timeout = timeout + self._executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix='ann-transport') + self._request_lock = threading.Lock() + self._closed = False + + def _read(self, size): + result = bytearray() + while len(result) < size: + chunk = self.process.stdout.read(size - len(result)) + if not chunk: + raise EOFError('worker closed the protocol stream') + result.extend(chunk) + return bytes(result) + + def request(self, operation, generation=0, payload=b''): + if len(payload) > MAX_BYTES: + raise ValueError('frame exceeds protocol limit') + if not self._request_lock.acquire(blocking=False): + raise RuntimeError('concurrent requests on one worker are not supported') + try: + if self._closed: + raise RuntimeError('worker connection is closed') + future = self._executor.submit(self._exchange, operation, generation, payload) + try: + return future.result(timeout=self.timeout) + except FutureTimeout: + self.cancel() + raise TimeoutError('worker request timed out; generation invalidated') from None + except WorkerError: + raise # A framed operation error leaves the stream synchronized. + except BaseException: + self.cancel() + raise + finally: + self._request_lock.release() + + def _exchange(self, operation, generation, payload): + self.request_id += 1 + self.process.stdin.write(HEADER.pack( + b'PVAN', 1, operation, self.request_id, generation, len(payload), 0)) + self.process.stdin.write(payload) + self.process.stdin.flush() + magic, version, op, request, gen, size, status = HEADER.unpack(self._read(32)) + if (magic, version, op, request, gen) != ( + b'PVAN', 1, operation, self.request_id, generation) or size > MAX_BYTES: + raise ValueError('invalid worker response header') + result = self._read(size) + if status: + raise WorkerError(status) + return result + + def capabilities(self): + return json.loads(self.request(0)) + + def index_info(self, generation): + rows, dimensions = struct.unpack('<II', self.request(6, generation)) + return {'rows': rows, 'dimensions': dimensions} + + def build(self, generation, vectors, dimensions=None): + dimensions = dimensions if dimensions is not None else len(vectors[0]) + if dimensions <= 0 or any(len(row) != dimensions for row in vectors): + raise ValueError('inconsistent dimensions') + values = [value for row in vectors for value in row] + payload = struct.pack('<II', len(vectors), dimensions) + payload += struct.pack(f'<{len(values)}f', *values) + self.request(1, generation, payload) + + def search(self, generation, queries, k, metric=0, allowed=None, per_query=False): + if not queries or not queries[0] or any(len(q) != len(queries[0]) for q in queries): + raise ValueError('inconsistent query dimensions') + if per_query and (allowed is None or len(allowed) != len(queries)): + raise ValueError('one allowlist is required per query') + kind = 2 if per_query else 1 if allowed is not None else 0 + values = [value for row in queries for value in row] + payload = struct.pack('<IIIII', len(queries), len(queries[0]), k, metric, kind) + payload += struct.pack(f'<{len(values)}f', *values) + if kind: + for ids in allowed if per_query else [allowed]: + payload += struct.pack('<I', len(ids)) + payload += struct.pack(f'<{len(ids)}I', *ids) + data = self.request(2, generation, payload) + offset = 0 + result = [] + for _ in queries: + count, = struct.unpack_from('<I', data, offset) + offset += 4 + row = [] + for _ in range(count): + row.append(struct.unpack_from('<Id', data, offset)) + offset += 12 + result.append(row) + if offset != len(data): + raise ValueError('unexpected trailing response data') + return result + + def unload(self, generation): + self.request(3, generation) + + def save(self, generation, path): + """Persist canonical FP32 vectors, not a vendor graph or PostgreSQL index.""" + payload = self.request(4, generation) + path = Path(path) + header = struct.pack('<4sIQ32s', b'PVAS', 1, len(payload), hashlib.sha256(payload).digest()) + temporary = None + try: + with tempfile.NamedTemporaryFile(dir=path.parent, delete=False) as stream: + temporary = stream.name + stream.write(header + payload) + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary, path) + finally: + if temporary and os.path.exists(temporary): + os.unlink(temporary) + + def load(self, generation, path): + with open(path, 'rb') as stream: + header = stream.read(48) + if len(header) != 48: + raise ValueError('truncated snapshot header') + magic, version, size, digest = struct.unpack('<4sIQ32s', header) + if magic != b'PVAS' or version != 1 or size > MAX_BYTES: + raise ValueError('unsupported snapshot format or size') + payload = stream.read(size + 1) + if len(payload) != size or hashlib.sha256(payload).digest() != digest: + raise ValueError('snapshot size or checksum mismatch') + self.request(1, generation, payload) + + def close(self): + idle = self._request_lock.acquire(blocking=False) + self._closed = True + try: + if not idle: + self.cancel() + self._executor.shutdown(wait=True) + try: + self.process.stdin.close() + except (OSError, ValueError): + pass + try: + self.process.wait(timeout=5) + except subprocess.TimeoutExpired: + self.cancel() + self.process.stdout.close() + finally: + if idle: + self._request_lock.release() + + def cancel(self): + """Invalidate the entire process/session; this is not GPU kernel preemption.""" + self._closed = True + if self.process.poll() is None: + self.process.kill() + self.process.wait(timeout=5) + + def __enter__(self): + return self + + def __exit__(self, *args): + self.close() diff --git a/test/ann/support/postgres.py b/test/ann/support/postgres.py new file mode 100644 index 000000000..42c3c961b --- /dev/null +++ b/test/ann/support/postgres.py @@ -0,0 +1,162 @@ +"""Test-only frozen-table SQL reference; not runtime integration.""" + +import json +import time + +from .client import MAX_BYTES, Worker + + +class FrozenIndex: + """Own a table freeze and private read-only transaction for a worker generation. + + Predicates are trusted psycopg SQL expressions, with values bound separately. + Keys must be unique non-null integers and vectors non-null FP32 pgvector values. + """ + + def __init__(self, dsn, table, schema='public', key='id', vector='embedding', + executable=None, backend='cpu-exact', max_rows=100000, + worker_timeout=30, statement_timeout_ms=30000): + self.dsn = dsn + self.table = table + self.schema = schema + self.key = key + self.vector = vector + self.executable = executable + self.backend = backend + self.max_rows = max_rows + self.worker_timeout = worker_timeout + self.statement_timeout_ms = statement_timeout_ms + self.last_search = None + self._connection = None + self._worker = None + self._keys = [] + self._ordinals = {} + + def __enter__(self): + import psycopg + from psycopg import sql + + if self._connection is not None: + raise RuntimeError('index context is already active') + if self.max_rows <= 0: + raise ValueError('max_rows must be positive') + if not isinstance(self.statement_timeout_ms, int) or self.statement_timeout_ms <= 0: + raise ValueError('statement timeout must be a positive integer') + self._keys = [] + self._ordinals = {} + self._connection = psycopg.connect(self.dsn) + try: + self._connection.execute("SELECT set_config('statement_timeout', %s, true)", + (str(self.statement_timeout_ms),)) + relation = sql.Identifier(self.schema, self.table) + # Lock first, then prohibit this private connection's own writes too. + self._connection.execute(sql.SQL('LOCK TABLE {} IN SHARE MODE').format(relation)) + self._connection.execute('SET TRANSACTION READ ONLY') + self._connection.execute('SET LOCAL gin_fuzzy_search_limit = 0') + metadata = self._connection.execute( + 'SELECT c.relkind, am.amname, a.atttypmod, t.typname ' + 'FROM pg_class c JOIN pg_am am ON am.oid = c.relam ' + 'JOIN pg_attribute a ON a.attrelid = c.oid ' + 'JOIN pg_type t ON t.oid = a.atttypid ' + 'WHERE c.oid = %s::regclass AND a.attname = %s AND NOT a.attisdropped', + (relation.as_string(self._connection), self.vector)).fetchone() + if metadata is None or metadata[:2] != ('r', 'heap') or metadata[3] != 'vector': + raise ValueError('only heap tables with a vector column are supported') + dimensions = metadata[2] if metadata[2] > 0 else None + vectors = [] + query = sql.SQL('SELECT {}, {}::text FROM {} ORDER BY {}').format( + sql.Identifier(self.key), sql.Identifier(self.vector), relation, + sql.Identifier(self.key)) + with self._connection.cursor(name='ann_build') as cursor: + cursor.execute(query) + for key, value in cursor: + if not isinstance(key, int) or isinstance(key, bool) or key in self._ordinals: + raise ValueError('keys must be unique non-null integers') + if value is None: + raise ValueError('null vectors are not supported by this prototype') + vector = json.loads(value) + if dimensions is None: + dimensions = len(vector) + if len(vector) != dimensions: + raise ValueError('inconsistent vector dimensions') + rows = len(vectors) + 1 + if rows > self.max_rows or 8 + rows * dimensions * 4 > MAX_BYTES: + raise ValueError('prototype corpus materialization budget exceeded') + self._ordinals[key] = len(self._keys) + self._keys.append(key) + vectors.append(vector) + if dimensions is None: + raise ValueError('empty unconstrained vector column has no known dimension') + self._worker = Worker(self.executable, self.backend, timeout=self.worker_timeout) + self._worker.build(1, vectors, dimensions=dimensions) + return self + except BaseException: + self.close() + raise + + def search(self, queries, k, predicate=None, params=(), metric=0): + try: + return self._search(queries, k, predicate, params, metric) + except BaseException: + self.last_search = None + self.close() + raise + + def _search(self, queries, k, predicate, params, metric): + from psycopg import sql + + if self._connection is None or self._worker is None: + raise RuntimeError('index context is not active') + self.last_search = None + started = time.perf_counter() + if predicate is None: + predicate = sql.SQL('TRUE') + if not isinstance(predicate, sql.Composable): + raise TypeError('predicate must be a trusted psycopg SQL expression') + query = sql.SQL('SELECT {} FROM {} WHERE {}').format( + sql.Identifier(self.key), sql.Identifier(self.schema, self.table), predicate) + allowed = [] + with self._connection.cursor(name='ann_eligible') as cursor: + cursor.execute(query, params) + for (key,) in cursor: + if key not in self._ordinals: + raise RuntimeError('source identity changed while frozen') + allowed.append(self._ordinals[key]) + # Heap/key order is not a promise of local ordinal order. + allowed.sort() + prepared = time.perf_counter() + try: + result = self._worker.search(1, queries, k, metric=metric, allowed=allowed) + searched = time.perf_counter() + # Detect loss of the lock-owning session while external work was running. + self._connection.execute('SELECT 1').fetchone() + except BaseException: + self.close() # A failed external request must not retain the freeze. + raise + result = [[(self._keys[ordinal], distance) for ordinal, distance in row] for row in result] + finished = time.perf_counter() + self.last_search = { + 'eligible_rows': len(allowed), 'query_vectors': len(queries), + 'eligibility_mapping_seconds': prepared - started, + 'worker_roundtrip_seconds': searched - prepared, + 'session_check_projection_seconds': finished - searched, + 'total_seconds': finished - started, + } + return result + + def close(self): + try: + if self._worker is not None: + self._worker.close() + finally: + self._worker = None + try: + if self._connection is not None: + self._connection.close() # Rolls back and releases the table freeze. + finally: + self._connection = None + self._keys = [] + self._ordinals = {} + + def __exit__(self, *args): + self.close() diff --git a/test/ann/test_batching.py b/test/ann/test_batching.py new file mode 100644 index 000000000..80e5cb04a --- /dev/null +++ b/test/ann/test_batching.py @@ -0,0 +1,40 @@ +"""Python invokes native C scenarios; scheduler implementation is C only.""" + +from pathlib import Path +import subprocess +import unittest + +EXECUTABLE = Path(__file__).resolve().parents[2] / 'ann/ann-batch-tests' + + +class NativeBatchTests(unittest.TestCase): + def scenario(self, name): + result = subprocess.run([str(EXECUTABLE), name], capture_output=True, + text=True, timeout=15) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertEqual(result.stdout.strip(), 'ok') + + def test_batch_and_result_lifetime(self): + self.scenario('batch') + + def test_admission_budgets(self): + self.scenario('admission') + + def test_queue_and_running_deadlines(self): + self.scenario('deadlines') + + def test_concurrent_producers(self): + self.scenario('concurrent') + + def test_validation_and_shutdown(self): + self.scenario('validation') + + def test_early_release_and_running_shutdown(self): + self.scenario('lifetime') + + def test_backend_error_propagation(self): + self.scenario('errors') + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_cuvs.py b/test/ann/test_cuvs.py new file mode 100644 index 000000000..c93edeafe --- /dev/null +++ b/test/ann/test_cuvs.py @@ -0,0 +1,62 @@ +"""GPU tests skip by default, but fail on missing SDK/device when explicitly enabled.""" + +import os +from pathlib import Path +import random +import sys +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT / 'test/ann')) +from support.client import Worker, WorkerError + + +@unittest.skipUnless(os.environ.get('ANN_CUVS_TEST') == '1', 'set ANN_CUVS_TEST=1 on a cuVS host') +class CuvsTests(unittest.TestCase): + def test_filtered_recall_reload_and_capabilities(self): + rng = random.Random(23) + vectors = [[rng.random() for _ in range(32)] for _ in range(4096)] + queries = vectors[::1024] + filters = [list(range(i, 4096, 4)) for i in range(4)] + with Worker() as cpu, Worker(ROOT / 'ann/ann-worker-cuvs', 'cuvs') as gpu: + caps = gpu.capabilities() + self.assertFalse(caps['exact']) + self.assertEqual(caps['metrics_mask'], 1) + self.assertEqual(caps['algorithm'], 'cagra') + cpu.build(1, vectors) + gpu.build(1, vectors) + for allowed, private in [(None, False), (filters[0], False), (filters, True)]: + truth = cpu.search(1, queries, 10, allowed=allowed, per_query=private) + actual = gpu.search(1, queries, 10, allowed=allowed, per_query=private) + for i, (expected, found) in enumerate(zip(truth, actual)): + self.assertEqual(len(found), 10) + expected_ids = {r[0] for r in expected} + self.assertGreaterEqual(len(expected_ids & {r[0] for r in found}) / 10, .9) + if allowed is not None: + eligible = allowed[i] if private else allowed + self.assertTrue(all(r[0] in eligible for r in found)) + self.assertEqual(found, sorted(found, key=lambda r: (r[1], r[0]))) + self.assertEqual(gpu.search(1, queries, 10, allowed=[]), [[]] * 4) + with self.assertRaises(WorkerError) as error: + gpu.search(1, queries, 10, metric=2) + self.assertEqual(error.exception.status, 2) + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / 'canonical' + gpu.save(1, path) + gpu.unload(1) + gpu.load(2, path) + self.assertEqual(gpu.search(2, [vectors[0]], 1)[0][0], (0, 0)) + + def test_failed_build_preserves_generation(self): + vectors = [[float(i), 1] for i in range(256)] + with Worker(ROOT / 'ann/ann-worker-cuvs', 'cuvs') as gpu: + gpu.build(1, vectors) + vectors[0][0] = float('nan') + with self.assertRaises(WorkerError): + gpu.build(2, vectors) + self.assertEqual(gpu.search(1, [[0, 1]], 1), [[(0, 0)]]) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_cuvs_gathered.py b/test/ann/test_cuvs_gathered.py new file mode 100644 index 000000000..acdeb848e --- /dev/null +++ b/test/ann/test_cuvs_gathered.py @@ -0,0 +1,72 @@ +"""Gathered search on a real GPU through the cuVS worker build. + +Set ANN_CUVS_TEST=1 after `make -C ann ann-worker-cuvs`; see ann/README.md. +""" + +import json +import os +import random +import sys +import unittest + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from test_server import (GATHERED, ROOT, STATS, Server, decode, expected, # noqa: E402 + frame, gathered, receive) + +EXECUTABLE = os.path.join(ROOT, 'ann', 'ann-worker-cuvs') + + +@unittest.skipUnless(os.environ.get('ANN_CUVS_TEST'), 'set ANN_CUVS_TEST=1 on a cuVS host') +class CuvsGatheredTests(unittest.TestCase): + def setUp(self): + self.server = Server('--backend', 'cuvs', '--lanes', '4', executable=EXECUTABLE) + + def tearDown(self): + self.server.close() + + def device_jobs(self, connection): + connection.sendall(frame(STATS, 1)) + return json.loads(receive(connection)[3])['device_jobs'] + + def search(self, connection, requests): + for request, (query, vectors, k) in requests.items(): + connection.sendall(frame(GATHERED, request, gathered(query, vectors, k))) + replies = {} + while len(replies) < len(requests): + _, request, status, payload = receive(connection) + self.assertEqual(status, 0) + replies[request] = [i for i, _ in decode(payload)] + return replies + + def test_matches_exact_results_on_device(self): + rng = random.Random(11) + requests = {} + for request in range(1, 13): + dimensions = rng.choice([3, 16, 96]) + rows = rng.randint(2048, 5000) + vectors = [[round(rng.uniform(-2, 2), 2) for _ in range(dimensions)] for _ in range(rows)] + query = [round(rng.uniform(-2, 2), 2) for _ in range(dimensions)] + requests[request] = (query, vectors, rng.choice([1, 10, 100, 400])) + with self.server.connect() as connection: + replies = self.search(connection, requests) + for request, (query, vectors, k) in requests.items(): + # Order among near-equal distances is not significant; callers re-sort. + self.assertEqual(sorted(replies[request]), sorted(expected(query, vectors, k)), request) + self.assertEqual(self.device_jobs(connection), len(requests)) + + def test_heavy_ties_stay_exact(self): + # Ties beyond the GPU candidate window must fall back to the CPU kernel. + vectors = [[1.0, 0.0]] * 3000 + [[5.0, 5.0]] * 100 + with self.server.connect() as connection: + replies = self.search(connection, {1: ([0.0, 0.0], vectors, 5)}) + self.assertEqual(sorted(replies[1]), list(range(3000))) + + def test_small_jobs_use_the_cpu(self): + with self.server.connect() as connection: + replies = self.search(connection, {1: ([0.0], [[2.0], [1.0], [3.0]], 1)}) + self.assertEqual(replies[1], [1]) + self.assertEqual(self.device_jobs(connection), 0) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_cuvs_index.py b/test/ann/test_cuvs_index.py new file mode 100644 index 000000000..54f5580c8 --- /dev/null +++ b/test/ann/test_cuvs_index.py @@ -0,0 +1,102 @@ +"""CAGRA shards built by scripts/ann/build_cuvs_index.py, served through SQL on GPUs. + +Needs a cuVS host: ANN_CUVS_TEST=1, ANN_PG_DSN for a local superuser, the +ann-worker-cuvs build, and Python packages cuvs, cupy, numpy, psycopg. +""" + +import os +import subprocess +import sys +import tempfile +import unittest +import uuid + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from test_server import ROOT, Server # noqa: E402 + +DSN = os.environ.get('ANN_PG_DSN') +PSQL = os.environ.get('ANN_PSQL', 'psql') +BUILD = os.path.join(ROOT, 'scripts', 'ann', 'build_cuvs_index.py') + + +@unittest.skipUnless(os.environ.get('ANN_CUVS_TEST') and DSN, 'set ANN_CUVS_TEST=1 and ANN_PG_DSN on a cuVS host') +class CuvsIndexTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.directory = tempfile.TemporaryDirectory() + # A small exact limit makes 2% filters take the filtered CAGRA route. + cls.server = Server('--backend', 'cuvs', '--lanes', '8', '--exact-limit', '500', + executable=os.path.join(ROOT, 'ann', 'ann-worker-cuvs')) + cls.table = 'ann_cuvs_' + uuid.uuid4().hex[:10] + cls.psql(f""" + CREATE TABLE {cls.table} (id bigint PRIMARY KEY, tenant int, emb vector(32)); + SELECT setseed(0.1); + INSERT INTO {cls.table} SELECT i * 3, i % 50, + (SELECT array_agg(random()::real) FROM generate_series(1, 32) WHERE i > 0)::vector + FROM generate_series(1, 40000) i; + """) + for dtype in ['float32', 'float16']: + path = os.path.join(cls.directory.name, dtype) + subprocess.run([sys.executable, BUILD, '--dsn', DSN, '--table', f'public.{cls.table}', '--key', 'id', + '--vector', 'emb', '--out', path, '--shards', '4', '--dtype', dtype], + check=True, capture_output=True) + cls.psql(f"SELECT ann_index_load('{cls.table}_{dtype}', '{path}')") + + @classmethod + def tearDownClass(cls): + cls.psql(f'DROP TABLE {cls.table}') + cls.server.close() + cls.directory.cleanup() + + @classmethod + def psql(cls, script): + result = subprocess.run([PSQL, DSN, '-X', '-q', '-At', '-v', 'ON_ERROR_STOP=1'], + input=f"SET ann.worker_socket = '{cls.server.path}';\n" + script, + capture_output=True, text=True, timeout=600) + if result.returncode != 0: + raise AssertionError(result.stderr) + return result.stdout.strip() + + def recall(self, index, filtered): + fltr = f', ARRAY(SELECT id FROM {self.table} WHERE tenant = q.tenant)' if filtered else '' + where = 't.tenant = q.tenant' if filtered else 'true' + return float(self.psql(f""" + SELECT avg(cardinality(ARRAY(SELECT unnest(a) INTERSECT SELECT unnest(b)))) / 10 FROM ( + SELECT ARRAY(SELECT s.id FROM ann_index_search('{index}', q.emb, 10{fltr}) s) a, + ARRAY(SELECT t.id FROM {self.table} t WHERE {where} ORDER BY t.emb <-> q.emb LIMIT 10) b + FROM {self.table} q WHERE q.id % 1200 = 0) x""")) + + def test_ann_recall(self): + for dtype in ['float32', 'float16']: + self.assertGreaterEqual(self.recall(f'{self.table}_{dtype}', False), 0.9, dtype) + + def test_filtered_graph_route(self): + for dtype in ['float32', 'float16']: + index = f'{self.table}_{dtype}' + self.assertGreaterEqual(self.recall(index, True), 0.9, dtype) + outside = self.psql(f""" + SELECT count(*) FROM {self.table} q, ann_index_search('{index}', q.emb, 10, + ARRAY(SELECT id FROM {self.table} WHERE tenant = q.tenant)) s + JOIN {self.table} t USING (id) WHERE q.id % 1200 = 0 AND t.tenant <> q.tenant""") + self.assertEqual(outside, '0') + + def test_exact_route_matches_sql(self): + # 100 eligible keys is under the exact limit, so results are exact. + mismatches = self.psql(f""" + SELECT count(*) FILTER (WHERE a IS DISTINCT FROM b) FROM ( + SELECT ARRAY(SELECT s.id FROM ann_index_search('{self.table}_float16', q.emb, 10, + ARRAY(SELECT id FROM {self.table} WHERE id % 1200 < 3)) s ORDER BY s.distance, s.id) a, + ARRAY(SELECT t.id FROM {self.table} t WHERE t.id % 1200 < 3 ORDER BY t.emb <-> q.emb, t.id LIMIT 10) b + FROM {self.table} q WHERE q.id % 3000 = 0) x""") + self.assertEqual(mismatches, '0') + + def test_distances_are_exact(self): + error = self.psql(f""" + SELECT max(abs(s.distance - (t.emb <-> q.emb))) FROM {self.table} q, + ann_index_search('{self.table}_float16', q.emb, 10) s JOIN {self.table} t USING (id) + WHERE q.id % 2400 = 0""") + self.assertLess(float(error), 1e-4) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_filtered.py b/test/ann/test_filtered.py new file mode 100644 index 000000000..00659881c --- /dev/null +++ b/test/ann/test_filtered.py @@ -0,0 +1,289 @@ +"""Opt-in native filtered CustomScan tests against ordinary PostgreSQL SQL.""" + +import os +import unittest +import uuid + + +@unittest.skipUnless(os.environ.get('ANN_PG_DSN'), 'set ANN_PG_DSN for live PostgreSQL tests') +class FilteredTests(unittest.TestCase): + def setUp(self): + import psycopg + from psycopg import sql + + self.psycopg = psycopg + self.sql = sql + self.dsn = os.environ['ANN_PG_DSN'] + self.db = psycopg.connect(self.dsn, autocommit=True, prepare_threshold=None) + self.db.execute("LOAD 'vector'") + self.db.execute('SET ann.enable_filtered_search = on') + self.db.execute('SET max_parallel_workers_per_gather = 0') + self.schema = 'ann_filtered_' + uuid.uuid4().hex + self.table = sql.Identifier(self.schema, 'documents') + self.db.execute(sql.SQL('CREATE SCHEMA {}').format(sql.Identifier(self.schema))) + self.db.execute(sql.SQL('CREATE TABLE {} (id integer, tenant text, terms tsvector, ' + 'embedding vector(3))').format(self.table)) + self.db.execute(sql.SQL("INSERT INTO {} SELECT i, CASE WHEN i % 101 = 0 THEN 'rare' " + "ELSE 'common' END, to_tsvector('simple', CASE WHEN i % 101 = 0 " + "THEN 'database search' ELSE 'storage' END), " + 'ARRAY[i * 0.123, i % 13, i % 17]::vector ' + 'FROM generate_series(1, 4096) i').format(self.table)) + self.db.execute(sql.SQL('CREATE INDEX documents_ann ON {} USING ann ' + '(embedding vector_l2_ops)').format(self.table)) + self.db.execute(sql.SQL('CREATE INDEX documents_tenant ON {} (tenant)').format(self.table)) + self.db.execute(sql.SQL('CREATE INDEX documents_terms ON {} USING gin (terms)').format(self.table)) + self.db.execute(sql.SQL('ANALYZE {}').format(self.table)) + + def tearDown(self): + self.db.execute('RESET ROLE') + self.db.execute(self.sql.SQL('DROP SCHEMA {} CASCADE').format(self.sql.Identifier(self.schema))) + self.db.close() + + def statement(self, predicate="tenant = 'rare'", ending='LIMIT 10', projection='id'): + return self.sql.SQL('SELECT {}, embedding <-> %s::vector AS distance FROM {} ' + 'WHERE {} ORDER BY distance, id {}').format( + self.sql.SQL(projection), self.table, + self.sql.SQL(predicate), self.sql.SQL(ending)) + + @staticmethod + def custom_nodes(plan): + nodes = [] + if plan.get('Custom Plan Provider') == 'AnnFilteredSearch': + nodes.append(plan) + for child in plan.get('Plans', []): + nodes.extend(FilteredTests.custom_nodes(child)) + return nodes + + def explain(self, statement, params=('[13.4,3,5]',), analyze=True, db=None): + db = db or self.db + prefix = 'EXPLAIN (ANALYZE, FORMAT JSON, TIMING OFF)' if analyze else 'EXPLAIN (FORMAT JSON)' + return db.execute(self.sql.SQL('{} {}').format(self.sql.SQL(prefix), statement), params).fetchone()[0][0]['Plan'] + + def assert_oracle(self, statement, params=('[13.4,3,5]',), db=None): + db = db or self.db + db.execute('SET ann.enable_filtered_search = on') + plan = self.explain(statement, params, db=db) + nodes = self.custom_nodes(plan) + self.assertEqual(len(nodes), 1, plan) + actual = db.execute(statement, params).fetchall() + db.execute('SET ann.enable_filtered_search = off') + db.execute('SET enable_indexscan = off') + db.execute('SET enable_bitmapscan = off') + expected = db.execute(statement, params).fetchall() + db.execute('SET enable_indexscan = on') + db.execute('SET enable_bitmapscan = on') + db.execute('SET ann.enable_filtered_search = on') + self.assertEqual(actual, expected) + return nodes[0], actual + + def test_btree_eligibility_and_projection(self): + node, rows = self.assert_oracle(self.statement(projection='id, upper(tenant), embedding')) + self.assertEqual(len(rows), 10) + self.assertEqual(node['Eligible Rows'], 40) + self.assertEqual(node['Distance Evaluations'], 40) + self.assertIn('documents_tenant', str(node['Plans'])) + self.assertEqual(node['Sort Method'], 'top-N heapsort') + + def test_gin_and_combined_filters(self): + for predicate in ["terms @@ to_tsquery('simple', 'database & search')", + "tenant = 'rare' AND terms @@ to_tsquery('simple', 'database')", + "tenant = 'missing'"]: + node, _ = self.assert_oracle(self.statement(predicate)) + self.assertEqual(node['Eligible Rows'], 0 if 'missing' in predicate else 40) + self.db.execute('SET enable_seqscan = off') + self.db.execute('SET enable_indexscan = off') + node = self.custom_nodes(self.explain(self.statement( + "terms @@ to_tsquery('simple', 'database & search')")))[0] + self.assertIn('documents_terms', str(node['Plans'])) + self.assertEqual(node['Distance Evaluations'], 40) + + def test_ties_nulls_offsets_and_spill(self): + self.db.execute(self.sql.SQL("UPDATE {} SET embedding = '[1,2,3]' " + 'WHERE id IN (101,202,303,404,505)').format(self.table)) + self.db.execute(self.sql.SQL("INSERT INTO {} VALUES (5000, 'rare', NULL, NULL)").format(self.table)) + for ending in ['LIMIT 3 OFFSET 2', 'LIMIT ALL OFFSET 3', 'LIMIT 5000', 'LIMIT NULL']: + self.assert_oracle(self.statement(ending=ending), ('[1,2,3]',)) + self.db.execute("SET work_mem = '64kB'") + node, rows = self.assert_oracle(self.statement("tenant IS NOT NULL", '')) + self.assertEqual(node['Sort Space Type'], 'Disk') + self.assertEqual(len(rows), 4097) + self.assertEqual(rows[-1][0], 5000) + statement = self.sql.SQL("SELECT id FROM {} WHERE tenant = 'rare' " + "ORDER BY embedding <-> '[1,2,3]' FETCH FIRST 2 ROWS WITH TIES").format(self.table) + node = self.custom_nodes(self.explain(statement, ()))[0] + self.assertNotEqual(node['Sort Method'], 'top-N heapsort') + self.assertEqual(sorted(row[0] for row in self.db.execute(statement)), [101, 202, 303, 404, 505]) + + def test_generic_prepared_parameters(self): + self.db.execute('SET plan_cache_mode = force_generic_plan') + self.db.execute(self.sql.SQL('PREPARE filtered_query(vector, bigint, bigint) AS ' + "SELECT id, embedding <-> $1 AS distance FROM {} WHERE tenant = 'rare' " + 'ORDER BY distance, id LIMIT $2 OFFSET $3').format(self.table)) + for query, limit, offset in [('[13.4,3,5]', 2, 0), ('[100,2,3]', 19, 3), + (None, 5, 4), ('[0,0,0]', None, 7), ('[1,2,3]', 0, 0)]: + execute = self.sql.SQL('EXECUTE filtered_query({}, {}, {})').format( + self.sql.Literal(query), self.sql.Literal(limit), self.sql.Literal(offset)) + actual = self.db.execute(execute).fetchall() + plan = self.explain(execute, ()) + self.assertEqual(len(self.custom_nodes(plan)), 1, plan) + self.db.execute('SET ann.enable_filtered_search = off') + expected = self.db.execute(self.statement(ending='LIMIT %s OFFSET %s'), (query, limit, offset)).fetchall() + self.db.execute('SET ann.enable_filtered_search = on') + self.assertEqual(actual, expected) + for limit, offset in [(-1, 0), (1, -1)]: + with self.assertRaises(self.psycopg.errors.InvalidRowCountInLimitClause if limit < 0 + else self.psycopg.errors.InvalidRowCountInResultOffsetClause): + self.db.execute(self.sql.SQL('EXECUTE filtered_query(NULL, {}, {})').format( + self.sql.Literal(limit), self.sql.Literal(offset))) + + def test_fuzzy_gin_and_cached_plan_guard(self): + self.db.execute('SET plan_cache_mode = force_generic_plan') + self.db.execute(self.sql.SQL('PREPARE guarded_query(vector) AS ' + "SELECT id FROM {} WHERE tenant = 'rare' " + 'ORDER BY embedding <-> $1 LIMIT 10').format(self.table)) + execute = self.sql.SQL("EXECUTE guarded_query('[1,2,3]')") + self.assertEqual(len(self.custom_nodes(self.explain(execute, ()))), 1) + self.db.execute('SET gin_fuzzy_search_limit = 100') + self.assertEqual(self.custom_nodes(self.explain(self.statement(), analyze=False)), []) + with self.assertRaisesRegex(self.psycopg.errors.FeatureNotSupported, 'gin_fuzzy_search_limit'): + self.db.execute(execute) + self.db.execute('SET gin_fuzzy_search_limit = 0') + self.assertEqual(len(self.db.execute(execute).fetchall()), 10) + + def test_snapshot_visibility_and_writes(self): + with self.psycopg.connect(self.dsn, autocommit=True, prepare_threshold=None) as reader: + reader.execute("LOAD 'vector'") + reader.execute('SET max_parallel_workers_per_gather = 0') + reader.execute('BEGIN ISOLATION LEVEL REPEATABLE READ') + _, original = self.assert_oracle(self.statement(), db=reader) + self.db.execute(self.sql.SQL("DELETE FROM {} WHERE id = 101").format(self.table)) + self.db.execute(self.sql.SQL("UPDATE {} SET embedding = '[0,0,0]' WHERE id = 202").format(self.table)) + self.db.execute(self.sql.SQL('VACUUM {}').format(self.table)) + _, old = self.assert_oracle(self.statement(), db=reader) + self.assertEqual(old, original) + reader.execute('COMMIT') + self.db.execute('BEGIN') + self.db.execute(self.sql.SQL("INSERT INTO {} VALUES (9000, 'rare', NULL, '[13.4,3,5]')").format(self.table)) + _, rows = self.assert_oracle(self.statement()) + self.assertEqual(rows[0][0], 9000) + self.db.execute('ROLLBACK') + self.assert_oracle(self.statement()) + + def test_rls_child_plan(self): + allowed = self.db.execute("SELECT rolsuper OR rolcreaterole FROM pg_roles WHERE rolname = current_user").fetchone()[0] + if not allowed: + self.skipTest('RLS fixture needs permission to create a temporary role') + role = self.sql.Identifier('ann_rls_' + uuid.uuid4().hex) + self.db.execute(self.sql.SQL('CREATE ROLE {}').format(role)) + try: + self.db.execute(self.sql.SQL('GRANT USAGE ON SCHEMA {} TO {}').format(self.sql.Identifier(self.schema), role)) + self.db.execute(self.sql.SQL('GRANT SELECT ON {} TO {}').format(self.table, role)) + self.db.execute(self.sql.SQL('ALTER TABLE {} ENABLE ROW LEVEL SECURITY').format(self.table)) + self.db.execute(self.sql.SQL("CREATE POLICY tenant_policy ON {} USING (tenant = 'rare')").format(self.table)) + self.db.execute(self.sql.SQL('SET ROLE {}').format(role)) + node, rows = self.assert_oracle(self.statement('id > 0')) + self.assertEqual(node['Eligible Rows'], 40) + self.assertEqual(len(rows), 10) + finally: + self.db.execute('RESET ROLE') + self.db.execute(self.sql.SQL('DROP OWNED BY {}').format(role)) + self.db.execute(self.sql.SQL('DROP ROLE {}').format(role)) + + def test_distance_function_execute_permission(self): + allowed = self.db.execute("SELECT rolsuper OR rolcreaterole FROM pg_roles WHERE rolname = current_user").fetchone()[0] + if not allowed: + self.skipTest('permission fixture needs a temporary role') + function = self.sql.Identifier(self.schema, 'private_distance') + operator = self.sql.SQL('OPERATOR({}.<~>)').format(self.sql.Identifier(self.schema)) + self.db.execute(self.sql.SQL('CREATE FUNCTION {} (vector, vector) RETURNS double precision ' + 'LANGUAGE plpgsql IMMUTABLE STRICT AS ' + '$$BEGIN RETURN l2_distance($1, $2); END$$').format(function)) + self.db.execute(self.sql.SQL('CREATE OPERATOR {}.<~> (LEFTARG = vector, RIGHTARG = vector, ' + 'FUNCTION = {})').format(self.sql.Identifier(self.schema), function)) + opclass = self.sql.Identifier(self.schema, 'private_ops') + self.db.execute(self.sql.SQL('CREATE OPERATOR CLASS {} FOR TYPE vector USING ann AS ' + 'OPERATOR 1 {}.<~> (vector, vector) FOR ORDER BY float_ops, ' + 'FUNCTION 1 {} (vector, vector)').format(opclass, self.sql.Identifier(self.schema), function)) + self.db.execute(self.sql.SQL('CREATE INDEX ON {} USING ann (embedding {})').format(self.table, opclass)) + self.db.execute(self.sql.SQL('REVOKE EXECUTE ON FUNCTION {} (vector, vector) FROM PUBLIC').format(function)) + statement = self.sql.SQL("SELECT id, embedding {} %s::vector AS distance FROM {} WHERE tenant = 'rare' " + 'ORDER BY distance, id LIMIT 10').format(operator, self.table) + role = self.sql.Identifier('ann_acl_' + uuid.uuid4().hex) + self.db.execute(self.sql.SQL('CREATE ROLE {}').format(role)) + try: + self.db.execute(self.sql.SQL('GRANT USAGE ON SCHEMA {} TO {}').format(self.sql.Identifier(self.schema), role)) + self.db.execute(self.sql.SQL('GRANT SELECT ON {} TO {}').format(self.table, role)) + self.db.execute(self.sql.SQL('SET ROLE {}').format(role)) + for enabled in ['on', 'off']: + self.db.execute('SET ann.enable_filtered_search = ' + enabled) + with self.assertRaises(self.psycopg.errors.InsufficientPrivilege): + self.db.execute(statement, ('[1,2,3]',)) + self.db.execute('RESET ROLE') + self.db.execute(self.sql.SQL('GRANT EXECUTE ON FUNCTION {} (vector, vector) TO {}').format(function, role)) + self.db.execute(self.sql.SQL('SET ROLE {}').format(role)) + self.assert_oracle(statement, ('[1,2,3]',)) + finally: + self.db.execute('RESET ROLE') + self.db.execute(self.sql.SQL('DROP OWNED BY {}').format(role)) + self.db.execute(self.sql.SQL('DROP ROLE {}').format(role)) + + def test_index_only_fallback(self): + self.db.execute(self.sql.SQL('DROP INDEX {}').format(self.sql.Identifier(self.schema, 'documents_tenant'))) + self.db.execute(self.sql.SQL('CREATE INDEX documents_cover ON {} (tenant, id) ' + 'INCLUDE (embedding)').format(self.table)) + self.db.execute(self.sql.SQL('VACUUM ANALYZE {}').format(self.table)) + self.db.execute('SET enable_seqscan = off') + self.db.execute('SET enable_bitmapscan = off') + plan = self.explain(self.statement()) + self.assertIn('Index Only Scan', str(plan)) + self.assertEqual(self.custom_nodes(plan), []) + + def test_toasted_vectors(self): + wide = self.sql.Identifier(self.schema, 'wide_documents') + self.db.execute(self.sql.SQL('CREATE TABLE {} (id integer, tenant text, ' + 'embedding vector(2000))').format(wide)) + vector = str([i / 2000 for i in range(2000)]) + for i in range(8): + self.db.execute(self.sql.SQL("INSERT INTO {} VALUES (%s, 'rare', %s::vector)").format(wide), (i, vector)) + self.db.execute(self.sql.SQL('CREATE INDEX ON {} USING ann (embedding vector_l2_ops)').format(wide)) + old_table = self.table + self.table = wide + try: + node, rows = self.assert_oracle(self.statement(ending='LIMIT 5'), (vector,)) + self.assertEqual(node['Distance Evaluations'], 8) + self.assertEqual(rows, [(i, 0.0) for i in range(5)]) + finally: + self.table = old_table + + def test_cancellation_cleanup(self): + function = self.sql.Identifier(self.schema, 'slow_eligible') + self.db.execute(self.sql.SQL('CREATE FUNCTION {} (i integer) RETURNS boolean LANGUAGE plpgsql STABLE AS ' + "$$BEGIN PERFORM pg_sleep(0.002); RETURN i > 0; END$$").format(function)) + statement = self.sql.SQL("SELECT id FROM {} WHERE tenant = 'rare' AND {}(id) " + "ORDER BY embedding <-> '[1,2,3]' LIMIT 10").format(self.table, function) + self.assertEqual(len(self.custom_nodes(self.explain(statement, (), analyze=False))), 1) + self.db.execute("SET statement_timeout = '20ms'") + with self.assertRaises(self.psycopg.errors.QueryCanceled): + self.db.execute(statement) + self.db.execute('SET statement_timeout = 0') + self.assert_oracle(self.statement()) + + def test_unsupported_shapes_and_disabled_path(self): + statements = [ + self.statement(ending='LIMIT 10 FOR UPDATE'), + self.sql.SQL("SELECT id FROM {} WHERE tenant = 'rare' ORDER BY embedding <-> %s::vector DESC LIMIT 10").format(self.table), + self.sql.SQL("SELECT id FROM {} WHERE tenant = 'rare' ORDER BY embedding <-> %s::vector NULLS FIRST LIMIT 10").format(self.table), + self.sql.SQL('SELECT id FROM {} ORDER BY embedding <-> %s::vector LIMIT 10').format(self.table), + ] + for statement in statements: + self.assertEqual(self.custom_nodes(self.explain(statement, analyze=False)), []) + self.db.execute(statement, ('[13.4,3,5]',)).fetchall() + self.db.execute('SET ann.enable_filtered_search = off') + self.assertEqual(self.custom_nodes(self.explain(self.statement(), analyze=False)), []) + self.db.execute('SET ann.enable_filtered_search = on') + self.db.execute(self.sql.SQL('DROP INDEX {}').format(self.sql.Identifier(self.schema, 'documents_ann'))) + self.assertEqual(self.custom_nodes(self.explain(self.statement(), analyze=False)), []) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_index.py b/test/ann/test_index.py new file mode 100644 index 000000000..8d49b976c --- /dev/null +++ b/test/ann/test_index.py @@ -0,0 +1,265 @@ +"""Loaded-index tests: file format, sharded tree merge, filters, and SQL functions. + +Worker tests need only the CPU build. SQL tests also need ANN_PG_DSN for a +superuser on this host (the server connects to the worker's socket); ANN_PSQL +overrides psql. On a cuVS host, ANN_INDEX_WORKER=ann-worker-cuvs plus +ANN_INDEX_BACKEND=cuvs runs the same checks against CAGRA files built by +scripts/ann/build_cuvs_index.py (see test/ann/test_cuvs_index.py). +""" + +import math +import os +import random +import struct +import subprocess +import sys +import tempfile +import unittest +import uuid + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from test_server import ROOT, STATS, Server, frame, receive # noqa: E402 + +LOAD = 11 +SEARCH = 12 +DSN = os.environ.get('ANN_PG_DSN') +PSQL = os.environ.get('ANN_PSQL', 'psql') + + +def f32(value): + return struct.unpack('<f', struct.pack('<f', value))[0] + + +def write_index(directory, keys, vectors, shards=1, source='-', dtype='float32', key='id', vector='emb'): + """Test fixture writer for the documented on-disk format (CPU route only).""" + order = sorted(range(len(keys)), key=lambda i: keys[i]) + keys = [keys[i] for i in order] + vectors = [vectors[i] for i in order] + os.makedirs(directory, exist_ok=True) + with open(os.path.join(directory, 'keys.i64'), 'wb') as out: + out.write(struct.pack('<%dq' % len(keys), *keys)) + with open(os.path.join(directory, 'vectors.f32'), 'wb') as out: + for row in vectors: + out.write(struct.pack('<%df' % len(row), *row)) + bounds = [len(keys) * i // shards for i in range(shards + 1)] + lines = ['pgvector-ann-index 1', f'dimensions {len(vectors[0])}', f'rows {len(keys)}', + 'metric l2', f'dtype {dtype}', f'source {source}', f'key {key}', f'vector {vector}', + f'shards {shards}'] + lines += [f'shard {i} {bounds[i]} {bounds[i + 1] - bounds[i]} shard-{i}.cagra' for i in range(shards)] + with open(os.path.join(directory, 'manifest'), 'w') as out: + out.write('\n'.join(lines) + '\n') + return keys, vectors + + +def load_frame(request, name, directory): + payload = struct.pack('<I', len(name)) + name.encode() + directory.encode() + return frame(LOAD, request, payload) + + +def search_frame(request, name, query, k, keys=None): + payload = struct.pack('<I', len(name)) + name.encode() + payload += struct.pack('<III', len(query), k, 0xFFFFFFFF if keys is None else len(keys)) + payload += struct.pack('<%df' % len(query), *query) + if keys: + payload += struct.pack('<%dq' % len(keys), *keys) + return frame(SEARCH, request, payload) + + +def decode_hits(payload): + count, route = struct.unpack_from('<II', payload) + return route, [struct.unpack_from('<qf', payload, 8 + i * 12) for i in range(count)] + + +def exact(keys, vectors, query, k, allowed=None): + scored = sorted((math.sqrt(sum((f32(a) - f32(b)) ** 2 for a, b in zip(row, query))), key) + for key, row in zip(keys, vectors) if allowed is None or key in allowed) + return scored[:k] + + +class IndexWorkerTests(unittest.TestCase): + worker = os.path.join(ROOT, 'ann', os.environ.get('ANN_INDEX_WORKER', 'ann-worker')) + backend = os.environ.get('ANN_INDEX_BACKEND', 'cpu-exact') + + @classmethod + def setUpClass(cls): + subprocess.run(['make', '-C', os.path.join(ROOT, 'ann'), 'ann-worker'], check=True, + stdout=subprocess.DEVNULL) + + def setUp(self): + self.directory = tempfile.TemporaryDirectory() + self.server = Server('--backend', self.backend, executable=self.worker) + + def tearDown(self): + self.server.close() + self.directory.cleanup() + + def fixture(self, name='docs', rows=600, dimensions=5, shards=4, seed=3): + rng = random.Random(seed) + keys = rng.sample(range(-10 ** 9, 10 ** 9), rows) + vectors = [[round(rng.uniform(-3, 3), 3) for _ in range(dimensions)] for _ in range(rows)] + path = os.path.join(self.directory.name, name) + keys, vectors = write_index(path, keys, vectors, shards=shards) + return path, keys, vectors + + def call(self, connection, data): + connection.sendall(data) + return receive(connection) + + def test_sharded_tree_merge_matches_exact(self): + path, keys, vectors = self.fixture(shards=5) + rng = random.Random(9) + with self.server.connect() as connection: + _, _, status, info = self.call(connection, load_frame(1, 'docs', path)) + self.assertEqual(status, 0, info) + self.assertTrue(info.startswith(b'rows 600 dimensions 5 shards 5 dtype float32')) + for request in range(2, 30): + query = [rng.uniform(-3, 3) for _ in range(5)] + k = rng.choice([1, 3, 10, 50, 700]) + _, _, status, payload = self.call(connection, search_frame(request, 'docs', query, k)) + self.assertEqual(status, 0) + route, hits = decode_hits(payload) + want = exact(keys, vectors, query, k) + self.assertEqual([key for key, _ in hits], [key for _, key in want]) + for (_, distance), (expected, _) in zip(hits, want): + self.assertAlmostEqual(distance, expected, places=4) + + def test_filters_restrict_results(self): + path, keys, vectors = self.fixture() + rng = random.Random(4) + with self.server.connect() as connection: + self.assertEqual(self.call(connection, load_frame(1, 'docs', path))[2], 0) + for request in range(2, 20): + allowed = rng.sample(keys, rng.randint(1, 80)) + # Unknown keys and duplicates are ignored; order does not matter. + sent = allowed + allowed[:3] + [10 ** 12 + request] + rng.shuffle(sent) + query = [rng.uniform(-3, 3) for _ in range(5)] + _, _, status, payload = self.call(connection, search_frame(request, 'docs', query, 10, sent)) + self.assertEqual(status, 0) + _, hits = decode_hits(payload) + want = exact(keys, vectors, query, 10, set(allowed)) + self.assertEqual([key for key, _ in hits], [key for _, key in want]) + + def test_errors(self): + path, _, _ = self.fixture() + with self.server.connect() as connection: + self.assertEqual(self.call(connection, search_frame(1, 'docs', [0] * 5, 3))[2], 3) + self.assertEqual(self.call(connection, load_frame(2, 'docs', path))[2], 0) + self.assertEqual(self.call(connection, load_frame(3, 'docs', path))[2], 3) + self.assertEqual(self.call(connection, load_frame(4, 'other', path + '-missing'))[2], 1) + self.assertEqual(self.call(connection, search_frame(5, 'docs', [0] * 4, 3))[2], 1) + self.assertEqual(self.call(connection, search_frame(6, 'docs', [0] * 5, 0))[2], 1) + # The connection survives errors and still serves stats. + self.assertEqual(self.call(connection, frame(STATS, 7))[2], 0) + + def test_corrupt_files_are_rejected(self): + path, _, _ = self.fixture() + with open(os.path.join(path, 'keys.i64'), 'r+b') as out: + out.truncate(8) + with self.server.connect() as connection: + self.assertEqual(self.call(connection, load_frame(1, 'docs', path))[2], 1) + + +@unittest.skipUnless(DSN, 'set ANN_PG_DSN for live PostgreSQL tests') +class IndexSqlTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + subprocess.run(['make', '-C', os.path.join(ROOT, 'ann'), 'ann-worker'], check=True, + stdout=subprocess.DEVNULL) + cls.directory = tempfile.TemporaryDirectory() + cls.server = Server('--backend', os.environ.get('ANN_INDEX_BACKEND', 'cpu-exact'), + executable=IndexWorkerTests.worker) + cls.table = 'ann_index_' + uuid.uuid4().hex[:12] + cls.psql(f""" + CREATE TABLE {cls.table} (id bigint PRIMARY KEY, tenant int, emb vector(4)); + SELECT setseed(0.5); + INSERT INTO {cls.table} SELECT i * 7, i % 9, + (SELECT array_agg((random() * 4)::real) FROM generate_series(1, 4) + WHERE i > 0)::vector FROM generate_series(1, 2000) i; + """) + rows = cls.psql(f"COPY (SELECT id, emb FROM {cls.table}) TO STDOUT").split('\n') + keys, vectors = [], [] + for line in filter(None, rows): + key, vector = line.split('\t') + keys.append(int(key)) + vectors.append([float(x) for x in vector.strip('[]').split(',')]) + cls.path = os.path.join(cls.directory.name, 'index') + write_index(cls.path, keys, vectors, shards=3, source=f'public.{cls.table}') + cls.name = cls.table + cls.psql(f"SELECT ann_index_load('{cls.name}', '{cls.path}')") + + @classmethod + def tearDownClass(cls): + cls.psql(f'DROP TABLE {cls.table}') + cls.server.close() + cls.directory.cleanup() + + @classmethod + def psql(cls, script, check=True): + result = subprocess.run([PSQL, DSN, '-X', '-q', '-At', '-v', 'ON_ERROR_STOP=1'], + input=f"SET ann.worker_socket = '{cls.server.path}';\n" + script, + capture_output=True, text=True, timeout=120) + if check and result.returncode != 0: + raise AssertionError(result.stderr) + return result.stdout if check else result.stdout + result.stderr + + def test_search_matches_exact_sql(self): + output = self.psql(f""" + SELECT count(*) FILTER (WHERE a IS DISTINCT FROM b) FROM ( + SELECT q.id, + ARRAY(SELECT s.id FROM ann_index_search('{self.name}', q.emb, 10) s ORDER BY s.distance, s.id) a, + ARRAY(SELECT t.id FROM {self.table} t ORDER BY t.emb <-> q.emb, t.id LIMIT 10) b + FROM {self.table} q WHERE q.id % 97 = 0) x + """) + self.assertEqual(output.strip(), '0') + + def test_hybrid_filter_matches_exact_sql(self): + output = self.psql(f""" + SELECT count(*) FILTER (WHERE a IS DISTINCT FROM b) FROM ( + SELECT q.id, + ARRAY(SELECT s.id FROM ann_index_search('{self.name}', q.emb, 5, + ARRAY(SELECT id FROM {self.table} WHERE tenant = q.tenant)) s ORDER BY s.distance, s.id) a, + ARRAY(SELECT t.id FROM {self.table} t WHERE t.tenant = q.tenant + ORDER BY t.emb <-> q.emb, t.id LIMIT 5) b + FROM {self.table} q WHERE q.id % 89 = 0) x + """) + self.assertEqual(output.strip(), '0') + + def test_join_returns_rows_and_edge_cases(self): + output = self.psql(f""" + SELECT count(*) FROM ann_index_search('{self.name}', '[1,1,1,1]', 3) s JOIN {self.table} USING (id); + SELECT count(*) FROM ann_index_search('{self.name}', '[1,1,1,1]', 3, '{{}}'); + SELECT count(*) FROM ann_index_search('{self.name}', '[1,1,1,1]', 3, ARRAY[NULL, -1]::bigint[]); + SELECT count(*) FROM ann_index_search('{self.name}', NULL, 3); + """) + self.assertEqual(output.split(), ['3', '0', '0', '0']) + + def test_errors(self): + self.assertIn('is not loaded', self.psql("SELECT * FROM ann_index_search('missing', '[1,1,1,1]', 3)", + check=False)) + self.assertIn('k must be between', self.psql( + f"SELECT * FROM ann_index_search('{self.name}', '[1,1,1,1]', 0)", check=False)) + self.assertIn('rejected', self.psql( + f"SELECT * FROM ann_index_search('{self.name}', '[1,1,1]', 3)", check=False)) + self.assertIn('unavailable', self.psql( + "SET ann.worker_socket = '/nonexistent/ann.sock'; " + f"SELECT * FROM ann_index_search('{self.name}', '[1,1,1,1]', 3)", check=False)) + + def test_stale_index_is_refused(self): + self.psql(f"INSERT INTO {self.table} VALUES (-1, 0, '[0,0,0,0]')") + try: + output = self.psql(f"SELECT ann_index_load('{self.name}_stale', '{self.path}')", check=False) + self.assertIn('ann index is stale', output) + finally: + self.psql(f'DELETE FROM {self.table} WHERE id = -1') + + def test_load_requires_superuser(self): + role = 'ann_index_' + uuid.uuid4().hex[:8] + output = self.psql(f"CREATE ROLE {role}; SET ROLE {role}; " + f"SELECT ann_index_load('x', '{self.path}');", check=False) + self.psql(f'DROP ROLE {role}') + self.assertIn('permission denied', output) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_native.py b/test/ann/test_native.py new file mode 100644 index 000000000..deb78c66a --- /dev/null +++ b/test/ann/test_native.py @@ -0,0 +1,161 @@ +"""Live tests of the native C access method against sequential SQL execution.""" + +from concurrent.futures import ThreadPoolExecutor +import os +import random +import unittest +import uuid + + +@unittest.skipUnless(os.environ.get('ANN_PG_DSN'), 'set ANN_PG_DSN for live PostgreSQL tests') +class NativeTests(unittest.TestCase): + def setUp(self): + import psycopg + from psycopg import sql + + self.psycopg = psycopg + self.sql = sql + self.dsn = os.environ['ANN_PG_DSN'] + self.db = psycopg.connect(self.dsn, autocommit=True) + self.schema = 'ann_native_' + uuid.uuid4().hex + self.db.execute(sql.SQL('CREATE SCHEMA {}').format(sql.Identifier(self.schema))) + self.table = sql.Identifier(self.schema, 'documents') + self.index = sql.Identifier(self.schema, 'documents_ann') + self.db.execute(sql.SQL('CREATE TABLE {} (id integer, tenant text, ' + 'terms tsvector, embedding vector(3))').format(self.table)) + self.db.execute(sql.SQL("INSERT INTO {} SELECT i, CASE WHEN i % 101 = 0 THEN 'rare' " + "ELSE 'common' END, to_tsvector('simple', CASE WHEN i % 7 = 0 " + "THEN 'database search' ELSE 'storage' END), " + 'ARRAY[i * 0.123, i % 13, i % 17]::vector ' + 'FROM generate_series(1, 4096) i').format(self.table)) + self.db.execute(sql.SQL('CREATE INDEX documents_ann ON {} USING ann ' + '(embedding vector_l2_ops)').format(self.table)) + self.db.execute('SET enable_seqscan = off') + + def tearDown(self): + self.db.execute(self.sql.SQL('DROP SCHEMA {} CASCADE').format(self.sql.Identifier(self.schema))) + self.db.close() + + def query(self, predicate='TRUE', limit=31): + return self.sql.SQL('SELECT id, embedding <-> (SELECT %s::vector) AS distance FROM {} WHERE {} ' + 'ORDER BY distance, id LIMIT {}').format( + self.table, self.sql.SQL(predicate), self.sql.Literal(limit)) + + def assert_oracle(self, predicate='TRUE', query='[13.4,3,5]', limit=31, db=None): + db = db or self.db + statement = self.query(predicate, limit) + db.execute('SET enable_seqscan = off') + plan = db.execute(self.sql.SQL('EXPLAIN (FORMAT JSON) {}').format(statement), (query,)).fetchone()[0] + if predicate != 'FALSE': + self.assertIn('documents_ann', str(plan)) + actual = db.execute(statement, (query,)).fetchall() + db.execute('SET enable_indexscan = off') + db.execute('SET enable_bitmapscan = off') + db.execute('SET enable_seqscan = on') + expected = db.execute(statement, (query,)).fetchall() + db.execute('SET enable_indexscan = on') + db.execute('SET enable_bitmapscan = on') + db.execute('SET enable_seqscan = off') + self.assertEqual(actual, expected) + return actual + + def test_selective_text_and_metadata(self): + for predicate in ["tenant = 'rare'", "terms @@ to_tsquery('simple', 'database & search')", + "tenant = 'rare' AND terms @@ to_tsquery('simple', 'database')", 'FALSE']: + self.assert_oracle(predicate) + rng = random.Random(42) + for _ in range(12): + self.assert_oracle(query=str([rng.random() * 500, rng.random() * 13, rng.random() * 17])) + + def test_nulls_and_spilled_sort(self): + self.db.execute(self.sql.SQL('INSERT INTO {} VALUES (5000, NULL, NULL, NULL)').format(self.table)) + self.db.execute("SET work_mem = '64kB'") + rows = self.assert_oracle(limit=10000) + self.assertEqual(rows[-1], (5000, None)) + self.assertEqual(len(self.assert_oracle(query=None, limit=10000)), 4097) + + def test_update_delete_abort_and_vacuum_reuse(self): + self.db.execute(self.sql.SQL("UPDATE {} SET tenant = 'rare' WHERE id % 29 = 0").format(self.table)) + self.db.execute(self.sql.SQL("UPDATE {} SET embedding = '[0,0,0]' WHERE id % 31 = 0").format(self.table)) + before = self.db.execute('SELECT pg_relation_size(%s::regclass)', + (self.schema + '.documents_ann',)).fetchone()[0] + self.db.execute(self.sql.SQL('DELETE FROM {} WHERE id % 2 = 0').format(self.table)) + self.db.execute(self.sql.SQL('VACUUM {}').format(self.table)) + self.db.execute(self.sql.SQL('INSERT INTO {} SELECT i, NULL, NULL, ARRAY[i,0,0]::vector ' + 'FROM generate_series(6000, 6999) i').format(self.table)) + after = self.db.execute('SELECT pg_relation_size(%s::regclass)', + (self.schema + '.documents_ann',)).fetchone()[0] + self.assertLessEqual(after, before + 8192) + self.db.execute('BEGIN') + self.db.execute(self.sql.SQL("INSERT INTO {} VALUES (9000, NULL, NULL, '[0,0,0]')").format(self.table)) + self.assert_oracle(query='[0,0,0]') + self.db.execute('ROLLBACK') + self.assert_oracle(query='[0,0,0]') + self.db.execute(self.sql.SQL('REINDEX INDEX {}').format(self.index)) + self.assert_oracle("tenant = 'rare'") + + def test_repeatable_read_and_uncommitted_insert(self): + with self.psycopg.connect(self.dsn, autocommit=True) as reader: + reader.execute('BEGIN ISOLATION LEVEL REPEATABLE READ') + original = self.assert_oracle(db=reader) + self.db.execute(self.sql.SQL("UPDATE {} SET embedding = '[13.4,3,5]' WHERE id = 1").format(self.table)) + self.db.execute(self.sql.SQL('DELETE FROM {} WHERE id = 2').format(self.table)) + self.db.execute(self.sql.SQL('VACUUM {}').format(self.table)) + self.assertEqual(self.assert_oracle(db=reader), original) + self.db.execute('BEGIN') + self.db.execute(self.sql.SQL("INSERT INTO {} VALUES (9000, NULL, NULL, '[13.4,3,5]')").format(self.table)) + self.assertEqual(self.assert_oracle(db=reader), original) + self.db.execute('COMMIT') + reader.execute('COMMIT') + self.assert_oracle(db=reader) + + def test_concurrent_writers(self): + statement = self.sql.SQL('INSERT INTO {} SELECT i, NULL, NULL, ARRAY[i,1,2]::vector ' + 'FROM generate_series(%s::integer, %s::integer) i').format(self.table) + + def write(start): + with self.psycopg.connect(self.dsn) as db: + db.execute(statement, (start, start + 299)) + + with ThreadPoolExecutor(max_workers=4) as pool: + list(pool.map(write, [10000, 11000, 12000, 13000])) + self.assertEqual(self.db.execute(self.sql.SQL('SELECT count(*) FROM {}').format(self.table)).fetchone()[0], 5296) + self.assert_oracle(query='[11001,1,2]') + + def test_existing_hot_chains_and_concurrent_build(self): + self.db.execute(self.sql.SQL('DROP INDEX {}').format(self.index)) + # Leave heap-page room so the unindexed updates can form HOT chains. + self.db.execute(self.sql.SQL('ALTER TABLE {} SET (fillfactor = 50)').format(self.table)) + self.db.execute(self.sql.SQL('VACUUM FULL {}').format(self.table)) + self.db.execute(self.sql.SQL("UPDATE {} SET embedding = '[13.4,3,5]' " + 'WHERE id % 17 = 0').format(self.table)) + self.db.execute(self.sql.SQL('CREATE INDEX CONCURRENTLY documents_ann ON {} ' + 'USING ann (embedding vector_l2_ops)').format(self.table)) + self.assert_oracle() + self.db.execute(self.sql.SQL('VACUUM FULL {}').format(self.table)) + self.assert_oracle() + + def test_maximum_dimension(self): + name = self.sql.Identifier(self.schema, 'wide_vectors') + self.db.execute(self.sql.SQL('CREATE TABLE {} (v vector(2000))').format(name)) + vector = str([1] * 2000) + self.db.execute(self.sql.SQL('INSERT INTO {} VALUES (%s::vector)').format(name), (vector,)) + self.db.execute(self.sql.SQL('CREATE INDEX ON {} USING ann (v vector_l2_ops)').format(name)) + self.db.execute(self.sql.SQL('INSERT INTO {} VALUES (%s::vector)').format(name), (vector,)) + rows = self.db.execute(self.sql.SQL('SELECT v <-> %s::vector FROM {} ' + 'ORDER BY v <-> %s::vector').format(name), (vector, vector)).fetchall() + self.assertEqual(rows, [(0.0,), (0.0,)]) + + def test_unlogged_and_temporary(self): + for kind in ['UNLOGGED', 'TEMPORARY']: + name = self.sql.Identifier('ann_lifecycle_' + uuid.uuid4().hex) + self.db.execute(self.sql.SQL('CREATE {} TABLE {} (embedding vector(3))').format(self.sql.SQL(kind), name)) + self.db.execute(self.sql.SQL('CREATE INDEX ON {} USING ann (embedding vector_l2_ops)').format(name)) + self.db.execute(self.sql.SQL("INSERT INTO {} VALUES ('[1,2,3]'), (NULL)").format(name)) + rows = self.db.execute(self.sql.SQL("SELECT embedding::text FROM {} ORDER BY embedding <-> '[0,0,0]'").format(name)).fetchall() + self.assertEqual(rows, [('[1,2,3]',), (None,)]) + self.db.execute(self.sql.SQL('DROP TABLE {}').format(name)) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_pipeline.py b/test/ann/test_pipeline.py new file mode 100644 index 000000000..54ecc2344 --- /dev/null +++ b/test/ann/test_pipeline.py @@ -0,0 +1,165 @@ +"""Live worker-pipeline tests for the filtered CustomScan, driven through psql. + +Requires ANN_PG_DSN for a superuser on the same host as this checkout, since +the server connects to the worker's Unix socket. ANN_PSQL overrides psql. +""" + +import os +import re +import subprocess +import sys +import threading +import time +import unittest +import uuid + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from test_server import ROOT, Server # noqa: E402 + +DSN = os.environ.get('ANN_PG_DSN') +PSQL = os.environ.get('ANN_PSQL', 'psql') + +SESSION = """ +LOAD 'vector'; +SET ann.enable_filtered_search = on; +SET max_parallel_workers_per_gather = 0; +SET ann.worker_chunk_rows = 256; +""" + +# Compare the worker route against local scoring for many queries and bounds. +COMPARE = """ +DO $$ +DECLARE q int; lim int; a int[]; b int[]; v vector; bad int := 0; +BEGIN + FOR q IN 1..25 LOOP + SELECT emb INTO v FROM {table} WHERE id = q * 37; + FOREACH lim IN ARRAY ARRAY[1, 7, 60] LOOP + PERFORM set_config('ann.worker_socket', '{socket}', true); + EXECUTE 'SELECT array_agg(id) FROM (SELECT id FROM {table} WHERE tenant IN (1, 2) OR id % 211 = 0 ' + 'ORDER BY emb <-> $1, id LIMIT $2 OFFSET $3) s' INTO a USING v, lim, q % 3; + PERFORM set_config('ann.worker_socket', '', true); + EXECUTE 'SELECT array_agg(id) FROM (SELECT id FROM {table} WHERE tenant IN (1, 2) OR id % 211 = 0 ' + 'ORDER BY emb <-> $1, id LIMIT $2 OFFSET $3) s' INTO b USING v, lim, q % 3; + IF a IS DISTINCT FROM b THEN bad := bad + 1; END IF; + END LOOP; + END LOOP; + RAISE NOTICE 'mismatches=%', bad; +END $$; +""" + +QUERY = ("SELECT array_agg(id) FROM (SELECT id FROM {table} WHERE tenant IN (1, 2) " + "ORDER BY emb <-> (ARRAY(SELECT 1.5::real FROM generate_series(1, 8)))::vector(8), id LIMIT 12) s") +EXPLAIN = ("EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT id FROM {table} WHERE tenant IN (1, 2) " + "ORDER BY emb <-> '[1.5,1.5,1.5,1.5,1.5,1.5,1.5,1.5]', id LIMIT 12") + + +@unittest.skipUnless(DSN, 'set ANN_PG_DSN for live PostgreSQL tests') +class PipelineTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + subprocess.run(['make', '-C', os.path.join(ROOT, 'ann'), 'ann-worker'], check=True, + stdout=subprocess.DEVNULL) + cls.table = 'ann_pipeline_' + uuid.uuid4().hex[:12] + cls.psql(f""" + CREATE EXTENSION IF NOT EXISTS vector; + CREATE TABLE {cls.table} (id int, tenant int, emb vector(8)); + SELECT setseed(0.25); + INSERT INTO {cls.table} SELECT i, i % 5, CASE WHEN i % 97 = 0 THEN NULL ELSE + (SELECT array_agg(round((random() * 3)::numeric, 1)::real) FROM generate_series(1, 8) + WHERE i > 0)::vector END FROM generate_series(1, 6000) i; + CREATE INDEX ON {cls.table} USING ann (emb vector_l2_ops); + ANALYZE {cls.table}; + """) + + @classmethod + def tearDownClass(cls): + cls.psql(f'DROP TABLE {cls.table}') + + @staticmethod + def psql(script, settings=''): + result = subprocess.run([PSQL, DSN, '-X', '-q', '-At', '-v', 'ON_ERROR_STOP=1'], + input=SESSION + settings + script, capture_output=True, text=True, + timeout=120) + if result.returncode != 0: + raise AssertionError(result.stderr) + return result.stdout + result.stderr + + def setUp(self): + self.server = None + + def tearDown(self): + if self.server: + self.server.close() + + def start(self, *args, env=None): + self.server = Server('--backend', 'mock-gpu', *args, env=env) + return f"SET ann.worker_socket = '{self.server.path}';\n" + + def explain(self, settings): + output = self.psql(EXPLAIN.format(table=self.table) + ';', settings) + return {k: int(v) for k, v in re.findall(r'(Worker Chunks|Local Chunks|Worker Candidates|' + r'Distance Evaluations): (\d+)', output)} + + def local_result(self): + return self.psql(QUERY.format(table=self.table) + ';').strip() + + def test_worker_route_matches_local_scoring(self): + self.start() + output = self.psql(COMPARE.format(table=self.table, socket=self.server.path)) + self.assertIn('mismatches=0', output) + + def test_explain_reports_worker_chunks(self): + stats = self.explain(self.start()) + self.assertGreater(stats['Worker Chunks'], 1) + # Backends re-score only returned candidates, plus any chunk shed to + # local scoring while the pipeline was full and NULL-vector rows. + self.assertLessEqual(stats['Distance Evaluations'], + stats['Worker Candidates'] + 256 * stats['Local Chunks']) + self.assertLess(stats['Distance Evaluations'], 2400) + + def test_inline_transport_without_shared_memory(self): + settings = self.start() + 'SET ann.worker_shared_memory_mb = 0;\n' + self.assertEqual(self.psql(QUERY.format(table=self.table) + ';', settings).strip(), + self.local_result()) + self.assertGreater(self.explain(settings)['Worker Chunks'], 1) + + def test_overloaded_worker_falls_back_per_chunk(self): + settings = self.start('--max-pending-mb', '0') + self.assertEqual(self.psql(QUERY.format(table=self.table) + ';', settings).strip(), + self.local_result()) + stats = self.explain(settings) + self.assertEqual(stats['Worker Chunks'], 0) + self.assertGreater(stats['Local Chunks'], 1) + + def test_worker_exit_mid_scan_keeps_exact_results(self): + expected = self.local_result() + settings = self.start('--lanes', '1', env={'ANN_MOCK_GPU_LAUNCH_US': '300000'}) + process = subprocess.Popen([PSQL, DSN, '-X', '-q', '-At', '-v', 'ON_ERROR_STOP=1'], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, + text=True) + replies = [] + thread = threading.Thread(target=lambda: replies.append(process.communicate( + SESSION + settings + QUERY.format(table=self.table) + ';\n' + + EXPLAIN.format(table=self.table) + ';\n', timeout=120))) + thread.start() + time.sleep(0.5) + self.server.process.kill() + thread.join() + output, errors = replies[0] + self.assertEqual(process.returncode, 0, errors) + self.assertEqual(output.splitlines()[0], expected) + # Whichever statement the exit interrupts scores its remaining chunks locally. + local = re.search(r'Local Chunks: (\d+)', output) + self.assertTrue(local is None or int(local.group(1)) > 0, output) + + def test_timeout_falls_back(self): + settings = self.start('--lanes', '1', env={'ANN_MOCK_GPU_LAUNCH_US': '2000000'}) + settings += 'SET ann.worker_timeout_ms = 200;\n' + started = time.monotonic() + self.assertEqual(self.psql(QUERY.format(table=self.table) + ';', settings).strip(), + self.local_result()) + self.assertLess(time.monotonic() - started, 10) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_postgres.py b/test/ann/test_postgres.py new file mode 100644 index 000000000..8653abdb8 --- /dev/null +++ b/test/ann/test_postgres.py @@ -0,0 +1,127 @@ +"""Opt-in live PostgreSQL tests, isolated in a uniquely named test schema.""" + +import os +from pathlib import Path +import sys +import unittest +import uuid + +sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'test/ann')) + + +@unittest.skipUnless(os.environ.get('ANN_PG_DSN'), 'set ANN_PG_DSN for live PostgreSQL tests') +class PostgresTests(unittest.TestCase): + def setUp(self): + import psycopg + from psycopg import sql + + self.psycopg = psycopg + self.sql = sql + self.dsn = os.environ['ANN_PG_DSN'] + self.schema = 'ann_test_' + uuid.uuid4().hex + self.db = psycopg.connect(self.dsn, autocommit=True) + self.db.execute('CREATE EXTENSION IF NOT EXISTS vector') + self.db.execute(sql.SQL('CREATE SCHEMA {}').format(sql.Identifier(self.schema))) + self.relation = sql.Identifier(self.schema, 'documents') + self.db.execute(sql.SQL('CREATE TABLE {} (id integer PRIMARY KEY, tenant text, ' + 'tags text[], terms tsvector, embedding vector(3))').format(self.relation)) + for i in range(64): + self.db.execute(sql.SQL('INSERT INTO {} VALUES (%s, %s, %s, ' + "to_tsvector('simple', %s), %s::vector)").format(self.relation), + (i, 'even' if i % 2 == 0 else 'odd', + ['database'] if i % 3 == 0 else ['storage'], + 'database query' if i % 3 == 0 else 'storage', f'[{i},1,2]')) + self.db.execute(sql.SQL('CREATE INDEX ON {} USING gin (terms)').format(self.relation)) + + def tearDown(self): + self.db.execute(self.sql.SQL('DROP SCHEMA {} CASCADE').format(self.sql.Identifier(self.schema))) + self.db.close() + + def test_combined_predicate_and_shared_query_batch(self): + from support.postgres import FrozenIndex + + predicate = self.sql.SQL("tenant = %s AND tags @> %s AND terms @@ to_tsquery('simple', %s)") + params = ('even', ['database'], 'database & query') + with FrozenIndex(self.dsn, 'documents', schema=self.schema) as index: + for query, row in zip(['[0,1,2]', '[24,1,2]'], + index.search([[0, 1, 2], [24, 1, 2]], 5, predicate, params)): + oracle = self.db.execute(self.sql.SQL( + 'SELECT id, embedding <-> %s::vector AS distance FROM {} WHERE {} ' + 'ORDER BY distance, id LIMIT 5').format(self.relation, predicate), + (query,) + params).fetchall() + self.assertEqual([v[0] for v in row], [v[0] for v in oracle]) + for found, expected in zip(row, oracle): + self.assertAlmostEqual(found[1], expected[1], places=5) + self.assertEqual(index.search([[0, 1, 2]], 5, self.sql.SQL('FALSE')), [[]]) + self.assertEqual(index.last_search['eligible_rows'], 0) + self.assertEqual(index.last_search['query_vectors'], 1) + self.assertGreaterEqual(index.last_search['total_seconds'], + index.last_search['worker_roundtrip_seconds']) + with self.assertRaises(TypeError): + index.search([[0, 1, 2]], 1, 'TRUE') + with self.assertRaises(RuntimeError): + index.search([[0, 1, 2]], 1) + + def test_freeze_blocks_all_external_mutations(self): + from support.postgres import FrozenIndex + + with FrozenIndex(self.dsn, 'documents', schema=self.schema): + self.db.execute("SET lock_timeout = '100ms'") + for statement in [ + 'UPDATE {} SET tenant = \'changed\' WHERE id = 0', + 'INSERT INTO {0} SELECT 1000, tenant, tags, terms, embedding FROM {0} WHERE id = 0', + 'DELETE FROM {} WHERE id = 0', 'TRUNCATE {}', + 'ALTER TABLE {} ADD COLUMN extra integer']: + with self.assertRaises(self.psycopg.errors.LockNotAvailable): + self.db.execute(self.sql.SQL(statement).format(self.relation)) + self.db.execute(self.sql.SQL('UPDATE {} SET tenant = %s WHERE id = 0').format(self.relation), + ('changed',)) + + def test_budget_cleanup_releases_lock(self): + from support.postgres import FrozenIndex + + with self.assertRaises(ValueError): + with FrozenIndex(self.dsn, 'documents', schema=self.schema, max_rows=1): + self.fail('oversized corpus accepted') + self.db.execute("SET lock_timeout = '100ms'") + self.db.execute(self.sql.SQL('DELETE FROM {} WHERE id = 0').format(self.relation)) + + def test_private_freeze_transaction_is_read_only(self): + from support.postgres import FrozenIndex + + with FrozenIndex(self.dsn, 'documents', schema=self.schema) as index: + self.assertEqual(index._connection.execute('SHOW transaction_read_only').fetchone(), ('on',)) + with self.assertRaises(self.psycopg.errors.ReadOnlySqlTransaction): + index._connection.execute(self.sql.SQL( + 'DELETE FROM {} WHERE id = 0').format(self.relation)) + + def test_freeze_loss_during_external_search_discards_results(self): + from support.postgres import FrozenIndex + + with FrozenIndex(self.dsn, 'documents', schema=self.schema) as index: + pid = index._connection.info.backend_pid + search = index._worker.search + + def lose_session(*args, **kwargs): + result = search(*args, **kwargs) + self.db.execute('SELECT pg_terminate_backend(%s)', (pid,)) + return result + + index._worker.search = lose_session + with self.assertRaises(self.psycopg.Error): + index.search([[0, 1, 2]], 5) + + def test_predicate_timeout_releases_freeze(self): + from support.postgres import FrozenIndex + + with FrozenIndex(self.dsn, 'documents', schema=self.schema, statement_timeout_ms=100) as index: + with self.assertRaises(self.psycopg.errors.QueryCanceled): + index.search([[0, 1, 2]], 5, self.sql.SQL('pg_sleep(1) IS NULL')) + self.assertIsNone(index._connection) + self.assertIsNone(index.last_search) + self.db.execute("SET lock_timeout = '100ms'") + self.db.execute(self.sql.SQL('DELETE FROM {} WHERE id = 0').format(self.relation)) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_remote.py b/test/ann/test_remote.py new file mode 100644 index 000000000..171358817 --- /dev/null +++ b/test/ann/test_remote.py @@ -0,0 +1,257 @@ +"""Networked GPU nodes: TCP transport, auth, routing, failover, and heartbeats. + +Needs ANN_PG_DSN for a superuser on this host (the test starts local +networked workers); ANN_PSQL overrides psql. Uses the CPU worker, so it runs +without a GPU; ANN_INDEX_WORKER/ANN_INDEX_BACKEND select the cuVS build. +""" + +import os +import socket +import struct +import subprocess +import sys +import tempfile +import time +import unittest +import uuid + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from test_index import IndexWorkerTests, write_index # noqa: E402 +from test_server import ROOT, Server, frame, receive # noqa: E402 + +DSN = os.environ.get('ANN_PG_DSN') +PSQL = os.environ.get('ANN_PSQL', 'psql') +TOKEN = 'test-token-' + uuid.uuid4().hex + + +def free_port(): + with socket.socket() as s: + s.bind(('127.0.0.1', 0)) + return s.getsockname()[1] + + +class Node: + def __init__(self, directory): + self.port = free_port() + token = os.path.join(directory, f'token-{self.port}') + with open(token, 'w') as out: + out.write(TOKEN + '\n') + self.server = Server('--backend', IndexWorkerTests.backend, '--listen-tcp', f'127.0.0.1:{self.port}', + '--auth-token-file', token, executable=IndexWorkerTests.worker) + deadline = time.monotonic() + 5 + while True: + try: + socket.create_connection(('127.0.0.1', self.port), timeout=1).close() + break + except OSError: + if time.monotonic() > deadline: + raise + time.sleep(0.05) + + @property + def address(self): + return f'127.0.0.1:{self.port}' + + def kill(self): + self.server.process.kill() + self.server.process.wait() + + def close(self): + if self.server.process.poll() is None: + self.server.close() + + +class WorkerAuthTests(unittest.TestCase): + def test_unauthenticated_connections_are_closed(self): + with tempfile.TemporaryDirectory() as directory: + node = Node(directory) + try: + with socket.create_connection(('127.0.0.1', node.port)) as connection: + connection.sendall(frame(8, 1)) + self.assertEqual(connection.recv(32), b'') + with socket.create_connection(('127.0.0.1', node.port)) as connection: + connection.sendall(frame(13, 1, b'wrong-token-value')) + self.assertEqual(connection.recv(32), b'') + with socket.create_connection(('127.0.0.1', node.port)) as connection: + connection.sendall(frame(13, 1, TOKEN.encode())) + self.assertEqual(receive(connection)[2], 0) + # Shared memory is refused over the network. + connection.sendall(frame(9, 2, b'/pgann.x')) + self.assertEqual(receive(connection)[2], 2) + connection.sendall(frame(5, 3)) + self.assertEqual(receive(connection)[2], 0) + finally: + node.close() + + def test_packed_filter_decoding(self): + with tempfile.TemporaryDirectory() as directory: + node = Node(directory) + path = os.path.join(directory, 'index') + keys, vectors = write_index(path, [-5, 3, 4, 300, 70000], [[1], [2], [3], [4], [5]]) + try: + with socket.create_connection(('127.0.0.1', node.port)) as connection: + connection.sendall(frame(13, 1, TOKEN.encode())) + receive(connection) + connection.sendall(frame(11, 2, struct.pack('<I', 1) + b'i' + path.encode())) + self.assertEqual(receive(connection)[2], 0) + + def varint(value): + out = b'' + while True: + byte = value & 0x7f + value >>= 7 + out += bytes([byte | (0x80 if value else 0)]) + if not value: + return out + allowed = [-5, 4, 70000] + # zigzag(-5) = 9, then gaps 9 (to 4) and 69996 (to 70000). + packed = varint(9) + varint(9) + varint(69996) + payload = struct.pack('<I', 1) + b'i' + struct.pack('<IIII', 1, 3, len(allowed), len(packed)) + payload += struct.pack('<f', 10.0) + packed + connection.sendall(frame(14, 3, payload)) + _, _, status, data = receive(connection) + self.assertEqual(status, 0) + count = struct.unpack_from('<I', data)[0] + hits = [struct.unpack_from('<qf', data, 8 + i * 12)[0] for i in range(count)] + self.assertEqual(hits, [70000, 4, -5]) # values 5, 3, 1 from query 10 + # A zero gap (duplicate key) is malformed. + bad = varint(10) + varint(0) + payload = struct.pack('<I', 1) + b'i' + struct.pack('<IIII', 1, 3, 2, len(bad)) + payload += struct.pack('<f', 1.0) + bad + connection.sendall(frame(14, 4, payload)) + self.assertEqual(receive(connection)[2], 1) + finally: + node.close() + + +@unittest.skipUnless(DSN, 'set ANN_PG_DSN for live PostgreSQL tests') +class RemoteSqlTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + subprocess.run(['make', '-C', os.path.join(ROOT, 'ann'), 'ann-worker'], check=True, + stdout=subprocess.DEVNULL) + cls.directory = tempfile.TemporaryDirectory() + cls.nodes = [Node(cls.directory.name), Node(cls.directory.name)] + cls.table = 'ann_remote_' + uuid.uuid4().hex[:12] + cls.psql(f""" + CREATE TABLE {cls.table} (id bigint PRIMARY KEY, tenant int, emb vector(4)); + SELECT setseed(0.3); + INSERT INTO {cls.table} SELECT i * 5, i % 20, + (SELECT array_agg((random() * 4)::real) FROM generate_series(1, 4) WHERE i > 0)::vector + FROM generate_series(1, 3000) i; + """) + rows = cls.psql(f'COPY (SELECT id, emb FROM {cls.table}) TO STDOUT').split('\n') + keys, vectors = [], [] + for line in filter(None, rows): + key, vector = line.split('\t') + keys.append(int(key)) + vectors.append([float(x) for x in vector.strip('[]').split(',')]) + cls.path = os.path.join(cls.directory.name, 'index') + write_index(cls.path, keys, vectors, shards=2, source=f'public.{cls.table}') + cls.name = cls.table + # Loads go to every node. + cls.psql(f"SELECT ann_index_load('{cls.name}', '{cls.path}')") + + @classmethod + def tearDownClass(cls): + cls.psql(f"DROP TABLE {cls.table}; DELETE FROM ann_index_catalog WHERE name LIKE 'ann_remote_%'") + for node in cls.nodes: + node.close() + cls.directory.cleanup() + + @classmethod + def settings(cls, nodes=None): + nodes = nodes or cls.nodes + return (f"SET ann.worker_address = '{','.join(n.address for n in nodes)}';\n" + f"SET ann.worker_auth_token = '{TOKEN}';\n") + + @classmethod + def psql(cls, script, check=True, nodes=None): + result = subprocess.run([PSQL, DSN, '-X', '-q', '-At', '-v', 'ON_ERROR_STOP=1'], + input=cls.settings(nodes) + script, capture_output=True, text=True, timeout=120) + if check and result.returncode != 0: + raise AssertionError(result.stderr) + return result.stdout if check else result.stdout + result.stderr + + def mismatches(self, search, exact, where='q.id % 211 = 0'): + return self.psql(f""" + SELECT count(*) FILTER (WHERE a IS DISTINCT FROM b) FROM ( + SELECT ARRAY(SELECT s.id FROM {search} s ORDER BY s.distance, s.id) a, + ARRAY(SELECT t.id FROM {self.table} t {exact} ORDER BY t.emb <-> q.emb, t.id LIMIT 7) b + FROM {self.table} q WHERE {where}) x""").strip() + + def test_ann_over_tcp(self): + self.assertEqual(self.mismatches(f"ann_index_search('{self.name}', q.emb, 7)", ''), '0') + + def test_hybrid_routes_agree_with_sql(self): + # 150 eligible rows per tenant: exact on this host at 1000, on the GPU node at 1. + for threshold in (1000, 1): + output = self.psql(f"""SET ann.gpu_min_eligible = {threshold}; + SELECT count(*) FILTER (WHERE a IS DISTINCT FROM b) FROM ( + SELECT ARRAY(SELECT s.id FROM ann_hybrid_search('{self.name}', q.emb, 7, + format('tenant = %s', q.tenant)) s ORDER BY s.distance, s.id) a, + ARRAY(SELECT t.id FROM {self.table} t WHERE t.tenant = q.tenant + ORDER BY t.emb <-> q.emb, t.id LIMIT 7) b + FROM {self.table} q WHERE q.id % 211 = 0) x""").strip() + self.assertEqual(output, '0', threshold) + # The filter is one read-only statement. + self.assertIn('ERROR', self.psql( + f"SELECT * FROM ann_hybrid_search('{self.name}', '[1,1,1,1]', 3, 'true); DELETE FROM {self.table}; --')", + check=False)) + self.assertEqual(self.psql(f'SELECT count(*) FROM {self.table}').strip(), '3000') + + def test_key_filter_routes_agree_with_sql(self): + for threshold in (1000, 1): + output = self.psql(f"""SET ann.gpu_min_eligible = {threshold}; + SELECT count(*) FILTER (WHERE a IS DISTINCT FROM b) FROM ( + SELECT ARRAY(SELECT s.id FROM ann_index_search('{self.name}', q.emb, 7, + ARRAY(SELECT id FROM {self.table} WHERE tenant = q.tenant)) s ORDER BY s.distance, s.id) a, + ARRAY(SELECT t.id FROM {self.table} t WHERE t.tenant = q.tenant + ORDER BY t.emb <-> q.emb, t.id LIMIT 7) b + FROM {self.table} q WHERE q.id % 211 = 0) x""").strip() + self.assertEqual(output, '0', threshold) + + def test_attach_and_catalog(self): + self.assertEqual(self.psql(f"SELECT ann_index_attach('{self.name}')").strip(), '3000') + self.assertEqual(self.psql(f"SELECT key_column || ',' || vector_column FROM ann_index_catalog " + f"WHERE name = '{self.name}'").strip(), 'id,emb') + self.assertIn('not in this database', self.psql( + "SELECT * FROM ann_hybrid_search('missing', '[1,1,1,1]', 3, 'true')", check=False)) + + def test_wrong_token_is_unavailable(self): + output = self.psql(f"SET ann.worker_auth_token = 'nope';\n" + f"SELECT * FROM ann_index_search('{self.name}', '[1,1,1,1]', 3)", check=False) + self.assertIn('unavailable', output) + + def test_failover_and_heartbeat(self): + spare = Node(self.directory.name) + try: + self.psql(f"SELECT ann_index_load('{self.name}_spare', '{self.path}')", nodes=[spare]) + self.psql(f"SELECT ann_index_load('{self.name}_spare', '{self.path}')", nodes=[self.nodes[0]]) + query = f"SELECT count(*) FROM ann_index_search('{self.name}_spare', '[1,1,1,1]', 3);\n" + process = subprocess.Popen([PSQL, DSN, '-X', '-q', '-At', '-v', 'ON_ERROR_STOP=1'], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, + text=True) + # Connect to the spare only, then list a second node and kill the spare while idle. + process.stdin.write(self.settings([spare]) + 'SET ann.worker_heartbeat_ms = 200;\n' + query) + process.stdin.write(f"SET ann.worker_address = '{spare.address},{self.nodes[0].address}';\n") + process.stdin.flush() + time.sleep(1) + spare.kill() + time.sleep(0.5) + started = time.monotonic() + process.stdin.write(query) + output, errors = process.communicate(timeout=60) + self.assertEqual(process.returncode, 0, errors) + self.assertEqual(output.split(), ['3', '3']) + self.assertLess(time.monotonic() - started, 5) + finally: + spare.close() + + def test_all_nodes_down_is_unavailable(self): + output = self.psql("SET ann.worker_address = '127.0.0.1:1';\n" + f"SELECT * FROM ann_index_search('{self.name}', '[1,1,1,1]', 3)", check=False) + self.assertIn('unavailable', output) + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_server.py b/test/ann/test_server.py new file mode 100644 index 000000000..ccaf4c973 --- /dev/null +++ b/test/ann/test_server.py @@ -0,0 +1,183 @@ +"""Pipelined Unix-socket server tests for gathered exact search.""" + +import json +import math +import os +import random +import socket +import struct +import subprocess +import tempfile +import time +import unittest + +ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +WORKER = os.path.join(ROOT, 'ann', 'ann-worker') +GATHERED = 7 +STATS = 8 + + +class Server: + def __init__(self, *args, env=None, executable=WORKER): + self.directory = tempfile.TemporaryDirectory() + self.path = os.path.join(self.directory.name, 'ann.sock') + self.process = subprocess.Popen([executable, '--listen', self.path, *args], + stderr=subprocess.PIPE, env={**os.environ, **(env or {})}) + deadline = time.monotonic() + 5 + while not os.path.exists(self.path): + if time.monotonic() > deadline or self.process.poll() is not None: + raise RuntimeError('worker did not start') + time.sleep(0.01) + + def connect(self): + connection = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + connection.connect(self.path) + return connection + + def close(self): + self.process.terminate() + self.process.wait(5) + self.process.stderr.close() + self.directory.cleanup() + + +def frame(operation, request, payload=b''): + return b'PVAN' + struct.pack('<HHQQII', 1, operation, request, 0, len(payload), 0) + payload + + +def gathered(query, vectors, k, metric=0): + values = [x for row in [query] + vectors for x in row] + return struct.pack('<IIII', len(vectors), len(query), k, metric) + struct.pack('<%df' % len(values), *values) + + +def receive(connection): + def exact(size): + data = b'' + while len(data) < size: + chunk = connection.recv(size - len(data)) + if not chunk: + raise EOFError + data += chunk + return data + + header = exact(32) + assert header[:4] == b'PVAN' + _, operation, request, _, size, status = struct.unpack('<HHQQII', header[4:]) + return operation, request, status, exact(size) + + +def decode(payload): + count = struct.unpack_from('<I', payload)[0] + return [struct.unpack_from('<Id', payload, 4 + i * 12) for i in range(count)] + + +def f32(value): + return struct.unpack('<f', struct.pack('<f', value))[0] + + +def expected(query, vectors, k): + distances = sorted((math.sqrt(sum((f32(a) - f32(b)) ** 2 for a, b in zip(row, query))), i) + for i, row in enumerate(vectors)) + if k >= len(distances): + return [i for _, i in distances] + threshold = distances[k - 1][0] + threshold += 1e-3 * abs(threshold) + 1e-6 + return [i for d, i in distances if d <= threshold] + + +class ServerTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + subprocess.run(['make', '-C', os.path.join(ROOT, 'ann'), 'ann-worker'], check=True, + stdout=subprocess.DEVNULL) + + def setUp(self): + self.server = Server('--backend', 'mock-gpu', '--lanes', '1', env={'ANN_MOCK_GPU_LAUNCH_US': '2000'}) + + def tearDown(self): + self.server.close() + + def stats(self, connection): + connection.sendall(frame(STATS, 1)) + _, _, status, payload = receive(connection) + self.assertEqual(status, 0) + return json.loads(payload) + + def test_pipelined_requests_match_exact_results(self): + rng = random.Random(7) + connections = [self.server.connect() for _ in range(3)] + sent = {} + for request in range(1, 31): + dimensions = 8 + vectors = [[rng.uniform(-1, 1) for _ in range(dimensions)] for _ in range(rng.randint(1, 300))] + query = [rng.uniform(-1, 1) for _ in range(dimensions)] + k = rng.randint(1, 20) + connection = connections[request % 3] + connection.sendall(frame(GATHERED, request, gathered(query, vectors, k))) + sent[request] = (connection, query, vectors, k) + for connection in connections: + mine = {r for r, value in sent.items() if value[0] is connection} + while mine: + operation, request, status, payload = receive(connection) + self.assertEqual((operation, status), (GATHERED, 0)) + self.assertIn(request, mine) + mine.remove(request) + _, query, vectors, k = sent[request] + ids = [i for i, _ in decode(payload)] + self.assertEqual(ids, expected(query, vectors, k)) + stats = self.stats(connections[0]) + self.assertEqual(stats['jobs'], 30) + # One lane with a 2 ms launch forces later jobs to share submissions. + self.assertLess(stats['batches'], 30) + self.assertGreater(stats['max_batch'], 1) + for connection in connections: + connection.close() + + def test_ties_at_the_boundary_are_retained(self): + with self.server.connect() as connection: + vectors = [[1, 0], [0, 1], [-1, 0], [0, -1], [3, 3]] + connection.sendall(frame(GATHERED, 9, gathered([0, 0], vectors, 1))) + _, request, status, payload = receive(connection) + self.assertEqual((request, status), (9, 0)) + self.assertEqual(sorted(i for i, _ in decode(payload)), [0, 1, 2, 3]) + + def test_invalid_frames_fail_without_closing(self): + with self.server.connect() as connection: + connection.sendall(frame(GATHERED, 1, struct.pack('<IIII', 2, 3, 1, 0) + b'\0' * 4)) + connection.sendall(frame(GATHERED, 2, gathered([0, 0], [[float('nan'), 0]], 1))) + connection.sendall(frame(4, 3)) + connection.sendall(frame(GATHERED, 4, gathered([0], [[2], [1]], 1))) + replies = {} + for _ in range(4): + _, request, status, payload = receive(connection) + replies[request] = (status, payload) + self.assertEqual(replies[1][0], 1) + self.assertEqual(replies[2][0], 1) + self.assertEqual(replies[3][0], 2) + self.assertEqual(replies[4][0], 0) + self.assertEqual(decode(replies[4][1]), [(1, 1.0)]) + + def test_admission_rejects_over_budget(self): + self.server.close() + self.server = Server('--max-pending-mb', '1') + with self.server.connect() as connection: + vectors = [[0.0] * 512 for _ in range(1024)] + connection.sendall(frame(GATHERED, 1, gathered([0.0] * 512, vectors, 1))) + _, _, status, _ = receive(connection) + self.assertEqual(status, 7) + self.assertEqual(self.stats(connection)['rejected'], 1) + + def test_disconnect_with_pending_work(self): + connection = self.server.connect() + vectors = [[float(i)] * 4 for i in range(2000)] + for request in range(10): + connection.sendall(frame(GATHERED, request, gathered([0.0] * 4, vectors, 5))) + connection.close() + with self.server.connect() as other: + other.sendall(frame(GATHERED, 1, gathered([0.0], [[1.0]], 1))) + self.assertEqual(receive(other)[2], 0) + self.assertIsNone(self.server.process.poll()) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/ann/test_worker.py b/test/ann/test_worker.py new file mode 100644 index 000000000..6bc4d6c0d --- /dev/null +++ b/test/ann/test_worker.py @@ -0,0 +1,162 @@ +import math +from pathlib import Path +import random +import struct +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import patch + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT / 'test/ann')) +from support.client import HEADER, Worker, WorkerError + + +class WorkerTests(unittest.TestCase): + def setUp(self): + self.worker = Worker() + + def tearDown(self): + self.worker.close() + + def assertStatus(self, status, function, *args, **kwargs): + with self.assertRaises(WorkerError) as error: + function(*args, **kwargs) + self.assertEqual(error.exception.status, status) + + def test_capabilities(self): + caps = self.worker.capabilities() + self.assertTrue(caps['exact']) + self.assertEqual(caps['backend'], 'cpu-exact') + self.assertEqual(caps['filters_mask'], 7) + self.assertEqual(caps['metrics_mask'], 7) + self.assertEqual(caps['storage'], 'fp32') + + def test_deadline_bounds_blocked_reads_and_writes(self): + for payload in [b'', bytes(1024 * 1024)]: + child = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(60)'], + stdin=subprocess.PIPE, stdout=subprocess.PIPE) + with patch('support.client.subprocess.Popen', return_value=child): + with Worker(timeout=.05) as stalled: + with self.assertRaises(TimeoutError): + stalled.request(0, payload=payload) + self.assertIsNotNone(child.poll()) + with self.assertRaises(RuntimeError): + stalled.capabilities() + + def test_transport_corruption_invalidates_session(self): + code = ('import sys; sys.stdin.buffer.read(32); ' + 'sys.stdout.buffer.write(bytes(32)); sys.stdout.buffer.flush()') + child = subprocess.Popen([sys.executable, '-c', code], + stdin=subprocess.PIPE, stdout=subprocess.PIPE) + with patch('support.client.subprocess.Popen', return_value=child): + with Worker(timeout=2) as broken: + with self.assertRaises(ValueError): + broken.capabilities() + with self.assertRaises(RuntimeError): + broken.capabilities() + + def test_filtered_batches_against_independent_oracle(self): + rng = random.Random(42) + vectors = [[rng.randint(-5, 5) for _ in range(7)] for _ in range(80)] + queries = [[rng.randint(1, 5) for _ in range(7)] for _ in range(5)] + filters = [sorted(rng.sample(range(80), n)) for n in [0, 1, 8, 25, 80]] + self.worker.build(1, vectors) + for metric in range(3): + for k in [0, 1, 7, 100]: + actual = self.worker.search(1, queries, k, metric, filters, True) + for query, ids, row in zip(queries, filters, actual): + expected = [] + for i in ids: + vector = vectors[i] + dot = sum(a * b for a, b in zip(vector, query)) + if metric == 0: + distance = math.sqrt(sum((a - b) ** 2 for a, b in zip(vector, query))) + elif metric == 1: + distance = -dot + else: + norm = sum(a * a for a in vector) * sum(b * b for b in query) + if not norm: + continue + distance = 1 - max(-1, min(1, dot / math.sqrt(norm))) + expected.append((i, distance)) + expected.sort(key=lambda x: (x[1], x[0])) + self.assertEqual([r[0] for r in row], [r[0] for r in expected[:k]]) + for found, want in zip(row, expected[:k]): + self.assertAlmostEqual(found[1], want[1], places=12) + + def test_shared_none_empty_and_ties(self): + self.worker.build(1, [[1, 0], [-1, 0], [0, 0]]) + self.assertEqual(self.worker.search(1, [[0, 0]], 4)[0], [(2, 0), (0, 1), (1, 1)]) + self.assertEqual(self.worker.search(1, [[0, 0]], 4, allowed=[]), [[]]) + self.assertEqual(self.worker.search(1, [[0, 0]] * 2, 4, allowed=[1]), [[(1, 1)]] * 2) + self.assertStatus(1, self.worker.search, 1, [[0, 0]], 2, metric=2) + self.assertEqual(len(self.worker.search(1, [[1, 0]], 9, metric=2)[0]), 2) + + def test_generation_and_failed_build_atomicity(self): + self.assertStatus(3, self.worker.search, 1, [[1]], 1) + self.worker.build(1, [[2]]) + self.assertStatus(1, self.worker.build, 2, [[float('nan')]]) + self.assertEqual(self.worker.search(1, [[1]], 1), [[(0, 1)]]) + self.assertStatus(3, self.worker.search, 2, [[1]], 1) + self.worker.build(2, [[3]]) + self.assertStatus(3, self.worker.search, 1, [[1]], 1) + self.worker.unload(2) + self.assertStatus(3, self.worker.search, 2, [[1]], 1) + self.assertStatus(3, self.worker.build, 2, [[4]]) + + def test_snapshot_reload_and_corruption(self): + self.worker.build(1, [[1, 2], [3, 4]]) + before = self.worker.search(1, [[1, 1]], 2) + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / 'snapshot' + self.worker.save(1, path) + self.worker.unload(1) + self.worker.load(2, path) + self.assertEqual(self.worker.search(2, [[1, 1]], 2), before) + data = bytearray(path.read_bytes()) + data[-1] ^= 1 + path.write_bytes(data) + with self.assertRaises(ValueError): + self.worker.load(3, path) + self.assertEqual(self.worker.search(2, [[1, 1]], 2), before) + + def test_transport_probe_is_not_search(self): + self.worker.build(1, [[1]]) + self.assertEqual(self.worker.request(5, 1, b'diagnostic bytes'), b'') + + def test_empty_index(self): + self.worker.build(1, [], dimensions=2) + self.assertEqual(self.worker.search(1, [[1, 2]], 10), [[]]) + + def test_invalid_inputs_and_stream_recovery(self): + self.worker.build(1, [[1], [2]]) + for ids in [[0, 0], [1, 0], [2]]: + self.assertStatus(1, self.worker.search, 1, [[1]], 1, allowed=ids) + self.assertStatus(1, self.worker.search, 1, [[math.inf]], 1) + self.assertStatus(1, self.worker.search, 1, [[1, 2]], 1) + self.assertStatus(2, self.worker.search, 1, [[1]], 1, metric=99) + self.assertStatus(1, self.worker.request, 2, 1, b'') + self.assertStatus(1, self.worker.request, 1, 2, struct.pack('<II', 2**32-1, 16000)) + self.assertEqual(self.worker.search(1, [[1]], 1), [[(0, 0)]]) + + def test_bad_frames_exit(self): + frames = [b'bad', HEADER.pack(b'PVAN', 2, 0, 1, 0, 0, 0), + HEADER.pack(b'PVAN', 1, 0, 1, 0, 2**32-1, 0), + HEADER.pack(b'PVAN', 1, 1, 1, 1, 8, 0) + b'bad'] + for frame in frames: + result = subprocess.run([str(ROOT / 'ann/ann-worker')], input=frame, + capture_output=True, timeout=5) + self.assertNotEqual(result.returncode, 0) + + def test_vendor_selection_never_silently_falls_back(self): + for vendor in ['cuvs', 'hipvs', 'fused-nvfp4', 'unknown']: + result = subprocess.run([str(ROOT / 'ann/ann-worker'), '--backend', vendor], + input=b'', capture_output=True, timeout=5) + self.assertEqual(result.returncode, 2) + self.assertIn(b'require vendor adapters', result.stderr) + + +if __name__ == '__main__': + unittest.main() diff --git a/test/expected/ann_filtered.out b/test/expected/ann_filtered.out new file mode 100644 index 000000000..e5da86321 --- /dev/null +++ b/test/expected/ann_filtered.out @@ -0,0 +1,115 @@ +-- Explicit loading also installs hooks when the first query has no vector literal. +LOAD 'vector'; +SET ann.enable_filtered_search = on; +CREATE TABLE t (id integer, tenant text, val vector(3)); +INSERT INTO t VALUES (1, 'a', '[0,0,0]'), (2, 'b', '[1,2,3]'), + (3, 'a', '[1,1,1]'), (4, 'a', NULL), (5, 'a', '[1,2,4]'); +CREATE INDEX ON t USING ann (val vector_l2_ops); +ANALYZE t; +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 3; + id +---- + 5 + 3 + 1 +(3 rows) + +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 2 OFFSET 1; + id +---- + 3 + 1 +(2 rows) + +SELECT id FROM t WHERE tenant = 'missing' ORDER BY val <-> '[3,3,3]', id LIMIT 3; + id +---- +(0 rows) + +SET plan_cache_mode = force_generic_plan; +PREPARE filtered_query(vector, bigint, bigint) AS + SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> $1, id LIMIT $2 OFFSET $3; +EXECUTE filtered_query('[3,3,3]', 2, 1); + id +---- + 3 + 1 +(2 rows) + +EXECUTE filtered_query(NULL, 3, 0); + id +---- + 1 + 3 + 4 +(3 rows) + +EXECUTE filtered_query('[0,0,0]', NULL, 0); + id +---- + 1 + 3 + 5 + 4 +(4 rows) + +DEALLOCATE filtered_query; +RESET plan_cache_mode; +-- Unsupported shapes retain ordinary PostgreSQL execution. +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]' NULLS FIRST, id LIMIT 3; + id +---- + 4 + 5 + 3 +(3 rows) + +BEGIN; +DECLARE c SCROLL CURSOR FOR SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id; +FETCH 3 FROM c; + id +---- + 5 + 3 + 1 +(3 rows) + +FETCH BACKWARD 2 FROM c; + id +---- + 3 + 5 +(2 rows) + +CLOSE c; +COMMIT; +-- An unavailable worker keeps exact local results. +SET ann.worker_socket = '/nonexistent/ann-worker.sock'; +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 3; + id +---- + 5 + 3 + 1 +(3 rows) + +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 2 OFFSET 1; + id +---- + 3 + 1 +(2 rows) + +EXPLAIN (COSTS OFF) SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 3; + QUERY PLAN +--------------------------------------------------------------------- + Limit + -> Custom Scan (AnnFilteredSearch) + Search Route: worker pipeline or exact CPU after predicates + -> Seq Scan on t + Filter: (tenant = 'a'::text) +(5 rows) + +RESET ann.worker_socket; +DROP TABLE t; +RESET ann.enable_filtered_search; diff --git a/test/expected/ann_vector.out b/test/expected/ann_vector.out new file mode 100644 index 000000000..7b4e6893b --- /dev/null +++ b/test/expected/ann_vector.out @@ -0,0 +1,95 @@ +SET enable_seqscan = off; +-- Exact CPU staging: NULLs are retained and predicates run in the executor. +CREATE TABLE t (id integer, tenant text, val vector(3)); +INSERT INTO t VALUES (1, 'a', '[0,0,0]'), (2, 'b', '[1,2,3]'), + (3, 'a', '[1,1,1]'), (4, 'a', NULL); +CREATE INDEX t_ann ON t USING ann (val vector_l2_ops); +INSERT INTO t VALUES (5, 'a', '[1,2,4]'); +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; + id +---- + 2 + 5 + 3 + 1 + 4 +(5 rows) + +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]' LIMIT 3; + id +---- + 5 + 3 + 1 +(3 rows) + +SELECT COUNT(*) FROM (SELECT * FROM t ORDER BY val <-> (SELECT NULL::vector)) s; + count +------- + 5 +(1 row) + +SELECT pg_index_has_property('t_ann', 'index_scan'), + pg_index_column_has_property('t_ann', 1, 'distance_orderable'); + pg_index_has_property | pg_index_column_has_property +-----------------------+------------------------------ + t | t +(1 row) + +-- Rescans use a new query vector for each outer row. +SELECT t.id, s.id FROM t CROSS JOIN LATERAL + (SELECT id FROM t t2 ORDER BY val <-> t.val LIMIT 1) s + WHERE t.val IS NOT NULL ORDER BY t.id; + id | id +----+---- + 1 | 1 + 2 | 2 + 3 | 3 + 5 | 5 +(4 rows) + +UPDATE t SET val = '[3,3,3]' WHERE id = 1; +DELETE FROM t WHERE id = 2; +VACUUM t; +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; + id +---- + 1 + 5 + 3 + 4 +(4 rows) + +REINDEX INDEX t_ann; +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; + id +---- + 1 + 5 + 3 + 4 +(4 rows) + +TRUNCATE t; +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; + id +---- +(0 rows) + +INSERT INTO t VALUES (6, 'a', '[3,3,3]'); +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; + id +---- + 6 +(1 row) + +DROP TABLE t; +CREATE TABLE t (val vector); +CREATE INDEX ON t USING ann (val vector_l2_ops); +ERROR: column does not have dimensions +DROP TABLE t; +CREATE TABLE t (val vector(2001)); +CREATE INDEX ON t USING ann (val vector_l2_ops); +ERROR: column cannot have more than 2000 dimensions for ann index +DROP TABLE t; +RESET enable_seqscan; diff --git a/test/sql/ann_filtered.sql b/test/sql/ann_filtered.sql new file mode 100644 index 000000000..d18fff687 --- /dev/null +++ b/test/sql/ann_filtered.sql @@ -0,0 +1,41 @@ +-- Explicit loading also installs hooks when the first query has no vector literal. +LOAD 'vector'; +SET ann.enable_filtered_search = on; + +CREATE TABLE t (id integer, tenant text, val vector(3)); +INSERT INTO t VALUES (1, 'a', '[0,0,0]'), (2, 'b', '[1,2,3]'), + (3, 'a', '[1,1,1]'), (4, 'a', NULL), (5, 'a', '[1,2,4]'); +CREATE INDEX ON t USING ann (val vector_l2_ops); +ANALYZE t; + +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 3; +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 2 OFFSET 1; +SELECT id FROM t WHERE tenant = 'missing' ORDER BY val <-> '[3,3,3]', id LIMIT 3; + +SET plan_cache_mode = force_generic_plan; +PREPARE filtered_query(vector, bigint, bigint) AS + SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> $1, id LIMIT $2 OFFSET $3; +EXECUTE filtered_query('[3,3,3]', 2, 1); +EXECUTE filtered_query(NULL, 3, 0); +EXECUTE filtered_query('[0,0,0]', NULL, 0); +DEALLOCATE filtered_query; +RESET plan_cache_mode; + +-- Unsupported shapes retain ordinary PostgreSQL execution. +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]' NULLS FIRST, id LIMIT 3; +BEGIN; +DECLARE c SCROLL CURSOR FOR SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id; +FETCH 3 FROM c; +FETCH BACKWARD 2 FROM c; +CLOSE c; +COMMIT; + +-- An unavailable worker keeps exact local results. +SET ann.worker_socket = '/nonexistent/ann-worker.sock'; +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 3; +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 2 OFFSET 1; +EXPLAIN (COSTS OFF) SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]', id LIMIT 3; +RESET ann.worker_socket; + +DROP TABLE t; +RESET ann.enable_filtered_search; diff --git a/test/sql/ann_vector.sql b/test/sql/ann_vector.sql new file mode 100644 index 000000000..7cb7601ce --- /dev/null +++ b/test/sql/ann_vector.sql @@ -0,0 +1,37 @@ +SET enable_seqscan = off; + +-- Exact CPU staging: NULLs are retained and predicates run in the executor. +CREATE TABLE t (id integer, tenant text, val vector(3)); +INSERT INTO t VALUES (1, 'a', '[0,0,0]'), (2, 'b', '[1,2,3]'), + (3, 'a', '[1,1,1]'), (4, 'a', NULL); +CREATE INDEX t_ann ON t USING ann (val vector_l2_ops); +INSERT INTO t VALUES (5, 'a', '[1,2,4]'); +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; +SELECT id FROM t WHERE tenant = 'a' ORDER BY val <-> '[3,3,3]' LIMIT 3; +SELECT COUNT(*) FROM (SELECT * FROM t ORDER BY val <-> (SELECT NULL::vector)) s; +SELECT pg_index_has_property('t_ann', 'index_scan'), + pg_index_column_has_property('t_ann', 1, 'distance_orderable'); + +-- Rescans use a new query vector for each outer row. +SELECT t.id, s.id FROM t CROSS JOIN LATERAL + (SELECT id FROM t t2 ORDER BY val <-> t.val LIMIT 1) s + WHERE t.val IS NOT NULL ORDER BY t.id; +UPDATE t SET val = '[3,3,3]' WHERE id = 1; +DELETE FROM t WHERE id = 2; +VACUUM t; +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; +REINDEX INDEX t_ann; +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; +TRUNCATE t; +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; +INSERT INTO t VALUES (6, 'a', '[3,3,3]'); +SELECT id FROM t ORDER BY val <-> '[3,3,3]'; +DROP TABLE t; + +CREATE TABLE t (val vector); +CREATE INDEX ON t USING ann (val vector_l2_ops); +DROP TABLE t; +CREATE TABLE t (val vector(2001)); +CREATE INDEX ON t USING ann (val vector_l2_ops); +DROP TABLE t; +RESET enable_seqscan; diff --git a/test/t/050_ann_wal.pl b/test/t/050_ann_wal.pl new file mode 100644 index 000000000..eb5cf0f7f --- /dev/null +++ b/test/t/050_ann_wal.pl @@ -0,0 +1,66 @@ +# Follow the existing access-method WAL tests, with an exact SQL oracle. +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 1); +$primary->start; +$primary->backup('ann_backup'); +my $replica = PostgreSQL::Test::Cluster->new('replica'); +$replica->init_from_backup($primary, 'ann_backup', has_streaming => 1); +$replica->start; + +my $query = q(SELECT id FROM tst ORDER BY v <-> '[13.4,3,5]', id); + +sub check_results +{ + my ($name) = @_; + my $expected = $primary->safe_psql('postgres', + "SET enable_indexscan = off; SET enable_bitmapscan = off; $query"); + my $actual = $primary->safe_psql('postgres', "SET enable_seqscan = off; $query"); + is($actual, $expected, "$name: exact primary results"); + $primary->wait_for_replay_catchup($replica->name); + my $replayed = $replica->safe_psql('postgres', "SET enable_seqscan = off; $query"); + is($replayed, $expected, "$name: exact replay results"); +} + +$primary->safe_psql('postgres', q( + CREATE EXTENSION vector; + CREATE TABLE tst (id integer, v vector(3)); + INSERT INTO tst SELECT i, ARRAY[i * 0.123, i % 13, i % 17]::vector + FROM generate_series(1, 2000) i; + INSERT INTO tst VALUES (3000, NULL); + CREATE INDEX tst_ann ON tst USING ann (v vector_l2_ops); +)); +check_results('bulk build'); +$primary->safe_psql('postgres', q( + INSERT INTO tst SELECT i, ARRAY[i,1,2]::vector FROM generate_series(4000, 5000) i; + UPDATE tst SET v = '[13.4,3,5]' WHERE id = 1; + DELETE FROM tst WHERE id % 2 = 0; +)); +$primary->safe_psql('postgres', 'VACUUM tst'); +$primary->safe_psql('postgres', q( + INSERT INTO tst SELECT i, ARRAY[i,1,2]::vector FROM generate_series(6000, 6500) i; + BEGIN; + INSERT INTO tst VALUES (9999, '[13.4,3,5]'); + ROLLBACK; +)); +check_results('insert update vacuum reuse and abort'); + +# An immediate shutdown exercises WAL recovery rather than clean buffer flush. +$primary->stop('immediate'); +$primary->start; +check_results('crash recovery'); +$primary->safe_psql('postgres', 'REINDEX INDEX CONCURRENTLY tst_ann'); +check_results('concurrent reindex'); +$primary->safe_psql('postgres', q( + TRUNCATE tst; + INSERT INTO tst VALUES (1, '[13.4,3,5]'), (2, NULL); +)); +check_results('truncate'); +$replica->stop; +$primary->stop; +done_testing(); diff --git a/vector.control b/vector.control index c9c4b6a0a..5b2013348 100644 --- a/vector.control +++ b/vector.control @@ -1,4 +1,4 @@ comment = 'vector data type and ivfflat and hnsw access methods' -default_version = '0.8.7' +default_version = '0.9.0' module_pathname = '$libdir/vector' relocatable = true
^ permalink raw [nested|flat] 2+ messages in thread

* Re: [pgvector/pgvector] PR #1053: Serve loaded indexes to many PostgreSQL hosts over the network
@ 2026-10-31 14:05  "chenqin (@chenqin)" <noreply+chenqin@github.com>
  0 siblings, 0 replies; 2+ messages in thread

From: chenqin (@chenqin) @ 2026-10-31 14:05 UTC (permalink / raw)
  To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>

@chenqin closed this.

^ permalink  raw  [nested|flat] 2+ messages in thread


end of thread, other threads:[~2026-10-31 14:05 UTC | newest]

Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-31 14:05 [pgvector/pgvector] PR #1053: Serve loaded indexes to many PostgreSQL hosts over the network "chenqin (@chenqin)" <noreply+chenqin@github.com>
2026-10-31 14:05 Re: [pgvector/pgvector] PR #1053: Serve loaded indexes to many PostgreSQL hosts over the network "chenqin (@chenqin)" <noreply+chenqin@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox