From: catidegla (@catidegla) <noreply+catidegla@github.com>
To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>
Subject: Re: [pgvector/pgvector] issue #1043: bit Jaccard distance returns 1.0 for identical all-zero bit vectors
Date: Mon, 16 Nov 2026 11:05:09 +0000
Message-ID: <gh-pgvector-pgvector-1043-c6914652828@github.com> (raw)
In-Reply-To: <gh-pgvector-pgvector-1043@github.com>
References: <gh-pgvector-pgvector-1043@github.com>
The three options are not equally valid for pgvector specifically, because `<%>` is an ANN distance backing an index, not just a pairwise metric, and that picks the answer. For the index to work a vector has to be its own nearest neighbor, i.e. `d(x, x) = 0` (or at least `d(x,x) <= d(x,y)` for every other y), and only option 0 gives that for the all-zero vector. With 1, the zero vector ties its own distance with its distance to arbitrary nonzero vectors, so it is no longer distinguished as the exact match; with NaN it is the worst case, because Postgres orders NaN greater than everything, so the zero vector becomes the furthest thing from itself and an all-zero query never retrieves the identical stored row. So on any dataset containing zero vectors, 1 or NaN silently drops recall for those rows, while 0 keeps them findable.
There is also an internal-consistency argument: Hamming `<~>` already returns 0 for the identical all-zero pair, so Jaccard returning 1 means the two bit-distance operators disagree about whether a vector is its own nearest neighbor. Option 0 realigns them, and it is why SciPy switched NaN to 0 in 2018, the same reasoning you half-noted, to satisfy `d(x,x) = 0`.
Concretely, building on the `src/bitutils.c` location above, the case to add is `union == 0` (both popcounts zero) returns 0, kept distinct from the `ab == 0` but `union > 0` case (disjoint nonzero sets) which correctly stays 1. So the guard becomes the union-zero check first, then the existing ab-zero branch.