From: yzxxxx888 (@yzxxxx888) <noreply+yzxxxx888@github.com>
To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>
Subject: [pgvector/pgvector] issue #1057: Cosine distance returns NaN or incorrect finite values for finite nonzero vectors
Date: Wed, 04 Nov 2026 02:38:59 +0000
Message-ID: <gh-pgvector-pgvector-1057@github.com> (raw)
## Summary
The cosine distance operator `<=>` returns `NaN` or an incorrect finite distance for finite, nonzero `vector` inputs with large or small magnitudes.
The issue reproduces with direct SQL expressions, without HNSW or IVFFlat indexes.
## Environment
- PostgreSQL: 17.11
- pgvector: 0.8.7
- Docker image: `pgvector/pgvector:0.8.7-pg17`
- Vector type: `vector` (float32 components)
These results were observed on the versions above. I have not verified the current development branch.
## Steps to reproduce
Enable the extension:
```sql
CREATE EXTENSION IF NOT EXISTS vector;
```
Run the following queries:
```sql
-- Large finite vector compared with itself
SELECT '[1e20,0]'::vector <=> '[1e20,0]'::vector
AS large_self_distance;
-- Two finite nonzero vectors pointing in the same direction
SELECT '[1e20,0]'::vector <=> '[1,0]'::vector
AS same_direction_distance;
-- Small finite nonzero vector compared with itself
SELECT '[1e-30,0]'::vector <=> '[1e-30,0]'::vector
AS small_self_distance;
```
## Actual results
| Query | Actual cosine distance |
|---|---:|
| `[1e20,0]` compared with itself | `NaN` |
| `[1e20,0]` compared with `[1,0]` | `1` |
| `[1e-30,0]` compared with itself | `NaN` |
All input components are finite and representable as nonzero float32 values. None of these vectors is a zero vector.
## Expected results
All three queries should return a cosine distance of `0`, within normal floating-point tolerance.
For nonzero vectors:
```text
cosine_distance(a, b) = 1 - dot(a, b) / (norm(a) * norm(b))
```
A nonzero vector has zero cosine distance from itself. Positive scaling should also preserve cosine distance, so `[1e20,0]` and `[1,0]` should have distance `0`.
The expected result is therefore finite and well-defined in all three cases.
## Normalization control
Normalizing the large vectors first produces the expected result:
```sql
SELECT
l2_normalize('[1e20,0]'::vector)
<=>
l2_normalize('[1e20,0]'::vector)
AS normalized_self_distance;
SELECT
l2_normalize('[1e20,0]'::vector)
<=>
l2_normalize('[1,0]'::vector)
AS normalized_same_direction_distance;
```
Both queries return `0`.
## Cause analysis
In pgvector v0.8.7, `VectorCosineSimilarity` accumulates the dot product and squared norms using `float`:
```c
float similarity = 0.0;
float norma = 0.0;
float normb = 0.0;
similarity += ax[i] * bx[i];
norma += ax[i] * ax[i];
normb += bx[i] * bx[i];
```
Source:
https://github.com/pgvector/pgvector/blob/v0.8.7/src/vector.c#L613
For the large self-comparison, the float32 products overflow, producing an indeterminate infinity/infinity calculation.
For `[1e20,0]` compared with `[1,0]`, the dot product remains finite, but the squared norm of the large vector overflows. This produces a cosine similarity of `0` and a distance of `1`.
For the small self-comparison, the squared components underflow to zero, resulting in a zero/zero calculation.
Converting the accumulated values to double for the final calculation cannot recover information already lost during float32 multiplication and accumulation.