pg.ddx.io  pgvector/pgvector GitHub issues and pull requests  
help / color / mirror / Atom feed
From: yzxxxx888 (@yzxxxx888) <noreply+yzxxxx888@github.com>
To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>
Subject: [pgvector/pgvector] issue #1057: Cosine distance returns NaN or incorrect finite values for finite nonzero vectors
Date: Wed, 04 Nov 2026 02:38:59 +0000
Message-ID: <gh-pgvector-pgvector-1057@github.com> (raw)

## Summary

The cosine distance operator `<=>` returns `NaN` or an incorrect finite distance for finite, nonzero `vector` inputs with large or small magnitudes.

The issue reproduces with direct SQL expressions, without HNSW or IVFFlat indexes.

## Environment

- PostgreSQL: 17.11
- pgvector: 0.8.7
- Docker image: `pgvector/pgvector:0.8.7-pg17`
- Vector type: `vector` (float32 components)

These results were observed on the versions above. I have not verified the current development branch.

## Steps to reproduce

Enable the extension:

```sql
CREATE EXTENSION IF NOT EXISTS vector;
```

Run the following queries:

```sql
-- Large finite vector compared with itself
SELECT '[1e20,0]'::vector <=> '[1e20,0]'::vector
    AS large_self_distance;

-- Two finite nonzero vectors pointing in the same direction
SELECT '[1e20,0]'::vector <=> '[1,0]'::vector
    AS same_direction_distance;

-- Small finite nonzero vector compared with itself
SELECT '[1e-30,0]'::vector <=> '[1e-30,0]'::vector
    AS small_self_distance;
```

## Actual results

| Query | Actual cosine distance |
|---|---:|
| `[1e20,0]` compared with itself | `NaN` |
| `[1e20,0]` compared with `[1,0]` | `1` |
| `[1e-30,0]` compared with itself | `NaN` |

All input components are finite and representable as nonzero float32 values. None of these vectors is a zero vector.

## Expected results

All three queries should return a cosine distance of `0`, within normal floating-point tolerance.

For nonzero vectors:

```text
cosine_distance(a, b) = 1 - dot(a, b) / (norm(a) * norm(b))
```

A nonzero vector has zero cosine distance from itself. Positive scaling should also preserve cosine distance, so `[1e20,0]` and `[1,0]` should have distance `0`.

The expected result is therefore finite and well-defined in all three cases.

## Normalization control

Normalizing the large vectors first produces the expected result:

```sql
SELECT
    l2_normalize('[1e20,0]'::vector)
    <=>
    l2_normalize('[1e20,0]'::vector)
    AS normalized_self_distance;

SELECT
    l2_normalize('[1e20,0]'::vector)
    <=>
    l2_normalize('[1,0]'::vector)
    AS normalized_same_direction_distance;
```

Both queries return `0`.

## Cause analysis

In pgvector v0.8.7, `VectorCosineSimilarity` accumulates the dot product and squared norms using `float`:

```c
float similarity = 0.0;
float norma = 0.0;
float normb = 0.0;

similarity += ax[i] * bx[i];
norma += ax[i] * ax[i];
normb += bx[i] * bx[i];
```

Source:
https://github.com/pgvector/pgvector/blob/v0.8.7/src/vector.c#L613

For the large self-comparison, the float32 products overflow, producing an indeterminate infinity/infinity calculation.

For `[1e20,0]` compared with `[1,0]`, the dot product remains finite, but the squared norm of the large vector overflows. This produces a cosine similarity of `0` and a distance of `1`.

For the small self-comparison, the squared components underflow to zero, resulting in a zero/zero calculation.

Converting the accumulated values to double for the final calculation cannot recover information already lost during float32 multiplication and accumulation.


view thread (3+ messages)  latest in thread

Message-ID: <gh-pgvector-pgvector-1057@github.com>
Permalink:  ../gh-pgvector-pgvector-1057@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox