pg.ddx.io  pgvector/pgvector GitHub issues and pull requests  
help / color / mirror / Atom feed
[pgvector/pgvector] issue #1057: Cosine distance returns NaN or incorrect finite values for finite nonzero vectors
3+ messages / 2 participants
[nested] [flat]

* [pgvector/pgvector] issue #1057: Cosine distance returns NaN or incorrect finite values for finite nonzero vectors
@ 2026-11-04 02:38  "yzxxxx888 (@yzxxxx888)" <noreply+yzxxxx888@github.com>
  0 siblings, 0 replies; 3+ messages in thread

From: yzxxxx888 (@yzxxxx888) @ 2026-11-04 02:38 UTC (permalink / raw)
  To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>

## Summary

The cosine distance operator `<=>` returns `NaN` or an incorrect finite distance for finite, nonzero `vector` inputs with large or small magnitudes.

The issue reproduces with direct SQL expressions, without HNSW or IVFFlat indexes.

## Environment

- PostgreSQL: 17.11
- pgvector: 0.8.7
- Docker image: `pgvector/pgvector:0.8.7-pg17`
- Vector type: `vector` (float32 components)

These results were observed on the versions above. I have not verified the current development branch.

## Steps to reproduce

Enable the extension:

```sql
CREATE EXTENSION IF NOT EXISTS vector;
```

Run the following queries:

```sql
-- Large finite vector compared with itself
SELECT '[1e20,0]'::vector <=> '[1e20,0]'::vector
    AS large_self_distance;

-- Two finite nonzero vectors pointing in the same direction
SELECT '[1e20,0]'::vector <=> '[1,0]'::vector
    AS same_direction_distance;

-- Small finite nonzero vector compared with itself
SELECT '[1e-30,0]'::vector <=> '[1e-30,0]'::vector
    AS small_self_distance;
```

## Actual results

| Query | Actual cosine distance |
|---|---:|
| `[1e20,0]` compared with itself | `NaN` |
| `[1e20,0]` compared with `[1,0]` | `1` |
| `[1e-30,0]` compared with itself | `NaN` |

All input components are finite and representable as nonzero float32 values. None of these vectors is a zero vector.

## Expected results

All three queries should return a cosine distance of `0`, within normal floating-point tolerance.

For nonzero vectors:

```text
cosine_distance(a, b) = 1 - dot(a, b) / (norm(a) * norm(b))
```

A nonzero vector has zero cosine distance from itself. Positive scaling should also preserve cosine distance, so `[1e20,0]` and `[1,0]` should have distance `0`.

The expected result is therefore finite and well-defined in all three cases.

## Normalization control

Normalizing the large vectors first produces the expected result:

```sql
SELECT
    l2_normalize('[1e20,0]'::vector)
    <=>
    l2_normalize('[1e20,0]'::vector)
    AS normalized_self_distance;

SELECT
    l2_normalize('[1e20,0]'::vector)
    <=>
    l2_normalize('[1,0]'::vector)
    AS normalized_same_direction_distance;
```

Both queries return `0`.

## Cause analysis

In pgvector v0.8.7, `VectorCosineSimilarity` accumulates the dot product and squared norms using `float`:

```c
float similarity = 0.0;
float norma = 0.0;
float normb = 0.0;

similarity += ax[i] * bx[i];
norma += ax[i] * ax[i];
normb += bx[i] * bx[i];
```

Source:
https://github.com/pgvector/pgvector/blob/v0.8.7/src/vector.c#L613

For the large self-comparison, the float32 products overflow, producing an indeterminate infinity/infinity calculation.

For `[1e20,0]` compared with `[1,0]`, the dot product remains finite, but the squared norm of the large vector overflows. This produces a cosine similarity of `0` and a distance of `1`.

For the small self-comparison, the squared components underflow to zero, resulting in a zero/zero calculation.

Converting the accumulated values to double for the final calculation cannot recover information already lost during float32 multiplication and accumulation.


^ permalink  raw  [nested|flat] 3+ messages in thread

* Re: [pgvector/pgvector] issue #1057: Cosine distance returns NaN or incorrect finite values for finite nonzero vectors
@ 2026-11-04 02:54  "ankane (@ankane)" <noreply+ankane@github.com>
  1 sibling, 0 replies; 3+ messages in thread

From: ankane (@ankane) @ 2026-11-04 02:54 UTC (permalink / raw)
  To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>

Hi @yzxxxx888, see #1033.

^ permalink  raw  [nested|flat] 3+ messages in thread

* Re: [pgvector/pgvector] issue #1057: Cosine distance returns NaN or incorrect finite values for finite nonzero vectors
@ 2026-11-04 02:54  "ankane (@ankane)" <noreply+ankane@github.com>
  1 sibling, 0 replies; 3+ messages in thread

From: ankane (@ankane) @ 2026-11-04 02:54 UTC (permalink / raw)
  To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>

@ankane closed this.

^ permalink  raw  [nested|flat] 3+ messages in thread


end of thread, other threads:[~2026-11-04 02:54 UTC | newest]

Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-04 02:38 [pgvector/pgvector] issue #1057: Cosine distance returns NaN or incorrect finite values for finite nonzero vectors "yzxxxx888 (@yzxxxx888)" <noreply+yzxxxx888@github.com>
2026-11-04 02:54 ` "ankane (@ankane)" <noreply+ankane@github.com>
2026-11-04 02:54 ` "ankane (@ankane)" <noreply+ankane@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox