pg.ddx.io  pgvector/pgvector GitHub issues and pull requests  
help / color / mirror / Atom feed
From: leemeii (@leemeii) <noreply+leemeii@github.com>
To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>
Subject: [pgvector/pgvector] issue #1049: sparsevec rejects semantically empty input with more than 16,000 explicit zero fields
Date: Mon, 26 Oct 2026 13:19:45 +0000
Message-ID: <gh-pgvector-pgvector-1049@github.com> (raw)

## Summary

The `sparsevec` input parser rejects a vector containing more than 16,000 explicitly specified zero-valued entries, even when the resulting sparse vector would contain zero non-zero elements.

For example, the following canonical representation is accepted:

```sql
SELECT '{}/16001'::sparsevec;
```

It returns:

```text
{}/16001
```

However, an equivalent representation containing an explicit zero at every position is rejected:

```text
{1:0,2:0,...,16001:0}/16001
```

The error is:

```text
ERROR: sparsevec cannot have more than 16000 non-zero elements
```

The parser applies the limit before parsing any values or determining how many entries are actually non-zero. Consequently, an input with 16,001 zero-valued entries is rejected even though its resulting sparse representation would have `nnz = 0`.

This appears to be an input-validation inconsistency between the documented error condition and the actual validation logic.

## Steps to reproduce

Run the following SQL using PostgreSQL with pgvector installed.

The script generates the full input dynamically, so there is no need to paste a literal containing 16,001 entries manually.

```python

import math
import os
import psycopg


DSN = os.getenv(
    "PGVECTOR_DSN",
    "dbname=postgres user=postgres"
)

DIM = 16001

# Seed: canonical empty sparse vector
seed = f"{{}}/{DIM}"

# Mutant 1: 16,000 explicit zero fields
mutant_boundary = (
    "{"
    + ",".join(f"{i}:0" for i in range(1, 16001))
    + f"}}/{DIM}"
)

# Mutant 2: 16,001 explicit zero fields
mutant_bug = (
    "{"
    + ",".join(f"{i}:0" for i in range(1, DIM + 1))
    + f"}}/{DIM}"
)

# Query vector for L1 distance
query = f"{{1:1,16001:2}}/{DIM}"


def parse_sparsevec(conn, value):
    with conn.cursor() as cur:
        cur.execute(
            "SELECT %s::sparsevec::text",
            (value,)
        )
        return cur.fetchone()[0]


def l1_distance(conn, vector, query_vector):
    with conn.cursor() as cur:
        cur.execute(
            """
            SELECT %s::sparsevec <+> %s::sparsevec
            """,
            (vector, query_vector)
        )
        return float(cur.fetchone()[0])


def reproduce():
    with psycopg.connect(DSN, autocommit=True) as conn:
        with conn.cursor() as cur:
            cur.execute(
                "CREATE EXTENSION IF NOT EXISTS vector"
            )

        # Baseline
        seed_parsed = parse_sparsevec(conn, seed)
        seed_distance = l1_distance(conn, seed, query)

        print("=== Baseline ===")
        print("Seed:", seed)
        print("Parsed:", seed_parsed)
        print("Metric: L1")
        print("L1 distance:", seed_distance)

        assert math.isclose(seed_distance, 3.0)

        print("\n=== Mutated Inputs ===")

        cases = [
            ("16,000 explicit zeros", mutant_boundary),
            ("16,001 explicit zeros", mutant_bug),
        ]

        for name, mutant in cases:
            print(f"\nCase: {name}")

            try:
                parsed = parse_sparsevec(conn, mutant)
                distance = l1_distance(
                    conn, mutant, query
                )

                print("Accepted: True")
                print("Parsed:", parsed)
                print("L1 distance:", distance)

                same_vector = parsed == seed_parsed
                same_distance = math.isclose(
                    distance,
                    seed_distance,
                    rel_tol=1e-12,
                    abs_tol=1e-12
                )

                print("Same vector:", same_vector)
                print("Same L1 distance:", same_distance)

                if same_vector and same_distance:
                    print(
                        "PASS: Semantic equivalence preserved."
                    )
                else:
                    print(
                        "BUG: Equivalent sparse vectors "
                        "produce inconsistent results."
                    )

            except psycopg.Error as exc:
                print("Accepted: False")
                print("SQLSTATE:", exc.sqlstate)
                print("Error:", exc.diag.message_primary)

                if name == "16,001 explicit zeros":
                    print(
                        "BUG REPRODUCED: Equivalent sparse "
                        "vector rejected before L1 evaluation."
                    )
                    print(
                        "Expected L1 distance:",
                        seed_distance
                    )
                else:
                    print("Unexpected boundary rejection.")


if __name__ == "__main__":
    reproduce()

```

### Complete SQL reproducer

Save as `repro_sparsevec_explicit_zeros.sql`:

```sql
CREATE EXTENSION IF NOT EXISTS vector;

-- Case 1: The canonical empty sparse vector is accepted.
SELECT '{}/16001'::sparsevec::text AS canonical_result;

-- Case 2: Generate an equivalent representation containing
-- 16,001 explicitly specified zero-valued entries.
DO $repro$
DECLARE
    dim integer := 16001;
    literal text;
    parsed text;
BEGIN
    SELECT
        '{' ||
        string_agg(i::text || ':0', ',' ORDER BY i) ||
        '}/' || dim
    INTO literal
    FROM generate_series(1, dim) AS s(i);

    RAISE NOTICE 'Generated dimensions: %', dim;
    RAISE NOTICE 'Generated explicit entries: %', dim;

    BEGIN
        EXECUTE format(
            'SELECT %L::sparsevec::text',
            literal
        )
        INTO parsed;

        RAISE NOTICE 'Explicit-zero result: %', parsed;

        IF parsed = format('{}/%s', dim) THEN
            RAISE NOTICE
                'Both representations produce the same canonical sparse vector';
        ELSE
            RAISE NOTICE
                'Unexpected canonical representation: %', parsed;
        END IF;

    EXCEPTION
        WHEN program_limit_exceeded THEN
            RAISE NOTICE
                'Explicit-zero input rejected (SQLSTATE %): %',
                SQLSTATE, SQLERRM;
    END;
END
$repro$;
```

Run it with:

```bash
psql -X -d your_database -f repro_sparsevec_explicit_zeros.sql
```

### Expected output

The canonical query should return:

```text
 canonical_result
------------------
 {}/16001
```

The `DO` block should report that the explicit-zero representation was rejected:

```text
NOTICE: Generated dimensions: 16001
NOTICE: Generated explicit entries: 16001
NOTICE: Explicit-zero input rejected (SQLSTATE 54000): sparsevec cannot have more than 16000 non-zero elements
```

The exact formatting of the output may vary by PostgreSQL version.

The reproducer exits successfully because it catches the expected exception. If the explicit-zero input is accepted, the script reports the resulting representation instead.

## Expected behavior

The parser should distinguish between the number of explicitly supplied entries and the number of non-zero elements in the resulting sparse vector.

Since zero-valued entries are discarded during parsing, an input containing 16,001 explicit zeros should be semantically equivalent to:

```sql
SELECT '{}/16001'::sparsevec;
```

The expected result is:

```text
{}/16001
```

At minimum, the implementation should not report that the vector contains more than 16,000 non-zero elements when all supplied values are zero.

If the limit is intentionally intended to constrain the number of input fields rather than the resulting non-zero elements, the error message and validation contract should reflect that distinction.

## Actual behavior

The parser rejects the explicit-zero representation before parsing its values:

```text
ERROR: sparsevec cannot have more than 16000 non-zero elements
```

The canonical empty representation succeeds.

The behavior depends on the textual representation rather than the resulting sparse-vector contents.


view thread (3+ messages)  latest in thread

Message-ID: <gh-pgvector-pgvector-1049@github.com>
Permalink:  ../gh-pgvector-pgvector-1049@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox