pg.ddx.io pgvector/pgvector GitHub issues and pull requests
help / color / mirror / Atom feed[pgvector/pgvector] issue #1049: sparsevec rejects semantically empty input with more than 16,000 explicit zero fields
3+ messages / 2 participants
[nested] [flat]
* [pgvector/pgvector] issue #1049: sparsevec rejects semantically empty input with more than 16,000 explicit zero fields
@ 2026-10-26 13:19 "leemeii (@leemeii)" <noreply+leemeii@github.com>
0 siblings, 0 replies; 3+ messages in thread
From: leemeii (@leemeii) @ 2026-10-26 13:19 UTC (permalink / raw)
To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>
## Summary
The `sparsevec` input parser rejects a vector containing more than 16,000 explicitly specified zero-valued entries, even when the resulting sparse vector would contain zero non-zero elements.
For example, the following canonical representation is accepted:
```sql
SELECT '{}/16001'::sparsevec;
```
It returns:
```text
{}/16001
```
However, an equivalent representation containing an explicit zero at every position is rejected:
```text
{1:0,2:0,...,16001:0}/16001
```
The error is:
```text
ERROR: sparsevec cannot have more than 16000 non-zero elements
```
The parser applies the limit before parsing any values or determining how many entries are actually non-zero. Consequently, an input with 16,001 zero-valued entries is rejected even though its resulting sparse representation would have `nnz = 0`.
This appears to be an input-validation inconsistency between the documented error condition and the actual validation logic.
## Steps to reproduce
Run the following SQL using PostgreSQL with pgvector installed.
The script generates the full input dynamically, so there is no need to paste a literal containing 16,001 entries manually.
```python
import math
import os
import psycopg
DSN = os.getenv(
"PGVECTOR_DSN",
"dbname=postgres user=postgres"
)
DIM = 16001
# Seed: canonical empty sparse vector
seed = f"{{}}/{DIM}"
# Mutant 1: 16,000 explicit zero fields
mutant_boundary = (
"{"
+ ",".join(f"{i}:0" for i in range(1, 16001))
+ f"}}/{DIM}"
)
# Mutant 2: 16,001 explicit zero fields
mutant_bug = (
"{"
+ ",".join(f"{i}:0" for i in range(1, DIM + 1))
+ f"}}/{DIM}"
)
# Query vector for L1 distance
query = f"{{1:1,16001:2}}/{DIM}"
def parse_sparsevec(conn, value):
with conn.cursor() as cur:
cur.execute(
"SELECT %s::sparsevec::text",
(value,)
)
return cur.fetchone()[0]
def l1_distance(conn, vector, query_vector):
with conn.cursor() as cur:
cur.execute(
"""
SELECT %s::sparsevec <+> %s::sparsevec
""",
(vector, query_vector)
)
return float(cur.fetchone()[0])
def reproduce():
with psycopg.connect(DSN, autocommit=True) as conn:
with conn.cursor() as cur:
cur.execute(
"CREATE EXTENSION IF NOT EXISTS vector"
)
# Baseline
seed_parsed = parse_sparsevec(conn, seed)
seed_distance = l1_distance(conn, seed, query)
print("=== Baseline ===")
print("Seed:", seed)
print("Parsed:", seed_parsed)
print("Metric: L1")
print("L1 distance:", seed_distance)
assert math.isclose(seed_distance, 3.0)
print("\n=== Mutated Inputs ===")
cases = [
("16,000 explicit zeros", mutant_boundary),
("16,001 explicit zeros", mutant_bug),
]
for name, mutant in cases:
print(f"\nCase: {name}")
try:
parsed = parse_sparsevec(conn, mutant)
distance = l1_distance(
conn, mutant, query
)
print("Accepted: True")
print("Parsed:", parsed)
print("L1 distance:", distance)
same_vector = parsed == seed_parsed
same_distance = math.isclose(
distance,
seed_distance,
rel_tol=1e-12,
abs_tol=1e-12
)
print("Same vector:", same_vector)
print("Same L1 distance:", same_distance)
if same_vector and same_distance:
print(
"PASS: Semantic equivalence preserved."
)
else:
print(
"BUG: Equivalent sparse vectors "
"produce inconsistent results."
)
except psycopg.Error as exc:
print("Accepted: False")
print("SQLSTATE:", exc.sqlstate)
print("Error:", exc.diag.message_primary)
if name == "16,001 explicit zeros":
print(
"BUG REPRODUCED: Equivalent sparse "
"vector rejected before L1 evaluation."
)
print(
"Expected L1 distance:",
seed_distance
)
else:
print("Unexpected boundary rejection.")
if __name__ == "__main__":
reproduce()
```
### Complete SQL reproducer
Save as `repro_sparsevec_explicit_zeros.sql`:
```sql
CREATE EXTENSION IF NOT EXISTS vector;
-- Case 1: The canonical empty sparse vector is accepted.
SELECT '{}/16001'::sparsevec::text AS canonical_result;
-- Case 2: Generate an equivalent representation containing
-- 16,001 explicitly specified zero-valued entries.
DO $repro$
DECLARE
dim integer := 16001;
literal text;
parsed text;
BEGIN
SELECT
'{' ||
string_agg(i::text || ':0', ',' ORDER BY i) ||
'}/' || dim
INTO literal
FROM generate_series(1, dim) AS s(i);
RAISE NOTICE 'Generated dimensions: %', dim;
RAISE NOTICE 'Generated explicit entries: %', dim;
BEGIN
EXECUTE format(
'SELECT %L::sparsevec::text',
literal
)
INTO parsed;
RAISE NOTICE 'Explicit-zero result: %', parsed;
IF parsed = format('{}/%s', dim) THEN
RAISE NOTICE
'Both representations produce the same canonical sparse vector';
ELSE
RAISE NOTICE
'Unexpected canonical representation: %', parsed;
END IF;
EXCEPTION
WHEN program_limit_exceeded THEN
RAISE NOTICE
'Explicit-zero input rejected (SQLSTATE %): %',
SQLSTATE, SQLERRM;
END;
END
$repro$;
```
Run it with:
```bash
psql -X -d your_database -f repro_sparsevec_explicit_zeros.sql
```
### Expected output
The canonical query should return:
```text
canonical_result
------------------
{}/16001
```
The `DO` block should report that the explicit-zero representation was rejected:
```text
NOTICE: Generated dimensions: 16001
NOTICE: Generated explicit entries: 16001
NOTICE: Explicit-zero input rejected (SQLSTATE 54000): sparsevec cannot have more than 16000 non-zero elements
```
The exact formatting of the output may vary by PostgreSQL version.
The reproducer exits successfully because it catches the expected exception. If the explicit-zero input is accepted, the script reports the resulting representation instead.
## Expected behavior
The parser should distinguish between the number of explicitly supplied entries and the number of non-zero elements in the resulting sparse vector.
Since zero-valued entries are discarded during parsing, an input containing 16,001 explicit zeros should be semantically equivalent to:
```sql
SELECT '{}/16001'::sparsevec;
```
The expected result is:
```text
{}/16001
```
At minimum, the implementation should not report that the vector contains more than 16,000 non-zero elements when all supplied values are zero.
If the limit is intentionally intended to constrain the number of input fields rather than the resulting non-zero elements, the error message and validation contract should reflect that distinction.
## Actual behavior
The parser rejects the explicit-zero representation before parsing its values:
```text
ERROR: sparsevec cannot have more than 16000 non-zero elements
```
The canonical empty representation succeeds.
The behavior depends on the textual representation rather than the resulting sparse-vector contents.
^ permalink raw [nested|flat] 3+ messages in thread
* Re: [pgvector/pgvector] issue #1049: sparsevec rejects semantically empty input with more than 16,000 explicit zero fields
@ 2026-10-26 21:10 "ankane (@ankane)" <noreply+ankane@github.com>
1 sibling, 0 replies; 3+ messages in thread
From: ankane (@ankane) @ 2026-10-26 21:10 UTC (permalink / raw)
To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>
@ankane closed this.
^ permalink raw [nested|flat] 3+ messages in thread
* Re: [pgvector/pgvector] issue #1049: sparsevec rejects semantically empty input with more than 16,000 explicit zero fields
@ 2026-10-26 21:12 "ankane (@ankane)" <noreply+ankane@github.com>
1 sibling, 0 replies; 3+ messages in thread
From: ankane (@ankane) @ 2026-10-26 21:12 UTC (permalink / raw)
To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>
Thanks @leemeii, fixed in the commit above.
^ permalink raw [nested|flat] 3+ messages in thread
end of thread, other threads:[~2026-10-26 21:12 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-10-26 13:19 [pgvector/pgvector] issue #1049: sparsevec rejects semantically empty input with more than 16,000 explicit zero fields "leemeii (@leemeii)" <noreply+leemeii@github.com>
2026-10-26 21:10 ` "ankane (@ankane)" <noreply+ankane@github.com>
2026-10-26 21:12 ` "ankane (@ankane)" <noreply+ankane@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox