From: PG Bug reporting form <noreply@postgresql.org>
To: pgsql-bugs@lists.postgresql.org
Cc: 125atishprak@gmail.com
Subject: BUG #19854: GiST distance ordering fails on NaN coordinates, Assert in computeDistance
Date: Mon, 09 Nov 2026 12:53:06 +0000
Message-ID: <19854-c255ec23cf0cb87c@postgresql.org> (raw)
The following bug has been logged on the website:
Bug reference: 19854
Logged by: Atish Prakash
Email address: 125atishprak@gmail.com
PostgreSQL version: 18.6
Operating system: Linux x86_64
Description:
Ordering a GiST index scan by distance fails when NaN coordinates are
involved. On an
assert-enabled build, one row with a NaN coordinate takes the server through
crash
recovery. These values are accepted by the input functions.
1. NaN in a stored box:
CREATE TABLE b(c box);
INSERT INTO b VALUES ('(NaN,0),(1,1)');
INSERT INTO b SELECT '(5,5),(6,6)' FROM generate_series(1, 5);
CREATE INDEX ON b USING gist (c);
SET enable_seqscan = off;
SET enable_sort = off;
SELECT count(*) FROM (SELECT c FROM b ORDER BY c <-> '(0,0)'::point) s;
Assert-enabled build:
TRAP: failed Assert("box->low.x <= box->high.x"), File: "gistproc.c",
Line: 1253
Without assertions (19rc1): returns 6.
Polygons and circles with a NaN coordinate crash the assert build the
same way.
Without assertions (19rc1), they fail instead with:
ERROR: inconsistent point values
2. NaN in the query point, with ordinary stored values:
CREATE TABLE p(c polygon);
INSERT INTO p SELECT '((5,5),(6,5),(6,6))' FROM generate_series(1, 5);
CREATE INDEX ON p USING gist (c);
SET enable_seqscan = off;
SET enable_sort = off;
SELECT count(*) FROM (SELECT c FROM p ORDER BY c <-> '(NaN,NaN)'::point)
s;
ERROR: index returned tuples in wrong order
This happens with and without assertions (tested with assertions on 18.6,
19rc1 and
master, and without on 19rc1). The query point is an ordinary parameter
here.
Expected: no assertion failure or internal error, whatever order NaN sorts
in.
computeDistance() in gistproc.c asserts that a box's low bound does not
exceed its high
bound, which does not hold when a bound is NaN, since its comparisons are
all false.
An AddressSanitizer build without assertions reported no memory errors in
these cases,
so this looks like a robustness problem and not a memory-safety one.
NaN handling in the geometric code has come up before, for example in
"Strange behavior
with polygon and NaN":
https://www.postgresql.org/message-id/CAGf+fX70rWFOk5cd00uMfa__0yP+vtQg5ck7c2Onb-Yczp0URA@mail.gmail...
I did not find this assertion reported. If this is considered part of that
known
problem, please treat this as one more example of it.
I have not written a patch.
Found by comparing index and sequential scans across the operator classes
listed in the
catalog. I reproduced and checked the result myself.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-bugs@postgresql.org
Cc: noreply@postgresql.org, 125atishprak@gmail.com
Subject: Re: BUG #19854: GiST distance ordering fails on NaN coordinates, Assert in computeDistance
In-Reply-To: <19854-c255ec23cf0cb87c@postgresql.org>
References: <19854-c255ec23cf0cb87c@postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox