Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xQiRl-00000000K20-0vzx for pgsql-bugs@arkaria.postgresql.org; Tue, 10 Nov 2026 09:47:21 +0000 Received: from localhost ([127.0.0.1]:57218 helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xQiRj-00000004f6p-02QA for pgsql-bugs@arkaria.postgresql.org; Tue, 10 Nov 2026 09:47:19 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]:45150) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xQOsw-000000013KQ-0w0y for pgsql-bugs@lists.postgresql.org; Mon, 09 Nov 2026 12:54:06 +0000 Received: from mahout.postgresql.org ([2001:4800:3e1:1::227]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xQOsv-000000001pp-1DBb for pgsql-bugs@lists.postgresql.org; Mon, 09 Nov 2026 12:54:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Message-ID:Date:Reply-To:Cc:From:To:Subject: Content-Transfer-Encoding:MIME-Version:Content-Type:Sender:Content-ID: Content-Description:In-Reply-To:References; bh=X7/jgvLUElrzBA59PuYs/V5JkWtIrYe85PqPzWmNK8w=; b=vwWHkUN7ENSE/0vOp+/m+czC5W GoujNgaE7NdPSFH8mEyHfJWmz92/ShafIyEgtRYVpZDOI9CNOFjUtT7JKEZ2AHGSjkS5hIeTud7FE bmYjiUwxAMrA1LnRdoeMqMpX3P8p3uuNhPepR2YH3KTg9YaYg3g29rCg5MaeUz2PfJmpzWKAGBvMo zz4GgOLF6wSULmWx6MCanFa4zj5Zg/gyKkopbVu65V6s+9VyaqFdfTNVincFO8eyDbPZSozMoEgYd D+HcV56Q13IZxn6DLSdxs8axuT3wdxkGezKa3Qi/9I1K9rGmuejbfbXUKBgxdB7qBOCPdwHGmcPXj JU/PwZLw==; Received: from wrigleys.postgresql.org ([2a02:16a8:dc51::60]) by mahout.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xQOsu-000000004Vo-1bki for pgsql-bugs@lists.postgresql.org; Mon, 09 Nov 2026 12:54:04 +0000 Received: from localhost ([127.0.0.1] helo=wrigleys.postgresql.org) by wrigleys.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xQOst-00000000IUj-1BJB for pgsql-bugs@lists.postgresql.org; Mon, 09 Nov 2026 12:54:03 +0000 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: BUG #19854: GiST distance ordering fails on NaN coordinates, Assert in computeDistance To: pgsql-bugs@lists.postgresql.org From: PG Bug reporting form Cc: 125atishprak@gmail.com Reply-To: 125atishprak@gmail.com, pgsql-bugs@lists.postgresql.org Date: Mon, 09 Nov 2026 12:53:06 +0000 Message-ID: <19854-c255ec23cf0cb87c@postgresql.org> X-Auto-Response-Suppress: All Auto-Submitted: auto-generated List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk The following bug has been logged on the website: Bug reference: 19854 Logged by: Atish Prakash Email address: 125atishprak@gmail.com PostgreSQL version: 18.6 Operating system: Linux x86_64 Description: =20 Ordering a GiST index scan by distance fails when NaN coordinates are involved. On an assert-enabled build, one row with a NaN coordinate takes the server through crash recovery. These values are accepted by the input functions. 1. NaN in a stored box: CREATE TABLE b(c box); INSERT INTO b VALUES ('(NaN,0),(1,1)'); INSERT INTO b SELECT '(5,5),(6,6)' FROM generate_series(1, 5); CREATE INDEX ON b USING gist (c); SET enable_seqscan =3D off; SET enable_sort =3D off; SELECT count(*) FROM (SELECT c FROM b ORDER BY c <-> '(0,0)'::point) s; Assert-enabled build: TRAP: failed Assert("box->low.x <=3D box->high.x"), File: "gistproc.c", Line: 1253 Without assertions (19rc1): returns 6. Polygons and circles with a NaN coordinate crash the assert build the same way. Without assertions (19rc1), they fail instead with: ERROR: inconsistent point values 2. NaN in the query point, with ordinary stored values: CREATE TABLE p(c polygon); INSERT INTO p SELECT '((5,5),(6,5),(6,6))' FROM generate_series(1, 5); CREATE INDEX ON p USING gist (c); SET enable_seqscan =3D off; SET enable_sort =3D off; SELECT count(*) FROM (SELECT c FROM p ORDER BY c <-> '(NaN,NaN)'::point) s; ERROR: index returned tuples in wrong order This happens with and without assertions (tested with assertions on 18.6, 19rc1 and master, and without on 19rc1). The query point is an ordinary parameter here. Expected: no assertion failure or internal error, whatever order NaN sorts in. computeDistance() in gistproc.c asserts that a box's low bound does not exceed its high bound, which does not hold when a bound is NaN, since its comparisons are all false. An AddressSanitizer build without assertions reported no memory errors in these cases, so this looks like a robustness problem and not a memory-safety one. NaN handling in the geometric code has come up before, for example in "Strange behavior with polygon and NaN": https://www.postgresql.org/message-id/CAGf+fX70rWFOk5cd00uMfa__0yP+vtQg5ck7= c2Onb-Yczp0URA@mail.gmail.com I did not find this assertion reported. If this is considered part of that known problem, please treat this as one more example of it. I have not written a patch. Found by comparing index and sequential scans across the operator classes listed in the catalog. I reproduced and checked the result myself.