Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Subject: BUG #19524: In `contrib/btree_gist` float4/float8 GiST index
 operations, handling NaN values with raw C operator
To: pgsql-bugs@lists.postgresql.org
From: PG Bug reporting form <noreply@postgresql.org>
Cc: 3020001251@tju.edu.cn
Reply-To: 3020001251@tju.edu.cn, pgsql-bugs@lists.postgresql.org
Date: Thu, 18 Jun 2026 07:52:50 +0000
Message-ID: <19524-9559d302c8455664@postgresql.org>
Auto-Submitted: auto-generated
Archived-At: 
 <https://www.postgresql.org/message-id/19524-9559d302c8455664%40postgresql.org>
Precedence: bulk

The following bug has been logged on the website:

Bug reference:      19524
Logged by:          Yuelin Wang
Email address:      3020001251@tju.edu.cn
PostgreSQL version: 19beta1
Operating system:   Linux (Ubuntu 24.04, x86_64)
Description:       =20

**Component**: `contrib/btree_gist/btree_float4.c`, `btree_float8.c`,
`btree_utils_num.c`

```sql
CREATE EXTENSION btree_gist;

-- Effect 1: EXCLUDE constraint bypass (float4)
CREATE TABLE reservations (
    room   float4,
    during tsrange,
    EXCLUDE USING gist (room WITH =3D, during WITH &&)
);
INSERT INTO reservations VALUES ('NaN'::float4, '[2025-01-01,2025-01-02)');
INSERT INTO reservations VALUES ('NaN'::float4, '[2025-01-01,2025-01-02)');
SELECT COUNT(*) FROM reservations;

-- Effect 2: RLS bypass (float8)
CREATE TABLE measurements (id int, val float8);
CREATE INDEX ON measurements USING gist (val);
INSERT INTO measurements VALUES (1, 'NaN'), (2, 1.5);
ALTER TABLE measurements ENABLE ROW LEVEL SECURITY;
ALTER TABLE measurements FORCE ROW LEVEL SECURITY;
CREATE POLICY hide_nan ON measurements FOR SELECT USING (val !=3D
'NaN'::float8);
CREATE ROLE lowpriv LOGIN;
GRANT SELECT ON measurements TO lowpriv;
SET ROLE lowpriv;
SET enable_seqscan =3D off;
SET enable_bitmapscan =3D off;
SELECT * FROM measurements ORDER BY id;
RESET ROLE;

-- Effect 3: index corruption after page split (float8)
CREATE TABLE t (val float8);
CREATE INDEX ON t USING gist (val);
INSERT INTO t SELECT 'NaN'::float8 FROM generate_series(1, 2000);
SET enable_indexscan =3D off; SET enable_bitmapscan =3D off;
SELECT COUNT(*) AS seqscan_count FROM t WHERE val =3D 'NaN';
RESET ALL;
SET enable_seqscan =3D off; SET enable_bitmapscan =3D off;
SELECT COUNT(*) AS indexscan_count FROM t WHERE val =3D 'NaN';
RESET ALL;
```

Expected vs actual output:

| Query | Expected | Actual |
|---|---|---|
| `SELECT COUNT(*) FROM reservations` | `1` (second insert blocked by
EXCLUDE) | `2` |
| `SELECT * FROM measurements ORDER BY id` (lowpriv, index scan) | `(2,
1.5)` only | `(1, NaN)` and `(2, 1.5)` |
| `seqscan_count` (Effect 3, seq scan forced) | `2000` | `2000` |
| `indexscan_count` (Effect 3, index scan forced) | `2000` | `0` |

The comparison functions in `btree_float4.c` and `btree_float8.c` use raw C
operators (`=3D=3D`, `<`, `>`) on `float4`/`float8` values. Under IEEE 754,=
 all
comparisons involving NaN return false, including `NaN =3D=3D NaN`, which
diverges from PostgreSQL's semantic ordering where NaN equals itself and
sorts above all finite values. This causes two independent failures:
`gbt_float8eq(NaN, NaN)` returns false, so `BtreeGistNotEqualStrategyNumber`
in `btree_utils_num.c:300` incorrectly concludes that NaN satisfies `!=3D
NaN`, and since `gbt_float8_consistent()` unconditionally sets `*recheck =3D
false`, the heap-level filter is never applied. Separately,
`gbt_float8key_cmp(NaN, NaN)` returns -1 instead of 0, violating strict weak
ordering and producing a corrupted GiST tree during `picksplit` that cannot
locate NaN entries.

The fix is to replace the raw C operators in `gbt_float4gt/ge/eq/le/lt` and
`gbt_float8gt/ge/eq/le/lt` with PostgreSQL's `float4_cmp_internal()` and
`float8_cmp_internal()`, which handle NaN correctly. The key comparators
`gbt_float4key_cmp()` and `gbt_float8key_cmp()` need the same update. As a
defensive measure, `*recheck` in the consistent functions should be set to
`true` when the query or key involves NaN.