public inbox for [email protected]
help / color / mirror / Atom feedBUG #19512: PG 17.10: SIGSEGV in build_minmax_path (planner) and hash_bytes (HashAgg executor)
2+ messages / 2 participants
[nested] [flat]
* BUG #19512: PG 17.10: SIGSEGV in build_minmax_path (planner) and hash_bytes (HashAgg executor)
@ 2026-06-05 01:49 PG Bug reporting form <[email protected]>
2026-06-05 14:10 ` Re: BUG #19512: PG 17.10: SIGSEGV in build_minmax_path (planner) and hash_bytes (HashAgg executor) Bruce Momjian <[email protected]>
0 siblings, 1 reply; 2+ messages in thread
From: PG Bug reporting form @ 2026-06-05 01:49 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
The following bug has been logged on the website:
Bug reference: 19512
Logged by: Kay
Email address: [email protected]
PostgreSQL version: 17.10
Operating system: Ubuntu 24.04 LTS, kernel 6.8.0-110-generic, x86_64
Description:
PostgreSQL 17.10 (pgdg, Ubuntu 24.04) crashes with SIGSEGV in two
unrelated code paths when planning / executing aggregate queries
against a busy TimescaleDB hypertable. Both stacks are entirely
inside core PG (the TimescaleDB planner hook is on the stack but
the crash address is below it in PG core, not inside the extension's
.so). Five cluster-wide cascade restarts in 2 hours today, plus 5
distinct events over the preceding 72 hours.
We are not seeing these on a 17.9 cluster running similar workload.
ENVIRONMENT
- PostgreSQL 17.10 (pgdg, postgresql-17_17.10-1.pgdg24.04+1, binary
mtime 2026-05-12, installed 2026-05-19)
- Ubuntu 24.04 LTS, kernel 6.8.0-110-generic
- x86_64 AMD EPYC, 16 cores
- TimescaleDB 2.27.1 (2.27.1~ubuntu24.04-1710)
- Workload: ~5-6M rows/h continuous COPY into a 4.7B-row hypertable
("svid_trace"), 1h chunk interval, hypercore/columnstore
compression with segmentby on a high-cardinality text column.
- Non-default GUCs relevant to crashes:
timescaledb.enable_vectorized_aggregation = off (workaround for #9902)
timescaledb.auto_sparse_indexes = off
timescaledb.enable_composite_bloom_indexes = off
timescaledb.enable_sparse_index_bloom = off
max_wal_size = 32GB
data_checksums = off
Everything else default.
CRASH 1 -- PLANNER SIDE, MIN/MAX PATH BUILD
Trigger query (issued by a small bash watchdog every 5 min; ran
without crashing for ~3 weeks and only started SIGSEGV'ing on
2026-06-05):
SELECT COALESCE(EXTRACT(EPOCH FROM (now() - max(ts)))::bigint, 999999)
FROM svid_trace;
svid_trace.ts is timestamptz. There is a btree on (ts) plus several
composite btrees containing ts as the second column.
PG log:
2026-06-05 08:31:24.857 KST [3337534] LOG: server process (PID 4192477)
was terminated by signal 11: Segmentation fault
2026-06-05 08:31:24.857 KST [3337534] DETAIL: Failed process was
running:
SELECT COALESCE(EXTRACT(EPOCH FROM (now() - max(ts)))::bigint,
999999)
FROM svid_trace
Stack trace (coredumpctl info 4192477):
#0 set_base_rel_pathlists (postgres + 0x452c59)
#1 query_planner (postgres + 0x474cbf)
#2 build_minmax_path (postgres + 0x4750e7)
#3 preprocess_minmax_aggregates (postgres + 0x4754c0)
#4 grouping_planner (postgres + 0x476f4b)
#5 subquery_planner (postgres + 0x47a653)
#6 standard_planner (postgres + 0x47a9a8)
#7 pgss_planner (pg_stat_statements.so + 0x6ced)
#8 timescaledb_planner (timescaledb-2.27.1.so + 0x8071e)
#9 planner (postgres + 0x560dcb)
#10 pg_plan_queries (postgres + 0x560ec2)
#11 exec_simple_query (postgres + 0x56248a)
Crash is inside set_base_rel_pathlists during MIN/MAX path
generation.
CRASH 2 -- EXECUTOR SIDE, HASH AGGREGATION
Trigger query (issued every 15 min; fired four times today, crashed
every time, 16 minutes apart):
SELECT time_bucket('15 minutes', ts) AS bucket,
eqp_id, recipe_id, COUNT(*) AS n
FROM svid_trace
WHERE eqp_id = ANY(ARRAY['EQ-A','EQ-B', /* 13 ids total */])
AND recipe_id IS NOT NULL AND recipe_id <> ''
AND ts > '2026-06-04 14:15:00+09'::timestamptz
AND ts <= '2026-06-05 08:30:00+09'::timestamptz + interval '15
minutes'
GROUP BY bucket, eqp_id, recipe_id
ORDER BY bucket, eqp_id, n DESC;
svid_trace is partitioned on ts (1-hour chunk interval). All chunks
in the requested window are compressed (hypercore columnstore),
except the most recent one which is uncompressed.
The query plan when it works (sort-based path, after Workaround 2
below) uses parallel Gather + Sort + GroupAggregate. The crashing
path uses HashAgg on a 3-key group (time_bucket, text, text).
Stack trace (coredumpctl info 4183753):
#0 hash_bytes (postgres + 0x72e480)
#1 hash_any (postgres + 0x1b4a29)
#2 FunctionCall1Coll (postgres + 0x373114)
#3 lookup_hash_entries (postgres + 0x38b178)
#4 agg_fill_hash_table (postgres + 0x38c071)
#5 ExecProcNode (postgres + 0x3b1c36)
#6 ExecProcNode (postgres + 0x375d73)
#7 pgss_ExecutorRun (pg_stat_statements.so + 0x3f85)
#8 ExecutorRun (postgres + 0x566f53)
#9 PortalRun (postgres + 0x568ae8)
#10 exec_simple_query (postgres + 0x5625e1)
Crash is inside hash_bytes while building/probing the HashAgg
table. FunctionCall1Coll on the stack frame just above hash_bytes
strongly suggests a Datum (likely the text recipe_id) being passed
to its hash function with a bad pointer / length.
WHAT WE RULED OUT
1. TimescaleDB: both crash PCs are inside postgres text segment.
The only TS frame on the stack (Crash 1) is timescaledb_planner,
well above the crash. We previously suspected the
ts_executor_end_hook issue we reported against 2.27.1; the new
backtraces show it is not.
2. Disk corruption: data_checksums = off here, so we cannot prove
a negative, but NVMe SMART shows zero media errors, zero
data-integrity errors, 4% wear, 100% spare. The crashing
queries succeed on rerun (e.g. Crash 2 returned 725 rows in
23 ms after Workaround 2), so chunk contents themselves read
out cleanly.
3. Workload spike: pg_stat_activity snapshots from the crash
minute show 6 active backends; load average 1.8-2.5.
WORKAROUNDS DEPLOYED
1. For Crash 1: replaced the hypertable scan with a watermark table
read in our application. SELECT max(ts) FROM svid_trace ->
SELECT max(last_start_ts) FROM extraction_watermark. The
crashing path (preprocess_minmax_aggregates) is only triggered
for the trivial MIN/MAX form on a single column. Any expression
breaks the optimization and avoids the crash.
2. For Crash 2: SET LOCAL enable_hashagg = off in the same
transaction before the GROUP BY. PG plans Sort + GroupAggregate
instead and runs cleanly.
After both workarounds, zero crashes in the following hour.
ADJACENT 17.10 RELEASE-NOTE FIXES WE NOTICED
We checked the 17.10 release notes
(https://www.postgresql.org/docs/release/17.10/) before filing.
Four items sit in or right next to our crash regions:
- "Fix incomplete removal of relation references in RestrictInfo
structs during join removal" (commit 53cb4ec1de7..). The commit
message mentions a use-after-free that only manifests "if the
freed space gets claimed by some List node before a Bitmapset
can be put there." Same memory-race shape as our
non-deterministic Crash 1 -- but our minimal trigger query has
no JOIN at all.
- "Check for nondeterministic collations before assuming
uniqueness from equality."
- "Fix incorrect logic for hashed IN / NOT IN with non-strict
equality operator (... could crash or give wrong answers in hash
aggregation)."
- "Fix 'no relation entry for relid 0' in set operations."
It is possible our Crash 2 is a still-unfixed sibling of the third
item. We have not been able to find a pgsql-bugs report for our
exact stacks (search on "set_base_rel_pathlists build_minmax_path"
returns 0 hits for the past year).
WHAT WE CAN PROVIDE ON REQUEST
- The .zst core files for both PIDs are kept locally for 7 more
days. We cannot upload them publicly (they include hypertable
buffer contents) but can run gdb on them and paste any specific
request.
- A second cluster running 17.9 + TimescaleDB 2.27.1 with similar
workload -- we can compare behaviour.
- Optional: we can downgrade one node to 17.9 and re-issue the
trigger queries to confirm 17.10 regression. Happy to do this
if it would help triage.
We will follow up with a minimal SQL+data reproducer if we can
produce one outside of production traffic -- open to suggestions
on how to construct one given the planner side appears to depend
on cumulative memory state.
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: BUG #19512: PG 17.10: SIGSEGV in build_minmax_path (planner) and hash_bytes (HashAgg executor)
2026-06-05 01:49 BUG #19512: PG 17.10: SIGSEGV in build_minmax_path (planner) and hash_bytes (HashAgg executor) PG Bug reporting form <[email protected]>
@ 2026-06-05 14:10 ` Bruce Momjian <[email protected]>
0 siblings, 0 replies; 2+ messages in thread
From: Bruce Momjian @ 2026-06-05 14:10 UTC (permalink / raw)
To: [email protected]; [email protected]
On Fri, Jun 5, 2026 at 01:49:46AM +0000, PG Bug reporting form wrote:
> The following bug has been logged on the website:
>
> Bug reference: 19512
> Logged by: Kay
> Email address: [email protected]
> PostgreSQL version: 17.10
> Operating system: Ubuntu 24.04 LTS, kernel 6.8.0-110-generic, x86_64
> Description:
>
> PostgreSQL 17.10 (pgdg, Ubuntu 24.04) crashes with SIGSEGV in two
> unrelated code paths when planning / executing aggregate queries
> against a busy TimescaleDB hypertable. Both stacks are entirely
> inside core PG (the TimescaleDB planner hook is on the stack but
> the crash address is below it in PG core, not inside the extension's
> .so). Five cluster-wide cascade restarts in 2 hours today, plus 5
> distinct events over the preceding 72 hours.
The only thing I can add is that it is possible for an extension like
TimescaleDB to cause a crash in the Postgres sections of the code.
--
Bruce Momjian <[email protected]> https://momjian.us
EDB https://enterprisedb.com
Do not let urgent matters crowd out time for investment in the future.
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2026-06-05 14:10 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-06-05 01:49 BUG #19512: PG 17.10: SIGSEGV in build_minmax_path (planner) and hash_bytes (HashAgg executor) PG Bug reporting form <[email protected]>
2026-06-05 14:10 ` Bruce Momjian <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox