Subject: Autoprewarm workers terminated due to a segmentation fault

From: Glauber Batista <[email-masked]>
Date: Tue, 9 Jun 2026 15:37:24 -0300

Hello,

I have an issue with the autoprewarm workers segfaulting during the service
restart. Sometimes, it successfully restarts after a few tries, but usually
I need to remove the autoprewarm.blocks file. My setup consists of a
primary server with two replicas and all of them present the same issue. I
have been using this setup for several years with no issues, but since I
upgraded to Postgres 18 I'm having it. This is a production database.

Details:

Postgres Version: PostgreSQL 18.3 on aarch64-unknown-linux-gnu, compiled by
gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, 64-bit

This is the related settings I'm using in postgresql.conf
```
shared_preload_libraries = 'pg_stat_statements,pg_prewarm'
pg_prewarm.autoprewarm = True
pg_prewarm.autoprewarm_interval = 300s
```

I'm using systemd to manage Postgres, but it also happens if I start
postgres using `pg_ctl`. So I ruled out a systemd issue.

This is the error message I'm seeing.
```
LOG:  restored log file "000000010000079200000015" from archive
LOG:  consistent recovery state reached at 792/15A586C8
LOG:  database system is ready to accept read-only connections
LOG:  restored log file "000000010000079200000016" from archive
LOG:  restored log file "000000010000079200000017" from archive
LOG:  background worker "autoprewarm worker" (PID 2350) was terminated by
signal 11: Segmentation fault
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted while in recovery at log time
2026-06-09 17:34:19 UTC
HINT:  If this has occurred more than once some data might be corrupted and
you might need to choose an earlier recovery target.
LOG:  restored log file "00000001000007920000000D" from archive
LOG:  restored log file "0000000100000791000000F8" from archive
LOG:  entering standby mode
LOG:  redo starts at 791/F804CB38
LOG:  database system is ready to accept read-only connections
LOG:  restored log file "0000000100000791000000F9" from archive
LOG:  restored log file "0000000100000791000000FA" from archive
LOG:  restored log file "0000000100000791000000FB" from archive
LOG:  background worker "autoprewarm worker" (PID 2522) was terminated by
signal 11: Segmentation fault
LOG:  terminating any other active server processes
FATAL:  could not restore file "0000000100000791000000FC" from archive:
child process was terminated by signal 3: Quit
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted while in recovery at log time
2026-06-09 17:34:19 UTC
HINT:  If this has occurred more than once some data might be corrupted and
you might need to choose an earlier recovery target.
LOG:  restored log file "00000001000007920000000D" from archive
LOG:  restored log file "0000000100000791000000F8" from archive
LOG:  entering standby mode
```
Changing the debug level to DEBUG1 didn't show anything useful, so I'm not
pasting it here.

At first, I thought it could be somehow related to this bug:
https://www.datadoghq.com/blog/engineering/unraveling-a-postgres-segfault/,
but investigating a bit it does not seem to be the case. Either way, I
compiled Postgres again using CFLAGS="-O0 -g -fno-strict-aliasing" to check
if it was something related to the code optimization for ARM64, but the
issue persisted.

Then, I used gdb to get the core file. Since it has been quite some time
since I didn't debug anything written in C/C++, I used Claude to guide me.
Here's some info:
```
(gdb) frame 0
#0  0x0000f5a0c6003854 in autoprewarm_database_main (main_arg=0) at
autoprewarm.c:649
649                blk = block_info[i];
(gdb) list
644
645                read_stream_end(stream);
646
647                /* Advance i past all the blocks just prewarmed. */
648                i = p.pos;
649                blk = block_info[i];
650            }
651
652            relation_close(rel, AccessShareLock);
653            CommitTransactionCommand();
(gdb) p *block_info
$1 = {database = 0, tablespace = 0, filenumber = 0, forknum = MAIN_FORKNUM,
blocknum = 0}
(gdb) p p
$2 = {block_info = 0xf5a07f600000, pos = 131072, tablespace = 1663,
filenumber = 28197, forknum = MAIN_FORKNUM, nblocks = 65329}
(gdb) p stream
$3 = (ReadStream *) 0xb82a946671f0
(gdb) p *(ReadStream *) stream
$4 = {max_ios = 0, io_combine_limit = 0, ios_in_progress = 0, queue_size =
0, max_pinned_buffers = 225, forwarded_buffers = 0, pinned_buffers = 0,
distance = 1,
  initialized_buffers = 9, read_buffers_flags = 0, sync_mode = false,
batch_mode = true, advice_enabled = false, temporary = false,
buffered_blocknum = 4294967295,
  callback = 0xf5a0c600330c <apw_read_stream_next_block>,
callback_private_data = 0xfffff758cea8, seq_blocknum = 2403,
seq_until_processed = 4294967295,
  pending_read_blocknum = 2403, pending_read_nblocks = 0,
per_buffer_data_size = 0, per_buffer_data = 0x0, ios = 0xb82a94667618,
oldest_io_index = 8, next_io_index = 8,
  fast_path = false, oldest_buffer_index = 9, next_buffer_index = 9,
buffers = 0xb82a94667254}
(gdb) p p.pos
$5 = 131072
(gdb) p p.nblocks
$6 = 65329
(gdb) p apw_state->prewarm_stop_idx
$7 = 0
(gdb) p apw_state->prewarm_start_idx
$8 = 0
(gdb) p block_info[131072]
Cannot access memory at address 0xf5a07f880000
(gdb) p &block_info[131072]
$9 = (BlockInfoRecord *) 0xf5a07f880000
(gdb) p block_info[131071]
$10 = {database = 0, tablespace = 0, filenumber = 0, forknum =
MAIN_FORKNUM, blocknum = 0}
(gdb) p block_info[131070]
$11 = {database = 0, tablespace = 0, filenumber = 0, forknum =
MAIN_FORKNUM, blocknum = 0}
(gdb) p block_info[1]
$12 = {database = 0, tablespace = 0, filenumber = 0, forknum =
MAIN_FORKNUM, blocknum = 0}
(gdb) p apw_state->prewarmed_blocks
$13 = 0
(gdb) p *apw_state
$14 = {lock = {tranche = 0, state = {value = 0}, waiters = {head = 0, tail
= 0}}, bgworker_pid = 0, pid_using_dumpfile = 0, block_info_handle = 0,
database = 0,
  prewarm_start_idx = 0, prewarm_stop_idx = 0, prewarmed_blocks = 0}
(gdb) info line autoprewarm.c:649
Line 649 of "autoprewarm.c" starts at address 0xf5a0c600382c
<autoprewarm_database_main+936> and ends at 0xf5a0c600384c
<autoprewarm_database_main+968>.
(gdb) list autoprewarm.c:640,660
640                {
641                    apw_state->prewarmed_blocks++;
642                    ReleaseBuffer(buf);
643                }
644
645                read_stream_end(stream);
646
647                /* Advance i past all the blocks just prewarmed. */
648                i = p.pos;
649                blk = block_info[i];
650            }
651
652            relation_close(rel, AccessShareLock);
653            CommitTransactionCommand();
654        }
655
656        dsm_detach(seg);
657    }
658
659    /*
660     * Dump information on blocks in shared buffers.  We use a text
format here
```

I found out that some parts of the autoprewarm were re-written recently
here:
https://www.postgresql.org/message-id/flat/CAN55FZ3n8Gd%2BhajbL%3D5UkGzu_aHGRqnn%2BxktXq2fuds%3D1AOR6Q%40mail.gmail.com
and I think it could be related, given the data present in the dump.

Also, I inspected my data to ensure it was not the culprit. I got the
database and filenumber from `bt full` and run the following queries.
```
blk = {database = 23583, tablespace = 1663, filenumber = 28197, forknum =
MAIN_FORKNUM, blocknum = 49}
```
So I queried the database (23583) and the filenumber (28197) using
```
SELECT datname FROM pg_database WHERE oid = 23583;
\c <datname>
SELECT relname, relkind, relpages, pg_relation_filenode(oid) AS filenode
FROM pg_class
WHERE pg_relation_filenode(oid) = 28197;
```
and it returned
```
   relname    | relkind | relpages | filenode
--------------+---------+----------+----------
 lccss_cc_idx | i       |    64805 |    28197
```
So I think it's not stale data or shrunk relation caused by VACUUM,
TRUNCATE, etc.

I'm only reporting with the latest data I collected, but this issue has
been happening since April, when I upgraded the database.

All that said, it seems there's a missing guard-clause at line 649. I
didn't spend much time reading the code, but it's clearly accessing a
position in the array that is not allocated.

Let me know if any extra information is needed.

Best,
Glauber Cassiano Batista

---

From: "Matheus Alcantara" <[email-masked]>
Date: Tue, 09 Jun 2026 18:06:09 -0300

Hi,

On Tue Jun 9, 2026 at 3:37 PM -03, Glauber Batista wrote:
> [ ... ] 

Thank you for the report!

I've managed to reproduce the issue with the following:

create table test_warm (id int, data text);
-- insert enough rows to generate too many pages on an index
insert into test_warm select g, repeat('a', 100) from generate_series(1, 5000000) g;
create index warm_idx on test_warm(id);

-- force read the index entirely into shared_buffers
select count(*) from test_warm where id > 0;

Then pg_ctl stop and pg_ctl start will start failing with the following logs:

2026-06-09 17:47:40.924 -03 [23025] LOG:  shutting down
2026-06-09 17:47:40.925 -03 [23025] LOG:  checkpoint starting: shutdown fast
2026-06-09 17:47:41.033 -03 [23022] LOG:  database system is shut down
2026-06-09 17:47:49.830 -03 [23172] LOG:  starting PostgreSQL 19beta1 on aarch64-darwin, compiled by clang-17.0.0, 64-bit
2026-06-09 17:47:49.842 -03 [23172] LOG:  database system is ready to accept connections
2026-06-09 17:47:49.917 -03 [23172] LOG:  background worker "autoprewarm worker" (PID 23182) was terminated by signal 11: Segmentation fault: 11
2026-06-09 17:47:49.917 -03 [23172] LOG:  terminating any other active server processes
2026-06-09 17:47:49.918 -03 [23172] LOG:  all server processes terminated; reinitializing

Wondering if the following would be enough?

  /* Advance i past all the blocks just prewarmed. */
  i = p.pos;
+ if (i >= apw_state->prewarm_stop_idx)
+         break;
+
  blk = block_info[i];

With this change on a fresh start I got very similar shared hit buffers
on the first and second execution, so I think that the
autoprewarm.blocks was reloaded successfully?

postgres=# explain(analyze, costs off, timing off, summary off) select count(*) from test_warm where id > 0;
                                    QUERY PLAN
-----------------------------------------------------------------------------------
 Finalize Aggregate (actual rows=1.00 loops=1)
   Buffers: shared hit=15965 read=70242
   ->  Gather (actual rows=3.00 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=15965 read=70242
         ->  Partial Aggregate (actual rows=1.00 loops=3)
               Buffers: shared hit=15965 read=70242
               ->  Parallel Seq Scan on test_warm (actual rows=1666666.67 loops=3)
                     Filter: (id > 0)
                     Buffers: shared hit=15965 read=70242

postgres=# explain(analyze, costs off, timing off, summary off) SELECT count(*) FROM test_warm WHERE id > 0;
                                    QUERY PLAN
-----------------------------------------------------------------------------------
 Finalize Aggregate (actual rows=1.00 loops=1)
   Buffers: shared hit=15970 read=70237
   ->  Gather (actual rows=3.00 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         Buffers: shared hit=15970 read=70237
         ->  Partial Aggregate (actual rows=1.00 loops=3)
               Buffers: shared hit=15970 read=70237
               ->  Parallel Seq Scan on test_warm (actual rows=1666666.67 loops=3)
                     Filter: (id > 0)
                     Buffers: shared hit=15970 read=70237

Thoughts?

--
Matheus Alcantara
EDB: https://www.enterprisedb.com
From 6a3e3e26c0328e15fc47091684143dc5bb8ef6fc Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <[email-masked]>
Date: Tue, 9 Jun 2026 17:59:20 -0300
Subject: [PATCH] Fix out-of-bounds access in autoprewarm worker

The read stream callback apw_read_stream_next_block() advances p->pos
through the block_info array until it reaches a block belonging to a
different relation/fork or hits prewarm_stop_idx. When the stream
finishes processing all blocks for a fork, p.pos may equal
prewarm_stop_idx which cause a segfault.

Add a bounds check before accessing block_info[i] to prevent the crash.

Author: Matheus Alcantara <[email-masked]>
Reported-by: Glauber Batista <[email-masked]>
Discussion: https://www.postgresql.org/message-id/CAO%2B_mTQgQyTYwDh%3DU8iTnsDmOGyWsZJjUV31SmEYwmw6_xY6Bw%40mail.gmail.com
---
 contrib/pg_prewarm/autoprewarm.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/contrib/pg_prewarm/autoprewarm.c b/contrib/pg_prewarm/autoprewarm.c
index ba0bc8e6d4a..a77e52c59ea 100644
--- a/contrib/pg_prewarm/autoprewarm.c
+++ b/contrib/pg_prewarm/autoprewarm.c
@@ -643,8 +643,15 @@ autoprewarm_database_main(Datum main_arg)
 
 			read_stream_end(stream);
 
-			/* Advance i past all the blocks just prewarmed. */
+			/*
+			 * Advance i past all the blocks just prewarmed. The read stream
+			 * callback may have advanced p.pos to prewarm_stop_idx, so we
+			 * must check bounds before accessing block_info[i].
+			 */
 			i = p.pos;
+			if (i >= apw_state->prewarm_stop_idx)
+				break;
+
 			blk = block_info[i];
 		}
 
-- 
2.50.1 (Apple Git-155)

---

From: Tomas Vondra <[email-masked]>
Date: Tue, 9 Jun 2026 23:44:28 +0200

> + if (i >= apw_state->prewarm_stop_idx)
So how does it get advanced past the prewarm_stop_idx? I've been unable
to reproduce it locally, maybe it's platform-specific. The original
report was from ARM, are you on ARM too, Matheus?

But AFAIK the code may not account for read stream callback updating the
pos to prewarm_stop_idx? The callback may end with (p->pos =
apw_state->prewarm_stop_idx), and that seems to be past the end of the
array.

That'd mean the proposed check is generally the correct way to fix this.
TBH it's not clear to me why this needs to set the *next* entry at the
end of the loop. Well, it does that so that the loop condition can use
'blk', but that seems a bit fragile / confusing, and no one noticed the
issue.

Maybe this would be a better way to write the while loop?

    while (i < apw_state->prewarm_stop_idx)
    {
        blk = block_info[i];

        if (blk.tablespace != tablespace ||
            blk.filenumber != filenumber)
            break;

        ...
    }

regards

-- 
Tomas Vondra

---

From: "Matheus Alcantara" <[email-masked]>
Date: Tue, 09 Jun 2026 19:25:55 -0300

> report was from ARM, are you on ARM too, Matheus?

Yes, I'm also on ARM. I also set pg_prewarm.autoprewarm_interval=10s on
postgresql.conf, not sure if it make more easier to reproduce.

> apw_state->prewarm_stop_idx), and that seems to be past the end of the

Yes, this is my understanding.

>             blk.filenumber != filenumber)

Is attached patch what are you sugesting? If yes, I agree that looks
better, it's more safe and easier to understand.

--
Matheus Alcantara
EDB: https://www.enterprisedb.com
From ebf5296026b6f07fb30821ee0603fff24817001b Mon Sep 17 00:00:00 2001
From: Matheus Alcantara <[email-masked]>
Date: Tue, 9 Jun 2026 17:59:20 -0300
Subject: [PATCH v2] Fix out-of-bounds access in autoprewarm worker

The read stream callback apw_read_stream_next_block() advances p->pos
through the block_info array. When processing the last block, it
increments p->pos to prewarm_stop_idx before returning. The callback
itself is safe because it checks bounds before accessing the array.

However, the caller assigned blk from block_info[i] at the end of the
loop body, before the loop condition was re-evaluated. When i equaled
prewarm_stop_idx, this accessed memory beyond the allocated DSM segment,
causing a segfault.

Restructure the loop to check bounds at the top and assign blk at the
beginning of the loop body, where it is always safe. This avoids the
need for an explicit bounds check at the end.

Author: Matheus Alcantara <[email-masked]>
Reported-by: Glauber Batista <[email-masked]>
Discussion: https://www.postgresql.org/message-id/CAO%2B_mTQgQyTYwDh%3DU8iTnsDmOGyWsZJjUV31SmEYwmw6_xY6Bw%40mail.gmail.com
---
 contrib/pg_prewarm/autoprewarm.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/contrib/pg_prewarm/autoprewarm.c b/contrib/pg_prewarm/autoprewarm.c
index ba0bc8e6d4a..f3569b12d11 100644
--- a/contrib/pg_prewarm/autoprewarm.c
+++ b/contrib/pg_prewarm/autoprewarm.c
@@ -572,16 +572,23 @@ autoprewarm_database_main(Datum main_arg)
 		 * valid forks or run out of options, we'll close the relation and
 		 * move on.
 		 */
-		while (i < apw_state->prewarm_stop_idx &&
-			   blk.tablespace == tablespace &&
-			   blk.filenumber == filenumber)
+		while (i < apw_state->prewarm_stop_idx)
 		{
-			ForkNumber	forknum = blk.forknum;
+			ForkNumber	forknum;
 			BlockNumber nblocks;
 			struct AutoPrewarmReadStreamData p;
 			ReadStream *stream;
 			Buffer		buf;
 
+			blk = block_info[i];
+
+			/* Stop when we reach a different relation. */
+			if (blk.tablespace != tablespace ||
+				blk.filenumber != filenumber)
+				break;
+
+			forknum = blk.forknum;
+
 			/*
 			 * smgrexists is not safe for illegal forknum, hence check whether
 			 * the passed forknum is valid before using it in smgrexists.
@@ -645,7 +652,6 @@ autoprewarm_database_main(Datum main_arg)
 
-			blk = block_info[i];
 		}
 
 		relation_close(rel, AccessShareLock);
-- 
2.50.1 (Apple Git-155)

---

From: Tomas Vondra <[email-masked]>
Date: Thu, 11 Jun 2026 16:40:37 +0200

> better, it's more safe and easier to understand.

Yeah, that's roughly what I had in mind.

After looking a bit closer, I think this bug was introduced by

    commit 6acab8bdbcda735ef47b1bb0ba2284d6c465cd88
    Author: Melanie Plageman <[email-masked]>
    Date:   Fri Apr 4 15:25:27 2025 -0400

    Refactor autoprewarm_database_main() in preparation for read stream

which happens to advance to the next block in a couple places

    blk = block_info[++i];

before we know it the incremented "i" is a valid element. The following
commit (d9c7911e1a5f adding the read stream) ends up doing the same
thing, except the index is incremented in a callback.

Melanie, do you agree with the proposed fix?

regards

-- 
Tomas Vondra

---

From: Melanie Plageman <[email-masked]>
Date: Mon, 15 Jun 2026 10:35:43 -0400

> Melanie, do you agree with the proposed fix?

Thanks all for investigating this! I agree that the proposed fix is
correct (both v1 and v2). I am a bit torn between the two versions.
v1:

i = p.pos;
if (i >= apw_state->prewarm_stop_idx)
    break;
blk = block_info[i];

preserves what I had intended in that it sets blk as close to where i
was advanced as possible. I had wanted this behavior because we have
the "fast-forwarding" that happens in a few places in that loop, and I
was afraid it would get quite confusing. The other thing I wanted was
the invariant that blk is always the i'th block when exiting the loop.
This was in case future code restructured the outer loop and added
other break conditions and then used blk.

However, I do hate duplicating the loop bounds checking in two places.
And future code that restructures the control flow of this function is
going to have to be extremely careful anyway. The combination of
multiple nested loops, "fast-forwarding" behavior, and skipping
forward X number of blocks with the read stream made the requirements
of this function's control flow quite complex.

v2:

            ForkNumber    forknum;

            blk = block_info[i];

            /* Stop when we reach a different relation. */
            if (blk.tablespace != tablespace ||
                blk.filenumber != filenumber)
                break;

has the advantage that we set blk directly before using it instead of
at the end of the previous iteration, which people may find less
confusing.

So, overall, I am fine with either approach. Whatever other hackers
find more clear is the right approach. In my attempt to make it clear,
I obviously made it incorrect -- which is far worse :)

- Melanie

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 26 Jun 2026 19:55:51 +0200

FWIW I've committed the v2 fix (I liked it more), and I've backpatched
it to 18.

regards

-- 
Tomas Vondra

---

From: Matheus Alcantara <[email-masked]>
Date: Fri, 26 Jun 2026 15:07:44 -0300

> FWIW I've committed the v2 fix (I liked it more), and I've backpatched

Thank you!

---

From: Melanie Plageman <[email-masked]>
Date: Mon, 29 Jun 2026 11:12:50 -0400

> FWIW I've committed the v2 fix (I liked it more), and I've backpatched

Thank you so much for doing this. I had it on my list for next week,
but a few other things got ahead of it. I really, really appreciate
you taking on committing it!

- Melanie

---

From: Glauber Batista <[email-masked]>
Date: Wed, 1 Jul 2026 16:16:45 -0300

Hello,

I finally got the chance and time to test the patch. I'm using it on a
replica for now and it's working as expected.

2026-07-01 19:09:35 UTC::@:[169848]:LOG:  database system is ready to
accept read-only connections
2026-07-01 19:09:35 UTC::@:[174803]:LOG:  started streaming WAL from
primary at B7E/24000000 on timeline 1
2026-07-01 19:09:35 UTC::@:[174804]:LOG:  autoprewarm successfully
prewarmed 0 of 131072 previously-loaded blocks

Thank you all for your hard work!

Best,
Glauber Cassiano Batista