pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedFix ReadStream handling when heap rescan strategy changes
8+ messages / 5 participants
[nested] [flat]
* Fix ReadStream handling when heap rescan strategy changes
@ 2026-11-03 06:40 Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-03 10:23 ` Re: Fix ReadStream handling when heap rescan strategy changes Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-11-17 21:54 ` Re: Fix ReadStream handling when heap rescan strategy changes Melanie Plageman <melanieplageman@gmail.com>
0 siblings, 2 replies; 8+ messages in thread
From: Bertrand Drouvot @ 2026-11-03 06:40 UTC (permalink / raw)
To: pgsql-hackers@lists.postgresql.org; +Cc: Melanie Plageman <melanieplageman@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
Hi hackers,
While working on [1], I noticed that a heap rescan can change whether a buffer
access strategy is used while retaining the existing ReadStream. If the strategy
is disabled, the stream keeps a dangling pointer after it is freed. If it is
enabled, the stream continues using its NULL strategy and the queue capacity
selected at creation.
The issue was introduced by b7b0f3f27241. Nazir pointed it out and Yuhang
proposed fixing it with a new read_stream_set_strategy() API in [2]. Since the
issue is independent of the streaming SampleScan changes, I wonder if it wouldn't
make sense to fix it separately, as in the attached?
0001 takes a different approach (than [2]) and recreates the ReadStream when
initscan() changes whether a strategy is active. Ordinary rescans continue to
reset and reuse the existing stream. It uses the existing ReadStream APIs, which
could also make the backpatches easier.
0002 contains a test. I'm not sure it is worth adding to the tree given the amount
of setup, but it demonstrates the issue.
[1]: https://postgr.es/m/auCYYbDyrQEwIsQq%40bdtpg
[2]: https://postgr.es/m/CAN55FZ3Lik2m9J%2BEQjzEHSMSBNgAXU5Myt8LFg%3DD2OQ0i_jUMQ%40mail.gmail.com
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments: download all (2 files, .tar.gz)
v1-0001-Fix-ReadStream-handling-when-heap-rescan-strategy.patch (text/x-diff, 6.6K) download
From f3b3c20c4291a0767eb50fb22fae0493f602a371 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 2 Nov 2026 14:18:02 +0000
Subject: [PATCH v1 1/2] Fix ReadStream handling when heap rescan strategy
changes
b7b0f3f27241 made heap scans keep their ReadStream across rescans. However,
initscan() may add or remove rs_strategy when the relation size or caller flags
change.
ReadStream stores the strategy pointer in its ReadBuffersOperation objects and
sizes its buffer queue when it is created. A rescan that enables a bulk read
strategy therefore leaves the existing stream using its NULL strategy and old
queue capacity. A rescan that disables the strategy frees it while the stream
still refers to it, leaving a dangling pointer.
Fix this by recreating the stream when the scan switches between using and
not using a strategy. When instrumentation is enabled, reattach the accumulated
statistics and preserve the maximum capacity across streams.
Reported-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reviewed-by:
Discussion: https://postgr.es/m/
---
src/backend/access/heap/heapam.c | 119 ++++++++++++++++----------
src/backend/storage/aio/read_stream.c | 3 +-
2 files changed, 76 insertions(+), 46 deletions(-)
91.7% src/backend/access/heap/
8.2% src/backend/storage/aio/
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 6a09a1a1517..bd3cb3b6b6c 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -352,6 +352,57 @@ bitmapheap_stream_read_next(ReadStream *pgsr, void *private_data,
Assert(false);
}
+/*
+ * Start the read stream used by a heap scan.
+ *
+ * The caller must call initscan() first, because it sets up the buffer access
+ * strategy passed to the read stream API. The scan must not already own a read
+ * stream.
+ */
+static void
+heap_scan_read_stream_begin(HeapScanDesc scan)
+{
+ Assert(scan->rs_read_stream == NULL);
+
+ if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN ||
+ scan->rs_base.rs_flags & SO_TYPE_TIDRANGESCAN)
+ {
+ ReadStreamBlockNumberCB cb;
+
+ if (scan->rs_base.rs_parallel)
+ cb = heap_scan_stream_read_next_parallel;
+ else
+ cb = heap_scan_stream_read_next_serial;
+
+ /* ---
+ * It is safe to use batchmode as the only locks taken by `cb`
+ * are never taken while waiting for IO:
+ * - SyncScanLock is used in the non-parallel case
+ * - in the parallel case, only spinlocks and atomics are used
+ * ---
+ */
+ scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL |
+ READ_STREAM_USE_BATCHING,
+ scan->rs_strategy,
+ scan->rs_base.rs_rd,
+ MAIN_FORKNUM,
+ cb,
+ scan,
+ 0);
+ }
+ else if (scan->rs_base.rs_flags & SO_TYPE_BITMAPSCAN)
+ {
+ scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_DEFAULT |
+ READ_STREAM_USE_BATCHING,
+ scan->rs_strategy,
+ scan->rs_base.rs_rd,
+ MAIN_FORKNUM,
+ bitmapheap_stream_read_next,
+ scan,
+ sizeof(TBMIterateResult));
+ }
+}
+
/* ----------------
* initscan - scan code common to heap_beginscan and heap_rescan
* ----------------
@@ -403,6 +454,18 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock)
else
allow_strat = allow_sync = false;
+ /*
+ * A read stream keeps the strategy passed at creation and sizes its
+ * buffer queue accordingly. End it before replacing or freeing that
+ * strategy. heap_rescan() will create a new stream after the scan is
+ * reinitialized.
+ */
+ if (scan->rs_read_stream && allow_strat != (scan->rs_strategy != NULL))
+ {
+ read_stream_end(scan->rs_read_stream);
+ scan->rs_read_stream = NULL;
+ }
+
if (allow_strat)
{
/* During a rescan, keep the previous strategy object. */
@@ -1270,52 +1333,9 @@ heap_beginscan(Relation relation, Snapshot snapshot,
else
scan->rs_base.rs_key = NULL;
- initscan(scan, key, false);
-
scan->rs_read_stream = NULL;
-
- /*
- * Set up a read stream for sequential scans and TID range scans. This
- * should be done after initscan() because initscan() allocates the
- * BufferAccessStrategy object passed to the read stream API.
- */
- if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN ||
- scan->rs_base.rs_flags & SO_TYPE_TIDRANGESCAN)
- {
- ReadStreamBlockNumberCB cb;
-
- if (scan->rs_base.rs_parallel)
- cb = heap_scan_stream_read_next_parallel;
- else
- cb = heap_scan_stream_read_next_serial;
-
- /* ---
- * It is safe to use batchmode as the only locks taken by `cb`
- * are never taken while waiting for IO:
- * - SyncScanLock is used in the non-parallel case
- * - in the parallel case, only spinlocks and atomics are used
- * ---
- */
- scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL |
- READ_STREAM_USE_BATCHING,
- scan->rs_strategy,
- scan->rs_base.rs_rd,
- MAIN_FORKNUM,
- cb,
- scan,
- 0);
- }
- else if (scan->rs_base.rs_flags & SO_TYPE_BITMAPSCAN)
- {
- scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_DEFAULT |
- READ_STREAM_USE_BATCHING,
- scan->rs_strategy,
- scan->rs_base.rs_rd,
- MAIN_FORKNUM,
- bitmapheap_stream_read_next,
- scan,
- sizeof(TBMIterateResult));
- }
+ initscan(scan, key, false);
+ heap_scan_read_stream_begin(scan);
/* enable read stream instrumentation */
if ((flags & SO_SCAN_INSTRUMENT) && (scan->rs_read_stream != NULL))
@@ -1387,6 +1407,15 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params,
* reinitialize scan descriptor
*/
initscan(scan, key, true);
+
+ if (scan->rs_read_stream == NULL)
+ {
+ heap_scan_read_stream_begin(scan);
+
+ if (scan->rs_read_stream && scan->rs_base.rs_instrument)
+ read_stream_enable_stats(scan->rs_read_stream,
+ &scan->rs_base.rs_instrument->io);
+ }
}
void
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index fddf6ef441d..b6269321aff 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -255,7 +255,8 @@ void
read_stream_enable_stats(ReadStream *stream, IOStats *stats)
{
stream->stats = stats;
- if (stream->stats)
+ if (stream->stats &&
+ stream->stats->distance_capacity < stream->max_pinned_buffers)
stream->stats->distance_capacity = stream->max_pinned_buffers;
}
--
2.34.1
v1-0002-Test-heap-rescan-strategy-changes.patch (text/x-diff, 3.1K) download
From a7b9bca6236a68f77ed2ee18eb08a9081345e768 Mon Sep 17 00:00:00 2001
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Date: Mon, 2 Nov 2026 14:55:40 +0000
Subject: [PATCH v1 2/2] Test heap rescan strategy changes
---
src/test/modules/test_misc/meson.build | 1 +
.../test_misc/t/016_heap_rescan_strategy.pl | 81 +++++++++++++++++++
2 files changed, 82 insertions(+)
97.9% src/test/modules/test_misc/t/
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index 5d81f5b13be..b5e98fd0233 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -24,6 +24,7 @@ tests += {
't/013_temp_obj_multisession.pl',
't/014_log_statement_max_length.pl',
't/015_temp_schema_exit_deferrable.pl',
+ 't/016_heap_rescan_strategy.pl',
],
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
diff --git a/src/test/modules/test_misc/t/016_heap_rescan_strategy.pl b/src/test/modules/test_misc/t/016_heap_rescan_strategy.pl
new file mode 100644
index 00000000000..01007a3c737
--- /dev/null
+++ b/src/test/modules/test_misc/t/016_heap_rescan_strategy.pl
@@ -0,0 +1,81 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Set a low threshold for selecting a bulk-read strategy.
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf', qq{
+shared_buffers = '1MB'
+max_parallel_workers_per_gather = 0
+});
+$node->start;
+
+$node->safe_psql(
+ 'postgres', q{
+CREATE TABLE growing AS
+ SELECT g, repeat('x', 1000) AS payload
+ FROM generate_series(1, 16) AS g;
+CREATE TABLE growth_done (done boolean PRIMARY KEY);
+
+CREATE FUNCTION grow_table(should_grow boolean)
+RETURNS integer
+LANGUAGE plpgsql
+VOLATILE
+AS $$
+DECLARE
+ inserted boolean;
+BEGIN
+ IF should_grow THEN
+ INSERT INTO growth_done VALUES (true)
+ ON CONFLICT DO NOTHING
+ RETURNING true INTO inserted;
+
+ IF inserted THEN
+ INSERT INTO growing
+ SELECT g, repeat('y', 1000)
+ FROM generate_series(17, 4000) AS g;
+ END IF;
+ END IF;
+
+ RETURN 1;
+END
+$$;
+});
+
+# Grow the relation during the second scan. The third rescan must recreate
+# its read stream with the newly selected bulk-read strategy.
+my $bulkread = $node->safe_psql(
+ 'postgres', q{
+SET enable_material = off;
+SET enable_memoize = off;
+SELECT count(*) AS scan_count
+FROM (VALUES (1, false), (2, true), (3, false)) AS v(ord, should_grow)
+CROSS JOIN LATERAL (
+ SELECT *
+ FROM growing
+ WHERE g >= grow_table(v.should_grow)
+ OFFSET 0
+) AS s
+\gset
+
+SELECT pg_stat_force_next_flush()
+\gset
+
+SELECT COALESCE(sum(reads), 0) + COALESCE(sum(hits), 0)
+FROM pg_stat_get_backend_io(pg_backend_pid())
+WHERE object = 'relation'
+ AND context = 'bulkread';
+});
+
+cmp_ok($bulkread, '>', 0,
+ 'a rescanned sequential scan uses its newly selected bulk-read strategy');
+
+$node->stop;
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Fix ReadStream handling when heap rescan strategy changes
2026-11-03 06:40 Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-11-03 10:23 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-11-04 03:32 ` Re: Fix ReadStream handling when heap rescan strategy changes shihao zhong <zhong950419@gmail.com>
2026-11-04 09:09 ` Re: Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 2 replies; 8+ messages in thread
From: Nazir Bilal Yavuz @ 2026-11-03 10:23 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Melanie Plageman <melanieplageman@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
Hi,
On Tue, 3 Nov 2026 at 09:41, Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> While working on [1], I noticed that a heap rescan can change whether a buffer
> access strategy is used while retaining the existing ReadStream. If the strategy
> is disabled, the stream keeps a dangling pointer after it is freed. If it is
> enabled, the stream continues using its NULL strategy and the queue capacity
> selected at creation.
>
> The issue was introduced by b7b0f3f27241. Nazir pointed it out and Yuhang
> proposed fixing it with a new read_stream_set_strategy() API in [2]. Since the
> issue is independent of the streaming SampleScan changes, I wonder if it wouldn't
> make sense to fix it separately, as in the attached?
Fixing it separately makes sense to me.
> 0001 takes a different approach (than [2]) and recreates the ReadStream when
> initscan() changes whether a strategy is active. Ordinary rescans continue to
> reset and reuse the existing stream. It uses the existing ReadStream APIs, which
> could also make the backpatches easier.
The main difference between Yuhang's ([2]) and your approach is that
[2] updates the existing stream through read_stream_set_strategy()
when the strategy is enabled but your approach ends and re-creates the
stream. Both approaches end and re-create the stream if the strategy
is disabled. It seems [2]'s performance is better because of this
reason but I am not sure whether it makes a measurable difference.
Although I didn't find a correctness problem with [2] during my
earlier review, I can see the simplicity benefit here. Recreating the
stream keeps strategy-dependent initialization in one place. That
seems a reasonable approach for backpatching.
> 0002 contains a test. I'm not sure it is worth adding to the tree given the amount
> of setup, but it demonstrates the issue.
I haven't reviewed 0002 yet, but I think regression coverage for this
bug would be worthwhile.
Minor readability suggestion:
+ if (scan->rs_read_stream && allow_strat != (scan->rs_strategy != NULL))
We can replace both '!=' with '=='. I find it slightly easier to read,
just a suggestion.
--
Regards,
Nazir Bilal Yavuz
Microsoft
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Fix ReadStream handling when heap rescan strategy changes
2026-11-03 06:40 Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-03 10:23 ` Re: Fix ReadStream handling when heap rescan strategy changes Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-11-04 03:32 ` shihao zhong <zhong950419@gmail.com>
2026-11-04 08:37 ` Re: Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 1 reply; 8+ messages in thread
From: shihao zhong @ 2026-11-04 03:32 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; pgsql-hackers@lists.postgresql.org, Melanie Plageman <melanieplageman@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
Hi Bertrand,
I am not sure the dangling pointer case can happen on master today.
initscan() only frees the strategy if SO_ALLOW_STRAT gets cleared or
the table drops below NBuffers / 4. The only caller of
table_rescan_set_params() is SampleScan, and a sample scan has no read
stream on master.
I don't think the table can shrink either, because truncation takes an AE
today.
This is something I really hope we can get fixed/improved.
Back to the main topic, I think on master the worst case scenario is the
one in your 0002, which causes more buffer pollution.
Thanks,
Shihao
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Fix ReadStream handling when heap rescan strategy changes
2026-11-03 06:40 Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-03 10:23 ` Re: Fix ReadStream handling when heap rescan strategy changes Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-11-04 03:32 ` Re: Fix ReadStream handling when heap rescan strategy changes shihao zhong <zhong950419@gmail.com>
@ 2026-11-04 08:37 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-04 14:40 ` Re: Fix ReadStream handling when heap rescan strategy changes shihao zhong <zhong950419@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Bertrand Drouvot @ 2026-11-04 08:37 UTC (permalink / raw)
To: shihao zhong <zhong950419@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; pgsql-hackers@lists.postgresql.org, Melanie Plageman <melanieplageman@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
Hi Shihao,
On Tue, Nov 03, 2026 at 10:32:25PM -0500, shihao zhong wrote:
> Hi Bertrand,
>
> I am not sure the dangling pointer case can happen on master today.
>
> initscan() only frees the strategy if SO_ALLOW_STRAT gets cleared or
> the table drops below NBuffers / 4. The only caller of
> table_rescan_set_params() is SampleScan, and a sample scan has no read
> stream on master.
>
> I don't think the table can shrink either, because truncation takes an AE
> today.
Thanks for looking at it!
I think that an extension can trigger it without shrinking the relation: start
a large sequential scan with allow_strat = true, then call table_rescan_set_params()
with allow_strat = false.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Fix ReadStream handling when heap rescan strategy changes
2026-11-03 06:40 Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-03 10:23 ` Re: Fix ReadStream handling when heap rescan strategy changes Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-11-04 03:32 ` Re: Fix ReadStream handling when heap rescan strategy changes shihao zhong <zhong950419@gmail.com>
2026-11-04 08:37 ` Re: Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-11-04 14:40 ` shihao zhong <zhong950419@gmail.com>
2026-11-14 21:34 ` Re: Fix ReadStream handling when heap rescan strategy changes Joao Rito <joaovictor.rito@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: shihao zhong @ 2026-11-04 14:40 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Nazir Bilal Yavuz <byavuz81@gmail.com>; pgsql-hackers@lists.postgresql.org, Melanie Plageman <melanieplageman@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
Hi Bertrand,
> I think that an extension can trigger it without shrinking the relation
I think you are right here and I missed that case. Not sure if any extension
really did it or not.
So my earlier mail was too strong. Core can't reach it, but an
extension can. It might be worth saying that in the commit message.
Thanks,
Shihao
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Fix ReadStream handling when heap rescan strategy changes
2026-11-03 06:40 Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-03 10:23 ` Re: Fix ReadStream handling when heap rescan strategy changes Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-11-04 03:32 ` Re: Fix ReadStream handling when heap rescan strategy changes shihao zhong <zhong950419@gmail.com>
2026-11-04 08:37 ` Re: Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-04 14:40 ` Re: Fix ReadStream handling when heap rescan strategy changes shihao zhong <zhong950419@gmail.com>
@ 2026-11-14 21:34 ` Joao Rito <joaovictor.rito@gmail.com>
0 siblings, 0 replies; 8+ messages in thread
From: Joao Rito @ 2026-11-14 21:34 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: shihao zhong <zhong950419@gmail.com>; Nazir Bilal Yavuz <byavuz81@gmail.com>; Melanie Plageman <melanieplageman@gmail.com>; Thomas Munro <thomas.munro@gmail.com>; pgsql-hackers@lists.postgresql.org
Hi Bertrand,
Thank you for the patch. I tested v1 (0001 and 0002) and it looks good to
me.
What I did:
- Applied both patches to current master (165af118c5). They apply cleanly
and build without warnings (--enable-cassert --enable-debug
--enable-tap-tests).
- make check passes (243 tests). The whole src/test/modules/test_misc
suite passes with the patch (16 files, 169 tests), including the new
t/016_heap_rescan_strategy.pl.
- I also ran the new test against unpatched master. It fails there on
"a rescanned sequential scan uses its newly selected bulk-read strategy"
(got 0, expected > 0). So the test does catch the bug, which I think
helps with the question of whether it is worth its setup cost.
- I ran the query from the test by hand (shared_buffers = 1MB,
max_parallel_workers_per_gather = 0) and looked at
pg_stat_get_backend_io() for the bulkread context:
- master: 0 reads, 0 hits in bulkread, with io_method = worker and
io_method = sync.
- patched: 468 reads, 104 hits in bulkread, with both io_method values.
The query returns the same 11997 rows in both builds, and there were no
assertion failures or errors in the server log.
One small thing: the commit message of 0001 has an empty "Reviewed-by:"
and "Discussion: https://postgr.es/m/"; without the message-id. I guess you
will fill those in later.
I only tested the case where the rescan starts using a strategy. I did not
exercise the other direction (the strategy is disabled on rescan, the
extension case you described), and I did not test parallel, bitmap or TID
range scans, io_uring, the back branches, or a meson build.
Thanks again,
Joao Rito
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Fix ReadStream handling when heap rescan strategy changes
2026-11-03 06:40 Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-03 10:23 ` Re: Fix ReadStream handling when heap rescan strategy changes Nazir Bilal Yavuz <byavuz81@gmail.com>
@ 2026-11-04 09:09 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
1 sibling, 0 replies; 8+ messages in thread
From: Bertrand Drouvot @ 2026-11-04 09:09 UTC (permalink / raw)
To: Nazir Bilal Yavuz <byavuz81@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Melanie Plageman <melanieplageman@gmail.com>; Thomas Munro <thomas.munro@gmail.com>
Hi,
On Tue, Nov 03, 2026 at 01:23:26PM +0300, Nazir Bilal Yavuz wrote:
> Hi,
>
> On Tue, 3 Nov 2026 at 09:41, Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > The issue was introduced by b7b0f3f27241. Nazir pointed it out and Yuhang
> > proposed fixing it with a new read_stream_set_strategy() API in [2]. Since the
> > issue is independent of the streaming SampleScan changes, I wonder if it wouldn't
> > make sense to fix it separately, as in the attached?
>
> Fixing it separately makes sense to me.
Thanks for looking at it!
> The main difference between Yuhang's ([2]) and your approach is that
> [2] updates the existing stream through read_stream_set_strategy()
> when the strategy is enabled but your approach ends and re-creates the
> stream. Both approaches end and re-create the stream if the strategy
> is disabled. It seems [2]'s performance is better because of this
> reason but I am not sure whether it makes a measurable difference.
OTOH, [2] calls read_stream_set_strategy() on every rescan when the bulk strategy
remains enabled. FWIW, I ran a few benchmarks, and regardless of which approach
was faster, the absolute difference remained below 0.15 microseconds per rescan.
Each measurement timed 3 million rescans, and the results were repeatable across
15 samples.
So I don't think performance strongly favors either approach.
> Although I didn't find a correctness problem with [2] during my
> earlier review, I can see the simplicity benefit here. Recreating the
> stream keeps strategy-dependent initialization in one place. That
> seems a reasonable approach for backpatching.
+1
> > 0002 contains a test. I'm not sure it is worth adding to the tree given the amount
> > of setup, but it demonstrates the issue.
>
> I haven't reviewed 0002 yet, but I think regression coverage for this
> bug would be worthwhile.
Yeah, I think the test is worthwhile. It initializes and starts a new cluster,
which is not that cheap (and our test code has been growing quite a bit lately)
but that seems justified here.
> Minor readability suggestion:
>
> + if (scan->rs_read_stream && allow_strat != (scan->rs_strategy != NULL))
>
> We can replace both '!=' with '=='. I find it slightly easier to read,
> just a suggestion.
Yeah, that's probably just a matter of taste. I find the existing links more
directly to "the requested state differs from the current state" but I'm happy
to change it if you feel strongly about it (I don't).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Fix ReadStream handling when heap rescan strategy changes
2026-11-03 06:40 Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
@ 2026-11-17 21:54 ` Melanie Plageman <melanieplageman@gmail.com>
1 sibling, 0 replies; 8+ messages in thread
From: Melanie Plageman @ 2026-11-17 21:54 UTC (permalink / raw)
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: pgsql-hackers@lists.postgresql.org, Thomas Munro <thomas.munro@gmail.com>
On Tue, Nov 3, 2026 at 1:40 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> 0001 takes a different approach (than [2]) and recreates the ReadStream when
> initscan() changes whether a strategy is active. Ordinary rescans continue to
> reset and reuse the existing stream. It uses the existing ReadStream APIs, which
> could also make the backpatches easier.
Now that initscan() has to concern itself with the read stream because
it has to reset or end the stream before changing the strategy, then
change the strategy, and then make the new read stream, I think the
separation between initscan() and making the read stream is no longer
useful. In fact, the ping-ponging has become confusing. I think your
version might actually have read_stream_reset() getting called twice
in some cases since read_stream_end() also calls it. Attached v2 just
moves all the read stream setup into initscan() and adds some flags to
make it more clear when we want to rebuild the stream/strategy.
> 0002 contains a test. I'm not sure it is worth adding to the tree given the amount
> of setup, but it demonstrates the issue.
I agree that zero test coverage of this is not good. I really would
have liked to find somewhere to put the test that wasn't a new TAP
test file that makes yet another new cluster. Alas, neither I nor my
LLM could come up with a good place. I did however, change the test to
use a cursor instead of a query that does a rescan. That takes away
the need for the plpgsql function and makes the test simpler IMO.
pg_stat_get_backend_io() doesn't exist in 17, so we'll have to modify
the test to backpatch there.
- Melanie
Attachments:
v2-0001-Recreate-read-stream-across-rescans.patch (text/x-patch, 10.9K) download
From 3888a3a2811639983a5f487da04e41e9c10a6dac Mon Sep 17 00:00:00 2001
From: Melanie Plageman <melanieplageman@gmail.com>
Date: Tue, 17 Nov 2026 15:57:28 -0500
Subject: [PATCH v2] Recreate read stream across rescans
Heap scan descriptors persist across rescans, but initscan() can add or
remove the buffer access strategy when the relation size or caller flags
change.
The ReadStream stored the strategy pointer and sized its queue capacity
based on the strategy at ReadStream creation, which was only before the
first scan. This meant that on rescan, enabling a bulk-read strategy
left the existing stream using its NULL strategy and old capacity.
And disabling the strategy left the stream pointing to freed memory.
Reset and reuse the stream when strategy use is unchanged. Otherwise,
end it before changing the strategy and create a correctly sized
replacement.
Reported-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Reported-by: Nazir Bilal Yavuz <byavuz81@gmail.com>
Backpatch-through: 17
---
src/backend/access/heap/heapam.c | 155 ++++++++++--------
src/backend/storage/aio/read_stream.c | 4 +-
src/test/modules/test_misc/meson.build | 1 +
.../test_misc/t/016_heap_scan_stream.pl | 54 ++++++
4 files changed, 142 insertions(+), 72 deletions(-)
create mode 100644 src/test/modules/test_misc/t/016_heap_scan_stream.pl
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 3bb33699349..67073fff6b7 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -360,8 +360,11 @@ static void
initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock)
{
ParallelBlockTableScanDesc bpscan = NULL;
- bool allow_strat;
+ BlockNumber nblocks;
+ bool had_strat = scan->rs_strategy != NULL;
+ bool use_strat;
bool allow_sync;
+ bool rebuild_stream;
/*
* Determine the number of blocks we have to scan.
@@ -377,10 +380,10 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock)
if (scan->rs_base.rs_parallel != NULL)
{
bpscan = (ParallelBlockTableScanDesc) scan->rs_base.rs_parallel;
- scan->rs_nblocks = bpscan->phs_nblocks;
+ nblocks = bpscan->phs_nblocks;
}
else
- scan->rs_nblocks = RelationGetNumberOfBlocks(scan->rs_base.rs_rd);
+ nblocks = RelationGetNumberOfBlocks(scan->rs_base.rs_rd);
/*
* If the table is large relative to NBuffers, use a bulk-read access
@@ -395,27 +398,47 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock)
* if you change this, consider changing that one, too.
*/
if (!RelationUsesLocalBuffers(scan->rs_base.rs_rd) &&
- scan->rs_nblocks > NBuffers / 4)
+ nblocks > NBuffers / 4)
{
- allow_strat = (scan->rs_base.rs_flags & SO_ALLOW_STRAT) != 0;
+ use_strat = (scan->rs_base.rs_flags & SO_ALLOW_STRAT) != 0;
allow_sync = (scan->rs_base.rs_flags & SO_ALLOW_SYNC) != 0;
}
else
- allow_strat = allow_sync = false;
+ use_strat = allow_sync = false;
- if (allow_strat)
+ /*
+ * Reset the stream before reinitializing state used by its callback. A
+ * stream keeps the strategy passed at creation and sizes its buffer queue
+ * accordingly, so end it instead when strategy use changes.
+ */
+ rebuild_stream = scan->rs_read_stream && had_strat != use_strat;
+ if (scan->rs_read_stream)
+ {
+ if (rebuild_stream)
+ {
+ read_stream_end(scan->rs_read_stream);
+ scan->rs_read_stream = NULL;
+ }
+ else
+ read_stream_reset(scan->rs_read_stream);
+ }
+
+ if (use_strat)
{
/* During a rescan, keep the previous strategy object. */
- if (scan->rs_strategy == NULL)
+ if (!had_strat)
scan->rs_strategy = GetAccessStrategy(BAS_BULKREAD);
}
else
{
- if (scan->rs_strategy != NULL)
+ if (had_strat)
FreeAccessStrategy(scan->rs_strategy);
scan->rs_strategy = NULL;
}
+ /* Publish the new size only after the old stream has been reset. */
+ scan->rs_nblocks = nblocks;
+
if (scan->rs_base.rs_parallel != NULL)
{
/* For parallel scan, believe whatever ParallelTableScanDesc says. */
@@ -491,6 +514,53 @@ initscan(HeapScanDesc scan, ScanKey key, bool keep_startblock)
*/
if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN)
pgstat_count_heap_scan(scan->rs_base.rs_rd);
+
+ if (scan->rs_read_stream == NULL)
+ {
+ if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN ||
+ scan->rs_base.rs_flags & SO_TYPE_TIDRANGESCAN)
+ {
+ ReadStreamBlockNumberCB cb;
+
+ if (scan->rs_base.rs_parallel)
+ cb = heap_scan_stream_read_next_parallel;
+ else
+ cb = heap_scan_stream_read_next_serial;
+
+ /* ---
+ * It is safe to use batchmode as the only locks taken by `cb`
+ * are never taken while waiting for IO:
+ * - SyncScanLock is used in the non-parallel case
+ * - in the parallel case, only spinlocks and atomics are used
+ * ---
+ */
+ scan->rs_read_stream =
+ read_stream_begin_relation(READ_STREAM_SEQUENTIAL |
+ READ_STREAM_USE_BATCHING,
+ scan->rs_strategy,
+ scan->rs_base.rs_rd,
+ MAIN_FORKNUM,
+ cb,
+ scan,
+ 0);
+ }
+ else if (scan->rs_base.rs_flags & SO_TYPE_BITMAPSCAN)
+ {
+ scan->rs_read_stream =
+ read_stream_begin_relation(READ_STREAM_DEFAULT |
+ READ_STREAM_USE_BATCHING,
+ scan->rs_strategy,
+ scan->rs_base.rs_rd,
+ MAIN_FORKNUM,
+ bitmapheap_stream_read_next,
+ scan,
+ sizeof(TBMIterateResult));
+ }
+
+ if (scan->rs_read_stream && scan->rs_base.rs_instrument)
+ read_stream_enable_stats(scan->rs_read_stream,
+ &scan->rs_base.rs_instrument->io);
+ }
}
/*
@@ -1204,7 +1274,10 @@ heap_beginscan(Relation relation, Snapshot snapshot,
scan->rs_base.rs_nkeys = nkeys;
scan->rs_base.rs_flags = flags;
scan->rs_base.rs_parallel = parallel_scan;
- scan->rs_base.rs_instrument = NULL;
+ if (flags & SO_SCAN_INSTRUMENT)
+ scan->rs_base.rs_instrument = palloc0_object(TableScanInstrumentation);
+ else
+ scan->rs_base.rs_instrument = NULL;
scan->rs_strategy = NULL; /* set in initscan */
scan->rs_cbuf = InvalidBuffer;
@@ -1270,60 +1343,8 @@ heap_beginscan(Relation relation, Snapshot snapshot,
else
scan->rs_base.rs_key = NULL;
- initscan(scan, key, false);
-
scan->rs_read_stream = NULL;
-
- /*
- * Set up a read stream for sequential scans and TID range scans. This
- * should be done after initscan() because initscan() allocates the
- * BufferAccessStrategy object passed to the read stream API.
- */
- if (scan->rs_base.rs_flags & SO_TYPE_SEQSCAN ||
- scan->rs_base.rs_flags & SO_TYPE_TIDRANGESCAN)
- {
- ReadStreamBlockNumberCB cb;
-
- if (scan->rs_base.rs_parallel)
- cb = heap_scan_stream_read_next_parallel;
- else
- cb = heap_scan_stream_read_next_serial;
-
- /* ---
- * It is safe to use batchmode as the only locks taken by `cb`
- * are never taken while waiting for IO:
- * - SyncScanLock is used in the non-parallel case
- * - in the parallel case, only spinlocks and atomics are used
- * ---
- */
- scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_SEQUENTIAL |
- READ_STREAM_USE_BATCHING,
- scan->rs_strategy,
- scan->rs_base.rs_rd,
- MAIN_FORKNUM,
- cb,
- scan,
- 0);
- }
- else if (scan->rs_base.rs_flags & SO_TYPE_BITMAPSCAN)
- {
- scan->rs_read_stream = read_stream_begin_relation(READ_STREAM_DEFAULT |
- READ_STREAM_USE_BATCHING,
- scan->rs_strategy,
- scan->rs_base.rs_rd,
- MAIN_FORKNUM,
- bitmapheap_stream_read_next,
- scan,
- sizeof(TBMIterateResult));
- }
-
- /* enable read stream instrumentation */
- if ((flags & SO_SCAN_INSTRUMENT) && (scan->rs_read_stream != NULL))
- {
- scan->rs_base.rs_instrument = palloc0_object(TableScanInstrumentation);
- read_stream_enable_stats(scan->rs_read_stream,
- &scan->rs_base.rs_instrument->io);
- }
+ initscan(scan, key, false);
scan->rs_vmbuffer = InvalidBuffer;
@@ -1375,14 +1396,6 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params,
* additional data vs a normal HeapScan
*/
- /*
- * The read stream is reset on rescan. This must be done before
- * initscan(), as some state referred to by read_stream_reset() is reset
- * in initscan().
- */
- if (scan->rs_read_stream)
- read_stream_reset(scan->rs_read_stream);
-
/*
* reinitialize scan descriptor
*/
diff --git a/src/backend/storage/aio/read_stream.c b/src/backend/storage/aio/read_stream.c
index fddf6ef441d..f005a93aa5b 100644
--- a/src/backend/storage/aio/read_stream.c
+++ b/src/backend/storage/aio/read_stream.c
@@ -255,7 +255,9 @@ void
read_stream_enable_stats(ReadStream *stream, IOStats *stats)
{
stream->stats = stats;
- if (stream->stats)
+ /* The same accumulator may span recreated streams of different sizes. */
+ if (stream->stats &&
+ stream->stats->distance_capacity < stream->max_pinned_buffers)
stream->stats->distance_capacity = stream->max_pinned_buffers;
}
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index 5d81f5b13be..5225aa280ca 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -24,6 +24,7 @@ tests += {
't/013_temp_obj_multisession.pl',
't/014_log_statement_max_length.pl',
't/015_temp_schema_exit_deferrable.pl',
+ 't/016_heap_scan_stream.pl',
],
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
diff --git a/src/test/modules/test_misc/t/016_heap_scan_stream.pl b/src/test/modules/test_misc/t/016_heap_scan_stream.pl
new file mode 100644
index 00000000000..a659b5e1eae
--- /dev/null
+++ b/src/test/modules/test_misc/t/016_heap_scan_stream.pl
@@ -0,0 +1,54 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+# Set a low threshold for selecting a bulk-read strategy.
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf', qq{
+shared_buffers = '1MB'
+});
+$node->start;
+
+$node->safe_psql(
+ 'postgres', q{
+CREATE TABLE growing AS SELECT 1 AS i;
+});
+
+# Start without a strategy, grow the relation, then rewind the same scan. The
+# rescan must recreate its read stream with a bulk-read strategy.
+my $bulkread = $node->safe_psql(
+ 'postgres', q{
+BEGIN;
+DECLARE c SCROLL CURSOR FOR SELECT FROM growing;
+MOVE FORWARD ALL IN c;
+
+INSERT INTO growing SELECT generate_series(2, 10000);
+
+MOVE ABSOLUTE 0 IN c;
+MOVE FORWARD ALL IN c;
+COMMIT;
+
+SELECT pg_stat_force_next_flush()
+\gset
+
+SELECT EXISTS (
+ SELECT
+ FROM pg_stat_get_backend_io(pg_backend_pid())
+ WHERE object = 'relation'
+ AND context = 'bulkread'
+ AND reads + hits > 0
+);
+});
+
+is($bulkread, 't',
+ 'a rescanned sequential scan uses its newly selected bulk-read strategy');
+
+$node->stop;
+done_testing();
--
2.43.0
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2026-11-17 21:54 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-03 06:40 Fix ReadStream handling when heap rescan strategy changes Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-03 10:23 ` Nazir Bilal Yavuz <byavuz81@gmail.com>
2026-11-04 03:32 ` shihao zhong <zhong950419@gmail.com>
2026-11-04 08:37 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-04 14:40 ` shihao zhong <zhong950419@gmail.com>
2026-11-14 21:34 ` Joao Rito <joaovictor.rito@gmail.com>
2026-11-04 09:09 ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-17 21:54 ` Melanie Plageman <melanieplageman@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox