From: Thomas Munro <tmunro@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Remove arbitrary cap on read_stream.c buffer queue.
Date: Thu, 27 Feb 2025 07:51:28 +0000
Message-ID: <E1tnYg3-000JQb-1a@gemulon.postgresql.org> (raw)
Remove arbitrary cap on read_stream.c buffer queue.
Previously the internal queue of buffers was capped at max_ios * 4,
though not less than io_combine_limit, at allocation time. That was
done in the first version based on conservative theories about resource
usage and heuristics pending later work. The configured I/O depth could
not always be reached with dense random streams generated by ANALYZE,
VACUUM, the proposed Bitmap Heap Scan patch, and also sequential streams
with the proposed AIO subsystem to name some examples.
The new formula is (max_ios + 1) * io_combine_limit, enough buffers for
the full configured I/O concurrency level using the full configured I/O
combine size, plus the buffers from one finished but not yet consumed
full-sized I/O. Significantly more memory would be needed for high GUC
values if the client code requests a large per-buffer data size, but
that is discouraged (existing and proposed stream users try to keep it
under a few words, if not zero).
With this new formula, an intermediate variable could have overflowed
under maximum GUC values, so its data type is adjusted to cope.
Discussion: https://postgr.es/m/CA%2BhUKGK_%3D4CVmMHvsHjOVrK6t4F%3DLBpFzsrr3R%2BaJYN8kcTfWg%40mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/55918f798bc2d1846eea0d805fbec91d4e1816e0
Modified Files
--------------
src/backend/storage/aio/read_stream.c | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-committers@postgresql.org
Cc: tmunro@postgresql.org, pgsql-committers@lists.postgresql.org
Subject: Re: pgsql: Remove arbitrary cap on read_stream.c buffer queue.
In-Reply-To: <E1tnYg3-000JQb-1a@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox