pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Thomas Munro <thomas.munro@gmail.com>
To: Daniel Gustafsson <daniel@yesql.se>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: David Steele <david@pgmasters.net>
Cc: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: Jakub Wartak <Jakub.Wartak@tomtom.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: WIP: WAL prefetch (another approach)
Date: Thu, 25 Nov 2021 23:32:07 +0100
Message-ID: <a6915477-c675-7959-28b2-5b8fa60f44fc@enterprisedb.com> (raw )
In-Reply-To: <CA+hUKGJ7OqpdnbSTq5oK=djSeVW2JMnrVPSm8JC-_dbN6Y7bpw@mail.gmail.com >
References: <CA+hUKGJ1=pOiNjSgXYJnjE3OyRtp8tjMRcON256e5EFpzPpgtA@mail.gmail.com >
<20210317214331.GF20766@tamriel.snowman.net >
<28c3352e-5388-6410-aae7-05907079e6c9@enterprisedb.com >
<CA+hUKGLHTjbBPFX6E1ZCGVyQD30-vWtH=N6EcHEzW13q+jh7FQ@mail.gmail.com >
<8dd2dcc2-0b6b-a4ab-c1b3-032c0a5ab1c3@enterprisedb.com >
<CA+hUKGKmkKfH1h9XG5ctYCgqeOHGdN5bHMoQMZLR=+kLY0PQzA@mail.gmail.com >
<CA+hUKGLsF8sEE+n3qoaV6+XjmYEwYoFVDbLPbYTGz7oGkL=iZA@mail.gmail.com >
<CA+hUKG+a4wn7B-qBu9SLc1HAbZRSKbp1Ep=rDeveM-UQz84+jg@mail.gmail.com >
<773932.1619022622@sss.pgh.pa.us >
<b971373e-5f05-e691-d83c-c72a4f67167b@enterprisedb.com >
<CA+hUKGLQqsNBXXZ6uXFi7L9iqx-4ZSuiAvKiPqDtRQs+yFD9ew@mail.gmail.com >
<CA+hUKGJKDibdw5HzOdgh+tFsA7=wwJ3O6p+qYwbgAD_xkLWA3Q@mail.gmail.com >
<CAOuzzgrn7iKnFRsB4MHp3UisEQAGgZMbk_ViTN4HV4-Ksq8zCg@mail.gmail.com >
<CA+hUKGKLakuWXsHuOXYYYyk-CiZKgcqXgzwKb61d9nrpgn5eww@mail.gmail.com >
<5B6070FF-5869-4A84-88BC-2D04EDCDB858@yesql.se >
<CA+hUKGJ7OqpdnbSTq5oK=djSeVW2JMnrVPSm8JC-_dbN6Y7bpw@mail.gmail.com >
Hi,
It's great you posted a new version of this patch, so I took a look a
brief look at it. The code seems in pretty good shape, I haven't found
any real issues - just two minor comments:
This seems a bit strange:
#define DEFAULT_DECODE_BUFFER_SIZE 0x10000
Why not to define this as a simple decimal value? Is there something
special about this particular value, or is it arbitrary? I guess it's
simply the minimum for wal_decode_buffer_size GUC, but why not to use
the GUC for all places decoding WAL?
FWIW I don't think we include updates to typedefs.list in patches.
I also repeated the benchmarks I did at the beginning of the year [1].
Attached is a chart with four different configurations:
1) master (f79962d826)
2) patched (with prefetching disabled)
3) patched (with default configuration)
4) patched (with I/O concurrency 256 and 2MB decode buffer)
For all configs the shared buffers were set to 64GB, checkpoints every
20 minutes, etc.
The results are pretty good / similar to previous results. Replaying the
1h worth of work on a smaller machine takes ~5:30h without prefetching
(master or with prefetching disabled). With prefetching enabled this
drops to ~2h (default config) and ~1h (with tuning).
regards
[1]
https://www.postgresql.org/message-id/c5d52837-6256-0556-ac8c-d6d3d558820a%40enterprisedb.com
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
prefetching.png (image/png, 44.9K) download
view image
view thread (174+ messages) latest in thread
Message-ID: <a6915477-c675-7959-28b2-5b8fa60f44fc@enterprisedb.com>
Permalink: ../a6915477-c675-7959-28b2-5b8fa60f44fc@enterprisedb.com/
Also on: postgresql.org/message-id/a6915477-c675-7959-28b2-5b8fa60f44fc@enterprisedb.com
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@enterprisedb.com, thomas.munro@gmail.com, daniel@yesql.se, sfrost@snowman.net, alvherre@2ndquadrant.com, andres@anarazel.de, david@pgmasters.net, 9erthalion6@gmail.com, Jakub.Wartak@tomtom.com, tgl@sss.pgh.pa.us, tomas.vondra@2ndquadrant.com
Subject: Re: WIP: WAL prefetch (another approach)
In-Reply-To: <a6915477-c675-7959-28b2-5b8fa60f44fc@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox