MIME-Version: 1.0
References: 
 <CA+UBfa=vDV8wbmAV0pgrx-FuJh+x8YOW23vJ90Jzr=14rV+9jA@mail.gmail.com>
 <OS9PR01MB12149A4E7927072A215AEED69F565A@OS9PR01MB12149.jpnprd01.prod.outlook.com>
 <CA+UBfakmkdtauuRsOVXFqhFVJt0nTdEadx94tJn+qG0Pe8Wjfw@mail.gmail.com>
 <CAN4CZFM7FV0VTNkujD=Mb7tNa+jkmEfnX7carvj95fY6Tp11FQ@mail.gmail.com>
 <CA+UBfamW6NuuMMQTDRPDQ0a9fWN_u2OvjEF98u3CfYKTBcOZMw@mail.gmail.com>
 <CA+UBfa=Dv-2tLSEKHJ0YFFH8PCTHxnX9rtVZeV8gd8q1a-GmYA@mail.gmail.com>
 <CA+UBfa=PKdShpSTTTSHwXdGPZnm2rGMKPjERNOdS0SG9t9CT3Q@mail.gmail.com>
 <CABPTF7WVW2x4XitXttHwCamSZcBn=Q+wLjf+M+MuEbZSAxqdDw@mail.gmail.com>
 <CAAAe_zCxg2NTG_i1erLQQr8Wn+6SQ3EMOmp+N4J58Xxb21g2BQ@mail.gmail.com>
 <CA+UBfa=qDfWB90w5AsmX4f3PbeeM++GbaoVagd9ff-DKQDLvWA@mail.gmail.com>
 <CA+UBfakz7G5FH8PjxWyFLmF+sWdqMVcvQRRM0vURmznafqOjQQ@mail.gmail.com>
 <CABPTF7XABSSwUPbnS+UE9OyeH-z3ihmdp9tOt3UJ4XcWZkE1DA@mail.gmail.com>
 <CA+UBfamzeXcdEbmhdOdjWn5X_YVt9n8xUpH5ZwmA7S8VWvaoXw@mail.gmail.com>
In-Reply-To: 
 <CA+UBfamzeXcdEbmhdOdjWn5X_YVt9n8xUpH5ZwmA7S8VWvaoXw@mail.gmail.com>
From: Xuneng Zhou <xunengzhou@gmail.com>
Date: Thu, 25 Jun 2026 15:47:20 +0800
Message-ID: 
 <CABPTF7Vz+p5dxUbKPxExaAVLujkbJjrpzXsOZbCqKMp_NvL3YA@mail.gmail.com>
Subject: Re: [WIP] Pipelined Recovery
To: Imran Zaheer <imran.zhir@gmail.com>
Cc: assam258@gmail.com, Zsolt Parragi <zsolt.parragi@percona.com>,
	Jakub Wartak <jakub.wartak@enterprisedb.com>,
	"Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CABPTF7Vz%2Bp5dxUbKPxExaAVLujkbJjrpzXsOZbCqKMp_NvL3YA%40mail.gmail.com>
Precedence: bulk

Hi Imran,

On Tue, Jun 23, 2026 at 9:27=E2=80=AFPM Imran Zaheer <imran.zhir@gmail.com>=
 wrote:
>
> Hi
>
> I am attaching the new series of patches.
>
> What has changed?
>
> * Rebased
>
> * The patch set is now split into two new patches. This will make the
> code easier to understand and review.
>
> * The v4-0003 patch contains code mostly related to keeping the
> recovery states synced between the startup process and the pipeline
> process. Most of these changes were required to make the streaming
> replication work.
>
> * The v4-0002 patch now only contains the consumer code that handles
> receiving the decoded records from the shmem queue and moving the redo
> loop forward.
>
> * The v4-0004 contains some basic tests to see if the pipeline worker
> is functioning as expected. More testing was done by passing
> PG_TEST_INITDB_EXTRA_OPTS=3D"-c wal_pipeline=3Don" before running the
> recovery test suite.

+1 for splitting the patch set into smaller components to make the
review process smoother.

> * Other than that, the cpu overhead during deserialization is
> optimized by skipping multiple copies of the decoded record and
> directly passing the pointer to the shmem queue. There is still some
> overhead visible during serialization that could be improved at the
> producer end.
>
> * Signal handling for the pipeline worker is improved so that
> promotion signals are sent to both the startup process and the
> producer worker by the postmaster.
>
>
> You will also find the new benchmarks attached [1] and the pdf report
> overview. A simple cpu profiling on the pipelined startup process
> shows that the cpu overhead during reading records has now been
> removed and offloaded to the producer worker.
>
> Before pipelining:
>
> Around 50% of the cpu time is spent on fetching the wal record. Note that
> in this workload pipeline is off so don't worry about the new func
> ReceiveRecord(), it's just a wrapper around ReadRecord().
>
>   Children      Self  Command   Shared O  Symbol
> -   98.85%     0.21%  postgres  postgres  [.] PerformWalRecovery
>    - 98.64% PerformWalRecovery
>       - 51.00% ReceiveRecord
>          - 50.78% ReadRecord
>             - 50.52% XLogPrefetcherReadRecord
>                - 49.61% XLogPrefetcherNextBlock
>                   + 25.33% XLogReadAhead
>                   + 22.32% PrefetchSharedBuffer
>                   + 0.76% smgropen
>       - 46.68% ApplyWalRecord
>          + 29.23% heap_redo
>          + 9.51% heap2_redo
>          + 4.74% btree_redo
>          + 1.11% xlog_redo
>          + 0.80% xact_redo
>
>
> After Pipelining:
>
> Here the only work needed to be done by the cpu is to get the decoded
> record from
> the queue. Other times (89.13%) cpu is worried about applying the wal rec=
ord.
>
>   Children      Self  Command   Shared O  Symbol
> -   98.74%     0.37%  postgres  postgres  [.] PerformWalRecovery
>    - 98.37% PerformWalRecovery
>       - 89.13% ApplyWalRecord
>          + 56.89% heap_redo
>          + 18.28% heap2_redo
>          + 8.01% btree_redo
>          + 2.02% xlog_redo
>          + 1.15% xact_redo
>       - 7.80% ReceiveRecord
>          + 7.63% WalPipeline_ReceiveRecord
>
> If the recovery process is not I/O bound then we would be able to test
> this cpu optimization. Doing pgbench on a workload that is fully in
> memory shows around 30% performance gains. You can see more
> benchmarking details in the attached drive link [1]

The perf result looks promising!

> Some comments related to attached pdf and benchmarking, it is showing
> that we can get more performance advantage out of the pipeline when
> most of the workload is running in memory i.e. we have enough shared
> buffers configured.
>
> If you want to do some experiments, please be my guest; I would be
> happy to see more testing. You can share what performance advantage
> you are getting from this. You can also refer to the benchmarking
> script that I have been using [2].
>
>
> Looking forward to your review, comments, etc.

I haven't had a chance for a meaningful review yet, but expect to do so soo=
n.

--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.