public inbox for [email protected]
help / color / mirror / Atom feedRE: [WIP] Pipelined Recovery
2+ messages / 2 participants
[nested] [flat]
* RE: [WIP] Pipelined Recovery
@ 2026-02-09 05:15 Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 2+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2026-02-09 05:15 UTC (permalink / raw)
To: 'Imran Zaheer' <[email protected]>; +Cc: pgsql-hackers
Dear Imran,
I feel your idea is very interesting. Here are very elementary comments from my
point of view.
01.
I feel the patch looks large and it is bit difficult to be reviewed. Can you
separate it into several parts, e.g., producer and consumer parts?
02.
The file name seems to be "xlogpipeline.{h|c}" per others.
03.
Can you add tests? it's also helpful to understand the patch.
04.
Some Assert() are commented out but they won't be accepted. Can you added a
function like AmWalPipelineProducer() and call there?
05.
Producer is implemented as the bgworker process, but we have another option to
be as a background process. Which one is better and why?
06.
Can we enable the pipeline with the streaming replication? I've tried but it
did not work well - the producer exit and recovery finished without any promote
signals. My expectation was that producer waits till new record was arrived.
```
...
LOG: starting backup recovery with redo LSN 0/02000028, checkpoint LSN 0/02000080, on timeline ID 1
LOG: entering standby mode
LOG: [walpipeline] started.
LOG: redo starts at 0/02000028
LOG: completed backup recovery with redo LSN 0/02000028 and end LSN 0/02000128
LOG: consistent recovery state reached at 0/02000128
LOG: database system is ready to accept read-only connections
LOG: [walpipeline] consumer: received shutdown message from the producer
LOG: [walpipeline] producer: exiting: sent=4 received=4
LOG: WAL pipeline stopped
LOG: redo done at 0/02000128 system usage: CPU: user: 0.00 s, system: 0.02 s, elapsed: 0.03 s
...
```
07.
As mentioned by Jakub, it is better to profile per process (startup and producer).
08.
I suggest to benchmark on larger environment. Do let me know if difficult - I can
try on my env instead.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: [WIP] Pipelined Recovery
@ 2026-02-16 06:53 Imran Zaheer <[email protected]>
parent: Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 0 replies; 2+ messages in thread
From: Imran Zaheer @ 2026-02-16 06:53 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: pgsql-hackers
Hi Kuroda.
Thanks for the feedback.
> 01.
> I feel the patch looks large and it is bit difficult to be reviewed. Can you
> separate it into several parts, e.g., producer and consumer parts?
>
Ok I will try to split it.
> 02.
> The file name seems to be "xlogpipeline.{h|c}" per others.
>
Noted, thanks.
> 03.
> Can you add tests? it's also helpful to understand the patch.
>
Right now I am trying to fix the existing tests under
`src/test/recovery/`. But sure we should add some pipeline
specific tests.
> 04.
> Some Assert() are commented out but they won't be accepted. Can you added a
> function like AmWalPipelineProducer() and call there?
>
Noted. Sure I will add AmWalPipelineProducer().
> 05.
> Producer is implemented as the bgworker process, but we have another option to
> be as a background process. Which one is better and why?
>
Yes I myself don't have any good answer for this question. For me the
bgworker api looked simple and good enough to start the pipeline
implementation.
> 06.
> Can we enable the pipeline with the streaming replication? I've tried but it
> did not work well - the producer exit and recovery finished without any promote
> signals. My expectation was that producer waits till new record was arrived.
>
> ```
> ...
> LOG: starting backup recovery with redo LSN 0/02000028, checkpoint LSN 0/02000080, on timeline ID 1
> LOG: entering standby mode
> LOG: [walpipeline] started.
> LOG: redo starts at 0/02000028
> LOG: completed backup recovery with redo LSN 0/02000028 and end LSN 0/02000128
> LOG: consistent recovery state reached at 0/02000128
> LOG: database system is ready to accept read-only connections
> LOG: [walpipeline] consumer: received shutdown message from the producer
> LOG: [walpipeline] producer: exiting: sent=4 received=4
> LOG: WAL pipeline stopped
> LOG: redo done at 0/02000128 system usage: CPU: user: 0.00 s, system: 0.02 s, elapsed: 0.03 s
> ...
> ```
Yes am working to fix the streaming replication, so far the
testing/benchmarking were done with archive recovery.
>
> 07.
> As mentioned by Jakub, it is better to profile per process (startup and producer).
>
Noted.
> 08.
> I suggest to benchmark on larger environment. Do let me know if difficult - I can
> try on my env instead.
>
Yes, that would be very helpful. I myself tried very hard to collect
some good benchmarks but have some hardware limitations. If you can
help me with this , that would be great. You can use the scripts[1]
that I used. Let me know if you need any help understanding or using
the scripts.
[1]: https://github.com/imranzaheer612/pg-recovery-testing
Thanks for your valuable feedback. That was very helpful.
Regards,
Imran Zaheer
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2026-02-16 06:53 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-02-09 05:15 RE: [WIP] Pipelined Recovery Hayato Kuroda (Fujitsu) <[email protected]>
2026-02-16 06:53 ` Imran Zaheer <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox