MIME-Version: 1.0
References: <cb014f00-66b2-4328-a65e-d11c681c9f45@gmail.com>
 <CAEvyyTj0rEsgcQOQgkARbRPbupHR_mc=TUzHBBLNzd8JByUUTw@mail.gmail.com>
 <4c1d0b97-a5f8-472c-afdd-bdeb09b93f33@gmail.com> <OS9PR01MB12149A6CEE200E1A5A88D9F14F563A@OS9PR01MB12149.jpnprd01.prod.outlook.com>
In-Reply-To: <OS9PR01MB12149A6CEE200E1A5A88D9F14F563A@OS9PR01MB12149.jpnprd01.prod.outlook.com>
From: lakshmi <lakshmigcdac@gmail.com>
Date: Tue, 17 Feb 2026 11:41:13 +0530
Message-ID: <CAEvyyTiQqd=rv3XUxc0YEaW-feopksBveZKKjVZNeSVG=GrY+A@mail.gmail.com>
Subject: Re: parallel data loading for pgbench -i
To: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Cc: Mircea Cadariu <cadariu.mircea@gmail.com>, 
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>, "tomas@vondra.me" <tomas@vondra.me>
Content-Type: multipart/alternative; boundary="0000000000003396a8064afee604"
Archived-At: <https://www.postgresql.org/message-id/CAEvyyTiQqd%3Drv3XUxc0YEaW-feopksBveZKKjVZNeSVG%3DGrY%2BA%40mail.gmail.com>
Precedence: bulk

--0000000000003396a8064afee604
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Mircea, Hayato,
I ran a few more tests on 19devel ,focusing on the partitioned case to
better understand the performance behavior.

For scale 500, the serial initialization on my system takes around 34.3
seconds. Using parallel initialization without partitions (-j 10) makes the
client-side data generation noticeably faster,But the overall runtime ends
up slightly higher because the vacuum phase becomes much longer.

However,when running with partitions(pgbench -i -s 500 --partitions=3D10 -j
10),the total runtime drops to about 21.9 seconds, and the vacuum cost is
much smaller.I also verified that the row counts are correct in all cases
,and regression tests still pass locally.

So it looks like the main benefit of parallel initialization shows up
clearly in the partitioned setup,which matches the expectations discussed
earlier.Just sharing these observations in case they are useful for the
ongoing review.
Thanks again for the work on this patch.

Best regards,
Lakshmi

On Wed, Feb 11, 2026 at 5:53=E2=80=AFPM Hayato Kuroda (Fujitsu) <
kuroda.hayato@fujitsu.com> wrote:

> Dear Mircea,
>
> Thanks for the proposal. I also feel the initalization wastes time.
> Here are my initial comments.
>
> 01.
> I found that pgbench raises a FATAL in case of -j > --partitions, is ther=
e
> a
> specific reason?
> If needed, we may choose the softer way, which adjust nthreads up to the
> number
> of partitions. -c and -j do the similar one:
>
> ```
> if (nthreads > nclients && !is_init_mode)
> nthreads =3D nclients;
> ```
>
> 02.
> Also, why is -j accepted in case of non-partitions?
>
> 03.
> Can we port all validation to main()? I found initPopulateTableParallel()
> has
> such a part.
>
> 04.
> Copying seems to be divided into chunks per COPY_BATCH_SIZE. Is it really
> essential to parallelize the initialization? I feel it may optimize even
> serialized case thus can be discussed independently.
>
> 05.
> Per my understanding, each thread creates its tables, and all of them are
> attached to the parent table. Is it right? I think it needs more code
> changes, and I am not sure it is critical to make initialization faster.
>
> So I suggest using the incremental approach. The first patch only
> parallelizes
> the data load, and the second patch implements the CREATE TABLE and ALTER
> TABLE
> ATTACH PARTITION. You can benchmark three patterns, master, 0001, and
> 0001 + 0002, then compare the results. IIUC, this is the common approach =
to
> reduce the patch size and make them more reviewable.
>
> 06.
> Missing update for typedefs.list. WorkerTask and CopyTarget can be added
> there.
>
> 07.
> Since there is a report like [1], you can benchmark more cases.
>
> [1]:
> https://www.postgresql.org/message-id/CAEvyyTht69zjnosPjziW6dqNLqs-n6eKia=
2vof108zQp1QFX%3DQ%40mail.gmail.com
>
> Best regards,
> Hayato Kuroda
> FUJITSU LIMITED
>

--0000000000003396a8064afee604
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">Hi Mircea, Hayato,<br>I ran a few more te=
sts on 19devel=C2=A0,focusing on the partitioned case to better understand =
the performance behavior.<br><br>For scale 500, the serial initialization o=
n my system takes around 34.3 seconds. Using parallel initialization withou=
t partitions (-j 10) makes the client-side data generation noticeably faste=
r,But the overall runtime ends up slightly higher because the vacuum phase =
becomes much longer.<br><br>However,when running with partitions(pgbench -i=
 -s 500 --partitions=3D10 -j 10),the total runtime drops to about 21.9 seco=
nds, and the vacuum cost is much smaller.I also verified that the row count=
s are correct in all cases ,and regression tests still pass locally.<br><br=
>So it looks like the main benefit of parallel initialization shows up clea=
rly in the partitioned setup,which matches the expectations discussed earli=
er.Just sharing these observations in case they are useful for the ongoing =
review.<br>Thanks again for the work on this patch.<br><br>Best regards,<br=
>Lakshmi<br><br></div><div class=3D"gmail_quote gmail_quote_container"><div=
 dir=3D"ltr" class=3D"gmail_attr">On Wed, Feb 11, 2026 at 5:53=E2=80=AFPM H=
ayato Kuroda (Fujitsu) &lt;<a href=3D"mailto:kuroda.hayato@fujitsu.com">kur=
oda.hayato@fujitsu.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_q=
uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2=
04);padding-left:1ex">Dear Mircea,<br>
<br>
Thanks for the proposal. I also feel the initalization wastes time.<br>
Here are my initial comments.<br>
<br>
01.<br>
I found that pgbench raises a FATAL in case of -j &gt; --partitions, is the=
re a<br>
specific reason?<br>
If needed, we may choose the softer way, which adjust nthreads up to the nu=
mber<br>
of partitions. -c and -j do the similar one:<br>
<br>
```<br>
if (nthreads &gt; nclients &amp;&amp; !is_init_mode)<br>
nthreads =3D nclients;<br>
```<br>
<br>
02.<br>
Also, why is -j accepted in case of non-partitions?<br>
<br>
03.<br>
Can we port all validation to main()? I found initPopulateTableParallel() h=
as<br>
such a part.<br>
<br>
04.<br>
Copying seems to be divided into chunks per COPY_BATCH_SIZE. Is it really<b=
r>
essential to parallelize the initialization? I feel it may optimize even<br=
>
serialized case thus can be discussed independently.<br>
<br>
05.<br>
Per my understanding, each thread creates its tables, and all of them are<b=
r>
attached to the parent table. Is it right? I think it needs more code<br>
changes, and I am not sure it is critical to make initialization faster.<br=
>
<br>
So I suggest using the incremental approach. The first patch only paralleli=
zes<br>
the data load, and the second patch implements the CREATE TABLE and ALTER T=
ABLE<br>
ATTACH PARTITION. You can benchmark three patterns, master, 0001, and<br>
0001 + 0002, then compare the results. IIUC, this is the common approach to=
<br>
reduce the patch size and make them more reviewable.<br>
<br>
06.<br>
Missing update for typedefs.list. WorkerTask and CopyTarget can be added th=
ere.<br>
<br>
07.<br>
Since there is a report like [1], you can benchmark more cases.<br>
<br>
[1]: <a href=3D"https://www.postgresql.org/message-id/CAEvyyTht69zjnosPjziW=
6dqNLqs-n6eKia2vof108zQp1QFX%3DQ%40mail.gmail.com" rel=3D"noreferrer" targe=
t=3D"_blank">https://www.postgresql.org/message-id/CAEvyyTht69zjnosPjziW6dq=
NLqs-n6eKia2vof108zQp1QFX%3DQ%40mail.gmail.com</a><br>
<br>
Best regards,<br>
Hayato Kuroda<br>
FUJITSU LIMITED<br>
</blockquote></div></div>

--0000000000003396a8064afee604--