MIME-Version: 1.0
References: <cb014f00-66b2-4328-a65e-d11c681c9f45@gmail.com>
 <CAEvyyTj0rEsgcQOQgkARbRPbupHR_mc=TUzHBBLNzd8JByUUTw@mail.gmail.com>
 <4c1d0b97-a5f8-472c-afdd-bdeb09b93f33@gmail.com>
 <OS9PR01MB12149A6CEE200E1A5A88D9F14F563A@OS9PR01MB12149.jpnprd01.prod.outlook.com>
 <CAEvyyTiQqd=rv3XUxc0YEaW-feopksBveZKKjVZNeSVG=GrY+A@mail.gmail.com>
 <TYRPR01MB121560B291DA3CD262CC7A09AF568A@TYRPR01MB12156.jpnprd01.prod.outlook.com>
 <CAEvyyTjPWfvJLn3c_G_zLRffZ3=YqzMYj6c5znaNxpHyZAg3XQ@mail.gmail.com>
 <10868918-cdf9-49dc-99af-8e8ccd6e368c@gmail.com>
 <CAEvyyTircZ-tHgap=J6Aog0CBgXp4Dqx6dHYyK1iqgfoT+8D_A@mail.gmail.com>
 <eb80b601-7600-4fa1-8b99-550d40c1745b@iki.fi>
 <ddca6fbd-1e99-4756-8b02-38f7962d464d@gmail.com>
In-Reply-To: <ddca6fbd-1e99-4756-8b02-38f7962d464d@gmail.com>
From: lakshmi <lakshmigcdac@gmail.com>
Date: Mon, 13 Apr 2026 11:44:18 +0530
Message-ID: 
 <CAEvyyTjt1_QXO_37h1hbVqWdONm+uopV74j3K2pS5VrLKmozsw@mail.gmail.com>
Subject: Re: parallel data loading for pgbench -i
To: Mircea Cadariu <cadariu.mircea@gmail.com>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>,
 "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
 "tomas@vondra.me" <tomas@vondra.me>
Content-Type: multipart/alternative; boundary="0000000000009b26e6064f5156c8"
Archived-At: 
 <https://www.postgresql.org/message-id/CAEvyyTjt1_QXO_37h1hbVqWdONm%2BuopV74j3K2pS5VrLKmozsw%40mail.gmail.com>
Precedence: bulk

--0000000000009b26e6064f5156c8
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Mircea, Heikki,

I tested the v3 patch on 19devel with larger scale factors.

The behavior looks much better now compared to the earlier versions. For
scale 100 and 500, I see clear improvements in overall runtime, and for
scale 2000, the total time is around 97s on my system.

The loading phase now runs concurrently across workers, and I don=E2=80=99t=
 see the
earlier serialization behavior anymore.

The VACUUM phase also remains relatively small (~6s for scale 2000), which
suggests that the previous overhead has been addressed.

I also verified correctness, and the row counts match the expected values.

Overall, the partitioned parallel approach looks solid and scales well in
my tests.

Thanks again for the work on this.

Best regards,
Lakshmi

On Sat, Apr 11, 2026 at 12:07=E2=80=AFAM Mircea Cadariu <cadariu.mircea@gma=
il.com>
wrote:

> Hi,
>
> On 07/04/2026 10:00, Heikki Linnakangas wrote:
> >
> > This all makes more sense in the partitioned case. Perhaps we should
> > parallelize only when partitioned are used, and use only one thread
> > per partition.
> >
> Thanks for having a look. I attached v3 that parallelizes only the
> partitioned case, one thread per partition. Results:
>
>
> patch:
>
> pgbench -i -s 100 --partitions 10
>
> done in 12.63 s (drop tables 0.05 s, create tables 0.01 s, client-side
> generate 5.98 s, vacuum 1.63 s, primary keys 4.96 s).
>
>
> master:
>
> pgbench -i -s 100 --partitions 10
>
> done in 29.29 s (drop tables 0.00 s, create tables 0.02 s, client-side
> generate 16.31 s, vacuum 7.78 s, primary keys 5.18 s).
>
> --
> Thanks,
> Mircea Cadariu
>

--0000000000009b26e6064f5156c8
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><p>Hi Mircea, Heikki,</p>
<p>I tested the v3 patch on 19devel with larger scale factors.</p>
<p>The behavior looks much better now compared to the earlier versions. For=
 scale 100 and 500, I see clear improvements in overall runtime, and for sc=
ale 2000, the total time is around 97s on my system.</p>
<p>The loading phase now runs concurrently across workers, and I don=E2=80=
=99t see the earlier serialization behavior anymore.</p>
<p>The VACUUM phase also remains relatively small (~6s for scale 2000), whi=
ch suggests that the previous overhead has been addressed.</p>
<p>I also verified correctness, and the row counts match the expected value=
s.</p>
<p>Overall, the partitioned parallel approach looks solid and scales well i=
n my tests.</p>
<p>Thanks again for the work on this.</p>
<p>Best regards,<br>
Lakshmi</p></div><br><div class=3D"gmail_quote gmail_quote_container"><div =
dir=3D"ltr" class=3D"gmail_attr">On Sat, Apr 11, 2026 at 12:07=E2=80=AFAM M=
ircea Cadariu &lt;<a href=3D"mailto:cadariu.mircea@gmail.com">cadariu.mirce=
a@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=
=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding=
-left:1ex">Hi,<br>
<br>
On 07/04/2026 10:00, Heikki Linnakangas wrote:<br>
&gt;<br>
&gt; This all makes more sense in the partitioned case. Perhaps we should <=
br>
&gt; parallelize only when partitioned are used, and use only one thread <b=
r>
&gt; per partition.<br>
&gt;<br>
Thanks for having a look. I attached v3 that parallelizes only the <br>
partitioned case, one thread per partition. Results:<br>
<br>
<br>
patch:<br>
<br>
pgbench -i -s 100 --partitions 10<br>
<br>
done in 12.63 s (drop tables 0.05 s, create tables 0.01 s, client-side <br>
generate 5.98 s, vacuum 1.63 s, primary keys 4.96 s).<br>
<br>
<br>
master:<br>
<br>
pgbench -i -s 100 --partitions 10<br>
<br>
done in 29.29 s (drop tables 0.00 s, create tables 0.02 s, client-side <br>
generate 16.31 s, vacuum 7.78 s, primary keys 5.18 s).<br>
<br>
-- <br>
Thanks,<br>
Mircea Cadariu<br>
</blockquote></div>

--0000000000009b26e6064f5156c8--