MIME-Version: 1.0
References: <cb014f00-66b2-4328-a65e-d11c681c9f45@gmail.com>
In-Reply-To: <cb014f00-66b2-4328-a65e-d11c681c9f45@gmail.com>
From: lakshmi <lakshmigcdac@gmail.com>
Date: Mon, 19 Jan 2026 14:55:43 +0530
Message-ID: <CAEvyyTj0rEsgcQOQgkARbRPbupHR_mc=TUzHBBLNzd8JByUUTw@mail.gmail.com>
Subject: Re: parallel data loading for pgbench -i
To: Mircea Cadariu <cadariu.mircea@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000df34a10648ba3e92"
Archived-At: <https://www.postgresql.org/message-id/CAEvyyTj0rEsgcQOQgkARbRPbupHR_mc%3DTUzHBBLNzd8JByUUTw%40mail.gmail.com>
Precedence: bulk

--000000000000df34a10648ba3e92
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Mircea,

I tested the patch on 19devel and it worked well for me.
Before applying it, -j is rejected in pgbench initialization mode as
expected. After applying the patch, pgbench -i -s 100 -j 10 runs
successfully and shows a clear speedup.
On my system the total runtime dropped to about 9.6s, with client-side data
generation around 3.3s.
I also checked correctness after the run =E2=80=94 row counts for pgbench_a=
ccounts,
pgbench_branches, and pgbench_tellers all match the expected values.

Thanks for working on this, the improvement is very noticeable.

Best regards,
lakshmi


On Mon, Jan 19, 2026 at 2:30=E2=80=AFPM Mircea Cadariu <cadariu.mircea@gmai=
l.com>
wrote:

> Hi,
>
> I propose a patch for speeding up pgbench -i through multithreading.
>
> To enable this, pass -j and then the number of workers you want to use.
>
> Here are some results I got on my laptop:
>
>
> master
>
> ---
>
> -i -s 100
> done in 20.95 s (drop tables 0.00 s, create tables 0.01 s, client-side
> generate 14.51 s, vacuum 0.27 s, primary keys 6.16 s).
>
> -i -s 100 --partitions=3D10
> done in 29.73 s (drop tables 0.00 s, create tables 0.02 s, client-side
> generate 16.33 s, vacuum 8.72 s, primary keys 4.67 s).
>
>
> patch (-j 10)
>
> ---
>
> -i -s 100 -j 10
> done in 18.64 s (drop tables 0.00 s, create tables 0.01 s, client-side
> generate 5.82 s, vacuum 6.89 s, primary keys 5.93 s).
>
> -i -s 100 -j 10 --partitions=3D10
> done in 14.66 s (drop tables 0.00 s, create tables 0.01 s, client-side
> generate 8.42 s, vacuum 1.55 s, primary keys 4.68 s).
>
> The speedup is more significant for the partitioned use-case. This is
> because all workers can use COPY FREEZE (thus incurring a lower vacuum
> penalty) because they create their separate partitions.
>
> For the non-partitioned case the speedup is lower, but I observe it
> improves somewhat with larger scale factors. When parallel vacuum
> support is merged, this should further reduce the time.
>
> I'd still need to update docs, tests, better integrate the code with its
> surroundings, and other aspects. Would appreciate any feedback on what I
> have so far though. Thanks!
>
> Kind regards,
>
> Mircea Cadariu
>
>

--000000000000df34a10648ba3e92
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><p>Hi Mircea,<br></p>
<p>I tested the patch on 19devel and it worked well for me.<br>Before apply=
ing it, <code>-j</code> is rejected in pgbench initialization mode as expec=
ted. After applying the patch, <code>pgbench -i -s 100 -j 10</code> runs su=
ccessfully and shows a clear speedup.<br> On my system the total runtime dr=
opped to about 9.6s, with client-side data generation around 3.3s.=C2=A0<br=
>I also checked correctness after the run =E2=80=94 row counts for pgbench_=
accounts, pgbench_branches, and pgbench_tellers all match the expected valu=
es.<br></p><p>Thanks for working on this, the improvement is very noticeabl=
e.</p><p>
</p><p>Best regards,<br>lakshmi</p><p><br></p></div><br><div class=3D"gmail=
_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Mon,=
 Jan 19, 2026 at 2:30=E2=80=AFPM Mircea Cadariu &lt;<a href=3D"mailto:cadar=
iu.mircea@gmail.com">cadariu.mircea@gmail.com</a>&gt; wrote:<br></div><bloc=
kquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:=
1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I propose a patch for speeding up pgbench -i through multithreading.<br>
<br>
To enable this, pass -j and then the number of workers you want to use.<br>
<br>
Here are some results I got on my laptop:<br>
<br>
<br>
master<br>
<br>
---<br>
<br>
-i -s 100<br>
done in 20.95 s (drop tables 0.00 s, create tables 0.01 s, client-side <br>
generate 14.51 s, vacuum 0.27 s, primary keys 6.16 s).<br>
<br>
-i -s 100 --partitions=3D10<br>
done in 29.73 s (drop tables 0.00 s, create tables 0.02 s, client-side <br>
generate 16.33 s, vacuum 8.72 s, primary keys 4.67 s).<br>
<br>
<br>
patch (-j 10)<br>
<br>
---<br>
<br>
-i -s 100 -j 10<br>
done in 18.64 s (drop tables 0.00 s, create tables 0.01 s, client-side <br>
generate 5.82 s, vacuum 6.89 s, primary keys 5.93 s).<br>
<br>
-i -s 100 -j 10 --partitions=3D10<br>
done in 14.66 s (drop tables 0.00 s, create tables 0.01 s, client-side <br>
generate 8.42 s, vacuum 1.55 s, primary keys 4.68 s).<br>
<br>
The speedup is more significant for the partitioned use-case. This is <br>
because all workers can use COPY FREEZE (thus incurring a lower vacuum <br>
penalty) because they create their separate partitions.<br>
<br>
For the non-partitioned case the speedup is lower, but I observe it <br>
improves somewhat with larger scale factors. When parallel vacuum <br>
support is merged, this should further reduce the time.<br>
<br>
I&#39;d still need to update docs, tests, better integrate the code with it=
s <br>
surroundings, and other aspects. Would appreciate any feedback on what I <b=
r>
have so far though. Thanks!<br>
<br>
Kind regards,<br>
<br>
Mircea Cadariu<br>
<br>
</blockquote></div>

--000000000000df34a10648ba3e92--