MIME-Version: 1.0
References: <cb014f00-66b2-4328-a65e-d11c681c9f45@gmail.com>
 <CAEvyyTj0rEsgcQOQgkARbRPbupHR_mc=TUzHBBLNzd8JByUUTw@mail.gmail.com>
 <4c1d0b97-a5f8-472c-afdd-bdeb09b93f33@gmail.com>
 <OS9PR01MB12149A6CEE200E1A5A88D9F14F563A@OS9PR01MB12149.jpnprd01.prod.outlook.com>
 <CAEvyyTiQqd=rv3XUxc0YEaW-feopksBveZKKjVZNeSVG=GrY+A@mail.gmail.com>
 <TYRPR01MB121560B291DA3CD262CC7A09AF568A@TYRPR01MB12156.jpnprd01.prod.outlook.com>
 <CAEvyyTjPWfvJLn3c_G_zLRffZ3=YqzMYj6c5znaNxpHyZAg3XQ@mail.gmail.com>
 <10868918-cdf9-49dc-99af-8e8ccd6e368c@gmail.com>
 <CAEvyyTircZ-tHgap=J6Aog0CBgXp4Dqx6dHYyK1iqgfoT+8D_A@mail.gmail.com>
 <eb80b601-7600-4fa1-8b99-550d40c1745b@iki.fi>
 <ddca6fbd-1e99-4756-8b02-38f7962d464d@gmail.com>
 <OS9PR01MB12149E0E2CAAB378D264A35B1F5242@OS9PR01MB12149.jpnprd01.prod.outlook.com>
In-Reply-To: 
 <OS9PR01MB12149E0E2CAAB378D264A35B1F5242@OS9PR01MB12149.jpnprd01.prod.outlook.com>
From: lakshmi <lakshmigcdac@gmail.com>
Date: Mon, 13 Apr 2026 17:21:06 +0530
Message-ID: 
 <CAEvyyThGm4NHDnfCGeCCOZ1_nrB=Eqct6y55GGuW0_UpTAsu3g@mail.gmail.com>
Subject: Re: parallel data loading for pgbench -i
To: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="0000000000001f9797064f560b2e"
Archived-At: 
 <https://www.postgresql.org/message-id/CAEvyyThGm4NHDnfCGeCCOZ1_nrB%3DEqct6y55GGuW0_UpTAsu3g%40mail.gmail.com>
Precedence: bulk

--0000000000001f9797064f560b2e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Hayato,

Thanks for your feedback.

I tried a few runs with different partition counts. From what I saw,
performance doesn=E2=80=99t always improve with more partitions=E2=80=94in =
fact, higher
partition counts increase VACUUM time and slow things down.

I also agree that having control over the number of workers (like using -j)
would help balance this better.

Regarding TRUNCATE, I noticed it=E2=80=99s already done earlier, so it migh=
t be
worth checking if the extra TRUNCATE is needed.

I didn=E2=80=99t see memory issues in my tests, but I understand it could b=
ecome a
concern with many partitions.

Thanks again for the suggestions.

Best regards,
Lakshmi

On Mon, Apr 13, 2026 at 12:53=E2=80=AFPM Hayato Kuroda (Fujitsu) <
kuroda.hayato@fujitsu.com> wrote:

> Dear Mircea,
>
> Thanks for updating the patch. Now each worker looks like not to create
> each
> child tables, just run TRUNCATE and COPY. But I'm unclear why the TRUNCAT=
E
> is
> needed here. Isn't they truncated in
> initGenerateDataClientSide()->initTruncateTables()
> before launching threads?
> Also, the current API is questionable. E.g., we cannot work in series if
> --partition is
> specified. And I'm afraid OOM failure may be more likely to happen if the
> table has
> many partitions.
> Is it possible that we can have -p again for the initialization? We can
> require
> partitions >=3D nthreads or partitions % nthreads =3D=3D 0 at that time.
>
>
> Best regards,
> Hayato Kuroda
> FUJITSU LIMITED
>
>

--0000000000001f9797064f560b2e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Hayato,<br><br>Thanks for your feedback.<br><br>I tried=
 a few runs with different partition counts. From what I saw, performance d=
oesn=E2=80=99t always improve with more partitions=E2=80=94in fact, higher =
partition counts increase VACUUM time and slow things down.<br><br>I also a=
gree that having control over the number of workers (like using -j) would h=
elp balance this better.<br><br>Regarding TRUNCATE, I noticed it=E2=80=99s =
already done earlier, so it might be worth checking if the extra TRUNCATE i=
s needed.<br><br>I didn=E2=80=99t see memory issues in my tests, but I unde=
rstand it could become a concern with many partitions.<br><br>Thanks again =
for the suggestions.<br><br>Best regards, =C2=A0<br>Lakshmi</div><br><div c=
lass=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_=
attr">On Mon, Apr 13, 2026 at 12:53=E2=80=AFPM Hayato Kuroda (Fujitsu) &lt;=
<a href=3D"mailto:kuroda.hayato@fujitsu.com">kuroda.hayato@fujitsu.com</a>&=
gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0=
px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear =
Mircea,<br>
<br>
Thanks for updating the patch. Now each worker looks like not to create eac=
h<br>
child tables, just run TRUNCATE and COPY. But I&#39;m unclear why the TRUNC=
ATE is<br>
needed here. Isn&#39;t they truncated in initGenerateDataClientSide()-&gt;i=
nitTruncateTables()<br>
before launching threads?<br>
Also, the current API is questionable. E.g., we cannot work in series if --=
partition is<br>
specified. And I&#39;m afraid OOM failure may be more likely to happen if t=
he table has<br>
many partitions.<br>
Is it possible that we can have -p again for the initialization? We can req=
uire<br>
partitions &gt;=3D nthreads or partitions % nthreads =3D=3D 0 at that time.=
<br>
<br>
<br>
Best regards,<br>
Hayato Kuroda<br>
FUJITSU LIMITED<br>
<br>
</blockquote></div>

--0000000000001f9797064f560b2e--