MIME-Version: 1.0
References: <CAKna9VaVsDzfOfOGu1+grStp9BBHFMKrH5DCEbbtGcQUWJ74KQ@mail.gmail.com>
 <CAPnRvGswoifDeosThKM5zkFZzph5u86ozYkrETpGD=oig14pOA@mail.gmail.com>
In-Reply-To: <CAPnRvGswoifDeosThKM5zkFZzph5u86ozYkrETpGD=oig14pOA@mail.gmail.com>
From: Lok P <loknath.73@gmail.com>
Date: Fri, 6 Sep 2024 10:12:49 +0530
Message-ID: <CAKna9VYFp4Bqc8-9vyb_AbZH4xpXnVUsM_2papyw10WMpbjiOA@mail.gmail.com>
Subject: Re: Faster data load
To: Muhammad Usman Khan <usman.k@bitnine.net>
Cc: pgsql-general <pgsql-general@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000cfb2d506216c0add"
Archived-At: <https://www.postgresql.org/message-id/CAKna9VYFp4Bqc8-9vyb_AbZH4xpXnVUsM_2papyw10WMpbjiOA%40mail.gmail.com>
Precedence: bulk

--000000000000cfb2d506216c0add
Content-Type: text/plain; charset="UTF-8"

On Fri, 6 Sept, 2024, 9:20 am Muhammad Usman Khan, <usman.k@bitnine.net>
wrote:

> Hi,
>
> You can use pg_partman. If your table is partitioned, you can manage
> partitions in parallel by distributing the load across partitions
> concurrently. Or you can use citus. It can be an excellent solution,
> especially for handling large data volumes and parallelizing data operations
>


Thank you.
The tables are partitioned. Also during index creation we are trying to do
it multiple partitions at same time from multiple sessions.But seeing out
of memory error in 5th or 6th session. And even each sessions taking 30mins
per partitions for index creation. Attach index partitions happening in
seconds though.

>
>

--000000000000cfb2d506216c0add
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto"><div><br></div><div dir=3D"auto"><div class=3D"gmail_quot=
e" dir=3D"auto"><div dir=3D"ltr" class=3D"gmail_attr">On Fri, 6 Sept, 2024,=
 9:20 am Muhammad Usman Khan, &lt;<a href=3D"mailto:usman.k@bitnine.net">us=
man.k@bitnine.net</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote"=
 style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><d=
iv dir=3D"ltr"><div><div dir=3D"ltr" class=3D"gmail_signature" data-smartma=
il=3D"gmail_signature"><div dir=3D"ltr"><div>Hi,<br><br>You can use pg_part=
man. If your table is partitioned, you can manage partitions in parallel by=
 distributing the load across partitions concurrently. Or you can use citus=
. It can be an excellent solution, especially for handling large data volum=
es and parallelizing data operations<br></div></div></div></div></div></blo=
ckquote></div></div><div dir=3D"auto"><br></div><div dir=3D"auto"><br></div=
><div dir=3D"auto">Thank you.=C2=A0</div><div dir=3D"auto">The tables are p=
artitioned. Also during index creation we are trying to do it multiple part=
itions at same time from multiple sessions.But seeing out of memory error i=
n 5th or 6th session. And even each sessions taking 30mins per partitions f=
or index creation. Attach index partitions happening in seconds though.=C2=
=A0</div><div dir=3D"auto"><div class=3D"gmail_quote" dir=3D"auto"><blockqu=
ote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc s=
olid;padding-left:1ex"><div dir=3D"ltr"><div><div dir=3D"ltr" class=3D"gmai=
l_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div></div=
></div></div></div><br></div><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">
</blockquote></div>
</blockquote></div></div></div>

--000000000000cfb2d506216c0add--