Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <pgsql-general-owner+archive@lists.postgresql.org>)
	id 1smKqt-004xFw-Hw
	for pgsql-general@arkaria.postgresql.org; Thu, 05 Sep 2024 22:21:19 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.94.2)
	(envelope-from <pgsql-general-owner+archive@lists.postgresql.org>)
	id 1smKqp-001J7H-Qf
	for pgsql-general@arkaria.postgresql.org; Thu, 05 Sep 2024 22:21:16 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
	by malur.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <jross@openvistas.net>)
	id 1smKqp-001J1v-FB
	for pgsql-general@lists.postgresql.org; Thu, 05 Sep 2024 22:21:15 +0000
Received: from luna.openvistas.net ([207.158.15.156] helo=openvistas.net)
	by magus.postgresql.org with esmtp (Exim 4.94.2)
	(envelope-from <jross@openvistas.net>)
	id 1smKqm-000LxJ-P0
	for pgsql-general@lists.postgresql.org; Thu, 05 Sep 2024 22:21:14 +0000
Received: (qmail 440 invoked from network); 5 Sep 2024 22:21:10 -0000
Received: from unknown (HELO ?10.0.26.96?) (jross@154.27.111.79)
  de/crypted with TLSv1.3: AEAD-AES128-GCM-SHA256 [128/128] DN=none
  by mail.openvistas.net with ESMTPSA; 5 Sep 2024 22:21:10 -0000
Message-ID: <640e21b1-a841-40ae-a5eb-e6ee548520a1@openvistas.net>
Date: Thu, 5 Sep 2024 16:21:09 -0600
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Faster data load
Content-Language: en-US
To: pgsql-general@lists.postgresql.org
References: <CAKna9VaVsDzfOfOGu1+grStp9BBHFMKrH5DCEbbtGcQUWJ74KQ@mail.gmail.com>
From: Jeff Ross <jross@openvistas.net>
In-Reply-To: <CAKna9VaVsDzfOfOGu1+grStp9BBHFMKrH5DCEbbtGcQUWJ74KQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
List-Id: <pgsql-general.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-general@lists.postgresql.org>
List-Owner: <mailto:pgsql-general-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-general>
Archived-At: <https://www.postgresql.org/message-id/640e21b1-a841-40ae-a5eb-e6ee548520a1%40openvistas.net>
Precedence: bulk

On 9/5/24 14:14, Lok P wrote:

> Hi,
>
> We are having a requirement to create approx 50 billion rows in a 
> partition table(~1 billion rows per partition, 200+gb size daily 
> partitions) for a performance test. We are currently using ' insert 
> into <target table_partition> select.. From <source_table_partition> 
> or <some transformed query>;' method . We have dropped all indexes and 
> constraints First and then doing the load. Still it's taking 2-3 hours 
> to populate one partition. Is there a faster way to achieve this?
>
> Few teammate suggesting to use copy command and use file load instead, 
> which will be faster. So I wanted to understand, how different things 
> it does behind the scenes as compared to insert as select command? As 
> because it only deals with sql engine only.
>
> Additionally, when we were trying to create indexes post data load on 
> one partition, it took 30+ minutes. Any possible way to make it faster?
>
> Is there any way to drive the above things in parallel by utilizing 
> full database resources?
>
> It's postgres 15.4
>
> Regards
> Lok

Try pg_bulkload to load the data--takes a little set up but it is very 
fast.  Do pay attention to the caveats.  For a performance test they 
probably won't be relevant.

https://github.com/ossc-db/pg_bulkload?tab=readme-ov-file

Jeff