Date: Fri, 4 Mar 2022 13:47:35 -0500
From: Bruce Momjian <bruce@momjian.us>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: aditya desai <admad123@gmail.com>,
	Pgsql Performance <pgsql-performance@lists.postgresql.org>
Subject: Re: Any way to speed up INSERT INTO
Message-ID: <YiJexxjLXukD8vhf@momjian.us>
References: 
 <CAN0SRDHkd=3j7_xu2G_mg0ku0uwdN+DDGxEn8gzO=9hb=UFtcw@mail.gmail.com>
 <3967140.1646419359@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3967140.1646419359@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/YiJexxjLXukD8vhf%40momjian.us>
Precedence: bulk

On Fri, Mar  4, 2022 at 01:42:39PM -0500, Tom Lane wrote:
> aditya desai <admad123@gmail.com> writes:
> > One of the service layer app is inserting Millions of records in a table
> > but one row at a time. Although COPY is the fastest way to import a file in
> > a table. Application has a requirement of processing a row and inserting it
> > into a table. Is there any way this INSERT can be tuned by increasing
> > parameters? It is taking almost 10 hours for just 2.2 million rows in a
> > table. Table does not have any indexes or triggers.
> 
> Using a prepared statement for the INSERT would help a little bit.

Yeah, I thought about that but it seems it would only minimally help.

> What would help more, if you don't expect any insertion failures,
> is to group multiple inserts per transaction (ie put BEGIN ... COMMIT
> around each batch of 100 or 1000 or so insertions).  There's not
> going to be any magic bullet that lets you get away without changing
> the app, though.

Yeah, he/she could insert via multiple rows too:

	CREATE TABLE test (x int);
	INSERT INTO test VALUES (1), (2), (3);
	
> It's quite possible that network round trip costs are a big chunk of your
> problem, in which case physically grouping multiple rows into each INSERT
> command (... or COPY ...) is the only way to fix it.  But I'd start with
> trying to reduce the transaction commit overhead.

Agreed, turning off synchronous_commit for that those queries would be
my first approach.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  If only the physical world exists, free will is an illusion.