pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Richard Huxton <dev@archonet.com>
To: Qing Zhao <qzhao@quotefx.net>
To: Kevin Barnard <kbarnard@speedfc.com>
Cc: pgsql-performance@postgresql.org
Subject: Bulk Record upload (was Re: Increasing number of PG connections)
Date: Mon, 2 Feb 2004 20:52:19 +0000
Message-ID: <200402022052.19253.dev@archonet.com> (raw)
In-Reply-To: <8133E3F7-55B7-11D8-AEAC-000A95AB8896@quotefx.net>
References: <401E4D27.26927.3254D8C1@localhost>
	<8133E3F7-55B7-11D8-AEAC-000A95AB8896@quotefx.net>

On Monday 02 February 2004 19:39, Qing Zhao wrote:
> I am new here. I have a question related to this in some way.

Hmm - no real connection I can see - might have been better to start a new 
thread rather than replying to this one. Also, it is usually considered best 
practice not to quote large amounts of the previous message if you're not 
replying to it,

> Our web site needs to upload a large volume of data into Postgres at a
> time. The performance deterioates as number of rows becomes larger.
> When it reaches 2500 rows, it never come back to GUI. Since the tests
> were run through GUI, my suspision is
> that it might be caused by the way the application server talking to
> Postgres server, the connections, etc.. What might be the factors
> involved here? Does anyone know?

You don't really give us enough information. What GUI are you talking about? 
How are you loading this data - as a series of INSERT statements, text-file 
with separators, from Access/MySQL etc?

In general, the fastest way to add a large number of rows is via the COPY sql 
command. Next best is to batch your inserts together into larger transactions 
of say 100-1000 inserts.

Two other things to be aware of are: use of VACUUM/ANALYZE and configuration 
tuning (see http://www.varlena.com/varlena/GeneralBits/Tidbits/index.php).

PG shouldn't have a problem with inserting a few thousand rows, so I suspect 
it's something to do with your application/GUI setup.

Hope that helps, if not try turning on statement logging for PG and then we 
can see what commands your GUI is sending.

-- 
  Richard Huxton
  Archonet Ltd



Message-ID: <200402022052.19253.dev@archonet.com>
Permalink:  ../200402022052.19253.dev@archonet.com/
Also on:    postgresql.org/message-id/200402022052.19253.dev@archonet.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: dev@archonet.com, qzhao@quotefx.net, kbarnard@speedfc.com
  Subject: Re: Bulk Record upload (was Re: Increasing number of PG connections)
  In-Reply-To: <200402022052.19253.dev@archonet.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox