X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id D18CBD1B4DA for ; Mon, 2 Feb 2004 20:52:26 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 84473-07 for ; Mon, 2 Feb 2004 16:52:26 -0400 (AST) Received: from anchor-post-35.mail.demon.net (anchor-post-35.mail.demon.net [194.217.242.85]) by svr1.postgresql.org (Postfix) with ESMTP id F41ABD1B4BD for ; Mon, 2 Feb 2004 16:52:22 -0400 (AST) Received: from mwynhau.demon.co.uk ([193.237.186.96] helo=mainbox.archonet.com) by anchor-post-35.mail.demon.net with esmtp (Exim 3.35 #1) id 1Anl3P-000G5v-0Z; Mon, 02 Feb 2004 20:52:23 +0000 Received: by mainbox.archonet.com (Postfix, from userid 529) id BC7B716401; Mon, 2 Feb 2004 20:52:22 +0000 (GMT) Received: from client17.archonet.com (client17.archonet.com [192.168.1.17]) by mainbox.archonet.com (Postfix) with ESMTP id 2434B1593E; Mon, 2 Feb 2004 20:52:20 +0000 (GMT) From: Richard Huxton To: Qing Zhao , Kevin Barnard Subject: Bulk Record upload (was Re: Increasing number of PG connections) Date: Mon, 2 Feb 2004 20:52:19 +0000 User-Agent: KMail/1.5 Cc: pgsql-performance@postgresql.org References: <401E4D27.26927.3254D8C1@localhost> <8133E3F7-55B7-11D8-AEAC-000A95AB8896@quotefx.net> In-Reply-To: <8133E3F7-55B7-11D8-AEAC-000A95AB8896@quotefx.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200402022052.19253.dev@archonet.com> X-Bogosity: No, tests=bogofilter, spamicity=0.000000, version=0.15.3 X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200402/16 X-Sequence-Number: 5540 On Monday 02 February 2004 19:39, Qing Zhao wrote: > I am new here. I have a question related to this in some way. Hmm - no real connection I can see - might have been better to start a new thread rather than replying to this one. Also, it is usually considered best practice not to quote large amounts of the previous message if you're not replying to it, > Our web site needs to upload a large volume of data into Postgres at a > time. The performance deterioates as number of rows becomes larger. > When it reaches 2500 rows, it never come back to GUI. Since the tests > were run through GUI, my suspision is > that it might be caused by the way the application server talking to > Postgres server, the connections, etc.. What might be the factors > involved here? Does anyone know? You don't really give us enough information. What GUI are you talking about? How are you loading this data - as a series of INSERT statements, text-file with separators, from Access/MySQL etc? In general, the fastest way to add a large number of rows is via the COPY sql command. Next best is to batch your inserts together into larger transactions of say 100-1000 inserts. Two other things to be aware of are: use of VACUUM/ANALYZE and configuration tuning (see http://www.varlena.com/varlena/GeneralBits/Tidbits/index.php). PG shouldn't have a problem with inserting a few thousand rows, so I suspect it's something to do with your application/GUI setup. Hope that helps, if not try turning on statement logging for PG and then we can see what commands your GUI is sending. -- Richard Huxton Archonet Ltd