Date: Wed, 04 Jun 2008 23:50:09 +0200
To: "andrew klassen" <aptklassen@yahoo.com>,
	"James Mansion" <james@mansionfamily.plus.com>
Subject: Re: insert/update tps slow with indices on table > 1M rows
From: PFC <lists@peufeu.com>
Cc: pgsql-performance@postgresql.org
Content-Type: text/plain; format=flowed; delsp=yes; charset=utf-8
MIME-Version: 1.0
References: <354284.6724.qm@web37304.mail.mud.yahoo.com>
Content-Transfer-Encoding: 8bit
Message-ID: <op.ub8uxvcrcigqcu@apollo13.peufeu.com>
In-Reply-To: <354284.6724.qm@web37304.mail.mud.yahoo.com>
User-Agent: Opera Mail/9.24 (Linux)


> I am using the c-library interface and for these particular transactions
> I preload PREPARE statements. Then as I get requests, I issue a BEGIN,
> followed by at most 300 EXECUTES and then a COMMIT. That is the
> general scenario. What value beyond 300 should I try?
> Thanks.

	Do you have PREPARE statements whose performance might change as the  
table grows ?

	I mean, some selects, etc... in that case if you start with an empty  
table, after inserting say 100K rows you might want to just disconnect,  
reconnect and analyze to trigger replanning of those statements.

> Also, how might COPY (which involves file I/O) improve the
> above scenario?

	It won't but if you see that COPY is very much faster than your INSERT  
based process it will give you a useful piece of information.

	I understand your problem is :

- Create table with indexes
- Insert batches of rows
- After a while it gets slow

	Try :

- Create table with indexes
- COPY huge batch of rows
- Compare time with above

	Since COPY also updates the indexes just like your inserts do it will  
tell you if it's the indexes which slow you down or something else.

	Also for insert heavy loads it's a good idea to put the xlog on a  
separate disk (to double your disk bandwidth) unless you have a monster  
disk setup.

	During your INSERTs, do you also make some SELECTs ? Do you have triggers  
on the table ? Foreign keys ? Anything ?
	How much RAM you have ? And can you measure the size of the table+indexes  
when it gets slow ?


>
>
> ----- Original Message ----
> From: James Mansion <james@mansionfamily.plus.com>
> To: andrew klassen <aptklassen@yahoo.com>
> Cc: pgsql-performance@postgresql.org
> Sent: Wednesday, June 4, 2008 3:20:26 PM
> Subject: Re: [PERFORM] insert/update tps slow with indices on table > 1M  
> rows
>
> andrew klassen wrote:
>> I'll try adding more threads to update the table as you suggest.
> You could try materially increasing the update batch size too.  As an
> exercise you could
> see what the performance of COPY is by backing out the data and
> reloading it from
> a suitable file.
>
>
>