pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Simon Riggs <simon@2ndquadrant.com>
To: Christopher Petrilli <petrilli@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: jd@commandprompt.com, "Marc G. Fournier" <scrappy@postgresql.org>
Cc: pgsql-performance@postgresql.org
Subject: Re: Sustained inserts per sec ... ?
Date: Tue, 05 Apr 2005 20:48:29 +0100
Message-ID: <1112730509.16721.918.camel@localhost.localdomain> (raw)
In-Reply-To: <5217.1112668589@sss.pgh.pa.us>
References: <20050401140421.P18194@ganymede.hub.org>
	<59d991c4050401113857086517@mail.gmail.com>
	<28353.1112388128@sss.pgh.pa.us>
	<59d991c4050401124638c4ab5d@mail.gmail.com>
	<1112388818.3487.34.camel@jd2.commandprompt.com>
	<59d991c4050401125956d6069d@mail.gmail.com>
	<59d991c40504040648457808e9@mail.gmail.com>
	<26454.1112629979@sss.pgh.pa.us>
	<59d991c405040409094e7bbd17@mail.gmail.com>
	<26669.1112631807@sss.pgh.pa.us>
	<59d991c40504041846bc39a6@mail.gmail.com>
	<5217.1112668589@sss.pgh.pa.us>

On Mon, 2005-04-04 at 22:36 -0400, Tom Lane wrote:
> Christopher Petrilli <petrilli@gmail.com> writes:
> > On Apr 4, 2005 12:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> do a test run with *no* indexes on the table, just to see if it behaves
> >> any differently?  Basically I was wondering if index overhead might be
> >> part of the problem.
> 
> > http://www.amber.org/~petrilli/diagrams/pgsql_copy500_pkonly.png
> 
> > I appologize, I forgot to kill the PK, but as you can see, the curve
> > flattened out a lot.  It still begins to increase in what seems like
> > the same place.  You can find the results themselves at:
>
> Yeah, this confirms the thought that the indexes are the source of
> the issue.  (Which is what I'd expect, because a bare INSERT ought to be
> an approximately constant-time operation.  But it's good to verify.)

Yup, indexes are the best explanation so far - block extension needs
some work, but I doubted that it was the source of this effect.

> Now some amount of slowdown is to be expected as the indexes get larger,
> since it ought to take roughly O(log N) time to insert a new entry in an
> index of size N.  The weird thing about your curves is the very sudden
> jump in the insert times.

Well, ISTM that the curve is far from unique. Mark's OSDL tests show
them too. What was wierd, for me, was that it "resets" when you move to
a new table. The index theory does accurately explain that.

Perhaps the jump is not so sudden? Do I see a first small step up at
about 4.5M rows, then another much bigger one at 7.5M (which looks like
the only one at first glance)?

> What I think might be happening is that the "working set" of pages
> touched during index inserts is gradually growing, and at some point it
> exceeds shared_buffers, and at that point performance goes in the toilet
> because we are suddenly doing lots of reads to pull in index pages that
> fell out of the shared buffer area.

So this does seem to be the best explanation and it seems a good one.

It's also an excellent advert for table and index partitioning, and some
damning evidence against global indexes on partitioned tables (though
they may still be better than the alternative...)

> The indicated fix of course is to increase shared_buffers.

Splitting your tables at 4M, not 10M would work even better.

..

Anyway, where most of this started was with Christopher's comments:

On Fri, 2005-04-01 at 14:38 -0500, Christopher Petrilli wrote: 
> This was an application originally written for MySQL/MYISAM, and it's
> looking like PostgreSQL can't hold up for it, simply because it's "too
> much database" if that makes sense.  The same box, running the MySQL
> implementation (which uses no transactions) runs around 800-1000
> rows/second systained.

B-trees aren't unique to PostgreSQL; the explanation developed here
would work equally well for any database system that used tree-based
indexes. Do we still think that MySQL can do this when PostgreSQL
cannot? How?

Do we have performance test results showing the same application load
without the degradation? We don't need to look at the source code to
measure MySQL performance...

Best Regards, Simon Riggs





view thread (44+ messages)  latest in thread

Message-ID: <1112730509.16721.918.camel@localhost.localdomain>
Permalink:  ../1112730509.16721.918.camel@localhost.localdomain/
Also on:    postgresql.org/message-id/1112730509.16721.918.camel@localhost.localdomain

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: simon@2ndquadrant.com, petrilli@gmail.com, tgl@sss.pgh.pa.us, scrappy@postgresql.org
  Subject: Re: Sustained inserts per sec ... ?
  In-Reply-To: <1112730509.16721.918.camel@localhost.localdomain>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox