X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 949075342C for ; Mon, 4 Apr 2005 14:48:47 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 12982-07 for ; Mon, 4 Apr 2005 13:48:46 +0000 (GMT) Received: from rproxy.gmail.com (rproxy.gmail.com [64.233.170.195]) by svr1.postgresql.org (Postfix) with ESMTP id 0B40D53426 for ; Mon, 4 Apr 2005 14:48:43 +0100 (BST) Received: by rproxy.gmail.com with SMTP id a41so1190937rng for ; Mon, 04 Apr 2005 06:48:47 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=DKn/ktsdU9H/xIO6Lrt0t7/4yx/fr3SLkhUqr5ATZjFoQ91Fy3Nm/BCSOCVjmxj3aKdEO99TPfs624IV/gRLlx2nMHxBfdLq71jGy/VVD7esVp4wr6MTrfdQXq4HPzgWn75wj2Y89LAswBJXzPH+etDdCbvWeQSDAQq5kjCbvFI= Received: by 10.38.69.34 with SMTP id r34mr394704rna; Mon, 04 Apr 2005 06:48:47 -0700 (PDT) Received: by 10.38.207.41 with HTTP; Mon, 4 Apr 2005 06:48:47 -0700 (PDT) Message-ID: <59d991c40504040648457808e9@mail.gmail.com> Date: Mon, 4 Apr 2005 09:48:47 -0400 From: Christopher Petrilli Reply-To: Christopher Petrilli To: jd@commandprompt.com Subject: Re: Sustained inserts per sec ... ? Cc: Tom Lane , "Marc G. Fournier" , pgsql-performance@postgresql.org In-Reply-To: <59d991c4050401125956d6069d@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit References: <20050401140421.P18194@ganymede.hub.org> <59d991c4050401113857086517@mail.gmail.com> <28353.1112388128@sss.pgh.pa.us> <59d991c4050401124638c4ab5d@mail.gmail.com> <1112388818.3487.34.camel@jd2.commandprompt.com> <59d991c4050401125956d6069d@mail.gmail.com> X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.27 tagged_above=0 required=5 tests=AWL, DNS_FROM_RFC_ABUSE, RCVD_BY_IP X-Spam-Level: X-Archive-Number: 200504/48 X-Sequence-Number: 11445 On Apr 1, 2005 3:59 PM, Christopher Petrilli wrote: > On Apr 1, 2005 3:53 PM, Joshua D. Drake wrote: > > > > > What seems to happen is it slams into a "wall" of some sort, the > > > system goes into disk write frenzy (wait=90% CPU), and eventually > > > recovers and starts running for a while at a more normal speed. What > > > I need though, is to not have that wall happen. It is easier for me > > > to accept a constant degredation of 5%, rather than a 99% degredation > > > for short periods, as it can cause cascade problems in the system. > > > > Could this possibly be a checkpoint happening? > > > > Also how many checkpoint segments do you have? > > Changes to the postgresql.conf file from "default": > > maintenance_work_mem = 131072 > fsync = false > checkpoint_segments = 32 I've now had a chance to run a couple more tests, and here's two graphs of the time required to insert (via COPY from a file) 500 records at a time: http://www.amber.org/~petrilli/diagrams/pgsql_copy500.png http://www.amber.org/~petrilli/diagrams/pgsql_copy500_bgwriter.png The first is with the above changes, the second contains two additional modificiations to the configuration: bgwriter_percent = 25 bgwriter_maxpages = 1000 To my, likely faulty, intuition, it would seem that there is a backup happening in the moving of data from the WAL to the final resting place, and that by increasing these I could pull that forward. As you can see from the charts, that doesn't seem to have any major impact. The point, in the rough middle, is where the program begins inserting into a new table (inherited). The X axis is the "total" number of rows inserted. The table has: * 21 columns (nothing too strange) * No OIDS * 5 indexes, including the primary key on a string They are created by creating a main table, then doing: CREATE TABLE foo001 INHERITS (foos); And then recreating all the indexes. Thoughts? Any advice would be more than appreciated. Chris -- | Christopher Petrilli | petrilli@gmail.com