X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 0C7C55E4075 for ; Sat, 7 Aug 2004 01:34:45 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 29299-05 for ; Sat, 7 Aug 2004 04:34:52 +0000 (GMT) Received: from mpls-qmqp-03.inet.qwest.net (mpls-qmqp-03.inet.qwest.net [63.231.195.114]) by svr1.postgresql.org (Postfix) with SMTP id 9FD855E46BF for ; Sat, 7 Aug 2004 01:34:42 -0300 (ADT) Received: (qmail 25435 invoked by uid 0); 7 Aug 2004 04:03:10 -0000 Received: from mpls-pop-03.inet.qwest.net (63.231.195.3) by mpls-qmqp-03.inet.qwest.net with QMQP; 7 Aug 2004 04:03:10 -0000 Received: from 63-227-127-37.dnvr.qwest.net (HELO ?10.0.0.2?) (63.227.127.37) by mpls-pop-03.inet.qwest.net with SMTP; 7 Aug 2004 04:34:49 -0000 Date: Fri, 06 Aug 2004 22:36:13 -0600 Message-Id: <1091853373.27166.260.camel@localhost.localdomain> From: "Scott Marlowe" To: "Martin Foster" Cc: pgsql-performance@postgresql.org Subject: Re: Performance Bottleneck In-Reply-To: References: <410FDC53.5080509@bigfoot.com> <4110FFF6.1030001@bigfoot.com> <41140D17.7070101@ethereal-realms.org> <41141322.3010200@bigfoot.com> <1091840178.27166.255.camel@localhost.localdomain> Content-Type: text/plain Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 (1.4.6-2) Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.1 tagged_above=0.0 required=5.0 tests=RCVD_IN_SORBS X-Spam-Level: X-Archive-Number: 200408/92 X-Sequence-Number: 7728 On Fri, 2004-08-06 at 22:02, Martin Foster wrote: > Scott Marlowe wrote: > > > On Fri, 2004-08-06 at 17:24, Gaetano Mendola wrote: > > > >>Martin Foster wrote: > >> > >> > >>>Gaetano Mendola wrote: > >>> > >>> > >>>> > >>>>Let start from your postgres configuration: > >>>> > >>>>shared_buffers = 8192 <==== This is really too small for your > >>>>configuration > >>>>sort_mem = 2048 > >>>> > >>>>wal_buffers = 128 <==== This is really too small for your > >>>>configuration > >>>> > >>>>effective_cache_size = 16000 > >>>> > >>>>change this values in: > >>>> > >>>>shared_buffers = 50000 > >>>>sort_mem = 16084 > >>>> > >>>>wal_buffers = 1500 > >>>> > >>>>effective_cache_size = 32000 > >>>> > >>>> > >>>>to bump up the shm usage you have to configure your OS in order to be > >>>>allowed to use that ammount of SHM. > >>>> > >>>>This are the numbers that I feel good for your HW, the second step now is > >>>>analyze your queries > >>>> > >>> > >>>These changes have yielded some visible improvements, with load averages > >>>rarely going over the anything noticeable. However, I do have a > >>>question on the matter, why do these values seem to be far higher then > >>>what a frequently pointed to document would indicate as necessary? > >>> > >>>http://www.varlena.com/GeneralBits/Tidbits/perf.html > >>> > >>>I am simply curious, as this clearly shows that my understanding of > >>>PostgreSQL is clearly lacking when it comes to tweaking for the hardware. > >> > >>Unfortunately there is no a "wizard tuning" for postgres so each one of > >>us have a own "school". The data I gave you are oversized to be sure > >>to achieve improvements. Now you can start to decrease these values > >>( starting from the wal_buffers ) in order to find the good compromise > >>with your HW. > > > > > > FYI, my school of tuning is to change one thing at a time some > > reasonable percentage (shared_buffers from 1000 to 2000) and measure the > > change under simulated load. Make another change, test it, chart the > > shape of the change line. It should look something like this for most > > folks: > > > > shared_buffers | q/s (more is better) > > 100 | 20 > > 200 | 45 > > 400 | 80 > > 1000 | 100 > > ... levels out here... > > 8000 | 110 > > 10000 | 108 > > 20000 | 40 > > 30000 | 20 > > > > Note it going back down as we exceed our memory and start swapping > > shared_buffers. Where that happens on your machine is determined by > > many things like your machine's memory, memory bandwidth, type of load, > > etc... but it will happen on most machines and when it does, it often > > happens at the worst times, under heavy parallel load. > > > > Unless testing shows it's faster, 10000 or 25% of mem (whichever is > > less) is usually a pretty good setting for shared_buffers. Large data > > sets may require more than 10000, but going over 25% on machines with > > large memory is usually a mistake, especially servers that do anything > > other than just PostgreSQL. > > > > You're absolutely right about one thing, there's no automatic wizard for > > tuning this stuff. > > > > Which rather points out the crux of the problem. This is a live system, > meaning changes made need to be as informed as possible, and that > changing values for the sake of testing can lead to potential problems > in service. But if you make those changes slowly, as I was showing, you should see the small deleterious effects like I was showing long before they become catastrophic. To just jump shared_buffers to 50000 is not a good idea, especially if the sweet spot is likely lower than that.