X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 19CE35E3AC5 for ; Sat, 7 Aug 2004 07:04:43 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 64405-10 for ; Sat, 7 Aug 2004 10:04:34 +0000 (GMT) Received: from pns.mm.eutelsat.org (pns.mm.eutelsat.org [194.214.173.227]) by svr1.postgresql.org (Postfix) with ESMTP id 7A71C5E37CF for ; Sat, 7 Aug 2004 07:04:29 -0300 (ADT) Received: from nts-03.mm.eutelsat.org (localhost [127.0.0.1]) by pns.mm.eutelsat.org (8.11.6/linuxconf) with ESMTP id i77A42h22217; Sat, 7 Aug 2004 12:04:03 +0200 Received: from [127.0.0.1] (accesspoint.mm.eutelsat.org [194.214.173.4]) by nts-03.mm.eutelsat.org (8.11.6/linuxconf) with ESMTP id i77A4RZ09440; Sat, 7 Aug 2004 12:04:27 +0200 Message-ID: <4114A900.5070607@bigfoot.com> Date: Sat, 07 Aug 2004 12:03:44 +0200 From: Gaetano Mendola User-Agent: Mozilla Thunderbird 0.7.1 (Windows/20040626) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Scott Marlowe , "pgsql-performance@postgresql.org" Subject: Re: Performance Bottleneck References: <410FDC53.5080509@bigfoot.com> <4110FFF6.1030001@bigfoot.com> <41140D17.7070101@ethereal-realms.org> <41141322.3010200@bigfoot.com> <1091840178.27166.255.camel@localhost.localdomain> <1091853373.27166.260.camel@localhost.localdomain> In-Reply-To: <1091853373.27166.260.camel@localhost.localdomain> X-Enigmail-Version: 0.84.2.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200408/94 X-Sequence-Number: 7730 Scott Marlowe wrote: > On Fri, 2004-08-06 at 22:02, Martin Foster wrote: > >>Scott Marlowe wrote: >> >> >>>On Fri, 2004-08-06 at 17:24, Gaetano Mendola wrote: >>> >>> >>>>Martin Foster wrote: >>>> >>>> >>>> >>>>>Gaetano Mendola wrote: >>>>> >>>>> >>>>> >>>>>>Let start from your postgres configuration: >>>>>> >>>>>>shared_buffers = 8192 <==== This is really too small for your >>>>>>configuration >>>>>>sort_mem = 2048 >>>>>> >>>>>>wal_buffers = 128 <==== This is really too small for your >>>>>>configuration >>>>>> >>>>>>effective_cache_size = 16000 >>>>>> >>>>>>change this values in: >>>>>> >>>>>>shared_buffers = 50000 >>>>>>sort_mem = 16084 >>>>>> >>>>>>wal_buffers = 1500 >>>>>> >>>>>>effective_cache_size = 32000 >>>>>> >>>>>> >>>>>>to bump up the shm usage you have to configure your OS in order to be >>>>>>allowed to use that ammount of SHM. >>>>>> >>>>>>This are the numbers that I feel good for your HW, the second step now is >>>>>>analyze your queries >>>>>> >>>>> >>>>>These changes have yielded some visible improvements, with load averages >>>>>rarely going over the anything noticeable. However, I do have a >>>>>question on the matter, why do these values seem to be far higher then >>>>>what a frequently pointed to document would indicate as necessary? >>>>> >>>>>http://www.varlena.com/GeneralBits/Tidbits/perf.html >>>>> >>>>>I am simply curious, as this clearly shows that my understanding of >>>>>PostgreSQL is clearly lacking when it comes to tweaking for the hardware. >>>> >>>>Unfortunately there is no a "wizard tuning" for postgres so each one of >>>>us have a own "school". The data I gave you are oversized to be sure >>>>to achieve improvements. Now you can start to decrease these values >>>>( starting from the wal_buffers ) in order to find the good compromise >>>>with your HW. >>> >>> >>>FYI, my school of tuning is to change one thing at a time some >>>reasonable percentage (shared_buffers from 1000 to 2000) and measure the >>>change under simulated load. Make another change, test it, chart the >>>shape of the change line. It should look something like this for most >>>folks: >>> >>>shared_buffers | q/s (more is better) >>>100 | 20 >>>200 | 45 >>>400 | 80 >>>1000 | 100 >>>... levels out here... >>>8000 | 110 >>>10000 | 108 >>>20000 | 40 >>>30000 | 20 >>> >>>Note it going back down as we exceed our memory and start swapping >>>shared_buffers. Where that happens on your machine is determined by >>>many things like your machine's memory, memory bandwidth, type of load, >>>etc... but it will happen on most machines and when it does, it often >>>happens at the worst times, under heavy parallel load. >>> >>>Unless testing shows it's faster, 10000 or 25% of mem (whichever is >>>less) is usually a pretty good setting for shared_buffers. Large data >>>sets may require more than 10000, but going over 25% on machines with >>>large memory is usually a mistake, especially servers that do anything >>>other than just PostgreSQL. >>> >>>You're absolutely right about one thing, there's no automatic wizard for >>>tuning this stuff. >>> >> >>Which rather points out the crux of the problem. This is a live system, >>meaning changes made need to be as informed as possible, and that >>changing values for the sake of testing can lead to potential problems >>in service. > > > But if you make those changes slowly, as I was showing, you should see > the small deleterious effects like I was showing long before they become > catastrophic. To just jump shared_buffers to 50000 is not a good idea, > especially if the sweet spot is likely lower than that. As you can see 50000 are less then 20% of his total memory and I strongly fell that 50000 is not oversized for his hardware ( as wal_buffers isn't), may be could be for his database activity but for sure that value ( values ) can not be source of problems. I'd like to have a wizard that could be run also for hours in order to find the good compromise for all GUC parameters , may be a genetic algoritm can help. Regards Gaetano Mendola