X-Original-To: pgsql-performance@postgresql.org Received: from spampd.localdomain (postgresql.org [64.49.215.8]) by postgresql.org (Postfix) with ESMTP id 587FD475F5F for ; Fri, 11 Apr 2003 12:16:20 -0400 (EDT) Received: from davinci.ethosmedia.com (unknown [209.10.40.251]) by postgresql.org (Postfix) with ESMTP id 5859F4762A2 for ; Fri, 11 Apr 2003 12:16:04 -0400 (EDT) Received: from [63.195.55.98] (HELO spooky) by davinci.ethosmedia.com (CommuniGate Pro SMTP 4.0.2) with ESMTP id 2956889; Fri, 11 Apr 2003 09:15:57 -0700 Content-Type: text/plain; charset="iso-8859-1" From: Josh Berkus Organization: Aglio Database Solutions To: "Ron Mayer" , "Matthew Nuzum" , "'Pgsql-Performance'" Subject: Re: Caching (was Re: choosing the right platform) Date: Fri, 11 Apr 2003 09:15:15 -0700 User-Agent: KMail/1.4.3 Cc: References: In-Reply-To: MIME-Version: 1.0 Message-Id: <200304110915.15060.josh@agliodbs.com> Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, hits=-31.6 required=5.0 tests=BAYES_10,IN_REP_TO,QUOTED_EMAIL_TEXT,REFERENCES, REPLY_WITH_QUOTES,USER_AGENT_KMAIL autolearn=ham version=2.50 X-Spam-Level: X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp) X-Archive-Number: 200304/122 X-Sequence-Number: 1628 Ron, > In a data warehouse we have here, we load about 3 million rows > each week; load time improved from about 9 to 7 hours > by breaking up such queries into expressions that only require > one sort at a time, and surrounding the expressions with > "set sort_mem=3Dsomething_big" statements to give it enough > space to not hit the disk. > > SET SORT_MEM=3D300000; > CREATE TEMPORARY TABLE potential_new_values AS > SELECT DISTINCT val FROM import_table; > ... > SET SORT_MEM=3D1000; > > Anyone else have similar experience, or am I doing something > wrong to need so much SORT_MEM? No, this sounds very reasonable to me. I do a similar operation on one of = my=20 systems as part of a nightly data transformation for reporting. Since I=20 haven't had to do those on tables over 150,000 rows, I haven't seen the kin= d=20 of RAM usage you experience. > Below is an example of another real-world query from the same > reporting system that benefits from a sort_mem over 32M. > Explain analyze (below) shows a 40% improvement by having > the sort fit in memory. Cool! That's a perfect example of sizing sort_mem for the query. Mind if= I=20 steal it for an article at some point? --=20 Josh Berkus Aglio Database Solutions San Francisco