pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Josh Berkus <josh@agliodbs.com>
To: Ron Mayer <ron@intervideo.com>
To: Matthew Nuzum <cobalt@bearfruit.org>
To: 'Pgsql-Performance' <pgsql-performance@postgresql.org>
Cc: ron@intervideo.com
Subject: Re: Caching (was Re: choosing the right platform)
Date: Fri, 11 Apr 2003 09:15:15 -0700
Message-ID: <200304110915.15060.josh@agliodbs.com> (raw)
In-Reply-To: <POEDIPIPKGJJLDNIEMBEOEAACJAA.ron@intervideo.com>
References: <POEDIPIPKGJJLDNIEMBEOEAACJAA.ron@intervideo.com>

Ron,

> In a data warehouse we have here, we load about 3 million rows
> each week; load time improved from about 9 to 7 hours
> by breaking up such queries into expressions that only require
> one sort at a time, and surrounding the expressions with
> "set sort_mem=something_big" statements to give it enough
> space to not hit the disk.
>
>   SET SORT_MEM=300000;
>   CREATE TEMPORARY TABLE potential_new_values AS
>      SELECT DISTINCT val FROM import_table;
>   ...
>   SET SORT_MEM=1000;
>
> Anyone else have similar experience, or am I doing something
> wrong to need so much SORT_MEM?

No, this sounds very reasonable to me.  I do a similar operation on one of my 
systems as part of a nightly data transformation for reporting.  Since I 
haven't had to do those on tables over 150,000 rows, I haven't seen the kind 
of RAM usage you experience.

> Below is an example of another real-world query from the same
> reporting system that benefits from a sort_mem over 32M.
> Explain analyze (below) shows a 40% improvement by having
> the sort fit in memory.

Cool!  That's a perfect example of sizing sort_mem for the query.   Mind if I 
steal it for an article at some point?

-- 
Josh Berkus
Aglio Database Solutions
San Francisco




view thread (14+ messages)  latest in thread

Message-ID: <200304110915.15060.josh@agliodbs.com>
Permalink:  ../200304110915.15060.josh@agliodbs.com/
Also on:    postgresql.org/message-id/200304110915.15060.josh@agliodbs.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-performance@postgresql.org
  Cc: josh@agliodbs.com, ron@intervideo.com, cobalt@bearfruit.org
  Subject: Re: Caching (was Re: choosing the right platform)
  In-Reply-To: <200304110915.15060.josh@agliodbs.com>
  References: <POEDIPIPKGJJLDNIEMBEOEAACJAA.ron@intervideo.com> <200304110915.15060.josh@agliodbs.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox