pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: John A Meinel <john@arbash-meinel.com>
To: Ken <ken@upfactor.com>
Cc: Richard Huxton <dev@archonet.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Help with tuning this query (with explain analyze finally)
Date: Fri, 04 Mar 2005 13:56:12 -0600
Message-ID: <4228BD5C.9020800@arbash-meinel.com> (raw)
In-Reply-To: <42289347.7040607@arbash-meinel.com>
References: <001001c51ef4$3849eeb0$6a01a8c0@KEN>
	<200503022152.13211.josh@agliodbs.com>
	<004601c51fbe$828bbb50$6a01a8c0@KEN>
	<200503030935.14537.josh@agliodbs.com>
	<001d01c5204a$b440edf0$6a01a8c0@KEN>
	<4227AA36.2080700@arbash-meinel.com>
	<000e01c5207a$1f668ac0$6a01a8c0@KEN>
	<42288529.7000803@archonet.com>
	<000e01c520d8$4f899a90$780ba8c0@javadude>
	<42289347.7040607@arbash-meinel.com>

John Arbash Meinel wrote:

> Ken wrote:
>
>> Richard,
>>
>> What do you mean by summary table?  Basically a cache of the query
>> into a table with replicated column names of all the joins?  I'd
>> probably have to whipe out the table every minute and re-insert the
>> data for each carrier in the system.  I'm not sure how expensive this
>> operation would be, but I'm guessing it would be fairly heavy-weight.
>> And maintaince would be a lot harder because of the duplicated
>> columns, making refactorings on the database more error-prone.  Am I
>> understanding your suggestion correctly? Please correct me if I am.
>>
>>> Can you turn the problem around? Calculate what you want for all
>>> users (once every 60 seconds) and stuff those results into a summary
>>> table. Then let the users query the summary table as often as they
>>> like (with the understanding that the figures aren't going to update
>>> any faster than once a minute)
>>
>>
> It's the same idea of a materialized view, or possibly just a lazy cache.
>
...

> This unrolls all of the work, a table which should be really fast to
> query. If this query takes less than 10s to generate, than just have a
> service run it every 60s. I think for refreshing, it is actually faster
> to drop the table and recreate it, rather than deleteing the entries.
> Dropping also has the advantage that if you ever add more rows to s or
> ss, then the table automatically gets the new entries.
>
Just as a small update. If completely regenerating the cache takes to 
long, the other way to do it, is to create insert and update triggers on 
s and ss, such that as they change, they also update the cachedview table.

Something like

CREATE TRIGGER on_ss_ins AFTER INSERT ON ss FOR EACH ROW EXECUTE
    INSERT INTO cached_view SELECT p.id as person_id, s.*, ss.* FROM 
<the big stuff> WHERE s.id = NEW.id;

This runs the same query, but notice that the WHERE means it only allows 
the new row. So this query should run fast. It is a little bit of 
overhead on each of your inserts, but it should keep the cache 
up-to-date. With something like this, I would have the final client 
query still include the date restriction, since you accumulate older 
rows into the cached view. But you can run a daily process that prunes 
out everything older than 31 days, which keeps the cachedview from 
getting really large.

John
=:->

Attachments:

  signature.asc (application/pgp-signature, 252B) download

view thread (26+ messages)  latest in thread

Message-ID: <4228BD5C.9020800@arbash-meinel.com>
Permalink:  ../4228BD5C.9020800@arbash-meinel.com/
Also on:    postgresql.org/message-id/4228BD5C.9020800@arbash-meinel.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: john@arbash-meinel.com, ken@upfactor.com, dev@archonet.com
  Subject: Re: Help with tuning this query (with explain analyze finally)
  In-Reply-To: <4228BD5C.9020800@arbash-meinel.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox