Received: from maia.hub.org (unknown [200.46.204.183]) by mail.postgresql.org (Postfix) with ESMTP id 63415634AF4 for ; Tue, 9 Mar 2010 12:13:57 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 02672-03-6 for ; Tue, 9 Mar 2010 16:13:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-bw0-f216.google.com (mail-bw0-f216.google.com [209.85.218.216]) by mail.postgresql.org (Postfix) with ESMTP id 3C4A4635CA2 for ; Tue, 9 Mar 2010 12:13:36 -0400 (AST) Received: by bwz8 with SMTP id 8so3302960bwz.3 for ; Tue, 09 Mar 2010 08:13:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=dmcEQE2m4Ux3ZCAMfxvVi+iaPTquwSIPsyeG6cZqJDU=; b=WD32q74iW8uT4HkzRlS22qTIxSKsVoYI4eSDZ3tbVBU38Z73kyfW341PesjuPwpVnn +yvl96+JEDTfaeqDU9QEInv74BVKZ6T++1gYmhB+krfHpmCPhqvOrlIETeUsLOtnKAP1 ZEr66D+w/tjbBLWYh4V8/DVi7jFwfI2xwQJ3g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Svg+UgqsO8Xw8nbmkKlWnmUq42Srfhmo2SMu5U4hqvm7FMzTJ+Mnz24bRZuoaFMOzF DNRs+zLWbu9KWmR+2NgvDuGGsUtVR0dYGCJVP6B8yXBshUzXlCy8mnJwjz274VPPlmvc bzFtJo7YKPYRsLOT08hzbunHkrtDPIVChqv/U= MIME-Version: 1.0 Received: by 10.204.138.79 with SMTP id z15mr2723708bkt.167.1268151214543; Tue, 09 Mar 2010 08:13:34 -0800 (PST) In-Reply-To: References: Date: Tue, 9 Mar 2010 10:13:30 -0600 Message-ID: Subject: Re: Extracting superlatives - SQL design philosophy From: Dave Crooke To: Merlin Moncure Cc: pgsql-performance Content-Type: multipart/alternative; boundary=00151747bff2aa14a80481607650 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.598 tagged_above=-10 required=5 tests=BAYES_00=-2.599, HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 201003/95 X-Sequence-Number: 37920 --00151747bff2aa14a80481607650 Content-Type: text/plain; charset=ISO-8859-1 Cool trick .... I didn't realise you could do this at the SQL level without a custom max() written in C. What I ended up doing for my app is just going with straight SQL that generates the "key" tuples with a SELECT DISTINCT, and then has a dependent subquery that does a very small index scan to pull the data for each row (I care somewhat about portability). In order to make this perform, I created a second index on the raw data table that has the columns tupled in the order I need for this rollup, which allows PG to do a fairly efficient index range scan. I had been trying to avoid using the disk space to carry this 2nd index, since it is only needed for the bulk rollup, and I then reliased I only have to keep it on the current day's partition, and I can drop it once that partition's data has been aggregated (the insert overhead of the index isn't as much of a concern). Alternatively, I could have lived without the index by sharding the raw data right down to the rollup intervals, which would mean that rollups are effective as a full table scan anyway, but I didn't want to do that as it would make real-time data extration queries slower if they had to go across 10-20 tables. Thanks everyone for the insights Cheers Dave On Tue, Mar 9, 2010 at 6:46 AM, Merlin Moncure wrote: > On Wed, Feb 24, 2010 at 4:31 PM, Dave Crooke wrote: > > This is a generic SQL issue and not PG specific, but I'd like to get > > an opinion from this list. > > > > Consider the following data: > > > > # \d bar > > Table "public.bar" > > Column | Type | Modifiers > > --------+-----------------------------+----------- > > city | character varying(255) | > > temp | integer | > > date | timestamp without time zone | > > > > # select * from bar order by city, date; > > city | temp | date > > -----------+------+--------------------- > > Austin | 75 | 2010-02-21 15:00:00 > > Austin | 35 | 2010-02-23 15:00:00 > > Edinburgh | 42 | 2010-02-23 15:00:00 > > New York | 56 | 2010-02-23 15:00:00 > > New York | 78 | 2010-06-23 15:00:00 > > (5 rows) > > > > If you want the highest recorded temperature for a city, that's easy > > to do, since the selection criteria works on the same column that we > > are extracing: > > > > # select city, max(temp) from bar group by city order by 1; > > city | max > > -----------+----- > > Austin | 75 > > Edinburgh | 42 > > New York | 78 > > (3 rows) > > > > > > However there is (AFAIK) no simple way in plain SQL to write a query > > that performs such an aggregation where the aggregation criteria is on > > one column and you want to return another, e.g. adding the the *date > > of* that highest temperature to the output above, or doing a query to > > get the most recent temperature reading for each city. > > > > What I'd like to do is something like the below (and I'm inventing > > mock syntax here, the following is not valid SQL): > > > > -- Ugly implicit syntax but no worse than an Oracle outer join ;-) > > select city, temp, date from bar where date=max(date) group by city, > > temp order by city; > > > > or perhaps > > > > -- More explicit > > select aggregate_using(max(date), city, temp, date) from bar group by > > city, temp order by city; > > > > Both of the above, if they existed, would be a single data access > > followed by and sort-merge. > > > > The only way I know how to do it involves doing two accesses to the data, > e.g. > > > > # select city, temp, date from bar a where date=(select max(b.date) > > from bar b where a.city=b.city) order by 1; > > city | temp | date > > -----------+------+--------------------- > > Austin | 35 | 2010-02-23 15:00:00 > > Edinburgh | 42 | 2010-02-23 15:00:00 > > New York | 78 | 2010-06-23 15:00:00 > > (3 rows) > > > > > > # explain select * from bar a where date=(select max(b.date) from bar > > b where a.city=b.city) order by 1; > > QUERY PLAN > > > -------------------------------------------------------------------------- > > Sort (cost=1658.86..1658.87 rows=1 width=528) > > Sort Key: a.city > > -> Seq Scan on bar a (cost=0.00..1658.85 rows=1 width=528) > > Filter: (date = (subplan)) > > SubPlan > > -> Aggregate (cost=11.76..11.77 rows=1 width=8) > > -> Seq Scan on bar b (cost=0.00..11.75 rows=1 > > width=8) -- would be an index lookup in a real scenario > > Filter: (($0)::text = (city)::text) > > Another cool way to do this is via a custom aggregate: > create function maxdata(data, data) returns data as > $$ > select case when ($1).date > ($2).date then $1 else $2 end; > $$ language sql; > > create aggregate maxdata(data) > ( > sfunc=maxdata, > stype=data > ); > > select (d).* from > ( > select maxdata(data) as d from data group by city > ); > > It does it in a single pass. Where this approach can pay dividends is > when you have a very complicated 'max'-ing criteria to justify the > verbosity of creating the aggregate. If you are not doing the whole > table, the self join is often faster. I'm surprised custom aggregates > aren't used more...they seem very clean and neat to me. > > merlin > --00151747bff2aa14a80481607650 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cool trick .... I didn't realise you could do this at the SQL level wit= hout a custom max() written in C.


What I ended up doing for my a= pp is just going with straight SQL that generates the "key" tuple= s with a SELECT DISTINCT, and then has a dependent subquery that does a ver= y small index scan to pull the data for each row (I care somewhat about por= tability). In order to make this perform, I created a second index on the r= aw data table that has the columns tupled in the order I need for this roll= up, which allows PG to do a fairly efficient index range scan.

I had been trying to avoid using the disk space to carry this 2nd index= , since it is only needed for the bulk rollup, and I then reliased I only h= ave to keep it on the current day's partition, and I can drop it once t= hat partition's data has been aggregated (the insert overhead of the in= dex isn't as much of a concern).

Alternatively, I could have lived without the index by sharding the raw= data right down to the rollup intervals, which would mean that rollups are= effective as a full table scan anyway, but I didn't want to do that as= it would make real-time data extration queries slower if they had to go ac= ross 10-20 tables.


Thanks everyone for the insights

Cheers
Dave

On Tue, Mar 9, 2010 at 6:46 AM, Merlin Moncure <mmoncure@gmail.com> wrote:
On Wed, Feb 24, 2010 at 4:31 PM, Dave Crooke <dcrooke@gmail.com> wrote:
> This is a generic SQL issue an= d not PG specific, but I'd like to get
> an opinion from this list.
>
> Consider the following data:
>
> # \d bar
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Table "public.bar&q= uot;
> =A0Column |=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Type=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 | Modifiers
> --------+-----------------------------+-----------
> =A0city=A0=A0 | character varying(255)=A0=A0=A0=A0=A0 |
> =A0temp=A0=A0 | integer=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 |
> =A0date=A0=A0 | timestamp without time zone |
>
> # select * from bar order by city, date;
> =A0 city =A0 =A0| temp | =A0 =A0 =A0 =A0date
> -----------+------+---------------------
> =A0Austin =A0 =A0| =A0 75 | 2010-02-21 15:00:00
> =A0Austin =A0 =A0| =A0 35 | 2010-02-23 15:00:00
> =A0Edinburgh | =A0 42 | 2010-02-23 15:00:00
> =A0New York =A0| =A0 56 | 2010-02-23 15:00:00
> =A0New York =A0| =A0 78 | 2010-06-23 15:00:00
> (5 rows)
>
> If you want the highest recorded temperature for a city, that's ea= sy
> to do, since the selection criteria works on the same column that we > are extracing:
>
> # select city, max(temp) from bar group by city order by 1;
> =A0 city =A0 =A0| max
> -----------+-----
> =A0Austin =A0 =A0| =A075
> =A0Edinburgh | =A042
> =A0New York =A0| =A078
> (3 rows)
>
>
> However there is (AFAIK) no simple way in plain SQL to write a query > that performs such an aggregation where the aggregation criteria is on=
> one column and you want to return another, e.g. adding the the *date > of* that highest temperature to the output above, or doing a query to<= br> > get the most recent temperature reading for each city.
>
> What I'd like to do is something like the below (and I'm inven= ting
> mock syntax here, the following is not valid SQL):
>
> -- Ugly implicit syntax but no worse than an Oracle outer join ;-)
> select city, temp, date from bar where date=3Dmax(date) group by city,=
> temp order by city;
>
> or perhaps
>
> -- More explicit
> select aggregate_using(max(date), city, temp, date) from bar group by<= br> > city, temp order by city;
>
> Both of the above, if they existed, would be a single data access
> followed by and sort-merge.
>
> The only way I know how to do it involves doing two accesses to the da= ta, e.g.
>
> # select city, temp, date from bar a where date=3D(select max(b.date)<= br> > from bar b where a.city=3Db.city) order by 1;
> =A0 city =A0 =A0| temp | =A0 =A0 =A0 =A0date
> -----------+------+---------------------
> =A0Austin =A0 =A0| =A0 35 | 2010-02-23 15:00:00
> =A0Edinburgh | =A0 42 | 2010-02-23 15:00:00
> =A0New York =A0| =A0 78 | 2010-06-23 15:00:00
> (3 rows)
>
>
> # explain select * from bar a where date=3D(select max(b.date) from ba= r
> b where a.city=3Db.city) order by 1;
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0QUERY P= LAN
> ----------------------------------------------------------------------= ----
> =A0Sort =A0(cost=3D1658.86..1658.87 rows=3D1 width=3D528)
> =A0 Sort Key: a.city
> =A0 -> =A0Seq Scan on bar a =A0(cost=3D0.00..1658.85 rows=3D1 width= =3D528)
> =A0 =A0 =A0 =A0 Filter: (date =3D (subplan))
> =A0 =A0 =A0 =A0 SubPlan
> =A0 =A0 =A0 =A0 =A0 -> =A0Aggregate =A0(cost=3D11.76..11.77 rows=3D= 1 width=3D8)
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 -> =A0Seq Scan on bar b =A0(cost=3D= 0.00..11.75 rows=3D1
> width=3D8) =A0 =A0 -- would be an index lookup in a real scenario
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Filter: (($0)::text =3D (c= ity)::text)

Another cool way to do this is via a custom aggregate:
create function maxdata(data, data) returns data as
$$
=A0select case when ($1).date > ($2).date then $1 else $2 end;
$$ language sql;

create aggregate maxdata(data)
(
=A0 sfunc=3Dmaxdata,
=A0 stype=3Ddata
);

select (d).* from
(
=A0select maxdata(data) as d from data group by city
);

It does it in a single pass. =A0Where this approach can pay dividends is when you have a very complicated 'max'-ing criteria to justify the<= br> verbosity of creating the aggregate. =A0If you are not doing the whole
table, the self join is often faster. =A0I'm surprised custom aggregate= s
aren't used more...they seem very clean and neat to me.

merlin

--00151747bff2aa14a80481607650--